Team Ownership - Who Owns Production

Christopher Slowe at UXDX Europe. Video: https://youtu.be/OWUHWGRM3z4

Readable transcript: edited from the recording's captions for readability (fillers and false starts removed, punctuation and section headings added). Wording is the speaker's own. Timestamps are positions in the video. Names marked [?] could not be verified against the audio.

Opening band: production ownership as a comedy

[00:00:00] Hi everybody, thanks for joining me, top of the morning on the first day. I think that makes me the opening band, so as the opening band I hope you find me entertaining but ultimately forgettable compared to the main event. I'm going to talk about production ownership, which you could really think of as, in some ways, a tragedy. But that's kind of a depressing way to start the day, so I'm going to instead go with production ownership, a comedy, as in how we could have done things.

[00:00:26] This is a topic really near to my heart. I've been at Reddit for most of its history, I kind of started three months in. Quick about me: founding engineer, as was already introduced, left and kind of came back, and I had a chance to kind of undo and then redo a bunch of mistakes, which is a delightful experience. When I say I have a lot of opinions, it means I've seen a lot of really broken things, most of which I broke. Breaking things builds character, and building character is expensive. So my goal for today is to say we're all in this together. That's true as much of a developer group as of this talk, that we're all in here at 9:30 in the morning on a Monday.

What Reddit is, and how it grew

[00:01:09] First, a really brief history. Start with, what is Reddit? Okay, kind of curious, who here uses Reddit? Very cool. Who here considers themselves a Reddit user? It's always a different group. I don't know what that is, it's like little lurkers. Reddit is a network of communities, and it's basically a platform where people can discuss their hobbies and interests with other passionate users, and really the cornerstone of Reddit is about conversation.

[00:01:44] The thing about Reddit is we're kind of a big deal, kind of a big site. I usually refer to us as the biggest little site no one's ever heard of, or at least pretends not to have heard of. We've got 330 million monthly active users, we do millions of votes per day. It's kind of a big deal. But let's talk a little bit about how we got there.

[00:02:02] How did we get started? Really far back in the day, the thing about Reddit that took us the entire history to realize is that Reddit is an example of a product that kind of out of the box just had this wonderful product market fit. So we opened up with — there was always this kind of three-month doubling cycle during the early years, which really meant that it kind of focused the mind on what we were doing at any given time, which is to survive the next doubling period. We got acquired really early and we were basically a wholly owned subsidiary for most of the first bunch of years, and this left an interesting mark on the company.

[00:02:33] So here is our employee growth as a function of time. I actually started the axis in 2008, where we started actually keeping records. This line at the floor goes all the way back over to about here, from 2005, so you can see it's a pretty gentle ramp up. If you recast this in terms of reach per developer it looks like this. This is the pageviews per month per developer, so you can see that basically everyone on this team is taking on a fairly significant load when it comes to what they're doing.

[00:03:04] The side effect of this is that everyone has to know what they're doing. If you don't know what you're doing, you're kind of screwed. In the early years it's like, managers? What are managers? You just kind of keep the site going, that's the point. We actually didn't have a PM until about 2014, 2015, just because it was mostly developer focused. This is also, I think, at a time when there's this myth of the 10x engineer, and really, fundamentally, 10x engineers and startups are very compatible. You have to kind of know what you're doing and be able to do it, or you don't really get a chance to make any headway.

Org charts, and who owns production in a startup

[00:03:37] For the more recent stuff, let's talk a little bit about org charts, which is kind of the important bit when you get a bigger team. We're now up to about 550 people in the company, about Twitter[?] engineers. Now, the hard part about this talk is being retrospective. To start with, I put myself in my shoes from 2008 and realized that if I'd seen this slide back then[?] I would feel a bit like this. You kind of have to take it with a grain of salt.

[00:03:59] So the nomenclature[?] here: this is production. In a startup, production is a block, and who owns production is the team. If you want, you can actually break that down into individual developers. Everyone knows their turf, so you end up with this situation where, if you go kind of from back end into front end — and this is kind of like the products they run across, well, it's only kind of one product — you get a bunch of devs and they're mostly overlapping. And you'll notice it's mostly pairwise, so you have an ability at any given time to work with somebody else, but usually you're kind of working alone, kind of drilling on your own area.

[00:04:30] Everyone knows what they're doing, all the gaps are filled, or at least the gaps you care about are filled, and everyone in this picture probably has a pretty complete idea of what production looks like and what everyone else is doing, which is kind of an important part. And the last thing is, for a startup there really only needs to be one product. Maybe at some point during the startup you decide to pivot into a different product, but it's still kind of one product.

[00:04:50] As you grow, the picture starts to get complicated, because you start getting redundancies like this. You start filling in the gaps, you start adding more people, you also have people who are less experienced, a little smaller circles, people adjust in their position, and you end up in this position where you get this kind of overlapping whip[?]. This is about the point at which you have to start thinking about teams, because you can't just have a monolithic team running at this point.

[00:05:13] Usually the canonical first thing you have is the split, so this front-end and back-end shift that happens in place, and in the middle we can put some scary air quotes around the API in the middle. But now when you actually have teams you really need to start thinking about an API, like an actual way that your front end and back end communicate, that's official, that's like a contract. You have a dedicated front-end team, it's mostly working in mobile clients or web or whatever[?] user technologies, and back-end teams generally working in whatever your stack is built in, in our case it was Python. This is also to some extent a function of just, you need to hire a manager, and it's easier to hire a manager to run an entire section of the stack rather than to basically choose them to run a product.

[00:05:52] As time goes on, the next thing that usually happens is you start splitting up your products. You build new products, you start building out teams, each one of these verticals here has its own bit of ownership. And oftentimes you'll see the same thing in the back end, but it usually happens in the front end first, where you have the canonical split that used to happen, which was you have the mobile team that would be in charge of all the mobile stuff. Now, of course, the mobile team is really your main group.

Where the problems start

[00:06:16] So things get more complicated. Of course you start getting more boxes, things start to split up. Let's talk about where the problems start to come into play, because this is actually, I think, the core of production ownership: what problems are you trying to solve at any given time?

[00:06:32] Let's take this simple setup. This, by the way, is usually called the layer cake. It's kind of a mix of vertical and horizontal teams. Well, first of all, you kind of have this API in the middle and it's really kind of unclear who owns it, because you've got overlapping considerations: the client teams use it, but the back-end teams kind of make it, so they all kind of have an opinion on it. So that's where the first kind of interface that requires a little bit of support comes into play.

[00:06:56] You've also got this kind of contention between product teams. You have the "what did you do?" kind of argument that happens between two product teams that have overlapping code bases. The one that I think happens more often is the "not it" — there's some sort of bug that happens, it's an interface between two teams, like, those guys did it, we don't have time to do it, it's not on our roadmap right now. So, mind the gap. And then last of all there is the full-stack drilling that happens, of, no really, I know how to write back-end code. And of course front-end teams and client teams want to have control over their destiny and therefore want to get involved in the back-end stack.

[00:07:36] The point that I wanted to get across first off is this is also all kind of the same problem. This kind of drilling that happens with full-stack development is really about this kind of horizontal tension. It's similar to two product teams that are fighting over what part of the product they run. So to some extent, all of these problems about who owns production are really a question of how do we get teams to interface efficiently. You've just got a bunch of people; how do you get them to coordinate?

Bugs, glue teams and guilds

[00:08:04] Well, here are some ideas and some solutions that we've seen work. Let's start off with bugs. Most of what happens when you have arguments about ownership of production is, it's really about where do you land the bugs. There are always bugs. In some cases the bugs are really clear-cut: P0 back-end bugs, probably you're down or you're in some degraded state. It's very clear that in that situation there needs to be somebody who's going to be paged and held accountable. What's a little less clear is that there's an entire class of front-end bugs that fall into the same category. If your checkout flow is broken because somebody pushed a bug that means now your JavaScript doesn't make the button work, it's probably a P0, that probably means somebody should get paged. It's still a situation where you actually need escalation.

[00:08:50] So the important part here is to start off with establishing SLAs around bugs. If you all agree on the level of what a bug is, you can all agree that actually somebody should get paged. What we started doing is we actually centralized all of our bug triage. This is across all of the product teams, across all the back-end teams. We make sure that everyone's part of a rotation to triage bugs, which means that every group starts to get familiar with other teams' bugs and kind of knows where they should land, or at least you can start to draw some lines that are more clear.

[00:09:18] Another thing you can do is to actually start adding teams that act as explicit glue in here. If you think about some of the things I said that were wrong, you've got these teams that don't necessarily have to coordinate on the client side, whereas in the back end it's much more of a single block, at least early on, for many of the early years. So you can just add a team in here. You can think of this as a support team that starts to build out what is the underlying infrastructure and the surface upon which people build products. We have this team, we call it web infra, and correspondingly[?] we call our back-end team the infrastructure team, as most teams do. The idea here is that even on a front-end stack there's kind of a back of the front end, there's a set of common libraries and modules that everyone uses, and you need a team that's able to kind of go in and start supporting that and make sure that you can build products on top of it efficiently.

[00:10:08] Another thing we can do is actually look at the individual composition of a team. So let's do our zoom-and-enhance magic. Here's what one of these stacks looks like. Generally what you have is a team which consists of a single director who's kind of running a bunch of engineering pods, almost, and these engineering pods are naturally kind of these inter-stack groups. So you have an EM who's running the team, you have a web engineer, you have an iOS engineer, an Android engineer, and kind of a back end, and of course you probably have a product manager and a designer and a whole set of things, but I'm kind of keeping it right now to engineering, just to be simple.

[00:10:45] The hard part with this kind of setup is that now what you've got is people distributed across teams of the same discipline who can't really naturally coordinate. There's no way to necessarily keep synced. So the trick we've employed is one that we actually stole from Spotify, which is the notion of guilds. A guild is kind of, you think of it as effectively a cross-cutting VT, like a virtualized team, and they're not so much teams as more notional ways to coordinate.

[00:11:14] In this case we can actually say there's a back-end guild, which is kind of semi-voluntary, semi just kind of show up and talk to your fellow back-end people and coordinate, and we have a front-end guild and mobile guilds. This means that everybody gets to complain about the stuff that bothers them the most about other teams. For example, back-end guilds, universally, it's like, oh my god, you won't believe the failure we just saw in production last week. Front-end guild, the one I hear the most is, two more JavaScript frameworks came up during the time of this meeting — actually, wait, no, three. And mobile guilds, I have to say, being kind of a back-end chauvinist, I'll actually warn you, those back-end guys are blaming us again for retries, is the one that I think I've heard the most.

Does it work for production? The Node shim story

[00:11:54] But the question is, does this now work for production? We solved some of the product contention. How did this work for the actual drilling into the back end? Well, first off, bugs are everyone's problem, that's a pretty easy one. If you can't find an owner of a bug, it's literally whichever team seems to be the least loaded, or whoever basically gets selected randomly from a group.

[00:12:19] When you talk about clients and servers, here's where things get complicated, because we can't actually employ guilds. Guilds are intended to be single discipline, so you can't start mixing back end and front end. There's different tooling, there's different languages, there's some contention about exactly what normal behaviors are, and so it's kind of hard to do that. So the thing that we thought about was, okay, what about this language gap? Can we do something with the language gap? In our case you've got generally Python on the back end, JavaScript on the front end, we have Objective-C in the clients. So what about cross-stack languages? This is generally where Node comes into play. Node's becoming much more common, especially in back-end stacks, as time goes on.

[00:13:00] Here's a short story of our foray into using Node. We just recently did a giant redesign of Reddit. This is a small change to the way the site works. Of course it launched perfectly, everything was totally fine, nothing ever went wrong. But here, hypothetically, is something that could have gone wrong. The idea was, as we're doing a front-end rebuild, let's see if we can actually increase developer velocity of the client engineers. In this case you have this problem if you're basically building out a new API on top of an existing API, and so, well, what about in the inner stack, let's just add a shim.

[00:13:32] So what we did is we built a shim sitting on top of our API. The idea here is not so much that we want to develop an entirely new API, but we want to add flexibility for the client engineers to actually develop their own API as they need it, and kind of figure out what the interplay is with our current API and what they write, and then the intention is long term we rebuild our API around this new paradigm. So a shim is born. We started to do it in Node, because again, client engineers are already working in JavaScript, so it was giving them a language they were used to dealing with, so, developer velocity.

[00:14:05] In this case we have a monolith that's kind of sitting at our core, and we also have other services that are kind of running around, so the idea is that when a request comes in — oh, yeah. As we're actually launching this, we want to make sure that as we're rolling this out, it's kind of done in a very deliberate fashion. So we were doing a whole bunch of beta testing with our new stack, we were making sure that everything could actually load up. The tricky part, of course, of building a new thing at scale means that any small problems are going to be magnified pretty rapidly, so we decided to do kind of a slow ramp-up on this one, and we launched it at two percent.

[00:14:39] Here's what the request graph looked like on our back end. That is not two percent growth, that's about a hundred percent growth. So this is one of those, what on earth is happening. The axis, if you can't see it, is in thousands of operations per second. So two percent causes about a doubling in our back end, which seems to be a problem.

[00:14:58] The way that we set this up was we kind of put this on top of all our back-end services, and a request comes in and it goes to this shim layer, and the first thing it says is, okay, who has to handle this? And it has a whole bunch of choices. You can think of it as the newer version of this that we have in places where we started building out GraphQL. The underlying idea of GraphQL is, I have a request, I don't really care about the response, but I have to figure out who should actually handle it downstream.

[00:15:23] In this case we had a whole bunch of decision points. We could actually ask the monolith, we could ask this other service called Gateway, we could start to kind of fiddle with it. And Gateway actually had the same kind of process going on, and there's this extra little arrow. So what's happening here? We have one request coming in, it turns into multiple requests going to our back end, that goes to our monolith, and basically DDoSes itself. So we were seeing cases where one request could turn into as many as 15 or 20 requests going to our back end. This is by definition bad, especially when we're talking about scaling up. This is where we got this fan-out, an extra factor of 25 roughly in the traffic load. So much cursing ensues at this point when we discover this is a problem.

Tooling, tracing, and what it cost the back-end teams

[00:16:07] So what do we actually do in this situation? We've got a team that is not really used to thinking in these ways. Well, fortunately we actually had a whole bunch of systems in place, we just hadn't universally deployed it. One of my favorite quotes is that the future is here, it's just not universally distributed. In this case we actually had tooling in place, we just never actually pushed it up into the Node stack. So there's this great tool called Zipkin. Zipkin does tracing, and tracing means for a given request you can actually look through the entire stack and figure out what sub-requests come from it, how it gets routed across all of your services.

[00:16:41] Of course, part of this is adding communication, so we actually had a chance to collaborate across all of our teams. It seems like a very basic idea, making sure your teams actually talk to one another. In this case the team had a goal and a way to measure it, and we could actually prioritize whether or not this is the right thing to do. This communication meant that we could surface to front-end teams, hey look, you're actually slamming the back end, and actually have them trace it down, with some tooling that was very straightforward to use. So a couple of deploys took away most of this. This is one of those where, when you actually figure out what's going on, it goes away very rapidly. So we spent a whole bunch of iteration time on that.

[00:17:19] So what do we learn here, is where I'm trying to go. What I'm saying right now is that as you're drilling, you can think of this as, you kind of squish the back-end teams. What we've done is we've increased the total surface area where you have collaboration between front-end and back-end engineers, and you also have, by virtue of the fact that we now start adding into production technologies that client engineers are used to developing in, it naturally means that back-end engineers aren't as familiar with them. And so you kind of jam into the stack this weird technology that no one really understands.

[00:17:56] So the back-end teams in this way become sort of a giant new single point of failure. This generally also kills motivation on both sides. You end up in a situation where you're kind of constantly firefighting, and now on top of that you're firefighting against technologies you don't understand as well. There's also lots of stuff going into production right now: you have not just client libraries you're shipping up to the client, you also have client libraries you're shipping to production servers, and you have to figure out how to scale that stuff up with tooling that might not be as clear. And don't get me started on JavaScript bundling, which I think is a tool of the devil right now, or at least the tools of the devil, as there is a new one every two weeks.

[00:18:35] So the important part here, for the back-end teams and for any team in a situation where you're constantly firefighting: stop the bleeding first is the important part, and then scale up the teams. Quick plug for this: there's a great book by Will Larson called An Elegant Puzzle, which is really a good description of how to scale up infrastructure teams at scale, and he has his basic tenets of, when you have a problem where you're hitting scale, sometimes the correct solution is in fact to hire. If you can't get ahead of the curve you're probably stuck and hiring is the only possible solution.

Takeaways

[00:19:12] So what we have done as the next iteration of this whole process, besides reassessing our desire to use Node in the deep stack, is actually to add a layer. When you have an unclear owner in the situation, add some owners. In this case we were adding an extra layer. The intention here is that we have a set of SREs whose entire job is to be embedded in the actual product teams and to act as the interface. This also adds a nice ability of us forming a guild effectively across the entire company, where they're able to coordinate, they're able to do things like share and iterate on instrumentation[?] and actually figure out how best to use the tooling that they have and build out new tooling at the same time. This is currently in kind of a slow roll, and so I'll let you know how that breaks, probably in a talk in this forum next year.

[00:19:58] But let's just go through overall takeaways, and the one thing you should have from this talk is a set of one, apparently. So, do QA on your slides, an important thing. But in this case, let's say that communication is definitely key, and especially that specialization is no excuse to stop coordinating. Part of the reason why we were actually able to deal with the mess that we had made with our shim was because we just had teams coordinating and actually talking about what's going on and drilling into it.

[00:20:28] I'd also say that all of the pictures I drew of production ownership were these kind of weird horizontal-vertical flips that happen. That's normal and healthy. In fact there should be some tension in the stack. I've heard of companies who will periodically just decide to switch from horizontal to vertical, and the nice thing about doing that is you get a chance to kind of have this implicit-to-explicit cross-pollination happening. So first of all you have your production teams and your product stacks, and you switch to, oh yeah, you know what, everybody who's doing JavaScript flips into a team, and then you work on that for a while. So it gives a chance to reconsolidate.

[00:20:58] The last thing here really is that vertical teams need glue. No matter what you're doing, you really can't have completely separated units across this chart. There has to be overlap. In the early startup days you're talking about overlap between people and actual developer life cycles. In the later days you actually need to have either explicit people acting as that glue, or you need to have ways to collaborate, or you need a team that kind of catches them all together. So with that, I'd like to say thank you.