Getting Your Pipelines Flowing

Ross McKinley at UXDX Europe. Video: https://youtu.be/k0FO18-5xog

Readable transcript: edited from the recording's captions for readability (fillers and false starts removed, punctuation and section headings added). Wording is the speaker's own. Timestamps are positions in the video. Names marked [?] could not be verified against the audio.

Time to player

[00:00:08] So before I begin, can I have a quick show of hands of who's a fan of the Star Wars franchise? Excellent. So I work at DICE. DICE is a games company and one of the games that we make is Star Wars Battlefront 2. So I'm going to go into a little bit of a story about how we shipped that game, and there'll be some more stuff as well soon.

[00:00:45] So I'm just going to cover what is it that we're going to learn in this talk. In EA we've got this concept called time to player, so I'll talk about that. We'll talk about how to ship a game. We'll look a little bit into the value stream map that we have in the company. And then one of the things that I really like to do is front-load a bunch of the effort of making games to make things go faster.

[00:01:15] So what is time to player? We've got this one idea that might be a prototype, or it might be a new feature, or it might be some new gun or a new map or some kind of cool thing that we want to put in the game. And what we do with that is, we've got a whole team of development people, and they might be QA testers or rendering engineers or physics people or any of the different disciplines in the company. And then we've got this sort of prayer area, where we sort of hope that everything works, and we polish it and we make it a little bit better, and then everything breaks. This is where the bulk of the polishing and getting it ready to ship to the players happens. And then we've got it in a player's hand, they're sitting on the couch and they're able to play the game.

[00:02:22] So time to player is this process. It's the time to go from the brain into the computers and making art assets and putting it onto a console. And this is really important to EA, so what players want is more content delivered to them faster. It's a changing industry, away from this big box product game and then in a couple of years' time it'll be another game. No one really wants that any more. People want these live service games where there's an update this week that has a new map in it, or there's an update in two weeks' time which has a new gun or a new hat or a new place to play, or maybe it's some extra balancing or something else in the game.

How the patch process used to work

[00:03:14] So next I'm going to talk about the patch process, which is how we got from this box product of Star Wars Battlefront 2 and brought it into this weekly patch release cycle. So let's say that we've got this big release and it's on a disc and we've shipped it to all the stores, and everyone can take that disc and put it in their PlayStation or Xbox or whatever. So let's just pretend that all that work has happened and we've got the full game.

[00:04:02] So the question then is, we're working on this second thing, and the second thing might be a new map or it might be whatever other feature it is that we want to have coming down the line, like maybe it's some new content. So normally what happens is, we've got our disc and we've got that version sort of ready to go, and it sits in the shops and all that, and our next version is in development and it's sort of just being made, so we don't really have it finalized, but it's in that development phase. So each of these red lines represents a totally distinct area in version control for us. We call them different branches, so a totally different branch for each red line.

[00:04:58] But then we realize, oh hey, that version that we shipped on disc, maybe it's got some problem. Maybe there's too many people who are not able to get past an enemy in one area of the game, or it's too challenging and it's not fun, and we'd have lots of metrics to back this up. So we would say, okay, we're going to have to do a hotfix to make people be able to play the game, while we're still working on all this content here that we don't really have ready to ship yet, like maybe that's scheduled for a couple of months down the line. So we make that fix and we ship it to release, so now the version that people are playing on their consoles is this new version where you're able to actually get past that level, and everything's looking pretty good.

[00:05:55] And then we realize, oh hey, that fix that we needed, we now need to take that into our new content that we're making, because we need to have all those changes that are on people's console, we also want to have them in our next release. We don't want the case where we ship one thing and then we ship version 2 and it doesn't have that fix that we had. So okay, let's put that into version 2. And this is a manual process: we do some work here and then we have to redo it over here, and some other things have changed, so okay.

[00:06:39] All right, so let's just say that happened, and now release 2.0 is getting a little bit further along. We're in that sort of prayer area where it's doing game testing, and it's getting a little bit more ready to release. But then we find out, okay, that bug fix that we just made, maybe that has changed something else. Now too many players are playing just one special class because it's become overpowered. So maybe we want to do something else, like, okay, maybe now we need to do some rebalancing.

[00:07:23] And we have this rebalancing thing in version 2 and it's actually ready to go. The thing that we're waiting on in release 2 is maybe some more art assets, or a new character is being made, or something else that's not ready yet, but we've got this balancing update that we are happy with, and that's just a bunch of numbers on a spreadsheet which can be validated. So we say, okay, we've got this thing in version 2 and we've got this problem, so let's bring that into release one. And we say, okay, but release one is live and release 2 is not live, so we have to do this weird thing.

[00:08:11] So we do this weird thing, and then we say, okay, now we've got this release 1.2, which is partly release 1.1 and partly release 2, and we make this new branch and we do all this stuff, and okay, it kind of works, and we say, let's ship that. And now we've got a couple of updates. Our release 2.0 is now a little bit smaller, but it's pretty much ready to go as well.

[00:08:47] So then we say, okay, we wait for the right time and we sort of count our blessings, we do a last test and everything was looking good, and it's like, okay, finally we can rest, release 2.0 is at the door. But we did a lot of weird stuff to get here.

Evaluating the process

[00:09:12] So at the start of this I said each one of these red lines represents a new branch in our version control history, and there's a lot of red lines on this. We've got, for each release, we've got a red line going one way and we've got all these weird ones that are coming in from different angles. So let's evaluate this process. Maybe we aren't doing things the right way here, or maybe something weird is happening, because it certainly feels weird and it's not a very pleasant way to work.

[00:09:56] So we evaluate the process, and we say, okay, this is kind of error-prone. We've got some work going on over here and then you've got some work over there, and we sort of do this matching thing, and there's a lot of figuring it out as it happens. So it being error-prone, it's probably not the best thing. I'm going to put a big red X on that just to evaluate it. Figuring it out as it happens, it's probably not the smartest way to release games. We should probably have a plan in place, all right, so that's also not the best thing.

[00:10:43] But the complexity is not too bad. It'll work pretty well if we don't have that many releases, if we're releasing like a DLC pack that might take a year or something to come out. So the number of patches that we're making to fix those bugs is pretty low, and DLC takes a long time, so that's pretty good, that's okay.

[00:11:11] But that is so much manual effort. Every time we make one of these packs and we do this weird cobbling thing, it's like, oh, okay, there's so much stuff to keep track of that has to be done by a human. It's like, was this change made in this update, or on this update, did this one make it down to the next update, did we bring things back in? It was a pretty weird thing.

[00:11:38] And each of those red lines was a release branch, and release branches at DICE and EA are interesting, because they require a whole bunch of different things. They need a big environment to be set up, we need to have an area for playtests to happen, we have to do all this extra stuff. And it turns out that that took two weeks every time we wanted to make one of these new branches. So that's not great either.

What if it's all patches?

[00:12:20] So this is sort of what we had, and from the perspective of the legacy games industry, where we're making a box product and a couple of months later, or maybe a year later, we make some DLC pack, this is fine. It works kind of well enough, and there's enough time for you to do the whole test cycle and for it to not blow up in your face.

[00:12:53] But what if, instead of there being three patches and then another release, what happens if it's all patches? What happens if we don't want to release once every six months, we want to release every week, or we want to release every day, or we want to release a little small bit of a feature to get that feedback loop happening as quick as possible? Maybe this is not going to work any more. And then if we're looking back, it looks real bad, all right, because each branch is taking two weeks to set up.

[00:13:41] So if we look back at time to player, we're trying to get that metric of how fast can we ship features. Well, the minimum is two weeks, and that's assuming that making the content is free, and it's not free. So this is the thing that we really wanted to look at. We wanted to look at this two-week setup time and see, this is unacceptable for us, because time to player, the absolute minimum, is two weeks, and we just have to do better than that.

The goals for a new release strategy

[00:14:23] So a good friend of mine, his name is Miles McCoy, he said this a couple of years ago: the fastest way to do something is not at all. So if you just don't do something, it takes zero time, and you can pretend to not do something and it's kind of the same time, or you can engineer a way to avoid doing this big setup thing.

[00:14:52] So we decided, hey, this release strategy that we had is probably not the best. Let's just brand that as the dinosaur way of doing things, and let's come up with a new strategy. So what are the goals for that? What are our lofty goals, which maybe we might not achieve, but at least we have to try? So we wanted to minimize the manual effort, the overhead in saying, this change happened here and then this other change is here, and then maybe one of those changes is there but this one isn't compatible with that one. That's not great.

[00:15:35] Avoid having a branch per release. What if we just had one release branch, or maybe it's a set of release branches? If we have one whole thing for every release and we want to release once a week, there's 52 weeks in a year, and the lifetime of a game might be 10 years or something. I don't know what those numbers are, but they're big numbers, and if it takes two weeks every time you set up a branch, that's just unacceptable.

[00:16:13] And then the thing that we wanted to really do was enable weekly patches. We want to enable the business to ship the features that our game developers are making, that our content creators are eager to see in the hands of players, so you get that feedback, you get that whole loop closed as soon as possible. And then we wanted to have one setup for all releases.

[00:16:45] So what I described before with the red lines, that chaos, was how we used to ship games. In February we did this big sort of evaluation that I just described, and we said, let's change the way that we ship games, let's deliver value to players as fast as we can. Or at least let's try.

The static infrastructure process

[00:17:12] So we came up with this solution for that. In February we had Star Wars Battlefront 2 that was live. There's a lot of controversy around that which I don't really want to go into too much, but I think I just did. Whoops. So we had to do lots of iteration really quickly to make things right with the community, and we've successfully done that.

[00:17:50] So in about a month, something like that, we're going to be releasing Battlefield 5, and Battlefield 5 is currently under development. It's going to be shipped on a disc just like the rest of our console games, and we're going to use the same process that we used for Star Wars Battlefront 2 to enable us to deliver quickly.

[00:18:23] So I'm going to run through that process. I call it the static infrastructure process, but they're just words, it could be anything. So we've got release one [?] and that's just sitting there. Let's pretend that that's already gone. In the last diagram it had gone, now it's not, it's in development, okay, and it moves forward. And we say, okay, release 2 is like we sort of know what's within that, and 2.1 is some other thing, let's not worry too much about that.

[00:19:02] And then as time marches on, release 2 is getting a little bit more defined, it's a little bit more polished, and then it's pretty much ready to go. So we move along, and then release 2.1 is behind it, because that's the way that time happens, and release 2.2 is some other thing that's way in the past, or the future, or whatever way time works, it's in development. Let's say that.

[00:19:36] And release 2.0 is over on the one side of the screen and it's ready to go. So let's say release 2.0 is ready to ship, release 2.1 is right behind it, so it's in this almost ready to ship area. Now we've got 2.2, which is whatever that is, and release 3 is way off, that's prototypes and features and stuff that we don't really know yet.

[00:20:09] So what happens now when we want to move forward a couple of days, or weeks, or hours or whatever? Well, it just sort of moves this way. So now 2.0 has already been shipped and 2.1 is ready to ship, it's ready to go out, it's gone through the first party certification process and everyone's looking happy at it. And if we want to make a hotfix here, we just make it where it is, and the automatic process is that it will go through to 2.2, and then it'll go through to 3.0, and changes that we make at any point in the branch just go out this way.

[00:20:59] That's the whole pattern here: if I make a change anywhere, it goes from the one side of the screen back this way.

Why the release process now matches the creative process

[00:21:09] And this is a more accurate picture of what it actually looks like. We've got something that's live, we've got something that's ready to go live, we've got 2.2 which is in that polishing phase and it's not really ready to ship, but we know what's going to be in there. Release 3.0 is, we don't really know what that is. Well, we've got a good idea of, maybe it's this map, or maybe it's this feature, or maybe it's this new character, but we're not totally sure yet. And way off in prototype land we've got feature A and feature B, and we just don't really know if they'll even make it into the game. Maybe that's a dinosaur map pack, or a big head mode, or whatever sort of fun stuff that the creative people want to do.

[00:22:03] And then the way this works is, we ship 2.1, 2.2 then becomes ready to ship, and as we're doing that we're making fixes and they're going back, they're trickling back, and everything's looking good. But then what happens here? We've got these features that were prototypes, and then we've got this release 3.0 and that's doing whatever it's doing, it's more defined now, and okay, where do we go from here? Well, we take those two features and we just say, we're going to release these ones, so they go into that pipeline and they say, okay, we're going to iterate on these, and maybe the teams on those features will increase, or we'll do that whole polishing thing, and they progress into the whole picture. But then these other features come along, and everyone's got these creative ideas, and maybe that map pack or the dinosaur map pack wasn't the best idea, or maybe it was. So all of these features are something else.

[00:23:18] And what's interesting about this whole picture is that it kind of reminds me of this. We've got these ideas, and then we've got this coding phase, and then we've got this polish phase as well, and on the one side we've got the things that have actually made it out to customers, to some players, and you can go and play those. So if we apply that on the map, we've got all these ideas on the one side, we've got this sort of deep coding area where it's no longer a prototype, it's getting somewhere good, and then we've got this polish phase where we're getting everything ready to ship, and then we've got something that's ready to ship, and then we've got whatever is actually on people's consoles and people can play them.

[00:24:10] So the beauty here is that the release process, which before was a bunch of manual stuff that didn't really make sense and was hard to understand and hard to keep track of, the release process now matches the artistic process. And this was radically different for us.

The wins, and the lessons

[00:24:33] So we wanted to engineer this new release strategy. We wanted to minimize the manual effort, avoid a branch per release, enable weekly patches, one setup for all releases, release process matches the artistic process. And what I described had one line going across, so it had one release pipeline. At the start there were red lines going zoom, zoom, zoom, and each one of those took two weeks to set up. At the start of every game project we set it up once, and that's our release process, and it doesn't matter how many releases we pop into that pipeline and how many come out at the other end, it's one pipeline.

[00:25:35] Now, the setup for that was more like three weeks than two weeks, but I let myself away with that one.

[00:25:45] So yeah, the big wins. We've got releases happen when the game team decide. I'm part of the build team, I no longer have to get involved, I don't have to micromanage each release. I don't have to think, oh, this change happened in one and we've got this thing happening and this other feature and these are conflicting changes. I don't have to do that. It frees up weeks of effort for every single release, and the build team can focus on actually the overall picture, like how do we do some cool stuff, maybe we want to use different technologies or some other new cool projects.

[00:26:26] And the time to player, which is a metric that we had at the start, which is the thing that delivers business value to the company and the thing that delivers value to players, is reduced by two weeks for every single release. That's not a lot if you're releasing once every six months. If you want to release once a day or once a week, that is absolutely critical.

[00:26:58] So the lessons that we learned were to use value stream mapping to map out the entire organization and say, well, we've got these two weeks that the build team are doing every single release, probably not the best idea. Front-load a whole bunch of effort at the start of a project: spend those three weeks so that you don't have to spend two weeks every single release. Keep your infrastructure static, but that's basically what I mean with the release branches, just don't do that if you can avoid it. And evaluate your processes, which for me is the most important thing here. Evaluate your processes, because we were doing this for years and we just didn't take the time to step back and say, is this the best way of working? That's me, thank you very much.