Reducing Time To Player With Static Infrastructure

Ross McKinley at UXDX Community: Stockholm. Video: https://youtu.be/cOpRleAFplE

Readable transcript: edited from the recording's captions for readability (fillers and false starts removed, punctuation and section headings added). Wording is the speaker's own. Timestamps are positions in the video. Names marked [?] could not be verified against the audio.

What time to player means

[00:00:11] I'm going to talk about reducing time to player using static infrastructure. I'm a developer, forgive me. What you'll get out of this conversation is what this metric, time to player, is. I work for DICE, we make triple-A games, they're really big, they're everywhere, everyone plays them. I'm specifically going to talk about the patching process, after we've made that big bang release. How do we make incremental updates to get players content that they want to play? And I'm going to talk about how we used to do this, and how we are starting to do this going forward.

[00:00:54] So, time to player. Time to player is this concept where, as a big games company, we spend months and months and months, and sometimes years, working on a game, and then all at once we release it to the public and we announce it, and this all happens in a couple of weeks. That's the traditional waterfall, very slow process of developing a product. I'm not really going to talk too much more about that. The second phase is, once the game goes live, how do you make patches, how do you fix bugs, how do you release new content? I'm going to focus on that.

[00:01:43] So, time to player is, we've got this idea from all the creative people in our company, we've got a bunch of developers, they start prototyping and hacking on things, we pray for a really long time, hope that everything works, and then finally players get it in their hands. I'm on the build team here in DICE in Stockholm. Time to player is the time to go from inception of an idea to a player's console or PC.

[00:02:21] I'm on the build team, we do all of our magic in this praying section. We try to stay out of the way of all of our developer engineers and creatives who are working on the left side of this diagram, and we take all the content and code that they create and see it through the entire process to a player's machine, be it a console or PC or whatever. And in that circle there, there's so many stakeholders. We've got QA, we've got outsourcing QA, we've got the audio teams. The audio teams will generally only start working on fixing the soundtrack and the explosions after they've been implemented, right, because they only know how long an explosion is going to last after that's been figured out. So that's just one of the teams, for example.

[00:03:31] So that's this metric, time to player. How do we reduce the amount of time it takes to get from the brain all the way over to the player's console? We'll talk more about the issues in the previous way of doing things in the next slide.

The triple-A patch release process

[00:03:58] So, the triple-A patch release process. This is how we actually do all of this. I'm going to get a little bit technical, I'm going to talk about our branching structure. So here we are making a release. Let's just pretend that we've done all the work and it's launch day and everyone's celebrating. At that point, some people are working on the next new feature. On launch day, all of the work that's gone into that is already way on in the process of going through compliance, getting all the legal sign-offs, going to a franchise holder to make sure that we're representing the characters correctly and that everything's good to go.

[00:04:52] In the meanwhile, all these new features are getting developed. After the release happens we then take all of that content and make sure that it's integrated into the new feature that's getting developed, so that we can make a patch on someone's console. Unfortunately games are really, really, really big, and once you deliver something to their console it's usually on a disk, and that disk is around 50 gigabytes in size. We want our patch to not also be 50 gigabytes in size, so we take the latest version, which is the one that's live in the world, and we do a comparison to say, oh, only this amount of data has changed, so it only has to take this much extra space on that person's machine.

[00:05:45] That's quite a complicated process, but it means that we have to be very careful about what version we're patching against. So this new feature now becomes release 1.1. Back in time, before this release has gone, someone else has started working on the next new feature, and these are all happening sometimes in parallel. Maybe this feature has started work months before the original release and hasn't got any of the latest content. So then that's our hot new feature again, and then we do this big complicated process of making sure that that's now up to date with whatever is live on other people's machines, and that's our hot new feature that we want to release, and then that becomes our release.

Two weeks for every branch

[00:06:44] So this is the way we used to patch games. In this circle here it's kind of not clear what's going on there. Every time that we make one of those branches, it costs us two weeks of development engineering time just to set up the whole pipeline for the new release. So the very first time that we're making the release, the two weeks has already been spent. The next time we're making a release, we have to spend two weeks of effort to warm up all of our build machines and do a bunch of technical stuff, which is just dead time, it's wasted, but we have to do that every time that we want to make a release. So if we have two releases that's four weeks. If we have ten releases that's just 20 weeks. Sorry, at the end of the day mathematics in my head is not my strong point.

[00:07:47] So this is fine if you're making maybe two or three releases a year, which is traditionally how video games would be made. You have like, oh, here's a brand new DLC pack, or we've got ten more levels to play. What we want to do moving forward is give incremental updates to our player base, so that we can keep the product alive for a longer amount of time and have more content, more updates, more bug fixes ongoing throughout the development cycle. So, two weeks to set up every branch, that kind of sucks.

[00:08:33] So what we did was, we stood back from that whole process and figured out, okay, well, we've got all these creative things happening, the release process looks almost the same every time that we make a release. It has to go to the same people, it has to go to compliance, the QA people still need to get it. It's the same thing. Why is it taking so much time every time we want to make a release? So how do we fix this?

The fastest way to do something is not at all

[00:09:05] This is really interesting, because Rory hasn't seen my slides so I don't know how he got this. I make this same quote in almost all of my presentations: the fastest way to do something is not at all. A good friend of mine, Myles McGuire, said this about ten years ago, and it's something that I've really taken to heart as a build engineer. What I want to do is just nothing. The fastest way for a build to happen is for it to just not ever need to happen.

[00:09:44] So how do we fix that two-week process of setting up a new branch? Well, what happens if we don't do it? That's what we started to do. We started to think, can we remove the build team from the process of making a release? Can we spend all that effort upfront? Can we keep the infrastructure static, so that we don't have to change things every time we want to make a release? The build team shouldn't be involved in the speed or the amount of releases that are happening. And then, let's do one setup for all releases.

How DICE ships games today

[00:10:30] So, how does DICE ship games today? Over the last couple of months we've gone through a bit of a design process and quite a lot of engineering to make this happen, and we think that we've got something nice here. I'm going to go through the whole process again, and bear with me, I know there's going to be a lot of arrows here.

[00:10:59] So let's say that we have released 2.0, then we have 2.1 which is in development, and that's going to be happening by some other team alongside the whole process of getting release 2.0 polished and going through all of the QA process. And 2.2 is somewhere else in the distance, that's the iteration, prototype phase, we don't really know what features are going to be in there or what bug fixes. And then we've got all of these features which are still in the idea phase.

[00:11:38] The way that releases actually happen from the perspective of a player is, the player sits on this side of the screen and on the console they've got release 2.0, now they've got release 2.1. And in that process we have this — let me go back. So here we've got 2.2, 2.1, 2.0, then we make that release and we've got 2.3, 2.2 and 2.1. So all of these new features are coming in on this side and just going through the same pipeline this end. Super easy and super straightforward, but this is totally revolutionary.

[00:12:32] So setting up this pipeline took us three weeks, and we never have to change it. So if we want to make a hundred releases in a year, it's three weeks setup at the start of the project. Obviously there's other things to do, right, I don't just sip margaritas in the sun. We do have a balcony though, which is really nice. As you can see, this will just stack up, so you've got now 2.2 is released, nothing else has changed in the picture.

[00:13:13] So this image here is roughly the same image as this. We've got ideas on the one side, we've got a bunch of development, we've got a little bit of prayer involved, of QA and bug fixing and so on, and then it releases to the player. That's how that maps onto the whole process. We've got all of our feature development on the one side, we've got our core engineering a couple of releases out, we've still got a lot of prayer and a lot of bug fixing and polish that happens, and then eventually it goes out to the world, or to a subset of our users if we're doing some sort of a beta test or something like that.

[00:13:55] But the key bit here is that nothing changes. We can push one release through this and the engineering setup is three weeks, or we could push a thousand releases through this and the engineering setup is three weeks. So that's how we got the build team out of the whole picture of blocking releases, and that's how we will end up shipping way more often. Because previously, if someone said, oh, we wanted to do four releases in a month, it's like, well, that's going to be eight weeks, so how is that going to happen?

What it freed up, and lessons learned

[00:14:40] Yeah, so that was a pretty nice win for us. Now releases happen when the game team decides, because the build team is no longer in that process. A game TD can say, oh, we'd like to release tonight, and it's like, okay, we don't do anything, good luck. It frees up weeks of effort every release, so now as a build engineer we get to focus on things like reliability, making sure that the build goes fast, rather than making sure that we're able to make a release. And we can focus on cool new projects, like engineering fun stuff, whatever that is.

[00:15:30] And the time to player here, which is the key metric, has been reduced by two weeks for every single release, except for the initial one, which is three weeks. But we can work on that, because we're able to work on these cool new projects, so we can slim that down at some point in the future.

[00:15:53] So the lessons learned here are to remove the build team from the process if possible, and spend the effort upfront. Sometimes this is a really hard thing to sell to someone, if you're saying, oh, it's going to take us longer to ship the initial release, people don't like that. But it's really easy to convince someone that actually you can now release thousands of things from here on, and it costs nothing. So that was a big win for us. I'll answer some questions if anyone has any.