The Technology Behind The Games

at UXDX EMEA. Video: https://youtu.be/4BJTiPkOb_I

Readable transcript: edited from the recording's captions for readability (fillers and false starts removed, punctuation and section headings added). Wording is the speaker's own. Timestamps are positions in the video. Names marked [?] could not be verified against the audio.

Introduction: a B2B organization inside a B2C company

[00:00:00] Good afternoon, everybody, and many thanks to John and the team at UXDX for inviting me to talk to you today. First and foremost, I'm not a product manager, and I'm probably speaking well beyond my expertise. But I wanted to do that thing that Rory mentioned, where we just talk about use cases. I wanted to give you a sense of how we plan technology development, how we plan our internal platform at King, and how we help the game developers who work inside King to ship those games.

[00:00:28] Let me first do a very quick introduction to myself. I've been working in and around the game industry for many years. I've been involved in a number of startups, starting with Havok and then most recently Swrve, and only in the last two years have I joined King. What I wanted to draw a conclusion from here was that my history of working with these companies was largely in a B2B mode: a business enterprise servicing other businesses as our customers. Swrve was a SaaS product, and Havok was really not quite a packaged good but a library and SDK product, but ultimately all our customers were always businesses and companies.

[00:01:06] With King it's different. King is ostensibly a B2C company. We create computer games, mobile games. But internally we're operating, or we try to operate, as a B2B product delivery organization, delivering our product internally to our internal customers, which are the game teams. I'm going to come back to that. That's essentially the theme of the talk: how do we configure ourselves in that way to deliver efficiently to the game teams?

About King and its games

[00:01:30] But first, a little bit about King, for those of you who don't know. King is a company founded in Sweden, in Stockholm, in 2003. Its biggest success so far has been the Candy Crush game, Candy Crush Saga. It was a huge success, launched in 2012, and it shot the company to stardom. Directly afterwards it went public, in 2014, and it was subsequently bought by Activision Blizzard in 2016. We have over 2,000 employees spread all around the world, and the team that I'm responsible for is a shared technology team, which is about 450 people based in Stockholm, Barcelona, London and San Francisco, as well as a number of other locations.

[00:02:10] King is well known for Candy Crush, but probably less well known is that we've released a number of titles across lots of different platforms: over 200 titles in the history of the company so far. A lot of those titles are still live, and they're the titles that our shared technology platform powers. More recently we started to develop new titles, and the latest release was Crash Bandicoot: On the Run, which is a very 3D title. This is interesting because it's maybe the first example of us taking advantage of some of the Activision Blizzard IP. Crash Bandicoot is a historic piece of IP created back in the early PlayStation One era, so it was really exciting to be involved in the development of that and the launch of that title earlier this year.

[00:02:54] I wanted to describe first the games, how they're put together and how we think about those games, particularly in terms of being live services. Hopefully many of you will either have played or at least be aware of Candy Crush Saga, which is the number one mobile game in the US at the moment. It's a super successful game: over 250 million players play the game every month, and we've got some loyal players who've been playing the game for over a decade. So it's really exciting to be involved in delivering this service. It's a deceptively simple game. At its core it's a puzzle game, a match-three puzzle game, and it's entirely free to play, so the majority of players playing Candy Crush don't pay in order to play.

[00:03:39] Essentially, in match-three you're looking to match three or more candies beside each other to unlock boosters and powers and to get points. Over time your skill level increases, because the complexity of the levels increases over time. That's the core thing: people come to play games and to have a challenge. Built on top of that, though, one of the things that King innovated very early on when developing these titles is the idea of progression in a puzzle genre. Rather than each puzzle experience being just an isolated event, it connects the achievement of the solutions to the puzzles over time into a coherent narrative. We call that the saga, and so we have Candy Crush Saga and Soda Saga. This brings a more longitudinal experience for players, so they get a real sense of progression and achievement, and we can also weave in an interesting narrative which changes over time and keeps that interest going in the game itself.

[00:04:37] So we think of the games at three different levels. One is the core gameplay itself, which engages the user to begin with: the switcher, the puzzle game. We've then got the metagame, as it were, which is the progression mechanic, this idea of making your way over time through many, many levels. And there are many levels: there are over 10,000 levels in Candy Crush released to date, and we're celebrating our 10k-level event later this year. And then on top of that again we have the gaming service itself, which is all the events, the seasonal content, the competitions, the community that we layer on top of the core gameplay experience. So you can think of it as an onion-skin game: the core, the progression meta, and then finally the community and the events system on the outside of that. And that's what the core platform enables.

[00:05:28] We think about these games as entertainment services, and they're evergreen services, where players are continually coming back to play the core game, but also to interact with the community, to continue to be challenged, and to get new events and new features exposed to them over time. Obviously we continue to release new levels to keep the game fresh for those players who are our most loyal constituents.

The shared tech platform

[00:05:51] So what powers those games? We have large numbers of live titles across many, many different platforms. Fundamentally, we have the shared tech platform, which is what I wanted to talk mostly about today. With the shared tech platform we have a core set of services, which can be split largely into our game engine; the game operations tools themselves; our economy management system, which also includes ads and in-app purchasing; the data infrastructure, our data warehouses, our analytics, our A/B testing solutions; and then the core game servers themselves and the databases that we use to keep persistence about player progression. All of this comes together into a single platform that all of our games sit on top of and communicate with on a constant basis, day to day.

[00:06:34] As I mentioned, we have over 250 million players every month, and they send us over 75 billion events per day. Our platform essentially sorts through those events, gets as much observability as we can on what's happening inside the game, and then allows our game teams to operate the games on behalf of our customers, the game players, to give them as great an experience as possible.

[00:06:54] Let's click in a little bit deeper into some of these areas. The game engine is our own proprietary 2D and 3D engine, called Fiction Factory. It has a long history inside of King, and its superpower, I guess, is its ability to deliver these very high-quality experiences to the widest possible breadth of platforms and low-powered devices. This gives us really great reach, so we can reach pretty much everybody on the planet with the games that we deliver. Internally it's a C++ engine, which is then cross-compiled onto the different platforms that we support.

[00:07:28] We have our economy management tools. There's a sophisticated set of tooling that allows us to manage the in-app purchasing, because we are a free-to-play game. The majority of players don't pay for the service, but we have a subset of our players who do pay, and they pay for things like boosters and cosmetic items and level progression advancements or free lives and things like that. We also have an advertising side of the business, but the economy of the games is really important for us to manage in fine-grained detail. This involves setting pricing for items, promotional activities, merchandising, all the types of things you'd expect in a traditional e-commerce platform, except this is all for virtual items.

[00:08:11] The economies themselves are getting increasingly sophisticated, and particularly with our more recent games, the economies are quite deep and have lots of different things that players can purchase or interact with. An example might be the skins that we sell, the cosmetic enhancements for the Crash player inside the Crash Bandicoot game. There are very different skins, which in some cases are made available for free, but in some cases are available for purchase inside the game.

[00:08:36] And I mentioned briefly the ads side of our business. Another important part of our economy is players who don't necessarily want to pay for any specific item in the game, but are happy to watch an ad in return for maybe a free life or a booster. In our case those ads are always opt-in, so no one is forced to watch them. But if you wish to make faster progression than you would normally, at the end of the level, for example, if you fail to complete it, then watching the ad confers some benefit. That's an increasing part of our business, and it's working very well.

Live operations and data

[00:09:06] We have a whole series of live operations tools. This is super important, because we need to be able to operate the game very effectively over the air, essentially without having to release new versions of the game client through the app stores, which has a certain latency and friction associated with it. So our goal is to be able to operate a huge amount of the game completely transparently via our internal operations tools. This can get quite complex, because there are multiple campaigns running at any given point in time in the game, and these can all interact with each other, with events running, with seasons. It's essentially managing a calendar of live events for this entertainment platform, and that's exactly how we think about operating it. Some of these events are specialized for particular cohorts of our players. Our early-life players might see one set of events, and our later-life players might see a different set of events, because they understand the game more. We can have a very differentiated experience depending on how much you've played the game historically.

[00:10:03] To give you an example of the complexity of these types of live operations, a campaign that's running right now in the States is our Candy All-Stars campaign, and that brings together very many parts of our company and our operations infrastructure. It includes leaderboards, it includes promotional activities, it includes invite and friendship mechanisms, it includes links with TV shows and our marketing department. So it's a huge multi-point marketing campaign and event for our players to compete against each other, ostensibly to find the best Candy player in the US. It's a really exciting event, and it's one that we ran last year in the UK, so this is something that we're really excited to expand into the US market. But that gives you a sense of the type of overlay that sits on top of the core gameplay mechanic itself.

[00:10:50] Driving all of this is data. We rely very heavily on observability, on knowing what's happening inside the game and how our customers are interacting with the games. We've got a lot of data coming in, and we produce a lot of internal metrics and analytics and reports that we derive insights from. We're a very heavily A/B-testing-influenced culture as well. With our games, given the large numbers of users we have, even small percentage changes in KPIs have a meaningful impact on our business. So we need all the tools at our disposal on the statistics side to sift through this data and look for statistically significant results which can be meaningful for our business. That's built into the core of how we design.

Team structure and the product portfolio

[00:11:32] So that's a very quick insight into our games, how they're structured, and some sense of the platform that we're building. That's what I have responsibility for: the shared technology platform driving those capabilities. So how do we go about planning and prioritization, and how do we go about structuring our team to deliver that service effectively?

[00:11:55] First off, it's the team structure itself, the org. We're essentially organized into six main departments, as it were, and each department has a set of sub-products that they are responsible for. We're implementing an inverse Conway, to coin a phrase. Really what we're trying to do is create departments which minimize the dependencies between the departments, so we optimize for the flow of communication. What we have done in configuring ourselves as a B2B organization is have engineering and product management peer-to-peer inside the organization, which I think is potentially slightly unusual, particularly for a gaming company. And more recently we've been layering on this concept of partner success, which is an equivalent to customer success, and something I'll come back to.

[00:12:36] So what do our product lines look like? I'm not going to get into the details of the products, but we have a broad spectrum of products that all come together into that single platform, a number of different product areas. At any given time we're having to manage this portfolio of products, like any product management team. Some products are designated for future investment; these are the areas where we think we can deliver future value. In some cases we're in maintenance mode, and in other cases we're trying to deprecate products which are end of life, where maybe we've produced newer versions and we need to move the teams onto those newer versions. So we're constantly balancing all these different product efforts at any given time.

[00:13:11] The resourcing that we have available to do that is always subject to change as well. We try to track the effort that we've put in across our teams over time, so we can understand, by team, the amount of time that we've managed to put into mission work, or new product development work; operational work, which you can think of as maintenance or reacting to issues; and then the unplanned work, which is typically associated with incidents that arise with the service or with third-party partners, or indeed unplanned things that we have to react to, like changes in legislation and maybe changes in the platforms that we support. Taken all together, this gives us a feel, looking forward, for how we can extrapolate our capabilities and our resources to plan effectively.

Internal customers: the inversion of customer discovery

[00:13:52] Who we're servicing is the customers, the internal customers, which are the game teams. Our game teams have quite a variety in their distribution of size. Candy is our largest internal team, and in some cases we've got single teams responsible for a multitude of titles. The games themselves follow a power-law curve of scale, so it's quite a challenge for us to balance the needs of all of these different customers in a rational way. But one of the tenets we have is that even though the biggest games are our most important customers, we're developing a shared platform, so we always have to ensure that any capability we introduce can be used by multiple of the teams that we're supporting.

[00:14:30] Having said that, it's a nice analogy to compare to the classic idea of customer discovery. If you're building a B2B company and you're looking, in the early stages, to figure out who's the market for my product or my service, you'll possibly follow Steve Blank and his idea of customer discovery. That's trying to discover the customers that, at any given point in time, best pattern-match to your current capabilities. Rather than trying to meet the needs of all customers, you're very deliberately choosing customers that map to your current capabilities, and over time, as you have identified those customers, you expand your capabilities and expand your market share as a result. So customer discovery is a really important part of B2B growth, to grow in a rational and managed way.

[00:15:15] For internal customers you don't get to do that, because the customer set is fixed. So you have a different problem. It's not customer discovery; it's minimum viable product discovery: what's the minimum set of products that are shared equally across all the different customers? You're looking for the shape of product that meets as many of the customers' needs as possible, but doesn't get too specialized to the needs of any given customer. So it's essentially an inversion of customer discovery.

[00:15:42] To do that, when we're going into planning cycles we try to keep a number of principles in mind. The first one is that idea of making sure that our services in our games are operable and data-driven. We really want all the services that we provide, largely speaking, to be deliverable in a way that doesn't require a client update. That means all these capabilities can be live and operated in real time by the game operators. We're always looking for features and products that have high impact potential, so they can drive real business growth in the various game teams, but also high repeatability: again, that idea of not producing features that are only usable by one given customer. We're looking for differentiated capabilities. If there's a good solution in the market, we want to use that. We're going to put our focus and efforts on areas where we can make a change, where we can actually be more competitive within our existing market.

[00:16:31] And over time we're always looking to reduce complexity and fragmentation. Any company of our scale has a legacy and a history, and it's quite a challenge to deal with that, deal with the technical debt, and deal with continually moving your platform forward in a way that's coherent. And I guess at the end of the day we're servicing the idea of minimizing the cognitive load of our customers. By that I mean we want a single set of SDKs, a single set of APIs, a single platform and a single user experience that's really coherent. That allows our game teams not only to operate efficiently, but also to interoperate, to share between them very efficiently as well, so we get a high degree of transferability of knowledge and skill across the company.

Annual planning: cascading goals

[00:17:11] So how do we do that? What does the planning actually look like? I guess we kick off, as anybody in a larger enterprise does, with corporate goals being established. I'm going to talk to you about a yearly cadence of planning, which I think is super common. Typically at the start of a year, leadership gets together and defines a set of goals for the company for the next period of time, at least articulating where we hope to be by the end of the subsequent year. This is also informed by longer-term strategy, of course, but I'm keeping a lens on the next year just for the purposes of this.

[00:17:42] That informs, in my case, my team's platform: what are the objectives we need to have in order to meet those corporate goals? I split them into two areas: one thinking about our platform and product, and on the other side thinking about the organization, or the people or process goals. They need to be considered at the same time. On the platform side, we're interested in keeping account of our technology strategy and where we want to be longer term, as well as making sure of the more tactical customer commits and the milestones that our customers, the game teams, have for delivery in the subsequent year.

[00:18:15] These then get cascaded through the organization. It's also really important to have a connection between the two different areas, because organizational objectives also cost, in terms of resources to implement, so you have to make sure these are considered at the same time. Ultimately you end up with the outputs of this process, which are your roadmaps, expressed in some way, and also sets of personal goals. The idea is that the goals at this level can all link right back to the top-level objectives set by the company.

Filling the blank canvas: the knapsack problem

[00:18:46] What you face, as any planner or as any product team going into the start of that year, is the blank canvas, the really scary blank canvas. You've got some idea of a planning horizon, let's just take a year, and you have some idea of resourcing and capacity, which is a dimensionless quantity that varies by team and by time and is really hard to reason about. But ultimately it's our job to fill this void, this blob of time and resource, with stuff to do during the year to hit our objectives. So we'll have framed objectives; we're trying to meet a certain set of objectives. We have a set of customers and requirements, and aligned responsibilities with those customers to deliver. We also have a product line that exists, the artifacts of our efforts, and that product line can largely be broken into areas that we're continuing to maintain and operate, and areas of new investment, things we think can move the needle by the end of the year.

[00:19:43] Our goal is to fill this space with stuff, with activity, with work, with resource, in order to meet all these goals. That turns out to be an expression of the knapsack problem, in a sense. It's an NP-complete problem, so from a complexity perspective it's not possible to optimize this, but you can do a good job of iterating your way towards at least a locally optimal, or an approximately optimal, solution. So here's how we do it.

[00:20:07] We start off by thinking about our technical debt, and we try to allocate a certain resource to technical debt, so we keep things moving in terms of our product and we have this progression of core technology that maps to future needs. We also have an idea of maintenance. Any given product that's in maintenance mode, we know it has a long-standing set of requests coming from customers, and we have issues that we need to resolve. Specifically for maintenance, we'd like to relate that directly to customer requests, in this case three customers.

[00:20:37] What we're trying to do here, to some extent, is limit the bandwidth that we spend outside of new product investment. Most companies will have some budget that they're trying to allocate, either for tech debt or for maintenance. In our case we try to think of something like a 40% rule. This is not science at all, but it's the rule of thumb that we use internally. Then within that, the space that's available is our opportunity to move the company towards the objectives that have been specified. Our job then is to fill the rest of the space with our product backlogs, and that's what we do. We communicate, we assemble, we discuss, we argue, and ultimately we come up with some map of the things that we want to do, streamed or themed by different product areas, in some cases driven by objectives, and in some cases driven by specific customer needs from time to time.

No plan survives contact: multi-resolution replanning

[00:21:28] After all that effort, we get this beautiful, pristine view of our product roadmap, and we're very excited and enthusiastic. Of course, anybody who's been involved in this area knows that that's largely nonsense. While that is pretty good for the first couple of weeks, over time things just deviate and get more and more chaotic as the world happens around us. No plan survives contact with the enemy is the classic quote, and that's certainly true of product roadmaps.

[00:21:55] But don't despair. I guess everyone knows what's coming next. We iterate our way towards a solution that is locally optimal. At different time epochs we replan. We reassess where we are in terms of the inputs that have come into the organization, and the changes in timings and changes in resources, and we get something that's closer to a staged plan. It's a resync: we allow it to get out of whack over the space of a quarter, but then we synchronize it at the end of the quarter.

[00:22:22] And it's not even as simple as that; that's far too high-level a view. In reality you build a process and a set of communication channels to be much more reactive at different scales in product planning. In our case this looks something like this. We have an annual planning process, which I've described. Then every quarter we review that, just to sanity-check that we still have the right objectives. Are we tracking to our key results and objectives? Are there major resourcing changes we have to make, or big changes in the business that we have to take care of? Then on a monthly cadence we're looking more at week-to-week, month-to-month, sprint-to-sprint resource allocations, and changes that might have come in from the outside, perhaps a legislation change or something we have to react quickly to. And then we have our daily behaviors, our daily ceremonies, the stand-ups and all the other activities that give us that very tactical, very high-resolution and very low-latency ability to react to the changes that we're seeing.

[00:23:17] At some level you can think about this as a multi-resolution view of your implementation. You start off with a very strategic view, the annual view, and over time you evolve towards that very tactical view. You need to have both ends of this spectrum active in your organizations. The annual gives us the directionality, where we want to land, and the tactical gives us the ability to react quickly to the reality on the ground.

Fallacies of internal customers

[00:23:41] This is driven by the visibility of what the customer requirements are. One of the fallacies of internal customers, which is the mode we're in, is that you should have perfect visibility of your customer, because they're sat right there beside you, or they're just in the next room, or on the floor above you, or whatever. That breeds a certain familiarity bias, as I'm going to call it. Particularly as companies grow, what you've relied on historically is interrelationships between people, and this means that your perfect visibility of your customer requirements starts to fragment dramatically over time.

[00:24:18] So reliance on old networks is a problem; it's one of those anti-patterns. You might have had two engineers, one in the product team and one in the game team, who had a great relationship and made stuff happen. That works really well until you have a new game team that doesn't have that relationship, and suddenly they're out in the cold. That's very tricky. And people move on, and some of these old networks get broken as a result of changes in your teams.

[00:24:42] The non-communication of changes, and the implicit assumption that things that have changed in priority in one place will be visible in another place, is usually wrong. You have to work very hard to keep track of mutual changes in prioritization and scheduling. These uncommunicated changes just propagate and build momentum over time.

[00:25:01] Also, within organizations, and any sales organization will understand this, you're essentially selling to an org. We, as an internal organization selling to internal customers, are effectively selling to a customer org as well. Just because we're in the same company doesn't mean that the leadership of that game team fully understands, or is fully aware of, what the engineering in that game team is doing. This is where some of those old networks can really fight against you, because while the right thing might be happening at an engineering level, it could be out of whack with what the prioritization from leadership is. So that misalignment in the org is something you really have to track. And ultimately, over time, the channels of communication just get too diverse, too many to keep track of at any given time, so you have to put in a rational mechanism for communication.

Alignment

[00:25:48] One of the things I think I've come away with, spending these last two years in King and comparing that to my previous years in B2B, is alignment. It's probably the single most important word, the word I probably hear most on a day-to-day basis: maintaining alignment between us as an internal product developer and our customers, the game teams. Alignment is super, super important, and this has many dimensions. From our perspective, the most important alignment is with the game teams themselves: understanding their roadmaps, understanding their release schedules, their projections, and when they're going to need certain capabilities. Because the game teams are the lifeblood of the company, and their requirements drive our requirements.

[00:26:32] There are other areas of alignment. Our technology leadership across games and in the central tech solution as well, thinking about tech debt and longer-term platform support. With our product areas, we need to align between the product areas on resourcing and prioritization. We need to be aligned with the overall world outside of us. The legal teams allow us to keep track of changes in legislation and changes in platform requirements. We have the platforms themselves, and by this I mean Apple and Google and Facebook and the other platforms that we support. They have a changing regulatory environment and a changing set of requirements that we need to map to. And finally there's our own King leadership, and the long-term strategy of the company and what we're trying to achieve.

Product management and partner success

[00:27:13] Coming to a conclusion: how do we go about implementing this alignment in an internal product mode? We used the classic tools of B2B enterprise, with product management and customer success. Product management is something that we introduced a number of years ago, and it's very, very effective. We have product areas with product management who are responsible for working with the customers, prioritizing their requirements, developing the use cases, and communicating the requirements and the prioritization to the engineering teams. Product management and engineering are peer level in our organization.

[00:27:48] That introduces its own problem, in the sense that we have a large product portfolio, and product management without something else is a real problem for game teams. If you think about it, an individual game team might be a small team that has to maintain multiple points of communication with all these product managers to keep track of what's going on. You can deal with that to some extent through a product management hierarchy, but it's also important to have someone on the inside, essentially. So we built our account management, or our customer success team. These folks work directly with the game teams and keep track of their specific requirements and roadmaps and schedules, and they're essentially our first warning signal. They aggregate their customers' requirements and interact with our product managers to make sure we're all kept informed and on track with their schedules.

[00:28:32] So, nearly to conclude: we're configuring ourselves as a B2B enterprise delivery organization. This is what a classic B2B enterprise delivery organization looks like, I think: engineering, product management, then you have support, account management, and solution engineering or solution architecture, which I'm going to loosely call customer success, and then sales. So how does that map? We also have engineering. We have support. We've introduced integration engineering, which is the first step towards that customer success capability. This is all within engineering, and integration engineering allows us to deal with customers that are smaller, that maybe can't keep track of the changes we're making to the platform quite as well, and it also does an education job in helping customers adopt new technologies. We have product analysis, which is a direct analog in B2B. And then we have introduced, and are scaling up, our partner success organization, which is effectively the equivalent of account management: that idea of keeping track of customer requirements, staying very close to their roadmaps and their delivery schedules, and representing that within our team. We don't really have the equivalent of sales; I guess partner success does that.

Conclusion: B2B lessons for internal platforms

[00:29:37] In conclusion: we started by saying I have a history in B2B, and now in King I find myself in a B2B2C mode, trying to be a B2B organization delivering to an internal set of customers. Just some observations on the relationship between those two. First off, as I mentioned, there's no customer discovery. All priorities arrive by alignment, and alignment is probably the word I come away with, what I've learned most about at King: how to align with the internal customers and with a larger organization.

[00:30:08] We don't have a sales renewal cycle, and anybody in B2B, and particularly in SaaS, knows that the renewal is nearly the most important event. The initial sale is important, but actually renewing shows you have a scaling, momentum-building business. We don't have that, so we rely very much on account management to understand what the customer is really requiring, and to make sure they can continue to take updated versions of our platform.

[00:30:31] We've no marketing funnel, in the sense that there's no SEO or SEM or marketing qualified leads or anything of that nature. So our emphasis is on visibility: the customers, the game teams, really understanding our platform, and us evangelizing the sharing of technology and promoting the adoption of our platforms. We keep all the customers' cognitive load to a minimum, and all on the same versions of the systems that we provide.

[00:30:55] The competition to our internal solution is DIY, rightfully so. So we try to set the centralization bar super high, so that we only share and only centralize technology if it's truly something that can be used across multiple game teams. If it's a single game team requirement, then they should have the resources to deliver that. And then finally, the power of shared OKRs is extreme, but you have to watch out for that familiarity bias. Don't assume that the knowledge transfers. You have to work really hard to align on OKRs and to ensure you're on track with your customers. Hopefully that gives you a sense of some of the experiences of trying to deliver solutions inside a company like King. Thank you very much.