King's Product Prioritization Framework (RICE Model) for 400+ Team Members!

Jaco Els at UXDX EMEA. Video: https://youtu.be/zwz_HlhxJ0U

Readable transcript: edited from the recording's captions for readability (fillers and false starts removed, punctuation and section headings added). Wording is the speaker's own. Timestamps are positions in the video. Names marked [?] could not be verified against the audio.

Introduction and a short history of King

[00:00:07] I'm talking about how we improved and evolved our product delivery process at King. My name is Jaco Els. I look after the product team for Shared Tech at King. I started working in the gaming industry with a lucky break, getting a job at Electronic Arts around 2011. I've spent most of the time since then working in different adjacent industries to the game industry, and I joined King about four years ago, firstly looking after the product team for our data and machine learning teams, and then about two years ago I took over product for Shared Tech.

[00:00:50] King is a company that was founded in 2003. It started as a business that made web games that it released through its own web portal. I know some of you might remember back then it was quite common to have web portals with hundreds of Flash games on them that people played in their browsers. The trend that followed that was Facebook opening up publishing of web games inside of Facebook, and the first business that really was successful with that was Zynga with a game called FarmVille. King followed that trend and started to publish their web games on Facebook, using the web portal as a way to quickly iterate new game concepts and game mechanics and then releasing the ones that got traction on Facebook.

[00:01:40] The first one that was really successful was Bubble Witch Saga, which was a bubble shooter on Facebook that grew really rapidly, and less than a year later King released what would become their biggest hit still today, a little game called Candy Crush Saga, that I think everybody's mom played at some point during the last 10 years. Facebook was big because it gave access to a large audience, but it was when mobile, the next trend after Facebook, came that that scale really took off. When King started launching their games, specifically Candy Crush, on mobile, the company really started to grow exponentially.

[00:02:22] Following the success of Candy Crush, the strategy was, and the common wisdom at the time was, you have to find your next big hit. So the company started to invest in building out a portfolio of IP, and we released a number of games during that period: Farm Heroes Saga, Pet Rescue Saga, other Candy franchise games. They all were successful in their own right and as standalone games, but none of it really ever matched the scale and success of Candy Crush, and Candy Crush still today is one of the biggest mobile games in the world. So the strategy in 2019, 2020 really moved from supporting and growing a portfolio to focused investment in Soda and Candy, because as the platforms evolved and the game evolved, the realization was that certain games actually have a lot of years in them, and you'll see that with all of the big IPs in mobile and on other platforms as well.

[00:03:24] To invest in a game the size of Candy Crush we need a lot of people. King is about 2,000 employees. We have studios across multiple cities: Berlin, Malmö, Stockholm, London and Barcelona. The two largest teams in the company are the Candy Crush team, who has about 500 people across seven product areas, and Shared Tech, where I work and where our product teams are, which covers four of what we call product domains that build tools and in-house platforms that the game teams use to create content.

[00:04:04] The four teams are, firstly, content creation focused tools: our game engine, tech artist tools. Then live operations tooling, so any ephemeral content or seasonal content that the game teams create is deployed into the game through a set of LiveOps tools that manage scheduling, targeting and the in-game economy tooling. The third area is data and machine learning, so this is analyzing, structuring A/B tests, and our machine learning and AI capabilities. And lastly, but definitely not least, is core platforms and infrastructure, which is our Google Cloud platform along with incident management, etc.

[00:04:47] So large groups, teams across those four domains, roughly 22 individual product teams that service the game teams. Inside of the game teams it's really about creating and crafting fun product player experiences that then get pushed out to players through live operations teams. Those teams are typically level designers, people who design seasonal content, so this time of year Halloween themed content inside of the game, and an operations team that deploys and tests and measures.

Growth, tech debt and the slowdown

[00:05:19] What we realized is, as the business went through the exponential growth, the best way was to hire smart people, have relatively autonomous teams that can move fast, identify opportunities and iterate quickly. We were very engineering focused in the early days. We're a culture of building things ourselves and not buying stuff off the shelf, and that allowed us to really support the rapid growth of Candy Crush through the years. But what started to happen was the accumulation of tech debt. As people move on, you get loads of in-house developed tooling; people eventually leave and those projects get abandoned.

[00:06:00] Candy Crush reached its 10th birthday recently, and as you can imagine, over a 10-year period a lot of debt accumulates, a lot of complexity, and what we started to see is a slowdown of our ability to iterate and deliver value to our players. This really started to show in the level of frustration, sometimes, that came from our stakeholders. With the complexity of the organization and the number of people that need to coordinate over a large number of teams, it's really hard to keep track of who's working on what, which teams are delivering on the projects and the requests that I have, why is stuff behind schedule, and how do I get this important new opportunity that the business has identified prioritized within what is a large book of work.

[00:06:51] These frustrations started to accumulate into getting this type of question quite often: if you have 400 people in Shared Tech, why is it that we struggle to get the things we need? This is a scene out of a movie called Office Space where the development manager is interviewed by some management consultants trying to explain what it is that he actually does day to day.

[00:07:15] So we started to look at this, and we saw the accumulating slowdown, the struggle to get value to our internal stakeholders, which meant we couldn't get value to our players, and we really started to reach an inflection point, and that really showed with specific projects. We went out to the market to do a build versus buy evaluation of a CRM system, and we identified that if we bought a market leading CRM system we could improve the quality of CRM and in-game content delivery to our players at a lower cost. We could deprecate a lot of legacy tooling that we've built over the years and essentially replace that with a market leading CRM system, and also all of those engineers that were maintaining and building in-house tooling could be moved to higher value projects.

[00:08:13] So, no-brainer. We put the business case forward, it got approved, and we expected to deliver the new system in about six months. The project ended up taking about three times the estimated duration. It took us 18 months to get the CRM system deployed, and when we got to that point it was with limited initial functionality. So why did that happen? Something that's supposed to be relatively simple took us such a long time, and this was not only happening on this one specific project; we saw it happening in other projects as well.

Misaligned priorities, no transparency and a complex organization

[00:08:56] So we started to have a look at our internal processes, and some of what Rory said in the previous talk really resonated. The first thing that we identified was that there's really a big misalignment in priorities between what Shared Tech does and how the game teams operate. Game teams, like most business teams, typically get measured in terms of their ability to drive revenue and grow revenue, which means they generally think in terms of quarters. They get given targets and that's how they're measured, so that's what people running the business day to day focus on. Whereas Shared Tech is a strategic partner that typically needs to run projects that go over multiple quarters. We optimize for things like operational efficiency, creating new capabilities that drive revenue. So we found that this misalignment in the way that we think about value and prioritization of work was really one of the key things that made it difficult for us to align on what work we should be doing next.

[00:10:01] The next problem was a real lack of transparency. With this culture of moving fast and being engineering led, that also went along with a very high level of autonomy that individual teams had. Autonomy is great for moving fast, but it's not good for ensuring that there's very good collaboration and connectivity between the efforts, and that went along with a severe lack of transparency. What we found, for instance, inside of Shared Tech, is if we had to answer the question "which teams in Shared Tech are doing work for the in-app purchase team in Candy?", that was a really difficult question to answer. All of these teams were working really hard and delivering value, but just understanding where the dependencies were was really, really hard, and unnecessarily so.

[00:10:53] This was made worse by a complex organizational structure. As I said with the teams in the previous slides, it's about a thousand people across multiple teams, and that being relatively flat. Once again, flat is supposed to make things more efficient, better decision making, faster decision making, but to figure out who I need to speak to about the one thing I need, that's sometimes very, very hard. A professional network inside of King was a superpower. If you knew the network and you knew who knew the person that you needed to talk to, that enabled you to move fast, but for new people joining the business that was super hard, to hook into that network.

[00:11:36] Lack of standardized processes: each of these teams were working their own way using their own tools. Some people were using Trello, some people using Jira. So to get a consolidated view, as I said, to get transparency, was super challenging, and that then ended up leading to projects that should take months taking years.

The new operating model

[00:12:06] So we reached this inflection point. The CRM project was a big one. We had another one related to player identity that took more than a year that really shouldn't have. Inside of Shared Tech, seeing the trends, seeing the challenges we were facing, we took a pause and really had a deep look at how we were operating and where the opportunities for improvement would be. We identified four areas where we would try to effect change, and we call that our new operating model.

[00:12:43] Firstly, we looked at prioritization. The point about prioritization is not just figuring out what order we're going to do stuff in, but getting a consistent way of talking about value. How do we consistently talk about value, and how do we use the same language around value inside of Shared Tech, but also in our conversations with our stakeholders, and have that aligned with how the stakeholders talk? For the game teams, that's primarily thinking in terms of revenue and efficiencies.

[00:13:17] Tracking. Tracking was very controversial in King because of the level of autonomy that teams had, and still have, and especially engineers were resistant to starting to track the type of work and how they work consistently, because it felt culturally like we were monitoring them. But that's not the intent. The tracking, and I'll go into it a bit later, gives you the opportunity to just have insight into what the teams are working on, so that we as a management team can support the engineering teams better.

[00:13:51] Alignment and communication: that's about creating transparency between teams and our stakeholders. And roles and responsibilities is about us just having an efficient way of working and a good understanding of what is the expectation of my contribution to the delivery. I'll go into each of these in a bit.

Prioritizing with RICE

[00:14:12] For prioritization we settled on the RICE framework. As I said, the value of using a framework is just a consistent way of talking about value, and it's not really about the specific framework, I believe. I think when you look at frameworks you should take a wide view of the frameworks out there and decide which one fits your business the best. For us RICE worked really well because it aligns with how we think about our business and how we think about value in our business. So we chose RICE. It's a well-known framework. We have a scoring system for each of these elements that produces a RICE score at the end that we use to stack rank priorities.

[00:14:55] Reach, for us, is really how we think about our business: whatever we do ends up in an experience inside of the game, or should be enhancing an experience inside of the game. So reach is the element of how many players will be impacted by a piece of work that we prioritize.

[00:15:17] Impact is really the key value measurement. We broke impact down by four elements. Does it drive revenue, so dollar value of additional revenue. Does it drive savings, so dollar value of savings that that feature or product will drive. Then we look at efficiency, so does the new capability drive efficiency; that efficiency then also gets converted into a dollar value realization. And last is risk mitigation. Risk mitigation for us is most often regulatory. There's increasing data privacy and child protection rules across the world that we need to implement, and if we don't comply, that's material risk to our business. So if we have, say, a new age rating that we need to apply in a specific geo, the risk is the value of our business in that geo, typically, and that's how we incorporate that into the framework.

[00:16:31] Confidence is typically how confident are we that we can drive the impact that we've assessed. The most confident rating is typically something like we have an A/B test with a proven outcome that it drives a specific revenue outcome, and at the lower end it's just somebody thinks this is a good idea, which gets a much lower rating. Effort is typically t-shirt sizing, small, medium, large, extra large, and the larger the effort, typically the lower the RICE score.

[00:17:02] So we get a RICE score out of this. We use the RICE scores to stack rank priorities, so we can talk in a consistent way about what we're going to do next. Now, the important thing is that the absolute RICE score value doesn't have any meaning. If it's got a 26 or a 42, that value is not really to be interpreted; it's about stack ranking the priorities that we have. Also, the rule is you should never prioritize something just because it has the highest RICE score. The framework is just meant to introduce consistency in the conversation about value, and not to be the canonical way of determining what we do next. So you're allowed to override it as a manager, and we encourage managers to use this as something that drives the conversation but not blindly follow the number.

[00:17:56] An important call-out here is on the impact, on the dollar value assessment. What's been really valuable for us is to include our finance business partners inside of the teams in that conversation, because what we found is the structured conversation around value added a lot to the ability to prioritize the right work. But also, as soon as you have somebody with a bit of finance knowledge looking at the forecasted value, it really sharpens up the numbers and gets you to a much more accurate forecast. It also means that these forecasts get included in our business planning, which means managers that go through the process and assess a value are accountable for that value. That's very important, because otherwise people tend to drive speculative things that don't really necessarily stand up to scrutiny. So that gets the value out the door, or at least gets consistency in how you assess value.

Tracking work by type and standardizing process

[00:19:01] Tracking by work type. As I said, this is relatively controversial in King. We classify work in four broad categories. Mission work is annual objectives, OKRs; these are the strategic investments we make and commit to annually. New requests is new requests that come in from our stakeholders during the course of a quarter and a year, new opportunities. Innovation work is research spikes, POCs, trying new things, and creating time for teams to do innovative things as well. And operational work is typically BAU support of the platforms day to day.

[00:19:41] The value of the tracking, the unlock for us, was really getting an insight into all of these autonomous teams that were working independently, and it created the ability for us to provide better support for these teams, because you would typically find trends. Why is a team not able to deliver their strategic plans for the year consistently? Because some teams can't. And that is typically either because there's too much operational work happening, BAU work, because of accumulated tech debt, or there's just a sheer amount, a large amount of requests constantly coming in from stakeholders.

[00:20:27] For instance, our game economy team provides the in-app purchase platform capability inside of Candy Crush, and that's the heartbeat of our business, and that's typically a team that really struggles to consistently work on their strategic investments, because on a regular basis there are business opportunities that flow into the team and things that the game teams want to try and experiment with, and it's a constant battle for what we do next. So if you can get your teams to track work and report work in as slick a way as possible, it really unlocks the ability to support those teams better and identify these trends.

[00:21:03] Next we looked at standardizing process and assigning roles and responsibilities. This is a copy of an internal slide where we first defined our process. None of this is new or revolutionary. The value of it is having a consistent way of working without being too prescriptive on how the teams operate. So, clear roles and responsibilities. Green on the slide here is product, blue is engineering, and the color represents ownership and accountability but not participation, and that's very important when you're settling in the way of working. You'd expect the engineering manager and the PM to work really closely together on all of these. For instance, effort estimation is not something the engineering team should go away and do on their own and then give the product manager a number. A PM and EM should be working really closely together on that.

[00:22:04] This process has allowed us to standardize our work, how we track work as per the previous slide, but also, because of the consistent process, it allowed us to create a single system of record where all work gets captured, which we opted for a system built in Jira. Jira is great and horrible at the same time, as anybody who's worked with it knows, but it is the standard, and the data model that's underneath it is pretty good, so you can capture a lot of metadata about the work that you're doing and customize that.

[00:22:40] That allows you to build what we call our SDDF [?] dashboard. This is a set of reports built in Jira that allow us to answer operational questions that I alluded to at the start of the talk. Which team is working on features or has features in queue for the game economy team, or which teams are working for a specific team inside of Candy Crush, and what's the queue length, how much work we have in the backlog, and how much work is queued up for adoption. So is there work that we've completed that is ready to go but is just waiting in a backlog, which is very expensive.

[00:23:23] This has allowed us also to really optimize the way we communicate. This creates transparency. All of our stakeholders have full access to this so they can have a look and a rummage around. So in areas where our product teams have built very close relationships and transparency with the stakeholder teams, this is then a really good way for them just to see a live version of what's in flight at any moment.

[00:23:48] We also use this to drive our annual processes, and this is a bonus unlock for us. When we did annual planning for our annual budgeting, long range planning and forecasting, it was a very laborious manual process, and with this, a piece of work that would have taken a team of senior people maybe two, three weeks, a finance individual did in maybe an afternoon. Because we could just extract everything: the costed value was there, estimated effort was there, so we could work with the team sizes and calculate a forecast of roughly how much of this we could get done in the next quarter. All of that's estimated for long range planning, but it's a massive efficiency gain.

What worked, what needs improvement, what's next

[00:24:36] So what worked well. As I mentioned, the consistent language of value is the most important thing, I think. Being able to prioritize allows us and enables us to work on the right next best thing, so that we can consistently be pushing the maximum value to our players. The reduction of cost of reporting, I just mentioned that, that's a big unlock. And then effective communication just creates a happier, better informed stakeholder, which enables us to build better relationships as well.

[00:25:13] Room for improvement. The teams will game the system with the scoring and the RICE. I think people still have the things that they believe should be at the top. This is generally not a big issue as long as you've created accountability for the work that the teams prioritize and for the forecasted value of an effort.

[00:25:36] The second one is really interesting in terms of motivating teams inside of Shared Tech. Revenue is not a good way to do that. Revenue is a lagging metric for a tech team; they're relatively far removed from it. So even though using revenue forecasting as a consistent way of talking about value worked, it created a bit of a disconnect between what the teams were doing day to day and the impact of that. So we have started to look at better ways of measurement and KPIs within the teams.

[00:26:15] And then the application of the RICE model was done inside of those four domain teams that I talked about in the beginning, and you would see discrepancies between the way the teams scored. Inside of the teams it was fine, things were stack ranked, but when you compared across the teams there were some inconsistencies, and then you had to get senior managers involved to help us prioritize when cross-team collaboration was necessary.

[00:26:41] Next, as I mentioned, we're going to tweak the impact measurement. We're looking at a set of KPIs around time to value, so what are the investments we make that make the game teams move faster and enhance productivity inside of the game teams. Stack ranking across all priorities: we built a single stack rank priority for all 400 people and what they work on next, and we review that on a bi-weekly basis to help the prioritization process. And then virtual teams. This is joint resource allocation, so it doesn't matter who your line manager is; we create initiative or project based teams that include resources from the game teams and the Shared Tech teams, and then we run these as projects that have a very clear remit, and we found that a very efficient way to move fast and drive productivity.

[00:27:43] Just one quick note on change management. Really important. This is underestimated in the beginning: structured change management with dedicated people working on change management, that feedback loop, driving change. Large organizations are resistant to change, more so than we would like to think, and having a structured effort around driving messaging and collecting feedback is super important to be able to drive change in ways of working like this. And that's it. Thank you very much.

Q&A

[00:28:17] Host: Thank you very much, Jaco. There's been a few questions, thank you for those. The first one, in terms of the RICE model, when you think about impact, how do you quantify impact?

[00:28:31] Jaco: Typically impact, with the current model, is quantified in a dollar value that we get to with a ranking system. So whether it's forecasted incremental revenue that a feature would provide, or cost savings, or the estimated value of efficiency, we work with a finance partner to typically convert that to a dollar value number, and then we use that as the impact of a piece of work.

[00:29:02] Host: And when you think about the standardization and getting processes and getting teams on board, particularly the large teams that you have, how do you incentivize them to follow the same process?

[00:29:18] Jaco: That's the hardest part. Driving change within large organizations is particularly challenging, and hence the call-out on change management. We did a lot of work in putting together a value proposition and a pitch, and we socialized that over a number of weeks, speaking to individual teams and managers, until we got to a point where we felt we had the buy-in from the teams that we could start pushing the change. We also, instead of running a pilot of an end-to-end initially in one team, rolled it out across all the teams but in incremental stages, to make the change as wide as possible but the incremental amount manageable at the same time.

[00:29:58] Host: Great. Just a note for everyone: if you do ask questions, we're clearly not going to be able to get through all those questions. After this event all of the speakers are going to put those answers to those questions and it will be shared on the UXDX Slack channel, so don't be shy sending questions, and don't be upset if your question isn't answered during this phase. Rory, have we got one more?

[00:30:23] Rory: Just a quick question for you, around getting RICE embedded into a corporate culture. How far up do you go? You're talking about the product team here, but do you have to then elevate your RICE scoring to senior leadership or the exec?

[00:30:40] Jaco: That's been a learning for us, because if you put a RICE score on a slide in front of senior people, they over-index on the value that they see. I think the value of the RICE score is on the feet-on-the-ground work that happens in terms of prioritization, and supporting that process of prioritization; that's where the value really is. By the time you get to senior management, it's important that you can justify the decisions that have been made based off of the RICE scoring and be able to articulate that, but that should be articulated more in terms of the outcome that we're delivering to the business and that value, and at that point of the conversation it's going to be less about the process that supported it. So although our senior management have understanding, and we took them through the details of the RICE process, it's not really part of the conversation with senior stakeholders.

[00:31:38] Host: Great, thank you very much, Jaco. Everyone, a round of applause.