Continuous Learning: Feeding UX With Monitoring

Jasmin Verollet-Dahncke at UXDX Europe. Video: https://youtu.be/zr2NDoXce-c

Readable transcript: edited from the recording's captions for readability (fillers and false starts removed, punctuation and section headings added). Wording is the speaker's own. Timestamps are positions in the video. Names marked [?] could not be verified against the audio.

Introduction: Next Games and The Walking Dead

[00:00:07] OK, I'm going to do this by hand, because I'm using my own thing, so this won't work anyway. We are here today and I want to tell you a little bit about how we work at my company, Next Games, and this is all about monitoring: data, and how we feed it in, and how it becomes features. I'm going to walk you a little bit through a player segmentation process, what data we have been gathering, and, with the kind of insights that we gain, how we develop features. What is our process there?

[00:00:41] On my agenda: I will tell you a little bit about myself and the company that I work for. Then I will also give a quick introduction to games as a service, because that's our business model, and I think it's really important to understand what KPIs are actually important for us in this kind of model. Then it goes to the beef: the monitoring and the feature development. So let's get started, because we're already behind.

[00:01:08] A little bit about me. I've been a UX designer for a couple of years. I started my career in 2006 as a graphic designer. For a really long time I did a lot of editorial; that's actually where my love for information hierarchy comes from, crafting sensible books, and then later digital products, where readability, likeability, a sense of understanding information, is for me at the core.

[00:01:40] The company that I work for is called Next Games. It's a mobile games company. We don't necessarily come up with our own worlds; we actually work with franchises. That means we join franchise holders, people that have big IPs and create cinema movies or series of books, and together with these companies that create these fantastic products, we create the games for these IPs, to bring to the fans to enjoy games based on them.

[00:02:18] Of the IPs that we work with, the biggest one is The Walking Dead. We actually have two Walking Dead games. The latest that came out is a location-based game where you can play like Pokemon Go, but instead of gathering cute monsters you actually shoot zombies with your friends. So it's a bit different. But our first Walking Dead game was a very classic RPG strategy game, and that is the one that I'm going to talk about. I worked really long on that, and we gained a lot of insights, and that's what I'm going to share with you.

[00:02:51] I want to give you a quick 30-second trailer for the game so you get an idea of what you actually need to do. Let's take a look. Actually there is some kind of sound; OK, but if we don't have any sound, that's fine too. Basically you shoot zombies, you slash them. It's just like in The Walking Dead series. You collect these awesome heroes that you just saw yesterday in the TV show, Negan with the super cool weapon. You can collect everything around that, you can level them up; a typical story role-playing game. What is specific about the genre of this game is that it's turn-based. That means you have a lot of time for thinking. This game is not super action, where you need to do things really quickly; instead you have time to think, and it's very strategic. The elements of the game and how to advance, that is what this game is all about.

Games as a service and free-to-play

[00:03:55] OK, let's go to the next slide. Games as a service. In mobile games, this new, well, new kind of revenue model that is really popular nowadays is free-to-play. Where this all comes from is the umbrella term games as a service, and that exists not just from the last decade; it started in the 90s with all the massively multiplayer online role-playing games. So this is nothing new. It's just something that has evolved with new models inside it. Free-to-play is just one model inside games as a service.

[00:04:34] What games as a service basically means, or what it was created for, is for us to gain revenue after an initial sale of a product. Back in the day, and also nowadays there are still games where you pay $60 and that is it, even though nowadays there are a lot of updates coming that you can pay for. Usually the classic model of monetization was this initial sale. You had a really long development pipeline, you paid for the game, and that was kind of it. Then usually the development teams moved on to the next thing, which would take many years.

[00:05:10] This has changed significantly. What came with this is these really massive worlds that we are crafting, that have a really big sense of community. That is really what is there at the core. All of these free-to-play games, or really many of them, still have this massively multiplayer online gaming thing at the core. They are all live, they are all online games that get continuously developed. So this is what this business is all about: to have this kind of product. I think many of you nowadays, in all the apps, try not to have these things that we make and forget about; we learn about them now with all this data that we have and try to improve them. That is the same way that this works.

Retention, engagement and conversion

[00:06:04] The important thing about this is the free-to-play part, because with free-to-play there are very specific KPIs that are important for us. These are the three main ones. Due to our games being free-to-play, players can freely download them and play with them. There is no initial sale. We haven't converted anyone; we haven't gained any monetary value in that sense. So we need to retain these players, which means that we need them to come back to our game. We need to engage them with our content. And then in the end, if we did things right and people actually had fun and had a good time, we can convert them into paying users, and this is where all of this clicks, and that's where the business lies.

[00:06:55] That is really important, because the monitoring that I will show you, the data that we collect, is all about how we improve this. How can we get our business to thrive? It is these three KPIs, key performance indicators, that we really care about. Engagement really stands for fun, and it's the same for retention. You come back to a product because you had a good experience. That is all what we are trying to do: create a good product, just like any one of you.

[00:07:31] We also have a little bit of, how would I say this: many of you work on apps that actually solve problems. Maybe with your app I can call a cab, or it takes care of my banking. You solve problems, so there's a bit more value, or I'm maybe even, to a certain extent, dependent. You make my life easier. But the products that we are doing are pure entertainment, so it's a little bit easier for people to drop them. Someone can download our game, and then either there were technical problems at the start, or they didn't understand something, or it just wasn't as engaging, and it's very easy to drop. So for us this whole retention thing, getting people to come back, is even harder. No one relies on having these apps. That is good to know.

[00:08:32] Something about the market also makes this whole retention thing important. I don't know how it is for you, but for us it's extremely expensive to do user acquisition. I think the current dollar value per user is around $8. And due to the nature of games, a lot of people churn. They try the game and they're like, "It's not actually what I was looking for, I'm not as interested." So just imagine this huge amount of users that you paid for coming to the game who don't stay. That is not good in this case, so that's why we work on these kinds of things.

Daily metrics and the need for player types

[00:09:16] So let's move forward to the monitoring. We have the same, probably, as you. We have daily metrics: daily active users, how many new users come in, what the average revenue is, and then of course retention, which is really important for us. Due to the nature of games, a player who logs in on day one has a very different experience from a player at day 730. So this metric is really important to us, because a player, due to the game's nature, has a different experience, because the game changes. You progress through our product at a very fast pace at the start. You unlock new features. The whole product can actually look visually very different to you depending on what day, or at what point of progression, you are. So there are a lot of very specific learnings and very specific parts of the game and the flow and the experience that we actually need to care about and look at.

[00:10:18] But the daily metrics only go so far. Of course, on a day-to-day basis you can find, "Hey, you rolled out your update, and something changed in the retention." Maybe there are technical issues, there are bugs; you can find stuff. But how do you really improve these basic KPIs, to retain players, to engage them, and maybe to convert them? For that you need to actually learn about your audience. You need to find the segments: who are these people that are actually interacting with your product? That's where we needed to find a way to find out about them.

[00:11:00] We came up with this initial set of questions about how we could go about identifying these audience segments. We also call them player types. So we asked questions. As you saw earlier, maybe, in the video, there are a lot of ways to fight against zombies, and we have a lot of features that do that, and they are all missions, different mission types. These are our main features where there's actually some kind of action. You interact with the gaming world, you do your strategy. It's also called second-to-second, the second-to-second kind of gameplay, where you maybe have most of the fun, let's say it like that.

[00:11:47] Anyway, we have a lot of these features that do similar things but in different ways. So we were like, "All right, what missions do players play? What is the favorite way of slashing these zombies?" Because there are so many; maybe by looking at that we can find out. From these questions more questions came up. For example, how do these players progress, related to this player type? Because there's a lot going on in there. Then there is also, of course, the long run. Again, due to the nature of a game that changes drastically throughout the lifetime of these players, with different things happening and unlocking, how do they transition between these player types? An early segment of players looks very different and has a very different experience, so when they move on and get more experienced, how do these players look and what are they doing? And then of course we also have business-related questions, like how are these players spending? We need to know.

What the segmentation found

[00:12:56] So we did this approach. We took one week of data. The main measure was missions played per player, and what kind of missions. It's a snapshot of the overall game at that time. That was the first thing that we set out to do, and this is what we found: buckets. The first bucket had four different player types, and they varied. This is a high engagement group. They played a lot of missions, which means they actually had fun. They loved the content that they engaged with, that they saw. They played, they kept coming back, and it also meant that they spent more. And then there was a second bucket with this kind of "a ready" [?] player. They weren't as engaged. They played very few missions, and of course they spent less. So we were like, "OK, we need to learn about them."

[00:13:56] A little bit more detail about these five player types that we found out about. In the engaged and highly engaged bucket there was the story player. Story mode is the very classic mode in games. It's curated content, where we have little bits of the Walking Dead world. We wrote our own things there, and you fight against zombies. The other mission feature we had was the grind mission. You repeat a similar mission to get a certain kind of item at the end of it. They loved it.

[00:14:32] The next audience segment that we found was the challenge player, where you join together with others to fight against the zombies and get points together to get really cool rewards. And the last ones that were really highly engaged were PvP players, players against players, and they were also extremely engaged. Something that we also learned at that point about the challenge and the PvP people: people who interact with people, and it doesn't matter if they play against each other or if they cooperate with other people, any kind of interaction with a real person leads to higher engagement. That was really interesting to see. These people who were in the features that set you up to work together or play against others had really great fun. That was the best: they created these skills together and then played against others. That worked really well.

The sleepers, and story players who became sleepers

[00:15:34] But then there was this one segment that didn't perform. We called them the sleepers. The sleeper was the low engagement group. So we needed to find out: what is it with these guys? What are the problems here? What is happening in the experience? What questions can we ask? The main thing with the data that we now had was: maybe it's not so bad. Maybe these sleepers actually become one of the highly engaged players. They become a story player, maybe they become a challenge player. So we wanted to find out about that. We looked into the data at how they transition between these different player types that we had just learned about, and it looked really grim. Instead of becoming one of these highly engaged players, they either stayed in the sleeper group, or worse, they churned. Churn in gaming means they quit your product and they never come back.

[00:16:45] This was the very first sign that, OK, we actually have a really big problem. We need to look deeper. So we needed to think: what else do we actually know about them by now? We knew that they were new players, because they had very low levels in the progression of the characters and in the gaming content. We knew that the mission they mostly engaged with was our story, and we knew that they churned really quickly and left our game.

[00:17:20] The one thing again that we had to go on here, from the data that we had already gathered, is story. We had pretty and highly engaged story players. This feature was performing, to our best knowledge. So going back to these player types, we were like, "OK, let's do the same kind of thing with the story player. Let's find out how story players transition between these player types. Do they become challenge players, grind players, or in the worst-case scenario, sleepers?" Well, it turns out that was the case. Even worse, instead of becoming highly engaged, they became sleepers, or worse, they left the game. After we saw that they were performing, playing what they loved, so what's going on, that these highly engaged players who love the product leave? And here again, to show it, it's pretty much the same. They had exactly the same behavior as a sleeper. So now we knew: OK, we have a problem.

Handing the problem to the game team

[00:18:38] So how do we go forward, and how do we do it? Now we have something to go on from the data analysts. Also, at any time while this kind of investigation is going on, the game designers and UX designers are usually involved in these findings, and help with formulating questions regarding the product, to find out a little bit more about these specific segments of your product's users.

[00:19:06] What we needed to investigate, what we learned from this, is that the early player flow is something that we should maybe take a closer look at. Because it's these early game players who become sleepers, that's what we found, and it's the story feature that is mostly used in the early game, because it is actually the very first mission feature that we introduce. So we have something to go on, and this is the point where we hand this over to the design team, to the game team, to actually take action and take this further. This is where I'm going to walk you through a little bit of how we do this, on a high level.

[00:19:47] So now we have something. We had an initial problem from data sets. We know there are these specific audience segments that are not performing, that have issues when playing or navigating through our product. How do we find out about this? This is when we start to look into more detail at, as I said, this early mission flow. What is actually happening there?

[00:20:11] This is also always a really good time to check in with your team, because, I'm not really sure how it is for you, but we really need to use our product every day. We test our product; everyone in the team tests almost every day. We have new builds with new features and small updates that we are working on, and we all make sure of what the experience is. On top of that we also play our games outside of this development cycle, in the live environment. So the team has probably gone through the same kinds of experiences that these players have gone through. There are usually a lot of really good insights and learnings, and I think it's probably the same for you. If you have tried out your own products, there are these very specific pain points where you are like, "Hey, I never got this. Have you considered this?"

Product goals and what the investigation found

[00:21:20] At this point we also define product goals. That means: what are the KPIs that we want to improve, and with what kind of ideas? This helps us create design pillars, top-level things we can return to when we need to make tough decisions. Sometimes development cycles can become long, and you might lose or forget why you initially set out to do this, so that really helps. We are still in the investigation phase, but we start defining these product goals: what are the initial problems that we're trying to solve?

[00:22:04] The main findings that we had, where we really needed to take a good look at the underlying systems of our game, were that early in the story mode, for example, the progress slowed down, and that was due to the difficulty rising. These are really specific game mechanics that underlie the games, how things really work, just by numbers. So they were easy for us to tweak and play test, to improve. That was really good. We found out it's getting really hard early in the story missions to play. And if we wanted to have this intended experience, then there should be something else that the players can engage with, but that wasn't the case.

[00:22:57] We also found out, from looking at these underlying game mechanics and from play testing ourselves, that other features were actually not unlocked. There was nothing else to go on. There was no other experience for you besides being punished in this one mode that just got way too hard. So of course I'm going to drop this thing; I don't have a lot of fun. These were the things that we found.

[00:23:25] Another one that we also wanted to address was that, besides things not being unlocked and things being too hard, there was something that they could do, but people actually didn't find the feature. This is also something that I learned from discussing with the team, and also from observing them playing, interacting with the product. I could see how people from the development team couldn't even find certain features. I asked them, "Hey, can you show me that screen?" and they couldn't quite get it right. This is a big red flag. If your own team can't navigate the product correctly, you should flag that right away.

Icons, text deserts and the feature brief

[00:24:12] Now we had some really main things, and we were like, "OK, definitely let's address all these." Here is also a picture of what our game looked like early, maybe two years ago. You had a very basic set of navigation with only icons when you started the game. There was this kind of camp that we knew from other games too. And this is one of these usability principles where we just didn't know better at the time: people couldn't recognize these icons. They couldn't remember what these things actually do, and there were a lot of them.

[00:24:54] The other thing that we had, besides this initial problem that people couldn't remember what these icons do: if you clicked on one of these to get into a really cool feature and do something cool, there were these vast deserts of text, and they were maybe not very pretty and not really engaging. So besides maybe forgetting what something does, you don't even get into a way to play anytime soon, because you are presented with layers of information that maybe weren't as important, and you still needed to press some buttons to actually get to the content that you wanted to play, to actually play the game. Those were things that we found out when looking at the product and playing with it.

[00:25:47] Something that we do that I can really recommend: I know documentation is often painful, and it gets outdated really quickly, especially when you work in agile and you iterate really fast. But it doesn't need to stay up to date. All you need is the basic description, some product goals, maybe some user stories, and later you can add wireframes. It doesn't even need to be as much as what we have put there, because this was a massive feature. The more vast or big this thing is that you do, the more info you have to go on, the better, of course. Because, as I said, these product goals and this feature brief work like a design pillar in the end. When you create these features you can go back later, if you have to make tough calls: "Hey, why did we do this? What were the initial goals that we set out to achieve?" That will help you make these tough calls at a later stage.

Wireframes, team feedback and the new hub

[00:26:49] Then we do a lot of the usual; I think it's the same for many of you. We do a lot of wireframes and sketching things out. Back then we still went to Photoshop; nowadays we do a lot in Sketch. And while we do this, we share a lot of these with the team too. We have a really constant feedback loop. I know it's not easy to share a lot of these things that you work on when they are not ready, but the earlier people can point out to you what could be problematic, the better. The more people, the better. I know it's hard, there's usually a lot of feedback, but it's definitely worth it.

[00:27:38] This is what we then ended up with. Besides all the underlying numbers that we addressed, like decreasing the difficulty in the story and unlocking other features earlier so players could interact with them, we also addressed the visual problem, and that was one of the biggest things that we did. We introduced this kind of accessibility hub. Instead of these scattered icons with features hidden behind them that do certain things, we created this one place: "Hey, these are all the cool things that you can actually do in our game. You can find them all here." We worked on the goal set, and you can see what kind of rewards you get. You can even see when they would unlock; there are timers and what have you. We also have other content features that you can play. So this is how we addressed these initial findings of people not finding things.

[00:28:40] In the end, all of these things we set out to do, but do you know that they will work? I feel like, OK, we just go by our best assumptions and by our experience, but you will only know later, when these products go live, whether you actually achieved what you wanted. Did it solve the problem? That's where I also feel it's hard to do these things. You go by your best assumption, you check in with your people, and you hope you hit home.

Team presentations and feature teams

[00:29:15] What we do then: we have this thing, well, actually we're not quite there, I'm still going through one round of iteration, but many of these things happen at the same time. What I like to do, especially when something is, well, this was a massive feature, it was a massive change to the game, is team presentations. We share with the team what is happening. We share the mock-ups, we share the product goals. Everyone is part of this journey of understanding why we are doing this. They can challenge your assumptions, they will challenge your designs, which is really a great help. If it's smaller things, you can just share them in the Slack channel. As long as there's some kind of tool where people can give feedback, I think that's really helpful.

[00:30:03] This is the initial reaction, at least from our teams: they have a lot of questions, which are usually really good things that you just haven't thought about. There's always something you haven't thought about. There's always something where, in your view, "Hey, I made it this way, so you use it this way," and then it turns out that they see what you've done and they think it does something completely different. It really helps a great deal. So you do a lot of this iteration with the team. You keep sharing the iterations, and the team stays involved at all times. You will never get to the point where everyone is happy, but you will get to the point where you feel, "OK, I have addressed the biggest issues that people have pointed out," and the rest needs to be verified when maybe the product goes live.

[00:30:58] Then there is another tool that we are using to make sure that these features that we have so meticulously planned out really make it into the game at a certain quality. It's called a feature team. Back in the day we picked people from the team who made features but also a couple of other things. We now have a team that actually takes the ownership. They are doing this one thing. They have the complete responsibility for shipping this feature and also iterating on it while developing. They do the testing, everything. They make sure that what we have set out with the design team, and with the team in general, is also what ends up in the game. Because usually you build something and there are also more learnings, so you want to have those things in there.

[00:31:55] Then they perform something called a component breakdown. That is just a functional spec: what kind of patterns do we need, what kind of logic is necessary for all these screens? From that, we make sure: is this feasible, and in what amount of time? All of these things are really important. The more ownership the people in your team have, the more, well, they will struggle with some stuff, but the better people can learn, if it's their responsibility to do these things.

Implementation, the redesign and going live

[00:32:28] And then we do the same, I think, as many of you. We implement, we iterate a lot, we do a lot of bug fixing, we polish. We check in with the other teams in-house. We maybe let your mother play, or your friends; I think that really helps. And then it all goes through QA.

[00:32:48] If you remember, this is what our game initially looked like, with these icons where people didn't know what hides behind them. We finally added text to them, so no one will ever forget what any of these are about. The other thing, if you saw, there was one feature in the bottom right that was leading to the story missions. We now instead have this hub where we have collected all of them. Initially it looked like this, and this is how it goes: you set out to do the thing, and this is what's actually in the game. A lot of things happen from this initial draft and the initial thing that you set out to address. There are usually a lot of business things that come in there that are important, or also feedback: maybe we want to have some kind of ads there, and we also need some timers. This is how things change throughout the development cycle, and also through being live and getting feedback.

[00:33:53] And you remember that when you opened the feature you had this unreadable kind of screen with all kinds of inputs in it. Now instead, when you click on one of these tabs, you are right away in the feature. This is how all the missions look. Any feature with a mission that you would interact with has the same kind of look, so you can right away start and get into the action of these features. There are a little bit fewer things in between, and it eases the flow for some of these things.

Learnings after release

[00:34:26] And, yeah, the big release: it's there. I have to say the big learnings from these stages: allocate time, not only to review the data again. For us, due to the nature of games, and because we work with franchises, we get a lot of feedback about what people love or hate. They are extremely opinionated and engaged. So we actually need to allocate a lot of time, at least one week or more, to come back to what we have done and what the perceived experience is for all of these players. Besides them writing about this, there's of course the new data that comes in, which shows whether the assumptions that you made are correct or not.

[00:35:21] Yes, I need to speed up, I can see. In games it's really hard. Again, you set out to do something, you try to verify with your team, and you find out that sometimes what you did is confusing, and things that you left out, that you thought were OK, in the end are missing. You need to allocate time to bring these back.

Summary and takeaways

[00:35:40] So what did we do? We monitored. Besides the daily data that you receive, try to make some snapshots and learn a bit more about the users that you are catering towards. How do their journeys look from day to day, from week to week, in very specific segments? You can learn a lot from this. Do a proper investigation, and involve everyone. There are great learnings among your team; everyone who works on your product knows it, and they might have really good ideas, so just involve them. And then in the end also test with them, and keep iterating.

[00:36:18] My biggest takeaways are: create really strong relationships between your analysts and the game designers or UX designers. There is so much potential there, it's really incredible, and I think it can always be improved, because often they might be working too siloed. Also, don't slack on the documentation. I know it's painful and it doesn't stay up to date, but it really helps you in the long run to see whether you designed what you set out to do at the start. And give your teams full ownership; there is no loss there. That's all. Thank you.