How Teams Integrate Design, Product & Devops

Ross McKinley, Anna Sitnikova, Anna-Karin Fossum at UXDX Community: Stockholm. Video: https://youtu.be/Z9p6l9vyCQ0

Readable transcript: edited from the recording's captions for readability (fillers and false starts removed, punctuation and section headings added). Wording is the speaker's own. Timestamps are positions in the video. Names marked [?] could not be verified against the audio.

How close are DevOps, product and UX in your teams?

[00:00:12] Host: The purpose of the talk today is about integrating teams: product, UX, DevOps. I'll start with you, Ross. Do you have a microphone in your hand? DevOps is at the end of the delivery cycle, I guess it's the releasing part. How close are your teams in terms of interacting with product and UX and understanding, from a DevOps perspective, all the insights and the research that goes on into the work that you do there?

[00:00:37] Ross: Right. So at DICE we've got a whole division which is focusing on research and prototyping, and we also build those prototypes. So we're involved right at the very beginning of the process of creating new games, creating new ways of playing, or using new features. We interact directly with the content creators, other technology folks, with the product directors in terms of where does this product want to go, and then what sort of new features.

[00:01:26] Host: Can I ask you the same? You're coming from the UX side, so how integrated are you with your developers and your DevOps teams?

[00:01:32] Anna-Karin: Right at the start of the program we always start with user research, and we're trying to involve all sorts of developers in that research phase, especially when it comes to prototyping and what kind of direction the project is going.

[00:01:50] Host: And Anna, from your perspective?

[00:01:54] Anna: Well, we tried different approaches. First of all, you have to understand that every team in our company works differently, so they choose what suits them the best, meaning they're still delivering the results but they organize around those values, those missions, in the way they want to work. So if you want to have UX research upfront they are free to do it, and we try to do it, but for my team specifically and for several other teams it didn't work.

[00:02:24] Anna: So we are a fully integrated team, cross-functional and quite autonomous, and right now UX and design as well as developers and analysts, agile coach, they're all team members of the same team. They're 100% to the same team and they're working toward the solution. It means when we're going out to talk to our customers and we're doing some research, everyone participates, everyone who is available at the moment with this phase. So let's say we need to do a survey with some customers, and the UX person might go there with an engineer because this engineer is available and also trying to achieve the same goal. So I would say fully integrated.

[00:03:25] Anna-Karin: We also tried the part where we actually involved the developers in the user research and usability testing, to watch or actually be part of the analysis, which is a very good thing to do because then they feel a bigger ownership of the mission.

The challenges of autonomous teams

[00:03:46] Host: One of the things I always find is when you hear stories of agile[?] and how companies are working, everything's perfect and everything's working great, but the reality is a little bit less clean and tidy in the background. So can you share some of the challenges that you found with that kind of autonomous team?

[00:04:07] Anna: I would say, if you put it into one challenge, probably making the team accept the fact that we have one backlog. So the team working towards one backlog, which is a representation of our mission, so basically how we're going to achieve those goals and missions. And making the organization understand that this backlog is one solid thing, and that the prioritization of this backlog goes by the product manager together as a team.

[00:04:42] Anna: It sounds like an easy thing, but the main challenge here is that eventually everyone in the team starts to think things like, well, my chapter would like to deliver something for the sake of a platform, or my peers in the UX group are doing this amazing research, I would like to join. No, you might, but that has to be accepted by the entire team and it has to be aligned with our missions. Because if it's not, then people start to work on many different things and you're never going to reach your goals.

[00:05:25] Anna: So as soon as your team members accept the thing that by bringing a small thing, or by introducing a new thing to the backlog, I might actually put the entire sprint at risk, if they start thinking this way then they immediately start to say no to a lot of things which are coming from the side. And it's a big challenge, because it's a lot of collaboration and communication, and it's a lot of confidence as well.

[00:05:56] Host: Thank you.

[00:05:59] Ross: I can jump in. At DICE we're about two years into our organizational transformation to deliver games faster to players, and one of the biggest challenges that my team faces is that when the process that exists doesn't work for us, we just throw it away and we try to find something that works specifically for us. Sometimes our challenge in that is about communicating outwards to the rest of the organization, to say, hey, this thing that everyone else is doing just doesn't work for this new team that you've created, and we're one of the key teams in this organizational transformation. A lot of folks just don't get that this way that we always used to work, that everyone is very accustomed to, is just not suitable. I think again it comes down to communication. That's definitely one of our key challenges.

[00:07:15] Host: Do you have any examples of something you threw out?

[00:07:20] Ross: Yeah, we don't keep a backlog anymore. We just work on the most important thing at the time. We found that our team was specifically under a huge amount of pressure, so our backlog just kept on growing and it was just not actually worth the overhead to do any effort there. So we just stopped and started working on our own stuff, and that has been super successful. We align the whole team around our core mission, which is to help DICE ship games, and everything that doesn't align with that, we just don't do. I think that has taken the rest of the organization a lot of effort to understand, and we did a terrible job of communicating that to the organization.

[00:08:31] Host: Is that an expression of that? [?] And do you have examples of challenges that you face? It's not all rosy.

[00:08:52] Anna-Karin: I was thinking about, from a user experience perspective, when you do the usability testing and you realize that you have to start to make some major design decisions and start all over again, and the developer has started maybe some kind of path. So I think that's a challenge: how we prioritize that, and also how you really fit that into the roadmap.

A/B testing: when the result is negative, and going from zero to two

[00:09:19] Host: Just touching on A/B testing there. Because A/B testing can happen at both ends of the cycle: you can A/B test before you build something, and then you may A/B test after you've released to production, so it kind of touches across all areas. And it ties in with the theme of hypothesis-driven development. What challenges specifically? You've touched on a challenge there. Is there anything else that you find with A/Bs, particularly if you don't get the results you're expecting or something like that?

[00:09:50] Anna-Karin: Yeah, we always try to do usability testing and user research, but then we also do A/B testing. And the challenge with A/B testing is really when it's negative, so you have to take it back and then you start the design process again and try to figure out also why it is negative, which is a lot of discussions.

[00:10:17] Anna: I would say my team, in the company at all, is the team that runs the most A/B tests. Alice[?] is smiling because he used to be working with us. We were actually building a lot of the foundations for how we run A/B tests. But the thing is, you learn this: when you go from zero to two A/B tests, that's the hardest journey. When you scale from two to many, it's much simpler. So yeah, we've successfully, I would say, managed this journey from zero to two.

[00:10:52] Anna: And our biggest challenge was actually how to allocate those customer journeys to a specific group of customers. We didn't want them to experience different parts of an A/B test. If you're in group A, we don't want you to suddenly see a flow from group B, C, E, whatever, however many flows we have. And we didn't find any tools out there which would help us to actually distribute those different journeys. So we needed to invest quite heavily into our own infrastructure, and we built the internal tool, Experimenter we call it, which is, right at the moment of your registration when you create an account, you're actually assigned a specific journey which follows you through the entire lifetime with us.

[00:11:48] Anna: And that was kind of a game changer, because before we were spending so much time in understanding, okay, this conversion looks better than that, but what did this specific user actually see, what kind of steps did they complete, where did they spend the longest time, and what do those kinds of users do next? And I guess it comes to prioritization. If you're willing to prioritize this kind of stuff, then you're going to end up with something, but maybe it doesn't work for everyone. Maybe not everyone needs that many tests.

[00:12:33] Host: So with the Experimenter, it keeps with somebody through their entire journey. How do you manage them? So you've run a few tests now, you want to roll one out. Does that finish, or do they have it over time? Because tests are going to keep multiplying and multiplying on everybody.

The five-minute thing that turned into a three-week cleanup

[00:12:54] Anna: This is a very, very good question. When we decide what to do with a specific group, we almost always try to think not just about A and B but also about who is in the middle of some journey. So in fact it's ending up with multiple groups, and we need to finish the experiment at some point. And with that, actually, it's not always been like this, because we once had a huge failure.

[00:13:24] Anna: We were running an experiment on a specific market where we were removing one of the steps of onboarding, and usually removal is more happy than introducing something, because you're actually killing something. So the experiment went quite well and it was clearly visible that we needed to roll out the experimental flow to the entire user base. But there was a group of users who were between chairs. They started this step like five years ago, or during the five years, and they never finished it.

[00:14:10] Anna: And that's where my favorite example of five-minute things comes up. There is no five-minute thing in product development. One of our developers was sitting with our stakeholder and they were happily discussing what to do next, and they decided to do a cleanup. So they decided to change the status of those users who were between chairs, who changed the step, to change their status from pending to okay, which was done immediately, because it's just a five-minute thing.

[00:14:43] Anna: So this five-minute thing ended up in a three-week cleanup, and our support lines were overloaded. From a user perspective it looked like this: you started with us maybe five years ago, you were starting onboarding somewhere, you collected five kronor from your customer, never finished onboarding and forgot about us. And five years after, boom, five kronor landing in your account. So, five-minute things ended up in a huge disaster, and that's what we joke around about: there's no five-minute thing.

[00:15:15] Host: Sorry, just touching on the same topic but coming at it from the other side. So you built your own product to manage A/B testing. How do you manage them from a technical perspective at DICE?

[00:15:29] Ross: Yeah, I think there are many strategies here, and I think you touched on it earlier. From the technical perspective, going from zero to introducing one A/B test is actually the significant, important thing. You need to start building up all the technology to support this long before you can launch your first A/B test. And once you have that created, you've actually gone through that journey of, how do we rerun the test, how do we keep measures and data, how do we even interpret any of this data? And then going from, okay, we've got one successful test, to now we're going to launch three or four, that becomes way easier over time. And then you get into that situation where you've got hundreds of tests running.

[00:16:29] Host: Any questions? No? Okay, so we are over time.