Experimentation for Product Managers - A Quick Guide

Sebastian Straube, Claus Reinhard at UXDX Community: EMEA Series: Design Process. Video: https://www.youtube.com/watch?v=mOdwXIOQwbc

Readable transcript: edited from the recording's captions for readability (fillers and false starts removed, punctuation and section headings added). Wording is the speaker's own. Timestamps are positions in the video. Names marked [?] could not be verified against the audio.

Introductions

[00:00:00] Sebastian: Welcome, everybody, to our talk about experimentation for product managers, a quick guide. First of all, we would like to introduce ourselves. My name is Sebastian. I'm a business agility and product coach at Accenture Business Agility, which is the unit within the large Accenture universe that focuses on large-scale agile transformations. Prior to that I was a product manager and product owner for different products, mainly in the field of e-commerce, but also in automotive, for example on a car configurator team, and also for different apps. My personal fields of interest are especially product discovery, product vision and strategy, and agility at scale. And with that, over to Claus.

[00:00:52] Claus: Thanks, Sebastian. Hi everyone, my name is Claus. Like Sebastian, I'm part of Accenture Business Agility, and I'm also a business agility and product coach, as Sebastian already stated. My special areas of expertise and interest are definitely in the area of learning and enablement, as well as product development and delivery, and I'm very happy to have this talk with you today.

Every step of product management rests on hypotheses

[00:01:27] Let's jump directly into our first content slide, which is a quote I brought to you from Eric Ries, the author of The Lean Startup. I guess most of you will know him or his book. The quote states: the only way to win is to learn faster than anyone else. How this is connected to the topic of experimentation for product managers, we will explore together in the next few minutes and the next few slides.

[00:02:03] If we look at product management in general, we have to admit it's quite a complex thing to do. I've brought in the design thinking double diamond, which I'm sure you know. We start to discover and define, and go on to develop and deliver, diverging and converging twice. This is a good example to see at a high level how much complexity there actually is within product management, because if we are working in product management, we are always working with hypotheses. Whether we do this explicitly or implicitly doesn't matter; we have a hypothesis at every single step of our product development.

[00:02:58] For example, if we are in discovery, we might have the hypothesis that we are actually asking the right questions to the right people. Then, when we decide on a problem to solve, we have the input from our discovery work, but there are still underlying assumptions, and the hypothesis that we are actually trying to solve the right problem with what we are picking and choosing. Once we have come to a well-defined problem and we're going into the solution area, putting in options for how we could solve that problem, then for every option on our plate we have the underlying assumption that this is a good approach to solve the problem. Otherwise we wouldn't pursue it.

[00:03:56] And when we decide on the approach and we are actually going to deliver it, there's still the underlying assumption and hypothesis that our solution will actually lead to the desired outcomes for our users and our customers. What makes product management such a complex art, I would say, is that it's not enough to be right at one point, with one of your hypotheses. For a successful product, you need to be right all the way through. Only if you're right all the way through will you actually have a product that makes a difference, a product that has the potential to be successful in the end.

[00:04:43] And of course, the sooner we learn that we are going the wrong way, the sooner we can adjust our course. If we learn too late that our hypotheses have been wrong, then we are wasting not only money and time, we are also endangering the trust that our users give to us and our product, and this might lead to the ultimate failure of our product, as we've seen in many, many examples.

Why products fail

[00:05:25] I've brought you a few examples of products that have failed. If you could click one more? Yes, perfect. I guess you know a lot of them. It doesn't matter whether we're talking about Google Plus, or Amazon's Fire Phone, or the experiment by Facebook to come up with some proprietary desktop screens and personal computers of their own. All of those have failed, and they have not failed because of technological challenges, or because the users didn't get it. They failed because assumptions and hypotheses that were made were falsified way too late in the process. Way too late we learned that this is just not going to work.

[00:06:29] By then we were already down the line. We had already invested quite a lot, and the only viable solution that remained was pulling the plug and not wasting any more money on those products.

Learning efficiency

[00:06:46] Jumping further, of course we have to ask: how can we avoid that happening to the products that we are building? The good answer is that there's actually an easy recipe for how we can try to avoid that, and the goal is learning efficiency. It might look a little bit complicated in this picture, but let me guide you through it. We define learning efficiency as learning precision divided by the required time and the required money that we have to invest. In other words, how much bang do we get for the buck, or how much learning do we get for our invested time and money?

[00:07:48] Learning precision, in our connotation, has a few different aspects to it. There's the amount of learning, there's the relevancy of learning, and there's the preciseness of learning. Do I learn enough? Do I learn the things that will really help me decide whether something is going in the right direction or not? And how accurate is what I've learned; how much can I actually trust my learning? That is learning precision. Now, if we imagine how we want to increase learning efficiency, we have three variables. Pretty easy: we could either increase learning precision, or we could try to decrease the time and money invested in our learning. All three variables, if handled correctly, will lead to higher learning efficiency.

Five steps to higher learning efficiency

[00:08:50] Okay, easier said than done. If we look at a few steps for how we can actually achieve that, we'll see that there is no rocket science behind it. It's actually pretty straightforward. We've brought you a few steps that can help you increase learning efficiency tremendously. The first one is to work with written hypotheses. Write down what your hypotheses currently are.

[00:09:24] As I mentioned in the beginning, we always work with hypotheses in product development, but most of the time I observe that they are implicit, and there are a few dangers to having implicit hypotheses. The first one is that if you have three or four or five people working on the same product and you have only implicit hypotheses, who ensures that you're all thinking the same? Who ensures that you all have the same hypotheses that you want to follow and verify, and that you're actually aiming for the same outcome? The other thing is that if we don't make hypotheses explicit, we quickly forget them. Being in the nitty-gritty details every day, we forget what we thought two weeks ago would be important.

[00:10:20] So the first thing, very easy: write down the hypotheses that are currently of the most importance for you to clarify. Our second step would be using tools, methods, frameworks, workshop formats, whatever is handy to you, in order to test these hypotheses, verify or falsify them, as quickly and inexpensively as possible. Choose from the variety of options that you have to address your hypotheses and learn from them, and try to choose a quick and inexpensive way.

[00:11:06] Now comes a very important next step. You've written down your hypotheses and you've tested them. Now what you have to do, of course, is course-correct based on your learnings. What's important for me is to course-correct from two perspectives. I think it's clear to everyone that you have to course-correct based on your underlying hypotheses. If you come to the result that whatever your assumption was is wrong, then great, you have to change that, and you have to think about what that means for your product.

[00:11:54] But what I would also recommend is to take a few moments and reflect on the method that you used. If you look again at the left-hand side of the picture, think about the time and the money I had to put in. Was this in a good relation to what I've learned from it? How accurate and how relevant is what I've learned with this method, with this tool? Think about whether you want to keep it or drop it. And if you want to keep it, think about whether there is any way you can increase your learning precision or decrease the required cost when you repeat that method the next time.

[00:12:44] The next step would be to minimize the time that you require for those learning loops. If you're doing this constantly, and you're reflecting both on your product hypotheses and on the methods you're using, you will automatically come to a point where you're getting more used to the methods and to the complete process. Then you will reduce the time you need and shorten your learning loops, which is great, because this will give you insights earlier, and insights earlier equal money saved.

[00:13:29] And then the last step, very important, and the only difficult one, because the other ones are very straightforward: you have to hang on. It is a thing of discipline, to really keep on doing what you're doing, keep on the path of improving your learning efficiency, and not drop it after two or three weeks of trying it out.

A toolbox of methods

[00:14:02] I've talked about picking and choosing from tools you can use to test your hypotheses. If we look at the toolbox that we have, we see this is not really a challenge. It might be a challenge to decide which one you like more, but it's not a challenge to find suitable ways of testing your hypotheses, because many, many people have put a lot of thought into how to test and validate assumptions. I've put some of the ones from my two books here on this slide. This is by no means a complete list of possibilities, and we could probably exchange for hours on different formats for how we can learn and test our assumptions at different stages of our product development.

[00:15:07] But the important thing is that it's not about the tool or the method that you're using. It's more about what fits your context, your stakeholders, and also your preferences. Then keep it as an actual toolbox. Try to keep rust off your tools, try to exchange broken tools for new ones that serve the purpose better, and try to hang on to the best tools you've already got in your toolbox, which you can use every day to do your work. So if choosing a method and tool is not the challenge, there has to be another challenge, and Sebastian will now talk to you about exactly the challenge that we see.

Matching methods to the stage of development

[00:16:09] Sebastian: Exactly. This challenge is about applying the methods we have correctly. It's important to consider which hypothesis I want to test in which step of product or product feature development. At the beginning, I'm at the point where I would like to understand the problems of my target audience in order to derive certain opportunities. Here I'm still very far away from developing actual code, and methods like problem interviews, for example, make the most sense. This is really the point where I would like to understand a problem as precisely as possible, including its context: in which situations does the problem occur, and how is the target group solving this problem right now, with certain alternatives or manual workarounds they use? Accordingly, the effort in this step is quite low. I only have to invite subjects from my target group and think of an interview script.

[00:17:21] When we move on to the second step, I want to verify that it's really worthwhile to solve a certain user problem. Here I can use methods from the previous slide, like jobs to be done, or I can create and analyze customer journeys. The effort here is already a bit higher, because usually I produce the first UX artifacts here.

[00:17:47] Now we leave the problem space and move over into the solution space. Once we have verified the user problem, it's about testing alternative solutions. One possibility here is to develop various clickable prototypes and test them against each other in user experience labs with test persons, maybe representatives of my defined personas. Again, the effort here is higher as I develop prototypes, and the more realistic, or as we say the higher the fidelity of the prototypes, the higher the effort, of course.

[00:18:31] Then in the very last step, I only start producing actual code. That means here I really develop a feature, and then I test the impact of that feature, for example in an A/B test in my real product. That's what we mean by code-based experiments. To summarize, the goal is to carry out tests and experiments as early as possible with less effort. As we've seen in the formula, this increases our learning efficiency and provides us with relevant findings early on, which, in case of doubt, prevents me from producing expensive code for features that the customer does not use in the end. If we start testing our hypotheses very early, we have a good chance to save the money and time we would have put into going in one direction.

Case study: a lifestyle companion app in Hamburg

[00:19:30] I've brought along a concrete example of a product where I accompanied the team as a product coach. At the time we developed "I know I know" [?], a lifestyle companion app for young people in Hamburg, the second biggest city in Germany. The background was that we had a German savings bank that was increasingly losing its connection to the younger target group, and it was thinking of new ways to revitalize this connection and also learn more about the younger target group, in order to later develop financial products and services specifically addressing younger people.

[00:20:16] When we started the product development process, our initial hypothesis was that young people often feel lost when starting out in a new city. Young people have just finished school and are moving out from home, most likely to a bigger city where they want to start studying or start their first job, and our hypothesis was that in this case they often feel lost when starting out. What we did was recruit representatives of our personas via Facebook. Facebook made it quite easy to target or filter by specific age and also by city, Hamburg. We invited them to our office and conducted problem-solution interviews to see whether the prospects confirmed our hypothesis. We gave ourselves a certain threshold number, and this number was reached, so for us we validated that we had a problem-solution fit here.

[00:21:24] Then in the second step we refined our hypothesis further. We made it more specific about what young people are looking for, and we said they are looking for going-out tips, perks and career advice. Again we conducted group interviews in our office, and based on those interviews we wanted to derive so-called jobs to be done: what jobs is this target group trying to accomplish? Then we matched the jobs with the topics in our hypothesis and confirmed that those jobs matched those topics. So again we proved the problem worthwhile to continue solving.

[00:22:10] Now we leave the problem space and move over to the solution space, where we had several solution ideas. Our hypothesis, and the most promising one for us, was developing a lifestyle companion app that gives young people a jump start in their new stage of life. What we did here was create two versions of a clickable prototype. One, at that point in time, some years ago, was a quite innovative chatbot user interface, and one version had a classical navigation. Within our product team we were very much convinced that, especially for a younger target group, the innovative chatbot user interface would perform better.

[00:22:59] But as always, the user knows better, and the chatbot interface failed. Even more important was that the results were still quite positive, as a large majority wanted the product in general, just not with the chatbot user interface. The effort for testing at this stage was much higher than in the problem space, because here we created high-fidelity prototypes.

[00:23:29] Then in the very last step, we started to develop and launch a minimum viable product of this app. Our hypothesis was that young people would regularly use the app to plan leisure activities. We tested this hypothesis via in-app tracking: tracking of retention, weekly usage metrics, Net Promoter Score, and continuous cohort analysis, of course. Here the effort was the highest, as we developed an MVP over several months.

Anti-patterns and biases

[00:24:13] Now, having seen the example and reflecting a little on the previous slide, I have to admit that experimental product development always reads so easily, but often we still let our tests get distracted and fall into old patterns. Here's where it's important to understand a few anti-patterns, or biases, that we keep falling into, so we can recognize them and then dismiss them immediately.

[00:24:45] One is the so-called action bias. Action bias means we prefer doing something instead of doing nothing. Usually that's fine, but not always, because it often leads us directly into the development of solutions. We skip the problem space and go directly to the solution space, whereas we've seen on the previous slides that testing hypotheses in the problem space often requires much less effort and tells us very early whether we should pursue a certain path or not. With action bias, we often skip the problem space.

[00:25:27] Then we have the sunk cost fallacy, which means we find it hard to let go of things we have already invested time or money in. This makes us reluctant to give up our direction and thus further increases sunk costs. For example, we have launched a certain feature and our users are not using it. Then we say, okay, I have another idea how to tweak this feature, and we work on it in the next sprint. Still not working, we do another tweak. Still not working. And we all know we should have given up much earlier; that would have saved us a lot of money and time for more valuable things to build.

[00:26:10] Third is confirmation bias, which means we look for proof or evidence that confirms our opinions or assumptions. This often leads us to develop in the wrong direction and burn money, so the result is quite similar to the sunk cost fallacy. And last but not least, we have incentivization, which means we do everything that leads to being rewarded, materially or socially, and this does not always mean the best for the product or our customers.

[00:26:43] Let's take the example that we have a very strong stakeholder in our organization who has certain feature wishes, and we just accept them because the stakeholder has influence on our bonus payment at the end of the year. But those features might not be the ones that are most valuable to our customers. We might get our bonus at the end of the year, but our product might not be successful as a result.

[00:27:13] Finally, when we look at those anti-patterns, we see that the first three are more human nature, whereas the last one is a structural problem that we often have in organizations and that we need to solve at an organizational level. I'm showing you those anti-patterns because for me it's important that you, as a product manager or a member of a product team, are aware of them, in order to be able to catch them when you fall into them and discard them immediately.

Summary and recommended reading

[00:27:53] We are slowly coming to the end of the presentation, and I would like to summarize how to increase learning efficiency. We have mentioned it already, but it's about practicing. The more experience you gain in experimentation, the better you become, so you need to run through the process several times: start with a hypothesis, test the hypothesis, derive a certain action, and repeat. Second, we've shown you that there's an immense amount of methods available. You need to find the toolkit that best suits your context and your product team.

[00:28:39] Then find your experimentation process. Use simple experiments to verify assumptions before coding. Really take your time to analyze and experiment in the problem space before jumping into the cost-intensive solution space. And you need to create a culture of experimentation: fail early and often, but celebrate lessons learned, not just successes. When you learn early that a feature idea might not be worthwhile to pursue further, that's also a success, because you have saved money and time early on. And when your context changes, you need to adapt your methods and your process to the new context, so let go of methods that no longer fit your environment.

[00:29:34] And last but not least, coaching. When you are lucky and have experienced product leaders in your organization, use them as coaches, mentors and trainers. They have a lot of experience we can benefit from. Another good possibility is to create so-called communities of practice, for example a product management community of practice or a product discovery community of practice, where you exchange with your peers across different product teams, learn whatever worked well in one team, and try those things in your team too. This is also a good possibility for exchange and for creating a culture of experimentation.

[00:30:21] Recommended readings: I think both books might be very popular with this audience. Marty Cagan's Empowered, which is about product leadership, and Teresa Torres's Continuous Discovery Habits, which is a hands-on product discovery manual, let's say. I'm also regularly publishing product management articles on my Medium blog, so if you would like to follow it, you can scan the QR code above. And with that, we are finished with the presentation, and I think we're opening up for questions.