Hypothesis Driven Development

Alexander Bertholds at UXDX Community: Stockholm. Video: https://youtu.be/Hm_rwLYMWbI

Readable transcript: edited from the recording's captions for readability (fillers and false starts removed, punctuation and section headings added). Wording is the speaker's own. Timestamps are positions in the video. Names marked [?] could not be verified against the audio.

What hypothesis-driven development is

[00:00:13] Alexander: Thank you for the introduction. I really liked one of the first things you mentioned today in your presentation, the T-shaped competence, and I hope that this talk will widen your T a little bit for all of you who work in UX and design, because I'm going to talk about hypothesis-driven development from a data perspective, going through the different parts you need to know in order to understand hypothesis-driven development as it looks today from the inside.

[00:00:53] Starting with what hypothesis-driven development is: it's not something invented by tech companies or anything. It's been around for ages, in use in science, and then for the last decade or two it's been adopted, slightly altered and used in tech more and more. It's a method or process that will help you to develop your product in the right direction, and the process consists of three major steps. You can make a different division of it, but I prefer to see it as three steps. You start by analyzing your users, and then you create a testable hypothesis of what features they need. Then you develop your feature and you test it. So instead of just shipping the feature, hoping that you have a good idea, the best idea from the start, you send it out to a subpopulation of the users and you see if it performs better than your old feature.

[00:01:54] In science this wouldn't be called an A/B test, as we do in tech; you would probably call it an experiment. Say that you were from a pharmaceutical [?] company, like AstraZeneca [?] or something like that. You would get a lot of lab rats, you would give them a disease, and then you develop a cure that you hope will cure this disease. You give this pill to half of the rats, and the other half won't get any pills at all, poor things. Then you wait and see if the rats who got the pill are cured, and that's how you know if your new pill is working. Hypothesis-driven development is doing the exact same thing, but with new features or new products. After you're done with the test, you learn. You learn from the test, you write down whether this feature was a success or not, and why.

A template for a testable hypothesis

[00:02:48] In order to create a good hypothesis, I like to use this template. Here are four bullets, four beginnings of a sentence, and I'm going to illustrate them with an example. Let's say that we were planning on rebuilding our signup page. Most companies have a signup page if you have some kind of users and they need to create an account or something. So we've been looking at the product usage data on the signup page, and we've learned, based on the insight that we received, that users abandon the signup page to return at a later time. Maybe we looked at the basic statistics on the signup page and were seeing that it has a lot of abandonment, and then we also see that some users, based on their cookies, returned at a later point in time and completed the signup flow. So we know that they start filling in the signup page, but then for some reason they drop out, and they return at a later time. So we have that insight.

[00:03:50] So we think that if we save the user's provided information, so we develop a new feature, it will make the signup flow less tedious, since retyping information isn't necessary. So we have the new feature involved, and this is the change in user behavior or user experience that we want to achieve. We'll know we've succeeded when the conversion on the signup page increases by at least four percentage points.

[00:04:18] This last step here is crucial for working with hypothesis-driven development. Because if you only have an idea, you develop a feature and you say, "Oh yeah, this is the user behavior we want to change," but you don't have a way of knowing if you reached a change in behavior or experience, then you don't know if your new feature was a success or not. So you really need to have this last step and decide: okay, if we get this change in behavior that we want, how will that manifest itself in our metrics?

Where to find data for insights

[00:04:57] When doing a hypothesis like this, one of the trickiest parts is the first step, getting the insight. To get insights you need data, so the question is where do we find the data to get good insights? As a data scientist, one of my favorite places is product usage data, which means that you track everything that your users are doing. What buttons are they clicking? How long are they in your product? How often do they use the product? Do they churn? What do their usage patterns look like? This is really powerful, and if you have a team of analysts that looks at this data, you will learn a lot, a lot of good ideas for future development.

[00:05:35] The drawback is that it's expensive to set up. If you're a small startup, maybe your first investment won't be hiring a data engineer and data scientists, building a pipeline, having a nice data warehouse in Redshift and so on. So it's good if you're mature enough to have it, but we need other ways that you can start with right away as a company as well. And we have user research. This is good for small and big companies. We have a lot of user researchers here, and the insights you can get from user research are super valuable. They can tell a different story than you can tell from just looking at usage data. And the best thing is that you can use it even if you're really small. You have a few users, and you can use user research, but you can also have it when you're a huge company like Microsoft or Facebook: going out, looking at your users, how do they actually interact with the product in real life?

[00:06:35] So if you can have a team of user researchers and a team of analysts looking at the product usage data, you have a really solid foundation for getting good insights in order to formulate good hypotheses. However, both of these are expensive, so I will give you two other options for data sources.

[00:06:55] One of them is using reports from customer support and sales. Like the user research people, they talk to users on a day-to-day basis. However, where the user researcher has a more holistic view of the user experience and the user journey, customer support and sales have a narrower focus. They will try to either sell the product, or they will try to just help the customer with a problem they have at hand. That means that when you listen to customer support, for example, you will probably mostly get ideas for new features that involve fixing small friction points or solving bugs and things like that, whereas a user researcher will be able to see the potential for new products and find new fields where you can use your product.

[00:07:42] And last we have experience, or gut feeling. It's not the most solid data point, if you can even call it a data point, but you shouldn't neglect it altogether. If you have someone with a lot of industry experience, that person is worth listening to, and the person can have really good ideas. Just make sure that you expose them to this framework of hypothesis-driven development, so you experiment on the features that are based on someone's gut feeling. Because if you do, then you know that if that person's gut feeling is completely off, you won't ship the bad product.

Running an A/B test

[00:08:22] So, in our example, we got the insight from looking at product usage data, and we developed this new signup page, and it's time to do the second step of the process: creating a test, creating an A/B test. Most of you know that an A/B test consists of splitting your users into groups randomly and exposing each group to one of the two features, either the new one or the old one, and then you compare the chosen evaluation metric between these two groups. In our case we chose conversion rate, so then we will see: do we get a better conversion rate for the new feature or for the old one?

[00:09:05] What's really important when you create an A/B test is that you do the group selection randomly. Because if you do, then the distribution of users in each group will be the same as the distribution of users in your overall user base. They'll have the same proportion of old people, the same proportion of young people, the same proportion from each market, and so on. If you split, say, by market or by gender or age, then you will have a bias in your experiment. You will end up doing an experiment and getting the result that feature A performed one way and feature B performed another way, but you don't know if that's because of the feature or because of differences in these two groups. That's why it's really important to do the group selection randomly, because then you know that all the difference between the groups is the feature itself.

How sure can we be of the result?

[00:09:53] In our example, we got the test result. In the old version, feature A, where we didn't save the information, we got a conversion rate of 52%. With the new feature we get a conversion rate of 54%. So we've got a small improvement. Then another big question arises, which people tend to fail on: interpreting how sure we can be that reality is reflected in these results. To answer that, we need to go into some statistics.

[00:10:23] Let's pretend we're running this exact same experiment 1,000 times, and for each experiment we register what conversion rate we got. We're running the same thing, testing the same features, not on the exact same group but on the same group size. On the x-axis here we have the different conversion rates received, and on the y-axis we have the number of experiments ending up with a certain conversion rate. What would happen if you did 1,000 experiments like this is that you would get a normal distribution of results.

[00:11:01] What does that mean? It means that most of the experiments will end up with a conversion rate close to the true conversion rate of your feature. The true conversion rate of your feature is not something you can know in reality, but here I just did a simulation from a normal distribution, so here I know. The thing is, when you run an A/B test, the result will be a random sample from a normal distribution with a mean at the true performance of your feature. As you can see, with a thousand experiments you will be unlucky sometimes. Say for feature A, in a handful of cases you will end up with conversion rates of 37%. And in our specific experiment, we ended up with feature B performing slightly worse than its average, ending up on 54%, and feature A performing slightly better than its average, ending up on 52%. So what I'm saying is that even if feature B is better than feature A, you will sometimes get another result. And the frequency of experiments that produce false results like this is determined by the selected significance level and power.

Significance level, power and minimum detectable effect

[00:12:18] Just a quick raise of hands: how many of you have heard about significance level? How many of you have heard about statistical power? Not as many. Okay. Significance level is the rate of experiments that will falsely tell you there is a difference between A and B even though there is no real difference. Typically it's standard to use 5%. You don't have to, and I'll talk more about that soon. This means that, say you run what we call an A/A test, where you're testing feature A against itself. Then you would expect no difference between the two groups. But if you were doing the test over and over again, 1,000 times, and you have a significance level of 5%, then in 5% of those experiments you'll say we have a statistically significant difference between these two groups, so feature A is better than feature A. Of course that's not the case. It just happens to be random fluctuations between the two groups.

[00:13:21] Then we have statistical power. That's the rate of experiments that will correctly tell you when there is a difference between feature A and feature B. So in our example, where we had a difference between feature A and feature B, it means that if we used a power level of 80%, we would detect that difference in 80% of the experiments we run.

[00:13:44] These two parameters are determined by the sample size you use and the minimum detectable effect. The sample size is simply how many users you have in each group, and the minimum detectable effect is the smallest change you will be able to detect with the chosen power. If you think that the conversion rate will go from, say, 50%, which could be the historical average, to 54%, then the minimum detectable effect is four percentage points. That's what we had in our hypothesis. We said that with this new signup flow we think that we will increase conversion by at least four percentage points, the minimum detectable effect. And the horrible thing is that any change smaller than that will be considered insignificant.

[00:14:32] Remember the result we had here: 52% for feature A and 54% for feature B. That means that we will say that there is no significant difference between feature A and feature B. There is a difference, but we can't say that it's not just due to random chance. So you might ask yourself now: why didn't you choose a smaller minimum detectable effect? If we had used one percentage point, then we would have detected that difference. As you might suspect, the answer is that it's not that simple. There's a balancing act in choosing these parameters. The significance level, the power, the sample size and the minimum detectable effect all interact with each other.

[00:15:17] Decreasing the minimum detectable effect will either give you a higher significance level and lower power, which is bad because you will get the wrong result more often, or it requires you to have a bigger sample size. And increasing the sample size will either give you a longer test time, or it won't be possible with your current user base. In our example of the signup page, it will give you a longer test time, because if you want to double the sample size, you need to run the test twice as long in order to let more users pass through your signup page. You can always do that, but you don't want to run an experiment for six months, because you want to ship your feature faster than that. And if you're testing something on your existing user base and, say, you have 10,000 active users, then you're limited by that. You can't do an experiment with more than 5,000 users in each group.

[00:16:10] The math behind calculating the sample size and how all these parameters interact is complicated, and if you're not too keen on the maths, there is a really great sample size calculator here, which I double-checked computes the correct result. You can just Google "Evan Miller sample size calculator" and you'll get it there. I can also send out the slides. I see a lot of people taking pictures, so I guess I will just send them out. Yeah, great.

Choosing the evaluation metric

[00:16:46] And finally we have the art of choosing the evaluation metric. Here we have a conversion rate, and you're not building a product with the goal of having high conversion rates in your product. You have another goal. Most of the time you want to generate revenue or profit. High-level metrics like these are very relevant to your overall goals, but they have an issue, in that they move really slowly. Say that we said, okay, we want to do a new signup page, and we think this will increase revenue by two percent. Then we would have to wait for these new users to become paying users, and then wait a few months for them to generate enough revenue, and it would take six months, twelve months or something like that to evaluate this experiment.

[00:17:35] So what we do instead is use low-level metrics, like conversion rates and user searches, something like that, and they are proxies for the overall goal. Because we imagine that, okay, if we increase the conversion rate, we'll get more new users, and if we assume that the average revenue per user is the same, that will generate more profit. They have the advantage, of course, that they can be evaluated immediately after the experiment is run. But the danger is the word "assumption." Did you hear that I said "assume that the average revenue per user would be the same"? How do you know that's true?

[00:18:12] When you're doing your hypothesis, you need to be really careful that the feature you're developing is not impacting the low-level metric in a good way and the overall goal in a negative way. Say that we create a new signup page, it improves our conversion a lot, but it hurts the average revenue per user. Then we might end up getting a bad result from something that looks like a success from the initial A/B test. So what you can do here is have a follow-up on all your experiments with high-level metrics. Half a year or one year later, you go back, compare these two groups that you exposed to the experiment, and see how they perform on the overall level. You'll be surprised how many times you'll have a really nice impact on the low-level metrics and no positive impact on the high-level ones.

[00:19:09] This is especially true when you're running campaigns and you try to reactivate customers by giving them free stuff or something like that. They usually give a huge spike in reactivation rate, the number of users reactivated, but when you look back at it six months later, they have churned again, and all you have is money spent on free stuff for users who never really wanted to use your product anymore.

Ask why in the learning step

[00:19:37] So I've talked about the analyzing step, where you can find the data and how you should work with product usage data and user research. I've talked a lot about the testing. And I just want, before we end, to encourage you, in the learning step here, don't settle for just looking at the A/B test result and learning whether feature B was better than feature A or not. Also ask yourself the question why. Say that feature B wasn't an improvement over feature A. Why was that? Go out and talk to the users who were exposed to this page, or analyze the usage data we have from them, and try to find an answer to that question.

[00:20:17] Because when you do that, when you put your data team and user research team on it, you'll immediately pop up to the first step in the process. You'll be analyzing the users, and you'll get new insights, new ideas and new features before you know it. If you have this process running and you work with hypothesis-driven development, I assure you that it will help you to develop your product in the right direction, which is what we all want. So thank you, if you have any questions afterwards.