A/B Testing At Scale

Tatiana Tretyak at UXDX Europe. Video: https://youtu.be/Y77LuaKZEcs

Readable transcript: edited from the recording's captions for readability (fillers and false starts removed, punctuation and section headings added). Wording is the speaker's own. Timestamps are positions in the video. Names marked [?] could not be verified against the audio.

Growing up curious in a small town

[00:00:00] Hi everyone. As was mentioned, my name is Tatiana Tretyak. I'm a product manager at Booking.com, and I've been at Booking for about three years. I'm based in Amsterdam, which is where I flew over from, and it's my first time in Dublin, so I'm very excited.

[00:00:20] But before we dive into the topic of culture and the cultural shifts you need to do experimentation, I want to share a story, or a fact you don't know about me, which is that I grew up in a really small town in Russia and actually moved to Amsterdam just with a job. When I say small town, it's like a 15,000 people small town. If you think of the size of Russia, it's like a little drop of the population. And as in every small town, people had very normal expectations of what normal is. I could go to school, work, maybe go to university, and it's all kind of planned out for you.

[00:01:00] But I always wanted to try new things, try different things. And I'm not talking about crazy hairstyles we all went through as teenagers. It's more like learning Photoshop at 12 because I wanted to change the hairstyle of my favorite actress, or learning Spanish in a town where no one speaks Spanish, for sure. And I didn't get much criticism, but the answer was always like, why do you bother, Tatiana, what's in it for you? And my answer was always, well, I'm curious, and if I fail at least I'll learn something.

[00:01:32] Well, this approach of I'll try, and if I fail I learn something, got me from my little hometown in Russia to a leading travel company in the world, where "at least we try and learn something" became the motto we use every day.

Letting customers drive the product

[00:01:48] So three years back I came to a company[?] which had almost 18,000 employees, so it's bigger than my hometown. No pressure. And there are around 200 product teams working on product development every day from Amsterdam, and they all do it with the goal of making it easier for everyone to experience the world. Of course every mission statement is managed to be ambitious and inspiring, but if you look at every single word of it, like, what's easier, who's everyone, what do you even mean by experience the world? And when you are sitting in[?] Amsterdam it's very hard to understand what customers actually want. So we made a decision that we'll let customers drive our product.

[00:02:37] And probably as all of you know in this room, predicting customer behavior is hard. Raise your hand if you ever predicted it right. Let's talk afterwards, because I want to know how you managed. And since predicting customer behavior is hard, the approach the company took is to run data and experiment on everything, every day. Every single pixel, any interaction, every feature has been tested, and it gave us an immense amount of data. We've built some tools to process this data, but in this talk I don't want to focus on the tools, because technology can be bought or replicated, but culture is something that you need to build from the beginning. Even if you have the most amazing tools in the world, the wrong culture will not let you experiment at scale every single day.

[00:03:32] So I would like to share with you four aspects of culture and organizational processes that actually give you this feeling that, okay, it's a new day, I'm going to run a new A/B test and I might fail, but I will learn something.

One: failure is okay, and what failure actually is

[00:03:48] The first one was actually that failure is okay, and I think it's been covered in previous talks in this room, so I might not want to reiterate the message too much. But failure is different for everyone. Statistically speaking, only one out of ten A/B tests is successful, which means nine are not. So if you obsess around the failures you basically will not have a life, because you only think about failed A/B tests. And another point is that breaking things is actually an amazing learning opportunity, because you learn how to not break them again next time.

[00:04:27] And what failure actually is, is a different question, because to me and my peers at Booking, failure is not a negative result. A negative result is actually a very, very good thing, because you managed to tackle a pain point of a user which is so bad that just by patching it you made it worse. It means that you found something where you have to dig deeper, something that requires more insights, more research, and you keep moving in this direction. The failure, or what we define as failure, is if you run an A/B test which you spent two weeks preparing and then three weeks running, and then you didn't learn anything, because that's just time wasted and you didn't get any insights out of that.

Two: experimentation is the key to agreement

[00:05:15] The second part is that experimentation is the key to agreement. Raise your hand if in the last month you disagreed with your designer or product manager on how the feature should look like. Imagine you have five teams that try to come to this agreement, or ten teams, or you have a CTO coming in saying we need this tomorrow, I don't care.

[00:05:41] Well, if experimentation is something that you do frequently, experimentation becomes a part of the decision-making process, and this way there is no right or wrong opinion, there is no right or wrong idea. Every idea is worth testing. And even if you are a hundred percent sure that this is something I'm going to roll out, testing and experimentation still becomes an important step in that delivery process, because even if you were hugely successful or hugely unsuccessful, experimentation will help you measure how successful you were, or unsuccessful, which happens nine times out of ten.

[00:06:18] And it also helps you to avoid HiPPOs, so the highest-paid people in the room, pushing their ideas. Because even if they're super confident with what they want to do, if the data proves them otherwise it actually gives you as a PM or UX person the ability to push back and do it right, and not just do it fast.

Three: data is king, but empathy matters

[00:06:40] Point number three: data is king, but empathy matters. I don't think it really matters the size of your company, but the moment you start getting your hands on data it becomes very tempting to just spend days on running analysis, running queries, analyzing the test results. But in the end, data and experimentation is not the what, it's not the reason your company exists, it's not something that will make the difference. In the end of the day experimentation is just how, it's the value of validated hypotheses. And in order to understand the what for this how, you actually have to work a lot on the qualitative side of things, so user research, customer interviews.

[00:07:31] And it's sometimes a bit tricky, it becomes a chicken-and-egg problem, because do you first start with research and then you run an experiment to validate the findings, or do you first run the experiment and if the findings don't make sense you invest in research? There is no right or wrong answer, because ideally you can do both. If you design your experiments in a way that you collect as much data as possible, every experiment can lead you to a new insight and a new idea of what needs to be investigated qualitatively.

[00:08:06] For example, if you run an A/B test that changes the checkout process and you suddenly see that overall great, people are booking, everyone's happy, but you see a drop in family bookings, so families booked less holidays, which is not great because families definitely need a holiday — it gives you an idea. Okay, something went wrong for families, let's talk to family bookers and understand why they didn't book, how can we improve the process for that. So experimentation can actually fire up your research. Or the other way around: if you work on something which you have no way to implement in terms of an A/B test, of course you first do research and then you test it through experimentation.

[00:08:50] And yes, so in my opinion data is king, but empathy is still very important. You shouldn't just run an A/B test every day, because then you lose touch with your customer and you lose the ability to understand what is actually the matter you're trying to solve.

Four: build a community to keep the culture strong

[00:09:05] And another very important point is that you have to try to create the community to keep the culture strong. This is something that applies both for startups and big corporations, because it takes a very long time to set up a culture, but especially in the times of rapid growth and rapid hiring, it's the first thing that can be broken very easily.

[00:09:32] If you're just a startup, it's a privilege to try to set up the culture you want, and then you can look at the user research and experimentation rules and how you trust data. But as you get more successful and you get bigger, you have more and more people coming in, and as you get more successful you also get more people coming in from other big companies who also have their rules of the game and they believe that this is the right rule of the game. So it's hard. Creating the culture is hard, maintaining it is even harder.

[00:10:07] At the very beginning when I joined, it was still possible to do one-on-one training with someone who created the whole culture, or talk to someone who is more experienced to ask for advice. But once you start having 50 people joining every month, it becomes not scalable to teach every person individually how the experimentation works and report the rules of the game.

[00:10:34] We found two solutions for how to solve it. One of them is work with your most passionate, most loyal employees to make them ambassadors. They become the go-to people to ask these questions, and they usually do it voluntarily, because they have been in the company for so long and they care about the matter, so you don't need extra incentives to make them do it. And they want to pass the baton, they want to share what they know, to keep this culture and keep the standards high.

[00:11:05] But it also doesn't go very far, because you can have just so many experts to cover a two thousand people organization. So another approach we've tested, and this worked, is also trying to create a community that does peer-to-peer reviews of each other's experiments. And it becomes just win-win-win for all three parties. You usually pair up someone who is more experienced in the organization with someone who is new, so people learn from each other how the experiments should be run and what is a good practice, what is the best practice.

[00:11:44] And since you actually review random experiments — so you just press a shortcut and you get a random experiment running in the company — it's also an amazing way to stay aware of what's happening in the company outside of your immediate scope. So you learn what's happening in other departments. And since you're leaving a review, you also help these people to understand what can I do better in future. And you don't need a massive organization to start doing it, because once the community gets started it then grows at a similar rate as your company grows, and it becomes self-sustainable.

[00:12:22] One thing to keep in mind though is, when you start this review process it's very important to explain why this is happening. Because if people just suddenly start leaving reviews on your experiments, you're like, why me, what did I do wrong, why out of all the hundred experiments running is it mine that got picked? So this can create a bit of animosity in the beginning, and not understanding, like, why are you grading me. So set up the rules of the game at the very beginning, and then the community will keep the culture strong.

Four things to take back to the office

[00:12:58] So I think these are four key ideas I would like to bring. Once you go back to the office on Monday — well, not Monday, Wednesday — remember that you have to revisit your definition of failure. A negative test is not a failure, a negative test is just one step in the right direction, hopefully, or which will be the right direction after some point. And try to get buy-in not just from your team but also from management, because we all sometimes have people who just want results and don't understand why they are not happening. And then it will just create an atmosphere which is more welcoming to experimentation, and failure becomes part of the game.

[00:13:44] Balance experimentation with qualitative research. Depending on the problem you're working on, you can either start with research and continue with experimentation, or the other way around. Try to make experimentation part of decision-making. Whether it's a project plan, your Jira[?] or whatever slide that you submitted with a roadmap, experimentation should be one of the aspects. You can never assume that whatever you've built will be right from the first go. And since you start communicating this to upper management, to peers, others will also start thinking this way. So whatever idea you push, always put a slot of time to experiment and test it.

[00:14:24] And of course try to build a community to keep the culture thriving. Even if you're at first just one person in a community, it's a good start and then it can keep growing further.

[00:14:34] So I have some links here, because I know a lot of people are interested more in the technical side of things and how the infrastructure is set up. I'm probably not the right person to tell you about it because I wasn't the one who built it, but on our blog at blog.booking.com there are plenty of articles written by my colleagues who explain this in a bit more detail. So if you're interested more in the technical side of things, that's the place you should go for. And we're hiring. Thank you.