The Superpowers and Shadows of A/B Testing: Balancing Data-Driven Success with Bold Innovation

Ryan Leffel at UXDX EMEA. Video: https://youtu.be/n8p9oAJb71s

Readable transcript: edited from the recording's captions for readability (fillers and false starts removed, punctuation and section headings added). Wording is the speaker's own. Timestamps are positions in the video. Names marked [?] could not be verified against the audio.

The superpowers of A/B testing

[00:00:09] Chris: Welcome, Ryan.

[00:00:10] Ryan: Thanks, Chris, appreciate it. All right, thank you all. I am super excited to be here today. What I'm going to talk about is something that can very much power innovation, but it can also hold us back, and this is A/B testing. A/B testing has a lot of superpowers. When used properly, it can bring us clarity and confidence. We have all the data we need to really make good decisions. But it also has a lot of shadows. There are biases, there are blind spots, and you can over-optimize what's already there. Today we're going to explore both sides. We're going to talk about the strengths and the traps, and hopefully come up with some ideas of how we could break through.

[00:00:53] How many in this room, and I can't really see a lot of people here, are using A/B testing? It feels like a good number of hands went up. Cool. Hopefully there's something here for you today. If you're not doing A/B testing, hopefully there's just some good practical insight into building products. A/B testing is a great way to try new things, to learn, to make adjustments. If you're building a product, you really should be optimizing.

[00:01:19] I love this quote: "Every idea is a bad idea. No idea performs the way you expect once you collide it with reality." This is by Marc Randolph. He was a co-founder and the first CEO of Netflix. There are a few things I really like about this. First, testing really is a humbling process. It forces you to test your assumptions and accept that you might be wrong. Another piece is that you never know if an idea works until you actually go out there and give it a try. So testing creates a really nice space to quickly experiment.

[00:01:56] I want to quickly talk through a few interesting examples of how we at Priceline have used A/B testing outside of the expected increased bookings. This is an email, and at the bottom you see a call-out for chatting with Penny. That's an AI travel assistant. So this isn't about booking. This is about driving more requests for support to Penny instead of customer service. It's not about booking; it's about saving the company money by reducing call volume.

[00:02:30] This is another example. We did a refresh, and instead of just rolling out the refresh, we did 22 tests. They were looking at adding in the silhouettes, the shapes of images, corner radiuses on buttons, elevations, for example. So instead of just rolling it all out, we actually tested into it. It wasn't about testing for wins. It was just making sure we're flat. Even a slight loser is okay. We just wanted to make sure nothing was really breaking by making some of these design enhancements.

[00:02:56] This is one of my favorites. If you all look at Priceline now on your mobile device, 50 percent of you will probably see the variant on the right. We're testing a collapsed navigation, so just a search bar, which allows us to lift content higher on the page, compared to the left-hand side, which is historically what you see on Priceline or any online travel agency: just a big search form. It's a really interesting test, and we really want to understand how it's working before we completely roll it out.

Every superpower has a shadow

[00:03:28] There are a lot of strengths and a lot of superpowers you have with testing. There's experimentation. Experimentation is a great way to test something, a very low-risk way to check out ideas. Precision is another one. Precision is all about isolating variables. With testing, you're able to very precisely target specific variables you want to test, so you can learn more about how those specific things are working. It gives you evidence. You don't need to guess; it's very much informed by data. And finally, scalability: you test something in one place, and it's likely you could roll it out in several others.

[00:04:05] Now, this is important. Every superpower has a shadow. For example, a shadow of experimentation is reactivity. You could test everything, but you could end up making a lot of small tweaks, constantly pivoting, not having a clear direction. A shadow of precision is tunnel vision: you can get very locked in on one specific thing and become blind to the bigger picture. A shadow of evidence is analysis paralysis: you're so stuck looking at data, trying to analyze data, that you're not actually moving forward. And complexity is a shadow of scalability. You test something in one place and roll it out in multiple areas, but without proper coordination, teams can lose alignment.

[00:04:52] The Icarus paradox is the idea that the very traits that lead to success can also cause failure. In Greek mythology, Icarus was given a pair of wings made by his father, Daedalus. Daedalus was a brilliant craftsman. He gave Icarus wings, and before Icarus took flight, Daedalus said, "Don't fly too close to the sun, because it can melt the wax on your wings. Don't go too close to the sea, because it could pull you down and make you crash." And Icarus got incredibly confident, flew too close to the sun and crashed into the sea.

[00:05:36] This is where the phrase "don't fly too close to the sun" comes from. He flew close to the sun, his wings melted, and he crashed. So really, the story here is that unchecked strengths can become weaknesses. Success requires not just playing to your strengths, but also recognizing what your blind spots are and how you can put up guardrails.

Biases

[00:06:02] I want to talk a little bit about some lessons I've learned through the years of doing A/B testing and building cultures of experimentation. The first is biases. As product people, marketers and designers, we use cognitive biases all the time to help influence people to make a purchase decision. But these biases can also impact the way we receive information.

[00:06:28] I want to talk a little bit about some of these biases and give an example: Groupon. Is anyone familiar with Groupon? Pretty good, awesome. First, I want to say I love Groupon. I've had several Groupon deals. I once had a business, and I've been a customer of Groupon. For anyone not familiar, it's a marketplace that allows you to get really good deals on services in your local area. Groupon realized really early on that one of its big drivers for pulling people in was emails.

[00:06:57] So they started sending an email a day, and they saw really great conversion, high open rates, clicks. It was looking great, and it confirmed that email was a very powerful channel. So they got a little bit confident. They started sending two emails a day, three emails a day, all the way up to five emails a day. And what started happening is some decline in the background. Users were getting overwhelmed. There were declining open rates, unsubscribes, user fatigue. So just because something works doesn't mean continuing to do it the exact same way, or doing more of it, will work better.

[00:07:36] A few cognitive biases come into play here. There's confirmation bias, the tendency to favor information confirming pre-existing beliefs. There's survivorship bias, the tendency to only remember tests that worked. And the sunk cost fallacy. The sunk cost fallacy is probably something everyone here is familiar with. You start doing something, you put investment into it, you have an executive team that really gets behind it, and it becomes harder to pivot or step away.

[00:08:08] The most important thing is knowing that there are biases. Being aware of what the biases are will help you avoid them. In this case, thinking about what could prove you wrong is very important. Pay attention to all the signals, not only the early ones that look good. And know it's okay to pivot. Set checkpoints to ensure you're heading down the right path and staying on it.

Culture and curiosity: be Josh

[00:08:33] Next, I want to talk about culture and curiosity. I want to introduce you to Josh. In the movie Big, Josh is 12 years old and he's stuck in a grown man's body. Anyone familiar with this movie? Awesome. So Josh is stuck in a grown man's body, and he's working at the MacMillan toy company. They think he's actually an adult, but he's 12 years old. In one particular scene, he's sitting in a boardroom full of executives, and they're pitching an idea for a new toy, a transformer, which is going to be the next big thing.

[00:09:07] I was going to play a clip, but I was warned there could be some tech issues and potentially some copyright infringement. So I put this together in storyboards, and we're going to talk through what happened. This actor's name is John Heard. You might remember him from Home Alone; he was the dad in Home Alone, and that's how we're going to refer to him from now on. So the dad from Home Alone is standing in front of the boardroom. He has all the stats, all the data on why MacMillan needs to build this transformer. This is going to be the next big thing.

[00:09:38] Meanwhile, we have Josh, the 12-year-old stuck in a grown man's body. In the middle of the meeting, he's playing with the toy. He's making it dance, taking it apart, putting it back together. The boardroom's all looking at him. Josh raises his hand and says, "I don't get it." Brave, he spoke up. The dad from Home Alone gives him this look: "What exactly don't you get, Josh?" He doesn't look happy about it. Josh just looks back at him and says, "A building that turns into a robot? That's not fun."

[00:10:13] So the dad from Home Alone comes forward, hands him a piece of paper and says, "If you haven't read the industry breakdown, you should. This is what it says. Here's all the data, here are all the facts. This is why we need to build this transformer." Josh raises his hand again, because he's brave and he's a 12-year-old in an adult body, and says, "I still don't get it."

[00:10:36] Everyone in the boardroom is looking at him. The guy in the back there, that's MacMillan, the CEO of the company. He says, "What don't you get, Josh?" It's a great example of leadership. Josh goes on to say it could have been a robot that turns into a bug; that would actually be a lot more fun. The dad from Home Alone gives him this look. He's not happy about it. He asks one of the best questions ever in cinema history: "A bug?"

[00:11:03] Meanwhile, the CEO leans in a little bit in the back, like, "This is really interesting." Bow tie guy over here is also paying a lot of attention: "Wait, so the robot turns into a bug?" All of a sudden you hear all this chatter in the executive room. They want to learn more about it. The dad from Home Alone is just like, "What is going on here? I'm giving you all the data, all the numbers." Meanwhile, MacMillan in the back is looking at Josh and says, "Well done, Josh. Well done."

[00:11:29] To me, this is a great example of culture and curiosity. There are a few things that happened here I want to call out. In order to have a healthy culture, you need psychological safety. A good culture makes it safe to speak up, even when something sounds naive. Josh wasn't dismissed when he said, "I don't get it." In fact, the CEO is the one who leaned forward and said, "What don't you get?" That's a great example of leadership.

[00:11:55] Also important is a learning mindset. A good culture values learning, not just agreement. The room had data and all the features, but no real sense of what users would love. Josh's questions opened up a space for understanding, and the execs didn't just tolerate it, they welcomed it. They listened. And finally, a good culture is about being user-obsessed. Your users are the ones who use your products, and it's important to listen to what they have to say. That is just as important as hearing what all the numbers are saying, because the users are telling you what those numbers actually mean.

[00:12:36] You might be asking, how do I create a culture that believes in curiosity? That answer is going to be different depending on where you work and what you do. That could be a whole other talk. But one thing I will tell you that is true is that it starts with leadership. This is one of my favorite quotes: "Leadership is a choice, not a position." That's Stephen Covey. So everyone in this room can change the culture in your company by asking questions, by challenging, by speaking up.

Optimization is the tool, innovation is the outcome

[00:13:09] Next, I want to talk about optimization being the tool and innovation being the outcome. I want to start by talking about local maxima. A local maximum is a point where small changes no longer yield significant improvement. I want to tell the story of local maxima by telling you what it looks like to break out of one, by talking about a cheeseburger.

[00:13:37] In the 1970s, McDonald's was absolutely crushing it. They were a machine, cranking out burgers, shakes and fries. They were all about speed, consistency and scale. If you were working on their corporate team, you were feeling great. One thing you might be talking about is how to make the burger better. So let's talk about the burger and how we could optimize it. You can ask: can we add more sesame seeds to the bun? Can we try a different cheese blend? Toast the bun slightly longer? How about a juicier patty? At the end of the day, you're still optimizing a burger. It's going to be a burger.

[00:14:17] Now, there was one franchisee at the time in California, and his name was Herb Peterson. He loved eggs; he loved eggs Benedict. And he had an idea: what if McDonald's served breakfast? So he got an English muffin, cheese, Canadian bacon and egg, and he created a special Teflon ring, small and round, to make a perfect size for a sandwich. This is how the Egg McMuffin was born.

[00:14:46] This broke every single rule in the McDonald's playbook. They didn't open before 11. They didn't serve eggs. They had no breakfast menu, no early staff, no morning supply chain. If you were in corporate, you saw this as risky and way off-brand. Herb tested it locally in his store, and people loved it. This is McDonald's hitting a global maximum. A global maximum is the optimal solution that can be achieved by testing different variations of a design, feature or strategy.

[00:15:19] Today, breakfast accounts for 25 percent of McDonald's revenue. It created new customer behavior, the morning drive-thru, and it opened up a completely new market: breakfast.

The Kano model and the local maxima trap

[00:15:34] I want to bring this into the Kano model. For anyone who's not familiar, the Kano model is a way to measure features to determine where you should continue to invest. This is a very simplified view of it, but I want to quickly talk through what it means. At the bottom of the pyramid, you have basic attributes. These could be things like a shopping cart, a checkout, product search: things you absolutely need for your product to work. They're expected. Customers expect to see these things when they come to your site.

[00:16:03] You have performance attributes, which are features that are not necessary but increase engagement. It could be a responsive site, multiple payment options, or filtering and sorting, for example. And you have your exciters. These are important. Exciters are things customers don't know they want, but they're really delighted to find them. Personalization done right, free shipping, buy online and pick up in store: those are examples of exciters.

[00:16:32] When you're optimizing, you're focused down here on the performance and basic features. That's where your focus is when you're optimizing, and this is also where the local maxima trap comes into play. So there are a few things you need to keep in mind. One, you need to take big swings, try new things, be a little bit risky. You need to ask, "What else?" It's not only about what you're doing; it's about other things you could be doing that you're not thinking of, like the Egg McMuffin.

[00:17:00] And lastly, always know the problem. As you start to test, as you get into any strategy, you might learn that the problem changes. It's always important to understand the problem you're trying to solve. And I think there's one really important piece of this: you want to solve the customer problem. It's easy to look and say, "Well, this is what the business problem is." But the business problem isn't always equal to the customer problem. So always understand what the customer problem is and ask how you could solve that. You want to get to the exciters, and you get there because you have a culture that believes in experimentation and a team that is curious.

[00:17:36] This is an important piece. You have the Egg McMuffin, and what I want to demonstrate here is how exciters over time become basic expectations. Not too long after McDonald's introduced the Egg McMuffin, every fast food chain was trying to figure out how it could do breakfast better. Now, if you're fast food and you don't serve breakfast, you're just not getting the business in the morning. People are going to pass. I realize we're talking about food, but think about literally any digital product out there. Something comes out, and within a very short time other people are doing it, and a lot of those features just become basic expectations.

The innovation flywheel: Burbn and Priceline

[00:18:14] This brings me to innovation. Innovation is the process of introducing something new, or improving something existing, leading to significant progress or change. A few key pieces to this. Many people think innovation is just creating something new. Innovation is also improving something that exists, and that's why A/B testing can be so powerful. But regardless of whether it's new or you're improving something existing, it's creating significant progress or change.

[00:18:46] Back to the pyramid. I want to talk a little bit about how innovation works with A/B testing, because there's an important flywheel concept I want to get out here. Incremental testing is ongoing tweaks to existing features. There's sustaining innovation, which is meaningful upgrades to maintain your market position. And then you have disruption: market-shifting innovations. You're optimizing at the incremental level, and when you do it right, that is going to help you get to sustaining, and that's going to help you get to disruptive.

[00:19:26] I'm going to tell a little story about a company called Burbn. They started off as a very cluttered app. They had check-ins, photos, messages, games. They were making a lot of small improvements, but it was still a little bit unfocused. They were like a Swiss Army knife, trying to do a little bit of everything. A little bit later, in 2010, they began to realize people were really using their photo feature. So they started to really lean in, sustaining based on photo sharing.

[00:19:59] A little bit later in 2010, they dropped 90 percent of the app. They went all in on photo sharing, created the first real visual culture around sharing photos, and rebranded themselves as Instagram. So that is an example of starting with incremental learning and reinventing.

[00:20:20] I think Priceline is a really good example of starting with disruption, coming down to incremental, and back up. Priceline was disruptive early on because they started with name your own price. Some of you might remember William Shatner and The Negotiator, which is usually the first thing anyone says to me when you work at Priceline: "Have you met William Shatner?" I have not, but I did see him on a Zoom call once.

[00:20:44] What they started was name your own price. Essentially, you could say, "This is how much money I want to pay." Your deal would get accepted or not, and then hopefully you'd go on your trip. It was bold. It was a new way of shopping. And it really helped with two key things. One, it helped get rid of unused inventory. And two, brands were able to maintain their brand equity, because they were privately accepting the bids without releasing what those deals actually were.

[00:21:18] A little bit later, Priceline started selling cars, mortgages, gasoline and groceries, all through name your own price. Then they leaned more into travel, focusing really only on travel. There was a dot-com crash in the early 2000s, so they scaled back a little bit and went all in, and now it was name your own price for airline, hotels and rental cars, nothing else. A little bit later, in 2004, and this is an example of sustaining, instead of just name your own price, Priceline introduced retail pricing. So you could either name your own price or pay the full retail price.

[00:21:54] After doing a lot of incremental testing on name your own price, a few key insights came out. Customers wanted speed and certainty over bidding. This brought us to launching Express Deals. It's not about bidding; it's hidden-name shopping. In other words, you could say, "I want to go to Berlin and stay in this neighborhood, in this type of hotel." You don't know what hotel you're going to get. You just know you're going to get the best price possible, and you find out what you're getting after you book.

[00:22:26] So disruption feeds the next cycle of testing and learning. One of the best things about Express Deals, outside of just being a great deal product, is that there's so much more to test. There's messaging, price drops, branding, placement, personalization. How do you get somebody to actually purchase something without knowing what it is? There's a lot you can learn from this, and over time a lot was learned about Express Deals.

[00:22:50] One key insight: travelers want more transparency upfront. This led to something called Price Breakers, which is what's known as a semi-opaque deal. A Price Breaker is essentially the same thing as an Express Deal, but instead of not knowing what property you're going to get, you see three properties, and you know you're going to get one of them. So it gives you a little bit more visibility into what you're going to get. In addition to creating another really good deal product, it gave a whole bunch of new things to test, and these new things to test are going to lead to more sustaining and eventually more disruptive innovation.

Use A/B testing to learn, not just optimize

[00:23:21] So incremental innovation becomes a flywheel for being disruptive. Use A/B testing to learn, not just optimize. When I first started working at Priceline, we were getting ready to launch the first loyalty program. Instead of investing a lot and jumping into it, we started putting VIP messaging on existing savings claims we already had. On any online travel agency, and here's a little secret for anyone who doesn't know: book a flight, you're going to get a cheaper hotel. Book a car, you're going to get a cheaper hotel.

[00:23:59] So we just started saying, "Priceline VIP: you booked a flight. Priceline VIP: you booked a car." We were giving people savings they were already going to get, just branded as VIP. We started to see a lot of signal, positive feedback on social: "Hey, I'm part of this VIP program with Priceline. This is great, I'm saving a lot of money." Then we went all in and built out a loyalty program. This gave us a lot of the signal we needed to do that.

[00:24:22] This is the way it looks: you have an idea, you collaborate, you test, you learn, and you get back to collaboration. A few key pieces to call out here. One, get to that first test as quickly as possible. Fail fast; that's one of the best things about this. If you fail, you either go back to collaboration or you realize it's time to move on. Maybe you're not solving the right problem. Maybe you're not asking the right question.

[00:24:47] People always ask, when you're doing A/B testing, especially at high velocity, when do you talk to customers? The answer is always talk to your customers. Talk to them before you test, talk to them after you test, talk to them while you're testing. The important thing is that you're talking to your customers. The last thing I want to call out here is this first test. In my opinion, this is your MVP. The MVP is the fastest way to learn how to build a product with the least amount of effort. So learn early, learn fast. You don't need full designs, perfect prototypes or months of research. Get to that first test.

Crash landings: normalizing failure

[00:25:22] Last, I want to talk about crash landings: failure. "There is no innovation and creativity without failure. Period." That's Brené Brown. What do these five products have in common? Anybody know? Show of hands. Exactly: mistakes, and they're all awesome. All these products started as mistakes. If you don't like one of these, there's definitely something else here for you. So all these products started as mistakes.

[00:25:50] So normalize failure, model failure. If you're a leader in this room, and everybody here can be a leader, make mistakes, talk about those mistakes, celebrate failure. It's not just about rewarding the risk. It's about celebrating what you get out of it, so that you learn, move forward and do something better.

[00:26:10] A few closing pieces. Beware of your strengths, because you need to know what your shadows are. Optimization is a great tool, and it's also a great way to get to innovation. Be Josh: be bold, speak up. Again, you can all be leaders. Make the breakfast sandwich: look beyond the obvious. Culture beats process. And lastly, the more you test, the more you learn. That's going to help you get to disruption, come back to the incremental, and get that flywheel moving. That's it. Thank you.

Q&A

[00:26:55] Chris: I think we have time for a couple of questions. I love the Big example. That's perfect, the dad from Home Alone. You had to get him in there. While you were walking through that, I was preparing this really killer question: in a travel company, how do you maintain that culture of being able to question things? But then you walked us through the history of Priceline, and innovation's kind of baked in. I have a feeling that in a lot of places, A/B tests very quickly become "I hope this works, because we can leave it out there." For those of us who aren't at a place like Priceline, who've had to fight the shadows of experimentation and A/B testing, too much Icarus, any tips on how we can change the game? How do we become Josh on day one?

[00:27:45] Ryan: It's a great question, and you just have to do it. I think that's really it. A lot of the time you see something that's not working, you just sit on it, and you don't want to speak up and say anything about it. So I would encourage anyone in this room to be as curious as you can possibly be and ask questions. If you see something that's not working, say something about it. If you have an idea, speak up. The worst thing that can happen is you get ignored or people say no. The best thing that could happen is somebody listens to you, like MacMillan, and it makes a change for the better. It really comes down to people, and that's all modeled through leadership.

[00:28:21] Chris: I also think, and it's something I always say in my classes, if you do that and people are jerks to you, maybe you're not at the right place. If you can't ask open questions, if you can't say a building robot sucks...

[00:28:32] Ryan: That's true.

[00:28:35] Chris: We've got a question here. Let's see, where was it? There's a very specific question about instances where A/B tests have failed, and learnings from them. You've probably seen more A/B tests than almost anybody in this room. Any principles you want to share?

[00:28:48] Ryan: For one thing, your win rate on tests is probably going to be around 20 percent. So roughly 70 to 80 percent, probably closer to 80 percent, of the tests you run are in fact going to fail. And it's those tests that fail that actually lead to the smaller percentage of tests that win. I think it's easy to celebrate the wins. We have a Eureka channel, as we call it, and it celebrates the wins. We don't actually have a losing channel, and I've been pushing hard on that. We should be talking more about the losers, because there would probably be nothing in Eureka if we didn't lose before we got to the win.

[00:29:30] Chris: That's nice. Continuing on a very practical point, any new AI tools in A/B testing that you'd recommend to the group?

[00:29:36] Ryan: There's some interesting stuff with AI out there. There are synthetic users out there right now, and you could use them. There are synthetic users that work in both qual and quant. We've been experimenting a little bit with synthetic users that work in the quant landscape. You can put in a hypothesis. It's not able to exactly do an A/B test, and it's not evaluating UI, but you give it the hypothesis and it will tell you what it thinks is going to happen based on the sample of synthetic users.

[00:30:09] We don't use it for testing, but it becomes a very interesting and handy tool to help frame the hypothesis, or to help the design team get on the right track ahead of time, because you have a little more confidence in what you're going to do. I've also heard of some interesting AI stuff that we're not using. I was at a conference a few months ago, and somebody was talking about AI they use to look at messaging. When they run messaging tests, they have AI evaluating the messaging, and if it's not hitting certain numbers for significance, it will actually rewrite the message and automatically relaunch it. I thought that was an interesting one also.

[00:30:44] Chris: And make sure to write those examples down, because they'll be changed in about 20 minutes. They'll do everything you said they don't do yet.

[00:30:51] Ryan: Yeah, right.

[00:30:52] Chris: One last one, and I'm going to paraphrase this a little bit, because I think there's a broader way to focus on the criteria taken into consideration. It's more about gatekeeping. Your Josh example is great. Josh happens to be a lucky 12-year-old who gets a seat at the table. What are some tips on making sure you have the right gatekeeping in check, so you're not betting on the wrong ideas, but at the same time keeping it open and encouraging a culture of curiosity?

[00:31:18] Ryan: It's the slide I showed that had the process. It's the collaborate piece. And collaborating isn't just the designer sitting in a room coming up with ideas. It's the designer sitting with somebody from product, somebody from tech, somebody from business, and everybody should feel free to explore and do workshops. We started putting together more of a two-hour sprint up front to really learn, come up with ideas and make sure we're solving the right problem. With that, you have one idea you know you want to move forward with, but you also have other ideas.

[00:31:51] It's also thinking about the biases, because those do come up. Whenever you go into a test, understand what could potentially go wrong. What data points should you be looking at outside of the key KPIs you're trying to win on? If you can collaborate and be mindful of your biases, I think it will help you get to a better place.

[00:32:10] Chris: I had one last one, about consensus: do you always have to go for consensus, or do you listen to the loudest voice, somebody who's very passionate? But we're not going to have time to ask that, so you've got to ask Ryan.

[00:32:16] Ryan: Ask me. I have opinions on that one.

[00:32:18] Chris: He can't wait to tell you. All right, thank you, Ryan.