Are You Proving Yourself Wrong?

Joseph Levin at UXDX Community: Netherlands. Video: https://www.youtube.com/watch?v=VYoTLbMNYtk

Readable transcript: edited from the recording's captions for readability (fillers and false starts removed, punctuation and section headings added). Wording is the speaker's own. Timestamps are positions in the video. Names marked [?] could not be verified against the audio.

How much time do you spend trying to prove yourself wrong?

[00:00:03] Good afternoon. I want to spend some time today talking about ways that we can prove ourselves wrong. I want to start by asking, how much time do you spend trying to prove yourself wrong? Maybe this feels counterintuitive, because we're all at a product development conference where we're all interested in building great products and trying to prove that those products work for our customers. So maybe trying to find the data that supports our latest hypotheses, or researching what is it that users love about our products, or maybe it's proving that that latest software patch that you rolled out actually solves that pesky bug that's been lingering around.

[00:00:44] But if you're anything like I used to be, you may actually be spending too much time trying to prove yourself right, and you would be better off trying to prove yourself wrong. Because I have found in my career that that small shift of mindset has made all the difference. I find it leads to better debates, better strategies, and I believe that striving to prove our assumptions wrong is what ultimately allows us to build products that are useful, unique and profitable.

[00:01:19] Now, I'm not pretending to suggest that this is a new idea. Proving ourselves wrong is at the heart of the scientific method. The late Richard Feynman once said, "We are trying to prove ourselves wrong as quickly as possible, because only in that way can we find progress." If you're not familiar with his work, Richard Feynman was a theoretical physicist, worked on the Manhattan Project and did a lot of things in quantum mechanics, and that's all way over my head.

[00:01:50] But what I find so inspiring about what he says here, and something that I believe we should bring into product development, is this power of trying to prove yourself wrong. What he's speaking to is the difference of evidence that is required when you're trying to prove something right versus trying to prove something wrong. When we're trying to prove ourselves right, it doesn't matter how elegant or beautiful our theory or hypothesis of user behavior may be, we can spend our entire life running one more experiment, one more test, to collect that experimental evidence that what we predicted our users would do is what we actually see them do in reality.

[00:02:30] But on the flip side, when we're trying to prove ourselves wrong, it only takes one counter example to recognize that our hypothesis is a little bit off, that there's something about human behavior we don't quite understand. I find that exciting, because it's in those moments that we can start to dig in, find deeper insights for our products, and ultimately build better products for our users. So today I would like to share three challenges that I have faced in my career, and how using the small mindset shift of trying to prove myself wrong has made all the difference.

Challenge one: better debates at Booking.com

[00:03:07] For the first challenge, I want to take us back to 2017. I was at that time the director of software engineering for the front end of Booking.com, so I was leading about 35 cross-functional teams all focused on trying to make it easier to discover, select and ultimately book a great place to stay for their travels.

[00:03:26] Now, at Booking.com we really use the experimentation method to its fullest. We run a lot of experiments, and when I say a lot, I mean a lot. We take tremendous pride in our data driven approach, and so we have built our own A/B test tooling that allows any team to test anything on the product, and then they can observe the changes in customer behavior and then make decisions. This gives us a tremendous amount of data, but that data only takes us so far, because data's not always actually black and white. The human still has to make the decision: is this the right thing for our users, is this the right thing for our business or our accommodation partners, should we keep this change in the product or not?

[00:04:17] And because those types of decisions are human, we also try to create a culture of healthy debate, of peer review, to really follow that scientific method, so that any change that a team makes can be seen by any other team. This helps on several fronts. One, it means that when something works and is successful, other teams can quickly copy it, it'll spread like wildfire across our product, helping more and more users. But on the other side, if we see something that maybe we don't understand, we don't agree with, then we can question and challenge and help to make the decisions that we're making for our users better.

[00:04:55] But these types of debates require perspective. We need to be able to look at things from different angles, different cultures, different types of users. And having different perspectives isn't always easy. I know I sometimes have struggled with it, with my own emotions, my own opinion, my own bias coming to the forefront and holding me back and holding the discussion back.

[00:05:21] Now I'm going to tell you about one of those times. It was all because of a small little change that one of the teams made. They made a change here on what we call the hotel card. This is how we show search results to our users: if you're searching for hotels you'll see a whole list of these cards for different hotels. What they were experimenting with was this little tag called Just Booked. They were trying to make it easier to reassure users that other users were also looking at this hotel, so using a little bit of social proof to say, hey, you're looking at this hotel, don't worry, other users have also looked at this hotel and they found it to be a good choice, maybe it's a good choice for you too. That was the goal, that was the outcome that this team was trying to solve for.

[00:06:08] Now, what sparked the debate with this change was the magic number. They decided to use 24 hours as a threshold of when we would label a hotel as just booked or not. What this means is that if the hotel had been booked in the last day, we would label it for other users as being just booked. And for myself and my product counterpart at the time, we weren't completely comfortable with this 24 hour threshold. We were wondering, what is the right threshold? Is it fair to say that something that was booked almost a day ago has just happened?

[00:06:45] Personally, for me, I was really worried about this. I was thinking, are we being disingenuous with our users? It seemed obvious to me that if something has happened a day ago, I can't say that it has just happened. Now you can maybe imagine that if I'm going in one-sided like that, the discussions with the team did not go very well, because they had the data, they had the conversion metrics that showed, hey, more users are finding a place to stay. So it was the team's conversion data versus leadership opinion. How do you solve that type of problem in a culture where we pride ourselves on being data-driven?

[00:07:23] And so our debates and discussions were just going in circles and circles and circles, until one person asked a simple little question: what is the counter-argument? This was an aha moment for me in my career, and I started to realize that I wasn't able to argue against myself, because I was saying, just booked, that's not good.

[00:07:44] So as a group we started to play with this idea. Okay, we're talking about hotels, let's look at other decisions people make in their lives. What about eating lunch? "I just had lunch." And so we start to create this balance, these scales of perspectives, saying okay, yeah, it seems kind of odd actually to say that if you just had lunch and that happened two days ago, that seems weird. But you may be saying, okay Joseph, you just found another argument for your own idea, you need to look the other way. And that's what we tried to do. Okay, let's look at bigger decisions, so maybe buying a house.

[00:08:20] In the group we started to realize, okay, yeah, this doesn't seem so odd, that if maybe a friend of yours bought a house a few weeks ago and they're still saying to other people, hey, I just bought a house. It's a big decision. So we started to realize that there's a spectrum here and it wasn't just black and white, and it helped me realize, okay, I wasn't seeing the whole picture here. But also, through the discussions, the team started to realize they weren't seeing everything.

[00:08:51] So ultimately what we were able to find is, okay, we're not seeing everything right away, so we needed to find the counter-arguments and use that as a signal to recognize, okay, are we moving this debate forward? I now like to use this as a flag: if I see debates just going in circles, checking myself, okay, am I able to actually argue against myself? Because if I can do that, then I recognize okay, I've shifted beyond my own emotions and bias and now I'm trying to find the best decision for this situation. And I also like to watch my teams, and when I'm facilitating with a group, okay, try to pull the same type of thinking out by asking, what is the counter-argument, how can you prove yourself wrong? Because then we can help move beyond our biases and ultimately create better debates and better decisions.

Challenge two: better strategies

[00:09:45] So that first challenge was about better debates. The second challenge was about creating better strategies. The year was still 2017 and I was still working on the front end of Booking.com, but I had made a transition in my role. I was now the director of product at Booking.com, and so this meant my responsibilities had changed. It wasn't so much about how we were building what we were building, but now it was helping to guide teams to make sure we're building towards the right strategies, building the best products we could for our users.

[00:10:18] But there was a pattern I was starting to see. Lots of teams were really just talking about how many tests they were running or how many experiments they had run, where a team may say, hey, I've run 20 experiments, or another team, I've run 50 experiments. We were using it like a vanity metric that made us feel good: we ran lots of tests, we obviously must be learning lots of things. And when I asked, what did we learn, too often I was disappointed with what we were saying. The answer was, well, we ran a lot of things and nothing worked. And then, well, what comes next? Well, we'll try a lot of different things and see what happens then. So it felt like we weren't actually focusing on the right types of learnings, we were just kind of running in a circle and circle and circle.

[00:11:11] So I started to work with the teams to say, okay, how can we actually focus on the learnings that matter, the learnings that move us forward? What I found helped was structuring our evidence and assumptions. Starting with these two things: what is the evidence we have for this problem space? Quantitative, qualitative, hard data, the things we consider to be true about this problem space. Okay, we have that, great. Now let's look at the assumptions. What are the beliefs, the hypotheses about this problem space that we're operating with? Because I find that it's the combination of both the evidence and our assumptions that informs our strategy. The strategy to me is simply a plan: how do we get from where we are today to where we want to be to achieve our goal? And it's fundamentally about the evidence and assumptions that go into that.

[00:12:14] So what we were able to do then is say, okay, having this evidence, having these assumptions, let's now rank the assumptions, and rank them by risk to the strategy. What I mean by this is that that number one assumption, that number one risk, if we were to prove that assumption wrong we would need to change the whole strategy. Because now we're able to see, okay, what is it we need to learn to really understand if we're on the right path? These are the learnings that are going to matter.

[00:12:45] Because now, by saying okay, assumption number one, let's use this robust debate that we've developed, let's look for the counter arguments and the perspectives, to figure out, okay, what kind of test could actually prove this wrong? Let's go try those. Because then, if we're able to actually scratch off that assumption, it is fantastic, because now we know we were on the wrong path. So we've short-circuited those circles we were running before, and now we've got a better direction because we know, don't go that way, explore this space. This is great because now you're able to get to your right strategy faster.

[00:13:23] And we were also able to sometimes find, hey, we've tested these assumptions, it seems like we actually are onto something with this, maybe let's move this section now into the evidence column. Because when we move and collect more evidence, then we're actually building more confidence again in our strategy. So the key point here was about clarifying which learnings matter most, and the most important is that focus on trying to prove yourself wrong, because by trying to prove our assumptions wrong we were able to build better strategies for our products.

Challenge three: better products and the outcome canvas

[00:14:02] So again, the first challenge was about creating better debates, the second challenge was about creating better strategies. The third challenge I faced was about building better products. Now we'll fast forward a little bit to 2018. I was still the director of product on the front end for Booking.com, but now I was seeing a different type of pattern appear. It was one of having too much of a short-term view in what we were prioritizing and investing in.

[00:14:31] What I mean by this is that we were jumping from small problem to small problem. We weren't showing the determination to actually try to solve bigger user problems. And what do I mean by a big user problem and a small user problem? A small user problem would be, if you think back to the example before about Just Booked, changing text, changing color, changing copy, these are little things about usability. What I wanted us to invest more in is actual user outcomes. How do we help customers find the right accommodation that meets their expectations, that they're going to enjoy on their trip? How do we make sure that they're able to filter and find just the right thing? How do we make sure that their special requests that they make to the hotel are actually fulfilled, or they get a notification if it can't be? These types of bigger challenges.

[00:15:29] And the root cause of this, that I started to understand, was that there was too much focus on a success rate, the success rate of experiments. The teams felt like, to be successful they needed to show that a significant amount of the experiments they ran were positive[?] and that we kept the change. So they were optimizing for the success rate. But what I wanted to happen was for us to take a step back and really try to prove ourselves wrong, so we could solve those bigger problems.

[00:16:01] But I was struggling to communicate my expectations. I had these conversations, but translating my thinking model wasn't working. What I found actually made this possible was to structure my thinking with what I ultimately came to call the outcome canvas. This was a way for me to codify my product thinking, how I like to think about a problem and approach solving it for our users.

[00:16:30] And it all begins with the outcome. This is the goal. What is the difference that we want to make in the life of the user? Because this is how we know, okay, are we building something that's useful and unique? And then let's think about the sizing. Okay, how many people can we help with this? Because with that outcome and that sizing we start to get a sense of, okay, is this the type of problem we want to invest in, is it a big enough problem that's going to be worth persevering and worth trying to figure out what the right strategy is?

[00:17:00] Next, outputs. Let's think about those short-term signals and metrics. How are we going to know that the users are actually reaching the outcome that we want? Now this middle bit is going to be familiar, because this is the strategic learning that I talked about before. I really feel like this is the most important part of the canvas, because this is the learning loop of trying to prove ourselves wrong. But again, we want to be determined to work here, because we figured out, hey, the outcome, the sizing, this is a meaty problem, this is good for our users and our business, we want to be working on this. So let's take the time to figure out what assumptions we need to prove wrong, what is the right strategy here.

[00:17:43] And then to think about, okay, what is the short term focus of our team right now, what's currently in here? Maybe you could think about this like, what's our current sprint or what are we working on this quarter? And we want then to also be clear about the long term roadmap, look a bit further ahead. Again this is based on our strategy: what are the milestones that we need to reach, maybe milestone one, milestone two, three, et cetera. And again we have confidence that this is the right way, because of the work we've done to try to prove our assumptions wrong, to build the right strategy.

Three buckets of impact

[00:18:16] Now, to bring it all together and to really make it clear for my teams that to be successful it wasn't just about the success rate of an experiment, there are going to be three buckets of impact that I wanted to see. I believe that high-performing teams will be able to show learnings, short-term impact and long-term impact.

[00:18:43] So learnings, again, it's about the better strategy. It doesn't matter how many tests you run or experiments. What matters is, are we iterating towards a better and better strategy? So those assumptions being invalidated, the evidence being collected. Then let's look at the short-term impact: what is the value that we are creating for users today? Again, we see that in the output signals, those KPIs, those metrics. And then let's talk about the long-term, the value that we are creating for users later. Again I see this as the progress against our roadmap milestones, so how are we moving towards our strategic future?

[00:19:24] What I found helped was then to ask teams, whenever we do status updates, let's talk about all three. Because then you made it more clear and it made it safer to say, hey, yeah, we've done a lot of learnings, but these are learnings that matter because they're actually moving us in the right direction. And I've also found that different products at different maturity are going to have a different balance of these three. I often don't expect that a team is going to have equal amounts of learnings, short term and long term, but it's going to depend on where the product is in this phase.

[00:19:59] But ultimately the takeaway here is to structure the thinking with the outcome canvas, because by piecing together all these elements you're able to more clearly see the different types of impact that we are developing, that ultimately helps us build better products.

[00:20:21] So really the key message I want to give to you today is: try to prove yourself wrong. Look for the counter arguments so you can have better debates. Clarify for yourself and your teams which learnings matter so you can build better strategies. And use an outcome canvas to create better products by pulling it all together. Now, I have found this thinking model and the outcome canvas has helped me, and if you'd be interested in trying it yourself, there is a download available on a LinkedIn article that I've written, or you can use the QR code here to scan and get a direct link. I would love to connect with anyone that would like to, and particularly if you disagree with some of the things I've said today, I would love to chat and hear your counter-argument. Thank you.