Solving Problems To Build A Compelling Product

Deborah Clarke at UXDX Europe. Video: https://youtu.be/mudBBPLJAsY

Readable transcript: edited from the recording's captions for readability (fillers and false starts removed, punctuation and section headings added). Wording is the speaker's own. Timestamps are positions in the video. Names marked [?] could not be verified against the audio.

Myths, truths and an atypical journey

[00:00:07] Hi guys. Today I'm going to talk about something that's very near and dear to all of our hearts, and that is solving problems. I'm going to talk through a few myths and truths. As you think about what's a myth versus what's the truth, the difference between those things is what's widely accepted, what are widely understood ways of going about things. I think that's part of the reason why you're here today: you're here to hear about people that are solving problems in new and unique ways.

[00:00:40] The reason why I want to talk about myths is because in debunking those myths, and in thinking about problems in different ways, that's the way you actually get to the secret sauce of product development. There is something atypical about that, right? It's not just accepting something, and it's not just going with whatever solution ends up coming about as part of solving a problem. It's actually about thinking about this atypical nature of problems, and different ways of going about solving those problems. So I'm going to talk to three today.

[00:01:16] I suppose I've been on a fairly atypical journey myself, in my life and in my career. I grew up in the United States. You probably can't tell that from my accent, or maybe you can with certain words. I then spent some time when I was younger in Australia. I then started my European journey, or European life, in France, and I've been in Ireland for the last six years. So if you just think about those different contexts, and the way that I've lived my life and my career, it definitely throws up different ways of thinking.

[00:01:51] Then, I started in art. I did my degree in art. I was in the gallery space, so I was doing lots of paintings and lots of drawings, and I ended up in business, crunching a lot of numbers and thinking about products in different ways. And then I also started my product development journey in beer. Beer was really fun, I enjoyed that a lot, but I actually like software a little bit better, which hopefully you'll see today.

What CarTrawler does

[00:02:15] I'll talk about CarTrawler, which is the name of the company. A lot of you probably haven't heard of CarTrawler, because, like me, it is a little bit atypical. It's not necessarily a commonly understood problem, because we're a little bit of a middleman in the software space. The way that we work is we have connections built with lots of different car rental companies around the world. We have about sixteen hundred different connectors built with all these different suppliers, or car hire companies, around the world. And then we work with lots of big partners in the travel space to actually plug our software into their systems. So it's quite a different way of approaching a product, and that in its own right means different things and different ways of solving problems.

Three myths

[00:03:07] As I said, I'm going to talk about three myths that I feel have definitely been debunked: things that are either commonly accepted in your spaces and look a little bit different for us, and may also look different for you. The first one is that it's hard. We often, I suppose, think about solving product problems as being something that's quite hard, and while in some ways you could agree with this statement, in other ways I'd say that it is a bit different.

[00:03:31] Second is that it's just about the user. Focusing on your end user, and actually understanding what your end user wants, is absolutely mission-critical to how you develop your product, but it's not just about that user. And the third is going after those big bang moments. We all want a product that's sleek and sexy, something that's different, something that's really exciting to look at. But ultimately, is it really about the big bang, or is it about an iterative process to get there? Or are there little things that you can do to touch up the product to make it better?

Three examples: downtown, compare and insurance

[00:04:05] I'm going to talk through three examples, very tangible UIs, and hopefully as we go through these slides you can see some of the UIs that I'm referring to. The first one is a map display at the back that I'm going to call downtown. That map display is solving for a cohort of our users that don't rent at airports. Typically, if you think about how you rent a car, you usually rent it for your holidays. You pick it up at the airport and then you drive it around while you're on your holidays. This cohort of user goes, "You know what, I'm going to my hotel. I'm going to rent my car while I'm in the city center." And so actually location is a lot more important to them than maybe some other things that you typically think of.

[00:04:49] We then have insurance. When you're renting a very high-value asset like a car that's worth around 30,000 euros, you want to be insured. You don't want to damage that car, you don't want to pay towards that. So insurance is a really important part of our funnel and our product. And then I'm going to talk a little bit about filtering as well, because I just love filtering. I think it's a really interesting way to think about how our users are comparing the content that they're coming across and what they're selecting.

[00:05:16] As I said, downtown is about location, and we know that 40% of our users actually cite location as being the primary driver. This is the primary reason why they're actually going after the car rental in the city center, and this is what's going to drive their decision as to which car they actually select. And there you go. Okay, well, that's interesting. So what? People want locations, so we've put a map display on there. That's great. We're trying to cater towards that interest, trying to build software products that work in that vein.

[00:05:46] And the "so what" of this is: here's our airport display. When you pick up a car from an airport, when you're selecting a car from an airport, you see this list view. You're going, well, that's quite different from a map. How people interact with that is very different to how people interact with a map. And the "so what" of this is that when you look at that display, the map converts at 50% less. That's something that doesn't quite add up, because there could be people that are picking up cars at airports and there could be people that are picking up cars in the city center, and they could be the same user, and yet we're converting at fifty percent less. So you're going, what's the story with that? How does that work?

[00:06:24] The second one I mentioned was around filtering, and how we actually make it easier for our users to compare cars. As you can see there, this is a list of the names of cars, a lot of car jargon: economy, compact, mini, very commonly understood terms in the car rental space. In compare, we looked at some interesting stats with our product insights team around how many people are looking at these sites and selecting cars, and how they're behaving on the site when they select those cars. What you can see from these numbers is that they're spending a lot less time on mobile, which is the display I just showed you, and they're selecting a lot more cars. So they're spending less time on that stage of selecting a car, and they're selecting a lot more. It means that it's really hard for them to see, whereas on desktop they're spending a heck of a lot of time and they don't select many cars.

[00:07:19] And you're going, okay, that's kind of interesting. But the "so what" of that is, when you look at our desktop car size filter and you look at our mobile car size filter, you go, well, there's a difference there in terms of user experience, in terms of how people are selecting those sizes of cars, and there's a bit more context there. Does that mean something? Between that 10.7% conversion rate and that 3.49% conversion rate, that's a huge difference. And we also know that our users use desktop and mobile to book our cars, and it is very much the same user. So why is there that 67% difference between those two displays?

[00:08:04] And then finally, insurance. As I said, insurance is an important component part of our funnel, of how people rent cars and how people think about renting cars. The reason for that is, as I said, it's a high-value asset. It's something that you're going to be driving around, belonging to a car hire company, and it's something you get upsold very aggressively at the desk, if any of you have gone through this experience before. So what we offer is exactly that. We go, "Right, not to worry, you can book this as part of your booking," and it's something that's important to you as a user as you think about car rental.

[00:08:32] What we found was that 53% of our users didn't understand the differences between these two insurance products, so much so that they actually called out the naming convention that we have for those two different insurance products. They said, "Yeah, I don't understand the difference between basic and premium." And that's interesting, because you're going, okay, they understand the products, because they're calling them by their names, but they actually don't understand the differences between those two things. And that's over half of our user base. That's a problem. And the "so what" of that component is that we actually saw conversion erode by about 10%. So it's a really important problem for us to solve, because this is such an important part of the journey and our funnel.

Myth one: it's hard

[00:09:13] If we go back to those three myths and we talk about the first one, it being hard, product development being hard, solving problems being hard: what I'd urge you to think about is this overwhelming feeling that we all have in a world of lots of different opportunities, probably 50-plus different ways of solving some of those problems. I spoke to lots of different ways of thinking about those problems, and I'm lucky to work in a company that has lots of opportunities to go after. We've got loads of different ways in which we could be approaching the product, and that's really exciting. But there's something about that overwhelming feeling that's important to harness, in a way.

[00:10:00] The way I liken it is to this screen. When you walk into a grocery store and you're hungry, and you have lots of different ambitions and lots of different opportunities you could be going after, or you're just genuinely hungry, ultimately you end up picking up lots of things you don't need, lots of things you don't want to eat, lots of things that are probably not even going to satisfy that hunger. So the point here is really that it's not so much about it being overwhelming. It's about creating the right level of focus, and knowing what you want when you go in.

[00:10:36] To bring it back to those three examples, the first being downtown, I'm going to talk about time to focus, time to break down the concept of an idea, and then the time to actually develop that particular feature. When you look at that time to focus, and when you look at the time to break down the idea or concept, you actually see that as these three features come together, there are different time differentials. And that time to focus is a very important part of what it actually means to realize a feature. If people aren't focused on it, no matter what intel you have, no matter what kind of understanding you have, you ultimately don't even get to the problem in the first place. You don't even get to breaking it down.

[00:11:21] So if you think about those two different component parts of what it means to bring about a feature, it really is much more about how much we know and the buy-in associated with that feature. If you know a lot about it and you have high buy-in, it's actually going to get to market a lot faster. If you plot those out on the graph and you think about where they fall in terms of how quickly they're getting to market, the one that ended up the highest, if you recall from the previous slide, is the thing that got to market quicker, because we had the focus. We knew exactly what we were going after, and we knew a lot about the insurance product. When we knew a lot about it and we had that focus, that's what ultimately got the feature out to market. So for me, it's less about being hard. It's more about taming that hunger, more about ensuring that that focus is in place and that you've actually got buy-in.

Myth two: it's just about the user

[00:12:15] To that end, if we think about our user, and we think about it being just about the user, the first thing I'd call out is that there are lots of different groups involved in bringing forward features. I'm sure in all of your companies you've got lots of different stakeholders involved in bringing about features and thinking about features. But every single one of these features, that downtown feature, was driven by user research. Everything that we knew about downtown and our downtown users was driven by user research. Same thing with compare, same thing with insurance.

[00:12:48] And you're going, okay, well then what's the difference? How did they get out to the market in faster ways or not faster ways? The thing that I'd urge you to think about is that it's not just about that user research. That user research absolutely focuses the problem in terms of solving a very real UI component, but getting to the answer had everything to do with a company-wide performance initiative, which just meant that people were focused on this, people were bought into this, and they understood that we needed to solve this problem, which is ultimately going to get you to that answer.

[00:13:22] So if we think again about those three different component parts of bringing a feature together, you've got your time to focus, your time to break down that concept, and your time to develop, with the time to develop often being the shortest part of the problem. Breaking down the concept, and that focus and that buy-in, are the most important component parts of what it means to realize a feature, to bring it into the market. And then ultimately your time to learning is very, very different depending on how you understand that focus and how you understand the problem.

[00:13:58] If you just focus on a user, or if you just use user research to try and advocate for solving a problem, what I'd say is, a hundred percent, you have to understand that user, and you have to go after the person that's ultimately using your site to solve the problem. But if you just focus on that, then you're going to be slower off the blocks, because you need to understand the wider context of the problem. You need to understand who the different stakeholders are that are in the mix, and how they're contributing to that problem, to really create that compelling product and feature.

Myth three: the big bang

[00:14:27] And the final thing, one of my favorite things to talk about, is the big bang activity. I love really beautiful products. I love things that are fun to interact with, and I really love moments in a product that are just sleek and really, really nice to use, and I think we all do in some way. But we also think about products like, "Okay, I'm going to have to entirely change this layout. The whole flow is going to have to change, because ultimately this doesn't work." We have those big, bold ideas about approaching our product. What I'd urge you to think about is solving those problems, again, in different ways.

[00:15:09] If we bring it back down to this downtown example, and the differences between that map and list display: the users said it was all about location, so we solved that. We have a map there, and we're saying, "Right, okay, we know it's about location, so we're going to give you a map to be able to make that selection." But then we saw that that display converted at fifty percent less than this display, and you're going, well, why is that the case? If our users are saying this is what it's about, why is that happening? And we know that they're very different user experiences. They don't look the same at all.

[00:15:42] So what we said was, what if we created a downtown list? What if we combined those two things together? As you can see, maybe there's a little map there that pops out into a modal, that ultimately gives that user that location intel and changes the user experience ever so slightly. It wasn't difficult to do. We had a map, we had a list. What if we pop a little tiny map in there, and then pop that out into a modal that we already had built? And what we saw was a plus 38% increase in conversion, which is massive for a business of our size. Just such a huge win, when you think about that problem in a really, really simple way, and you don't go for the all-singing, all-dancing, thinking about every single component part of it. You actually test and try one version of that.

[00:16:33] The next piece is on compare, as I said, with filtering. The biggest difference between these two things, can anybody tell me? Yeah, images. Yep. There's images on this one and there's no images on this one. They've all got the same names. One's a carousel and one's a bit more of a list, so that is a difference as well. So we said, okay, right, what if we actually made it easier to compare? We understood that problem again, and we looked at those different user experiences. What if we actually changed that to be images and little blocks, similar to what we had on our desktop display, knowing that there was that big difference between them?

[00:17:16] We knew that this one converted at 3.49%. This one converted at 4.56%, which is about a 30% increase. And you're just going, that was a really simple thing. Why didn't we do that before? Why didn't we cop on to that, and why didn't we think about it in that way? But again, a very simple change. Not about being complicated, not about changing the whole filtering flow or how we thought about compare. But this will drive that understanding even further, because again, you've got the product to market and you're learning from it.

[00:17:49] And then finally, on insurance. As I said, users were saying, "Right, I don't understand the differences between these insurance offers. I don't understand the basic and premium products." We saw that 10% drop in conversion, and we knew that the user experience was unclear. People were able to call out the insurance product names, they were able to talk about them, but they really didn't understand what they were. So we decided to change one very simple thing. Anybody know what it is? Nope.

[00:18:19] We changed it from basic and premium to limited and premium. We changed the words. We changed the names of these products. People were calling out those words; they were calling out those things as things that they didn't understand. And what we saw was, again, an 11% increase. Now, to be fair, we experimented with lots of different wording changes here, so this was definitely, I think, the third test we did. But it was that notion that when you change small things, they have impacts. Even when we changed one word, so we changed it to "standard" instead of "premium," what we saw was a minus 6% drop. So you go: that's minus six percent, that's plus eleven percent. Those are huge swings, and all we did was change the words based off of that core insight, that people don't understand what these products are. They don't understand what these products do.

Small moments of delight

[00:19:08] One of my favorite things, as I said, about products is those small moments, those iterative moments. I've got up here, on the left-hand side for you guys, one of my favorite apps, Deliveroo [?]. Anyone use it? Yeah. One of the things I always do is I always give a tip. And why do I give a tip? Because it's a small moment of delight. When you click on that little plus sign, you get this little burst of excitement, and then you see your little smiley face, and it grows even bigger when you give bigger tips. Every single time, I'm locked in. I just enjoy pressing that button, and then you think, "Oh, well, I'm giving someone a tip, this is a nice thing to do." I want to urge you to think about that: don't underestimate these really small things, these really small moments of delight. They're so important to what it means to realize a product.

[00:20:01] I do have a few examples from our own product. When users compare cars, they click on the little heart and it pops up into the little comparison display, so it enables them to compare a little bit better. When we put that live, we saw a plus 18% increase. We also did something different on this display, which you probably can't even see, because it's such a small, nuanced way of accounting for things. When users added, say, an additional driver to their car hire, or added other things to their basket, what we did was we wiggled the little basket up in the top right-hand corner. You go, well, that's such a small change, it doesn't actually mean anything. But it actually led to a plus 7% increase in our conversion, once again. So it's not about these big, bold changes. It is about the small moments, the small moments of delight that you bring and incorporate into the product to solve those problems.

Debunking the myths

[00:20:55] So if we think about this in new and different ways and debunk those myths: it's not hard. It's just a little bit more about focus, focusing on what you're actually trying to go after, and really encapsulating that in a succinct and understood way. It's not just about the user. If you just focus on the user, you don't actually get the product to market, and you're not actually learning from it, because there are so many other component parts to thinking about that product. And don't go for the big bang all the time. Don't redesign the flow and think about everything in new and different ways every single time you approach a feature. Think about that nuance, think about those small things and those small changes that really, really make a difference. Or just have fun. Easy way to remember that one.

[00:21:43] When you think about having fun, I said at the start of my talk that solving problems to me is fun. I think that's why a lot of us are here today: because we enjoy solving problems. And when you're solving problems, I urge you to think about these contexts in different ways, and be this guy. Approach the way in which you engage in an environment, modify your way of thinking about things, and debunk those myths in an atypical way. Don't go after the way that everybody's telling you to think about things. Actually think about problems in new and different ways. Thank you.

Q&A

[00:22:31] Host: Thank you, Deborah, that was great. Right before lunch, we have time for a few questions for Deborah. Anybody have some questions? Yeah, we have one right here. Great, keep your hand up.

[00:22:59] Audience: Can you hear me? Yeah. Those stats that you showed, did you test in production or pre-prod?

[00:23:05] Deborah: We do both. We do usability testing and prototype testing before, depending on how different the feature is going to be, and depending on usability issues that we've seen in the past that could be difficult, say a carousel interaction where we're not quite sure if that's going to work. But we do most of our testing actually in production, which is really around what actually happens when you put it out into a live environment, and what it affects.

[00:23:31] Audience: And what's the average sample? How many users do you test for each?

[00:23:37] Deborah: It depends, really. Sometimes we're testing different channels, or we're testing, say, different markets. We look at, say, the US market versus maybe the Portuguese market or the Spanish market, that kind of thing. Obviously when you test different sizes of populations, when you test different software features in those markets, what you find is that sometimes it can take longer to get to a testing result. That's the nuance of the A/B testing space, which is that it has to do with two things. One is the size of the test, the size of the traffic either side. And the other piece is how big an uplift you get. If it's small, then it takes a really long time. If it's a large uplift, like plus 11%, then it'll take no time at all. So it really depends on the test. It depends on the feature.

[00:24:30] Audience: Thank you.

[00:24:31] Host: Great questions. Okay, any other questions? Yep, one over here. Great.

[00:24:35] Audience: Hi, my name is Nicholas. My question is, how do you cycle through these different changes? Because if you do a lot of things at the same time, you probably won't get an exact measurement of the impact of each change. So how do you go about choosing your tests and cycling through them?

[00:25:02] Deborah: Yeah, that's a great question. There are a couple of things. One is that we test lots of different things, and sometimes what we choose to do is test different component parts of the feature at different times. For example, on the wording test, all we tested was the wording. On the downtown list display, you can see that we obviously changed a number of different elements, even though they were really small changes. And sometimes what we do is split out some of the changes across lots of different branches of the funnel, and we go about testing those different things.

[00:25:38] What we use is something called a multi-armed bandit approach, whereby we go after calculating which arm of the test is winning. We calculate the probability that it's going to win, and then we funnel more and more traffic into those arms of the test when we see that one's winning over another, based off of that probability. So it means we get to test results a lot faster.

[00:26:01] Audience: Yeah, thank you.

[00:26:04] Host: Great. I think we have time for one more question before lunch. Anybody have a final question? Okay, great. Well, thanks, Deborah, really appreciate it.