Why Failing Fast is Not Good Enough

Praful Poddar at UXDX APAC. Video: https://youtu.be/ocG-5C41ToM

Readable transcript: edited from the recording's captions for readability (fillers and false starts removed, punctuation and section headings added). Wording is the speaker's own. Timestamps are positions in the video. Names marked [?] could not be verified against the audio.

Introduction

[00:00:00] Thanks, everyone, for joining today. Good morning, good afternoon, good evening. The reason I'm starting off with this picture is I'm truly excited, mainly for two reasons. The first: it's 2021, and I'm hoping and wishing that this is a much better year for all of us across the globe, from a variety of perspectives. And the second: the opportunity to talk to you and share some of my humble learnings from my experience. I'm very excited to be talking to some people who are looking to learn something new today. I also want to thank UXDX for putting this forum together. I think it's truly an interesting forum to network with like-minded people and learn something new for myself today as well.

[00:00:45] A little bit about me: I'm a product guy. I like using logic a lot. I'm learning how to keep things more and more simple; ironically, though, that's really difficult. On the personal front, I do enjoy reading and playing a bit of squash, although COVID did put a bit of a roadblock to that, but I'm getting back to it slow and steady. And one of the advantages of being at home is that I'm building new skills: my four-year-old is keeping me up to date on the latest in Play-Doh and Lego.

[00:01:17] For the ones that are curious about how this is going to span, I put this together as a spoiler, and I'm taking a risk here that people don't jump across. If I have timed myself right, I will spend around 20 to 22 minutes discussing the topic with you. Right now we are just setting this up at the beginning. I have a few learnings that I've tried to condense into three broad points that I want to talk to you about, by taking you through a very interesting story that we did on the ground, and then I'll conclude everything towards the end with a quick summary. I would urge you not to jump there already, because it will be a good experience for you to listen in on all the learnings, because they're fairly well connected and will make more sense.

Why failing fast is not good enough

[00:02:01] Let's get right into it. This is the topic that I wanted to discuss: why failing fast is not good enough. A provocative statement, and purposefully so. Don't get me wrong at all, I am a big fan of failures. In some of the projects and the teams that I've worked with before, I think we've celebrated failure, learned so much from failing fast, and applied those learnings to take more impactful products to users. However, when we reflect and really analyze those in a little bit of depth, there is almost another dimension, a different angle, which I feel needs to be managed and understood about failures, and what I want to take you through is how we uncovered some of those.

[00:02:43] We really went back and started to reflect, as a group, as a team, on how the last few quarters of last year had gone by, and what we had released as successes and failures. I'll try to condense these into three points that we took away from all those analyses that we did. The first was that while projects were going on, successes were happening, failures were happening, the broad sense of "are we getting smarter?" just didn't seem to be there. That was reflected in people not talking about learnings as much as we would have liked.

[00:03:17] The second: when we tried to do a high-level analysis of different kinds of failures in terms of size, nature and team, and laid them out over a timeline, the frequency of the failures, now versus a few quarters back, was broadly remaining the same. So we were not really improving on that front either. And the third one, to our surprise: we actually had no objective view on what this was costing us, how much we were pumping into these failures, and what net ROI we were able to generate. Were we really contributing back to business and consumer value in the end?

[00:03:52] Armed with these three insights, what we decided to do was take a different path this time around: pick up a team, pick up a project and try something differently, with the perspective of optimizing our failures based on what we learned from this reflection and retrospective. What I'm going to do next is take you through the story of how we implemented some of these learnings on the ground, what we did, what worked, and condense them into three points that you can take away as practical applications for yourself.

The setting: car inspection at OLX Autos

[00:04:23] Before I get into the story itself, let me set this up well. This is the organization I work for, OLX Autos. Essentially what we're trying to create is a differentiated used-car buying and selling experience for users. Very simply put, a person who wants to sell their car comes to us and sells the car; we refurbish it, shine it up and then sell it on to the buyer. A simplified view: it's a C2B2C model. Our platform also allows people who are interested in selling cars to end consumers themselves to do so. We facilitate that, not just by helping buyers and sellers connect, but also by providing a lot of services on the ground to make sure that the transaction can be much more seamless and a better consumer experience. So this is the setting.

[00:05:07] This is a key component that happens currently in a used-car process: inspection. I'm sure most of us are quite familiar with what happens in a car inspection. It's a fairly technical product that you're trying to sell and buy, and inspection is used to understand the objective condition of the car and arrive at an accurate price. This process is super important to this journey. Although a lot of advances have happened in this area over the years, it's still quite manual. You still need a guy who's fairly expert to go through this checklist of spots, look at both digital and physical data, and make a judgment call.

[00:05:49] But things are changing as well. What's really happening across the board is that a lot of companies and people are trying this out. We're trying to change this process itself. We're almost trying to say, hey, can we solve this better through technology, not just making it a better consumer experience but also making it much more sustainable and scalable? So picture yourself as a consumer, doing this entire process by walking around your car with just your phone, and we make that happen. That was the hypothesis, or the problem, that we were trying to solve in the area that we took up to apply these new learnings that we had come across with respect to failures. Let's dig into what we learned.

Learning one: define your quarter-inch hole

[00:06:29] The first one really was that, by nature, we as product, technology and design folks are in a rush to build. We want to take new solutions to consumers very, very fast. What happens a lot of the time because of that is we don't end up spending enough time defining what we're solving for first. Let me quote Theodore Levitt here, a professor from Harvard. He says, in the context of people wanting to buy quarter-inch drills: "People are not actually looking for quarter-inch drills. They're looking for quarter-inch holes." So it's really the core need that we're going after.

[00:07:07] Let's extend that further. It's actually not the quarter-inch hole. They're actually looking for the painting on the wall. Take it a step further: they're actually looking for the aesthetics that it brings to the room, the feeling I get when I wake up to that painting. If you continue to do this exercise, you keep going to figure out the real user need that you're trying to solve for, and to deliver an impactful solution. Here's how that helped.

[00:07:32] Bring back our example of building this state-of-the-art inspection process. Before we jumped into building something, we tried to ascertain where we should focus when we built the solution. Should we focus on the seller, in which case we'd probably be solving for convenience and bringing transparent prices to this user? Or should we focus on the buyer, in which case we're trying to solve for trust: with this car, am I getting a good deal or not? Will this car not break down six months later? After enough deliberation and thought, we were able to arrive at the conclusion that what we're actually trying to solve for in the end is really the buyer, while our initial bent was, hey, this is probably a seller-oriented solution.

[00:08:16] How did that help us? It really helped us to design the experiment around what people are going after. This is an inspection report, and what we're essentially trying to do here is gather these data points through this new experience that we've built. There are two ways to approach it. We can either collect all the data points that the seller can give very conveniently, or we can focus on the data points that the buyer needs to build trust, to almost have the confidence to put down an advance amount online. Armed with the information that it is a buyer-focused experience, we were able to build the right experiment, one that solved for buyer trust, which could actually lead to some seller inconvenience in the end.

[00:08:53] Basically, what we did was avoid a bad experiment design by focusing on the problem, which helped us not go down the path of a false negative or false positive, essentially non-usable learnings coming out of a particular thing. That was learning one.

Learning two: fail frugally, before you invest

[00:09:09] Learning two is really to understand the cost that comes along with failure, and how these costs shoot up as we get more and more invested in a particular solution. Let me again bring back our example. For the solution that we were building, the initial thought, the initial idea, was, hey, this will probably be a video-oriented solution. We'll get people to just walk around the car with their phones, both interior and exterior, and give us a view through a video. Then we can just take this video, slice and dice it, and get all the answers that we need for the decision making that we wanted to do.

[00:09:46] There were a lot of reasons contributing to this kind of thought process. A lot of platforms had come up with video-oriented solutions in the consumer space in the markets that we were operating in, a lot of acceleration was happening there, and there were options. People were getting very comfortable with videos in general across the board. Network and data speeds were also improving, so it was becoming easier to support videos on the mobile network.

[00:10:08] To add to that, we had also built somewhat of a solution in another category of our platform. Our platform also allows users to sell anything used through our product, and this is an example of an electric appliance, a mixer-grinder. We had built a solution where, if a seller wants to showcase the working condition of this particular product, they can make a quick video which the buyer can use for decision making. And the thought there was, hey, why don't we just repurpose this for the car category, and we'll be up and running with the solution. You're very tempted to go down this path and get invested in this idea very soon.

[00:10:44] But going back to our reflection, we said, hey, we should first figure out if there is enough evidence for us to move along this path, and really reduce our chances of failure. We did two things. The first one was, hey, can we be really brief and try this out today without even building anything? We have our stores, we have inspection engineers, we have consumers, and we have video capabilities. They just might not be in our product, but people are fairly comfortable using WhatsApp video calling or their native phone cameras. Can we just utilize that?

[00:11:15] So we started to do some on-the-ground pilots. We got people who were not interested in bringing the car to a store to get on video calls with our inspection engineers. It was almost like a guided experience, where the inspection engineer was running them through the process. And if people were not ready to do that, we gave them a little guidance and said, can you make a video and just send it to us? That would give us a bit of learning: can we actually utilize these information points?

[00:11:37] The second: we were not the only ones in the market trying to solve this right now. There are a lot of SaaS-based organizations, insurance companies, OEMs and other competitors trying to build something similar, and there's a lot happening in the market too. So we went out and spoke to a lot of people, and we came across a bunch of solutions, from AR to VR, hardware, software, video.

[00:12:00] Piecing these two together, and very, very critically examining what would work for our consumers, what benefit we would get, and what happens in the markets that we operate in, we came out with a completely different conclusion. We came to the conclusion that video is not the path to go. Let me try and explain why. First of all, it's not a super consumer experience. Picture yourself trying to make a two-, three-minute video around your car. It's so awkward. You don't know whether you should be talking while you're making the video or not. If you're trying to open the car, and if you're trying to show us the engine, the camera's going all haywire. It's not a good experience.

[00:12:37] Secondly, the moment you switch your camera from photo to video, the quality dips significantly. The quality that you get from the video is much lower than what we would expect. And third, the size of the video that gets created, which is multiple hundreds of MBs in the end: it's just not practically feasible to get that uploaded and shared in an optimized manner. So video was clearly not the path for us to go. In fact, what we realized was that a very simple, smart image capture experience, which guides how these images should be taken with the smartphone, where it's more or less a tap-tap experience for the user to give us a bunch of information about the car, works beautifully for us to get all the information that we were trying to get with this video.

[00:13:18] So you see what we did. If I look at this cycle of product development, rather than failing at the gates of a launch, where we would have invested a lot in this idea, built it, launched it and figured out, hey, it's just not working, we actually failed really frugally. We failed at the concept stage itself, which required a lot fewer resources and a lot less cost. That's learning number two.

Learning three: stagger discovery and delivery

[00:13:42] Learning number three, and this one's very interesting, so bear with me on this. I'm sure most of us are familiar with OKRs. I think it's a fantastic tool and strategy to give teams really actionable goals that they can own. But I do feel there are some challenges when they get implemented on the ground, in what really happens practically, at least in the digital environment that I work in. Think of a quarter-on-quarter OKR cycle. In a quarter, what happens is all of the work around discovery, which means, hey, what are we supposed to build, gets front-loaded into the quarter. So everybody's scrambling to figure out what gets built, and all of the launches and deliveries happen towards the back end of the quarter.

[00:14:21] That's not the most optimal use of resources. The kind of resources we need when we are doing discovery work or delivery work are very different. And if all our initiatives and projects are at the same stage of discovery or delivery, we are putting a lot of pressure on these resources. What that does is bring down the quality significantly, and we are increasing our chances of failure. So we looked at this and said, hey, how can we systematically, and on purpose, change this?

[00:14:49] Just to give you an idea, this is a very oversimplified flow. What you would see in a quarter is that the problem, opportunity or hypothesis work will probably be at the front of the quarter, and the launch and the analysis will probably always be happening at the back, with a bunch of these things happening for all the initiatives that are there in the OKR. So resources like design, research, prototype testing and product are all constrained doing problem and opportunity analysis for all the initiatives at the front of the quarter, which does not provide as much quality as you would expect, again bringing up the chances of failure.

[00:15:23] So we systematically tried to change that. The same example: let's assume we had these five or six initiatives that we wanted to pick up. We wanted to integrate with an OEM to get the car's service history. We wanted to change a bunch of things in the UI. We also wanted to do some data science things, where we can extract text from the image, and then play around with the form that we had in terms of configurations and ordering, so that we could get the best possible information.

[00:15:49] Rather than having everything happen at the same time, we mixed it up within a particular timeframe. We said, hey, all these projects need to be systematically driven so they're at a different point in the life cycle at any given time. Some will be more in discovery; some will be more in delivery. What that effectively did was ease the pressure on the resources, so they could work on this with quality, lessening our chances of failure going forward.

[00:16:14] If I extend this a little bit further, picture that you have a lot of these initiatives you need to work on within a particular timeframe, and how you would organize or plan for them. Think of having different kinds of things happening along the product life cycle at a particular point in time. What that does is lessen the pressure on the resources that work on this, for good, better, best quality. And we could use this in multiple capacities. Picture a product manager or design manager that has to do this for one project but multiple tasks. We can zoom into the tasks and say, can I plan my tasks so that they're all at different stages at a particular point in time? And we can zoom out a level, where a manager needs to plan for multiple teams: can I get teams to be working at different stages as well?

[00:17:08] This has also done two things, particularly in view of stakeholder management and expectations. One, it allows a lot of room for us to spend time on the larger initiatives. In this particular example, look at initiative four, where we need to give ourselves a lot of time to do discovery work. What that's really done is, while all of the other initiatives are constantly delivering something, we've got this cushion to work on this in a bit more depth. It also gives a lot of visibility, not just of the progress but also of the process that product follows, to stakeholders, and manages expectations a lot better. So that's learning number three.

Summary

[00:17:44] Those are the three learnings I had. I want to bring all of this together in a quick summary. The first one: do spend significant time defining what your quarter-inch hole is. What we're essentially trying to do there is avoid bad experiment design and going down the path of false positives or false negatives. The second one is thinking about the costs that you incur by failing at a later stage, and trying to validate at each stage to save on that cost later in the journey of product development. And the last is to ease the pressure on the teams that need to work within a particular timeframe in the life cycle: try to spread the work systematically across all your initiatives at a particular point in time.

[00:18:26] You've been a great audience. Thank you so much for listening and spending the time today. I want to leave you all with this one quote, a quality that I value a lot for myself and that has shaped what I am and what I do today: continue to stay curious, keep learning and keep an open mind. Take care. Cheers.