Preventing Churn Through XPs

Daniele Joubert at UXDX EMEA. Video: https://youtu.be/KIMQCyKvnAI

Readable transcript: edited from the recording's captions for readability (fillers and false starts removed, punctuation and section headings added). Wording is the speaker's own. Timestamps are positions in the video. Names marked [?] could not be verified against the audio.

Uber Eats delivers things that aren't food

[00:00:00] Hello. I'm Daniele Joubert, and I will be chatting to you guys about preventing churn through XPs. I currently work at Uber Eats in South Africa, and I'm the head of growth and consumer for sub-Saharan Africa. It's been an incredible journey working at Uber Eats, and we've learned so much. I'm really excited to take you guys through everything around XPs and how to actually implement them and build them into your daily strategy.

[00:00:29] To start with, Uber Eats. Uber Eats is delivering beauty products and flowers and toothpaste and firewood and baby necessities and pet food, and all of these things that are not food. It's definitely not in the name. I think a lot of people around the world ask how Uber Eats went from delivering restaurant food, as hot and as quickly as possible, to their clients, to delivering everything else that is not food. And the crux of my talk is: how do you innovate and build better products for your clients, not just by listening to them?

[00:01:19] The agenda for today: listening to your customers is only the beginning. Then I'll take you guys through setting up your team for success when it comes to experimentation. We'll also chat through some ideas around guidelines for setting up experiments. And then I want to show you a case study in South Africa of an experiment that we ran, which again shows that the behaviors of eaters are sometimes a lot different from what we expected them to be.

Listening to your customers is only the beginning

[00:01:48] Listening to your customers is only the beginning. So how did Uber Eats get where we are today? The business started as a ride-sharing business, which you all know: this whole idea of, can you get a ride, can you go where you need to be at the click of a button? From there, we were one of the creators of this idea of a gig economy, building and creating earning opportunities for people in our network. And because we'd solved this idea of moving people from point A to point B at the touch of a button, this idea of delivering food started. We started testing it in the U.S.: can we get it right? Can our solutions and our logistics solve not just moving people, but moving food from point A to point B as effectively and as efficiently as possible?

[00:02:45] But let's jump to 2022. Suddenly the world got hit by a pandemic. Lockdown got announced, and no restaurants were able to operate. As Uber Eats, this is what we were doing. But as a business, we'd learned to adapt and to diversify and to make plans, and we were really able to enable our clients to get what they needed during difficult times, by ensuring safety and less contact between people. So we went from delivering food to delivering pharmaceutical products and groceries and liquor, anything that consumers needed that they could not get because of the lockdown restrictions.

[00:03:31] So, this idea of listening to your customers. If we had listened to our customers, we would have stayed a virtual food court. This is what it was: a virtual food court, one place where you can get all your favorite brands, order your favorite food and get it delivered in time. But by not just listening to our customers, by running XPs, by testing new product features, by doing trials, we've actually become an entire virtual mall. If I had asked people five years ago, "Hey, do you think Uber should be delivering jewelry or clothing?", most people would probably have said, "No, no, no, that's not a product fit." And today we are delivering just that: anything at the click of a button, in the convenience of your own home or a location where you need that specific thing.

Setting up your team for experimentation

[00:04:28] So how did we get here? How do you set up a team for success when it comes to experimentation? How do you set up your team to be able to build experiments and learn what your eaters, your clients, want without them being able to articulate it themselves? This entire idea starts with understanding your users in a data-conscious world. The first thing is: get to know your users. What is the information that you have on the behaviors, the decisions, the choices that they make in their experience with your product? For us, it could be when they order, what they order, what their basket size is, how frequently they order, and what some of the pain points and decision-making points are for your consumer in the journey.

[00:05:20] The second thing is: decentralize your data. Yes, keep personal information under lock; make sure that it is as safe as it can be. But give your team and the rest of the business access to information. The only way to truly create a data-driven culture is by giving them access to the information that they need in order to make better, data-driven decisions. And then finally, if you know your customers and you have access to your data, run experiments. Build cohorts based on behaviors, run experiments as quickly as possible, measure them as accurately as possible, and really use the results to adapt your strategy.

[00:06:05] But what are the fundamentals of making experimentation part of your business? The first fundamental is that it needs to be entrenched in what you do every single day. The number of problems and the number of unanswered questions that you have about your current users' needs and your future users' needs and behaviors just increases every single day. So in order for you to truly understand them better, to really improve your product, your experiences and your business performance overall, you really need to make experimentation a critical part of your strategy and execution. It shouldn't be an afterthought, or once a year, or only on big changes. It should be entrenched in every single decision that you make.

[00:06:59] The second thing: in order to make experimentation entrenched in your business, you need the technology to support it. Running multiple experiments weekly at the same time is very time-consuming. Focus on building technology platforms that allow teams to set up experiments very quickly. Make sure the technology not just allows them to set up the XP, but also to run the results, the XP analysis of basic stats [?]. If you have to ask your team to run these results manually every single time, it reduces their ability to run faster, create more experiments and ask more questions. So build the tech that allows them to do that on every single product that you develop, on every single question you have, on every engagement, as quickly as possible, and you'll see how quickly you can learn.

[00:08:01] And then the last fundamental is focus. When experimentation becomes entrenched in your company, and when you have the technology that allows you to quickly execute and validate your assumptions, the world becomes your oyster. But as soon as the world becomes your oyster, the skill lies in your ability to prioritize your assumptions. Make sure you are focused on only trying to test one thing at a time. Test one thing at a time, but run as many tests as possible. Focus on assumptions that could drive a higher return on investment. This could be from a financial perspective, like conversion, or it could be from an experiential perspective. Just make sure that there is one specific thing that you're focused on for that experiment, and only try to validate that assumption.

Why this matters: retention is king

[00:08:58] So why does this matter? In the past, acquisition was a given. A company's ability to get a user on the platform was the most important thing, because the friction to move between products or companies was high. Most companies invested a lot of money in getting their customers onto the platform, getting them to use it, getting them to download, and once they were there, they were sticky. But in today's world, and the technology environment that we work in, it's so easy for most consumers to jump between products and between companies. Banks, something that was very hard to move between a few years ago, are a click of a button.

[00:09:42] So in today's life, retention is king. The only way in which you can really drive and improve your retention is to drive loyalty, to improve their experience, to diversify your product, to create the delight moments that they love, and also product updates. But you can only achieve this if you entrench experimentation and make it part of your life, in order to give your consumers updates and experiences and loyalty experiences that they didn't even know they wanted, to be less dependent on market research, and to test your assumptions and ideas in a real-world environment.

Guidelines for setting up XPs

[00:10:28] So what are some of the guidelines for setting up XPs? The first guideline is that your objective, what you're trying to achieve, and your hypothesis are two different things. Your objective needs to be linked to a business question that you're trying to answer or achieve or solve for, and your hypothesis should be linked to what you will be testing specifically. The second thing is: spend time formulating your hypothesis. A lot of the time, something that might seem very straightforward ends up being very complicated, and something that might seem very complicated ends up being very straightforward. But if your hypothesis is clear and you know exactly what you are testing for, it is much easier to set up your entire experiment.

[00:11:21] The next thing is to clearly define your primary and your secondary KPIs. A lot of the time in experiments, the results could show improvements in your primary KPIs, but where people get caught up is unintended consequences. So ensure that you consider both the good that you were trying to achieve, and might have achieved, as well as the metrics that might have had a negative effect. What was that, and how do you balance the positive with the negative to truly understand what the impact of this change would be on your business?

[00:11:59] The fourth thing is to create a controlled environment, as much as it's feasible. This is a real-world experiment, and this is not a theoretical environment. So yes, exclude your treatment groups as much as possible from other elements and other engagements that could influence the results, but do not try to isolate them so completely that the results of your experiment are skewed and not a true reflection of what the results would be if you made it live in a real-world environment.

[00:12:35] The fifth thing is: think about the execution before you launch. The experiments that you do need to be linked to something that you can change, implement and impact. If you test assumptions that provide incredible results, but it's not something that you can deliver on, you will actually be wasting your time. So again, when it comes to focus, make sure you're testing something that will drive business value, as well as something that you can implement or that can impact your strategy, in order to get the most from it.

[00:13:09] And then lastly, test one hypothesis at a time. You take one hypothesis using different treatments, versus trying to test different hypotheses using one treatment. Really focus, to ensure that you know what you'll do with the results, because this will impact what it means for you going forward and the business results that you're trying to achieve.

Case study: choice architecture in promotions

[00:13:37] I want to end with a very, very interesting case study that we did in South Africa. The case study was around how we can use choice architecture to improve our conversion rates on the promotions that we run. The business objective was improving our conversion on promotional campaigns in order to increase our orders and our gross bookings, and our top-line performance in general. The hypothesis was that we really wanted to test and utilize behavioral heuristics, one specific one called choice architecture, in some of our campaigns, to see what the impact would be on our conversion rates for the users that we targeted.

[00:14:24] How we decided to set up this experiment: for treatment one, we would send three promotions in the same campaign to the same cohort of eaters, and allow the user to choose which of these promotions they would like to use, and then see whether they ended up using it, whether they ended up placing an order, and also what the value of those orders would be for the business. The second, third and fourth treatment groups would each only get one promo code, and each one of those promo codes would be linked to the same value as one of the promo codes in treatment one.

[00:15:06] Our primary KPIs: we wanted to test what the incremental impact would be to our business. What was the user uplift, what was the trip uplift, what was the gross bookings uplift that we would achieve if we ran these promos, across the different treatment groups? Our secondary KPI, however, was efficiency. It doesn't help if we target treatment one and it drives a lot of growth, but it actually costs us a lot more than what the other promotional engagements would have cost us.

[00:15:42] It's quite interesting if you see some of the engagements that we built. On the choice promo, treatment one, the users got emails and push notifications and in-app experiences, all linked to choice. "Hey, pick a promo that speaks to you. Get 75 Rand off because you're a solo eater, or get 150 Rand off because you're ordering for a family, or if you're spontaneous, get 99 off your order, but spend it in the next three hours." Again, gamifying their choices, but really giving them the option to select one of the options.

[00:16:19] This is an example of just treatment four and the 99-off promo. It was, "Hey, come into the app and get 99 off your next order." Again, we had emails and push notifications, and we had the same number of communications on treatment one as well as all the other treatment groups that we targeted.

[00:16:42] So if I asked you, person X, if you were a logical person and you got either one of these promotions, which one do you think would have received the most orders? Which one would have driven the best business performance? On the one side, it's busy. There's a lot of information, and the title is all around the choice and picking an option, versus this one [?], which is very straightforward: 99 off. I don't even think you can get a better deal than that anyway.

[00:17:17] But the results were so interesting. If you take treatment one and compare it with the averages of treatments two to four, the user uplift was 204 more, the GB uplift was 61 more, and on the secondary KPIs, the cost per incremental eater and the cost per incremental gross booking were a lot less. So the conclusion of this experiment was that by giving people a choice between three options instead of just giving them one, we got a higher uplift. The campaign drove a lot more efficient results. We created a gamified experience, which also increased our retention and engagement of the consumers that we targeted.

[00:18:07] For me, this is another example, a proof point, of how important it is not just to make assumptions about the effect your product updates or initiatives will have on your eaters' or your clients' behavior, but rather to really embed experimentation in your daily life in order to test your assumptions and really see what the impact would be on your business. So this is my conclusion for the day. I hope you enjoyed it, and I'm really looking forward to seeing a lot more companies build experimentation into their product design, as well as into their daily operations. Thank you so much for having me, and have a wonderful day.