The problem of asking customers what they want

If I were to ask people five years ago, "hey do you think Uber Eats should be delivering jewellery or clothing?", most people would have probably said "no, that's not a product fit" and here we are today delivering just that. Anything at the click of a button in the convenience of your own home or a location that you know where you need that specific thing.  

If we listened to our customers we would have just been a virtual food court. This is what Uber Eats was. It was a virtual food court where you could get all your favourite brands, order your favourite food and get it delivered on time. But by not just listening to our customers, by running experiments, by testing new product features and by doing trials we've actually become an entire virtual mall.

Distilling insights

You need to be less dependent on market research and test your assumption and ideas in a real world environment. To do this the first step is to get to know your users. What is the information that you have, the behaviours, the decisions, the choices that they make in their experience with your product? For us it could be when they order, what they order, what is their basket size, how frequently do they order. Figure out the pain points and decision making points for your consumer in the journey.

The second thing is to protect the data. Keep personal information under lock and make sure that is as safe as it can be. But give your team and the rest of the business access to information in line with legal parameters. The only way to truly create a data-driven culture is by giving them access to the information that they need in order to make better decisions that are data driven.

The number of problems and the number of unanswered questions that you have about your current users needs and your future users needs and behaviours just increases every single day. In order for you to truly understand them better, to really improve your product, you really need to make experimentation a critical part of your strategy and execution. It shouldn't be something that's an afterthought or once a year or on big changes, but it should be entrenched in every single decision that you make.

Entrenching Experimentation

In order to make experimentation entrenched in your business, you need the technology to support it. Running multiple experiments weekly at the same time is very time consuming. Focus on building technology platforms that allows teams to set up experiments very quickly. Make sure the technology not only allows them to set up the experiment, but also to review the results. If you have to ask your team to review these results manually every single time, it reduces their ability to run faster and create more experiments and ask more questions.

So build the tech that allows them to do that on for every single question they have on every single product that you develop and you'll see how quickly you can learn.

The last fundamental is focus. When experimentation becomes entrenched in your company and when you have the technology that allows you to quickly execute and validate your assumptions, the world becomes your oyster. But as soon as the world becomes your oyster, the skill shifts to your ability to prioritise your assumptions. Make sure you are focused on only trying to test one thing at a time but run as many tests as possible. Focus on assumptions that could drive higher return on investments, either from a financial perspective like conversion, or from an experiential perspective. And make sure that there is one specific thing that you're focused on for that experiment and only try to validate that assumption.

Market Challenges

In the past, acquisition was king. A company's ability to get a user on the platform was the most important thing, because the friction to move between products or companies was high. Most companies invest a lot of money in getting their customers onto the platform. So the most important thing was getting them onto the platform, getting them to use it or getting them to download it. And once they're there, they're sticky. But because in today's world and the technology environment that we work in, it's so easy for most consumers to jump between products, between companies. Banks and other things that were hard to move between a few years ago are now just a click of a button.

In today's life, retention is king. And the only way in which you can really drive and improve your retention is to drive loyalty by improving their experience or diversifying your product. But you can only achieve this if you entrench and make experimentation part of your life in order to give your consumers updates and  loyalty experiences that they didn't even know they want.

Managing Experiments

The first guideline is that your objective of what you're trying to achieve and your hypothesis are two different things. Your objective needs to be linked to a business question that you're trying to answer or achieve or solve for. And your hypothesis should be linked to what you will be testing specifically.

The other thing you want to do is spend time formulating your hypothesis. A lot of the time something that might seem very straightforward ends up being very complicated and something that might seem very complicated ends up being very straightforward. But if your hypothesis is clear and you know exactly what you are testing for, it is much easier to set up your entire experiment.

The third thing is to clearly define your primary and your secondary KPIs. A lot of the times in experiments, the results could show improvements in your primary KPIs. But where people get caught up is with unintended consequences. Ensure that you consider both what the good is that you were trying to achieve as well as what might be some of the metrics that had a negative effect.

The other thing is to create a controlled environment as much as possible. This is a real world experiment. This is not a theoretical environment. So yes, exclude your treatment groups as much as possible from other elements and other engagements that could influence the results. But do not try to isolate them completely so that the results of your experiment are actually skewed and it's not really a true reflection of what the results would be if you had to make it live in a real world environment.

Also, think about the execution before you launch. The experimentations that you perform need to be linked to something that you can change, implement and impact. If you test assumptions that provide incredible results, but you don't have the ability to deliver on them, you will actually be wasting your time. When it comes to the focus, make sure you're testing something that will drive business value as well as something that you can implement or that can impact your strategy in order to get the most from it.

Lastly, test one hypothesis at a time. You should test one hypothesis using different treatments versus trying to test different hypotheses using one treatment.

The End Result

Our experiments have led to a higher uplift in sales and have increased our retention and engagement of the consumers that we targeted.  

We made many mistakes along the way, but they serve as another proof point of how important it is not just to make assumptions about what your product updates or initiatives will deliver. You need to embed experimentation in your daily life in order to test your assumptions and really see what the impact of this would be on your business.