Assumptions, Bets, Experiments: How To Manage The Opportunity Tree

Amna Askari at UXDX Community: Oslo. Video: https://youtu.be/PdWXtb795tU

Readable transcript: edited from the recording's captions for readability (fillers and false starts removed, punctuation and section headings added). Wording is the speaker's own. Timestamps are positions in the video. Names marked [?] could not be verified against the audio.

Reach and the land of opportunity

[00:00:00] Hey guys, I know everyone must be pretty tired by now, so I'm going to try my best not to bore you and keep this exciting.

[00:00:20] I'm also not going to ask how many people have heard of Reach, because probably none, so I'm not going to get any hands on that one. I'll quickly tell you about Reach and a little bit about me, and then we'll get started with the talk. Reach recently renamed because they want to be reaching everyone in the UK. They're the UK's largest content and news publisher.

[00:00:44] Some of these brands you may know, you may not know. They house all of these brands. There's local brands, regional brands and national news brands. And according to Comscore data, which obviously I've taken from one of the official slides of the company, apparently we're the sixth largest site by reach, just after Microsoft.

[00:01:06] We had one million page views in January and approximately around 38 to 39 million active users, which is quite a lot for a news company. We also have subsidiary brands like In Your Area on top, and what they do is hyper local news. A lot of what we do is provide information to people, do in-house and other advertising, and try to make them have fun and keep them engaged.

A reality check on teams, outcomes and features

[00:01:34] Before getting into the details of hypothesis driven development, let's look at it from the very, very outside. All of us over here are trying to do one thing. It's us, and then it's our desired outcome, and there lies a land of opportunity in between that. For a lot of us the desired outcome is different. For some of us it's keeping people happy, for some of us it's making a lot of cash, and for some of us it's making a real difference in the world. But we've got this land of opportunity that sits between us and the desired outcome.

[00:02:06] In an ideal world, what we do is build something together, and we've got a product that bridges a gap between us and the desired outcome. Job done, we can go home. But we need a reality check, because things don't work like that.

[00:02:23] What actually happens in the product and development world is that there's a lot of things that are different. Firstly, we're not just one whole. We consist of fragmented teams. There are a lot of different people that we work with, and we need to get them to be on the same team. Secondly, the desired outcome is also not just one thing. It's a bit fuzzy, there's lots of different things going on, and everybody in all those different fragments also has a different opinion of what the desired outcome is.

[00:02:48] Quite often we don't really have happy users, lots of cash, and making a difference in the world. We have angry users, falling statistics, things that are breaking all the time. And somehow we need to amalgamate everybody to manage that and survive, and also grow at the same time.

[00:03:08] Also, looking at the middle bit, the line of opportunity, i.e. the product, which we think is a straight line, actually is not. It consists of loads of different features that are interconnected and interdependent. There's legacy features somewhere in there. Some of them die sometimes. And we want to come up with new features, and we're like, how do we do this? So it's a pretty difficult problem.

[00:03:33] Before I move on, I just wanted to get an idea of how many people work in design, development and products. Everyone that does design, raise their hand. Quite a few product and engineering. Not that many. Cool.

Well-defined goals and data-driven decisions

[00:03:50] So this is the question. How do we mitigate risk and manage complexity in order to get to where we want to, which is the desired outcome? The way we do this first is, according to the UXDX model and a lot of the things that we've all been talking about, to wrap around our teams and try to have a holistic effort towards a goal. Instead of having a fuzzy desired outcome, we have well-defined goals that come from top down. And in the middle, we make data-driven and informed decisions, and we keep contributing towards those well-defined goals and getting feedback from what we're doing.

[00:04:34] I don't know how many of you take data-driven decision-making very importantly in your organization. Some people do and some people don't. But from our experience, we found that every time we make quantitative and qualitative-backed data-driven decisions, we've always, always made progress. And the thing is that once you've done that, it's not the end of it. You need to keep getting response from your users, from the way that you're doing, and then keep going.

[00:05:00] This is what it should look like, as opposed to... I'm just going to show you the last one for comparison.

A real example: getting more inbound social traffic

[00:05:09] I'm going to give you a real-world example of how we did this at Reach, and take you through the story so it's easy to follow. Let's start with the first goal. The goal was, we want as many as possible readers to consume at least 50% of an article. That makes sense.

[00:05:29] Now we find that the readers coming in from social sources rather than search engines like Google consume more of the article, but this is only 5% of our inbound traffic. Something to consider here is that the way we knew this is because we had measurements in our article that gave events back to say how many people consume 25%, 50%, 75%. If we didn't have that, the first thing to do was to put in those markers and measure those analytics.

[00:05:57] Now we know that people coming in from social can consume more of the article. What should we do now? Now we have a well-defined goal: increase inbound social traffic to 10%. How do we do this? What can we do as a product, design and engineering team as a whole?

[00:06:15] I'm just going to show you another example of how we get to the same question with a different overarching goal. In this case it's, we want to grow our inbound traffic monthly by at least 3%. That's also reasonable. Facebook has changed their algorithm and inbound traffic from them has dropped.

[00:06:35] I don't know how many people work in the media industry, but Facebook did recently change their algorithm last year, which meant that they wanted people personally to share more and they wanted pages, official pages, to get less traffic via the things that they were posting. That put us into trouble. Things like the Daily Mirror posting things wasn't getting as much traffic as it was. So our inbound traffic from Facebook had dropped.

[00:07:01] Now we have another well-defined goal, which is increase inbound social traffic from other social sources. And we're back to the same point, which is what can we do as a product, design and engineering team. We want to increase inbound traffic from social sources, and ideally from several social sources.

Narrowing down to a first test

[00:07:19] Now we move on to the execution phase. The thing is that there's still a lot of uncertainty and quite a lot of opportunity and quite a lot of risk where we are now, but compared to where we could have been if we didn't have those well-defined goals, it's much, much less. Let's see how we can narrow this down further.

[00:07:37] Now we're like, okay, cool, we need to increase our traffic from social. What social elements do we have on the page? We've got some social follow buttons and we've got some social share buttons. Which ones should we optimize? We don't know. Also, we've got loads of different kinds of pages. We've got section pages, article pages, home pages. They all have social icons on them. Which ones should we test on? We don't know. Also, how do we go about this redesign? What is making the current social icons ineffective, and how can we make them effective? A little bit lost.

[00:08:06] But to the rescue come a few things, and I think I would really encourage you to use these in your toolkit. The first thing is go back to your well-defined goals. One of the goals that we've set is that we wanted to quantitatively increase the amount of traffic by X percent. In that case, we need to maximize whatever impact that we're having. Social follow has way less of an impact than a social share in terms of generating traffic.

[00:08:36] For example, if I share something on Facebook, five or six people would post it. Whereas if I follow a page, I have to wait for that page's post to come in my newsfeed and then I'll go and share it. It's a bit of a more long-winded process and it doesn't have immediate turnover. Cool, we'll go with social shares.

[00:08:53] What kind of page? Again, this has a quantitative reasoning towards it. Most of my traffic comes into article pages. People want to share articles because that's what sparks interest. Cool, we'll go with that. Let's start with that.

[00:09:05] How to redesign? This is why UX research and market research really comes into play. We looked at what other people were doing, and that is a really, really good thing to do. It's not copying, it's just being inspired. Often people's egos come in the way and they're like, oh, we're not going to follow other people. But honestly, everything that people do is taken from each other, and it's fine to look at what other people are doing.

[00:09:30] We saw that BuzzFeed does quite a lot of testing and they stuck their social sharing bar at the bottom of the page. We're like, this is interesting. Our social icons are just at the top of the page. Maybe we should try doing that. Maybe we should make them sticky. And then the more UX research that we did, we found that lots of people were actually sticking their social bar. So we're like, okay, this has some kind of backing.

[00:09:51] Also analytics. This could also guide which aspects of the page people are reading, and that's something that we did initially. We had scroll depth analytics to see how much people were reading with respect to where the traffic was coming from.

Brainstorming, the hypothesis and the metrics

[00:10:05] Now we've got a little bit of an idea of how to start an initial first test. This is, by the way, an actual screenshot of how we got to all of those ideas. We were lucky that we could brainstorm with journalists, because they also work for the organization and they were giving us insight about users and how they read articles. What we did is, back to the drawing of the big blue circle around the team, we got everybody together on board. That allowed us to really be divergent in the way that we were brainstorming. We used the little green dots to pick the things that people thought were the most important.

[00:10:40] Now we should actually give people access to be able to share more, and maybe put the share icons in more than one place, and maybe also stick them. The hypothesis was, the share icons are only in one place and we need to make them more available. Now the action is, let's make them more available and perhaps sticky.

[00:11:03] I'm going to get into the details of how we set up tests in a little bit and I don't want to overwhelm anyone, but essentially what you do is you say, cool, I want to tweak what I have in A, B and C way. The first way I tweak it is that I put the share buttons not only at the top, but I've also put them in the middle of the article and at the end of the article. Oh no, it's fine, I keep talking.

[00:11:26] And then the second thing that we did is that we'll stick them at the bottom like BuzzFeed, but we'll also stick them at the top, and then we'll see what happens. And then eventually we got a winner from that test, which was essentially the top sticky bar. So we're like, cool, that satisfied our hypothesis and we're going to go over sticking the share bar on top.

[00:11:43] I need to change the slide right now, maybe we should come back. But basically the next part of this is that the way that we evaluated which variant actually won was using metrics.

[00:11:58] And can I also ask you guys, how many of you guys have done A/B testing in the work that you do? Cool. Within an A/B test, for those of you who don't know what it is, it's essentially just doing what I said. It's sending out different versions of your product to your users and then measuring how they interact with those differences in a positive or negative manner.

[00:12:20] To see how they interact with them, you need to set up metrics. Whether you use third party tools or in-house tools, which I'll get into in a bit, you need to be sure of what exactly you're measuring and then how that contributes to your result. For example, if you just go with a vanity metric by poking in the dark and you say time spent on page, page views, and that's it. For this test, time spent on page: if you spend more time on the page, that does not indicate that you're engaging more with the share bar. Often, if you spend less time on the page and you click share if you find it in the middle of the article, that could be a positive thing. So you need to think back and really think about what metrics you're going to be using.

[00:13:01] We chose to go with A, the most obvious option, which is how many share buttons are people clicking on. That's the primary one, but that came back in perfect time. And we also wanted to monitor overall page views and time spent on page. Obviously, if overall page views are increasing because people are sharing more, that's a very good thing. So choosing your metrics wisely is very, very important.

The next round of hypotheses

[00:13:25] We got the top sticky one as the winner and everything was all good and we're done. But not so fast. The first step is often just the first step, because it's the first step in figuring out what I need to do to get to my eventual goal.

[00:13:43] As you can see on the left, we've got that sticky variant that basically won. After that, we wanted to test a few other hypotheses. We wanted to see how we should write the call to action next to the share bar[?]. Should we say share this article? Should we say share this? Should we say share? I know that it seems like a very, very little difference, but you would be surprised on how much that made a quantifiable difference in the results.

[00:14:08] We also decided to change the colors of the icons, keep them with the brand colors or keep them with the original colors of the social networks. And we also decided to change the number of icons that we have. I don't know if you guys remember, but one of the initial things was give more options other than Facebook. We thought that because things are moving a bit more personally, we use WhatsApp, we use Facebook Messenger, and we actually decided to put email, which is a bit of a rare option.

[00:14:35] But after we did user research, one of the tools that I said helps simplify the discussion, we found that a lot of our audience are slightly older. They're like between 45 to 60, and they're way more comfortable with sharing via email. And after we got the results from these tests, we found that the variant with email actually won, and we ended up going with that. So if you go to one of our sites, a share bar actually looks a little bit like this now.

[00:15:03] It did take, and this was actually two tests broken up, it wasn't just a singular other test. And I would go and break it down in terms of hypothesis and tasks, but I think it was a bit short on time. This is the winning variant, and we did it.

A/B tests, multivariate tests, client and server side

[00:15:18] But now I'm going to go into a little bit more detail on how all of this works. With no test running, this is your version A. This is your standard product going to everyone. And with the test running, you have a slightly tweaked version, as I said.

[00:15:36] Who here knows the difference between A/B testing and multivariate testing? Cool. PWA man[?].

[00:15:45] Basically what it is, for example, let me try to think of a very simple example. If you think that a blue button is going to be more engaging than a red button, and when you have to click pay now and you're an e-commerce company, that's an A/B test. Version A has a blue button. Version B has a red button. You know exactly what would make version B win over version A, i.e. the color.

[00:16:08] But often you don't want to just check one dimension. You want to see how a combination of differences makes an effect. That is when you go into a multivariate test. Not only do you have two different things, which is the heading and the image. Let's call the heading and the lower thing the image. Not only do you want to see which heading or which image is the best, you want to see the combination of which heading and which image is the best.

[00:16:32] A lot of third party tools like Google Optimize have this inbuilt into that platform, and I'm going to talk about it in a little bit. I would highly encourage you to just even mess around with it, and you don't have to be a developer to actually get that started. You can just pick, change two things on your page, and then it will set up the variants for you. Does that make sense? Cool.

[00:16:56] I'm going to get into a little bit of the building side of this, and I would heavily encourage all product and design people to also understand how this works, because that helps drive discussions with developers and it makes you understand what is possible and what isn't.

[00:17:12] It's going to be client or server side. For those who said you guys use A/B testing, how many of you use a third party? Okay, one person. And in-house? The rest. A lot of third party testing tools are usually client side.

[00:17:33] Which means what actually happens is that after your page is delivered to the user, it's the same page, the same version that's delivered, there's a little bit of a script that comes about because of the third party testing tools. It's like, hey, I need to make this change according to the experiment that you set up in my software. And then it blocks the rest of the site working and it installs that script, and that implements the changes.

[00:17:55] This is usually suitable for more lightweight changes. Like for example, changing colors of buttons, more visual things if you're changing some CSS. But for a more heavyweight change, i.e. if you're implementing, like we did one test where we changed how videos were served to our users, we made them from just simple videos to more of an interactive story feature. For that, you can't really have a script coming in and being like, hey, the video that's already on the page, I'm going to change that into a story feature.

[00:18:24] So you want to deliver a different version of the page to the user from the get-go. That is server-side testing. You can see that over here, the people that are building and making a difference are client side, it's on the page after it's delivered to the user. But server side is when you actually, before delivering to the user, you implement that change. So you really need to think about how heavyweight or how lightweight your change is.

[00:18:47] Also, how important is time to interactive? If you need to deliver the site to your users at lightning fast speed and you need to give them something as soon as they've asked for it, some companies have that as a priority, some don't, you cannot do client-side testing. Because if something goes wrong in the test and you lose out on time and that blocking script doesn't work, things aren't ready. And server-side testing is better.

[00:19:12] But then the problem with server-side testing is that if you have a CDN layer, I don't know how many people know about this, all it is is that it's a cache and a lot of companies have it. If you have a CDN layer, then server-side testing gets a little bit tricky, because you need to make sure that the CDN is on board with the variant. Because obviously if the original version of the product is cached, the CDN is like, hey, I'm just going to give that version, and then you won't get the difference that you're trying to implement in the first place. So these are just a few concerns that you should keep in mind when you're deciding what kind of test to do.

In-house or third party, and writing test code you can ship

[00:19:46] I know I mentioned this quite a lot before. This is whether you should do an in-house or a third party. How many of you guys have used Google Optimize? A few. I would really encourage you to check it out. There's a free version. And what I was saying earlier is that anyone can basically go to the card[?] version of your site and make a little bit of a game[?]. You don't need to know any HTML or CSS. And the UI for it is really nice in how it sets up the variants.

[00:20:14] What it has that I think is really, really cool is in-built analytics and reporting. One thing that happens when you tweak your product and send out a new thing to your users, like you change the button from red to blue, is something called the novelty effect. Your users are more likely to click on something that's new and shiny in the beginning, but that's not representative of how effective it is. It's just something that happens. So when you're doing analytics in the end and you're analyzing which variant won, you need to keep that in mind and ignore that initial peak. Third-party software like Google Optimize does that for you, so that's very nice.

[00:20:53] It also tells you when you should end your experiment, and that sounds much easier than it actually is. And it depends on how many people you're shipping the test out to. Many people say that two weeks is a good experiment time, but Google Optimize, in a lot of our tests, has gone for more than two weeks, around three to four weeks, but it tells you, hey, you should probably end your experiment now because the statistical differences are enough. But that's something also to keep in mind.

[00:21:18] Also resilience. When you're paying for something, the free version of Google Optimize is pretty good too, and Optimizely, but they've kind of taken care of that. So if you have an in-house A/B testing tool, it needs to be very well implemented into your architecture, and you need to be able to rely on it, because if, for example, you're shipping out a test to 50% of your users and something goes wrong with the test, you're in trouble.

[00:21:44] And also stakeholders. I said that many people can use the third-party software because the UI is quite simple. So if you want to get everybody on board and engage with the test, it's a very good thing to do. And often if you've built your own server-side testing framework and it's behind the scenes, people in UX and design don't really know what's going on. But you can always tell them and you're really sure.

[00:22:04] And also how close to production code. This depends on whether you have an in-house tool or whether you're using a third-party tool. In a client-side test, like I said, there's a little script that says, hey, I'm going to implement these changes on the page. You define those changes when you're doing them in Google Optimize or in your own tool or whatever. Sometimes you just want to get the change out for the test.

[00:22:33] Now the problem occurs when one of the variants wins and you need to implement it into production. If you've just done something very, very quick for the purposes of the test, you have an entire new development lifecycle to implement all of those changes into production.

[00:22:49] So to the engineers out there, and to teams in general, I would heavily encourage you to always keep in mind that this test could potentially win, and we want to cut our development time as much as possible. Write your code the way your production code is written. This doesn't mean that you write finalized or incredible code, but just in the spirit of how things are organized in your production code base, and not just a super hacky solution, because literally every time we've regretted doing that.

[00:23:18] This is a slightly more technical thing, and it's a fun thing to do. I don't know how many of you know about it, but using Chrome, you have this thing called source snippets. And this is what we use to just get the test quick started outside of Optimize, outside of internal tools or whatever. What the source snippets does is exactly what I said that the client side testing does. It basically just runs a script after your website is open.

[00:23:45] You can do this on any site you want. You can pick any site, go to your dev tools, and select a source snippet, write a custom bit of JavaScript, and just be like, I'm going to change these elements. I can change the style of them. I can move them around. I can do whatever. And just run that script and it'll do it for you. So I would really encourage developers to use that as a kind of sandbox to get started with the tests.

[00:24:07] And is that super confusing, or did that kind of make sense? It's basically just called source snippets, so anyone that uses dev tools will be able to find it in the tabs. And that's it.

Q&A

[00:24:31] Amna: Do we have any questions? Okay, we have loads. Mr. Banger[?].

[00:24:36] Audience: Do you use control groups? And we recommend to use them. Meaning, you take 5% of all your users if you have one million, and you actually don't do nothing. So you can go six months back and go to management and see, if you didn't hire me, you actually lose this amount of revenue or sites or whatever.

[00:24:55] Amna: As part of the test, you have to have one variant that is the control. It's best explained with this. This is like normal life. But then, out of all your users, you say let's test on 50% of them. Within that 50%, let's say that we have four variants. That means 10% get the original and then 10% get each of the variants. So you can compare with the control out of the box. And in Google Optimize, that's a default. You have to have your original version as a control so you can compare.

[00:25:32] Amna: And then what you can do to go back to the business and be like, hey, this made a really big difference, is that even though your stats for these experiments are just specific to that time, you can scale that. Just use the numbers and be like, I'm getting this many page views in two weeks from my new variant, this is how many pages I'm getting from my current variant, my current system, and then in six months time this is how we would project. And that usually works.

[00:26:00] Amna: There was a question here.

[00:26:01] Audience: Do you have any best practices or tools that are good for native apps?

[00:26:05] Amna: I'm not aware of what we can do within the app framework, but you can basically use server-side testing for native apps. Because native apps connect to a back end, and all the server-side testing stuff happens via your back end. So I would really encourage you to do it with that.

[00:26:25] Audience: And Google Optimize is also server side?

[00:26:28] Amna: Google Optimize is cool in the sense that the UI you can use, but it's client side. But you can also use it for server side. Once you set up the variants, each variant, you have an experiment ID for your experiment, and you have other details that you can basically implement in your server-side code. What's really awesome is that once you've done that, you can use all of those tools for reporting. It works the same way.

[00:26:56] Audience: You said initially that there was a goal for having users read the whole pages. And then you started to focus on sharing. And was that two different cases, or was that within the same pool of goals, so that you actually had a goal to get more users to read the whole article, but also to get more shares? And if you put the sharing on top, it would be more in the middle. It would be more tempting to actually share the story and forget about reading the whole article. So what were the results? Did you ditch the read the whole article and go for the higher of more sharing, or what was the result of that?

[00:28:01] Amna: That was a long-ish question, so I'll try to answer it bit by bit. You know when we usually say things like, this is my goal, or this is what I have to do, it seems very black and white and linear, when actually it's not. Goals is a very vague term. And the reason why I took those fuzzy dots in the beginning is because there are a lot of things that we want to achieve.

[00:28:26] Amna: And often, the strategy that I took in that instance was essentially working from a top level goal and breaking it down. And like you said, often, sometimes, what you get to when you break it down might jeopardize the top level goal. In our case, it didn't. We actually did have an indirect increase in the amount of people that were finishing to read the article. But in that situation, I think you actually stick to what you're working with. And then, after you get the results back, if it seriously jeopardizes the thing that you were initially trying to do, you just don't do it.

[00:29:01] Amna: The way that you monitor that is through metrics. You measure the article consumption, how many people crossed 50 and 75%. You also measure the time on page. And you also measure the number of shares clicked. And then you look at your priorities. For us, it's as many page views as possible, and also as many people reading as much of the page. Those are two different things. So we saw how the test changes actually caused those things to change. And by looking at how those things compared, we were like, this makes sense.

[00:29:36] Amna: Something interesting that happened was that overall page views were increasing. But when we looked a bit closer, and we found for the particular networks, it was actually a little bit more complicated. Twitter and Facebook, there were less page views actually coming out of it. Because, for example, if you share a post, less people are likely to click on it, if you for example share on a WhatsApp or email or Facebook Messenger group. We also did a derived metric in that instance. We calculated the density of people that actually came back to our site over the people that it was shared with.

[00:30:18] Amna: So you do have to micro-analyze every little bit and keep looking back at your initial metric. And see, am I jeopardizing what I wanted in the beginning? No, let's keep going down. But you have to keep re-referring.

[00:30:31] Audience: So you actually achieved both?

[00:30:33] Amna: Yeah. I mean, it worked out. I'm giving you a success story right now. I could also talk to you about how we massively messed up. But, next talk. Thank you very much.

[00:30:46] Host: Cool, thanks.