A Formula For An Experiment Driven Culture

Ruben de Boer at UXDX Community: Amsterdam. Video: https://youtu.be/rHMd_1DSjN0

Readable transcript: edited from the recording's captions for readability (fillers and false starts removed, punctuation and section headings added). Wording is the speaker's own. Timestamps are positions in the video. Names marked [?] could not be verified against the audio.

Introduction and Sanoma

[00:00:12] All right, thank you for the lovely introduction. I'd like to start off with a question. Who here has an experiment-driven culture at their organization? A few. Why not? Very good, there are people looking, they probably have their hands up. All right, I'd like to talk about experiments: a formula for an experiment-driven culture, how we do it at Sanoma and at NU.nl. Let's dive into it.

[00:00:44] For a short introduction: as introduced, my name is Ruben. I'm the head of CRO at Sanoma, and actually not just CRO, luckily. It's a centralized team, and I'll explain more about that later on. Besides my work at Sanoma, in my free time I'm also setting up my own company, Conversion Ideas, in which I create and sell online courses, as well as write a book on psychology and conversion optimization. Later this year I want to help charity organizations for free and blog about it, and hopefully inspire people to do so as well.

[00:01:17] For those who don't know Sanoma, most Dutch people will know these brands. Sanoma is a huge publishing company. I already see some issues with the slides, but it goes on. Dutch people know a lot of these brands. In this presentation I'll highlight NU.nl; you'll see the logo right here. NU.nl is the largest and best news website of the Netherlands. Sanoma is a publishing and education company active in most northern European countries, and Sanoma Netherlands, with its portfolio, reaches up to 80 percent of the Dutch population each month. So it's quite big. Okay, and interesting, technical problems. All right.

Getting to the first success

[00:02:17] I believe that getting to an experiment-driven culture starts with the first success, and the first success is to prove that experiments work and that experiments contribute to the future success and development of an organization. Without that first proof of success, I see a lot of companies being stuck with only a few experiments a month or even a year. But this success, what is it? How do we get it? How did it start at Sanoma, how did it start especially at NU.nl? How do we get to this first success? Unfortunately this one works. How do you get to this first success without hitting obstacles all the time? How do we get the ball rolling? What does this success actually look like?

[00:03:06] I think achieving that first success is very important, but it's also very, very hard. Recently CXL and PW [?] published their research on the state of the conversion optimization industry in 2018. If you haven't seen it, it's a very good read. Not a lot of text, a lot of diagrams, it's really fun, and you get some great insights. From this research I got two very important and, I think, actually pretty shocking conclusions.

[00:03:36] The first one is that most companies, over 68%, do not execute more than four tests per month. If you really want to get a cultural change, if you want to get people on board with experimentation, you do need a lot more proof to actually show that it works, and not even four per month is not a lot. If you consider that most of the time there's a specific team, or people across teams, or a single dedicated person working on experimentation, they should have many more experiments and much more proof for the company, for their colleagues and for your management, to show that experiments really work. So that's the first shocking conclusion.

[00:04:20] The second one is probably even worse, and I'll read it aloud. This was literally stated in the research: you probably do not want to live in the Netherlands and work for the publishing and media industry. This is actually in the research. So somewhere in my career I made a wrong turn, the wrong choice when it comes down to salary. Okay, I have to say, of course, working at Sanoma is a lot of fun, very diverse, you get loads of traffic, just the salaries. Okay, let's stick to the first one: companies not testing enough.

The formula: chance times frequency

[00:05:00] I believe that the first proof of success, and actually the whole success of your experimentation program, can be boiled down to a simple formula. This is it: success is chance of success times frequency. A little bit of imagination here. Okay. Success in this case is simply the number of good experiments, the A/B test winners. At NU.nl, the news website, we worked on the redesign and we wanted to change the first image of a section. This is the economy section. Fifty percent of website visitors saw A and fifty percent saw B, and in B we saw that more articles were being read with the bigger image. So a successful experiment.

[00:05:51] Chance is all the time you spend on your research, on your hypothesis, on your design, when you're coding or doing quality assurance, et cetera. All the time you spend on increasing the chance of success. And frequency is simply the number of experiments you run. For instance, let's say you have a 30 percent chance to find a winner in an experiment and you run 10 experiments per month: you end up with three winners a month. In the case of a twenty percent chance of success, if you run twenty experiments per month, you have four winners. This sounds pretty straightforward and pretty easy, but I believe if we really focus on this and we're really aware of this, we can become much more successful and really kickstart that experimental culture.

[00:06:35] Let me give a few examples. I gave this presentation some months ago at NU.nl as well, and I was talking about this formula. At that time at NU.nl I was responsible for experimenting and A/B testing, but all the code from A/B tests had to be checked by one guy who was responsible for quality assurance. Because what we saw was that if he doesn't check the code, one in forty or one in fifty experiments would break the website or produce errors. So this guy was making sure that the code was actually working.

[00:07:07] This increases the chance of success a tiny little bit. Maybe we go from one in 40 tests not functioning properly to one in 200. But this guy had to check all the code and all releases from the whole development department. So all the tests, all experiments, were somewhere in his backlog, and we had to wait for him to approve the code before we could launch the experiment. So the chance of success increased a little bit, the frequency plummeted, and so did the success of experimentation. After this presentation, at NU.nl we stopped with that quality assurance. Now, luckily, the code of experiments doesn't have to be checked by this single guy, and we have much more success.

[00:07:51] Another example at Sanoma. A lot of Dutch people will know it: it's a big furniture brand, vtwonen [?], and they have a magazine. The e-commerce team is merged with an inspiration team. The inspiration team does all the information about the magazine, the magazine itself, its blog, information about gardening, et cetera. From this we saw a very large chance to get all those people from the inspiration section to the shop and have them convert. We were very sure we would get a big uplift in conversions from the shop if we could test in the inspiration section.

[00:08:28] But those people who do the inspiration part did it for years, and they're very, very proud and fond of the brand, and experimentation was not part of that brand. So before you could set an experiment live, you needed lots and lots of emailing, lots of meetings, lots of calls. So very high chance, very low frequency. And in fact, and I know you guys will keep this a secret, there are still some designers and developers who hate experimentation at Sanoma.

Lowering the chance, raising the frequency

[00:09:02] Because I was working with this formula and talking to some colleagues about it, it really helped me set the focus for my agenda as well. Let's say you have a team responsible for experimentation, so you're a designer, developer, conversion specialist, et cetera, and they spend 200 hours a week on experiments, and you need 40 hours for one experiment. That means five experiments per week, 260 per year. If your chance of success is 25%, you have 65 winners, and if a winner is worth 100,000 euro, you have 6.5 million extra revenue because of your experiments. And this is really pretty good, especially when you look at the research.

[00:09:41] But if you start switching around the hours, let's say you spend less time on your experiment, so you lower the chance a little bit. You spend a little less time on your research, you do a couple of meetings less, and you code a little less beautifully. In that case you need only 25 hours per experiment, and that makes 8 experiments a week, 416 per year. Your chance drops a little bit, to 20% instead of 25, but you have more test winners, and still, if the value per winner is the same, you end up with 8.3 million extra revenue, which is an uplift of 1.8 million. Thank you.

[00:10:17] That's because of lowering the chance and increasing the frequency. Of course, it works the other way around: if you have a very hard time finding your A/B test winners, your experiment winners, start spending more time on your chance. But I believe in 99 out of 100 companies that we see doing experiments, lowering your chance and increasing your frequency works much better.

[00:10:42] So always ask yourself: does the design need to be pixel perfect? I'm not sure this is the right conference to ask this question. But for your experiment, does the design need to be pixel perfect, or does it just need to be good enough for the experiment to work? Does the code have to be beautiful for this test to work? Do we need hours of data mining, or do you have enough insights after one or two hours? And do you really need to attend a meeting, or can you set up an experiment at the same time? This all helps to get much more success and get to that first proof of success, to kickstart an experimental mindset.

A growth mindset and working around bottlenecks

[00:11:18] But I believe you can get even more out of this. We see a lot in growth hacking blogs where they speak about a growth mindset, and a growth mindset is basically: I cannot fail, I can only learn. I love to challenge myself, and when others are successful I'm not jealous, it inspires me. I really think we do need a growth mindset, and we do need to see experiments as a fun challenge, especially to kickstart that culture. Because going back to that research from CXL and PW [?], when they asked about the biggest challenges, the biggest frustrations in setting up experiments and getting to that culture, if you look at the top seven, five of those are real obstacles to getting that first success, to achieving that first proof that it actually works.

[00:12:07] If you have the right mindset, you might start being creative about bottlenecks. What I also hear a lot in different companies is, "We want to do a lot of experimentation, but there's an IT bottleneck. I have to wait for IT. I'm at the bottom of the backlog, they have so many releases, so I can't test as much as I want." What I always say is, okay, drop your frequency to zero for two weeks and start learning to code yourself. Of course it's not fun. We don't want to code; we want to experiment, do research, design. But by learning coding yourself, you are able to do many more experiments and build the proof, and build a business case to get a front-end developer on your team, and after that you don't need to do any coding any more at all, ever, hopefully.

[00:12:54] You can elaborate on this, because we hear it a lot: people say there's no budget. It depends on time, of course, and your design tools, but take the formula for success. For your chance, you can do research on the websites of the competition, heuristic evaluations, it's all free. Hotjar is a great research tool; the first month is only 30 euros. If you're completely new to CRO or optimization, you can find yourself a nice, good course for 10 to 15 euros online. That's chance. For frequency, you can optimize for free. Let's say you need to learn coding: you can find yourself a good course for 10 to 15 euros online again. So getting to the first success costs only 50 euros. So budget limitations, I mean, and you're not dependent on IT anymore, because you can code yourself. This really can help you kickstart that success and get a lot more experiments.

[00:13:48] So I think if we are aware of this, we start changing our agenda, changing our time, and really get to the first success. I'd like to quote a colleague of mine in marketing. It's a Dutch quote, "stop met lullen, start met testen" [?], for those who don't understand it. It basically says stop talking and start testing, or maybe: stop bullshitting, start testing.

How NU.nl and Sanoma work today

[00:14:17] With this mindset, that's how it started at NU.nl, and it's how we work today in the UX team. Let me explain how we work today at NU.nl. We try to test everything, or at least as much as possible. We work on a redesign based on insights from research, and this is a couple of versions of the mobile navigation, and there are a lot more variations of this. We don't only do A/B testing; we also have our own usability lab, and I can really advise every company to have its own usability lab. It really helps you get a lot of valuable insights, but also invite developers, designers, product owners and stakeholders to see what's happening. It gives them a feeling for it and gives them more of a user-centric mindset.

[00:15:05] That doesn't mean that everything goes well at Sanoma. These are actual graphs from brands of Sanoma that did a redesign. These are their conversion rates, and you see them plummet after a redesign. We can't help every brand, which is why we try our very best with NU.nl as well as the other big examples.

[00:15:21] How we're organized: like I said, the data and performance team is a central team. It's like an in-house agency, with 20 experts in these fields. We help the top brands and the top online KPIs of Sanoma, and those are quite a lot. We have weekly meetings to discuss research findings, hypotheses, tests and all the learnings, to really challenge each other and see whether what works in one brand works in another as well. We have brainstorm sessions for smaller brands. Like I said, we cannot help every brand, but we try to help them with fun brainstorm sessions. We have a checklist for rebuilds and redesigns, so those graphs hopefully never happen again. And we document our progress and learnings, so we can keep building upon our success and build on experiments.

[00:16:13] Also, someone in my team is responsible for teaching the business about this mindset. That's Bernadette [?]; you can ask her questions in the break. We do that by sending out newsletters, cases, presentations and workshops, where a lot of people show up, and it really helps to kickstart this culture. And we have our consultation hour; I actually had to look up this word, it's "inloopspreekuur" [?] in Dutch. Every Tuesday morning people are free to walk into our department and ask us questions about data and performance. In that way we can also help the smaller brands of Sanoma.

[00:16:51] At NU.nl, several people are responsible for setting up and running experiments, and they get help from the data and performance department. We test everything. We don't only test designs or research insights; we also test artificial intelligence, personalization and new functionalities, new business propositions. We try to test everything here.

[00:17:17] We have a weekly optimization meeting where the chief editors are there as well, and the chief editors are like the highest managers of NU.nl. We discuss research insights, experiment insights, new functionalities; we discuss everything. Occasionally we also invite others, like people from the editorial team, to see what we're doing, for them to understand the mindset, and of course we are interested in their ideas as well.

[00:17:47] And of course we have our newsletters within NU.nl. Whenever I set an experiment live, everyone gets an email, and whenever we get the learnings from an experiment, everyone gets an email again. And recently NU.nl started with an innovation newsletter. So we send a lot of information to people at NU.nl, and we really do encourage them to think with us, to come up with ideas, and also to learn from our learnings. And last but not least, to actually get that growth mindset, we really celebrate our failures, big, with champagne, with glitter, because every failure is learning.

[00:18:26] So that's how we're organized right now, and what we're doing right now. We still should improve a lot, but we do have the base. We do have the experimental culture, the experimental mindset. But it all started with one manager who hired one CRO specialist, who got that first success, got the ball rolling, and now we find ourselves where we are right now. Thank you.

Q&A

[00:18:48] Host: Questions? I'll just repeat this for the video as well. Curious to know: how well can you code by now?

[00:19:09] Ruben: Very good question. I actually started learning coding before I was a manager. I was a CRO specialist and the IT department was a bottleneck, so I started learning coding. I can set up around 90% of experiments myself, and I think for my team it's between 50% and 90%. If things get too tough, that's when we align with development.

[00:19:35] Host: I have another question.

[00:19:48] Ruben: Good question. I should give a good answer. We don't run a lot of tests on translations [?], because we're a Dutch company, a Dutch publishing company, having only Dutch tests. We do test copy every now and then, but it's not the biggest part of our testing. Actually, we have lots of brands, and for some brands some things work and for others they don't, and we're still trying to figure out what's kind of universal and what's not. So we do run some copy tests.

[00:20:22] Host: One final question, just for time purposes.

[00:20:31] Audience: Thank you for the nice talk. You mentioned that you send the results of every experiment by email to people to keep them involved. But how do you cope with the common problem where people get 20 emails every week, because you do a lot of experiments, so it's hard to follow all of them, and at some moment they just stop reading all this information?

[00:20:58] Ruben: Firstly, we're not at the point where we have 20 experiments a week. We do have a lot, but we haven't had that problem yet, fortunately. If there are a lot of tests done within one week, I try to put them in one email and make it as short as possible, and when there's one test done in the whole week, I elaborate a little bit on it. But I haven't had that problem yet. I hope to get it soon, because it sounds like a wonderful problem to have. Okay, thank you.