Continuous Research to Enable Product Teams

Mike Brown at UXDX EMEA. Video: https://youtu.be/QtgEH9WmMag

Readable transcript: edited from the recording's captions for readability (fillers and false starts removed, punctuation and section headings added). Wording is the speaker's own. Timestamps are positions in the video. Names marked [?] could not be verified against the audio.

The four objections to user research

[00:00:00] Hello everyone, I'm Mike Brown, head of design at reed.co.uk. Really excited to be back at UXDX conference. I really enjoyed being here last year and it's exciting to be here on the main stage this time. Today I'm going to be talking to you about continuous research and the successes we've had in enabling our product teams. But first I just want to start by asking everyone a few questions. Can everyone raise their hand if user research or UX research is an important part of your work? I would have figured that at this conference.

[00:00:31] But now a few more questions as well. Can you raise your hands again if you've experienced any of these questions or statements in your work, if it's ever happened to you. First of all, we don't have enough researchers. Anyone ever experienced that? A few people. We don't have budget for user research. Yeah, a few more. What's the value of user research, can you define that for me? Anyone experienced that one? And then my personal favorite: we just don't have time for user research. We'd love to be doing it, but there's just not enough time.

[00:00:58] I saw a lot of hands going up. These are obviously problems and statements that myself and my teams have experienced often over the years, and this talk is going to tell you how we've answered each of those problems in the work that we do at Reed.

[00:01:12] First, a little bit about me. I've been leading design organizations for quite a while, mostly in recent history at large organizations in the UK and globally, but right now I lead at reed.co.uk. We're one of the UK's leading recruiters, and essentially that's just a recruitment platform, basically a giant job board. It's quite a complicated product and company, because on the one side we have the experience that we deliver to people searching for jobs, so there's people posting jobs, people applying to jobs. But then on the other side of the business we have the people who are actually posting the jobs that they want to be applied for. A lot of those people are professional recruiters, but then also a lot of people who might be posting a job for the first time. So we have two very different types of user groups and we're delivering two very different types of experiences at the same time. You can imagine how important it is to understand these users and to properly understand their needs.

[00:02:07] But when I started at Reed, unfortunately there was really a low UX maturity in the business. There wasn't really much research being done, or if it was, it was often using the wrong methodology, meaning that the insights weren't that useful. Product decisions were sometimes based on data but not always, meaning that a lot of bad decisions were being made. But one of the things that we did have going for us is that we had a hardworking, well-regarded design team who were keen to make changes, and I'm going to tell you how we did that.

Eurosport, the labs, and what COVID exposed

[00:02:37] But first I just want to tell a little story. Back in 2019 when I was leading a different organization, it was a talented design team at Eurosport Digital, which is part of a much larger global organization called Discovery. I think since I left they've been buying other companies, they bought Warner Brothers, they're now called Warner Discovery, a really huge organization. This was a really exciting time because Eurosport was basically a digital sports platform with live broadcasts, big news sites all across the world. It was a big global company all through Europe.

[00:03:08] We were ramping up for some big transformational projects and also ramping up for the Tokyo Olympics, which were going to happen in 2020, so it was a really exciting time. We were growing the team, adding more and more scope, more and more things we were doing. Because we were growing so fast we hadn't actually added any dedicated researchers into the team. We had a big host of UX designers, visual designers, still separate at that point, and voice designers as well for some of our experiences, but we hadn't actually put dedicated researchers into the team.

[00:03:35] But that wasn't a problem for us, because most of our qualitative research looked like this. We tended to do it in user research labs, and when we did we'd hire a lab and then we'd also hire a researcher to help us out at that time. The UX team would plan the research, then the moderators would help us refine it, and then they would actually moderate the research. This worked really well for us. It felt like quite an efficient process. We could just jump in and do research whenever we needed to.

[00:04:05] But then early in 2020 this happened. COVID hit and absolutely everything moved online. It was a really challenging time. You can see how happy everyone is in the team, you can see the joy in the team. Could have been just because I was talking. But what this meant is that not only was there no in-person collaboration, which was a really hard thing for a team that was used to working so closely together, but crucially for us there were no labs. We couldn't get into the labs to do any of our research, and that meant for us no researchers as well, because we'd relied on hiring the researchers who worked in the labs.

[00:04:36] Over time we moved to remote testing, we changed our processes, and eventually we managed to get dedicated researchers. But overall research just remained really hard. What this had shown for us, what it really made clear, was some of the problems. We never had enough researchers for everything that we wanted to do. We had a lot of products, a lot of complicated products, a lot of different areas, and through no fault of their own the researchers became bottlenecks. As hard as they worked, they couldn't possibly handle all the different things that we needed to do, which slowed everything down.

[00:05:12] And on top of that the researchers were always having to switch context. For them to do their job properly they had to be across everything that was going on in every single product, and every time they came into a new research project they had to understand not only what had happened before but everything that was being planned, so they could properly talk through the prototypes, properly understand what we were trying to do. I really felt that things could be better.

Democratizing research: the product teams run it

[00:05:33] So when I moved to reed.co.uk I decided to try and do it differently. To try and answer this first problem, we don't have enough researchers, what we did is we democratized research. I think a lot of people when they hear this term imagine different teams, they imagine different parts of the business just having a free-for-all: product managers running in doing their own research separately, different stakeholders just running in being able to do research. That's not what we mean by democratizing research. It's still a process that we follow quite closely, but the key thing for us is that the product teams, rather than handing off their research, plan and conduct all their own research.

[00:06:11] We do have dedicated researchers in the team, and what they do, rather than having to be handed off to and doing the research themselves, is they facilitate and support the research. That differs depending on the level of the team. For some of our more junior designers, what they might need is a lot more support. The researchers might actually be quite hands-on, helping them choose the right methodology, helping them plan it, maybe even jumping in and doing some of the sessions to show them how it should be done. For the more senior members of the team, for the more senior established teams, it's more of a consultancy. They check in, help with all the planning, make sure everything is going right, then let them get on with it.

[00:06:49] That's really supercharged the amount of research that we can do. But crucially for the dedicated user researchers in the team, it means they have a lot more capacity. Now, rather than racing around trying to do all the product research, they actually have the capacity to take on larger pieces of research which the business desperately needs. That might be really large pieces which stretch across multiple products, or too large in scope for one team to bring. Or something that's really been powerful for us is they do a lot of forward thinking, a lot of forward research, which powers our strategy for the next year.

[00:07:17] So there's been lots of advantages to this, but the biggest one for us is that in the first year after we implemented this we went from a dedicated researcher doing occasional research to our democratized research, and our research increased by 1300% year on year. That's testing at least once a week, usually more than that, and this is completed studies, not starting things. We've kept that cadence going for the years after that as well.

Budget: building research into process, strategy and OKRs

[00:07:42] It's all very well doing all the research, but what if you don't have the budget to do the research? This is the next thing I want to answer for you. Back to this happy team for a second. As sad as everyone looks in this photo, this was actually the high point of our COVID experience, because after this, when things got really tough for the business, things started to shrink and unfortunately one of the first budgets that was cut was our user research budget.

[00:08:07] It was quite a shock to us, because we'd been doing a lot of research and there seemed to be broad understanding and support for it within the business. But as soon as the environment got tough and cuts had to be made it was seen as a nice to have. This really impressed on me and on our team that we hadn't done the work on explaining to the business and making them understand how important this was, not just to our design work but to our product work, and how we're actually risking our decisions by not doing research. We hadn't done the work to embed this.

[00:08:35] So the first thing I did when I started at Reed, which is something I do at every organization, was to collaboratively build out our processes and our principles that the design team and the wider product team would be following. This is probably something which looks quite familiar to most people here. It's loosely based on the Double Diamond, but this is something that we've agreed across the organization: that all our design work, everything that we do, goes through three large stages, discovery, exploration and then solution. For each of those we also have the activities that we'd expect to be done, the minimum and the maximum, depending on the size, the scope, the subject, what we'd be doing.

[00:09:12] But crucially research is a really big part of that. Research is reflected in each of them, and so the business now understands, or at least the product organization understands, that when we do design work inevitably we're going to be doing some research as part of that, big or small.

[00:09:27] But that wasn't really enough, because I understood that it's not just the product organization that needs to understand, it's the wider business as well. When I started I understood from leadership they wanted to increase the amount of research we did, so we actually built that into that year's strategy and set specific OKRs company-wide for the amount of research we complete. So that was monthly targets, quarterly targets on completed studies of research. What that did is that helped us to develop a cadence. The teams became used to doing research all the time, and then expectations of user research: they became familiar with doing it, they expected that whenever we started on something, even something quite small, we would be speaking to users, we would be gathering those insights.

[00:10:07] This was really powerful, this helped. And what we found is this wasn't actually needed in the second year. In the second year we didn't need to be setting this in our company OKRs, because we built that into our routines, and we've kept that cadence in subsequent years.

Making the research visible across the business

[00:10:20] But it's all very well doing all the research. If people aren't aware of it then you haven't established that buy-in. So one thing that we do is we make sure that all of our research, everything that we do, we produce reports, and all of that's available in our research library so that it's accessible to the whole business. They can download and read those. We're always looking for ways to improve this as well.

[00:10:41] But more than that, we take every opportunity to share and present our research. Obviously as a first step, presenting the research back to everyone in the team or the affected stakeholders. But more than that, we run wide showcases in our design organization for the whole business which are well attended, where we take the time to take people through our design decisions and our research decisions. That's been quite influential in making the wider business understand what we do. And we look for every other opportunity as well, whether that's other company-wide or other places, any meeting where we can have a chance to present our research, we do. What this does is it means that the business understands the amount of research that we're doing and they understand why it's important.

[00:11:17] Putting that all together: like most businesses here, I think, things have been tough over the last six months or maybe longer. The business has slowed down, some cuts have had to be made, but I'm pleased to say that at Reed there's never even been remotely a discussion of cutting our research budget. It's fundamental to how we work now and people understand why it's important.

Proving value by showing the risk you avoided

[00:11:37] It's all very well doing all the research, understanding why it's important, having that buy-in, but how do you actually demonstrate the value of the research? How do you get that budget in the first place?

[00:11:50] There's a lot of great examples online, a lot of other companies that you can point at, other statistics. I think these things are quite easy to find. I always find that lots of phones come up when I put slides like this up. This can be quite persuasive when you're trying to establish budget for research. A lot of senior stakeholders will be quite influenced by this. We can show big established tech companies that have used UX research as part of their DNA, UX and their research, and that works really well. But at the end of the day that will only get you so far, because people want to know what's the value of user research to their particular business.

[00:12:29] Jared Spool has a relatively concise quote on this one: "Before we can identify the right UX success, problem, value and progress metrics, we must know what our key UX outcomes are." This is absolutely right, and it's what we did over time at reed.co.uk. You have to understand what the UX outcomes are, then you can set baselines and establish the right metrics to be properly measuring the value of the work that you're doing, the UX and the research that you're doing over time.

[00:12:57] But Jared goes on to say the only way to know what our UX outcomes are is to spend serious time learning about our users and their challenges. This is absolutely true. It does take time, it does take effort, it's the right thing to do, but it takes time.

[00:13:11] So what about if you're starting out? What about if you're first trying to establish and get this budget, establish the understanding of the value of user research? How do you do that at the start? What we've found, and I've had a lot of success over the years, is actually looking at it from the opposite direction, concentrating initially on the negative value that you get from user research. Because often, almost every time we test, we have failures, things that we thought might be true weren't true. These failures can be very powerful, because they can be quantified, and when they're quantified those can demonstrate how much risk you're taking by not testing.

[00:13:46] To take just one example from our time at Reed, we do a lot of work on our mobile app. We're constantly optimizing, looking for things to improve. We'd had a feature that was suggested by multiple users in reviews and also in some of our testing. They'd been asking for a feature that they'd seen in competitors and other apps. We thought it was a great idea, the product team were ready to start building it, but we just asked, can we quickly test this? We have a few questions about the value of it, can we just test this? We did one day of user testing and what we found is that no one actually used it. No one used this feature, because it wasn't actually answering their user need in the context of Reed. In terms of our product it didn't have any value. In our research we even tracked down a person who suggested the feature. They didn't use it either.

[00:14:33] What that actually showed is that one day of testing, with just one product designer working with a bit of consultancy with a researcher, prevented two weeks of waste of development, because that's what it would cost to build this feature. And that's at least ten times the labor cost, because that would have taken developers, testers, product managers, all the work that would have gone into building this feature with no user value and no business value. So these kind of savings can be quite influential, because you can show how just doing small bits of research can de-risk so much of the work and stop us going down the wrong path.

[00:15:06] This is something that's not an isolated example. We find these kind of examples almost every time that we test, and so what we do is we constantly try and reinforce that value. For every piece of research that we do we always start with research objectives, to make sure that we understand what we're trying to do. At the same time we list out the hypotheses that we have, how we think the research is going to go. And we don't just do this within the design team or within the wider product team, we get everyone involved to be submitting what their hypotheses are, whether that's wider stakeholders.

[00:15:40] Then at the end, once we've completed the research, we track how we fared, and almost every time it looks like this: some of our hypotheses were correct, we had a supposition of what was going to happen, what our users wanted, but then some of them were wrong, or some of them inconclusive, we may have to do some more research. What this does is it constantly reinforces the value of our research, the fact that we're de-risking the product development of the company every time that we research. It's also a great education for our stakeholders as well, because I'm sure everyone's worked with an opinionated stakeholder or two who believes that they know what the user wants. Once they see a few of these results they start to understand that they're not the user.

Time: planning, scheduling and recruitment

[00:16:18] So the final question I want to answer is often the most vexing with research: we don't have time for user research. We've got budget, we've understood the value, we want to do lots of research, but people just think it doesn't fit into their processes. I've had this statement plenty of times in my career. Usually it comes from people who've worked in an environment where research became a bit of a blocker. They want to move really fast. I think everyone does these days. We want to release things fast and learn from the data that way. When we actually ask, we need to understand the problem better, let's do some research first, people see that as a blocker.

[00:16:54] So for us what we wanted to do is just speed up our processes so fast that we could work within the existing processes. The way that we do that is through planning. Essentially we have a dedicated monthly research schedule where we set a rough time every month when each team is expected to do research. They can obviously move this around, people are trading these things all the time, but essentially it sets that expectation that at a certain time each month each team will be testing, and then they can look at their roadmaps and their backlogs and plan around that to make sure that they're getting value out of that testing slot every month.

[00:17:29] And recruitment is really key for us. When we actually looked at what was taking the most time, when we were trying to make our research process faster, we actually found that recruiting the participants was the most time-consuming thing. This took the most time: actually getting people, booking them in, confirming, reconfirming, rebooking when people cancel. This was one of the things that just took the most time with all of our research.

[00:17:51] So by planning that ahead. We're quite lucky at Reed in that we have a very large active website, so when we want to recruit candidates, the people who are looking for jobs, we can put surveys on our site and actually get large numbers of participants quite quickly. So that's a cheap and fast way for us to do that. On the other side of the business, on the customer side, it's much harder, because these people are all generally professional recruiters, they're quite busy, they're time contended. That takes a lot more planning. But the crucial thing for us is that with planning ahead, often as far as a month ahead, we can ensure that when we're starting our testing the candidates are ready to go, they're ready to be tested.

[00:18:32] I can imagine that a few people in the audience are saying, well, we can test quite quickly, just run out and do some guerrilla testing, that'll get it done quite quickly. We really don't believe that's the right way to test. We take a huge amount of care in the people that we test with. There was a Nielsen Norman study a few years ago which showed that only 5% of the population have higher computer-related abilities, and the scary thing is that almost all of them are centered in the cities close to where we work. So if you just went out and did guerrilla testing within a few kilometers of where we are now, you'd be testing with people who are in the top one or two percent of computer literacy, using apps, using computers. Now that does not reflect the wider population in Ireland or the world, and it certainly doesn't reflect our wider user base.

[00:19:18] So we take extra care in our recruitment and make sure that we're recruiting against the people that we know are actually our audience. We've created business-wide personas for both sides: on the one side for our candidates, these are the people who are looking for jobs, then on the other side the customers who are posting jobs. We keep these updated. Every month we examine all the insights that we've gained and then look to see if these need to be updated, so they're constantly living, breathing documents. We use these to make sure that our recruitment, that the people we're testing with, are the right ones and that we're getting that non-biased view.

[00:19:51] Putting that all together, it means that we're really well built for agile. When we need to, we can plan research, analyze and act within one sprint, or within a two-week sprint, with no degradation of quality, and fitting in within the existing processes. It's not always necessary, but this is something that we can do when asked, and that's removed a lot of the objections to doing the amount of research that we like to do.

Where Reed is now, and what's next

[00:20:16] When you actually put that all together, what does Reed look like now? We've gone from a team that did research only every now and again, and not very well, to one that's researching every week with a really wide range of methodologies, constantly getting the insights and the data that we need to power the right decisions. The business doesn't just expect to have research, they're constantly demanding it now. Often before we even get asked to do a new big piece of work or learn about something, the business is coming to us and actually asking how can we be doing research up front to actually understand what the user needs are. What this means is that all our major product and design decisions now are based on data and research, when previously they may not have been. And most importantly, we can do it as fast as we need to.

[00:21:03] So where are we going? We've obviously had a lot of success. To be honest, it's kind of worked out even better than I expected, but we're constantly wanting to improve things, and so there's a few things that we're looking to keep adding to what we're doing now. The first one is really trying to do more of qual and quant together. Historically what we've tended to do is look at the user problem, what are we trying to learn, and then choose the right methodology, whether that's qual, we do a lot of qualitative research, or different quantitative methods, whether that might be surveys, first click tests, five-second tests, all these things that might give us large amounts of data.

[00:21:41] But the challenge we're setting the teams now is, can we be doing both each time? In the same time that we do one study, can we actually do both? Because if we work hard and do them concurrently we can actually get the results all back at the same time. This is something that we're working on now. We're having early results, it's not working as well with all teams, but this is something that can obviously give us even more insights to power our solutions. But more than that, it's really good for stakeholder buy-in, because as much as we've increased the UX maturity of the business, there are still some people we work with who will only be convinced by large numbers.

[00:22:12] The final thought, the thing that we're working on, is AI. We think a lot about AI and research, and not in terms of synthetic users or any of the other suspicious things that have been popping up online, but just more about how can AI make us even faster. We're already using AI, we've started experimenting with using AI to crunch through our research analysis with really great results. We can see how much faster this is going to make all of our processes, and I think we're in the early stages of generative AI and what it can do. I fully expect that within a short period of time this will have delivered us big value in almost every part of research ops, and we should be able to do all the parts of research so much faster.

Key takeaways

[00:22:55] Finally, just to leave you with the key takeaways from my talk, for how, if you implement these, you could have a successful continuous research program as well. First of all, democratize research. When you can move not just the planning but the conducting of research to product teams, supported and facilitated by UX researchers, you can really supercharge the amount and the quality of research that you can do, while freeing up more capacity for the larger strategic research which your business needs.

[00:23:28] It's really important to gain buy-in and budget for your research. You can do that by having really clear processes that everyone follows, ensuring that the UX research and your processes are understood across the whole business, and then of course making sure that everyone knows about it, constantly communicating the successes and sometimes the failures of your research as well.

[00:23:49] And on that note as well, the best way to show value of UX research is to understand UX outcomes and then measure metrics on top of that. But you can get real value by looking at the negative value first, actually showing how regular research de-risks all your product decisions. That's something that can be very persuasive in establishing UX value and user research value. And finally, with planning comes speed. The more planning you can do ahead of your research schedule, doing quality recruitment but planning this ahead, the faster you can move, and that means that you can deliver within your agile sprints.

Q&A

[00:24:25] Host: So like when you were democratizing research, you also said that planning ahead is really vital. What is it like when you're planning ahead with members who are not in your exact team?

[00:24:33] Mike: Sure. A lot of the planning, again, is something that we tend to empower the product teams to do themselves, so that planning would be done within that product team. Probably one thing I should have said is that we have a distributed design organization, so we're not a central design team. Each of the product designers sits within that product team, so they do that planning with the product managers and the other members of the team. They have daily discussions about the backlog and the roadmap. I'd say all the stakeholders involved really know what's going on, and part of this has meant that they're all invested and understand our research schedule.

[00:25:09] Host: Perfect. And let's say you work in an organization where your stakeholders are not directly involved in the team day-to-day. What are some techniques you could use to bring them in in the planning phase?

[00:25:19] Mike: Interesting, into the planning phase. Well, for that one I'd probably just say increase communication. I wouldn't change any of the processes that we're doing. I think you'd still be aiming to set the schedules, set that upfront planning, but I think what you'd have to do is just be widening the scope of the people who are involved in those discussions, making sure that those are visible. At the moment, probably the answer to that is our research schedule is largely public, but it's not put everywhere. I'd probably be making that, it's public on our internal Confluence site, but I'd be making that much more visible and making sure that everyone who's involved is constantly aware of it.

[00:25:58] Host: So transparency is a good solution here. Yeah, nice. Okay, let's go to questions. We have: how is your research library being used, and how do you manage to keep it organized and up to date?

[00:26:08] Mike: It's a really great question. A few months out from this talk I was hoping that we'd, so we're in the process of trying to get the budget signed off to move to a much better research library. Our long-term plan is to move to Dovetail or potentially another vendor, depending on discussions here, but it hasn't happened yet. We've had some delays in getting that particular budget signed off despite extensive efforts. So at the moment we use Confluence, which is available to the whole business. We post all our research studies there as promptly as possible, so they're all there. It means that our research insights are searchable by study, people can find that. But we want to get to a future, hopefully quite soon, where it'll actually be organized by insight as well and people will actually be able to search by keyword term and find nuggets of research which will help them essentially build on the research we've already done. It may not be necessary every time to do new qualitative or quantitative research. We may be able to make product decisions based on other recent research.

[00:27:08] Host: Wonderful. I think a lot of the themes here for democratizing research is you have to look at transparency in all your steps, because if you democratize it you also need to make sure the insights are valuable and findable very quickly. Which kind of leads into a good segue here, which is: do you have a concrete example of a research OKR?

[00:27:27] Mike: Well, the concrete example that we used, to be honest it's a pretty blunt OKR but it was really effective for us, is literally the amount of research studies that each team was completing. The annual OKR was setting for each team, you must complete this many studies per month, and then the OKR was achieved if they did complete or exceed that number. Over time, our plan at first was to do that in the first year and then in subsequent years change the metric to start to talk about the quality and the value that we were deriving from that research. We found it wasn't actually necessary to track that afterwards, because we established that cadence.

[00:28:05] Host: Okay, great. By show of hands, how many of you have a benchmark or a quota or a metric for how much research must get done, and how many of you actually track how much research gets done? For people online, there's like four hands. So I think we have some work to do here. All right. Okay, how did you come up with your personas?

[00:28:30] Mike: Well, I don't think our personas look that radical. I think they're probably familiar to people who have used personas. We were really keen to have business-wide ones. It's a mixture of, when we first created them we reviewed every insight that we had from all of our research over the previous couple of years, and then did some dedicated qualitative and quantitative research on two sides, on both candidates and the customers, to specifically answer extra questions about those, and that's how we built them out.

[00:29:01] Mike: I think the elements of them have really been led by our UX researchers. I was happy to empower them to come up with the final solution. I think they've added some really nice elements. There's all the key things you'd expect on there, key needs, pain points, but what they've done is they put little mini journeys onto each of them showing sentiment throughout, which has really been quite nice for all of our wider stakeholders as well who may not understand how personas work, because they can actually see the pain that some of our different personas are having at certain parts of the key user journeys and how those contrast between them.

[00:29:36] Host: Nice. Well, keeping track of time, I think we can move all further questions, just find Mike anywhere. Let's have a round of applause. Thank you.