Democratising Research: Does it work?

AJ King, Emma Andrews, Mike Brown at UXDX EMEA. Video: https://youtu.be/1KsH7ZgZvCQ

Readable transcript: edited from the recording's captions for readability (fillers and false starts removed, punctuation and section headings added). Wording is the speaker's own. Timestamps are positions in the video. Names marked [?] could not be verified against the audio.

Introducing the debate

[00:00:00] Emma: How's everyone feeling? Good? Give me a cheer if you're hungover. Give me a cheer if you're not hungover. See, they're more sprightly, aren't they? It's just after lunch and there are people milling around. I'd love some more people to come and sit for this debate, because I think it's going to be a spicy one. So how about when I say countdown, everyone does a big clap and cheer like they're having the best time ever, and we can get some more people sat down. Does that sound good? Yeah? All right: three, two, one. Brilliant.

[00:00:35] Emma: It's the second day, it's the afternoon, and I wanted to get you all woken up. But I'm not up here to clown around. We are here to talk about democratizing research. Before we dig in, just to introduce myself: my name is Emma. I actually started my career in consumer insights, doing surveys and focus groups, and if you haven't guessed by the t-shirt, I'm now at FullStory, working in digital experience intelligence. Democratizing research and collecting user research is something I'm really passionate about, and I can't wait to hear what the two sides are going to be today.

[00:01:12] Emma: Democratizing research is a bit of a buzzword at the moment. Everyone's talking about how we can democratize our data, but it's nothing new. I'm sure everyone here has had issues trying to access user data, and maybe, even if you got access, trying to understand it. How do we use it best to our advantage? We're still trying to figure it out. I don't know the answer; that's why I'm excited that you two are going to help with that. We're all aiming to be more data driven in our decisions, but does that mean giving everyone access to all user research data, or should we have a more structured approach? What's going to be the best way? These two guys are going to be quite polar opposite. I've asked them to be quite extreme in their two views, so it's going to be fun.

[00:02:10] Emma: The two experts we have on stage today: on my left here is Mike Brown, who is head of design at reed.co.uk, one of the UK's leading recruitment platforms. You spend most of your time managing designers and researchers. You have a B2B and a B2C marketplace, so you're very busy doing that and finding the data to do that. Mike works in London and lives in Glasgow, a long commute, but he's actually from Australia, so don't mention the rugby; keep that on the down low. And Mike, you said you are a cliche in that you love to barbecue whenever the sun is out. That's true, which isn't very often in Glasgow, but he does his best. I'm doing this like it's a dating show format.

[00:03:02] Emma: Mike will be arguing for democratizing user research, and then AJ is team against. AJ is a senior UX designer at Ocado [?], focusing on robotic and automation hardware. In his spare time AJ is an amateur actor, and actually has a show of Spamalot happening next week, and is missing final rehearsals to be here today with you all. Thank you. And the last time that AJ was in Dublin, he broke a rib. So any broken ribs this year?

[00:03:40] AJ: No, not yet. Far less scathed this time round. Dublin's been kind the second time, really. Emma [?] may break my rib later, but you heard it here first if it happens.

[00:03:54] Emma: Right, so these are our two brave participants today. Let's get started with the debate. I know we've had a few of these formats so far, so if you haven't already, make sure you scan the QR code to join, and then you'll be able to vote. There'll be four questions. You two will have two minutes for each question, and after each question you all get to vote on who convinced you the most. We'll then do closing arguments, and we'll have the overall winner. Does that sound good? Mike's doing last-minute prep there. Mike's already in the zone.

[00:04:22] Mike: I'm quite nervous.

Are product teams more efficient?

[00:04:24] Emma: Mike, you're actually going first. The first question we're starting off with: are product teams more efficient with or without democratizing research?

[00:04:36] Mike: It's vastly more efficient moving to a democratized research process. We did that at reed.co.uk, and year on year we increased the amount of research that we completed, the number of studies we completed, by 1300%.

[00:04:50] Emma: That's it?

[00:04:50] Mike: No, I've got more, but that's really the headline stat, by doing it properly. I think when people hear democratized research, they just think of chaos, where everyone in the business can run around doing their own research without much quality or guardrails. But the way we do it is that the product teams own the whole process. They don't just plan the research; they conduct it themselves, with the support of dedicated researchers. The researchers are there to help and support, but crucially they're no longer bottlenecks. You don't have to wait for the availability of the researcher and hope that you have enough time. It means that the teams are in control of how they do the research and when they do the research, but with the support of the researchers to maintain that quality.

[00:05:32] Mike: What that means is that we do so much more than we did before. We went from a business that tested very occasionally, and not very well, to one that's now testing every single week. And that means that product teams can be vastly more efficient, because all the decisions they're making are now based on insights and data.

[00:05:49] Emma: All right, time to spare. Thank you, Mike. AJ, do you want to have your two minutes, starting now?

[00:05:58] AJ: Sure. Your teams would be more efficient if you don't do it. That's about it. I want to talk about something practical when we talk about this, because I love the idea. The product team: it's a good vision. They get involved, they own that process. But training them up to do that process well might be more of an issue. And then as people turn over, your process is only as good as the people who are running it.

[00:06:22] AJ: I want a quick show of hands of people in the room: who in this room is someone that might end up doing research if it was democratized in your company? Maybe team leads, maybe product people. Yeah, a smattering of hands. Now, I'm not talking here about designers. Someone over in this corner yesterday amazingly said, "I wouldn't trust a UX designer who doesn't do their own validation research." I fundamentally agree with that. That's not what we're talking about here. But put your hands back up if you're those people, those people in the crowd who might have to do this if we decided to do it. Keep those hands nice and high. There's a smattering of you. Those of you out there, assuming you work a 40-hour week: put your hand down, you people with your hands up, if you are consistently working less than your contracted hours a week.

[00:07:10] AJ: A couple of hands down, most hands still up. My point on this is that it's a capacity piece. What we end up doing is putting someone else's job on someone who is already overworked, who is already trying to do a whole bunch of stuff, and that means that they're going to be stressed or they're going to skimp on something else. In the short term you end up with overworked, stressed people. Long term, maybe we risk burnout, higher turnover and knowledge loss from our organization. None of these things are efficient for your product team. In the initial moment maybe it feels great, because you're starting to get more data coming in, but I'd argue that long term all you're doing is stressing people out and pushing them past capacity by moving to a more generalized way of doing things.

[00:07:55] Emma: Well done. Now it's time for voting. We know how you're going to vote. Who were you more convinced by? Are you more convinced to move towards democratized research, or against? Let's see. Do a bit of a drum roll. Oh, that's close. It's close, I don't know. All right, so I think, Mike, you just pipped this one. Well done.

Are product teams more effective?

[00:08:29] Emma: To make it fair, we're going to swap who goes first next time, so AJ, I'm afraid you're up again. And this time, rather than talking about efficiency, we are asking: are product teams more effective with or without democratizing research?

[00:08:43] AJ: Sure. Mike brilliantly made the point I was hoping he'd make in the last question: that if you democratize, you end up with more data at the point of decision making. So arguably, if that's true, yes, you end up being more effective. You end up going down fewer rabbit holes. My point about capacity is way off, because you spend less time wasting on your product and more time working on the right sort of things.

[00:09:08] AJ: I think the trouble with that, though, is that research is a complicated discipline. To do it right and to do it properly, you do need a level of expertise and a level of focus on it in the moment. When you go and do your research, the data you get back is never really clear. You have to dive into it; you have to go and try to understand it more. Not having the time to deep dive into what you're looking at means you're going to end up with, at best, shallow insights. At worst, you're going to end up drawing the wrong conclusions from the data you've got. And that's assuming that before all of that, you managed to phrase the questions you want to ask in the right way, which is harder than it sounds, and you actually collected the data you were looking to collect.

[00:09:54] AJ: I'd argue that you probably end up being less effective, because you collect the wrong information, you don't analyze it to the depth you need to get true insight about what's right for your product, and then we end up going down way more rabbit holes because we're focused on the wrong areas. Whereas if we just left it in a specialized field, then maybe we would have clearer insights about what to do.

[00:10:13] Emma: Great. Mike, your rebuttal.

[00:10:17] Mike: I think it's interesting, because AJ was actually making one of my points for me, which is that essentially I completely disagree with what he was saying. When you democratize research and you actually put research in the hands of the product team and give them the ownership of it, you do get a deeper level of understanding, because they're researching the products that they work on day to day. The best researchers in the world, when they work across a complex company like ours, which has multiple very different products, can't be on top of every single detail in every single product. They're constantly having to context switch when they're moving from subject to subject, testing different things. They can't possibly have the level of detail and level of understanding that the product teams who work on this stuff day to day have. So when the product teams own the entire piece of research, they're going to have a deeper understanding, and the results are going to be better. And this isn't theory for me. This has been borne out by the results that we see every week at reed.co.uk.

[00:11:17] Mike: The biggest thing, I think, when we're talking about making product teams more effective, is that democratizing research leads to more research. We've proven that. With more insights, teams are going to be much more effective. They're going to be making all their decisions based on customer insights, not just hunches. I've seen teams that don't do so much qualitative research tend to make most of their decisions based on data and analytics, which is better than no data at all. But it means that you're constantly just understanding what's happening. You can see what's happening, and then you might have hypotheses about why it's happening, but without actually understanding why, you're constantly taking educated guesses. Doing more and more research makes you more effective, and democratized research is the way to do that.

[00:12:01] Emma: Brilliant. All right, everyone ready for the next vote? Right, thumbs ready. Let's see. Oh, okay, this one looks like it's going more in your favor, Mike. I'm sorry, AJ.

[00:12:23] AJ: That's okay. We have time. We have two more questions.

[00:12:26] Emma: All right, we're going to go for around a 70/30 split here. Thank you, voters.

Does it make the business more effective?

[00:12:32] Emma: Right, Mike, you're going to be up again now. We've been talking about product teams being more efficient and more effective, but now we want to look at the business as a whole. Does it make the business as a whole more effective?

[00:12:49] Mike: It absolutely does. I think we can carry on the same point I was making about the product teams, but on a larger scale, because with more insights and data for the business to use, the business can make better decisions. But when you actually widen the scope, one of the great outcomes for us when we democratized research is that it obviously vastly increased the amount of product research that we did, but it also took day-to-day research activities away from the researchers. They spend roughly half their time facilitating and supporting all the product research. That frees up 50% of their time, which we use for larger scale strategic research. That's been immensely valuable to our business, because it means that we're now constantly gaining insights from larger pieces that are looking ahead and help power our strategy. It helps us be doing research on our future, which makes our strategic decisions much more sound.

[00:13:42] Mike: Another advantage for the business when you do democratized research is that when you're increasing the number of people who are researching, it widens the number of stakeholders who are involved in and understand the research, and that's really going to embed the importance and understanding of research across the business. There'll be many more people who believe in it. There's much more chance of the insights actually being trusted and used as well. When everything is outsourced to one team or one set of researchers, there's a chance that some of the product teams may not trust the insights. When they're actually doing it themselves, when they've been responsible for it, when they're involved at every step of the process, there's much more chance that they're going to, first of all, trust the insights and then also act on them as well. It's very effective for a whole business to democratize research.

[00:14:27] Emma: All right, very nice.

[00:14:29] AJ: Okay, that's great. I love that we're just making the same points but from the other side. For sure, when you democratize research, you end up with the product team trusting what they've done more. They're going to action those insights more. So I think for this it then harms the wider business. Let's play the practical scenario out again. As a researcher, the idea is that we drift off and do higher level, more strategic research. We drift off on that amazing adventure that as researchers we bang on about forever wanting to go and do. We come back with a whole load of insights that are at a high level, that will strategically improve our product, but maybe don't mean too much to the individual product teams on the ground. It takes a while to distill them down to them.

[00:15:10] AJ: And then when we go and have that conversation with them, they've just spent the whole of the last however long doing their own research, which they trust more because they did it, and they didn't come with you on your journey. You turn up with a whole load of stuff, and their backlog's full, because the backlog's always full. But now it's not just full of stuff they think they should be doing because it seemed like a good idea to a product manager; it's now filled with stuff that their research says they should do. And when your higher level research says maybe we should change some of these things, you end up in a situation where it's your research or my research. By democratizing it, we almost end up with too much research in that capacity.

[00:15:52] AJ: What does that do practically to the team, the culture and the dynamics? Well, in one scenario, research wins, and you're left with a whole bunch of product team members saying, "Why did we bother? I worked over capacity to do my own research, and it freed you up to come and tell me why I was wrong. So why did I bother?" And then if we as research don't win, if the product team wins, then absolutely fine, but again, the whole benefit of democratizing research is to go and do that higher strategic stuff. So I think it can help, but at the practical team level, unless you embed how you're going to feed those two feeds of research in, it becomes a bit of a nightmare for backlog management.

[00:16:29] Emma: Thank you, just in time. All right, voting time again. Who are you more convinced by? For the business as a whole, does democratizing research make the business more effective? Oh, the emotion: that it might waste people's work time. Okay, right. AJ, this one's gone very much in your favor. So I think the feeling from the group here is that it helps product teams be more effective, but the business as a whole, maybe it might not. Fantastic.

Does it risk the value placed on user research?

[00:17:07] Emma: Okay, final question. AJ, you're going to come up again. Does democratizing research risk the impact of, or value placed on, user research?

[00:17:20] AJ: Unsurprisingly, I'm going to say yes, it does risk it. The reason it does, changing from the practical tack to thinking about how we get people to do research, is that it's a very different way of learning when it is your job and your craft and what you are doing day to day, versus if it's something that's been thrust upon you as a side part of your job. If you are in a product team, hopefully you go in with the same attitude that a researcher whose full-time job it is to do research does. At the end of a project, you look at it, you introspect, you work out what went well and what didn't go so well, and you build your craft by doing that.

[00:17:59] AJ: But in some of the teams I've worked with, we're all busy, and so we democratize research just for the impact, which is to get data out the end. So you end up with a whole bunch of people, and your researchers aren't full-time trainers, that you've sheep-dipped through two, maybe three days of UX training, and then you send them out into the world to go and do everything. But they're busy, so as soon as they get their data back, they don't stress about whether the job was done well. They just care about what the data says, whether it's good data or not, and they carry on in that way.

[00:18:32] AJ: It's the effect of learning, and it's Dunning-Kruger, which basically says there's a point, when you start to learn something, where you get super confident really quickly, even though you don't have much knowledge, and then you become super not confident straight after that, and then you slowly build. I think ultimately what we potentially end up doing is training people in a really exciting and engaging way and sending them out with the best of intentions, and we end up with a bunch of people who are really, really confident, but maybe not improving their skill in this area. That leads us to an organization where we have shallow data that doesn't really tell us very much over the long term. Then senior leadership look at it and go, "User research doesn't tell us very much beyond the obvious. We could have guessed. Why are we bothering doing it at all?" And then none of us end up doing research, because we can just best guess if we're only looking at shallow research.

[00:19:22] Emma: I said I'd be strict.

[00:19:25] Mike: Look, I can understand AJ's concerns, because the way he spoke about democratizing research reflects, I think, the fears that a lot of people, a lot of teams and a lot of companies share. But that's only if you do it the wrong way, because I can promise you, from having implemented it properly with the proper processes, that when you actually democratize research properly, there is no risk at all. If you use his example, where teams get a bit of training from a researcher and then never hear from them again, that could lead to decreases in quality and decreases in engagement. But the right way to do it is to have researchers who are constantly involved; they're just not having to do it themselves.

[00:20:05] Mike: In our example at Reed, the researchers are involved in every piece of research that we do. They help to support, they help to plan the research, they help to choose the right methodologies, and they ensure the quality. Depending on the team, some teams might have more junior members, and the researchers might be more hands-on. They might even run a few of the sessions themselves to make sure that the examples are being set and the quality is high. For more senior teams, with senior designers who've been doing this for a long time, it's more consultancy. But the crucial thing is that quality is always there. If you democratize research properly, what it actually does is increase the quality of your insights, because you're doing so much more. There's so much more research in the business, so many more people in the business are being exposed to it, and actually the value of it increases.

[00:20:49] Emma: Right, the last voting round on the individual questions. Who convinced you more this time on the impact of, and value placed on, user insights overall? It's a landslide. You guys are thinking more access equals more value placed. Oh, but it's tilting a little bit. All right, okay, three to one.

[00:21:19] AJ: That's not too bad. I'm okay with that. Yeah, good.

Closing arguments

[00:21:23] Emma: Wow. We now have time for closing arguments, so you both have two minutes to finish up. Who wants to go first for closing arguments? Oh, we didn't decide this, did we? I just realized that. Technical issue, guys. You won, you choose.

[00:21:40] Mike: Well, you can go first.

[00:21:42] AJ: I had a feeling you'd say that. Again, Mike is right. I think if you do this properly, then it works really well. In the few instances where I feel like I've fully managed to democratize product teams and work with them, it's been absolutely amazing. But I still end up, because they've seen the value of it, embedded with me in the teams that they work with. So it has not in any way led to me having more time, or time back. What it has done is put me embedded in a team, where I feel research should be anyway. So it has a positive effect, but it has not had the effect that people want from democratizing research, which is to free your researchers up to go and do other things.

[00:22:24] AJ: Ultimately, I think if you do do it properly, theoretically it's an amazing idea. Practically, for the reasons I've stated, I just don't think it works very well. I think you end up with people who are overconfident, over capacity and stressed by doing something that is not what they were being paid to do beforehand. We end up with really shallow data, and we just make bad decisions as a result of it. But we're confident in those decisions, and there's lots of data coming in, so perceptually maybe it all looks good. But I would argue that under the hood, not a lot of good's going on.

[00:22:58] Mike: Right. I do feel bad that AJ's had bad experiences with the early attempts to democratize research, and it hasn't worked out for him and he hasn't had the results. But I can tell you from our experiences that we've had the opposite: it's been a huge success. I want to finish how I started, by really emphasizing the fact that in the first year that we moved to democratized research, the amount of research, the number of studies we completed, went up by 1300%. We went from a business that wasn't researching very much, didn't speak to customers very often, and wasn't making decisions based on insights and data, to one that's now testing at least once every week. The business is now powered by these insights. It means that we're making the right decisions. We understand our customers so much more.

[00:23:43] Mike: The benefits have been huge, and they're across the business as well. It hasn't just vastly increased the amount of research that we're doing; as I said before, the quality has increased as well. Because the product teams own their research, they're closer to it, they understand it more, they use it better, and they can get deeper insights, because they understand the subject much more than someone who's just coming in and out. And the other big advantage for us as a business is that it's freed up our researchers to take on these larger strategic pieces of research, which they didn't have the capacity for before. So it's given other benefits to the business. It means that we're more forward-looking. We're basically delivering much more product research than we were before, all those insights that we needed, and these larger pieces as well.

[00:24:25] Mike: It's been a huge success for our business, and I'd recommend every business do it in the right way. As I emphasized, you can't do this without researchers. I think the view that some people have of democratized research, where it's chaos, doesn't have to be that way. With proper planning, proper guidelines and proper processes, I believe every business could reap the benefits that we have.

[00:24:47] Emma: Great, thank you both so much, and you kept it clean. Nice. I wish you could have seen the emojis there. People were very impressed by the results that you've had; lots of shocked faces going up.

[00:25:03] Emma: Just to summarize, because I think that was a really interesting debate, we're going to do the final vote now to see who wins. Are you more convinced that, when it comes to product teams, it can give them more time back? Do they have more data, so that they can easily make better decisions? When it comes to testing, does it mean that they have a deeper understanding? They're doing more research, they have a deeper understanding, and then they gain more insights. Or is it the other way around, where the time it takes to train product people means they have less time to do their job? Do they get it wrong because they don't have as much expertise, so they have to do more training and take more time again? And when it comes to the business, one side argued that researchers get more time back because product teams are empowered to do their own research, whereas AJ was saying, well, not really, because you still have to be very involved in the process. I love the learning curve point: they might be overconfident and then realize they don't know as much, because they can't get as deep into it.

[00:26:06] Emma: Those were the two sides that were argued. We discussed how we'll do the winner. We want you two facing forward so you can't see the results. I'm going to face this way so I can see them.

[00:26:19] AJ: I'm not hugely confident here, I won't lie. We'll see.

[00:26:22] Emma: Right, go. I'll hold your hands. All right, we're going to see. It's all right, guys. It's still moving.

[00:26:33] AJ: I feel like if you don't win, it's a sham of democracy.

[00:26:40] Emma: All right, has everyone finished their votes? It's still changing. That comment just lost you a few points, AJ. Nice. All right, and the winner is... Mike. Well done, you can turn around. It was 60/40.

[00:27:02] AJ: So I'll take close.

[00:27:03] Emma: Yeah. All right, well done, very good. Can we get a round of applause, please, for AJ and Mike?

Q&A

[00:27:11] Emma: Now you have the chance to ask questions. If you have any questions for either of them, or for both of them, please pop them in, and we have a couple of mics that we can go around with as well. Feel free to vote for the questions that you're more interested in us answering.

[00:27:35] Emma: The first one: how to convince the CEO to bring the first researcher into the company. He gets the value but still thinks it needs to be democratic; maybe he doesn't think they need to invest in that resource. So when would be a good time to do it, with business growth, and how do you convince the CEO? Who would like to go first? Any thoughts on that?

[00:28:01] Mike: It sounds like a bit of a complex question, because it sounds like they understand the value of research but not of actually having a researcher. I think you have to really emphasize the quality of the research that you want to do, to show the value. I talked in my talk yesterday about how to show the value of research. A really quick way to do that is to show how it de-risks the company. If you show the examples, if you show the negative outcomes from research, the things that failed and that the business then chose not to do, and if you quantify that, that can very easily show the value every time you research. So in that argument you'd probably be saying: if we have a researcher coming in, the quality of our research and the quality of our insights will go up, we'll be able to do a lot more, and that'll de-risk the company. Then the more you do, the more you can show the value of the research.

[00:28:59] AJ: Yeah, I would agree. Your CEO has an interest in particular things, so talk to them on that level, around value add. Essentially what they're basically saying right now is that the insights we gain from democratized research that show we're going in slightly the wrong direction are worth less than just paying an annual salary for someone. I think if you put it in the context of those hard numbers, it's a slightly easier sell: put a 12-month contract out, see who you get in, and then you can make a decision based on the return of that. There's always an element of speaking the language of the people who are in that position. If the CEO is saying they see the value but they're not quite there, then it's about pushing and saying: this is the number you can put on it. Is it a number you're willing to take a chance on and go with?

[00:29:52] Emma: If anyone has any follow-up questions from any of these, do feel free to pop your hand up or stand up, and we'll keep an eye out and go around with a mic. No initial questions on that. The next one: what criteria or attributes do you consider when selecting individuals to conduct research, aside from the researchers themselves? That kind of fits with the earlier question as well. How do you find the right people?

[00:30:16] Mike: It's a great question. I think our long-term intention is that our teams are now conducting so much research that over time we'd expect that multiple members of the product team could be the ones running the research, because they're all so well-versed. They attend so many sessions, they watch it. In practice, almost every single time, it's the designers in the team that end up conducting the research. They're the closest to it, so I think they're the logical person to do that, because they're close to the design process. Any modern designer who has been trained in UX is going to be empathetic and understand users, and should be able to adapt to the processes and understand how to do it. But I think that over time, when you're doing a huge amount of research, really any member of the team who has that empathy and understanding of the process could step up to do the research.

[00:31:07] AJ: Honestly, I think willingness. It works really well if people are enthused, if it's something that they're interested in giving a go, no matter what level they sit at. Having that level of enthusiasm downplays my point that they then won't learn and won't become competent in it. A base level of enthusiasm, and then the support, like you talked about, Mike, as you go through it: I think that will really raise up the right people. So I'd argue there's not really a right criterion more than that.

[00:31:35] Emma: All right, thank you. Okay: how do you deal with centralization of output, so it's accessible by everyone in a company, with less duplicated research? That's something that's common once you have many teams doing their own research. Any practical tips?

[00:31:54] Mike: It's a great question. At the moment we have a research library where everything is published. That's essentially how we centralize it all. It's accessible to everyone, and they can read every single report. We publish a report after every piece of research that we do, and it's all accessible to the business. We want to move to a better system. We're looking at vendors for a proper research library, where people will be able to search by insights, not just by studies. We're not there yet, but that's where we want to get to. I think that'll make it really efficient for everyone in the business to quickly find what they're looking for.

[00:32:27] Mike: We don't really have a problem with duplicated research, because, as we said, even though the teams are all doing their separate research, there's still the oversight of the research team. The researchers are involved in every piece of research, and they can be that filter to ensure that there is no duplication. If they see two teams are looking for the same insights, then they'll make sure that we just do one study to get those insights. So we've never really had a problem with duplication.

[00:32:55] Emma: Are they the ones that check and write the reports before they go out to the rest of the company?

[00:33:00] Mike: No, the product teams own that whole process. We have set templates. We've made it as fast and as easy as possible, so we have templates for everything: moderation guides, research reports, all these kinds of things are templated, so that people can just grab them and go really quickly.

[00:33:18] Emma: All right. Anything you want to add?

[00:33:20] AJ: I have nothing to add.

[00:33:22] Emma: All right. Oh, this is one for you, Mike, and it's a bit spicy. Every research project went up by 1300%. By how much did feature and product delivery go down, if your product teams were doing so much research?

[00:33:40] Mike: Well, the thing is, at the same time that we increased research by 1300%, we also drastically increased the speed at which we could do it. We sped up all of our processes. It's not as though the capacity in the team was suddenly all being used for research, because we do it quite quickly. When we need to, we can deliver a piece of research, plan, conduct, analyze and then get the results back, all within an agile sprint. So we can move quite fast. Rather than taking time away from building features, I'd say it has probably just increased the quality of what we do. Before we were doing all this research, the teams were spending a lot of time building things that didn't work, because they didn't have the insights to make the right decisions. Now that we're doing more research, it means we're much more efficient, because we're making the right decisions.

[00:34:33] Emma: Right, that was a strong comeback to that question. Very strong comeback. Karina asked: "I liked how you were nodding when the opposing party was speaking, which suggests you might actually think the truth is in the middle. You're able to speak freely now. What would you consider?" This is a good point, and it's why I mentioned at the beginning that we tried to force them to take very opposing views. I'd love to hear from both of you. So AJ first: how do you really feel?

[00:35:01] AJ: Okay, I'm just scanning the crowd. I'm getting a bit of a nod from Rory that I'm allowed to speak freely. A bit of a nod, okay, that's fine. Yeah, for sure the truth is in the middle. As I said, I've tried some of this stuff, and some of it has worked. For me, my approach to this comes out of having too few researchers to do research at the speed you need to make good product decisions. I have 28 stakeholders that I deal with across about 21 products. That's quite a lot. If you break it down, they get about a week and a half of my time a year on average, if I was equitable.

[00:35:41] AJ: What I've done is go in, really understand the products, and work out which ones have critical user journeys, where they require a lot of my time and this is an important space for us to be doing research. Then I've got three tiers. There's a middle ground where I do consultancy work for them, like a lot of centralized teams might, where if they've got a thing they'll come to us, and otherwise they do a bit of research themselves. And then I try to democratize any of my products that sit in the lower category. I do a lot of both B2C and B2B, and some of that B2B is internal to the company. As brutal as it is, we spend more time doing B2C research, and less time with me embedded doing internal B2B research. That's where the value of democratizing comes in for me. It frees those teams up to do it. I'm a bit of a support to them, but there's an explicit understanding that I won't be with them for the whole journey. So yeah, the middle ground is probably right.

[00:36:45] Emma: Middle ground, Mike, or are you...?

[00:36:47] Mike: To be honest, not really. I think we've been really successful with it, but I can understand people's fears, because if people just wanted to say they're democratizing research but didn't have researchers, for instance, or didn't have really clear processes and guardrails to make sure the quality was high, I could see it being a bit of a disaster for some companies. So I think that if it's done properly, with the constant support of researchers, it's a really positive thing. I know that when you talk about this, some researchers are really afraid of the concept, because it sounds like it's going to put them out of work. Whereas I think the opposite is true. It just means that there's going to be more and more research being done, which means there's more need for researchers.

[00:37:34] Mike: It's a bit like the conversations we're constantly having about AI. Some people are worried that AI is going to take their job. Actually, I believe it's just going to make everyone's job so much faster. It just means we'll all be so much more effective, and there'll be even more work that we can do. I think it's a similar thing.

[00:37:49] Emma: Brilliant. To be utilized more rather than to replace.

[00:37:52] Mike: Right, exactly.

[00:37:55] Emma: For sure. Okay, we've run out of time to answer these final questions, but thank you so much, audience members. Give yourselves a round of applause for asking great questions and being involved. I'm telling you now, AJ and Mike will hang around the front here, so I know there were some additional questions; feel free to come and ask them. I promise they don't bite. Thank you. All right, thank you.