ReOps as the engine for Continuous Discovery

Julian Della Mattia at UXDX Community: Continuous Discovery. Video: https://youtu.be/THvBg9y8Gi4

Readable transcript: edited from the recording's captions for readability (fillers and false starts removed, punctuation and section headings added). Wording is the speaker's own. Timestamps are positions in the video. Names marked [?] could not be verified against the audio.

Introduction

[00:00:00] Good evening, good morning, good afternoon. I know we are in many time zones, probably, so thanks a lot for joining. Today we will be talking about continuous discovery, in particular in my case about ResearchOps as the engine for product discovery, or how can it become the backbone for your continuous discovery practice.

[00:00:22] A quick intro about me. If you want, you can scan the QR code and you can have all my information, contact details and everything. I'm the UX research leader at Kiwi.com and I also have an agency called the180, and I've been helping teams in different parts of the world, especially in Europe and in South America, to build the research infrastructure from scratch, and in particular coaching teams who are experimenting with continuous discovery from the research perspective.

Get out of the building: how we got here

[00:00:52] To start, I want to take you on a little bit of a journey to the past. I want to take you back in time, and since this is quite a product topic, I want to start with this quote. Maybe some of you know it, some of you remember it: get out of the building. This is a quote coined by Steve Blank, the author of The Four Steps to the Epiphany and The Startup Owner's Manual. This quote is, I think, around 20 years old.

[00:01:24] With this, what Steve was trying to explain is that back at the time companies were used to building products behind closed doors, with no customers around, in complete isolation. Steve believed that this had a lot of problems, especially when those products actually hit the market. So he was encouraging teams, especially startups, to get out of the building and talk to people, talk to customers, show them things, show them stuff, discuss ideas and so on. Basically he was advocating for user input. This was later called customer development, and many things came afterwards.

[00:02:02] Fast forward in time. Many, many things came into play. We have the boom of design thinking, the rise of UX, the growth of UX. We have many voices advocating for the user, user centricity. We have all these topics that actually bring us to where we are today. I'd say that nowadays I feel confident enough to say that now organizations are investing time and resources in talking to customers. In many cases this translated into having a dedicated researcher, or a research specialist, or actually having complete teams of researchers. I know in the last couple of months, or especially this year, we kind of took a step back in this regard because of all the layoffs and so on. I know it's a tough situation and I want to acknowledge that, but I'm convinced to say that we are in a better position nowadays regarding this than when Steve Blank wrote that quote 20 years ago.

Two phenomena: democratization and continuous discovery

[00:03:03] With this we have, as I said, the emergence of UX research as a discipline, of the UX researcher as a role. But then other things came up, especially in the last couple of years, so I'm going to mention two particular phenomena. These are the rise of research democratization, which means that people who are not research specialists are conducting research. I'm not going to open this topic up and say, okay, what do I think about research democratization, is it good, is it bad. I'm not going to go there. I'm just going to acknowledge it and say, okay, this is happening, and more and more companies nowadays are going this way, so this is getting trendy.

[00:03:43] And then we also have another phenomenon, which is the rise of continuous discovery, which is powered by Teresa Torres's framework, or book, which is great. I love it, by the way. But this type of work in terms of the product trios, which are basically product managers, product designers and engineers — of course the researcher around, and Teresa explains how to incorporate research into this, but the protagonists of this story are the trios — this means this can happen also in companies that have no researcher.

[00:04:17] So these two phenomena actually take us to the situation where researchers are not the only ones talking to customers nowadays. Is this good, is this bad? Well, it depends. Sorry for a very researchy answer, which is it depends. We researchers love to say that. But I think it goes case by case, and as I said before, I've been working with teams who were experimenting with continuous discovery, and from what I saw so far, especially in the teams that don't have a researcher around — because when you have a researcher in the team or in the company it's a little bit different and there's kind of a balance there, and it gets a little bit on track, but that's normally not the case. Usually teams who were experimenting with this are, as Teresa Torres said, the trios.

What goes wrong: the common patterns

[00:05:12] And I have to say it gets tricky. Normally when I work with clients I get on board and I try to assess how they're working so far, what they did so far, how they've been approaching the whole continuous discovery thing, and normally what I see kind of looks like this. It's a little bit of a train wreck, I have to say. It gets a little bit chaotic, especially considering that these are people who are not specialists in research. I'm not saying it always goes this way, but so far this is what I observed, and I identified common patterns and common problems these teams have faced.

[00:05:55] So the first thing I saw was sloppy recruitment, meaning the teams usually — and this is a big black hole in teams who are trying out continuous discovery — many teams just reach out to customers a little bit randomly, maybe in uneven batches. Maybe for two weeks they have a lot of people to talk to, in the upcoming two they have nobody. Then random customers with different profiles, no tracking of who are you reaching out to. So really, this is really bad. Probably I would say this is one of the most recurring problems I saw.

[00:06:33] Then, a lot of the output got lost, or usually gets lost. I saw teams who were interviewing customers or talking to customers, and a lot of things got noted down, and then they were scattered notes, and then when they went into a discussion they discussed something and they just picked a couple of things, and then the rest just got lost. So a lot of the meaty stuff, or maybe something that could be useful in the future, got completely lost on the way.

[00:07:04] Then, the wrong tools. That's also something that's quite common. They're using insufficient tools for what they want to achieve, for example for unmoderated testing or moderated testing. Or they're using, for example, tools like the opportunity solution tree the wrong way. They use it, maybe they build it on Google Sheets, or they use it on a quarterly basis or a semester basis instead of continuously, so they're just misusing the stuff.

[00:07:31] And then this is the point where normally all the researchers in the audience kind of cringe a little bit, and it's bias. With bias I mean introducing bias, especially in the interviewing part. You researchers out there, you probably heard this many times, especially when you're conducting interviews with people who are not researchers. This is the kind of questions like, so would you buy this for 10 euros? Or, so which one do you prefer? Or, is this what you like? This kind of thing, quite often. So a lot of bias there.

[00:08:07] Then I also saw no triangulation, and with no triangulation I mean they were not giving dimension to the opportunities, they were not using different types of data. So they just got something and they just went with it. In most cases they had the N-of-one[?] approach, that means one customer said something, so okay, we should go now and build this, or fix this, or change this. So they didn't give enough polishing to this opportunity. No triangulation happening there.

[00:08:40] And I also saw a lot of cherry picking. With cherry picking I mean cherry picking in different ways. So on one hand we have the cherry picking of user input, or whatever they got from the interviews. For example, many teams, when they went to the interview, some customers said something they liked or that kind of aligned with what they were expecting, and they said, okay, this is great, let's just write it down. And when they were a little bit indifferent, or they were not really aligned with what they were expecting, it's like, yeah, okay, maybe this is not the right customer, or maybe they didn't get it. So cherry picking what they got from the interviews.

[00:09:21] And cherry picking in general in terms of the process. The framework of the process is meant to work in one way, and they were like — because the idea is that it should work well, there is, of course, some space for adaptation, but you're supposed to work one way, so this is effective — but in many cases they were just picking random parts of the book or the process and they were randomly adapting it. So a lot of cherry picking there too.

[00:09:50] To sum up, I think that the situation ended up in many ways. So on one hand we had the people who kind of cherry picked, or kind of disrupted the process a little bit, or changed the guidelines a little bit, and then of course it failed and they ended up with the feeling that, okay, this is not useful. So in some cases it was research is not useful, in other cases it was, okay, continuous discovery is not useful. But it's sad, because this is an amazing framework and this is super useful and can be super powerful if you use it the right way.

[00:10:27] And in other cases, which is even worse, sometimes you think you're making progress, but if you don't see these mistakes that are little, maybe they're so subtle that you don't notice them at the right time, it can happen that you build the wrong product. If you have bias here and there, if you have this bad recruitment, in many ways you can make mistakes, and these mistakes can kind of snowball, and in the end you build something that's wrong, and that's not a good situation to be in.

ResearchOps as the backbone

[00:10:57] So the problem is not the teams, the problem is not the process itself. The thing is, teams start doing discovery without a backbone, without a proper structure behind it. In order to make the most out of a continuous discovery practice you need to have the proper backbone in place. So this is where ResearchOps can chip in and can bring the most value.

[00:11:22] Many of you probably are familiar with ResearchOps, many of you maybe aren't, so to give a quick definition: ResearchOps is the people, mechanisms and strategies that set user research in motion. This is a beautiful definition by the ResearchOps community. You can probably find others online. But if you want to put it very, very simply — maybe of course it doesn't paint the full picture, but it's pretty straightforward — I'd say that ResearchOps is infrastructure for research.

[00:11:52] There are many ways in which ResearchOps can create value, and normally I get this question speaking about ResearchOps: yeah, okay, we don't have researchers in the team, so we don't need to worry about this now. I also hear stuff like, yeah okay, we'll think about that later, maybe let's try if this works. Well, let me tell you something: you should be working on ResearchOps from the start. If you start as you start your continuous discovery practice, you can make the most out of it, you can see the impact and the productivity and the output of the continuous discovery multiply, because you're building it on a strong foundation, you're building it on the right infrastructure.

One: participant management

[00:12:34] I want to tell you five things ResearchOps can help you with, so after this talk, maybe tomorrow, you can gather your team, if you're doing continuous discovery, and you can say, okay, we need to polish these things, or we need to think of ResearchOps in a different way. So the first thing ResearchOps can help you out with is participant management.

[00:12:54] This, I'd say, is a black hole in many teams. There are many things that can go wrong, and this is where ResearchOps can create value right away. Continuous discovery works on the premise that we should be talking to customers each week, having weekly touchpoints with customers, and in order to make this happen we need to have — I don't want to say an automated process in place, but at least an established or well oiled process in place.

[00:13:23] So you need to know how to reach out to them, you need to have emails ready, you need to know how often are you reaching out to them, who did you contact, who answered, who didn't, response rate. So you need to track this properly, on one hand. Then you also want to incentivize people to join your interviews. People are not giving out their time for free, so you need to give something back. In some cases it's Amazon gift cards, or credits on your product, or money. You have to figure that out. And you need to track who did you give what, especially for finance and everything, so you need to be tidy with that.

[00:14:06] Then, I don't know the exact rules in every country, and I probably have people from different countries here, so I don't know the details, but I'm going to talk about Europe, and I know that in some states in the United States it also works like this: you cannot simply blast emails like it's 2004. You need to be compliant with the regulations, you need to be compliant with the law. We have the GDPR law, and the data policies. Privacy matters and so on, they need to be complied with. So if you send emails to them, they have to opt in to those emails, you have to give them the possibility to unsubscribe, you need to tell them where you will store the data. Be pretty careful with that, because it can get you into trouble, because you'll be breaking the law. So you need to be careful and tidy in that regard.

[00:14:56] Plus, you have NDAs. If you're showing stuff that's maybe something that you're working on in your company, you need to be careful with that too. Plus I have to say that it's not the same if you're working B2B or B2C. You need to find the right solution for your team, for your company. This also applies to incentives: it's not the same to incentivize or to give something to regular customers and to companies. So this is a place where ResearchOps can create a lot of value right away. You can set your practice straight, on track.

Two: training the team in research

[00:15:30] Then another important point is training the team in research. I know many ResearchOps professionals don't necessarily come from research, but most of us do, and this is where we can actually help the team step up. And this is where we can address the bias thing that I mentioned before, by training the team on what are the methods they can use, when to use which method, when quantitative data, quantitative methods, when qualitative, the different types, how to use those types, how to conduct interviews the proper way, or usability testing, or surveys.

[00:16:09] Training them in how to use this can really make a difference, and this can also be translated into templating. For example you can create a template or guide for your team so they can start, especially those who are as experienced in research as others — it can help them. Okay, this is how you ask a question. You can create a question bank for your team and they can pick some of them from there until they get used to it, until they get to the next level.

[00:16:33] This way you can ensure the quality of the research output. Otherwise, one of the things we face normally is that when people who do research who are not researchers themselves, who are not specialists, they may create subpar output, because it's normal, they are not specialists in that. It's like maybe researchers don't know how to design, and it's fine. You need to help them in this regard. So this is where you can add some value.

Three: a repository

[00:17:00] Then we have the repository thing. With repository I don't mean this full-blown whole library thing, built in a really cool way with taxonomy and everything, and tools like Airtable or Dovetail or Condens. These tools are amazing, of course, and if you can, and if you have the possibility, use them and build a repository there. But for the practice here, at least you need to have a place where the output of your research can live and you can revisit later on, and you can search and you can look.

[00:17:34] Because sometimes it happens that, as I said, if you conduct interviews and you don't store this output somewhere it can get lost, and you're losing — well, that of course costs money, time, effort. So maybe there's something that you found out that maybe is not useful now but it can be useful in the future. Maybe you can triangulate it with something else later on, or you can spark new ideas later on. So you need to build a place where all this can live.

[00:18:02] Building a repository doesn't have to be fancy, it doesn't have to be super complicated or complex. Just an MVP, whatever you feel fit. Just try to track that and put it there so the whole team can benefit. And if this is owned by ResearchOps, or by the person in charge of the ResearchOps, this can take a load off the team.

Four: tooling, and five: triangulation

[00:18:24] Then fourth is tooling, and as I quickly hinted before, we have tooling in two ways. You have the proper software tools, meaning you can maybe empower the team, arm the team with different tools. Let's say you want to do unmoderated testing for some of the solutions: okay, here you go, Maze. If you want to recruit users: okay, User Interviews, off you go. If you want to, I don't know, maybe have transcriptions of the interviews you're doing, and you want even AI to create clusters for you: okay, you have Looppanel, for example. So having the ownership of all these tools, or suggesting tools for the team, can really help them make the most out of it.

[00:19:13] And then we have triangulation, which is a very powerful thing. Many teams overlook this part, and this is key, because this gives dimension and gives volume to the insights you collect. So triangulation: Teresa, sorry, mentions this in her book. She says that insights should be triangulated, meaning you should be combining different types of data. You should be combining qualitative and quantitative, and maybe you can connect it with other things.

[00:19:43] So with helping with triangulation, I mean ResearchOps can be the connector or the enabler of research throughout the organization, and can help you mix different sources and different types of information or data. For example, you can connect your team with, let's say, customer support. You can connect your team with, let's say, sales. You can connect your team with other departments. This whole end-to-end connection can be led by ResearchOps. So this is a very powerful point that many teams overlook, and it's key for the quality of your insights and for actually assessing the opportunities properly.

Why, how and when to start

[00:20:30] So I went over what can ResearchOps help you with. The following question that comes along is why. Why should you invest time, effort and money in ResearchOps? And the answer is pretty simple: if you are doing continuous discovery, and if you want to make the most out of this practice, you need to have the proper infrastructure in place, otherwise your effort can fall short, can fall apart, or can definitely fail. As I said before, there are many causes, there are many ways this can fail, and ResearchOps is the base, so that it doesn't fail, or at least it doesn't fail the way you thought it would.

[00:21:14] Then the third one would be how. How should we start with this? Because in many cases, okay, what should I do, I don't have a ResearchOps specialist in the company or the team. So here you have many ways to go about this, it's a little bit like Choose Your Own Adventure. You can either hire a ResearchOps specialist, which there are plenty in the market if you look for them, or you can appoint someone in your team to do this. It could be, maybe, if you have a researcher in your team, it could be this person kind of leading these things, or maybe you can split the load between different team members or different roles. You do as you please. And in many cases, if none of them are fully knowledgeable in ResearchOps things, you can hire a coach and you can have your team trained in ResearchOps. That's basically what I've been doing. This is not self-advertisement, I mean there are plenty of us out there, but we can really help you out.

[00:22:12] And the last question that comes is when. When should I start looking at these topics, when should I start thinking of ResearchOps? And I normally answer the same thing. I usually answer back that if you're doing continuous discovery, the best time to start with your ResearchOps, or building your research infrastructure, was yesterday, and the second best time is today. So normally when they ask me when should you start, I'm asking, okay, what are you waiting for to start? Okay, thanks everybody. So now let's go for Q&A.

Q&A

[00:22:49] Host: Excellent, thank you very much, Julian. I really enjoyed it. I'm a massive fan of ResearchOps, so it was great seeing another perspective on getting started even before you have research.

[00:23:01] Julian: Thank you.

[00:23:02] Host: So, as Julian said, if you have any questions please feel free to type them in on whichever platform you're watching this on, and I'll collate them and put them to Julian. But while you're thinking of your questions, I'm going to kick off with one question that came to my mind. You kind of touched on it at the end, saying how, and maybe you hire somebody external or you ask somebody internally to do this. What jumped out at me is it seems like every company is maybe reinventing the wheel here a little bit, because every company is kind of going, how are we going to deal with GDPR, how are we going to deal with tracking our participants. The training courses are likely very similar between companies, of how are we going to train people on doing research. Is there more of a product offering, or is there some kind of templates out there that people can just download, best practices or anything, rather than trying to build it from scratch themselves?

[00:24:04] Julian: Well, in the case of everything GDPR related, or law related, I'm very wary of recommending any templates or resources. It's not that I'm just kicking the ball outside, but what I'm saying is that normally if you have to deal with regulations and law, you better check with your legal team. Even if you find templates online, even if you have resources, just check with your legal team, because they are the experts in that and they're the ones responsible for that side. So GDPR things aside, always check with your legal team. Of course, even if you know some stuff, you can then discuss and you can maybe go into details while you put this into practice, but I'd recommend tackling it together with the legal team. So that's on one hand.

[00:24:49] Julian: And then in terms of ResearchOps in general, I think there are a lot of resources out there. The ResearchOps community, which is a Slack community, and also they have a lot of articles, a lot of things. Many people are creating a lot of things online, so you have to look up a little bit. There's no centralized page where you can say, okay, I want to learn everything about ResearchOps. It's not happening, hopefully in the future. But so far there are some, and I would start there. Check the community, talk to people who are in ResearchOps, and we can point you out to other people who are doing this. But still, this is a pretty — it's not a completely new field, but it's something that is shaping up lately, it's been shaping up in the last couple of years. That's what I would say. So we're still finding a centralized way, I would say.

[00:25:37] Host: Brilliant. And we've had two kind of comment slash questions come in. I think Lola has just said it's been insightful and she's currently in the journey, and she said it's going to be very helpful for getting everybody aligned on the strategic goals. So I think that's just a thank you. But then we have a question from Ricardo, and you touched on this a little bit, but I'll ask it again: in terms of UXR team size, when would you start the ResearchOps practice?

[00:26:06] Julian: As soon as you can. That's the answer. The thing with ResearchOps is it comes — I'm not going to say an afterthought, but it comes a little bit later in the mind of many hiring managers or people who are doing the strategy of a team, and I see it the other way around.

[00:26:24] Julian: And the reasoning is this. If you have a team of, let's say, five researchers or six researchers, and you don't have clear processes in place, and you're kind of just jamming a little bit — okay, everybody, we know what we are researching, maybe we align with stakeholders and we get the roadmap right and so on — but if everybody's doing something their own way, then it will be harder to standardize that. If you start with ResearchOps from the beginning, maybe with your first or even second researcher — but again, I say I would hire a ResearchOps specialist before I even hire a researcher — so they help you build the practice, every time you onboard a new researcher it would be easier for them to understand and to actually hit the ground running.

[00:27:11] Julian: Because ResearchOps also has one part — I haven't mentioned this here because this is also related particularly to continuous discovery — but one of the things ResearchOps is responsible for is onboarding. So creating the proper onboarding experience for newcomers is also part of that. So if you have clear processes in place, you have a clear onboarding, and that can help you scale the team. So that's why I said it should be right at the beginning. So I wouldn't wait, if you had the chance.

[00:27:42] Host: Great. Well, that actually brings us to time, so thank you very much, Julian, and I hope everybody out there enjoyed it as much as I did.

[00:27:51] Julian: Thank you, thank you for having me. It was a pleasure. Thanks everybody.