Prioritize Research Like a Pro: A Framework for High-Impact Projects

Alli Carone at UXDX Community: Product Growth: Data, CX, and Research Strategies. Video: https://youtu.be/42_ReJi6QrU

Readable transcript: edited from the recording's captions for readability (fillers and false starts removed, punctuation and section headings added). Wording is the speaker's own. Timestamps are positions in the video. Names marked [?] could not be verified against the audio.

Introduction and why we need to prioritize research

[00:00:00] Alli: All right, so today I'm going to be talking about prioritizing research like a pro, which is my framework for high-impact projects. That's me. I'm Alli, an American living in Amsterdam for five years now. I traded the sunny shores of Hawaii for the very gloomy canals of Amsterdam, but I do absolutely love living here. I do UX research at Tiqets, which is a travel tech scaleup in the Netherlands. You can plan and book all sorts of bucket-list travel experiences on there, and it's a really great platform, so I really recommend you check it out. I love travel myself. I've been to 70 countries, and my goal for my entire life is 100 countries, so I'm almost there. In my free time I teach spin classes, so you can find me on the spin bike in Amsterdam if you live here. Also in my free time I do some mentoring on ADPList, so I've linked that here, and I'd love it if you connect with me on LinkedIn as well.

[00:00:56] Getting into it here: I'm going to share some ideas with you today, and I want you to think about them as another tool in your toolbox. Take what resonates with you and then tweak what doesn't. You can tweak the stuff that I'm going to talk about to fit your organization. I want you to think about the bigger picture, the context of your organization, and see how you can adapt these ideas to it. This talk is for anyone who takes part in the research process. It can be researchers, but also designers and PMs, anyone in product who wants to figure out how to introduce the research that's going to have the most impact. I want you to leave here today feeling empowered to make really great research happen at your organization. I'm going to do that by sharing five criteria that have really helped me spot opportunities for high-impact research projects. I've integrated these criteria into a framework for prioritizing research that I'm going to share at the end as well.

[00:01:51] Why are we talking about this? Why do we need to prioritize research? Why can't we just do all of the studies that we want to do? It's because we're going to feel a lot like this guy. You probably do feel like this guy from time to time, because I definitely do. I am the only researcher at my organization; I'm a one-woman research team. I noticed a really big problem a couple of years ago, where I was taking on way too many projects at once. I was really excited to do all the projects and study all the questions, but I was spread way too thin. What I noticed was that the quality of my work really went down. And because I was so excited, I also didn't do the proper checks and balances before I took on research projects, to make sure that these were high-impact projects. My insights ended up falling flat, and now they live in the graveyard of my user research repository. Rest in peace.

[00:02:43] So here I was, drowning in work quite like this guy, and sometimes that work didn't even really have a high impact. I knew I needed to make a change here. You might be in a similar boat as a solo researcher or part of a small research team, or you could be a designer that does research, or a PM that does research. As much as we do want to take on every single project because they're fun and they're interesting, we're going to wind up feeling like this guy if we don't find a way to get a little bit choosy with what we take on. To be super clear, I am not advocating for less research. I'm advocating for better research, for higher-quality research, for higher-impact research, and I'm going to talk about how we can reach that today.

What research impact is

[00:03:27] First of all, what is impact? What do I mean by research impact? I can tell you what impact is not. Impact is not insights themselves, and impact is not running X number of studies per quarter. I used to think that if I delivered a report that had 30 insights in it, then that was going to be a very impactful report, or if I did 15 studies in one quarter, then I was going to have a lot of impact at my organization. But impact doesn't quite work like that. Impact is what happens after your insights are presented. Impact is the effect that your insights have on your organization. It's how they tick certain numbers up in terms of business objectives, but also improve the user experience of your platform. So impact is the stuff that happens after your research is done, after your insights are delivered.

[00:04:14] I'm going to share a quick framework for impact from Kenish Shukla [?]. He splits it into global impact and local impact. There's global impact, which is research's effect on key business goals and outcomes. This is what we usually think of: metrics like conversion, number of transactions, number of baskets, increasing net margin, these hard metrics. But then there's also local impact, which is a little bit trickier to measure. This is research's impact on internal processes: improving efficiency, informing product vision, things like that.

[00:04:51] And I'd like to add this to it: user impact. You could easily argue it's built into global impact, because businesses can really only achieve their business goals if they have an understanding of their customers and they're solving problems for their customers. But I did want to include it in here as well, to make it super clear that impact is also about improving the user experience, with some key metrics like user satisfaction or task success rate, things like that. All three of these are how research plays a role in product impact.

[00:05:26] So impact is about wearing the researcher hat as much as we wear the business hat too. As we all know, making a great product, an impactful product, is going to start with a deep understanding of your users, and as researchers we do this super duper well. We get really excited about studies, about understanding our users, about our impact, about our insights. But it's equally important to have that business mindset as well. We need to run studies that are going to provide value for organizations and empower them and enable them to achieve their business goals as well.

[00:06:01] That's what this framework is really about. It's about embracing that researcher hat, but trying on the business hat for size, checking ourselves out in the mirror and seeing if we like what we see, seeing how our organization reacts to us wearing that business hat. Because I promise that that reaction is going to be really good. This is a great image from Carol Rossi, by the way, who needs no introduction from me, but she's a great UX research leader with a lot of information on research impact, so I'll link her at the end of my presentation. Just to recap, what we've learned so far is that not every single question that comes through our organization merits a full-blown research methodology. That's because we don't have enough resources for this, but also, from a business perspective, it's probably not the best choice.

Five criteria: scope and risk

[00:06:46] Let's get into the meat of it. How do we actually spot those opportunities for high-impact research? What are some of the guiding criteria that we can use to make sure that we're on track for impactful projects? And here they are, boom. First we've got scope. Scope is: why is this project important? What user problems is it addressing? What business goals is it addressing? It's about connecting the research to a much bigger context. When you research in this space, things that are sufficiently important to organizations, that's where real value is going to lie. So you should skip projects if they're not tied to a user need or a business need, or ideally both. This is going to be one of the very first questions you should ask yourself when you get a project request: why is this project important to users, or the business, or both?

[00:07:37] Another criterion is going to be risk: product risk. Whenever you're building a product or features or ideas or designs or concepts, there are a lot of assumptions that we're making about it. You're assuming that the product will work for your audience, you're assuming that they're going to use the product, you're assuming that they're going to buy the product. You do need to be comfortable with some level of assumptions and uncertainty, because you can't know absolutely everything about how it's going to work before releasing it out into the wild. But you can, and you should, write down all of your assumptions about the product or the feature or the design or the idea, and figure out which ones are riskier and which ones are a little bit less risky. You're going to invest in testing those riskier assumptions. Don't test the assumptions that you're pretty confident in; we're going to test the riskier assumptions.

[00:08:30] I'm going to make this a little bit more concrete by bringing in the four product risks here. Is it valuable, is it usable, is it feasible, and is it viable? These risks are probably no surprise to people who work in product. It's a tenet of product to start from a deep understanding of your users and then build something that's usable and has product value around that. But I do bet that some of you are looking at this and saying, Alli, research can't address all of these product risks. And you would be 100% correct in saying that. Assessing product risk is a much bigger strategy that goes beyond just research's role. It involves product managers and engineers and tech leaders and a host of other product and tech folks.

[00:09:14] But research really can come in handy to de-risk, especially value and usability risk. As researchers we can provide a lot of clarity around who our users are and what their problems are, so that the solutions and features and products that you're building are solving the most crucial problems. We can get really, really deep into that problem space, so that we're helping our organizations to build the right thing. The overall point here is that there are going to be assumptions with every product, and we need to be aware of what they are and prioritize testing the riskiest ones, because that's where the biggest learning opportunity is going to be. I wish I had tons of time to dive more into product risk, because it's a really important conversation, but I did link some resources for it at the end of the presentation.

Action capacity, clarity and alignment

[00:10:02] Next up is action capacity. Does your team have the capacity to act on the insights that you're going to learn in this project? Any time you take on a project, you should be really clear with the stakeholders that they need to make changes based on what we learn, and you need to get confirmation from them that these are changes that they can make in a decent time period, let's say in a couple of sprints or within the quarter. Are they going to be able to act on those insights that you learn? If they can't, then that's going to go lower on the priority totem pole, on the research totem pole.

[00:10:38] Then you've got clarity. If we already know enough about the topic, then there's really no need to do further research on it. What you want to do is first think about your research questions and your research goals, and explore other sources to answer them first. Internally, this can look like looking at previous research that was done in this organization already. It might need an update if you look back on it, but check that research first. Check with your data team first, check with your market research team, your marketing teams, check with your customer support teams, to see what we know about the topic before going full steam ahead with the research project. You can also look at these awesome external sources like Baymard Institute and Nielsen Norman Group, or if you're in a certain industry there might be some published trends about that industry. Of course, if there's not enough and there's still very little clarity, then research. But you should just do these checks first, to make sure that you're not re-researching something that you already know enough about.

[00:11:39] And finally we've got alignment. You can think of alignment as a final check before actually proceeding with the research. This is where you're going to make sure that all your research stakeholders have gotten eyes on the project, that you have buy-in where you need it, that you have all the perspectives that you would need, and you want to make sure everybody's on the same page about the learning goals. If you have one stakeholder who has a certain learning goal, and you didn't talk to this other stakeholder who's involved but might have a different learning goal, then you're missing out on 50% of the insights and 50% of the impact that you could really have there. So the key here is getting aligned with everyone before you move on with research, and like I said, it's a last check before you proceed.

[00:12:19] I know that was a lot, and I know you might be thinking, okay, I have all this research work I already need to do, and now I need to apply all these criteria for high-impact projects to it. But I want you to think about this as just a conversation starter with your team. If you're a researcher, you're probably going to own this prioritization process, but you do not have to go at it alone. I want you to think about these criteria as something you can talk about with your team, perhaps adapt to your team's needs if you do need that, and get everybody talking about high-impact research and thinking about high-impact research with you and alongside you.

Generative and evaluative research

[00:12:57] How do we apply these criteria to the different research methods? As you know, there are two main types of research. This is probably old news to everyone, so just very briefly. You have generative research, which is higher level, it's more strategic. It's about understanding who your customers are and their problems, their needs and their goals. You do this type of research when you can't answer a couple of really key questions about your users: who are they, what are their problems, and what are their current solutions to those problems? If you do this type of research really, really well, you're going to make better product decisions down the line, because you know exactly what makes your users tick.

[00:13:36] Then on the right-hand side you have evaluative research, which is testing your ideas, your designs, seeing how well or not well they're working with your users. In this type of research you're collecting data on how to improve those designs, those concepts, those ideas. Some of the classic generative and evaluative techniques are listed here, and they're also in my framework.

The prioritization flowchart

[00:13:59] Time to integrate all that we've learned so far into the prioritization framework, and here is a snapshot of it. Boom. As you can tell, it's a flowchart. I've plugged all these criteria that we just talked about into the flowchart, into bubbles, into simple yes or no questions that are going to guide you to a decision of whether or not research is wholly necessary. Then there's often a recommendation about what type of research you should do, what kind of method you can apply, based on how you've gone through the flowchart and the project considerations that you've answered in these bubbles.

[00:14:34] It starts off by recapping these high-impact criteria, and you don't have to remember these; these are all in the flowchart. Then we're going to get into it by first figuring out your research goals. Is the project you've been requested to do going to follow the generative research flow or the evaluative research flow? There's a separate flowchart for both, so you don't need to follow all of it at once. You're going to follow the flow that pertains to the type of research that you're doing.

[00:15:03] I'm not going to go through every single bubble in the flowchart with you. The link is going to be there for you to explore and put your own projects through, but I do want to give some examples of how I've applied these criteria in the flowchart. We've got scope here. In this framework I'm asking you to ask yourself if the idea that you've been proposed is addressing a key user problem or business objective. If you can answer yes, then you're moving on to the next part of the flowchart, and this is a new criterion coming in; that's why it's a different color. But if you can't tie it to a bigger business problem or user need, then the framework is a little bit ruthless, and it says, hey, we're going to deprioritize that project for now. The scope needs to be there for us to move forward.

[00:15:49] Here's another example from the flowchart, and this is the risk criterion coming in again. This one is definitely something that you want to discuss with your team, with your product leaders, with your PM, et cetera, and see what you think. This is where you have all of your assumptions about the product or idea. You write them all down, and you ask yourself if you've found some risky assumptions here. Usually there's going to be some sort of risk involved in your assumptions, because we can't know everything before we build and release. But if you don't have any risky assumptions in that case, then you say no, and the project is going to be deprioritized from there. Because you can test everything before release, but you probably shouldn't test everything if you have a decent level of confidence that your assumptions are sound and good. That's where you're going to get to this space.

[00:16:37] But if you do have risky assumptions, which is likely the case with any sort of innovative product, then the next question will be: which risk, or which risks? That's where the four product risks are going to come in, and the recommendations on how to de-risk them, especially for usability and value risk, because that's where research can really contribute. But all of them are in that framework, and some de-risking methods are in that framework as well.

[00:17:01] Going through the flowchart, you're either going to finish on bubbles that say deprioritize, meaning this might not be a high-impact project based on not passing the criteria, or, if it did pass the criteria, there are bubbles that are going to give you a choice of methods to pick from, based on all of the questions that you've answered in the flowchart. Let's say you're in the generative flow, the top one. You might end up on a bubble that says you could select from a range of qualitative methods, user interviews or diary studies, and choose which one fits your project the best.

[00:17:34] One more note on this before I move on. The flowchart addresses projects before they go live, before the feature is ever built or released. So you won't see release and monitor as a suggestion, and you won't see A/B testing as a suggestion. Of course, the only way that you can know whether something is going to be successful or not is actually putting it out in the market and seeing how it performs. But since this framework is about prioritizing research projects and choosing the right methodology for impact, it doesn't include that for now. It is my ultimate goal to make this a much bigger thing that encompasses all stages of the product development process, and have it be really useful also for product managers, who can choose what to do based on different criteria that they go through in the framework. But for now it's just the pre-release phase. So what this framework does is turn those five criteria for high-impact projects into an easy-to-follow flowchart that guides you to a prioritization decision.

Adopting the framework in your workflow

[00:18:32] I want to speak about how you can adopt this framework into your workflow, how you can make it part of your work process. Step one is having a centralized place to gather all of your research requests. You might already have this, but if you don't, it's really, really important to have this space where you can collect the information that's going to help you prioritize. You need to collect information like the learning goals, the background associated with it, so the business objectives, the user problems, the team's capacity to act on the insights, things like this, all in one space. I have an example here: I use a Jira board for this, and you can totally copy this template and use it yourself. But the idea is that you need to find something that works for you. It could be a Google Doc, it could be a Google Form, it could be an Excel spreadsheet, it could be something like this. You have to find a space where you can collect the information that's then going to help you prioritize the project.

[00:19:27] The next step here is putting the projects through the framework. Once again, I recommend this be a collaborative thing. Don't do this by yourself, especially at first. Bring this into your organization, run some sessions with the framework with your product teams, and for each request that comes in, go through the framework with the PM or the designer who's requesting it and see where it falls together.

[00:19:50] Step three is a little bit trickier part of the flow. This is where you're going to communicate your decision to stakeholders. As I said before, the framework is going to land you either at deprioritize, if it doesn't pass all the criteria, or take the research request on, because it does pass the criteria. It's best to communicate this in as simple and straightforward a way as possible, but there are probably going to be feelings involved. We get really attached to projects, and if you are rejecting someone's project, there are probably going to be some stickier feelings that you need to work through. If that happens, I really recommend going back into the framework together and finding exactly where it didn't fit, finding the criterion that it got stuck at, and then seeing if there's anything else you can do besides saying a straight-up no.

[00:20:40] Other options besides a straight-up no: rethinking the project a little bit. With that stakeholder, go back to the drawing board, reframe the project, think about how you can change it a little bit so that it fits the goals and hits the criteria of a high-impact project. That might take a little additional work from the stakeholder and from you, but if that's what it takes to make a project more high-impact, then you might decide to take that on. You also could potentially support with some lighter research methods, so writing up a super simple in-app survey or in-website survey that gets some answers to the questions that you're looking for, but doesn't have you do 15 user interviews. That could be another option as well. Or you could also empower team members to take the research on themselves. I personally work with loads of designers and loads of PMs who are really savvy with research, really good at research, and have experience talking to users themselves. In that case there might be a way that you could democratize it, if you feel that that's necessary based on the unique context of the project.

[00:21:46] Once again, I want you to think of these criteria and this framework as a conversation starter. Make it a collaborative team thing and adapt it to your team's needs. This is a really sticky topic that I've tried to package into a framework, and the way it is now has worked really, really well for my organization. But if something isn't working well for your organization, you can change it. You can delete a bubble, you can add a bubble. The framework is totally downloadable; it's a Figma Community file. So you can adapt it to your organization and use it as a starting point to prioritize, but figure out how it fits within the context of how you work and your processes.

[00:22:29] Just to finish up here: if you do end up striking this balance of prioritizing research projects, you are going to free up more time to have more impact at your organizations. You're going to have more time to execute, track and share high-impact projects. You're going to have more time to work on organizational strategies, like strategic initiatives. You're going to have more time to educate and empower your organization about your users. The quality of your work is going to improve, and this is exactly what's going to make you indispensable as a user researcher. And just finishing up here with the resources slide: I have the link to the framework, some info on product risks, and some of the research leaders that I mentioned before. Thank you so much for listening.

Q&A

[00:23:13] Host: Thank you, Alli. That was really nice, almost taking a very complex process and just saying, okay: yes, no, yes, no, yes, no. Thank you for sharing with us, and Tom has also said great presentation. We have a few minutes for Q&A, so if anybody has any questions out there, please do post them on whichever platform you're watching this on and I'll put them through to Alli. But I'm going to get started with one question, because as you were presenting, the second question you had was, are there any risky assumptions? That was the second step. Does that mean that this is more for evaluative research, because you're looking at a solution at that point? Or can this also work for, as you mentioned, the generative research side of things?

[00:24:00] Alli: In my framework, risk is only in the evaluative research. Risk is these purple bubbles. There's a lot to it, because it is a really complex topic, but it is only in the evaluative flow. Because in the generative flow and discovery research, you're not introducing any risk to your platform by getting in deep with your users and understanding your users. Where you're introducing risk is when you have a product or a feature that you're introducing that might pose some sort of risk. So there's a lot of emphasis in the evaluative flow, but it's part of the generative flow too, because you're trying to reduce risk by, like I said, getting to know your users really, really deeply.

[00:24:38] Host: Excellent. Okay, so yeah, the two different sides are generative and evaluative. I missed that, but that's great. And the other one that I was thinking about, because you mentioned it and I think it's a good point, is team capacity. You don't want your research not to be used; the worst thing is always it sitting on a shelf. But is there also a risk that if you're looking at just capacity, you might deprioritize work that could have really good insights?

[00:25:11] Alli: Yeah, absolutely. You shouldn't just look at capacity; that shouldn't be it. I think there is definitely a time and place for research that could come into play next quarter, let's say. Say a team knows that a topic is going to be really, really interesting for them half a year from now. I think the best product teams have a vision of what's coming half a year from now, and that could be something that you're researching now, so that you have those deep insights when the time comes. But I think the real question here is, it has happened to me before with some projects, where I've just been so excited to take it on and it actually isn't on their roadmap. So it's just those checks and balances of making sure it can get on the roadmap, whether you've agreed upon the timeline or not. In my opinion it should be within a quarter, but if you have this agreed-upon timeline of, okay, this is going to come in later this year, that's also okay. Just that you have that understanding of, okay, we are going to act on these insights, and we know when, and this is when.

[00:26:05] Host: Really, it's useful. There's nothing more frustrating than doing research and letting it sit and decay on the shelf. Right, we have a question from Angelie [?] about where we can get the links. If you can share them, Alli, in our chat, I'll post those out to everybody, because it'd be great if you can just share the link to your framework. That would be great, and I'll share it out with everybody who's watching live at the moment.

[00:26:31] Alli: Okay, awesome. Yeah, I'll definitely share it.

[00:26:35] Host: Great. We have got a last question coming from Amog H [?]. Just conscious of time, actually, what we'll do is I'll post it on our community Slack channel, and I'll invite you, Alli, as well, and hopefully you can jump in there and answer the question. But I'm just conscious of time, so just want to say thank you very much once again, Alli, for sharing your insights.

[00:26:59] Alli: Thank you for listening.