The Impact of Research Beyond the Report

Kendall Avery at UXDX APAC. Video: https://youtu.be/mL2HVjJbQtc

Readable transcript: edited from the recording's captions for readability (fillers and false starts removed, punctuation and section headings added). Wording is the speaker's own. Timestamps are positions in the video. Names marked [?] could not be verified against the audio.

Finding impact without metrics

[00:00:00] Hello everyone, and welcome. My name is Kendall, and I'm very excited to chat with you all today about the indirect impact of research beyond the report.

[00:00:10] I lead the user experience research team at Uber for the rider experience, and this is a topic that's really close to me because it has shaped the way that I lead research for myself and my team over the past few years at Uber. To kick this presentation off, I just want to start with a quick story.

[00:00:32] A couple of years ago I was getting ready to make my next career transition. I was looking for my next opportunity, and as I was in the process of getting all my materials together, updating my portfolio and my resume, I was working with different recruiters, and consistently I was hearing the same thing: make sure that you highlight your impact. On the surface it sounded totally fair. Of course I would want to highlight my impact. But for some reason this got me really nervous and anxious, because when I thought about the impact that I was having, I didn't have a lot of metrics or numbers that I could point to to show the impact I was actually having on my work and my team.

[00:01:09] I didn't have things like CSAT scores I could point to. I didn't have a lot of launch metrics; many of the projects I'd worked on hadn't quite made it to a meaningful launch for our users. In fact, the project I was most proud of was still in an early concept phase and hadn't even made it out to users. So I was getting really nervous and anxious: how am I supposed to highlight my impact if I don't have any numbers to show for it? At the same time I was thinking, I've been in this role for over two years. What have I been doing if I haven't been having impact? I have to have been doing something.

[00:01:45] This is the moment that I sat down and started to write down what I had been doing for the last few years, and this is where that light bulb moment happened for me. I realized that some of the activities I had been doing really were impactful, just not in the way that I had thought about it before. For example, I was convincing skeptical stakeholders about the value of research. I was introducing new research methods to the team, expanding the quality and capabilities of the insights we were actually able to capture from our users. I was even onboarding new designers and helping them to lead their own research, so we were actually able to scale our practice.

[00:02:26] When I looked at this list of things I had been doing, each of them was actually an example of impact. It just wasn't the way that I had framed it before. Ever since this realization, I've been really intentional about leaning into all of these indirect, or at times seemingly immeasurable, forms of impact in my role here at Uber.

[00:02:52] When you think about what it means to have impact, it's to have a strong effect on someone or something. It's not always something that needs to be measured with metrics and data. It can just be a matter of recognizing when and where you're actually having these effects on others, and then how you can continue to have these effects and impact in your work moving forward.

[00:03:14] Today I'm excited to chat with you about the three techniques that I've found have set myself and others on my team up for success, to ensure that we're increasing and maximizing the impact that we're able to have with our teams. I'm going to touch on building authentic relationships, the importance of bringing the team behind the scenes, and then what it means to actually go beyond the research report and make sure that your impact lives on even after your research study is completed.

Building authentic relationships

[00:03:45] To kick it off, building authentic relationships. This to me is the foundation that the rest of your research impact is going to build on top of. It's what's going to set you up as a credible resource that your teams can go to when they need research guidance or support, and it's going to help you expand the reach and impact that your work can possibly have.

[00:04:07] There are a few ways that I like to think about building these relationships. I start by identifying who your allies are. These are going to be the folks that will help elevate your research and make sure that you're able to action your insights. To start, just take a look at who you're working with directly on your projects. Maybe these are designers, product managers, engineers, data scientists. Whoever it is, take a look at who is in that core network and start to build relationships with them individually.

[00:04:38] The way that I like to think about this is: get to know them as people first and co-workers second, because this is where that authenticity comes in. If you're not really interested in building this connection, the person at the other end of the conversation is going to be able to tell. So really focus on how you can set up a strong foundation of good, authentic relationships with your team from the very beginning.

[00:05:00] Then, once you've built these relationships with your immediate team, start to expand your network outward a little bit. Who are the partners of your partners? Who are the people that your core network is engaging with most frequently? Who are their stakeholders, and what matters to them? Set up some time to just introduce yourself to these folks and establish yourself as the research counterpart to their work. Even if you're not going to be engaging with these partners on a regular basis, you'll still be setting yourself up as a familiar face when your research does in fact intersect with their work.

[00:05:34] Then lastly, identify who your influencers are. These are going to be the decision makers, the people that have a lot of influence in your team. Try to understand what matters most to them, how they have engaged with research in the past, and what's worked and what hasn't. All of this is going to be important for you to understand how you might need to frame your research insights so that they're set up to be as impactful as possible for the people who are really able to action your insights and recommendations. This is where it can be really helpful to have that group of allies built around you who can also help elevate your work and make sure it's having the maximum impact possible.

Research as a partner, not a resource

[00:06:15] Next, I like to think about reframing the role of research within the team. To me this means thinking about research as a partner and not as a resource. When I think about research as a resource, the way that usually shows up is that a researcher is brought into the process maybe after a project has already been kicked off. We're asked to swoop in, deliver some insights, and then leave and move on to the next work. But if we reframe our role to that of a partner, we're actually setting ourselves up to be a member of the team from the very beginning.

[00:06:51] There are a couple of ways you can set yourself up in this way. Number one, really embed yourself in all of the different team meetings and rituals, not just the ones that center around research. This means going beyond a research kickoff and a research report readout as your primary points of connection with your team, and making sure that you're attending stand-ups, design critiques, product reviews, and any other meetings where decisions are actually being made. These are all places where your research can actually inform decisions. It's going to set you up as a visible member of the team, and it's also going to help you gather valuable insights about what the team cares about the most, which might even be things that didn't initially come up in that research kickoff.

[00:07:36] Second, think about continuing this partnership with stakeholders even after the research is completed. If research is treated like a resource, the expectation is that we just leave and move on to the next project after the research is completed and the insights have been handed off. Instead, try to find a few ways you can stay involved with the team after the research phase is complete. Maybe you're doing this by setting up a regular one-on-one, either biweekly or monthly, or maybe you're just attending the different stand-ups that your team is involved in. However this shows up for you, the key piece here is continuing to make yourself that visible team member.

[00:08:15] Which leads me to my last point: act like you belong, because you do. This is your chance to speak up in meetings, share your perspective, and really be that active part of the team. This is a piece that I used to really struggle with, because I used to see my role as a researcher as just sharing the insights and the recommendations. Anything beyond that, I felt like I was overstepping my role as a researcher. I was just here to tell the team what I learned, and that was it.

[00:08:45] But eventually I realized I was actually embedding myself in these team meetings, I was showing up with the team, and I was starting to be empowered to contribute more to the conversation. I realized I had a seat at the table, and it was actually up to me to take advantage of that seat. So I started to share my perspective and my thoughts a little bit more often, and what I saw is that sometimes I was bringing a brand new perspective that the team hadn't even considered before. Remember that once you embed yourself as a member of the team, you belong there, you deserve to be there, so take advantage of that.

Knowing when to compromise

[00:09:23] Finally, as you're considering how you can build these authentic relationships, it's also important to remember how to compromise and when it's time to compromise. Maintaining a strong relationship is more important than digging your heels in and saying this is the right research and this is the right approach. Building trust is a long-term goal, and it's going to require some compromises along the way.

[00:09:47] One of the ways I see this becoming a bigger issue is when UXR can't support a research request, or when the request that comes in isn't necessarily the best candidate for research at this time. It ends up feeling like we aren't able to be a good partner or keep that relationship strong, because the research team is saying no for one reason or another. Whatever the reason you're in this situation, I try to find different ways that I can continue to support the team and keep them moving forward while maintaining that positive relationship with research.

[00:10:21] One of the ways I think about this is identifying some alternative solutions to keep the team moving forward. Is this a question that could be answered by desk research, where we can just see what's been done before? Could we bring in a vendor to support this project if we don't have the capacity to do it ourselves? Maybe this is a chance to take a look at what other research is happening right now and say, we can take this one question you've got and fit it into another study that's already happening. Maybe that one question on its own didn't merit a full study, but we can still answer it in conjunction with another effort that's already ongoing.

[00:10:59] Maybe your role as a researcher is to not do the research at all, because there's actually a team who's better suited to help this team get the information they need. Maybe you're pointing them to other insights functions like data science or market insights, and you're able to do this because you've built those relationships up front and you know who's in your network. Really just try to find ways you can keep that relationship with your partners strong and also keep them moving forward.

[00:11:28] As you think, okay, I've started to build these relationships, am I having impact yet? There are a couple of different things I like to look out for. Number one, you might be receiving more research requests. This is usually one of the first ways I've noticed the impact taking shape. You're starting to receive requests to conduct new research or to find past research, and overall you're just being the go-to person for your team whenever they have any type of research question. It's important to note that just because you're getting more research requests doesn't mean you necessarily have to take them all on. This is where the power of compromising comes in.

[00:12:07] Another thing you might start to notice is that you're brought into the process a little bit earlier. Research is now seen as a key part of the kickoff process, because you've started to embed yourself as a member of the team. Maybe you're doing more upfront research instead of being brought in at the end, and you're doing research that informs and defines the rest of the roadmap instead of validating a solution before it goes to launch.

[00:12:32] Finally, you start to notice that you have a larger network of stakeholders. You've built the core network and the extended network, you've found out who those influencers are, and you're building intentional relationships across the organization. This opens you up to more opportunities for your research to inform their work and to have impact across the business. Pay attention to how many people have shown interest in your work, how many people are reaching out to you about it, and where they're coming from, to realize just how large a network you've started to create.

Bringing the team behind the scenes

[00:13:06] Next, I want to talk about what it means to bring your team behind the scenes. As researchers, I think we sometimes forget just how much we know about our craft. We've been doing it for a number of years, and it starts to feel second nature to us. Unfortunately, that means sometimes we're unintentionally leaving our partners and stakeholders behind in the process, because we just know what to do next. But the more we involve our partners and stakeholders in the process, the more often we're actually aligned on the goals and outcomes, and I've found it can create a lot of empathy for the role and the work that we do as researchers as well.

[00:13:46] I like to start by getting my team on board with understanding what certain research methods are best used for and the point in time when they're going to bring the most value to a project. As the saying goes, when all you have is a hammer, everything looks like a nail. I've definitely found myself in situations where my partners and stakeholders look at research and say, well, we can run a survey, an interview or a usability study, and that's it. It's just because they don't realize how many tools we actually have at our disposal to answer all the different questions they have. So it's not unlikely that your team might be coming to you with the method in mind first.

[00:14:24] To combat this, I like to start by explaining the purpose and trade-offs of different methods until I actually reach that point of understanding with the team. It's not enough to say this is the method we're going to use and just move on with it. I have to make sure that I'm bringing my team on board and keeping those relationships strong. I focus on describing the method I'm proposing for the team, how we're actually planning to conduct the research, why I chose this method over others, the other options that were available to me, and what the team can expect to learn from these insights. I've found that without all this alignment up front on all the different pieces, the research runs the risk of my team disagreeing with it or not actioning it, because it's not what they expected to learn and they're not sure what to do with the results.

[00:15:18] This leads me to my next point: align with your stakeholders to understand how they're planning to action the research. This is where it's important to figure out whether this research is a must-do effort or whether it falls into the nice-to-have category. In my experience, I've seen this happen a lot with surveys. Sometimes a team wants to run a survey to learn how users are responding to a new product or feature. The way I like to reframe this is to flip the question back to them. I ask: let's say, and this is unlikely, every single respondent hated it. Would your team actually do anything with that data? And if so, do you know what you would do with it?

[00:16:04] If the answer is no, maybe this isn't the right time to do the research, or maybe this isn't the right research method, and we need to go back to the drawing board and make sure we're asking questions of our users that get us useful data, that we know what we're going to action, and what we're going to do to keep the business moving forward. Sometimes one of the best things a researcher can do is determine that there's not actually a need to do research at all, and point to other ways we can get the insights we're looking for. So it's really important to make sure your stakeholders are aligned on the expectations of the methods, where they fit best in the process, and what they can expect from the research insights after the study is completed.

Showing the value of research

[00:16:46] Sometimes, though, despite your best efforts to get everyone on board up front, it's still challenging to bring stakeholders along, especially if you're introducing a new concept or method. This is where I find the power of showing the method, or the value of the research, really comes into play. A specific example: a couple of years ago I was working with a new product partner, and we were reviewing the research questions they had. My proposal was to run an unmoderated study. We would launch the study, ask users to accomplish a task and answer a few questions, we'd get video clips back within probably hours, and the team would be able to move forward pretty quickly.

[00:17:29] It seemed to work with the research questions they had, and it definitely worked with my timeline as a researcher and my ability to support. But the team was skeptical. They had never done an unmoderated study before, and they were really nervous about the fact that there wasn't going to be a researcher there guiding the participant if they had any questions, helping them out, or digging deeper to get the insights.

[00:17:50] This is where it's important again to remember those compromises. I decided to suggest: let's go ahead and run the unmoderated study, let's just find out and learn. If we find that we're not getting the results we're looking for, or the insights that are going to help us make the next decision, I will set up some time to follow up with users and do a set of interviews, which was a method that was familiar to the team. So we launched the unmoderated study, and, probably to the surprise of all of us, it was so successful that the product manager came back a week later asking when we could run another one. This is just an example of how powerful it can be to show the value of the research to your team.

[00:18:35] There are a couple of tips I consider when leveraging this technique. First, consider introducing a new research method in conjunction with a method or approach that the team is already familiar with. For example, maybe you're introducing a card sort, which is a new method to them, and you'll do it in addition to an interview. This way you can temper any concerns they have, because you're introducing something they're familiar with in addition to something that's a little bit newer.

[00:19:04] Secondly, invite your team to attend the sessions live. This is one of the best ways to make sure they're getting access to the insights in real time, and I think it's an especially great tool when you have skeptical stakeholders who question the legitimacy of the research: can we really trust this, we only learned it from six people, we only talked to six people, whatever it may be. This is a way you can allow your team to see the patterns start to form for themselves, and understand what we're actually doing as researchers and why these insights and findings are so trustworthy and credible to the business.

[00:19:42] Finally, as you're developing your research report, prioritize the video clips and direct quotes in your research. This way you can make sure you're addressing your team's research questions or concerns with those clips or quotes, which again strengthens the validity of the insights you're capturing. I've also found it's a great way to separate yourself as the researcher from a controversial finding. You're maintaining that strong relationship, and you're putting it front and center and saying, look, this isn't me saying this, this is the users, and this is why we need to pay attention to and action this finding. So you're showing the value of research by putting your users front and center, showing how these methods can be introduced and the value they bring, and all the different things your team is going to learn that they maybe didn't have access to before.

The power of language

[00:20:36] Then finally, it's important to recognize the power of language. Research terminology is tricky and at times just confusing. There's a ton of jargon in the craft, and I've found that if we haven't set the right expectations up front, we're increasing the likelihood of miscommunication down the road. One of the first ways I like to address this is to align on what we mean when we're talking about different terms. For example, does your team believe that user testing is the same thing as usability testing? What about concept testing versus usability testing? This came up very recently: I was running a study, and we had a very long discussion on the difference between findability and discoverability.

[00:21:20] I was also recently talking with a colleague, and halfway through the conversation we realized we had both been talking about user testing, but she was talking about running a live A/B experiment and I was talking about running prototype usability testing with our users. It happens all the time, but the most important thing is to make sure you are aligned up front on the terminology you're using and the expectations of it, making sure everyone's on the same page, because you're going to move a lot more quickly and efficiently through your work.

[00:21:51] Another thing to consider when you think about language is how it impacts the expectations your team has of your work. One word I am particularly mindful of is validate, or validation. In my experience, I've often found that when teams are seeking validation, they're looking to confirm their assumptions, confirm that they're right about something. While it's completely valid for us to run research that evaluates or tests whether something is actually working as well as we hope it will, what I like to do is slightly reframe the language to evaluate or investigate.

[00:22:30] It's a subtle change, but it really opens the team up to understanding that there might be a change we actually have to make. This isn't just going to tell us that something works and there are no changes to be made. If we're investigating or evaluating, we're open to the possibility that we might need to do something on our end to deliver a better product experience. And if the team is not comfortable with that, if this isn't going to work with their timeline, if we aren't in a position where we're actually going to be able to make changes, then maybe research isn't the best tool for us at this point in the process. So make sure you're using language in a way that communicates not only what you're doing but what the expectations of the team are as well.

[00:23:15] There are a couple of things to look out for to identify where and when you're having your impact. First, you're going to start to see that your team comes to you with questions instead of methods. This is one of my favorite things when you start to see it happening. You're no longer getting requests to run a survey or launch a usability study, whatever it might be. Your team's not coming with method-first recommendations or suggestions; they're coming to you with questions. This means they really trust you as the researcher to determine the right method and the right time to introduce it.

[00:23:47] You're also going to find that you're working on more of the high-priority efforts. This is likely because your team understands the value of UXR. They know you have a lot of different ways to answer their questions, you're an embedded part of the team, you're a partner instead of a resource. And you know when it's the right time to say no, this isn't a method or a research study that's going to get us the most valuable information at this point in time. So you're refining the type of work you're doing, and you're only working on those high-priority efforts that are going to bring a lot of value to your team.

[00:24:20] And finally, you'll start to notice that your team is speaking your language. This is another fun one to look out for. Your team is aligned up front on the methods and knows what to expect when the research is completed, but when you start to hear your stakeholders use terminology that you've introduced to them, that's when you can really know you're starting to have an impact. Your team is using this language on their own. I've seen this happen with my team with the whole findability versus discoverability thing, and it's a great feeling to see that they've internalized the language, what it means, and how it impacts the decisions they're making.

Beyond the report: cite your research everywhere

[00:24:57] Finally, I'm hoping this section addresses a common fear that I know I have felt and I've heard from other researchers too: that our work ends up sitting on the proverbial shelf, never to be looked at or referenced again. You've done all this hard work. You've recruited your users and talked to them, you did the analysis and the synthesis, and you've created this beautiful report at the end of it, and you don't want your work to end there. This whole section is about how you can make sure that even after your report is completed, you're maximizing its impact and reach with the stakeholders who are going to be able to action it the most.

[00:25:36] One of the easiest ways I've found to make sure this happens is to cite your research literally everywhere. To me this is a lower-touch way to make sure your research is reaching the broadest audience possible. I do this by sharing it in Slack messages and dropping it in Zoom chats or Teams chats. If I'm looking at a document somebody's asked me to review, I'm leaving comments and linking the research directly there. I'm just making sure the research work I have done is shared as broadly as possible.

[00:26:12] Let's say I'm sitting in on a team meeting, and I'm presenting or listening to the team present and ask questions about something, and I notice that a theme is starting to come up that we've actually done research on before. This is the moment I will link the research into the Zoom chat. I'll just drop it in there, I might add a little bit of context, but it's a way to get it out in front of the audience while it's on topic and relevant, because they're discussing this work right now. What I'll notice is that as soon as I send it out, I see people joining and taking a look at the document, and right then and there I know I'm starting to have impact. So find as many opportunities as possible to link out your sources and the research you've been doing, and make it really visible that you've been having an influence and impact on the work that's relevant for your team today.

[00:27:13] In addition to citing your own work, it is your responsibility as the researcher to be the voice of the user in every single room. I like to think about this as being the squeaky wheel: the person who's always reminding the team of what we've learned and how it's impacting the user's experience. One of the ways to do this is to diversify the rooms you're in. Attend meetings with different cross-functional partners, and make sure you're able to highlight your research insights there and how they directly relate to their work. This is also your chance to highlight where certain business decisions might actually be going against the insights or recommendations from the UXR team.

[00:27:49] Sometimes the business just moves really quickly, and without constant referencing of your work and constantly bringing it up to the team, there's a chance for your research insights to unfortunately be left behind. As you are constantly referencing it, bringing it up, reminding your team about what you've learned, the impact on the users, and how you can make sure you're delivering the optimal user experience, it's important to remember that while you should be the squeaky wheel, don't be too squeaky.

[00:28:16] What I mean by this is: pick your battles. Where does it make sense to say, no, we absolutely need to solve this for our users, this is high priority? And when is it time to say, we've learned this, I've shared it enough times, and the team is going in a different direction, so how can I make sure my insights can at least inform that new direction? Just make sure you're maintaining that compromise mentality to keep those relationships at a really good level.

Connecting the dots

[00:28:48] And then finally, as you're thinking about how your own research can live on, remember that it can also be your responsibility to help the existing work of others live on as well. I like to think about this as connecting the dots. How does my work relate to work that others have done? Maybe this is work from other researchers or data scientists, maybe it's the market insights team, anything that tangentially connects to the work I'm doing. It can also be a way to elevate the work of others by finding those connection points, and it really helps to strengthen the narrative and triangulate the insights you've captured by saying, look, we've seen this in other data as well.

[00:29:31] As you're doing this connect-the-dots work, you might even realize that some insights you've captured in your own study actually have a pretty strong impact on another team, maybe even more strongly than on your own. So you're also starting to create opportunities to proactively identify and propose new research questions for the business, and again you're expanding the reach of research. This happened very recently for us at Uber. We just kicked off a brand new research effort because three different researchers all had different insights that at face value didn't seem connected. But when you really looked at it, there was a strong connection point, which led us to a brand new research question that all three of them are now working together to tackle. It creates a lot of opportunities to make sure you're looking at the business from every single angle, and elevating the work of others, not only yourself.

[00:30:26] As you start to put these techniques into practice, hopefully you're going to start seeing impact when your research is being cited when you aren't in the room. It's tricky to say you're going to see your impact when you aren't in the room, but there are some ways you'll start to notice it's happening. Other people are going to be citing your research for you. Maybe you're looking at a document, or any work that somebody else has created, and they're referencing your research as an input to the decisions they're making. Maybe you're in the room when somebody else mentions that you've done this great research and this is what they learned from it. The more other people start citing your work, the more you can trust that it's being cited when you're not even there to speak to it, and a lot of this can happen because you are citing your own work as often as possible.

[00:31:14] The next thing is that you'll find your work is sticky. What I mean by sticky is that it's referenced continuously, and people are coming back to look at it over time. It's not something that has an immediate spike and then dissipates. People have found value in it, and they want to keep referencing it and keep it top of mind. This is actually a place where you can start to use numbers and metrics to measure the impact. At Uber we use the Google suite, so I rely a lot on the Google activity dashboard to see who's been viewing my report and how often it's been viewed in the past few weeks or month. Did it have an initial spike and then people forgot about it, or are there consistent viewers over time?

[00:32:00] So try to find ways to make sure your work is sticking for the people it needs to stick with the most, and also start looking at where it's being shared. Which leads me to my last point: your research is starting to impact the business and not just your team. You're connecting the dots, and you're finding where your research actually has a greater impact on other parts of the business, not just your own. Part of this comes from all those strong relationships you've built, the foundation we talked about at the beginning. And you might even be receiving more research requests from beyond your immediate team, so you're starting to see that the impact and reach of your research is getting broader and broader with every project you're doing.

Closing thoughts

[00:32:44] Thank you all for staying with me. I know we've covered a lot of content. I just want to close with a few final thoughts on how you can ensure your research is having the maximum impact possible within your teams and your organization. We touched on three techniques for increasing your impact, and it's important to know that these techniques are not meant to be used in isolation. It's the combination of all of them together that is the force multiplier for the amount of impact you can have with your teams. Impact tends to build upon itself, so setting a great foundation means you're setting yourself up for success and maximizing the amount of impact you're able to achieve.

[00:33:29] And finally, just because you can't easily measure it doesn't mean it's not impact. Start to take note of all the different effects you're having on your team, no matter how big or small. Check in with yourself every couple of months and take note of all the different changes you're seeing and the subtle behavior changes. Maybe you're noticing the team is adopting a certain language, or you have that one stakeholder come to you with a research question instead of a method. Take note of all of these, because while you're in it you may not be able to see the change happening, but upon reflection you should start to notice that you're having a pretty great impact on your team and the business overall.

[00:34:15] Thank you so much, and thank you to UXDX for having me. I hope a couple of interesting tidbits stood out to you that you can apply back to your work. My name is Kendall Avery, and we are hiring, so please take a look at uber.com/careers. Thank you so much.