Roadmap To Establishing A User Research Program

Yael Gutman at UXDX Europe. Video: https://youtu.be/3e8d8S7reJc

Readable transcript: edited from the recording's captions for readability (fillers and false starts removed, punctuation and section headings added). Wording is the speaker's own. Timestamps are positions in the video. Names marked [?] could not be verified against the audio.

Why data-driven decisions matter

[00:00:00] Hello, everybody, and thank you for joining me today for the session Roadmap to Establishing a User Research Program. My name is Yael Gutman, and I've introduced data-driven decision making to three organizations, building up user research programs from the ground up. It's the tips and tricks that I've collected throughout my career that I want to share with you today, so that when you go on your own path, you can do so as smoothly as possible, with the least obstacles.

[00:00:26] But first, a little bit about why this is so important to me. I'm obsessed with identifying opportunities and finding the key to the solutions. But throughout my career, I've been in situations where I wasn't clear on how we got to the path we're on. There were three main contributors to this. First, receiving top-down directives on what solution to build, with little to no discussion of what problem we're trying to solve. In other cases, where we did discuss what problems we're trying to solve, it was something based on organizational folklore or executive intuition, and not on empirical data. Another challenge was that some in the organizations believed they knew their users so well, there was no need to gauge the feedback from users.

[00:01:10] To combat these challenges, I started using data. And I found that when providing the same stakeholders and executives with data, they were more than happy to review the thinking and change strategy. And so this became my strategy, and how we started making more informed decisions.

[00:01:32] And this is backed, of course, by studies. A McKinsey & Company study from 2018 found a clear correlation between good design and superior financial performance. A Walker study from 2019 estimated that by 2020 customer experience will be the key contributor to brand loyalty, overtaking price and product quality. They also found that 86% of buyers are already willing to pay more for a better customer experience. Zappos and JetBlue are great examples of companies that put user experience at the top and benefit from it financially.

The challenges

[00:02:11] But although there's clear evidence of why companies should invest in this, there's still a way to go. Many companies are finding it difficult to adopt methodologies that will get them there. The same McKinsey study found that 40% of companies do not talk to their customers during development. A UserZoom study that focused on surveying UX professionals found that the biggest challenge of UX professionals is getting UX research and testing to be part of the product development process. 49% of their professionals stated that executive buy-in is a challenge.

[00:02:51] These are some of the challenges that I faced in my organizations that can give some color. Limited resources: some organizations believe that they don't have the budget or the time for user research. So today we're going to talk about how you can start really small, nimble and cheap, to really get the organization psyched and excited about user research and its impact on the organization.

[00:03:12] Next, there are organizational structural issues that make it difficult for user research to be adopted. One we discussed before, when the organization already believes they know the users and there's no need for user research. Another obstacle is where user research is available in the organization, but it's only in specific areas, done by specific teams. It's not across the whole organization. Another area that's challenging is when departments that we should be collaborating with don't really understand the benefit of user research, and they may find it to be stifling their creativity or slowing down the development. And today we'll talk about how you can turn those colleagues into ambassadors.

[00:04:00] So this is the roadmap we're going to review: the four steps of the roadmap, and we'll go through each of them. And storytelling is an element that grew throughout the longevity of the program itself.

Establishing a baseline and a toolset

[00:04:16] So the first order of business is to understand the baseline of what your users are doing and what they're thinking, and to identify the toolset to capture that data. In order to provide insights to your organization, you need two types of data sets. One is the quantitative data. What are the users actually doing? What are they clicking on? What pages are they on, on your website or on your app? The other type of data set is the qualitative data, which is why the users are doing what they're doing. And you need both of those in order to get the complete, holistic view of what your opportunities are. And there are choices that are reasonably priced, and three methodologies that will allow any organization a low entry into the user research world.

[00:05:07] So, for quantitative. When I joined my current organization, we had Google Analytics set up, the free version. It wasn't implemented in a holistic way. And so the first order of business was just to straighten that out, so we could have a baseline. Next, we did want to start with A/B testing. We tried the Google Analytics free A/B testing tool. We have a one-page application, so it wasn't flexible enough for us. But I definitely recommend that you check it out if that's something that would work with your platform. We ended up, after securing investment, integrating with Optimizely, which is the current tool we use for A/B testing, and we're very happy with it.

[00:05:51] CrazyEgg is a tool that I've used in the past for heatmaps. It's limited in its capabilities, but for the beginning stages you can find it very helpful. We've since moved on to Clicktale, now Contentsquare, for heatmaps as well as recordings.

[00:06:11] For user research in my current organization, we started with user interviews and user testing. And to ease the organization into it, we did it with no additional cost to the organization. I planned and executed the interviews and the tests with a colleague. So it was just within our regular roles, no additional cost. I'm very lucky: my current organization is a member organization, and so we have easy access to members. We ask our membership team for the types of users we're interested in, and they provide us with participants. But regardless of what type of organization you are, you can start really small with user interviews and user testing, just to prove the validity of this methodology. And once you prove the methodology, you can then go and request more investment in the program.

[00:07:06] You can also use Usertesting.com, which is very reasonably priced. It's an online tool where you can set up your actual test, and then observe the participants, which Usertesting.com also provides.

[00:07:22] For surveys, I started, again, very cheap, when I started in this organization, with SurveyMonkey. Believe it or not, it was a thousand dollars a year, and it served our purposes. It was limited. It wasn't as flexible as we needed, which is why eventually we moved to Qualtrics, so we could create more flexible ways of targeting our audience and prompting the survey. But SurveyMonkey was a very good tool for us.

Storytelling: sharing insights

[00:07:52] Next, once you start establishing user research on your own projects, as an individual contributor, it's time to share your insights. And you should share them as soon as you get them, so that your department, your management, and then eventually the organization get to hear about user research and its impact on KPIs and strategy. And when you storytell, it's really about telling the story behind the data. You should share the insights, package up the data so that it's meaningful to your audience, and don't make them do mental gymnastics in order to understand the point. Just feed them the learnings that you've established, and provide them in a digestible format.

[00:08:38] Use plain English. Sometimes we forget that the terms that we use in our day to day are not shared with the whole organization. So really use language that the organization, on the whole, understands. And lastly, you have to adjust your message to the audience. When we present to the management and to the board, it's the bite-size nuggets, the most important learnings that we had. When we share it with the broader team or with the working teams, we go more in depth and share the whole flow. So it really depends on who you're talking to.

[00:09:14] So this is an example of a fictitious project of implementing chat. And this is something that we would share with the management. 67% of issues were resolved during chat. That was the goal of the project, and so that's what we would message back to them. And we're also providing cues in terms of the visuals. Issues resolved are in green, which just visually means this is good. And the 12% that's indicated here, which is where the member still has to take some action, is in a more red tone, indicating just by the color that that's something that we should look into.

[00:10:00] When we would share information with a working team, we would provide additional information that wouldn't be of so much interest to executives. In this case, we would share the issue types and the issue complexity. And this is because the working team could use this information in order to make decisions on the strategy and how to serve customer service.

Expanding research to the team

[00:10:26] After you've established user research on your own projects, it's time to expand and integrate it into the rest of the team. A challenge that I faced was that not everybody on my team was inclined to do user research. And in order to tackle that, we mandated it. We integrated it into the product development lifecycle as a must-do within the project. But to make it easier for the team to manage user research, to plan it and to execute it, I templatized everything I could think of. I templatized a discovery deck, I templatized a test plan, I templatized the test results of an A/B test, I templatized a post-launch report. The idea of templatizing was to provide my colleagues with a template that would make it easy for them to focus on the content and not on the structure itself.

[00:11:32] Another thing that I did was that I leveraged potential ambassadors within my team. I identified specific staff that were more inclined to do user research, and then really focused on them. I helped them plan the research, monitor it, and then deliver the results. And this of course improved their rationalization of why they did a project, but what it also helped with was establishing the importance of user research within the department itself.

Facilitating ideation

[00:12:08] Once you've established yourself as an individual contributor, and then expanded it to the rest of the team, it's really important that we are able to provide an environment for creativity and for research ideas. When we started A/B testing, the team was so excited. We had so many ideas. But at some point after, the enthusiasm went down, and once we'd gone through all the low-hanging fruit, we hit a dry spell. And it was a real challenge to come up with testing ideas.

[00:12:40] And so, to combat that, we did two things. One, we leveraged partners that we already had within our organization. We started sharing problem areas with those partners, and we solicited advice from them, from experience that they've had with other customers in a wide range of industries. And they provided us with solution ideas as well as testing ideas.

[00:13:09] Another thing that we did was that we expanded how we collect ideas internally. We created an inclusive research think tank, and we invited other teams to participate. In our case, we invited our Technical team, QA team, as well as Customer Service. And what we identified was that by including them in these sessions, we were able to get additional points of view that we didn't have before. And the breadth of ideas that we were hearing was much wider. And the benefit of making this think tank inclusive was that these participants from other teams became ambassadors, and went back to their teams and told their colleagues about the user research program.

[00:13:58] But these think tanks were not a success from the get-go. The first couple of sessions did not result in new ideas. And it was at the end of one of these sessions that a colleague from the Technical team asked me, "How do you actually come up with ideas?" It was so enlightening. Product and design are tasked and expected to solve issues, whereas the tech team is usually charged with execution. And so it's not part of their daily routine to think of ideas. So we changed the format of those sessions. Instead of asking the team members to come in with ideas for testing, we asked them to come in with problem areas that they've identified within the user experience. And it was within that session that we discussed the problem area and the challenges, and organically came up with possible solutions and testing ideas.

[00:14:56] Another challenge that we faced was that not everybody was comfortable suggesting ideas. There was this sense that it was a little bit dangerous: what will happen if it doesn't succeed? So in order to combat that, we started publicly celebrating failures, as they were once called, but we called them learnings. So when we have an A/B test that doesn't hit the KPI that we had expected, we share that just as much as we share those that do. And we celebrate the fact that we didn't go on to implementing something, and getting technical resources to do something, that would not have been as impactful as we had hoped. We also emphasize how good these learnings are, so we know these are all building blocks towards additional tests.

[00:15:46] Another thing that I do personally is that I share when I thought, "Oh, I thought this test would do this, or would do that," and how humbling it is to learn that it didn't. And the last thing is that we try to make these sessions fun. Of course, this was before COVID, when we did meet together, and we would do these contests where whoever guesstimated closest to an A/B test result would receive a gift card. So it's always good to make it as engaging as possible.

Research as part of the culture

[00:16:24] This is the last phase, where research is really part of the organizational culture and indispensable to the organization. This is where we are now, and this is a continuous phase where we want to stay. This is where the product teams that initially didn't understand why we were asking these questions, and may have said that this was stifling their creativity and putting a pause in development, are now coming to us requesting data so that we can make more informed decisions.

[00:16:55] Through our continuous socialization of results within the organization, we've had other departments come to us and request that we help them with their research needs. So apart from being very gratifying, this is also an opportunity to further solidify user research as a very important component in the organization. And this is where management starts expecting user research. If they don't hear about user research, they ask for it, and they expect it to be part of the delivery.

[00:17:34] So to summarize: just start. You can start small and you can start low budget. And you can start in a way that proves to the organization how impactful it can be. Once you prove this on your individual projects, as an individual contributor, it's time to spread the knowledge and integrate it within the product lifecycle itself, so that all projects benefit from user research. And the further you extend it, the more important it is to facilitate ideation. Come up with ideas so the ideas are fresh, and you can continue to optimize and learn more. And the most important thing is to continue to share and socialize with your team, with management, with executives. That is the way that you get to a healthy, stable and long-lasting program. Thank you so much for joining me today.