Experts at Scale: Solving Problems with Process, Professionalism, and Politics

Kevin Newton, Brandi Amm at UXDX USA. Video: https://youtu.be/4OvR2K-j570

Readable transcript: edited from the recording's captions for readability (fillers and false starts removed, punctuation and section headings added). Wording is the speaker's own. Timestamps are positions in the video. Names marked [?] could not be verified against the audio.

Three principles for scaling research

[00:00:00] Kevin: Hi, everyone, and thank you for tuning into our talk. My name is Kevin Newton, and I lead the user experience research team at LinkedIn, charged with scaling the impact of UXR while maintaining rigor.

[00:00:09] Brandi: I'm Brandi Amm. I'm a research program manager here at LinkedIn who works with our user experience research team. I work with Kevin's team to identify problems and produce solutions, processes and guidelines at scale. We are here to present our simple process for scaling research to achieve continuous learning about products.

[00:00:30] Kevin: Thanks, Brandi. First, we will explain the framework we use to scale, and then, most importantly, walk through two case studies to highlight how the framework comes to life in the real world. Now, if you're watching this before the UXDX conference, I hope you'll join me on Wednesday, June 16th, 2021, at 10:55 AM Pacific time to discuss continuous discovery in practice live with Teresa Torres, facilitated by Flavia Neves. With that, let's get started.

[00:01:04] This framework is our guiding compass for what, why, when and how we create successful programs of research at LinkedIn. It's quite simple: three principles. You'll notice they're very similar to how user experience is approached more broadly. This is intentional. When we consider scaling research, we think of it like building a product that will be used by teams for creating continuous discovery. So you can see: you've got to identify a problem, design a solution, and then make it easy and consistent. Let's dive a little deeper into each principle.

[00:01:41] Identify a real problem. Just like when a company creates a new product, it's crucial that whatever is created solves a real problem. Only this problem exists within the company or organization that you work for, instead of out in the public. To deepen the analogy with startup culture, this step, when done correctly, ensures that the process built to scale research is an aspirin, something that someone needs to relieve a pain, as opposed to building a process that is a vitamin, something that could provide benefits sometime in the future but doesn't feel necessary in the present moment.

[00:02:17] A simple way to do this when it comes to scaling research is to ask yourself two simple questions. What needs, if any, aren't being met? And why aren't they being met? You may need to do a little research: talk to your stakeholders and colleagues to discover what the actual problem is. However, there is a danger here. Don't scale research just because you can, or because you were asked to by your boss. It will be exponentially more difficult if you are not solving an actual problem.

[00:02:46] Once you have identified an actual problem, it's time to ideate on how you might solve that problem. Similar to the design thinking process, these ideas should not be limited by how you would need to solve this problem. This step really is about answering the question on the screen: what must be true for this problem to be solved? If the problem is, for example, that you have no dedicated researchers, what you would need to be true is that you have someone who can conduct research on a consistent basis.

[00:03:21] Like the last principle, this has a trap too. It would be easy to say, "We need to hire a researcher, because we don't have anyone to do research," or "We need to secure headcount for a researcher." And the trap here is that there are many ways to solve the problem when the criteria do not include the solution. In this example, having someone to conduct research on a consistent basis is not the same as needing to hire a full-time researcher.

[00:03:50] Once you have a solution-agnostic criterion, then you can find solutions that are both easy and consistent. At this point, the question and the answer are all about efficiency, keeping in mind that the research needs to be rigorous and valid, of course. Let's take the same example, needing someone to do consistent research. What's the most efficient way to make that true? Do you have headcount? Great, then you can hire a full-time researcher. Do you have budget instead? Maybe you need to get a contractor or hire an agency to start doing some research for you. Do you have a research team within marketing? Could you maybe partner with them to run consistent surveys? It's not the deepest learning in user experience, but it could help show the value that then leads to more investment.

[00:04:35] Like the other two principles, there's one thing that you really want to stay away from during this step: do not over-complicate it for those who are going to carry out the process. That second part is a very important qualifier, because the process or system or program that you design to solve the problem could in fact be super complicated in the background. But for those running it and participating in it, the process and experience should feel easy and consistent. So there it is: three simple principles. Now Brandi's going to walk you through our first example of how these principles come to life in the real world. Brandi?

Lemonade stands: identifying the problem

[00:05:17] Brandi: Thank you, Kevin. The first case study we will be speaking about is Lemonade Stands. For context, in the beginning of creating programmatic work at LinkedIn, we loved to tie them all together with fun food names that helped represent the type of research being done. Not only was this a way for us to add humor, but it was also a way for those who are not aware of UXR to gain some idea of how the programs work. So what is a lemonade stand? Lemonade stands are quick 30-minute sessions where internal LinkedIn employees can sign up and provide feedback on about-to-launch design flows and interactions. These evaluative sessions revolve around concept testing or usability testing with our internal employees.

[00:05:59] Following the framework Kevin spoke about before, our first step was to identify a real problem. In this case, the problem came to us while working well with our partner teams: we still often found ourselves in the position where we were not meeting the needs of our design and product teams. Our first question was, what needs are not being met? First, we needed to be able to do more evaluative work on designs that we were getting ready to launch. While we were certainly doing some of this work, it was often lengthy, and there was always the ever-concerning point of that looming timeline. With more products being launched, we needed a system that worked in a timely fashion at scale.

[00:06:35] Second, we had internal employees who were not only using the product but were some of our biggest evangelizers to our external customers. While there were some internal avenues to give feedback, we wanted to create a more robust system, as well as a way for our employees to feel that their insights were being heard and implemented.

[00:06:57] As we all know, part of solving the problem is being able to identify the root of that problem. Our next step was to ask why. Why did these problems exist to begin with? Well, ultimately it came down to the most basic of problems: a lack of resources and a lack of established processes. We did not believe we had the research and research operations resources to support doing the evaluative work, but also doing it at a larger and quicker pace. We had no processes for quick evaluative research at scale, as we'd been hoping for. We needed to reduce the friction, create and streamline our processes to use fewer resources, and bring our stakeholders along for the ride.

Lemonade stands: designing the solution

[00:07:37] Now that we had defined the true problem, we next had to ask ourselves a few questions to start designing a solution. What must be true for us to successfully design a solution? Well, our first thought was that we had to reduce the bandwidth required for the researcher. This led to the idea that the lemonade stands must be designer-led. With the designer as the researcher, we were reducing the time the researchers needed to be involved in sessions and increasing our partners' involvement in the research process. This was an ingenious idea, because it was a way for our partners to be fully present in the research process, while also creating empathy and deepening the collaboration between our teams.

[00:08:13] We next had to identify how the researcher would be involved. We have found that outlining roles and responsibilities is a key way to reduce friction. So we put the researcher in the role of the consultant. They were a key partner in the process, but leaned on the designer to lead, while providing a foundation of knowledge needed to be successful in doing this type of research. They could also be a resource for the designer, so the designer felt supported and felt that we were setting them up for success.

[00:08:45] Our next, and of course, in my humble opinion, the most crucial step was streamlining operations. While streamlined processes are in every operations professional's handbook, this definitely wasn't an easy task. We had to consider all the touchpoints in the research process for our research operations team, and then either eliminate the touchpoint with tools and templates, or reduce the bandwidth required for our research program managers, lab operations and research coordination team. Well, I'll tell you, this could keep me up at night, and I'm really only partially joking. Lucky for me, I had a great partner to work through these problems with.

[00:09:16] The last solution we had to design was around closing the loop with our internal employees. At LinkedIn we have a saying, "Relationships matter," and we take this very seriously. Our relationship with our internal customers matters. Their voice is important and it's well worth hearing. So when we were designing a solution for the problem of having no internal customer voice, we wanted to make sure that this was a relationship that not only helped us but also helped the customers who came into work every day at LinkedIn. We wanted to be able to gather these valuable insights while also connecting with our internal voice and making sure that they felt heard.

Lemonade stands: making it easy and consistent

[00:09:55] Once we had identified the problem and designed a solution, it was then time for us to consider how we would make the solution easy and consistent. What was the most efficient way to make those things true? Well, our first step was to create a self-sign-up sheet through our internal project tracking tool that designers could use to sign up for a lemonade stand. We then attached this to an easily remembered internal site. This site allowed designers to see upcoming dates and sign up at their own convenience. Once they signed up, a researcher and a research operations person would be notified, and they could be assured that the dates were confirmed for sessions.

[00:10:29] Research operations then worked with our researchers to create planning templates for designers. These templates would then be reviewed by the researcher. This would offload the burden of planning from the researcher to the designer and create consistency between different groups of sessions. The consistency streamlined the process and allowed us to go from multiple planning meetings down to, if needed, one. Oftentimes it could still be done asynchronously.

[00:10:54] Through outreach and evangelizing the program, we were able to create an internal database of all eligible LinkedIn employees who would volunteer to participate. This probably required the most time from both teams. We spent multiple meetings working with our internal employee leadership to align on how this program could help both teams. Over time, we expanded the database as more internal employees heard of the program and wanted to be a part of it.

[00:11:21] And lastly, we created a spreadsheet template for the results. This was then reviewed by the user experience researcher. We would leverage the spreadsheet to close the loop. This included not only sending comms about insights to the relevant design and research teams, but also to our internal employees, so they could see their feedback was being used and moving us forward in the process. This is how we used the framework to create lemonade stands and solve the problems we were seeing within this area of business. Now I'll pass it off to Kevin, and he'll speak about another program we created, Project Ripple.

Project Ripple: the problem

[00:11:55] Kevin: Thanks, Brandi. Project Ripple is the code name for our efforts to quantify the user experience. It's called Project Ripple for two reasons. First, we knew we would want to start small and spread across the company, very much like a ripple in still water. And second, "Ripple" is the title of a song by the Grateful Dead, a legendary San Francisco jam band. There's a line I will mention in just a minute that fits this initiative perfectly.

[00:12:24] Let's take a look at the framework. Step one is identify a real problem. When we asked ourselves what needs aren't being met, it was obvious to us in the design organization when it comes to this area: there was no consistent assessment of the user experience at scale. Sure, we tested things in the lab. We uncovered problems to be solved and we gave directional feedback on design choices, but there was no way to measure whether or not the experience was getting better. We have a lot of metrics that guide decision making at LinkedIn, but none of them capture what it feels like to use the product. They certainly don't do so in a continuous and consistent way.

[00:13:06] Another important problem to solve was design not really having a seat at the numbers table. When a designer made an argument for why an experience should be like X, Y or Z, if a PM or a data scientist or a market researcher had numbers to suggest that the experience was doing fine as it was, there wasn't much design could do to convince them or argue against that. We needed a guiding number.

[00:13:33] When thinking about why these needs weren't being met, there was a process reason: we didn't have a process within design to measure the user experience. There was a professional reason: we had built the UXR team at LinkedIn to exclusively conduct qualitative research. That's what we were known for. We were a team that dove deep into the lives of our customers and members to understand their experience, and then presented a case for why LinkedIn should care about this problem or outcome. Lastly, there was a political reason. We already had so many metrics that functions outside of design were not particularly interested in just another number.

[00:14:22] Now we come to that Grateful Dead lyric that I promised. It goes, "Ripple in still water, when there is no pebble tossed, nor wind to blow." The company wasn't asking for this. There was no catalyst, like a pebble or wind, but we knew that this was a real problem within design, and we believed if we could solve it for them, for us, the value would spread from there. So that's what we did.

Project Ripple: what had to be true

[00:14:47] We set out thinking what would have to be true to solve these problems. We knew that to have a clear process, the methodology would need to be airtight: rigorous, valid, repeatable. This would increase both our credibility in a new space, quantified UX, and the longevity of the program, by allowing any new hire to learn how to conduct Ripple studies with ease. We also needed a way to consistently run studies with 300 participants. We typically recruited 8 to 20 participants for our UXR studies, so this was a departure for us, and we needed to figure it out.

[00:15:30] In terms of the team, we needed to ensure that qual researchers could conduct quant analysis. Thinking back to when I introduced the framework, remember that the "how" these are going to happen doesn't necessarily need to be settled at this stage. It can be; it doesn't have to be. All we knew is that qual needed to be able to do quant. And lastly, as far as the political reason goes, the value offered to the product organization needed to be blindingly obvious. Simple.

[00:16:07] Not only did we need to make those things true, but we needed to make it easy and consistent as well. So we asked ourselves, what was the most efficient way to make those things true? Well, the first one: developing the process. We immediately partnered with an expert in the field to develop the methodology. This saved us the time of trial and error and prevented us from reinventing the wheel. Although we did make LinkedIn-specific modifications when necessary, we leaned on the vendor, the agency, to help us develop this process and this methodology.

Project Ripple: making it easy and consistent

[00:16:46] Next, we once again worked with the operations team to create templates that could be leveraged by any UXR to create a Ripple study that would look almost identical, the only difference being the content tested. The metrics collected, the questions asked, when they were asked, where they were asked and how they were asked would all be the same every time. We leveraged a software tool named UserZoom to collect the larger amounts of data, and worked with the operations team to create a straight-to-participant recruiting strategy.

[00:17:29] Typically, there's a step where we recruit people who indicate they are interested in participating in a UXR study, and then from that list we invite them to the actual study. Well, this new strategy cut out the middle step, where we simply assess interest, and just invited everyone who clicked straight to the actual UXR study.

[00:17:50] As far as the qual researchers doing quant, that was a challenge in two ways. First, it wasn't their skillset, and it wasn't why we hired them in the first place, so we didn't expect it to be their skillset. And second, it wasn't necessarily an area they were excited to learn or explore. Having a background in anthropology, I know a lot of qualitative researchers, and I have a feeling there aren't too many of them who get super excited about studying hypothesis testing statistics on a Friday night.

[00:18:24] So we took the same approach we took with the lemonade stand: how can we make this easy and consistent? The answer was to build a self-updating calculator in Excel, which, let me tell you, is super complicated, but it feels easy. This allowed any UXR to follow the study templates that we had developed with operations, create a Ripple study in the same way, download the raw data from UserZoom and essentially paste it into the Excel calculator. And then instantly have a dashboard that tells them in plain language which groups, if any, have a meaningful difference, a statistically significant difference, in how they experienced the product being tested. It was pretty cool.

[00:19:21] Lastly, we have had an overwhelmingly positive response from our product partners. They often ask, "When are we running another Ripple study? What is the future of the program? When will it be run consistently on all of the products?" Additionally, we are working with other research functions to connect our metric to the popular metrics used within LinkedIn. You can think NPS, or CSAT, customer satisfaction, or even data science mappings of how people use the product.

[00:19:57] Our goal is to be able to connect our metric with those metrics. To use NPS as an example, we might be able to say, "If you move the Ripple score by X points, you can move NPS by Y points." Eventually we hope to have a program that is measuring our experiences in a continuous way, complete with a self-service dashboard to explore product scores and what issues are causing those scores, so that really any team member at LinkedIn could just go there and understand how their product is doing and what the experience feels like for their users. And that is Project Ripple.

Closing

[00:20:42] So there it is: three simple principles, each with a fairly strict warning. Because we've found that if you solve a real problem for someone, understand what needs to be true in the design of the solution, and then find the most efficient way to make those things true by creating ease and consistency for everyone involved, research will scale itself. When it does, the value of UX goes up in the company, and the team learns together more continuously and more consistently. So thank you for listening. If you have any questions, feel free to come to the Q&A on the 16th, or find us on LinkedIn, and let's start a conversation. Thanks.