What is a Rolling Research program?

Michele Ronsen at UXDX Community: Rolling Research and Customer Feedback. Video: https://youtu.be/1Xz9f0_B_lo

Readable transcript: edited from the recording's captions for readability (fillers and false starts removed, punctuation and section headings added). Wording is the speaker's own. Timestamps are positions in the video. Names marked [?] could not be verified against the audio.

Introduction and background

[00:00:00] Host: What is a rolling research program? Welcome, Michele.

[00:00:07] Michele: Hi everyone, I'm super happy to be here. Thanks for having me. The UXDX team asked me to talk about rolling research, and I'm prepared to share a ton about it with you. I'm going to go ahead and start my slideshow. Hopefully you can see the cover slide. Can I get a confirmation that everything is looking good and you're good to go? Awesome. I've also created some resources for you, so if we could share our resources, that'd be great.

[00:00:42] For those of you who might not know me, my name is Michele Ronsen. I'm a user research executive coach and educator. I teach people around the world how to break into the field and how to upskill. I do a ton of corporate trainings and team trainings. I am a classically trained designer who moved into research about 12 years ago, and I founded my latest firm, Curiosity Tank, about four years ago with the explicit purpose of helping people learn how to ask better questions and make more confident decisions.

[00:01:22] I'm super grateful to LinkedIn: LinkedIn named me a Top Voice influencer in 2020, the first user researcher ever, so I communicate a lot on LinkedIn. I'm super passionate about user research, and I'm also a practicing researcher. For context, here are some of the questions that I'm asked to explore most frequently, and this will set the stage for our conversation today as it relates to rolling research, because rolling research typically only relates to evaluative questions. What we have here are both evaluative and generative questions. Generative research questions typically happen in the beginning of the product or service development cycle, and evaluative research questions occur once we actually have something to evaluate.

[00:02:20] I also lead a program called Ask Like a Pro, in which I take user researchers through every single phase of the design process, and mentor them and provide guidance as they're working on actual research projects. You may hear me refer to a lot of things in terms of Ask Like a Pro, and some of the resources that I shared with you come from the Ask Like a Pro learning and training series. If you haven't had a chance, you'll want to download these resources, because there's also an hour-and-a-half event focused specifically on rolling research. It's a panel discussion where I invited the leaders of the rolling research programs from Zillow, Pinterest, Google and Roku, and you can hear firsthand about their programs as well.

What rolling research is

[00:03:22] What is rolling research in a nutshell? It's an iterative method. It's a continuous approach where we're conducting studies on a regular, ongoing basis. More frequently, teams are conducting studies with distinct start and stop times. Rolling research occurs on a regular cadence. It can follow the same product or service throughout the development life cycle, or it can rotate from question to question or team to team. It's hugely beneficial. Mostly, more UX-mature teams have rolling research programs in development, but it really allows for super in-depth understanding, continuously, over a period of time.

[00:04:16] It's super well suited for agile teams because it continually gathers feedback, and then the teams incorporate it into the design or development process iteratively. Rolling research can be implemented through a variety of methods, and I'm going to get more into this a little bit later. The frequency varies depending on the needs of the team or the organization, or what's being developed. The most common experience I have personally with rolling research, and developing rolling research programs, is on a weekly cadence. As you'll hear from the other experts I had the panel discussion with, it fluctuates from weekly, to twice a month, to two weeks on and two weeks off, to monthly. None of them were running it on a quarterly basis, but some people might consider things like NPS scores or CSAT scores rolling research. Those sometimes get referred to as rolling research too, but that's not really what I'm going to talk to you about today.

[00:05:25] Rolling research on a weekly, bi-monthly or even monthly basis requires a super clear structure, flexibility and adaptability, and we need all those things in place for it to be deployed successfully. The most common thing about rolling research is that it uses the same something. It uses either the same methodology, the same recruiting approach, the same schedule or the same output, and it's usually a combination of these things, so that we can do it week after week after week and keep delivering on it. Something is consistent, and that consistency is required for us to be predictable, and to time it in order to commit and deliver on this rolling rhythm. I'm going to give you some more examples of that.

[00:06:20] The most common thread here, again, is predictability. The program structure, the program cadence, the type of questions rolling research programs can explore and who they explore them with really vary, but predictability is the most common thread.

Questions, participants and stakeholders

[00:06:39] What questions does rolling research explore? Typically it explores evaluative questions. Some examples of evaluative questions are what you see here. They're evaluative in the sense that there is something for the respondents to evaluate. The product or service has been developed to a point where there is a storyboard, a paper prototype, or even a low- or high-fidelity prototype. Those are all visual stimuli, but sometimes it's going to be audible stimuli: think about a call script, or a VRU, or a conversation with Alexa, or Alexa telling you a joke, or something like that. It can be audible, it can be visual, or it can be literal: a product description or write-up, or a comparison of different products and services according to different tiers of service. The stimuli can also vary depending on the team, but we typically include evaluative questions in rolling research because it's more tactical.

[00:07:59] What kinds of participants are included? This is a really good question and one that I field a lot. The truth is that it can include all types of participants and all segments, but recruiting is definitely a significant consideration when building a rolling research program, because you need to have those people ready, within arm's reach, to be solicited, recruited and scheduled on that weekly or bi-monthly or monthly basis. Recruiting is a challenge to definitely consider when you're building any sort of rolling research program, but there are many ways to resolve it, as I've explored my own solutions and as I've had conversations with many other teams about how they go about it. I'm happy to answer any specific questions too.

[00:08:55] People also ask which stakeholders are involved, and how. I hate to say it varies, or I said it depends, which is also true. The stakeholders who are involved in the rolling research program vary by team, as does the person who's leading the study. Sometimes there's a dedicated rolling researcher who is focused 100% on the rolling research program. Sometimes it's a staff researcher, and staff researchers roll in and out, or they'll do one month on, three months off, one month on, three months off, and it rotates through the team. Other times there might be a research contractor dedicated to it. In other instances other disciplines or functions are running rolling research, and those are scenarios in which we're training designers, PMs, engineers, sales, L&D or marketing.

[00:09:58] The stakeholders themselves can also vary, because those are the people bringing the questions forward for exploration. The question sets might arise from the design team, the product team, the engineering team, marketing, whomever, and that influences the stakeholders who are involved. Stakeholder expectations also vary widely. Teams often require a research sponsor or research lead to oversee the work if it's not being run through the research program. There may be a research lead or research sponsor to make sure that the questions are the right size, that they're appropriate for the rolling research program, and that they align with the broader goals, and they also help ensure quality work. But the stakeholders themselves can represent different functions in the organization as well.

[00:11:01] I know some programs require stakeholders to contribute at a certain level. Some may be required to watch all of the sessions live or participate in debrief sessions. The program structures really vary, as do the stakeholder expectations, who the stakeholders are, and who is leading the research. There's no one right formula. The formula is going to be highly dependent on the culture you're operating in, the resources, the talent and skills, the commitment, the adaptability, the flexibility, the team, the project and things like that.

Benefits and drawbacks

[00:11:51] There are many different benefits and drawbacks to rolling research. I love developing, designing, deploying and conducting rolling research. It's super quick. You get a tremendous amount of impact versus the time invested. I love the predictability of it, and my stakeholders love the predictability of it. It's a terrific approach to align the team to a very specific timeline. The deliverables, by which I mean the final outputs, tend to be much leaner than in a typical research project. I love the fact that it's repeatable, it's formulaic, and it has a templated process. Again, that process is going to vary a little bit from team to team.

[00:12:43] It's also a terrific opportunity for new researchers to break into the field. It's a terrific opportunity for new researchers to work on a lot of different studies very quickly, which will expedite their learning and increase their experience, and they're able to see how the learnings are applied very quickly. Think about any researcher, let alone a new researcher, conducting rolling research for six months. They could easily have 18 studies under their belt in that amount of time, and that's a tremendous amount of experience to acquire in such a small amount of time.

[00:13:24] Some of the cons that come with rolling research programs are that it's difficult for a single researcher to maintain this over a long period of time. There's a lot of burnout. I know some teams build in blocks of time for recovery, or after X number of rolling research studies they roll on to a different type of study with a different methodology. There's somewhat of a lack of variety in terms of methodology, because most of the questions are evaluative. There's also a lot of context switching. You could be running a project for a different team, or on a different product or feature, every week or twice a month, so you're constantly acclimating to your environment. I think that's super fun, but it's not necessarily a good fit for everyone.

[00:14:21] Another consideration is that the person conducting the rolling research really needs to acquire a certain degree of methodological expertise, and this is in lieu of subject matter expertise, because the subject can change every week. Again, it's not for everyone. It's not necessarily a good personality fit, or a good pace fit. I think it's totally fine. I think it's a great way to explore, try new methods, improve your skills and increase your pace. It's also a great exercise, especially for academics looking to transition into industry, because the pace is so different, and rolling research requires you to be so much more succinct in your communications and your outputs.

[00:15:13] As I mentioned earlier, there are some recruitment constraints. You need some recruitment support in place, whether that's a dedicated recruiter, internally or externally, or a participant pool that you can regularly draw from. All of these things have certainly been resolved at many teams around the world. You also want to ensure that, just because rolling research by nature is lean, lean doesn't mean superficial, and that action is indeed taken.

How programs vary and what they share

[00:15:48] Here's a quick overview of some of the pros and cons of rolling research. I also mentioned quite a variety in terms of structure, benefit, stakeholders and methodologies. The variations differ in terms of the methods that are acceptable to one program versus another, and in recruitment processes and approaches. Also the outputs: the outputs are much leaner than typical research studies, but the deliverable type and the fidelity can really vary. Some teams, I know, have a specific number of participants or a specific number of questions that they will not exceed. For example, we'll speak with five people each week and we'll ask them five questions max. So there are some variables in terms of quantity of participants and questions.

[00:16:53] I talked a little bit about how to resource a rolling research program. I've seen dedicated staff, I've seen shared and varied staff, I've seen rotations and contractors. Sometimes senior talent is leading the rolling research program, sometimes junior talent. Sometimes the studies are blind, sometimes they're branded. The stakeholders themselves, and their expectations and requirements, can also differ quite a bit. In addition, each team has its own culture, and each industry has its own culture, so there are some cultural differences as well. And the intake process: how will you accept, evaluate or vet whether something is appropriate for user research, and for that team and that particular method? Recruitment, output, resourcing, things like that also vary.

[00:18:00] But there are a lot of similarities too. As I mentioned earlier, stakeholders are critical. How and when stakeholders contribute to the process does vary, but there always needs to be a stakeholder, and there are always repeatable processes in place. These repeatable processes typically include the research question and its size, the scheduling and the recruitment, alignment with the sprint, how someone signs up or gets into the queue, how it's vetted, and what's discussed. There are typically very stringent cutoff dates to get the stimuli into the rolling research program. Stakeholder involvement is always required to some degree. And the outputs, again, can really vary as well.

What the outputs look like

[00:19:01] What do rolling research outputs look like? I mentioned they're typically a little bit leaner. I've shared here some of my simplified deliverables from rolling research programs that I have run. What you see here is a concept study. I had five concepts for the redesign of a healthcare website, the desktop experience. I ultimately built this program out, and we ran over 180 studies in 18 months in a rolling research program. We were conducting one, two or three studies a week across four segments, to develop it from these, which were two of the initial five concepts, all the way through to handoff to the development team, for every single part of the website.

[00:19:58] For instance, I would conduct research on Thursdays. I would analyze and synthesize on Fridays. I would present very lean deliverables to the design team and the engineering lead on Mondays. This is an example of my visual markups; this is what I would present. Monday evening we would decide what we were going to test the following Thursday. Tuesday I would begin my recruit. Wednesday I would author my discussion guide and schedule, and designs were due Wednesday night for Thursday testing. It's very, very quick, and the impact of this is just enormous.

[00:20:41] For the client, I would share deliverables like this. What you see on the left was week one; you can see the date here is October 13, 2015. This was seven, almost eight, years ago, for my first rolling research program. Week two: you see what we tested and the key takeaways, and the date there is April 21st. Week three. Oops, I think I went backwards. There we go. We're progressing in fidelity as we move through the product development cycle, but again, these deliverables are extremely simplified. You have to simplify a lot of things in order to deliver on that cadence.

[00:21:31] The outputs vary, but they're going to be consistent within the team. I know some teams have a strict nine-slide format that includes very specific things. Other teams use only a top-line report, but they have very specific consistency in other areas. My preference has always been visual markups, because I work really closely with the designers, and that's a very easy way for me to communicate what's working well, what's not, what's resonating, where the opportunities are, and what we might want to explore or do differently. There are lots of other variables as well.

Does rolling research make sense for your organization?

[00:22:10] Does rolling research make sense for your organization? That's going to depend. It's going to depend on whether your team has the structure, or the wherewithal to develop the structure, and the discipline, and what sort of guardrails you're going to put in place: what will the rolling research program be built to handle, and what will it not? There's also talent required to conduct the studies. Clearly someone with previous rolling research experience will be in a better position to build a rolling research program, because they have some transferable knowledge, but not necessarily to maintain the program, and those could be two separate people. You also want stakeholder buy-in, and there needs to be a real commitment to execute on the learnings. Some sort of recruitment process is going to be required for you to deliver on the cadence, and adaptability.

[00:23:09] Those are just a few of the things that I would encourage you to consider if you're interested in building rolling research at your organization. I know that all of these companies have really robust rolling research programs. It's very popular, and the impact is enormous. The progress and the iterative nature: you really can't get any better than building a rolling research program to align the team, get everybody on the same page, and commit to deadlines and to improving and iterating on the same cadence.

[00:23:51] Again, here are some resources that I put together for you. I invite you to look at them. There's also a link in there to the one-and-a-half to two-hour panel discussion with rolling research leaders from other organizations. I invite any questions that you might have. Thank you very much.

Q&A

[00:24:13] Host: Excellent, thank you very much, Michele. That was really interesting. I enjoyed digging into all the pros and cons as well, because it's great just saying everything's perfect, but in reality there are always challenges.

[00:24:30] Michele: There are enormous challenges to building a rolling research program.

[00:24:34] Host: Absolutely. If anybody has any questions, and I know some people were posting some from the last session, please do post in whatever platform you're in. But I had one or two questions coming out of that that I'd love to dig into. In Continuous Discovery Habits, Teresa Torres talks about ways you can try to automate the recruiting of your interviewees. One challenge that I've always seen teams face is this: if you give that structure where you're always doing your testing on a Thursday, you're trying to minimize the context switching for the team by having it on a set date. How does that work with customers' availability, because maybe that doesn't work for the customers? How do you balance those two competing things, the customers' availability and the context switching of the team?

[00:25:37] Michele: Again, the repeatable nature of a rolling research program is paramount. You have many more customers than you do rolling researchers or rolling research projects, so that's why building a recruitment strategy and a recruitment approach that allows you to deliver on whatever cadence your team has set up is so important. In my instance, for the healthcare organization, I built my own participant pool, and I continued to feed the pool constantly, week after week after week, with new questions. I was thinking four weeks in advance: what are we going to want to learn about then? Okay, we're going to want to get feedback from people who have chronic conditions. We're going to want to get feedback from healthcare brokers. We're going to want to get feedback from people who are healthcare proxies, because we were testing all of these different things in the site as we developed it. You're constantly thinking ahead to build your pool, so that you have the right people to tap into when you need them.

[00:26:52] Host: That's expertise. Just on that, with all the different styles of people or personas or use cases, do you mix your generative with your evaluative? As you mentioned, generative usually happens earlier, but I think there's some good value in trying to understand generative throughout, as a kind of unstructured learning throughout the project. What's your approach to mixing a little bit of generative and evaluative? Do you do them in the same session or in very separate sessions?

[00:27:31] Michele: I think that is going to be really dependent on what the question set is. If the overarching question is what's working well and what's not, that's purely evaluative. If the question set is what's working well, what's not, and how can we improve it, then there are typically some more generative questions in there. I have no problems working on that simultaneously. One of my favorite question sets to explore is: are we building the right thing, versus are we building the thing right? Those are two very different questions.

[00:28:16] In the rolling research programs that I've built and the studies that I've conducted, I am very comfortable jumping to both sides, provided of course that everything is happening according to that very structured approach. If I have the questions on Monday, I can draft the discussion guide on Tuesday, recruit on Wednesday, get the stimuli Wednesday night, and test on Thursday. As long as it's meeting the schedule, that to me is one of the most fun parts, because it introduces more variety, and it also allows you to explore and flex different muscles. But some programs would say no, we only do evaluative, we only do X or Y.

[00:29:14] Host: All right, well, that actually brings us to time. If anybody has any questions, please do keep sending them in, and we'll put those through to Michele on the Slack group. But that's it for tonight, so thank you very much, Michele.