Formative Research Sprints To Adapt Products To The New World

Anshul Agarwal at UXDX Community: Bengaluru. Video: https://youtu.be/2aygokcXLE4

Readable transcript: edited from the recording's captions for readability (fillers and false starts removed, punctuation and section headings added). Wording is the speaker's own. Timestamps are positions in the video. Names marked [?] could not be verified against the audio.

LinkedIn events and the world changing

[00:00:00] I'm Anshul. I work on LinkedIn events. LinkedIn events is a feature through which you can create and join professional events. These events could be meetups, online workshops, seminars. If you go to your LinkedIn profile, you can find it there on the left panel of the homepage where I have a pink dot on the screen.

[00:00:18] Two types of people make these events. One is the organizer, who is the event's host. Organizers create the event, decide the details, invite attendees, and add speakers. Another is an attendee, who has either accepted an invitation to attend an event or joined an event. Organizers can grow their business. Attendees can network with others and learn new skills.

[00:00:43] When LinkedIn events came into being, it was built on the premise that offline conversations lead to online conversions. You went to an in-person event. You met someone. You connected with them on LinkedIn, reminded them of the context of your in-person conversation, went into a chat, and asked for a referral to a job at their company. Some lucky ones get the job. Some not so lucky ones repeated the process again. All in all, we were a platform for people to take their offline conversations online, and LinkedIn events was a step to streamline that process for our members. We wanted to provide a platform for people to take offline conversations forward, hopefully leading to conversions online.

[00:01:35] Yet before we could completely figure it out, the world changed. When COVID-19 hit, there were quarantines and lockdowns. There were a lot more things, like a vaccine hunt. But one thing that wasn't there was people meeting in groups. Face-to-face events got completely wiped out. Initially, companies canceled their events. Then after a while into the COVID situation, they started thinking about getting their events online. Offline conversations were not that relevant anymore because they were not happening.

[00:02:08] The times were changing fast and so were the user expectations. Our product was already built on the premise of taking offline conversations online. We were exploring features to do so. But now was the time to take the whole face-to-face experience online. We had to rush the roadmap. We had to hasten to make a product befitting the changed times. But how? How do we do that? That's when I come into the picture.

Why a formative sprint model

[00:02:34] As a user experience researcher, I helped my team build an understanding of user behavior. Our team was iterating fast. New features were being released every week or so. And we wanted to do a reality check: are we even building for what users want? We decided to adopt a formative sprint model. Formative research is simply when the product is taking form, usually before production. In fast-moving times, a sprint model allows you to test ideas rapidly. It's an optimized way to move faster and make better decisions.

[00:03:08] We designed this research to adapt quickly. COVID-19 had changed our users. People's behaviors and preferences had shifted. And it's not that people's behavior just shifted over last year. Even as we progressed through 2020 and people settled into the new normal, their behaviors kept shifting, and so did our findings. Things which were relevant three months before were not anymore. And things which we were blindsided to emerged as new insights.

An insight: speakers became more important

[00:03:34] Let me give you a snapshot of an insight that emerged to help us adapt our product to the new world. That's where you can see how impactful this could really be. One thing we found was that speakers became more important, as people wanted to be attentive on a virtual meeting only for specific slots. They were already overloaded with online meetings. They were working from home, juggling responsibilities, and wanted to devote their time and attention to only the speakers who mattered most to them, who were most relevant to them.

[00:04:12] While attending an event from home, your kid may have an online class or a pet might need care, or it may be the time to cook lunch. Office work may demand concentration as well. Multiple speakers in an event competed for attendees' limited attention. So the right information on speakers became the deciding factor for attendees to be attentive for specific slots. We didn't use to have a speaker section and details on the events page, but now we do. We show the speakers below the event details: their designations, number of followers, etc.

[00:04:51] This is a great example of an insight which was very specific to the changing times, and wasn't there before COVID happened. Earlier, what was more important to people was to know more about the industry they work in, to meet new people, to expand their knowledge on a topic dear to them. But speakers weren't the first thing they wanted to know about earlier. This was something new for us.

One sprint cycle, week by week

[00:05:17] Let me take you through one of these formative sprint cycles, so you can understand better how this works, and hopefully can set up such a sprint at your own company. I designed these sprints bi-weekly. I started with week zero to schedule for week two. Scheduling takes its own time, anywhere from two to three weeks, even if you do it yourself or you have a research operations team helping you. I'll tell you how you get participants, but let's go through a cycle first.

[00:05:50] In week one, I work on the artifacts. I seek my stakeholders' inputs on research questions and assumptions as part of the research brief. I maybe prepare it one day, and the next day I share it with everyone to input and comment on. I prepare a discussion guide on Wednesday, and then again pass it through my stakeholders on Thursday. I like to keep my Fridays free.

[00:06:11] In week two, I hit the field running, talking to approximately six users over a period of three days. It's field day one, field day two, and field day three. On Thursdays, I close the round with a top line report. But as you can see, it takes some meticulous planning beforehand. How can you make it all come together seamlessly?

Researching your own team first

[00:06:39] The first thing I do is I start by researching my own team. I figure out the product stage they are in, what answers would help them make product decisions, and at what time these answers would have the most impact. So I get clarity on what we are confident about, and what the open questions are. This is the most important thing, because sometimes I feel I have to have all the answers, but that is not the case. I can always bounce the open questions back to my stakeholders and get clarity.

[00:07:08] As an example, you may be confident about the business context which prompted this research, or the research goals, but you may not know the type of audience to cover for this research. Do I talk to people who are existing event attendees, or those who are prospective attendees? Should they be users of a particular feature of the product? What audience do we even want to develop this for, the features we are working on? And the stakeholders may be split on this. Some may want a bigger audience size, with multiple variables, whereas some may be focused on a niche segment. So you bring in an even more important question: which type of audience to exclude?

Bulk recruits

[00:07:53] Once you get clarity on the audience or open questions, the best way to get these people repeatedly every other week is by a bulk recruit. A bulk recruit is nothing but a list, a combined list, where you shortlist a pool of participants to recruit from on an ongoing basis. When your team is designing and iterating fast, having such a pool of prospective participants will help you plan a scheduling calendar, which scheduling folks can take care of while you take care of the other important stuff.

[00:08:27] One beautiful thing about a bulk recruit is that you can consider a lot of variables while creating this pool. And then based on the feature you are testing, you can handpick the ones which are closest to the feature at hand. What are the different types of events? What are the different types of attendees? Even different types of organizers, like we discussed. Another nice thing is that you can refresh a bulk recruit after a quarter or so. You may have utilized a lot of participants from it during your bi-weekly tests. You may have added variables. Just add those and you have a new list for yourself for the next three months or so.

Involving stakeholders in the discussion guide

[00:09:06] For the discussion guide, one rule of thumb that I always advocate to UX researchers is to involve their stakeholders in the research process as quickly and deeply as possible. Become a team rather than an individual working on a research project. I recommend any researcher look at a research project not as their own, but as the whole team's. It's best to gather research questions from your stakeholders so your research can answer direct business questions. Your research will have maximum impact that way.

[00:09:40] I typically do it by a scoping workshop. Right after I have had my kickoff, I would invite my stakeholders from various functions to fill out a spreadsheet, where I would have primary research questions, secondary probes, and maybe priority of these questions, because we can only cover so much in one of these cycles.

Moderated and unmoderated sessions

[00:10:04] When it comes to executing the interviews, the best way is via moderated sessions. A moderated session allows you to pick the brain of your participants where you feel they are giving something worthy of interest. You can do them through a video conferencing tool these days; there are a ton of them available in the market. That's the closest we can come to one-on-one sessions.

[00:10:26] Another thing that I tried for these bi-weekly tests, and that worked out super nicely for me, was unmoderated testing, especially when the team was running short on time. Unmoderated testing is done via a tool like UserZoom, as you might know. These sessions run on their own: you just set them up with a flow and then let the participant run through them on their own. You can also set conditions. That is another beautiful part of this unmoderated test: you let it fragment on different trajectories based on user responses. They recently went to an event, or they went to an event six months back, and then you could pick their brain based on those fragmented questions.

[00:11:07] It's a challenge to do it marvelously in the first go, I would say that. And it may require some learning, but since there is no need to schedule a participant, it's much faster. You can just launch a study and receive results within just a few hours, or overnight if you are looking for a global recruit.

Debriefs and watch parties

[00:11:28] Finally, when it comes to interpreting the insights, which is the most challenging and time-consuming part for a lot of researchers, I recommend keeping it simple. I focus on providing answers to business questions, and I would emphasize the importance of the debrief here. I usually share a debrief template with my observers beforehand, asking them to jot down their key observations. And afterwards, I simply hold a debriefing immediately after the session, structured around the observers' informal notes. I have found it to be the best source of preliminary findings.

[00:12:04] Your stakeholders have more business context than you, and they have a handle on features that will go into production sooner rather than later. Some of the best insights have come from them during my sprints. I usually take these findings forward with a little expansion and share them over email or as a top line report. When you are running studies at a bi-weekly or tri-weekly cadence to move rapidly, this is your best option.

[00:12:31] Watch parties are another thing I tried with my stakeholders to make them feel more involved in the insight synthesis process. Especially with unmoderated sessions, they work best. What I would do is open up the tool itself, like UserZoom, and show them two to three videos, each 15 to 20 minutes long. I believe that no research report can deliver the same jolt of insight as when the team actively observes some user sessions.

[00:13:01] What I noticed was that they start getting into discussions as soon as they have watched the first session. They start discussing what they should be changing, brainstorming or rethinking. They start building consensus right then and there. That's exactly what you want. You want your team to come to consensus faster based on research findings.

Wrapping up

[00:13:22] That's the process that I followed to design the formative sprints. They were a great help in the changing times. They just require some meticulous planning on recruiting, scheduling and partnering with stakeholders. But I would say they are a great experiment to try and see where it takes a product.

[00:13:40] To sum it up, I would leave you with this double diamond diagram as a reminder that formative is when the product is taking form, usually before production. In changing times, your product is ever evolving, befitting itself to changing user behaviors. Understanding those behaviors and how they change over time could just be the key to unlocking usability and usefulness from your product. Thank you.