Make user research agile

Danwen Wu at UXDX Community: APAC Online Series. Video: https://youtu.be/PAaRiARXuB8

Readable transcript: edited from the recording's captions for readability (fillers and false starts removed, punctuation and section headings added). Wording is the speaker's own. Timestamps are positions in the video. Names marked [?] could not be verified against the audio.

Introduction, and why agile matters for research

[00:00:00] So today I'm very excited to introduce this topic, how to make research agile. So yeah, let me start off with some introduction first. I'm a design manager at Vyond, of the product team, and I have 10 years' experience in user experience and user research related roles. I enjoy finding the problem and also discovering the insights and setting the motion toward transformation that helps our users. So yeah, this is about my background.

[00:00:35] And I also mentioned a bit about Vyond, so I will also quickly give some introduction about Vyond. So Vyond is an online platform which allows the user to build an animation just by dragging and dropping the asset to the stage and customizing look and feel just with a few clicks. So yeah, later I'm going to introduce a bit more about the product later. So yeah, this is my introduction.

[00:01:04] So for today's topic, I think many of you should already know what agile is, but just in case, allow me to quickly explain what agile is. So agile is an iterative approach to software development that helps the team to deliver value to their customer faster with fewer headaches. So instead of betting everything on a big launch, a team delivers work in small increments, and requirements and plans and results are evaluated continuously. So the team has a natural mechanism for responding to the changes quickly.

[00:01:48] And why do we have to consider agile when running research? So most of the time there are not enough research resources. I think many of you should already know. So in this case we have to plan resources carefully, to make sure that all projects can work on the right user problem.

[00:02:06] So by adopting this agile approach on user research, I see that there are three advantages. So the first advantage is we don't get stuck at only one point, instead we are feeding the insight to the next loop. And second is that we can also align with other teams by running the sprint together, so we are not really isolated. And third is we also can build a user friendly, or user oriented, culture in our team by continuously feeding the insight to our organization. So this is the three main reasons I feel that having agile when running the research is very important.

[00:02:51] So for today's agenda, I will be sharing four topics about how to run user research in an agile way. So the first, starting from the beginning, planning the project, and until the last, about how to share the insights. So the first topic, how to keep iterating the research approach and keep the research solution flexible. So this is the first one.

[00:03:21] And second topic, I'm going to talk about how to plan research resources and also empower research ability to other team members so that we can utilize our research resource better. And number three, creative. So in this part I'm going to talk about how to be creative when we experience the research resource limitation. And the last but not the least, storytelling. So how to storytell your user insights and make a bigger impact to your organization.

Iterate: launch the research when it is 80% done

[00:03:57] So let's find the first topic, iterate. So yeah, for the first one, iterate research. So during the research, it's also very simple, sometimes it feels like it's very similar to when you are building a product. So instead of just spending lots of time to prepare a perfect research plan, in our team we prefer to launch the research when it's 80% done. We prepare the high level question and then learn it through the research process.

[00:04:29] Sometimes in the middle of the research we found our interview questions were not specific enough, or instructions were not clear, or even we realized some of the research methods were not suitable for their projects. So in this case we research and keep moving forward.

[00:04:46] So some people might ask, can we really test research on our user? So let me quickly expand on how our team works. So we usually test an interview question, or a flow, or do the testing on our internal user first if it's possible. So the internal user including our internal employees who use our tool every day, or our regional reseller, or even our loyal or power users. They are very kind and willing to share their thoughts, and not only on the product itself but also they are willing to share the feedback on our research question or interview flow.

[00:05:32] So yeah, after a few interviews of the beta launch, then the research team, we will try to regroup again and review about our current research, including the question and flow and testing plan and even the research methodology, whether there is anything that needs to be modified. And then if everything's okay then we will do the official launch, to start to recruit our target user and run the testing. So by doing this approach we can make sure that we keep getting more insights and then try to update our research approach.

[00:06:20] Yeah. So there are a few things that we can try to pay more attention to during the initial interview or research. So including whether the beginning instruction is clear enough for our user to know. And then, within the time limitation, whether the number of questions and tests is enough for them to answer, or maybe too much, overwhelming. And during testing, whether the question or test is too difficult or too simple.

[00:06:51] And whether the question and test is clear enough for them. And yes, sometimes the question may not be really clear, and then we don't really want to define it as too much of a leading question, so in this case how can we better rephrase the questions? And then, during the flow, whether the flow is smooth enough, and then whether the testing, all the flow, aligns with their current scenario. So in this case, yeah, we don't really get stuck at one point, we keep feeding the insight to the next loop.

Break the big question into smaller questions

[00:07:24] And the second tip I would like to share is about avoiding working on too broad a question at the very beginning. So yeah, we try to break the big question into smaller questions. So yeah, by doing that, a big question breaks into a few different smaller questions, and we can see that has a few different advantages.

[00:07:50] So the good advantage by breaking the question, the first is about prioritizing. So by breaking that, we have to focus on the most important topic and prioritize the resource on the most urgent topic. So we cannot work on all the projects at once. And second is, after breaking that, we can share the project with other team members to work together, without holding the project on your own, so we have more resource to work together. And third, frequency. So by doing that, having more smaller questions, we can have more constant sharing, so other teams can work on the solution space in parallel, so they also get the feedback all the time.

[00:08:42] So I'm going to share one of the examples that we worked on recently about marketers. So we would like to know, how can a marketer, this job role, how can they create a video and how can Vyond help them to achieve their job role? So in this case the big question is, how can Vyond help the marketer? So breaking this big question, we have a few different smaller questions. So one research is about understanding the tool usage: why do they choose that specific tool? And another smaller question is, we would like to know about their background, including the working schedule and then what the environment is like, so where they work, and then how about, for example, the internet connection, everything about that, and then device usage.

[00:09:41] And then the third smaller question is whether they have any communication issue with their audience. And then the last is, how can we determine the marketer's purchase decision? What do they care about before they make the actual purchase decision? So yeah, this is one sample. So by breaking that big question into the smaller questions, we can focus on the more important topic, and we delegate the project to the different team members, and third is about we continuously share the insights, and also other people can get the feedback all the time.

[00:10:25] And number three, the third tip to make user research be more agile is to try to review all the topics again before starting the projects. So yeah, in this case, if you have a few different topics, I would suggest that you can try to categorize, or divide, into two different groups. One is a known issue, the other one is an unknown issue. So in this case we don't really need to run a full research from the very beginning, we can shift our focus on a different approach.

[00:11:00] For example, for the known issue we can run design validation, A/B testing, or preference testing, to understand which solution is better to resolve this issue. However, for the unknown issues, we can work more like exploratory research, to spend more time to understand their user mental model.

Empower: giving research ability to the team

[00:11:31] Okay. Yeah. So the first one is about iterate, and moving to the second part, empower. So I think in our team we don't really have many researchers. So by this situation we wish that we can empower our research skill to other team members. So other team members including, for example, designer, project manager, so that if they have any urgent task they can work individually by themselves.

[00:12:04] So how to achieve that? So I think for talking about empowering research ability to the team member, I think the first important thing is that we have to raise their curiosity and familiarity first. So we need to make sure that there is an easy way to receive the feedback. So as you can see, there are two examples. So we have our own Vyond community. In this community our power users share about their thoughts and any pain point they experience, and sometimes we even do some discussion with the user on the Vyond community.

[00:12:45] And for the right hand side, the right hand side is our release notes. So every release, we will write an article or video to introduce that new feature, but we will also make sure to include a one-minute survey[?] to collect their feedback. So after collecting lots of feedback, the feedback will be shared among the team so that we can build the empathy to our user problem among the entire team. So the first is, familiarize with our user first.

[00:13:16] And second is to give them the research power. So we built our internal research playbook. So we practice a few research methods by ourselves first and try to internalize the research method to fit our company's product development workflow. So as on the left hand side, you can see that we try to introduce a few different guidelines if you would like to select the suitable research methodology by considering your time, resources, and then whether qualitative or quantitative. So we give them the guideline about how to choose the suitable research method.

[00:13:58] And for the right hand side, we also try to clarify the job and responsibility of the team. For example, if you work on the research, who is the contact point you can try to find, and then including some of the to-do list, which can help our team members not to be afraid to do the research by themselves. So they can just quickly follow the guideline and workflow which are all listed on our research playbook.

Templates that safeguard research quality

[00:14:32] And then, third, it's also very important to keep our research quality. So we don't really just try to prepare the playbook, we also try to prepare a variety of templates. So in this case, the first we prepare is a research plan. So in the research plan we ask the people who want to conduct the research to consider who are your user, and then what is the known and unknown issue, and also about which research method you want to choose, and what is your action plan.

[00:15:11] And also we have, if you want to do some user testing, then how to better list out all the tests which you want your user to complete. And then if you want to do some user interview, here is the template you can consider to use, an interview question guide. We tell you what is a good and what is not a good question, so that you can just use this template to maintain the quality. So yeah, we try to safeguard the research quality by preparing a variety of templates.

[00:15:44] And another good advantage of this template, not only about safeguarding the quality but also we can change people's behavior. For example, in the past, every time when we do the insight sharing, then some of the team members, they propose the design solution too quickly, without considering what is the core value and what can we deliver to our user. So those types of critical or important information were always missed in the past research.

[00:16:17] So to solve this problem, we use this template as a reminder. So as you can see in this reminder, that we, as a user, before considering the solution, before considering how might we, we have to consider how can we turn the insights into the user goal, and then what is the value that our product can bring to them, and consider the solution as a later step. So by doing that, I feel like a template is a great tool to make sure that some of the key information cannot be missed during the presentation.

Workshops, customized service, and growing the method

[00:16:58] And the fourth technique is we also have a workshop often. So once we have any new workflow, as I just introduced earlier, or if we find any good practice from our designer, we share with the team and then learn together. So we explain it as an open door. So in the workshop everyone works together as knowledge sharing. So yeah, this is a good way for all the people gathering the knowledge and learning together.

[00:17:32] And number five, this is another good technique I would like to share with everyone: for our team we also try to provide more customized service based on team members' requests. Because some team members maybe they are new, but some team members have already run a few different research before. So in this case they don't really need the full support but they may need a bit of additional support.

[00:17:58] So in this case we break down the research flow and then solution into smaller tasks or steps. So as you can see, we try to list out, before the research plan, research during the research, and after research, and then by each step we estimate the working days of each step. And then other team members, if they need support, they can just select the service what they need only. So by doing that, we can help the team collaborate better and also provide a more flexible research service when planning the projects.

[00:18:40] And then number six, the last, how to empower our user. Definitely we need to grow their research method. So for the researcher in our team, the role is more like a consultant, or trying to handle more difficult research, and also at the same time they are the people who are sharing the knowledge. So we also continuously share the research knowledge to the team based on the current project type. So sometimes we have a more usability issue that needs to be fixed, sometimes we have some information architecture issue that needs to be fixed. So we will try to present the knowledge based on our current project needs.

Be creative: recruiting enterprise users

[00:19:21] Okay. Then moving to the third topic, be creative. So yeah, I think be creative is a very interesting topic. Sometimes being creative not only applies for the designer, but also we need to be creative as a researcher. And one of the stories I would like to share with everyone is how we recruit the users.

[00:19:45] So for Vyond we target enterprise users, but I think many of you should know that recruiting enterprise users is very difficult, because they are busy and sometimes they cannot really receive any gift card or incentive from our side. So in this case, instead of just waiting for their response passively, we initiated a few activities to provide a consulting service to help them to learn tips and tricks to build better videos.

[00:20:23] So each session is tailored for our enterprise needs based on the video project they are working on. So for example, right now they are working on a video for presenting information more clearly, so they will share the video with us and then we will review their project and give them some feedback. So by providing this consulting session service, actually it's really helping a lot for our enterprise user to share their time for us to interview, and also we can see the video they are working on and discover some of the issues. So I will say this is a very win-win situation for both sides.

[00:21:08] So yeah, from this story I feel like being more creative when you are designing how to reward and benefit our user interview is very important when you are trying to recruit your users.

[00:21:24] And for the consulting session, we offer a few different places for them to choose, but we ourselves always try to host a consulting session at our client's site. So yeah, if we can travel to our client's office, then we can see a lot more things that we couldn't see before, including for example some of the environment factors, including their internet connection, devices, their workplace and the environment, etc.

[00:21:54] So in this slide, this is one picture taken during one of the on-site visits. So we were surprised to see how our user breaks Vyond's limits by integrating Vyond along with other Adobe Suite tools, and also how to display it along with the studio. So it's a very impressive experience. So it's always great to have that chance to visit our customer's office.

[00:22:25] And also I think, yeah, the last is I would like to share the tool we use often to recruit the user and conduct the testing quickly. The tool is called usertesting.com. So during the COVID pandemic it became very difficult to travel, and sometimes we have some time difference issue. So our team, we use UserTesting to run the user interview, user testing, and even design preference testing. So it's quite efficient, if your company also experiences the same issue about you cannot really have enough time or have a chance to travel.

Storytelling: sharing insights in the organization

[00:23:07] Cool. Then we are moving to the last part, about storytelling. So yeah, we spend a lot of the time as a researcher to plan interviews and do the testing, so definitely we would like our insight to be more impactful and even be more memorable for our audience. So in this part I will tell you how can we share our insights in our organization.

[00:23:34] So the first, design discussion. So sometimes we notice that if we just have a pure research sharing only, some people feel like it's too official and the audience feel a bit of stress to share their ideas and their thoughts, which causes less interaction during the presentation. So our team, we want to make the sharing to be less official, and we want to share this to be more casual and more frequently.

[00:24:04] So we make the research insights and join the design discussion together. So which can boost, not only about sharing lots of insight for the design discussion, which can help the designer to boost the idea by considering and referencing a user pain point, but also for the research team side we can have a better idea, or even get more inspiration about what is our next project that we can work on.

[00:24:41] Yeah. And second, as I mentioned, casual. So we also found that showing evidence directly to the audience could be very convincing. So we try to just put some of the user clips, video, and highlight quotes. So which can help our audience be even more immersed, themselves, into the user scenario. So in this slide you can see that I present the user screen at the left hand side, and the right hand side is just a video with some quotes. So I feel like by using this tip, it's kind of not too official, it's a bit casual, but also at the same time it's very convincing to the other people to remember user insights.

[00:25:34] However, sometimes it's hard to always have a video or have the picture to present the user pain points, and especially sometimes when we travel to our client's office we are not allowed to take any photo or videos over there. So in this case, how can we better visualize the message and immerse our audience into the user scenario? So our solution is using our own tool, Vyond, to visualize the story that happened when there's no video available. So I'm going to quickly show you how we do that.

Visualizing a story when there is no video

[00:26:12] So this is one of the examples. This is the story that happened during a site visit. So as I mentioned earlier, sometimes we travel to our client's office to teach them how to better use Vyond. So at the time, when we arrived at the lecture room and then when we were about to log in to the public computer, and the reason why we use a public computer is because our personal devices were locked at the reception because of a security issue. So we are not allowed to use our own computer, we can only use a public computer.

[00:26:49] So when we were going to type our username and password to log in, suddenly the system triggered the two factor authentication. So it requires you to not only type your email but also asks you to log in to your email and click the link, type the link, in order to proceed. However, all of our devices were locked at the reception, so we were not allowed to use our own computer to open our email box, which made the situation get worse. So yeah, I think you guys might feel a bit confused if I just try to explain by plain text, and we don't really want this kind of thing to happen during our presentation.

[00:27:38] So how did we solve this issue? So yeah, we use our own tool. So this video, I'm going to quickly teach you guys how we can use Vyond to visualize this story.

[00:27:52] So at Vyond, you can just copy-paste the text and the script and find a few different keywords, and then at the left hand side we have a lot of the scene library. So you can just use the scene library to visualize, for example, the environment and then the scenario. And so as I mentioned earlier, we were in the lecture room, we were in the reception, we are doing a login, logout, login issue.

[00:28:22] And so after adding those, then we can even try to customize the character. So in this case I'm putting myself in the video, and then also adding another lecturer in this video. So once we add those characters in the video, actually we can make the presentation be more attached to our audience, because the people shown in the video are someone they are already familiar with.

[00:28:49] And then we try to customize the asset. So we mentioned that we are giving the lecture of Vyond, so we can put our own logo by uploading the UGC file, and also do some of the branding stuff by changing, for example, the color, and then putting some of the screenshots. So putting some of the key screens and also trying to change the color, and then let's see how it looks in the end.

[00:29:22] So after that we try to screen capture a few different views, so that we use the storyboard to tell them that our personal devices were locked at the reception, so that's why we cannot use our personal device. And when we were arriving at the lecture room we were very excited to share, however, when we were going to present how it works, the view shows the two factor authentication, and we had no idea how come we log in to our personal email. So all the people were waiting, so we felt very pressured. So in the end we had to call our colleague, to ask a colleague to log in to the email from their own.

[00:30:03] So by using this story it can better visualize the message. And what can be even better is, in the end I try to export it as a video. So let me quickly play the video to you, so you can see how we can turn the plain text into a more insightful story.

[00:30:58] Okay. So sorry, I only spent five minutes to make this video, so I haven't really had a chance to add some of the voiceover into the story. But anyway, the message I want to present to everyone is, when you have no assets or media available, sometimes using some of the tool to visualize your message can help your story to be more convincing, or to be more engaging to your audience.

Summary

[00:31:30] Cool. So here are all of my sharings. So yeah, to sum up, how to make research to be more agile. So the first, try to iterate and test your research so that you can get the insight quickly. And second, empower, so we can delegate our research work and focus on the high value projects. And third, be creative. So by being creative you can shorten your research time and even have better research results. And then last, storytelling. So by using a good storytelling way, we can bring the impact to our organization in an engaging way.

[00:32:15] Okay. So yeah, hope you guys enjoy my sharing, and hope my sharing also brings some of the inspiration when you guys are doing research. So thank you so much, and the bottom part is my personal information. Yeah, happy to receive if you have any question. Thank you.

Q&A

[00:32:35] Host: Thank you very much, Danwen. That was really, really insightful. I enjoyed that a lot because there was so much detail about how you work internally. So one thing, actually, and please, if anybody out there has any questions please do just add them on whichever platform you're using and I'll pass them through to Danwen. I would love to get a little bit of insight into how your team is structured. So when you were saying all of that work that you do in empower, are you a centralized kind of design team supporting multiple other teams, or how does the structure work? Who is responsible for the empower work and who are the users of that work?

[00:33:16] Danwen: Yeah. So our team, we have two different teams. So one is a design team, the other one is a research team. So as I mentioned earlier, as a researcher we are more like a supportive role, or consultant role, or bringing knowledge to the people. So if people have any question or any concern, if they want to run the research on their own, they can consult our team. So for example, at the very beginning, if they propose some of the research methodology, for example a user interview, we can tell them, for example, what kind you can use, we can help them to draft the high level question and also help them to review the detail. So you can take the research team more like a consultant side for them. And also the design team size is a lot bigger than the research team, so that's why the only thing we can do is like this.

[00:34:12] Danwen: And the second part is that sometimes we also try to shift our focus based on our company strategy. For example, these days our team focuses on AI related projects or growth related projects. So in this case we have to assign the researcher to this team as a full-time person to work together. So in this case the researcher, they are not a consultant any more, they are not a part-time person any more, they are full-time, so in this case this researcher will take care of all the research projects.

[00:34:49] Host: Brilliant.

[00:34:50] Danwen: So we are very flexible.

[00:34:52] Host: Yeah. The other thing that just jumped out to me is, particularly in the empower, so you've got the community, the playbooks, the templates, the services, the knowledge sharing, there's a lot of work there. So do you have anything like, if somebody was listening to this and going, wow, it's very scary to try and replicate all of that, is there any kind of guides or anything that people could use as a starting point on the internet that you could share?

[00:35:22] Danwen: Yeah, I have to be honest, actually some of the ideas were from UXDX. So I remember, yeah, you guys are also sharing some of the researchers, how they prepare those templates. I remember even like they prepare, for example, how to write the email to invite people to join the interview. So actually I also try to research a lot on the internet about how to write those kinds of information. But for our team, my role is more like to do some benchmark and then find the best practice.

[00:35:56] Danwen: However, I feel like it's still very important to internalize these resources. So I work closely with, for example, our operation team, and even HR team, and even our copywriter, to try, for example, how to make our message feel more Vyond, and to be more, for example, based on the project goal.

[00:36:20] Danwen: So what I would say will be, the first, try to find the information and resources, and UXDX is a great place that I always try to find. And second is, find the colleague who are also interested in this topic and then you share those content with them, and then they also try to internalize those content and customize for your own audience. So I will say that I also spend some time to prepare these resources, but I will make sure that every research project ends and I have something to carry to the next projects. So I think for the people, if you guys are working on more and more projects, you're going to have a lot more knowledge later.

[00:37:03] Host: Brilliant. All right, well thanks a million for sharing, and I hope everybody out there enjoyed it as much as I did. So once again, thank you very much.

[00:37:13] Danwen: Thank you so much. Thank you.