Blitz Research: More Insights in Less Time

Kevin Philpott at UXDX Community: USA Series: Design Process. Video: https://www.youtube.com/watch?v=GcF8IJcuFXU

Readable transcript: edited from the recording's captions for readability (fillers and false starts removed, punctuation and section headings added). Wording is the speaker's own. Timestamps are positions in the video. Names marked [?] could not be verified against the audio.

What blitz research is and what you'll learn

[00:00:00] Kevin: Hey folks. Today we're going to talk about blitz research, which is really just a fancy phrase for how you can do research faster, which of course we always want: to do everything faster. I'm going to go through the five areas that I found helpful in terms of cutting down a research timeline.

[00:00:23] As a bit of background, my name is Kevin Philpott, head of UX at Embroker Insurance. I've worked in design leadership and done that for various different organizations like GEICO, Gnomes [?] and IBM. I'm on a number of tech councils and UX councils, work regularly published on Forbes, and on George Washington University and Richmond University advisory groups. I'm not teaching at Berkeley. I did read a book that mentioned Berkeley once, but that's about as close as I get there to Jim, unfortunately. You might have heard me doing a couple of talks in a bunch of various different places, or maybe annoying you with a post on LinkedIn that got shared with you.

[00:01:14] Let's get right into what we're actually going to learn, what you're going to get out of this. I'm going to talk about the value of having an alignment meeting when you're doing your research. We're going to talk about how you can look at existing data to speed up how you do your research: do we have some information on this already? We're going to talk about unmoderated research methods. We're going to talk about spending less time designing. And I'm going to share a couple of what I would call project management power phrases, which sounds very fancy all together as I read that out loud, but it's some things that you can say in meetings that can help move things on a little bit.

[00:01:54] Caveat: there's no silver bullet. I'm not going to stand up here today and say that if you need to do a project that requires research, and it actually requires two days' worth of research, you can cut that down to one hour. You probably cannot. But this is a bit of a discussion, thinking about the context that we talked about earlier, where you have all of these different actors in your organization, engineers, product, multiple projects, and you're juggling different timelines. Sometimes you just don't have the exact amount of time that you would like to do your research, and some research is better than none.

[00:02:39] Sometimes you have to get creative, and these are some things that I've used myself over time that have helped me cut down the time that I spend on research. Sometimes when somebody asks you to do some research in an amount of time that isn't possible, you will have to say no to that. There is not a one-size-fits-all solution for this, of course. But with that caveat, let's go into our first item, which is having an alignment meeting.

The alignment meeting

[00:03:08] This might seem like more time, but I can tell you from personal experience that 30 minutes of an alignment meeting can save you 30 hours or more later on. What are you doing in an alignment meeting? You're setting expectations with all the various stakeholders that would be interested in this piece of research about what you are trying to achieve and how much time you actually have to do it. That way you don't get a bunch of scope creep and additional questions that come in at the end. You've set that expectation that maybe we've only got two or three days to do this.

[00:03:45] You can even set up the expectation that folks are not going to get a super nice, well-designed PowerPoint presentation with lots of videos spliced together. It can be a verbal readout if you need to do it faster. It can be just bullet points. So long as your group know that's coming, that's often quite acceptable. You can waste a lot of time documenting your research and not moving towards the outcomes for the business that you're actually trying to move towards.

[00:04:18] You want to bring in a representative from each stakeholder group that will be viewing or is interested in your research. Part of that is just that it makes it better from an integrative thinking perspective. What that really means is you are, for example, bringing in sales and seeing what they really want to get out of this. You're bringing in service and seeing what they really want to get out of this. You're thinking about your research from the customer's side and seeing what they really want to get out of this.

[00:04:45] When you go through that exercise, sometimes you find these really nice synergies, and you're really trying to maximize the value of the piece of research you're doing across all those different stakeholders. You're not going to be able to please everyone all of the time, but if you go through that exercise, you often get out in front of "this person wanted this from the research", instead of getting that at the end when you're presenting the research, or when you're halfway through.

[00:05:13] It's helpful to share the goal of the research. You can send it in writing afterwards if you want, with a summary of the alignment meeting, to make sure that everyone remembers once you get to the end what was agreed to.

[00:05:27] It's very helpful to have a business case. The reason it's helpful to have a business case, even if you are a UX designer, is that you don't want your research to get reprioritized or canceled in the middle of it. That is of course going to be detrimental to your timeline. Even if you're in UX and you're sitting there thinking, "Oh man, how am I going to put together a business case?", or maybe you're in engineering and you're thinking the same thing, just work with somebody else. You don't have to do it yourself. Somebody is requesting the type of work related to this research. Work with them, and see if you can put together a business case about why you are interested in this area.

[00:06:05] It really helps to try and quantify the value whenever you can, so that if it comes up, you have that. Many, many projects in many, many organizations don't actually have that quantified value, so you're just increasing the chances that it's not going to get canceled.

[00:06:22] In this alignment meeting, you will probably have a lot of folks adding different things that they want, and if you have a short amount of time, you probably can't do that. In certain cases you can just say, "Yep, that's a great idea, let's add that to our research backlog for the next iteration." Because you simply can't add in everything else on a short timeline. You don't necessarily want to tell people no, because you may be able to do it in the future, but you want to talk it away as something that we'll look at later on: we'll assess the value of it and we'll see if we can prioritize it.

Look at the data you already have

[00:06:56] All right, you've had your alignment meeting. Do you even really need to go and recruit people for your research? Maybe you already have the data that you need. Recruitment can take time, so you can check and get creative about the data sources that you can consider that might answer your research question.

[00:07:19] It's surprising how many folks really don't know what data is available in other departments in the organization. Maybe not surprising: these are big behemoths of organizations in some places. But it's worth asking about things like recorded calls, notes in customer relationship management systems like Salesforce, or whether there is an inbox with customer emails. One of the really nice things about these types of things is that when you set up, say, a user interview, you're really artificially creating that situation. You're coming up with questions, and you're trying to get people to imagine what they were doing. But these are in-the-moment pieces of data. Especially recorded calls: you can get some really great insights from those, I've found.

[00:08:09] Social media: if you're looking for opinions, you can use tools like Social Searcher or uvrx [?], and you can search to see what's being said about your company, if that's something that you're interested in. Or your company plus a keyword, if you're interested in perspectives about a certain thing that your company does.

[00:08:29] You can also look at session recordings. If you have tools set up like Mouseflow or Datadog (there's a whole host of them), you have tools that can record what users do online. You may just want to filter down from those recordings into key areas that you're interested in, and you might not need to do usability tests. You might be able to observe those recordings and see from them what the problem area is.

[00:08:59] Review websites are a good one as well, if you're trying to see, for example, in a competitive analysis, not just what your competitor has, but what people think about what your competitor has. You can go to review websites: Trustpilot, Yelp, G2.com. And you can look at your own company as well. What are you finding? Are there recurring patterns and themes coming up about where you could improve your experience?

Set yourself up for the future, and focus

[00:09:29] A lot of the things that I'm talking about here, you can just do, and the information is there. But you can also be pretty proactive here if you want to set yourself up for success in the future. If you have a research archive, use it, look at it, search through it. If you don't have one, set one up. Dovetail I've found to be a great tool for this. It really is amazing how much research gets done because people weren't able to find a previous test, or it gets lost in some set of folders somewhere. Being able to store and have that information, so you don't have to repeat research, can be quite helpful.

[00:10:13] Feedback forms: if you can set those up on your website, you may then be proactively collecting some data around different parts of your digital experience that you can go to afterwards, instead of needing to start from scratch when a need arises.

[00:10:32] Quantitative data sources: marketing surveys. Your company may have some of those. You may Google online for third-party surveys related to the topic that you're interested in that might give you some helpful information.

[00:10:49] And Jim talked about this when he talked about focus. You might not need to research everything. A really good example of this is you want to improve your digital experience. You might just want to go to your web analytics tool and look for drop-off in your funnel, in your experience flow. Using the same example, the shopping cart or a shopping flow, maybe you find that on a certain page you've got the majority of the drop-off. So you're not researching everything. You may be able to do a much more focused piece of research. You're always trying to descale the amount of research that you are doing to be just enough to make the decision that you want to make. Again, that fits very tightly with some of the stuff that Jim was talking about.

Unmoderated usability tests

[00:11:43] All right. We talked about recruitment of research participants sometimes taking time, and we're going to talk a little bit more about unmoderated methods. Unmoderated usability tests are usability tests where you typically buy a platform like UserTesting.com, or any number of platforms out there. There's tons of them. They often have a pool of participants. You can set up a test, someone will pick up that test, and you have written the test script in advance, and they go through that themselves.

[00:12:21] There are a number of advantages to this. First off, if you're doing moderated tests where you have to talk the participant through, you have to do the calendar scheduling and figure it out, whereas this happens on its own once you set it up when you do remote unmoderated usability testing. Tests can run in parallel. Depending on the type of participants and the type of testing that you're doing, it could easily happen, if you've got a very generic audience, that you have multiple tests going on at the same time and uploaded in a matter of a couple of minutes. Usually I would say, at least in my experience, for most tests, if you've got a general audience, you want to give it a day or two days.

[00:13:13] I'd also say if you're under time pressure, over-recruit by two users in these tests. There's a lot of professional testers out there at the moment, people that are just faking the tests to get paid the money. If you're really under time pressure, you want to account for that a little bit, so you don't have to get to the end and then redo the tests and launch it again. If you really care about speed, you might want to over-recruit a little bit for that.

[00:13:40] Keep it simple. One of my favorite usability tests is to start people on Google or on a blank page. A blank page is better, because you're not leading the folks as to what search method to use or where to start, and you just ask them, "Go ahead and do X." One of the benefits of an unmoderated remote usability test is that you can't interfere with that test. You can't ask questions like you could in a moderated session, which might give you different behaviors than if people just went and did it themselves.

[00:14:20] Another really great thing about this type of test is that often people focus on the experience of their products in the context of just their own product. They'll start people in their own quote flow, for example. But really, some of the best information I've found is when you start people and you see where they shop and who they look at, even before they get to your experience. What are the things that they're thinking about in the broader context before they even get to you? This is a super simple usability test, and it's super, super valuable. It's a good example of something that you can set up, run, and potentially get feedback on within a day or two, and that of course is helpful and easier than taking potentially a week or two weeks to recruit people to come in and do some moderated testing.

[00:15:14] What I've also found with unmoderated tests, if I want to be really, really efficient, is that maybe I'm looking for a certain type of experience, people having a certain problem. You can add a little post-test question to say something like, "Briefly describe what you liked least about your experience," or "what you liked most about your experience." If you're running a bunch of these tests, that can help you identify, without necessarily watching all of the videos, which ones you want to look at. Now, there's a big difference between what people say and what people do, so obviously the ideal scenario is that you actually watch the videos that you are putting out there. But if you're under extreme time pressure, this might at least help you prioritize which ones you want to look at first.

Unmoderated interviews and niche recruitment

[00:16:08] Unmoderated interviews: with a lot of these usability testing platforms as well, you can just change it up a little bit and basically get unmoderated interviews. Here's an example from TryMyUI; you could do this with a bunch of other tools, though. I might start this out with the scenario: "We will begin this session on a blank page, and you will be asked a series of questions about [insert your experience here]." Then: "Think about a recent time when you did this experience. Please describe that process from start to finish in detail. What did you like about that? What did you dislike? What would you like to have changed about that experience? Is there anything else you might want to add?" That, again, is a super simple test around an experience that you might want to explore, and you can launch that unmoderated and get those results back potentially very quickly.

[00:17:06] The other thing that I'll call out here is I've used User Interviews (userinterviews.com) for recruitment when I needed to go for a more niche audience. Some of these platforms like UserTesting.com or TryMyUI tend to have quite generic users, a general population. If you're trying to go after, I don't know, doctors or small business owners, you might need to go for a more targeted approach. I found User Interviews to be good for that. You can set up your screener. They're really pretty quick; you might get results back in a day or two for that. That can be an option if your need is a little bit more niche.

[00:17:50] If you have very niche needs, that does tend to take you longer to get the recruit set up, and it can cost you more. There's of course all sorts of different ways that you can try and get around that, like maybe going to conferences and trying to test people at those conferences. But it is certainly a more difficult task if you have a very niche need.

[00:18:12] One thing I'll also say is if you are exploring different partners for recruitment, a really good question to ask them is if they have recruited this type of user before. It is not ideal to enter a relationship with a recruitment provider where it's their first time trying to figure out how to get these people. That can of course extend your timeline. All right, so that's unmoderated interviews.

Less time designing

[00:18:42] Looks like a mistake. All right: less time designing. You could test with low-fidelity wireframes for certain concepts. You don't need to have everything built out to the nth degree of fidelity. Sometimes you can get some really good feedback with that.

[00:19:00] Another thing that you can do (again, this requires you to be a little bit more proactive) is having design systems. You might not have to design everything out in high fidelity, because you already have the components on your website. That's having a design system coded, which is what I was just talking about, or even just a whole bunch of templates in Figma or whatever tool you're using, so that you can copy and paste some of these bits and pieces and just edit them as you go. Again, that can save you a lot of time. You have to build that up, of course, but it's much better than designing from scratch.

Project management power phrases

[00:19:46] Last one here. I'm going to talk about some phrases you can use to help you move things along. Instead of saying, "When can you, stakeholder, get me this piece of information that I need for my research, or make a decision?", ask "How soon can you?", because that puts the emphasis on: this is actually urgent, I need this, give me the earliest possible time that I can get it.

[00:20:16] Sometimes also you'll be in a meeting and you'll be trying to gain consensus. A phrase that I've found super helpful is: "Based on what we have observed" (you might state what those observations are), "I hypothesize we should trial X" (you make the suggestion on how to move forward), "unless there are any violent objections." There are actually a lot of things going on here that make this an effective way to communicate.

[00:20:41] First, you're starting with "based on what we have observed": the evidence. You're now not necessarily introducing your own opinion into this. "I hypothesize": you're not saying, "Hey, this is definitely the best way to go." It's just a hypothesis to test. Sometimes when you're very definitive, people get defensive about that, and it can start a whole conversation. The next thing is you're putting forward a path of action, and then others can chime in on that. You're not waiting for the group to come up with something. And the last thing is "unless there are any violent objections". There'll always be little things that could be done better, but you're saying, "Hey group, this is what we're going with, unless there are any big objections," and that's often a good way to move the group forward.

[00:21:35] Another one is just making sure you set those expectations and time frames: "As a next step, we'll meet with whoever by Y date to determine Z." Make sure you're setting that expectation for the next step. Too often people leave meetings and it's like, "Oh man, we should have said what we're going to do next when we were in that meeting." You can keep up a nice cadence each time of always setting the next deadline, to keep the project moving forward.

[00:22:03] The last one: you will all at some point run into the scenario where you're looking for some confirmation or feedback online. It could be through Slack, it could be through your emails, and there's a group of different people. When you ask for that feedback, obviously you want to put in a date. Sometimes if you don't put in a date, it can just drag on and drag on. But even sometimes when you put in a date, if you're not really assertive about the timeline, it'll drag on, and people do other things and they don't respond.

[00:22:40] If you're in a group chat, you can say, "Please review this thing by this date and let me know if you object before we proceed." That means if you get a non-response by that date, you can move forward. Are there going to be people that get upset with that? Yes, of course there are. Again, no silver bullet here, but it's often a much more effective way than letting decisions sit out there and not being able to move your project forward.

[00:23:06] That is all the stuff that I wanted to run through with you today. I'm on LinkedIn; again, you might have been annoyed by some of my posts in the past. If you search Kevin Philpott, which is a fairly unique name, you will find me for sure. Feel free to check me out, and if you have questions, feel free to connect with me and ask them. This has been absolutely delightful. I'm at the end, and I'm going to hand back over.

Q&A

[00:23:40] Host: Great, thank you very much, Kevin. Another great presentation for tonight. And don't worry, I think LinkedIn is fair game for everybody to send out their posts, so I wouldn't be worried if you're annoying anybody.

[00:23:54] Kevin: Oh, I'm not. Just to be super clear about that, absolutely not worried. [Laughter]

[00:24:03] Host: Great. There's a couple of things that came up as you were going through your presentation that I thought it'd be good to ask. The first thing that you started with, and I thought it was a good point, is the alignment meeting. You mentioned one thing that I thought was interesting there, which was that you're going to get people giving their pet ideas, or wanting to get whatever they can put in. I was a little confused about whether at that point you have a very clear objective, or is this more of a discovery of what you are going to research? And if it's the latter, how do you effectively say no to people, or are you taking everything on board?

[00:24:46] Kevin: Ideally you have some sense of what you want to do, or what the business value of this request is, going in. One thing that is super helpful as a designer in general is that anytime somebody comes with a request, the business value is accompanied with it. That will filter out a lot of requests. And then when you go into the meeting, you've got a very specific thing that you would like to do. It is of course more challenging if you get into that and it's super exploratory, and you have to direct the conversation. I've done those as well.

[00:25:22] I think what you have to do in those types of situations is listen to the opinions and then suggest, based on that, what you think the research should be, and try to corral that conversation into "this is what we would like to achieve". If there are other things that come up, you might decide to prioritize those, stack-rank them, and say, "Cool, first what we're going to do is look at how we might understand the most important thing." Then you can deflect some of those questions: "Okay, that's good, let's get to that after we've figured out how we want to do this," or "let's put this in our backlog." But it's certainly more difficult, and it's an area that you practice over the years.

[00:26:05] Host: Okay, I'm just hearing backlog, prioritization, stack rank right here. I wonder, do we get Jim back in here to suggest some quick experiments?

[00:26:15] The other one, moving on to data: one thing I'm always a bit fearful of relying on. You gave some examples there of people calling in to a call center, or maybe some CRM. The negative feedback tends to outweigh the positive, in that when people are happy they feel less of a need. Some people do, and it's fantastic when you receive that positive feedback, but the majority of the time it's people who are negative. Do you feel that maybe you're getting biased in your opinions, because 99% of people are quite happy and then you've got this vocal 1%?

[00:26:57] Kevin: I think, bigger picture, you're always trying to triangulate your research methods. If you have information from surveys that can suggest what the overall level of satisfaction is, then you can go into the research understanding and knowing that. The other thing is that often you're going in and you want to understand more about a phenomenon, or more about opportunities to improve. And you've done a bit of research, hopefully, so after a little bit of time you can tell: is this an outlier, or is this something that comes up pretty frequently?

[00:27:33] It can be helpful to understand that single case in depth and see if you see patterns across, as opposed to just one particular instance of it. And even if it's not a massive problem for other users, it could still end up being a benefit or a delighter for them. I think you just have to go into the research with the right frame of mind in terms of how you're actually going to interpret or communicate that data afterwards, to be responsible about that.

[00:28:04] Host: Great. The last thing is more a comment than a question, but there was a whole heap of different tools that you mentioned throughout. Is there something that you would recommend for people who don't have access to those tools, around getting budget or getting sign-off and approval? Is there a business case there? How would you suggest people move forward?

[00:28:30] Kevin: Those tools can be incredibly expensive, UserTesting.com for example, but there are cheaper versions of those tools. That's one way you can approach it. If you need to build the business case as to why those tools can be important, sometimes what you unfortunately have to do is start without those tools and show how long it takes to do this on your own. You can calculate the cost over time of how much it's taking you as a designer to do this type of stuff. And also, from a business perspective, it's deeply unsatisfying each time to have to wait potentially two to three weeks to get insights on your stuff, and that's if you're really planning it out, if you're doing that type of recruitment manually.

[00:29:26] One thing to do is to take what you're currently doing, record the cost of it, record the time it takes, record your time as a cost, and try to make the business case that way. Because some of these tools, again, are not all super, super expensive, and you can sometimes put down on paper, "Hey look, we could pay for this, or I can keep doing this," and it almost makes the decision for itself.

[00:29:54] Host: Brilliant. Well, thank you very much. We've unfortunately run out of time, but again, I really enjoyed your presentation, and I hope everybody watching took a lot of value. Thank you very much.

[00:30:07] Kevin: Love it. Thanks very much, appreciate it.