Beyond Interviews: UX Research Methods for Everyone

Nadine Piecha at UXDX EMEA. Video: https://youtu.be/eIlQQdocxNc

Readable transcript: edited from the recording's captions for readability (fillers and false starts removed, punctuation and section headings added). Wording is the speaker's own. Timestamps are positions in the video. Names marked [?] could not be verified against the audio.

Introduction and what we'll cover

[00:00:08] Thank you for the introduction, and hello, good afternoon everyone here and online. You already heard, my name is Nadine, I'm the head of UX research at FREENOW. I cannot hide it, my accent is German, but I live in beautiful Spain now, working for FREENOW. I have a background from different industries. I've worked in B2B, I've worked in startups, I've worked in billion dollar companies like eBay, like Oracle. So a little bit from here and there, and I'm bringing some of that background and some of those experiences here today for you.

[00:00:38] And I'm bringing something else. I don't have free t-shirts, I don't have free books, but I have five-euro vouchers for the first five people that speak up. This talk will be a bit of a forum style, so I want you to participate. I want to hear about your experience as well, or your struggles as well. The first five people who will speak into this cube later will get a voucher if you come afterwards to me, and you can ride through Dublin for five euro less.

[00:01:05] What are we going to talk about today? A variety of research methods, and how you all can get involved in applying different research methods beyond just interviews. And also how to overcome struggles, struggles that you might face in your company, something like, I don't speak the language of the user, people don't believe me, people don't listen to research, it's not clear what value it has. We'll talk about those obstacles and how to overcome them, and hopefully all of you will help me find solutions or give some tips to the rest of the audience.

[00:01:38] When I talk about user research, what do I mean by that? Probably that's very clear to all of us. We've heard the term a million times, there's lots of user research tools out there. I just want to make one point very clear: I'm talking about systematic investigation of users. Who are the users, what's their context, what are their pain points? And I'm specifically referring to user research in the traditional understanding of interviews, surveys, so reaching out and interacting with users. I'm very aware that there are data analysts and that data analysis also is a very important part of research, and I love working closely with data analysts, yet in the talk I'm referring to the more traditional user research.

Who is in the room

[00:02:21] I always like to feel the vibe a little bit in the room, sorry for the people online, but I would like to ask people in the room, raise your hands if you are a user researcher, just so that I see how many user researchers we have here in the room. Okay, quite a few. So I hope there's some inspiration also for you in here on what methods to use, or maybe also how to involve your stakeholders or your colleagues that are non-researchers into the conversations.

[00:02:49] Whom else do we have here? Do we have data analysts in the room, someone who works with data? One, not much more. I'd love to see more data faces in research talks as well, because I really think we are pretty close peers and our knowledge can complement quite nicely. So hopefully next time I'll see even more data experts here as well. Do we have designers, PMs, UX writers? Okay, now we're seeing more hands. Very happy to have you all here, because you're super crucial for research. Research is a team sport, and you learn nothing if it's only the researcher doing the research and coming back with the ultimate answers. You need to be part of it. You need to say what you want to learn, what decisions you want to make, and ideally also be part of the research, applying research methods yourself.

[00:03:35] So let's go into that first question, and from here on we can switch to the polls. Actually you can open your phones again and answer the polls that are coming up, and then hopefully we can see them on the screen as well. Think about your company, your team. Who typically does user research in your company? You should see the poll now, and then we can hopefully see the result on the screen as well.

[00:04:07] All righty, so far only researchers. Let's hope that picture changes a little bit. Okay, we can see more answers coming in. And it seems like we have quite a few lucky people here who have researchers in their teams, so we can push the research to the researchers. There are a lot of designers, and there are PMs and POs also doing research in the company.

[00:04:40] Do we have any designer in the room who's doing research themselves? Anyone who feels like sharing and getting the FREENOW voucher? What kind of research do you do? Maybe you have an example of something that was easy for you to do. Anyone wants to share, any designer, on what type of research you're doing? I don't see any hands. Help me out. Okay, there's one.

[00:05:12] Audience: Hi, yeah, I'm a designer who does research, because frankly we are only UX designers, so we don't have a research department, so we do it. A lot of the time when we get to do it, we can do it via job watchings or interviews, or direct conversations with clients in the end. That's how we mostly do our research, because it's hard for us in particular to use research tools in my position in particular.

[00:05:47] Nadine: Nice, thanks for sharing. Yeah, that's the most common scenario that I would probably also guess, that designers also know how to do usability testing, flows, testing designs, prototypes, and talking to users. Do we have someone else who wanted to share?

[00:06:05] Audience: Yeah, so again, similar, designer. So we typically do kind of qualitative stuff, but that is often down to interviews, or just getting sentiment analysis or stuff that is out there in the world.

[00:06:19] Nadine: Nice, thank you for sharing. And you were a designer as well? Yeah, exactly. Do we have product managers in the room as well who have done some of the research themselves? Anyone who can speak up? I see one hand up there with the...

[00:06:41] Audience: Yeah, so I'm a PM working at a startup, so we don't have a dedicated research team either, and I do most of the research in the company. We usually do usability testing, interviews, heat maps, and a lot of data analysis that I get from data people.

[00:06:57] Nadine: Nice, thank you for sharing. Yeah, I've been in a startup before as well, and I know that kind of feeling, like everyone has to do everything. You don't have the big team, you don't have a lot of resources, so everyone has to get hands on. Really cool that you're actually interacting with users yourself as well, and data analysis makes a lot of sense as well. Thanks for sharing. Anyone else, any other PM who wants to share maybe a recent experience, an example?

[00:07:26] All righty, thanks for sharing. What I've seen from my experience: designers quite often very hands-on involved when it comes to testing designs, prototypes, concepts, getting feedback on ideas, while I've seen PMs probably a bit more strong on data, on understanding maybe the business and the revenue, the KPIs that we want to impact. It doesn't have to be that way. Ideally both can do both, and everyone can be involved in all the different methodologies.

Which methods teams actually use

[00:07:58] We can go to the next poll then from here. So who does research, we have established: different roles, ideally actually all the roles in the company. Everyone who works on building products for users will be involved in one way or another also in user research. Next question: which methods were used for user research in the past? User interviews, online surveys, focus groups, co-creation, eye tracking, ethnography, or maybe you have other methods that I didn't even list here.

[00:08:31] All righty, we're getting the responses in. And the picture starts looking like what I expected to see here. Almost everyone does interviews, a lot of people do surveys, and then all the other methods, all the beautiful other tools that we have available, are not used very often.

[00:08:55] That makes a lot of sense, and that's what I do as well, what our team does as well. User interviews usually is the default method. It's the one that always is top of mind. It's the one where you can in a relatively easy and fast way learn a lot of things. Just keep in mind there's a lot of stuff you won't learn from user interviews.

[00:09:11] I have one experience that I can share already. I work for FREENOW, so we are testing, or we are investigating, not just the passengers, like maybe all of you can use the FREENOW app, but we're also investigating the drivers. I've spoken to a lot of drivers from Ireland and I never really realized how the car looks like, where the phones are, how many phones they have in the car, that there's a camera recording the car to the inside. I just didn't know because I hadn't seen that. And for that it's super important, that's the context that the users are in. That's super important to have in mind when we're designing for this small screen on the app. What's the real life, what's the context, what does it feel like? So for that, for example, guerrilla research, that's a really important method. It's a really important add-on to go into the field to see the context.

[00:10:01] So that much about the interviews. Online surveys also seen as a very useful tool. A lot of companies do a lot of those online surveys, and everyone thinks, oh yeah, it's a fast way to get quick feedback, I can learn fast, everyone can design a survey, everyone can send a survey. Just one piece of advice: be careful, because rubbish in means rubbish out, and you might not even notice. With surveys, with interviews you do, because users give you feedback, or they look confused, or they say I don't understand the question, can you repeat it, can you rephrase it. In an online survey no one will. They just answer whatever, you get numbers for whatever, and you take decisions on that, whatever that might not have been the right question. So if you have the chance to involve a researcher for online surveys, I'd really recommend it.

Focus groups, co-creation and the side effects of variety

[00:10:43] About the other methodologies, we seem to have someone who has experience with focus groups or co-creation. I think that would be super interesting to hear, if there's anyone here who wants to share their experience, maybe with a method that is not user interviews and not surveys. I see a hand up over there, very cool, I'm already excited.

[00:11:07] Audience: Hi. So I produce data products, well, my team produces data products, so more dashboard types of things. And I work for Imperial College, so we had some students involved from the computing department. We gave them their own data and then we said, what would help you learn, visualize this in a way that would help you learn. And then we took that, and we did focus groups with the students with their own data, and they helped run those focus groups as well, so kind of a big co-creative environment. And then we obviously put the most beneficial things into production. So that's how we've done it.

[00:11:53] Nadine: Very cool, thanks so much for sharing. That sounds super interesting, and that's even focus groups and co-creation in one, like a group session, a workshop session, where they can actually become creative and build something that makes sense. Maybe one important comment on the co-creation, because sometimes I get that feedback, a little bit of hesitation, like, are users really the best advisers on what to build? No, they are not, and we're not asking. When we do co-creations like this, we're not asking them to really build the product. We're asking them to interact, to think about it, and from what they prioritize, from how they think, from what they discuss, that's when we learn what's the mental model, what's the approach, what do they prioritize. And that informs our decisions. But in the end, obviously you as PMs, as designers, as engineers, you know better what is possible and what can be built. But yeah, super interesting tool.

[00:12:46] We've also done co-creation sessions. Actually someone from my team in Dublin has done co-creation sessions with drivers, FREENOW drivers, to understand how could we improve the interface of the app for drivers when they're not driving, when they want to learn how much money they made, or how many jobs they can accept tomorrow, or how much they achieved today. So that was super exciting as well, very interesting projects.

[00:13:11] And then as a side learning: whenever you introduce another method that is not just interviews and not just surveys, you also can see how the excitement in the whole company goes up, in the leadership team goes up, amongst all the stakeholders that you work with. The first time in a long time when we did focus groups, really with a one-way mirror and having lots of people on the other side discussing, end users on one side and on the other side the product teams, you could really feel how excited everyone was and how everyone showed up. While for interview number 500, not the same amount of people show up and listen. So even to increase customer centricity it sometimes helps. It's not just because you get better insights, but also because you get more attention overall.

[00:13:53] And then last but not least, using many methodologies from my experience also helps to keep the researchers engaged and keep the researchers growing, and give them new challenges and make them become better researchers. So yeah, for us it really works in FREENOW to use not just user interviews and online surveys, but really combine, and invent your own methodologies as well.

The obstacles poll: resources and budget

[00:14:16] There's one more poll that I have prepared, that is about the obstacles. I said in the beginning we also want to talk about the obstacles that you face. So now I want to know from you, what is it that makes research challenging in your teams? Is it the lack of expertise, so maybe no researchers, I think we already heard from someone, or researchers in the team? Resources, not enough researchers or not enough budget? The return on investment, the value of user research not being clear? Speed, research slows us down? Language barriers, teams don't speak the language of the users? Or leadership buy-in? What is it that makes you struggle?

[00:15:07] All righty, I guess we already have one winner: resources. So not enough researchers for everything we want to do, and not enough budget for everything we want to do. Anyone who is in that situation currently want to share a little bit about that? Anyone who gave that answer can share what's the situation, so we have an example to work with. We have a hand up already here.

[00:15:33] Audience: Hi. It feels like some of these issues are somewhat connected. Often you don't have enough resources because leadership aren't bought into the value that research brings, so therefore they don't understand the value of research and you can't convince them to bring in a researcher. I recently had an instance where we had a researcher leave, promised they would backfill. The manager, who was a non-UX person, was like, oh well, I went and spoke to the other product people and the engineers and stuff and they said they don't really need user research. And so you're going, oh, but I don't know if they know that they need it, even though they do. So there's a lot of that going on as well.

[00:16:21] Nadine: Yeah, thank you for sharing. And I'm smiling even though it's actually a sad story to hear about, they don't know that they really need researchers, or research being done. Thank you for sharing. So maybe anyone else has advice, what can we do if we focus really on that there's not enough resources, we need to backfill that position, or we don't have the budget, maybe we cannot incentivize or we cannot get that other tool. How do you overcome these obstacles? Anyone has any story you can share, any tip, any trick that helped you to overcome the obstacle of not enough resources?

[00:17:00] Okay, sharing the problem was easier than thinking about the solution. Learning taken. Do we have anyone with any idea? Yes, nice.

[00:17:11] Audience: It's not a very unique or very all-around solution, but what I found helped us a lot was actually creating personas. Because whenever we could get our hands on some users, we would talk to them, talk to them also about their co-workers, talk to them about the other people they knew who would use our product, and then we would scale that as personas, because we knew we wouldn't really get a lot of quantitative feedback. And this then helps us create better user stories, even if we can't always challenge them. And another thing we do is we talk to the people who get the tickets from the users. If we can make a survey, we can ask them what the problems are that the users are reporting, because a lot of the time we can also extrapolate some problems in our design from that.

[00:18:04] Nadine: Yeah, cool, thank you for sharing. That's very good advice. There's a lot of research already available, you just have to find it. Maybe it's not every time generating a new research study or running a new round of interviews, but you can look at what you already did in the past, look at the persona that is a nice summary of what you already learned and what you've already investigated in the past, or even look at support tickets, or I don't know what else could be available, app reviews or NPS comments that are coming in on a regular basis. So even if just accessing that, it doesn't cost you any extra money and you have learnings from users. Very cool, nice, thanks for sharing. Anyone else, how to overcome obstacles like not enough researchers or not enough budget available?

[00:18:56] Audience: Yeah, I believe if you don't have enough researchers you can outsource the study, if you have budget. And then if you have budget but not enough researchers, that's that. But if you have researchers but not enough budget, you can look at different types of research, such as test it on your own or with some internal participants. If they have more or less relevant experience, background, it is also helpful. You can look at some BKM[?] standard requirements that products should follow, usability heuristics, and you can check it by yourself, and it would be helpful as well.

[00:19:40] Nadine: Nice, yeah, thank you for sharing, both very good points. Outsourcing if you have the budget. There are certain types of methodologies or certain types of studies that are a bit easier to outsource. That's typically the bigger, more longer term studies. Things like online surveys are very good to outsource to people with expertise even outside of the company. When you actually want to do agile prototype testing, adjustments, maybe there it is a little bit more difficult to outsource. But yeah, outsourcing if you have budget but no researchers, definitely a good option. The other way around, have your researchers do low-cost research, like what you already shared, also any sort of guerrilla type research, or reaching out to friends and family if you have researchers but you cannot give incentives. There's so much the researchers can still do. Well, depending on the product, depending on the user segment obviously, but in my past, in my experience, I've done so many research studies without any budget. You just need to find creative ways. In our case now it's easy, we just go in a taxi and talk to the driver, for example, or I talk to my friends, to my family as well, about how they order taxis, and that doesn't cost me anything, neither time nor money.

Leadership buy-in and the value of research

[00:20:51] What do we have as number two? It's actually difficult to decide. Okay, so maybe to the point of what you also raised in the beginning, no, it's all a little bit connected. The leadership team doesn't really buy in, maybe the value of research is not clear, and then the leadership team doesn't act on the research. Any advice from anyone in the room on how we can solve that? What can you do to get leadership closer to the end user, to the user research? Any tips or anything that you've seen working for you?

[00:21:31] Silence in the room. No one works with senior leadership teams? All the researchers that had their hands up in the beginning, what are you doing?

[00:21:41] Audience: For me, I think just documenting the risks that we're having to accept by not doing the research, and conveying it in the language that those stakeholders do understand, and having that backlog of risks clearly documented and presented has helped me in the past. Often we would kick off projects by doing some sort of risk assumption mapping, so you'll have all of your risks mapped out, just giving it that severity and saying, okay, we could just go and deliver, but if we do, these are the risks that we're having to accept by moving forward. So kind of speaking in their language, because I know sometimes research can be kind of a trigger word, and it has helped my team in the past.

[00:22:27] Nadine: Yeah, thank you for sharing. So two very important points that you talked about. One is making very explicit and very clear what the risks are of not listening to users or not even doing research, and the other is adjusting to the language they speak. My advice also would be to build relationships and get to know these people. In our case, for example, the CEO offers lunch meetings to anyone, so I was like, I want to go on lunch with you, so just to talk and get to know the stakeholders or senior leadership team, to understand what are their concerns, what are their worries, what are the top of mind issues at the moment, so you can adjust to that a little bit. Very cool. Any other tip or trick from the audience? Sorry, we have more hands raised and only one mic coming.

[00:23:16] Audience: I think, again, similar to what was just said there, starting small and doing a very small kind of experiment, like a fake door or something like that, has been very useful for me in the past. The last company I was in, there was a round of circular arguments about which feature we should build first for, say, video sharing, so whether we needed video or screen sharing first. And by having a fake door put in there, that costs very little time on the development purpose, we were able to say conclusively, by doing this we know that user intent, we measured that, we don't need to build this right now. And that can get you some kind of trust. But again, just that establishing trust has worked for us in the past.

[00:24:00] Nadine: Yeah, nice, thank you for sharing. So starting small, and then also building some kind of case studies or examples that you can repetitively probably use also to make your point. No, this is one example where it worked. And if you can link it to return on investment and how many millions you helped save the company because research helped you find out something, even better. Did we have more hands raised? I think there's another one.

[00:24:32] Audience: I've been working as a product designer, and most of the time most of the stakeholders, their opinions need to be heard, and whenever we are going ahead with a couple of options, everyone has their own opinions and it's really hard to figure out which one to proceed with. So what I have done is I've created a couple of research hypotheses, and we'll create some null hypotheses, and we'll do some research around it, and we'll show it to them, like, this is the hypothesis, this is what's going to happen if we go ahead with this. And with the results the stakeholders will be convinced. So that's what I've done.

[00:25:08] Nadine: Nice, thank you for sharing. And while you were talking you were just reminding me of another trick that I've tried, I think I actually did it in FREENOW, where I also asked all the stakeholders, everyone with their opinions, okay, what do you think will happen, what do you think the user will respond? And then you show them, look, you don't know, they actually responded something else. Or we confirm what they already knew, but at least then we have data to prove it. And if you can prove them wrong, even better, it makes a very clear point of research being needed. Thank you for sharing.

Closing

[00:25:41] I guess the lights turning off slowly are a sign for us to close down little by little. I can hardly see the audience, so maybe I'm getting to the end. I hope we covered some of the obstacles that you actually experience in your everyday life. If you have more obstacles that you wanted to share or that you want to talk about, just find me around in the room or find me tomorrow and we can totally discuss all the obstacles. I think from all the different companies that I've worked in, I have quite a few obstacles that I went through and that I can talk about. Are there any questions? Because here I see the questions, here I see the ending slide.

[00:26:24] I guess it's time to wrap up the time on this stage. I heard I'm the last one speaking on this stage today, so we're closing on this stage for now. I was thinking, oh, I should pick my words wisely, and what else would you do than asking ChatGPT what can I do to close the day of the conference when I'm in Dublin. And ChatGPT was kind enough to help me, so I have prepared something that I have to cheat and read from here now. You've survived the talks, the coffee runs and the struggle to find an outlet for your phone. Remember, conferences are like Irish weather: unpredictable, sometimes intense, but always worth it for the experience. So let's take all the ideas we've gathered today and, like a good Irish pub, let them brew overnight. So hopefully see you tomorrow. That was ChatGPT, not me. I think the final closing words will happen on the other stage, so see you there, and then have fun tonight.