Q&A With Mackenzie Guinon & Carmen Broomes

Mackenzie Guinon, Carmen Broomes at UXDX Community: USA East. Video: https://www.youtube.com/watch?v=CFvhoKfYpjY

Readable transcript: edited from the recording's captions for readability (fillers and false starts removed, punctuation and section headings added). Wording is the speaker's own. Timestamps are positions in the video. Names marked [?] could not be verified against the audio.

When stakeholders say research isn't needed

[00:00:01] Rory: Thank you, Jovi [?], and welcome, Carmen and Mackenzie.

[00:00:06] Carmen: Hello, thank you.

[00:00:08] Rory: Excellent, great to have you along. I really enjoyed watching your talk, so thank you very much for sharing it with us. I'm a massive advocate, as you can probably guess, of investing more in creating that culture of customer focus. As you've seen the format, I'm just going to do a bit of rapid-fire Q&A with you. The first thing, and it's one of the challenges that I have, is that I don't think I've ever met somebody who says they don't understand the customer. So when they come along with something, they're like, "Oh, well, for this change we don't need to do any research. This is obvious, people need this. We can just skip straight ahead to development, we don't need to bother." Have you encountered that, and how do you deal with situations like that?

[00:01:00] Mackenzie: Do you want to start, Carmen?

[00:01:03] Carmen: Yes, I can. I think we've all experienced that at some point in time. What I like to do is ask a lot of questions. When I'm in meetings with stakeholders, I do a lot of asking: "What do we know about this? How did we arrive at this conclusion? Where did we get this information from?" What I've found is they actually come to that realization themselves, and I don't need to say it. So I think it's very important as UX researchers to stay curious, and then help our stakeholders arrive at that conclusion for themselves. I don't know, Mackenzie, if you want to add anything.

[00:01:45] Mackenzie: Yes, I'll second what you're saying: the curiosity and asking questions. I think it is hard. You brought up, Rory, that often what makes people good at their job is knowing the customer. So even to approach it in the way of "you don't know, and I do" is pretty much the battle mindset we get into. Having empathy is their bread and butter: "Why I get paid so much as a product manager is that when a customer has a question, I know the answer. I can help, I can drive you towards a solution."

[00:02:14] And to bring out that we come from a really different perspective: we aren't the experts on the product. "I don't know the things you know. What do you not know about the user? What have you ever seen that you weren't able to explain?" I think the list of myths and questions that product managers, engineers and designers have is very long. So if you can shift the conversation from your knowledge versus my knowledge to "I understand you know a lot that I don't know," and, like Carmen said, then ask them questions about how we know what they know, what we don't know, and what they wish they knew, I think you can really shift the energy of that conversation.

[00:02:54] Rory: I love that. I've heard somebody else describe it as saying, "Look, I might be asking stupid questions," so you're downplaying it so that the person doesn't feel threatened or anything. That's fantastic.

Fitting generative research into the first fifteen minutes

[00:03:08] Rory: I'm going to jump into some things you've mentioned, because I love the idea. A lot of companies invest in the usability testing side of things, but that generative research up front is often lacking. You mentioned that you tried to jump in and piggyback on your usability tests, and the first 15 minutes was an interview. What can you get out of 15 minutes? Whenever I've asked generative questions, it just opens up a rabbit hole and I want to keep digging and digging and digging. So how do you structure it to make sure that you're good with your 15 minutes?

[00:03:47] Mackenzie: I'll jump in, because I'll just say I have felt like an outsider sometimes in the research community. When I became a researcher, I loved warm-ups. I wanted to warm up, and some people would say, "Ask their favorite color, ask a softball." And I was like, "I want to ask you about something that leads us towards the conversation." So something relevant, but often starting with, and I talk to recruiters a lot, "How long have you been in your career, in your role at your current organization, and before?" Then we start to talk about their responsibilities, but they're already being a human and they're not on a call with Handshake.

[00:04:25] So when I landed at Handshake, and Carmen is my manager, which wasn't that long ago, and she believed in that too, I did backflips, because I think it is a research identity to believe that that is a way you can get away with 10 to 15 minutes of generative on the front of a usability test. That wasn't exactly an answer to the question, just my deep belief. I think the word you used that I would lean on there is structure. I end up asking those same warm-up questions for the most part, and I adjust based on the study, in almost a longitudinal way, over different studies and tons of participants.

[00:05:01] The first thing is, if I really want deep data, I use it in that longitudinal way. Part of what I'm doing is just getting people excited, and also recognizing there's a human on the other side of the line. I've had people tell me in the first 15 minutes about promotions, about personal things. Then we go into the debrief, and I always start my debrief with what stood out to someone, and it's frequently a human story that stands out, so that's what they bring up, no matter what the research objective was that we were there for in the first place. I don't know if that helps.

[00:05:35] Rory: Yes. It's almost like sales. In sales they always say the first thing to do is make a personal connection. Carmen, did you have something there?

[00:05:47] Carmen: Yes. I think what it also does is create this appetite for deeper insights. For example, you may hear something and you can't get everything you want out of it, and then what I've experienced is observers who are listening in on the session say, "Well, I really want to know more about this." Then we're able to say, "Okay, we can spin up a research project focused on that," especially if there are patterns and we start to hear it over and over again through multiple research sessions.

[00:06:19] I love how Mackenzie put it. This is something we both practice, not only here at Handshake but also at other companies: having a set of questions that we ask participants over several studies. What that does is you're able to start to collect insights, and you can pull those together into a report. And if you want to go deeper and gather deeper insights, and create a new study, you can do that from there.

How do you know when you have enough?

[00:06:55] Rory: Excellent. Scott has asked a question, which is the age-old question in discovery research: how do you know you have enough data or insights, and when do you stop?

[00:07:11] Mackenzie: Do you want to go first, Carmen?

[00:07:13] Carmen: Yes, that's great. One of the things that I did is I created a charter, a UX charter, and one of our values in that charter is continuously learning. That means that we are probably not going to learn everything in one study, but we're going to continue to learn, and as we go we'll gather deeper and richer insights. So one of the things I would say is always go back to your research questions and work closely with your stakeholders, because you want to make sure that you have buy-in from them, I should say.

[00:07:56] Because if they're not bought in, if it's "we don't have enough data, I'm not sure," then they're not going to use your insights, they're not going to use your results. So be collaborative with them and make sure that they're part of the process. And then also, we may not know everything, but identify what those things are, knowing that we are continuously learning. That's part of our process as researchers.

[00:08:24] Mackenzie: I love what you said there, Carmen. I feel like I've had that same conversation with a stakeholder: "Are we done yet? Are you sure of what we found?" "No, I'm not convinced." That can be fun. I definitely love Carmen's comment that we are continuously learning. Each study doesn't have to be the last study we do. And I remind myself that as a researcher I've been going down rabbit holes since I was a kid: why, why, why, why.

[00:08:52] But one of my early mentors in research taught me the term saturation: this feeling that you get in these interviews you're doing, where you start to have a sense of where people are going to go. I talk about that a lot with my product manager, engineering and design partners that I bring in with me. We're never going to know everything, we're never going to know every edge case, but when do we start to feel like we have a really deep understanding and awareness, and we're starting to gain a sensitivity, relating back to our talk, in these interviews?

[00:09:25] I'm at the tail end of eight interviews on a really, really complex topic. I didn't complete them all today, this week, but I finished one, we got off, and in the debrief people went, "I feel like I know what they're going to say." I don't want to overbuild their confidence, but I think that's a mark of saturation. You're starting to get a good sense, and you're really sitting in the seat, wearing the hat, of the people you're talking to.

[00:09:55] Rory: Excellent. I actually have a load more questions that I'd love to ask, but unfortunately we're out of time. Thank you very much, Carmen and Mackenzie, and thank you for the talk. If you haven't seen the talk, I highly recommend it to everybody out there. I'll be talking to you again shortly. See you soon.