Q&A With Homer Gaines

Kendall Avery at UXDX Community: USA West. Video: https://www.youtube.com/watch?v=CjOqqaDyods

Readable transcript: edited from the recording's captions for readability (fillers and false starts removed, punctuation and section headings added). Wording is the speaker's own. Timestamps are positions in the video. Names marked [?] could not be verified against the audio.

Running a remote participatory design study for the first time

[00:00:00] Host: Hello and welcome, Kendall.

[00:00:01] Kendall: Hi, thanks so much for having me. Excited to be here.

[00:00:05] Host: Excellent. I really enjoyed your talk. The purpose of this session now is that I'm just going to ask a few questions, but if anybody in the audience has any questions that they want to send through as well, I'll then ask Kendall those questions. Let's jump right in. Designing a remote participatory design study: I really enjoyed the talk. I loved how honest you were about how everything was new and you were learning as you went. Since that was new, what would you have done previously when faced with this kind of challenge?

[00:00:42] Kendall: I think one of the extra challenges was that remote everything makes it a little bit harder to connect with people. It's the kind of thing that I had seen done before by other people on the team, but not people that I'd ever met before. I had just seen their research done and I was like, "Okay, this has existed," but it had only existed in person. I wish I had had a greater community. I did a lot of Google searching: how have other people done this, does it exist? It felt a little bit too new, or at least too new to me. I really wish I had had a community of people to say, "Oh my gosh, you've run this before, what were your pros and cons? What are some ways that I can learn from your mistakes, and what are some ways that I can do it a little bit differently?"

[00:01:31] I did reach out to a lot of my colleagues internally who had run workshops remotely that would traditionally have been done in person. But I wish I had just had more people to go to and bounce ideas off. Because it was so new, in general or at least to the audience I was working with, it felt a little bit more like, "Let's just try it out and hope it works."

Convincing the team to try the method

[00:01:56] Host: Excellent. From your story, it was great walking through it and some of the challenges that you faced. Was it difficult to convince people to give it a shot, or who did you have to convince?

[00:02:14] Kendall: Well, myself first, that it was the right thing to do, that it was the right method, because there's always that imposter syndrome that creeps in: did I even know enough about this method to pitch it? So I think that was the first convincing. But in general, with a couple of my studies where they've been non-traditional methods, things that go beyond interviews and usability studies: that's traditionally what's requested when I'm working with my design and product partners and they say, "Hey, can we do some UX research?" So it's a little bit of convincing to say yes, but is this the right way that we can answer this question?

[00:02:54] There was a little bit of hesitation, I think, from the team, saying, "Do we really want to just let people design this home screen? Is that really what we're going to learn best from?" I think the biggest pushback was, "We're not going to use those designs, so why would we even ask them to do it in the first place?" So the convincing was really more about the rationale behind the method and why we were doing it: that we weren't just looking for these designs at the surface level, but it was really the level beneath it, of why they were creating these designs. Once we aligned everyone around that why, that this method was just a way to get us there, we seemed to have a lot of folks on board.

[00:03:33] We also included some things that were familiar. We started off with an interview at the beginning of each session, so there was a little bit of familiarity of, "Okay, this is running how research normally runs. It's not totally brand new and I feel uncomfortable observing." And that's another piece: having the team sit in and observe. It becomes less of me being the translator of "I talked to some people, I created a report, and now I'm sharing it with you," and more bringing them in and almost convincing them that way too. But I did have it in my back pocket: "Oh my gosh, if this completely fails, I'll need to have a nice backup plan so that I can still have the team on board and they'll still ask for research."

[00:04:15] Host: Great. And was the interview up front your backup, or what was your backup plan?

[00:04:21] Kendall: I think it would have ended up being having some designs. Our design team already had design ideas, and I thought about whether we should just show the ideas. But I wanted to make sure that we weren't relying too much on our own assumptions. It's easy to design for the ideal user in the ideal use case, especially when you're sitting at home designing only with Figma and not with real people. So I wanted to make sure that I could give people the opportunity to tell us what mattered most to them, show us what mattered most to them. But if this completely failed and really wasn't going to get us those insightful data points that we were looking for, it was still going to be worthwhile to put these concepts in front of users, talk to them and get a little bit of feedback from them. I just felt like there was more opportunity that we could have explored, and I think it ended up being really beneficial for our team.

When participants rebuild the existing UI

[00:05:13] Host: Great. You mentioned that some people almost rebuilt the existing UI. How did you try to coax people out of that comfort zone of "When I open up the app, this is what I see, so this is what I'm familiar with," to really get an understanding of what it could be?

[00:05:40] Kendall: What was interesting about that was, after they designed the screen, I had to ask the question, "How do you think this compares to your current experience?" That's the moment they would look and say, "Oh my gosh, it's exactly the same." So they didn't even realize that they were doing this, which I took away as such a strong insight: the ways that we think about our products and our apps and the things that we use become so second nature that we almost don't even understand the why behind it. So asking them to create the screen, even though it ended up looking familiar to them, still helped me understand what about each of those components they brought in was important and why, because that's really what we were looking for: the why for everything.

[00:06:24] We also found some interesting insights from people seeing something for the very first time, then looking at their phone and saying, "Oh, it's been there the whole time, I never knew it." So just realizing how strong some of these patterns can be: we get into a routine, we use a thing a certain way. Discovery has been a big topic of discussion: how do we get people to discover things, and what's the right way to do that? I thought it was actually okay that they ended up creating the same screens, because that was a good signal to us that what we thought was working maybe wasn't having the same impact as we hoped, and it really forced us to take a look at other directions we could go.

Pushback on sample size

[00:07:05] Host: Excellent. And I guess it's a familiar pushback for research: you had 19 participants. Did you get the pushback of, "What does 19 tell us?"

[00:07:17] Kendall: We did. I would almost say 19 was more than we would normally do, which ended up feeling okay. One of the reasons that felt okay was that we tested in multiple markets. Originally the study was eight participants in the US and eight participants in Mexico. That was when we had only tested with Rides users, and we saw this theme coming out and thought, "You know what, we should double-check with our Eats users too, to see if the same theme is there." So we actually added three Eats-only users to see if that pattern was there, and it was. For me, I was concerned: this is only three people, is that enough? I'm sure that's the opposite of the pushback we get, of "Is it enough?" And it ended up actually being that those three people had such a strong signal to confirm some of the other insights we'd gotten from the other participants that it was like, "No, this is really a strong theme that we're seeing here."

[00:08:17] Once you're able to show your team, and this is the importance of bringing your team along into those interviews and sessions with you, they can start to see the saturation hit: "I've talked to four people and we've seen the same thing every single time," or, by the eighth participant, you're really not getting that much new information. When you can see it happen, it's a lot easier to believe that 19 people is enough. But absolutely, we get that pushback all the time, and it's one of the reasons we partner really closely with our data science teams, to make sure we can get that triangulation: "I talked to 19 people here, but look, here's a hundred thousand people, and they're saying the same thing, just in a different way."

Prioritizing many offerings in a small space

[00:09:03] Host: Excellent. I love the book How to Measure Anything: even if you just get one piece of information, it can be hugely insightful if you have nothing so far. The last question, since we only have a minute: you alluded to a change in strategy, but how do you solve the problem of having so many offerings and only a small bit of space? Are there any tips you could give to other people facing a similar issue?

[00:09:33] Kendall: I think the biggest thing our team is learning is prioritization: getting really crisp on what the user cares about and what the business cares about, and then figuring out how to focus and prioritize. We've been circling around this topic for a while now, and one of the biggest things we keep coming back to is: what's our vision, what's our strategy? If we can just prioritize and focus on delivering a great experience for one or two of these offerings, we can start to figure out how we scale that way. It felt a little bit like we were, as we say, boiling the ocean, trying to tackle too much at once and asking how to expand all these different product areas at the same time. So it's really thinking about where the right place to focus is, where you're going to have the most impact, and then building off of that and applying those learnings to the next version of growth for your company. We're still using the insights. I pulled them up today with the team and we walked through them again; they're still having a lot of impact. But prioritization is definitely critical.

[00:10:41] Host: Great. Well, thank you very much for your talk and for your answers. I look forward to chatting with you again soon.

[00:10:49] Kendall: Thank you.