Designing A Remote Participatory Design Study

Kendall Avery at UXDX Community: USA West. Video: https://www.youtube.com/watch?v=udw6IJYsG6w

Readable transcript: edited from the recording's captions for readability (fillers and false starts removed, punctuation and section headings added). Wording is the speaker's own. Timestamps are positions in the video. Names marked [?] could not be verified against the audio.

Introduction and Uber's growing pains

[00:00:00] Hi everyone. Super excited to be here today talking about how we can bring participatory design online and how we've done it at Uber, and thank you so much to UXDX for inviting me to speak on this topic today. My name is Kendall Avery. I lead our user experience research team for the rider experience here at Uber. Previously I worked in the enterprise space for SAP Concur, and I graduated from the University of Washington's Human Centered Design and Engineering department in 2016.

[00:00:27] Today we're going to talk a little bit about the problem that our team at Uber faced, the setup for how we conducted this new research method for our team, some of the outcomes that we benefited from by using this method, and then finally close with a few reflections that I had on this entire experience.

[00:00:45] To set the scene a little bit: ever since Uber came onto the market more than a decade ago, we've been growing and expanding our business in lots of different ways to allow people to move all around the world. Then we started expanding into products like food delivery and grocery delivery, really expanding our offering so that we could be the company that allows users to truly go anywhere and get anything.

[00:01:13] As we were growing as a company, we started to realize that there were maybe a few growing pains. Through all of this change, we grew really fast. Our mobility offerings expanded super quickly; we thought about all the different ways we could allow people to move. And then COVID happened, and we had to adjust quickly to new lines of business and make sure that we were actually supporting what users needed in that moment, when maybe it wasn't necessarily the need to move around and get from A to B.

[00:01:48] We saw that we now had a huge number of offerings to support new use cases that users maybe weren't familiar with before, because the whole world looks a little bit different. So our team wanted to know: how can we tell customers the story of what Uber can offer without getting in their way? We know that our users come to us with a purpose, and we were starting to introduce a lot of new things to them, but we wanted to make sure that we were introducing them in a way that didn't interrupt the core intent that brought them to the product.

[00:02:22] Our teams have been tackling this in a couple of different ways. The first was really focusing on our home screen. This is the first interaction that a lot of people have when they come to our app, and over the past few years we've experimented with different ways that we can start highlighting some of these offerings to our users. We used an Uber Eats pill in the top right corner of the screen, moving into a navigation bar at the bottom, and then started to highlight more of these offerings. But as you can see, we started to run into a bit of an issue where we almost had too much to share with our users, and we were worried that we were running out of space.

Choosing a method

[00:02:56] Our teams came together, product, design and research, to ask how we could make sure that we weren't running out of space and not being able to truly tell the full story of what Uber can provide to our users. We asked our team a couple of different questions. We wanted to know, from the user perspective, how do our customers currently think about all of these different services that Uber offers today? And since we spent so much time looking at our home screen, we also wanted to understand what that ideal home screen experience looks like for our customers. After aligning our product, design and research teams around these two questions, the research team, myself and my research partner included, were off to figure out how we could actually tackle these questions.

[00:03:45] The first place we started was with methods. What were the right methods that we could bring in to really help us understand both of these core research questions? The first place we started was interviews. We wondered, maybe we could talk to users and have them explain how they use Uber today and how these different products and services actually fit into their day-to-day lives. But this wasn't really going to help us understand the home screen issues we were trying to understand, and how users engage with the home screen today.

[00:04:16] Then we thought maybe we could do concept evaluations: get some designs from our design team, put them in front of users and see how they react. The concern we had with this method was: what if we were designing concepts that didn't actually meet the full range of user needs, because we were building them with our assumptions in mind and not basing them on user needs that really existed? We wanted to make sure that we weren't limited in this approach, since it felt like we were at a really foundational part of this entire process.

[00:04:48] So we decided: what if we could actually combine these two together? What if we could allow our users to both show and tell us what their experience means to them and what it actually looks like, and maybe even show us what that ideal home screen experience could really be for them?

[00:05:07] We landed on participatory design. If you're not familiar, participatory design is a technique that invites the participant to actually be a part of the design process. It invites them to be a creator along with the team and gives them a different way to communicate what matters most to them. Traditionally this is done in person. It can be in a lot of different forms of media: you can use whiteboard markers, you can use sticky notes, some teams even use Legos. It's really meant to be a creative process that allows users to communicate in a new way and really show us what they mean when they talk about their own experiences.

[00:05:50] This was a super exciting realization for us. We thought this was going to be the right method to use for our project. We thought it was going to really help us understand how they think about those different services today. We could still interview them and get a little bit of context there, but when it really came down to that home screen design, this was the perfect opportunity for them to show us what meant the most to them.

[00:06:11] There were a couple of issues that we realized we were also going to face with this method. One, this was brand new to me. I had never run a participatory design study before; this was going to be my very first time running it. And even for my team, this was the first time most of them had ever seen it done too. So we knew that the first thing we were going to have to do was align on the benefits that come from using this method and why it was the right one, get everybody on board, and make sure we all knew what the outputs were going to be.

[00:06:42] The second issue we faced was that not only was this going to be the first time I had ever used a participatory design method, but given that we were conducting this research in March of 2021, and we were still conducting everything from a work-from-home environment, this was not just a participatory design study. This was a remote participatory design study. That meant we had to figure out not only how to get the team on board and align on how we were going to create these outputs, but also what tool we could use to actually conduct this online.

What participatory design is and is not

[00:07:18] As I mentioned, we had to bring everyone on board, and the first place to start was aligning on what participatory design is and what it is not. Participatory design is a method that allows us to understand user needs. It's a great place for users to really show us what they mean, and to understand what matters most to them. It's a form of visual storytelling: a way to give them a different communication tool than they might otherwise have, to really show us what they mean.

[00:07:46] It's also a way to encourage divergent thinking. This was really important as well. One of the reasons we didn't want to do concept testing was because we didn't know the full range of concepts that could be created, and participatory design is a great place to really explore the full range of creativity and make sure that whatever our ultimate solution is, it's something that actually works for our users.

[00:08:09] Now a little bit about what participatory design is not. It is not final designs. This was one of the first questions that came up. No, we were not asking our users to design the product for us. Instead, we were asking them to inspire us, to help us understand what mattered and what we should consider, and how we could take all of those thoughts, feelings and emotions about their current experience and bring them into a design that our extremely talented Uber design team was going to bring to fruition.

[00:08:38] Secondly, it's not rigidly confined. One of the reasons participatory design is such a great method is because it allows for so much creativity. So we wanted to make sure that our team was fully comfortable with being a little bit uncomfortable, introducing things that maybe we hadn't even considered ourselves, and making sure that we were allowing our users to be as creative as possible throughout this entire process.

Setting up the study: Figma components and Mural

[00:09:07] Now that we had everyone on board, the next thing to do was to figure out how we were going to create the study and, especially in a remote environment, get all of our users to be able to participate. The first place we started was by collecting all of our materials into Figma. This is the design tool that we use here at Uber, and this is where our designers had already started to think about different components that we could maybe introduce on the home screen.

[00:09:34] We needed alignment with our product, design and research teams to say, these are the different types of components that we want to put in front of users. We wanted a mix of things that already existed today, things that were a little bit new and innovative, and then also things where we didn't even know how they were going to perform, but this was the right format to introduce them. When we conducted the study, it was also important that we captured a number of different perspectives, so we actually ran it both in the US and in Mexico City. As you can see here, all of our components, once they were brought into Figma, were also localized and translated, so that our Mexico City participants were seeing them in their actual context, and the home screen designs that they came up with were things that truly looked familiar to them.

[00:10:21] So first we got everything together in Figma, and then we had to figure out how to take this traditional method of pen and paper, sticky notes, glue sticks, tape, all of that, and bring it online. What we needed there was the right tool to allow us to do it. When we were thinking about what was important, we knew that we needed a tool that was going to be collaborative, shareable and easy to navigate, and I'll touch on the importance of each of these.

[00:10:52] First, collaborative. We knew that it had to be something that our users were actually able to truly be a participant in. It wasn't going to be valuable if it was just them telling us what they needed and us moving things around for them. We really needed to make sure that they were in control of the entire experience. So we took a look at some remote whiteboarding solutions, tools like Miro, Mural and FigJam; there are plenty of others on the market as well. For us, we ended up using Mural. This is a tool that the research team was already using for some of our synthesis efforts, so it was a familiar tool for us. But take a look at what your company already uses, or tools that you're familiar with, if you're thinking about running your own remote participatory design study.

[00:11:36] Now we knew that we had a collaborative tool that was going to allow multiple people to engage with the product at the same time. That way we could make sure that it wasn't just a one-sided effort, and that our participants in the study were truly going to be invited to co-create with us.

[00:11:54] The next important thing was that the tool was shareable. We had to make sure that our participants were actually going to be able to join the canvas that we had created for them to conduct the study, and we wanted to reduce the barrier to entry by making sure that it wasn't something they needed to create an account for. With Mural, one of the ways we did this was by using their visitor links. This is something we could also bring a little bit of governance around: we could password-protect the links and make sure that any confidential information wasn't shared beyond the bounds of the study, and we could also remove access as soon as the study was completed. This way we knew that people didn't have to create an account; it was going to be super easy for them to join the session. And we could also make sure that we were protecting any proprietary information that was only meant to be seen within the study.

The practice exercise

[00:12:46] Finally, the last piece I touched on was easy to navigate. This is important because as technology professionals, we are using technology every single day, so what I find easy to navigate isn't necessarily going to be easy for somebody else. One of the things that we'd also learned through our remote research was that we had users of various levels of technical ability. We wanted to make sure that we weren't limiting access to their insights because somebody wasn't able to engage with this tool.

[00:13:20] We did this by starting off with a practice exercise. We created a practice exercise at the beginning of every single study, and we encouraged our participants to really get comfortable using this tool by dragging and dropping these icons over to this iPhone screen frame, and basically show and tell us what their current home screen looks like. Now, this wasn't something we were actually capturing data on; it wasn't truly part of the study. But it was super important for a number of reasons.

[00:13:51] One, this got them comfortable with the tool and the drag-and-drop functionality. This was going to be core for the rest of the entire experience, so this was our chance to get any bugs figured out, any troubleshooting tips, and make sure that people were really comfortable with the mechanisms of Mural before we dove into the meat of the study, where we really wanted to gather the most insights.

[00:14:16] Secondly, this was super important because it also allowed us to practice the think-aloud protocol and get users comfortable with the amount of information and detail that we were going to want them to share during the study. So even though this was a practice exercise and we weren't capturing data from it, we treated it as if it was part of the study. We would ask probing questions, we would follow up and ask users to share their rationale with us, and just make sure that they were comfortable talking through the decisions they were making. By the time we moved over to the main exercise, they already knew what was expected of them, and we could jump straight into immediate value for both the participant and the rest of the team.

Running the study and synthesizing visually

[00:15:02] We had our components in Figma, we had our practice exercise set up, and then finally we brought it all together into the canvas that we used for actually conducting the study. After completing that practice exercise, we brought our participants over and essentially turned them loose to create their own home screen design. We made sure to bring in a variety of components that, again, were all aligned on between our design and product teams. And to encourage some of that creativity, we brought in some create-your-own components, which made sure users understood that if there was something they needed that they didn't see here today, it was still encouraged that they bring it into their designs.

[00:15:45] We conducted this study with 19 participants across the US and Mexico City, and at the end of it we had 19-plus home screen designs. The benefit we also found with this remote method was that after every single study was completed, we had this artifact of what their home screen design looked like, and we could bring them all together and actually synthesize our study results visually.

[00:16:15] This was also a new method for us. Traditionally we would code our interviews and re-watch video clips. We still had all of that, because our sessions were conducted via Zoom, but this gave us another way to quickly synthesize and see patterns across all the designs that our users had created. We were able to bring in all of the different designs, and also all of the different things that were not included in the designs themselves. That was a super important piece as well: not only did we want to know what was included, but what was the rationale for things that were not included. We brought them all together and were able to quickly see themes and patterns emerging, and even used Mural's comment feature to leave comments directly on some of the components and bring in direct quotes from our users to help explain the rationale for their design decisions.

[00:17:08] This was super beneficial for our research team, but also for our partner teams. This is something that we actually shared out while the synthesis was ongoing, and it was another way that we could encourage our team to be immersed in the insights as they were happening, and not feel like there was a big delay between the research being completed and the report being shared out. It was a way to incorporate everybody into the entire process and make sure that they were capturing the insights at basically the same time that we were.

[00:17:39] And then finally, some of the outputs from our designs. This is the type of artifact that each user created, and it was really helpful for us to see that mental model come to life, see what mattered to them, understand why, and really challenge some of our own assumptions around what we thought mattered to users. We even asked some of our participants to take a look at the screen that they designed and take a look at their app today, and we saw that some people were even designing the experience that they already had, because that mental model was so strong, and the way that they use our products today really drove how they created that home screen design. It was super helpful for us to have this visual representation of what mattered to our users, and a really concrete way for us to make sure that we were designing products that met their needs and not just based on our assumptions.

What changed for the team

[00:18:34] So the study is complete. We have all of these different artifacts, we're doing the synthesis, and we get these really great insights. What does this actually mean for our team? For Uber, this meant we actually changed our strategy. The outputs from this research really informed, or I guess challenged, a lot of the assumptions that we had, and took us in a new direction. It even led us to pause some of our engineering efforts, because we wanted to make sure that we were building the right thing and not following our assumptions from before. So it was really valuable that we did this research at this point in time.

[00:19:13] The next is that it led us to fast-follow research. We were immediately able to pivot to new research questions that came out of this round of research. We quickly turned our focus to say, "Okay, we've learned all this new information. What are the new unknowns, and how can we go out and tackle those?" Through some of that fast-follow research, we were also a little bit more open to trying new things. We experimented with different research methods, like a five-second study or unmoderated card sorting, things that were still a little bit unfamiliar to our team. But there was more comfort now that we had tried a method that was new to all of us and it had really paid off. So in our fast-follow research, we were able to be a little bit more open and creative, and make sure that we were doing whatever it took to get the insights that we needed to make confident product decisions moving forward.

Reflections: when remote works and when it doesn't

[00:20:10] The project on the whole was a success for the research team and for Uber. What did this mean going forward? What were some of the reflections and takeaways that I had from this project? Well, the first was that this was a remote study because it had to be. We were in a work-from-home environment, and there wasn't even the possibility to do this in person. But it also got us wondering: is remote still a viable option for future research studies when we want to use participatory design?

[00:20:44] Some of the things we considered: absolutely, one of the great benefits we got from using this method was that we really had global reach for our users. We talked to people in rural areas and urban areas, folks that, had we done this in person, we wouldn't have gotten the same variety of responses from. We have offices in metropolitan locations, and you tend to recruit people that live locally to those areas, so we ended up getting a more diverse group of participants by doing the study remotely.

[00:21:17] Secondly, as I mentioned, our synthesis was all contained in the tool, which was super helpful for expediting the synthesis process altogether. Had we done this in person, we might have added a bit of overhead trying to figure out how to take pictures of all the different artifacts, where to store them, and how we were going to conduct synthesis on all these different physical materials. Having them all consolidated in the same place sped the process up a little bit.

[00:21:44] And finally, because it was done remotely and over Zoom, it was really easy for our partner teams to observe the interviews and participatory design while they were happening. This is something that's really important for us on our research team. We always encourage our partner teams to attend research sessions whenever possible. This was a great way to make sure that the participant felt comfortable while they were designing and didn't feel like they had a bunch of eyes on them, but it still allowed our partner teams to observe quietly in the background and get the insights in real time.

[00:22:19] Now, that's not to say that remote participatory design is always the way it should be. There are plenty of reasons not to do it remotely. For example, technical limitations definitely came into play. We had some participants who, despite the practice exercise and all the information we gave them up front about how to join the session, still had technical issues, either interacting with the participatory design itself or even just getting access to the tool. We had some participants whose laptops were down; they had to join on a mobile device, and the study just wasn't going to run the same way. So we had to adapt on the fly and make sure that we were still able to get value out of these sessions. But technical limitations were definitely in the mix, and it's something to consider. If your core audience that you're testing with has technical limitations to begin with, then you might not want to do a remote version of this method.

[00:23:13] Secondly, because it's remote, the creative mediums that you're able to have your participants engage with can feel a little bit limiting. As I mentioned, we did everything with Figma. We chose to go high fidelity. You can go with a range of fidelities and even use wireframes, but you still end up being limited to these 2D mediums to interact with. So if you really want to encourage a lot of creativity and a lot of divergent thinking beyond the digital bounds, you might not want to do a remote study, and you might want to opt for something that's a little bit more physical, in-person and creative, like Legos or pom-poms, or who knows what it could be.

[00:23:57] And then finally, we did have to limit the number of exercises that we conducted with our participants. We had ideas for all these different ways that we could ask them to show us different mental models, and we had to scale it back, because we know that Zoom fatigue is real. We didn't want participants getting tired after spending too much time on a study talking to a computer, so we had to limit the number of exercises we introduced to them. If you're thinking about your own research needs and there's a lot you want to cover, that's another thing to consider, and a reason you might choose an in-person version of participatory design.

Tips and a checklist for success

[00:24:34] Now, a couple of tips for success for this study. Number one, allow for creativity. Participatory design is most successful when everybody feels empowered to be their most creative selves. So really think about whether your study merits this full range of creativity, and how far you can push the bounds of that. The more that you, your team and your participants know that this is highly, highly encouraged, the better the results you're going to get.

[00:25:05] Second, be flexible. Especially in a remote environment, things are going to go wrong. So think about all the different ways you can troubleshoot in advance, and make sure that you're super comfortable with the tools that you're using. But also, if things go wrong, that's okay. Like I mentioned, we had a couple of participants who weren't able to fully engage with the study the way that we had hoped they would. We adjusted: we turned it into more of an interview instead, or I would drag and drop the items for them and ask clarifying questions. We really tried to get as much out of it as we possibly could, and I think that's the most important thing when you're conducting research with users anyway: make sure that you're finding whatever valuable information you can when you're speaking with them. The time is so valuable, and it can be so limited, that even if you get one interesting insight, I think it's still worth it.

[00:25:59] Finally, pilot test. This is so, so, so important. Pilot testing is critical to running this type of research, especially since there are so many moving parts. You've got your collaborative tool, you've got the technical limitations of the users that you're working with. I always make sure that I'm pilot testing with somebody who's completely unfamiliar with the product. That way I'm also understanding if my questions make sense for them, and if there's anything that I haven't considered or have missed because I'm a little bit too close to the product. So make sure that you're practicing this before you get in front of users. That can help make sure you know all the troubleshooting steps to help them, and also possibly expand the bounds of what you thought your test was capable of in the first place.

[00:26:46] Finally, a checklist for success. Number one, determine your research questions. Make sure that the research questions you have are going to be well suited to a participatory design method. Not every single research question is well suited to participatory design. It's definitely not a one-size-fits-all method, but make sure that this is something where you really feel like you're going to get benefit from having your participants both show and tell you what that mental model looks like for them.

[00:27:14] Secondly, gather those materials and stimuli. Make sure that you have all the different things that you're going to ask your participants to create with, have them all together, and get alignment with your product and design teams to make sure that everyone agrees that this is what's going to be tested. Next, select that remote collaboration tool. Find something that you're familiar with, play around with a couple, and figure out the one that you think is going to work best for you and your team's needs. Then set up that practice exercise. Make sure that it's something that maybe fits within the realm of the study, but that's going to allow you to practice the troubleshooting and get participants comfortable with that think-aloud protocol at the beginning of the study. Next, set everything together with your canvas. Pilot test, pilot test, pilot test. And then finally, launch and iterate the study as needed.

[00:28:02] Thank you so much for your time today. I hope you feel inspired, and possibly interested in trying out remote participatory design with your teams. Thank you again to UXDX for having me, and check us out at Uber. We are hiring. Thanks again.