AI as a design partner: how to get the most out of AI tools to scale your process

Kelly Dern at UXDX Community: Mastering AI Tools and Cognitive Insights for a Captivating UX Portfolio. Video: https://youtu.be/dOr58sp0UHU

Readable transcript: edited from the recording's captions for readability (fillers and false starts removed, punctuation and section headings added). Wording is the speaker's own. Timestamps are positions in the video. Names marked [?] could not be verified against the audio.

Introduction: using AI to scale collaborative product design

[00:00:00] It's really great to be here today. I'm going to talk about AI as a design partner and how to get the most out of AI tools to scale your process. I am a senior product designer at Google working on the video AI program, but I also work as an instructional designer for the product inclusion and equity team, and I consider myself to be a systems designer, which is different from design systems, so I work on designing things at scale. Previously I've been on some other teams, like at some health tech startups and then also at Google Nest.

[00:00:37] So today we're going to talk about using AI to scale collaborative product design, and during our journey we're doing a deep dive into designing for all and what that means at scale, and how you can use AI tools to systematize your user journey creation process. We're going to talk through some additional AI-powered design sprint methods, and then augmenting versus automating your design process.

[00:01:05] When we talk about AI you might hear GenAI, or gen AI, or AI. What I'm talking about today is using massive data sets of text and code to learn patterns of human behavior and language in order to generate coherent and meaningful content. And content can be components, or text, or imagery, or video, and we're seeing all kinds of combinations now, which is really exciting.

[00:01:33] Why is this topic important to us and why is it important now? We're at the beginning of this wave of AI, which is really exciting, and I think as product creators we have the opportunity to build in better processes at the very beginning. As a product creator, I'm talking about anyone who's a designer or researcher or writer, anyone involved in the digital product creation process. And these people are very busy, they're balancing many priorities all at the same time, which could include your business goals, or a leadership ask, or user needs, or team goals, and it's just a lot of balls to keep in the air, and sometimes users or features or a goal or something can fall through the cracks.

[00:02:28] But with GenAI tools you can now balance all of these things at the same time and be more systematic in your process, and you can leverage AI to design for everyone. So those features or tools you may have left out during manual processes, now you can build in systems from the very beginning so that you're not unintentionally excluding anyone. So you can have a more systematized versus manual process.

[00:02:56] But this sounds really great, so how can we do this in practice? I'm going to talk through a couple of different case studies, and I'm going to go through a lot of content, so please save questions and we can talk about it at the end.

What designing for everyone really means

[00:03:12] I'm going to talk specifically about using GenAI for product inclusion and equity design. So what does designing for everyone really mean? Product inclusion and equity is the process of building belonging through all of our products by designing for the most marginalized voices at every phase of the product creation process. That is the goal of design for product inclusion and equity, and I think with GenAI tools we can now actually do this, instead of thinking of it as kind of an afterthought in our process.

[00:03:43] I think there's a couple of different pillars. The first one is documentation, such as documenting user flows, such as critical user journeys. Some teams call these things jobs to be done, or user stories. I'm going to call these critical user journeys, or CUJs, throughout this talk. The next pillar is ideation, so during your collaborative product design methods such as design sprints or workshops. And then the third pillar is taking all of this information and then synthesizing, organizing and prioritizing it. In this way, as product creators you can be more systematic, you can scale yourself, and you can have more speed and execution, and you can design more intentionally to avoid exclusion.

[00:04:30] Which is in this famous quote: if you do not intentionally, deliberately and proactively include, you will unintentionally exclude. A recent Kantar study shows that consumers actually really want this as well. For the majority of consumers, it's important to them that the companies they buy from are actively promoting diversity and inclusion in their practices and their products and in their features.

[00:04:55] Let's look at some other data more in depth as well. Just kind of keep these numbers in the back of your mind as we're going through this. One out of seven: what is this referring to? Nearly one out of seven, or 15% of the world's population, have a disability, which is a huge amount of people. How about one out of five? This is people older than 50 by 2050, which is also a large group of people. What about 700 million? By 2050 there could be over 700 million people with disabling hearing loss.

[00:05:36] So what these numbers are showing is that there are actually large parts of the population who have all kinds of abilities, which we refer to as disability, and they actually might be, instead of the minority, they might be a large group of your users, or even potentially the majority of your users, in your product creation process. But as product creators we know that we are designing for everyone, and we do our best to make sure that we're doing that. So we're designing for situational and temporary disabilities, which could mean you're cooking, or your hands are busy, or you're on a bumpy train and you're moving around a lot, or you're in a large room, or you're hard of hearing. Another example could be you've broken your arm, you're carrying a large package or something, you can only use one hand. And also permanent disabilities, such as being blind, or you're neurodivergent, or color blindness and things like that.

[00:06:36] We're also designing for experiences such as multimodality. So this means if you're using an app on your phone, you want to be able to go to your tablet or your smartwatch and have a seamless experience across all of those devices.

[00:06:52] And we know that we're designing for everyone, but sometimes things fall through the cracks, as I mentioned before. So what happens when we don't do this? These are some headlines that I've pulled out and I'll read through a few of these. "Alexa, why are women 47% more likely than men to get seriously injured in a car crash?" Or, "How self-tracking apps exclude women." Or, "Google fixed its racist algorithm by removing gorillas from its image labeling tech." Or this one a few years ago: "Twitter's audio tweets revealed an accessibility miss and now the company wants to fix it." So Twitter had released a feature that didn't have an audio component, so it was a big accessibility miss for the product. And we can avoid these things from happening by systematizing our critical user journeys.

Dimensions of identity and intersectionality

[00:07:47] As individuals we all have different dimensions of identity, which are hidden or visible identities which define who we are, how we think and how we interact with others. So that can be your education, or culture, or age, or experience. I'm going to use this list of dimensions of identity for this exercise, but these are often changing, some are being added or refined. For our purposes we'll use this list. And these identities can intersect and inform new, complex identities. So your age might interact with your gender or your race or your ethnicity, and creates... these individuals have really complex needs and identities which are not defined by the individual identity alone.

[00:08:32] And if we take the dimensions of identity and apply it to our systematizing process, what we can do is write inclusive user flows using the dimensions of identity from the very beginning. Next we can consider intersectionality, so these are the people who identify with multiple groups, traits and attributes, and then how these things work together to define complex identities not described by the individual attributes alone.

[00:09:04] And if we're going to break down user journeys even more: "as a low vision user I'm able to complete device setup in under 30 seconds." And if I'm rewriting this using the dimensions of identity, I could do something like "as a Spanish-speaking user," "as a non-tech-savvy user," and I can repeat that with all of the other dimensions of identity. And as you see, if you're actually doing this for all of the dimensions, and then intersectionality, it can be a huge long list and a very manual process, and you still might miss some of these different dimensions.

Prompting: breaking down a critical user journey

[00:09:38] So I'm going to break this down even more using GenAI. If you're breaking down what is a critical user journey, which we'll need for the exercise: as a user type, I want to goal, so I can task. In this example I'm going to use this hypothetical app that I made up called the Paw app[?]. So, as a dog owner I want to use the Paw app to find people near me to walk my dog by searching nearby. If you're following along, or you want to do this later, you can open Bard, ChatGPT or another conversational AI tool of your choice. I've tried it with both of these and this exercise works great.

[00:10:16] First you want to make sure you have the elements of a good prompt design. So you need to describe in detail your step-by-step guide by mapping out the experience of the tasks and steps. Next you want to define the persona, so who is this model pretending to be? In this case we're prompting with the dimensions of identity. And next, provide good examples: what kind of formatting or writing style are you looking for, are you expecting them to return that answer in a very specific format? You need to prompt the model with that. And also the constraints, and this is very important, because if you don't provide enough defined constraints the model can maybe hallucinate or not provide helpful information. And then next, provide the chain of prompts, so you might have to go back and refine the original ask and probe for more detail. So if originally you prompt asking for a CUJ and it doesn't work for you, you need to go back and refine it rather than assume that it's just not working.

[00:11:20] So if we're going to open Bard or ChatGPT, let's give this a try. You can take the dimensions of identity and ask the model to write 10 different examples per identity, with the constraint of one-word answers. So you take accessibility and age and gender and the whole list, and this will output a list which we'll use in the next exercise.

[00:11:43] Using this example, I've taken my prompt here, which is my hypothetical app: Paw app is a dog walking mobile app, a two-sided marketplace that connects dog owners, yada yada yada. And then below I put the prompt: "Using the above product for reference, rewrite this critical user journey and write additional user journeys for each dimension," and here is where I input my user journey. And then I ask it to replace the dimension, which is "blind," with the following dimensions of identity, and here is where you put that list that you output in the previous exercise. And if you want to go into more detail and be more refined, I would provide as much detail about your feature or product as possible. This could be a huge long list, like this prompt here. I won't go through all of this, but if you are following along, we can share these slides out later if you want to see.

[00:12:42] And then you can also do this with other things, like replacing "blind" with temporary disabilities, or permanent disabilities, or all kinds of things. And then you ask it to generate a creative task based on those identities, and then go back and repeat with a different dimension of identity. It is possible you can ask the model to do all of the dimensions at once, but I find it's easier to do one at a time.

[00:13:12] So let's look at this example. I asked the model to output a list of 10 different dimensions of identity and it put it into a table for me, and then I gave it the constraints and details about my app, and then asked it to rewrite my CUJ based on those different dimensions of identity. And you'll see that it comes up with a very interesting list. Some of them are a little off the mark, so I might want to go back and refine my previous prompt, but this gives me a really good starting place of critical user journeys based on my dimensions of identity. You can also go back and add additional dimensions of identity and make this list much longer.

[00:13:56] Now you can take this list and prioritize by your categories, such as business metrics or user types or user goals. You can also use conversational AI tools to find themes or overlap areas, and it can identify those for you, all in minutes.

[00:14:16] Now let's talk through scaling this with other prompt framing. You can try this with other types of users or modalities. We were talking about multimodal design: you can ask it to reframe the CUJs based on a mobile app or desktop or an audio-only device, for example. And then also going through and doing temporary disabilities, situational disabilities or permanent disabilities. For example, using the GenAI tool: what are the different permanent, situational or temporary disabilities? Second step is your product prompt, and then "using the above product for reference, rewrite this user journey," and then replace that dimension with the following situational disabilities. It's the same process over and over again. If you do try this method, I would love to hear how it works for you. I've tried this on several teams and products and it's been very helpful.

Other sprint methods: how might we, affinity clustering, abstraction laddering

[00:15:18] Now let's talk through a few other sprint methods. Leveraging AI for how might we statements. How might we's are part of your workshop or design sprint process, and they reframe insights into opportunity areas. So for example, how might we onboard our users to the app with two clicks or taps? So again you take your product prompt: "using the above product for reference, write 10 different how might we statements that focus on the following business priorities." I've given it the constraint of multimodal design, more inclusive design, or fewer clicks to find a dog walker, and it outputs a list. And you can ask it to output a hundred and then go back and refine.

[00:16:08] Next let's talk through affinity clustering. Affinity clustering is a method that I use a lot. You take large amounts of information that you have ideated on with your team using stickies or whiteboards or things like that. It used to take several hours and many people to come up with these ideas, but now you can, in several minutes, use a GenAI tool to find themes between these statements. This list on the left is an example based on my previous prompt. And it just allows you to be more efficient with your time.

[00:16:53] Abstraction laddering: this is also a really fun one to use with GenAI tools. Abstraction laddering is a technique used to dig deeper into a problem by gradually moving from concrete details to more abstract details. So you start with the initial problem statement, you ask why to get more abstract, which is a little bit more outside the box thinking, and you ask how to get to a more definite statement. The reason I believe GenAI tools are really great at this is that they're really good at abstract thinking, or I think even going into the realm of hallucination, so as a product creator you can use this tool as a way to jump your thinking. But we'll talk a little bit more about that later.

[00:17:37] So looking at this example: what features of the Paw app are important to you as a dog owner? Concrete would be easy scheduling on the Paw app, and more abstract would be things like knowing my dog is being taken care of, or even more abstract is a stress-free life for myself, or the ability to focus on my work and other commitments, or a sense of balance and well-being. And where I think this gets more powerful is when you ask it to come up with hundreds of abstract thoughts. It might get you, or get your team, beyond maybe your inside-the-box thinking, and can get you thinking about things that maybe hadn't come across your team.

Augmenting versus automating, and key takeaways

[00:18:21] Next let's talk about what this means for product creators: augmenting versus automating your process. A quick note on bias. We want to use these tools to augment but not entirely replace our design process. These models are not 100% perfect. I think in the previous talk we were talking a little bit about bias, so we must stress test the output. As we've seen with some tools that have come out, for example Midjourney is trained on certain images of people which might have bias data such as racial bias or gender bias, age or occupational bias. So we want to avoid continuing to reinforce this bias. We do want to make sure that we're either building in mechanisms for detecting the bias in these models, or also just, as the product creator, auditing them ourselves.

[00:19:19] So let's put this all together and just talk through the key takeaways from all of this. A product creator is balancing a lot of things at the same time, and we can use these tools intentionally to scale user journeys using the dimensions of identity. So you can take a critical user journey, put it through a GenAI tool filtering by the dimensions of identity, and you can have a scaled user output. And you can also do this with intersecting identities, so using a critical user journey and then filtering through the different dimensions and then asking it to review multiple different dimensions at the same time.

[00:20:05] And to recap a few of the things that we talked about today, ideas that you can take back to your team or your different products. We talked through systematizing user journeys to scale to all users, and remember that there's bias in the data, and it's augmenting but not entirely automating your process. You can either build in guardrails into your model quality itself, or you as a product creator are doing the model quality. You can scale this to other design sprint methods, so today we talked through user journeys, how might we statements, abstraction laddering and affinity clustering. I think you can also do lots of other types of design sprint methods and I'd be curious to see what you all have tried. And remember a good prompt structure, and don't be afraid to go back and readjust your original prompts.

[00:21:04] Thank you so much for your time today, and please reach out if you have any questions about anything, or if you've tried these methods I'd love to hear your feedback. I've linked my website here with some other information about things I work on, and also my newsletter, and I look forward to hearing from you all.

Q&A

[00:21:29] Host: Excellent, thank you very much, that was really good. It's probably one of the most frequently asked questions, is how can I really use AI to speed up my work, and it was a very good practical example, so thank you for sharing.

[00:21:44] Kelly: Of course.

[00:21:45] Host: And just a reminder to everybody out there, we do have a few minutes now, seven minutes, for Q&A, so please ask any questions or any things that that talk triggered and we'll pass those questions through to Kelly. But I'm going to start off, and I'm going to unfortunately be the cold-hearted capitalist now and say, even with AI speeding that up, why should we spend our time and our money doing this, as a pure cold-hearted capitalist, when 80% is good enough? We're going to get enough money and we're going to get dwindling returns on that 20%.

[00:22:25] Kelly: Well, I think you can see from those example headlines I pulled out that people are still falling through the cracks, and so it can also be that your company could be losing money by missing out on designing for certain users. So although they might be a minority, it could be perceived as your company is not inclusive, or your company has excluded some users. Also I think these tools allow us to work more efficiently so we can do these things. Before, it would have taken us a lot of time to ideate and all these processes and potentially design for them, but now I think we can do this much faster, and I'm very excited about things that Figma is working on, hopefully to automate some of our design processes so this won't be so repetitive. So I feel like now we can do things that used to be very manual, and although it might seem like it's a smaller group of users, they are part of your user group and you can be saving your company money by including them in your process.

[00:23:42] Host: Excellent. And I know from my personal experience working with airlines, there's a lot of legislation coming in now as well, so it's not just purely about the bottom line, it's reputation, brand reputation, which is very, very important. There are legal fines for not accommodating particular groups of people as well.

[00:24:03] Kelly: I have not heard about the airline stuff, but that's interesting.

[00:24:07] Host: It's international travel, so transatlantic. If you're just doing short haul I think it's fine, but for international, and I think they've mentioned that they're going to extend it to short haul in the future, so airlines have to be very accessible in their journeys.

[00:24:23] Host: Cool. We have, it's more of a logistics question from Mary, she's asking, is there a link to Kelly's deck? That was super valuable. So yeah, if you can share your deck we'll post that out to people as well.

[00:24:37] Kelly: Yes, I can definitely do that.

[00:24:40] Host: Excellent. So we have a question from Cindy: how do we balance our energy using AI and correcting the responses that AI gives us, because sometimes I spend more time adjusting the response than it would be if I just did it myself?

[00:24:56] Kelly: Well, I think that's where you can do things at scale. So if you get a long list and you say I want to refine all of this in this way, I think you can ask the GenAI tools to do it for you. But you are getting into a state where, instead of creating 10 CUJs for example, you're doing hundreds, so you have to be able to use these tools to design at scale. I have heard of people doing things like, far more complicated, like sampling the data: if you have a list of hundreds, for example, having a script that looks at a couple of different CUJs just to spot check them, to make sure they're doing what they should be doing, or they're representing the users in the correct way. In an ideal world you would have your different user groups represented as actual people and they would be the ones that can actually audit this full list to make sure that you're representing everybody. That's not possible for a lot of companies and teams, so it does add in some admin overhead, but I think it's also worth it. I think if you can figure out how to review them at scale it should be able to help you.

[00:26:20] Host: Yeah, that's some great advice. I think it's a skill that you just get better at, and once you've learned it then the time saving increases and increases.

[00:26:31] Kelly: Yes, definitely.

[00:26:33] Host: And one other kind of point, just if you do have questions do send it in, but it's just, you mentioned something there and I thought it was quite good. In programming there's a concept of pair programming where two developers work on the same machine, because it's just that back and forth conversation where it helps them think of better things. And I see in my personal work, I see ChatGPT, or sorry, Bard, as kind of almost a pairing partner, where I can ask it questions and it just gives me some ideas. It's not going to give me the perfect answer that I can just run with and go away, but it gives me enough of a starting point. I find it very helpful.

[00:27:18] Kelly: Yeah, that's a really good way to think of it. It's kind of like pairing with the tool in order to create user journeys or workshops or things like that. Yeah, I like that framing.

[00:27:34] Host: Well, excellent. We don't have any more questions coming in and we're just about on time, so I just want to thank you once again, Kelly, for sharing that amazing anecdote of how you're using AI. Thank you.

[00:27:47] Kelly: Thank you for having me.