AI for the Real World: Creating considered Products through Research and Design

Esra Gokgoz at UXDX Community: EMEA Online Series. Video: https://youtu.be/dw4l97fQ5NI

Readable transcript: edited from the recording's captions for readability (fillers and false starts removed, punctuation and section headings added). Wording is the speaker's own. Timestamps are positions in the video. Names marked [?] could not be verified against the audio.

Introduction and background

[00:00:00] Host: ...to welcome Esra.

[00:00:02] Esra: Hi, hi everyone. Firstly, thank you so much for having me. Really excited to be with you and to talk to you about designing thoughtful, considered AI products by using research and design.

[00:00:17] A little bit about me. My background is in design, but I prefer to think of my work in these sorts of themes that you see here. I've always worked in what I would call a more strategic capacity, a strategic design role as it were. And I've always also tried to do that at the early stages, so early stage ventures, early stage initiatives, to work with some of that ambiguity and to bring some of the toolkits, mindsets, all kinds of designerly things, before potentially any actual design activity is even taking place.

[00:00:55] And I've had the joy of doing that in a number of design and innovation practices at BCG, at Salesforce, and now at Microsoft. So at Microsoft I'm part of the data and AI organization, and we're a customer-facing part of the organization. We work with Microsoft customers on their biggest data and AI challenges, and we do that across all sorts of industries, all sorts of different problems. So anything from, say, detecting defects in manufacturing, to helping forecast or predict demand for retail. So some of the stuff that I'll be sharing with you today is going to be a cross-section from that very broad experience, and again from those very early stages of working through these potential products or potential services.

The narrative on AI, and why it isn't just a technical problem

[00:01:46] I don't know if some of you maybe already work with AI products or have at least implemented AI features. Maybe you have that direct experience. But regardless, I think that the narrative on AI is not always particularly helpful. There's a whole lot of headlines telling us either that it's going to solve all our problems or create the worst problems of all. And you might actually start to think that maybe it's a little bit of a buzzword. I think that's also certainly true in certain contexts that it's used. But I think at the very least it starts to maybe feel like a bit of magic, well, just magic actually. Or like a bit of a black box, maybe something that only people with a particular kind of expertise get to play a part in or actually define.

[00:02:36] And that's not really true, in my opinion at least. Like any technology, its value is in its use and its context. Yes, it is technically complex to develop and deploy AI solutions, I never want to discount that, but also it's not just a technical problem. In a lot of the kind of work that we do in this enterprise context, it's helping people, for instance, correctly manufacture really sensitive electronic equipment, or to make medical interventions. And especially in those contexts where the stakes are high and there's a lot of embedded expertise, in that context it's really important that we create things that are fit for purpose, and that we design things for the real world with all of its complexities and all of its contradictions and nuances. And for me, the best way to do that is through research and design.

[00:03:37] And when we think about the role of research and design in this context, it goes far beyond just thinking about usability or a great experience. As someone who's always championed this, I think this is always true, regardless of whether you're working with AI or not. But again, when the stakes are so high, and because of the nature of AI systems, we have to fundamentally consider what design does in this context.

[00:04:05] There are characteristics of AI systems basically that require that. So they're not deterministic. They adapt over time. We don't explicitly always tell them what to do, and thus we can't always predict exactly how they'll behave in different conditions, for instance. And they also have a lot of autonomy. So for the most part when we use them we kind of delegate our reasoning or our decision-making to these systems or these features. And all of these combined mean that it can be a little bit opaque, and also can fail in certain circumstances, and that can be difficult to anticipate. And all of that creates a very different landscape for research and design.

Who is the user, when the system reimagines the relationship?

[00:04:49] So my team that I'm on is called the human-AI experience team. And at first that might just seem like fancy language, a different way to say UX. But it's not, I promise you. It's actually due to the nature, this fundamentally different nature, of working with AI systems. So if we think about users, that word starts to mean something very different when we think about what AI can do for us. Quite often it's actually fundamentally reimagining that relationship.

[00:05:21] So if we take something like autonomous vehicles, in that context, who even is the user anymore? There's a driver that's not really driving. So it starts to really shift that idea. Maybe we also then actually have to consider more the, say, pedestrian that's crossing the street, the neighborhood that this vehicle is passing through, the city, what it means for the transport system, what it means for this wider ecosystem of actors, say insurance companies or mechanics, and maybe even actors or forces that don't necessarily have a human voice, like say the environment.

[00:06:00] So in this kind of context where we fundamentally are reimagining the relationships between these things, it almost becomes redundant to think about design as working between the user and the technology. It's a very limited scope and is really unhelpful. And instead I think design and research can become something that helps us explore this much wider context and this much wider set of relationships.

[00:06:29] And I've also been talking a lot about the human-centered part of this, but obviously that's not it. To create a well-rounded product there are often many, well actually often quite competing, forces at play, and things that pull us in different directions. So yes, we have to think about how this affects humans and society, but also we have to think about users. We have to think about our organization and what's technically possible, what data is available and so on. And this will look different for you as well, depending on what you're working on and what your organization is. But I'm sure we all recognize these forces, or some version of this, in our world.

[00:07:09] If you have been keeping up with the latest developments in AI, certainly in the past few months there's been a lot of new developments, a lot of hype, dare I say, around say DALL-E or ChatGPT. And I think it's been, even as people that are relatively close to the field, incredible to see, and also surprising to see just how much is possible. And all of that I think centers around this part: we're just pushing the boundaries of what is possible, and we're just using what's available to us to try and achieve new, different, cool things really. And there's absolutely nothing wrong with that. I actually also really believe in that as a source of innovation and as a source of opportunity. But that can't be it. We can't stop there. We have to think about these other forces and try to reconcile some of these other forces, and actually work within these creative tensions to make sure that we're creating something that is considered and well-rounded.

"AI will help us with our low value tasks": low value according to whom?

[00:08:18] So, some examples of the kind of considerations that we have to bring, that I think can be powered by research and be powered by design. A really common refrain in the AI space, maybe you've come across this, is that AI is going to help us with our low value tasks. I think I actually do agree with part of the sentiment, because it suggests a partnership between humans and AI, and I think that's a really appropriate way to think about it. But if you actually pause and think on this sentiment a little longer, you kind of start to ask questions like, okay, value, what kind of value? Functional, emotional, financial probably. And also low value according to who? According to the person that's doing the task, the person that maybe is suffering from the problems?

[00:09:10] So what is high value to one person in one context may not be in another. And a really mundane example of this is, if I was say packing up my belongings and wrapping up my things because I needed to move house, that for me is an activity I would love to outsource. It's tedious. I am very happy to have that be automated or to have assistance with that task. But say wrapping gifts for loved ones for the holidays, that's very different, even though it's functionally a very similar task. It's actually something that I might take a lot of joy in and that's also very personal to me. So both of these might be equally tedious to somebody else altogether.

[00:09:55] And that was a silly example, but you can see how this manifests in the real world when we use AI systems in the real world and we center a very particular type of value. So what a business stakeholder, who's say several degrees removed from the task at hand, thinks is high value or low value might be at odds, or in tension, with what the person actually doing that work values.

[00:10:22] And I think it's really also very easy to see how these things come about. It's not like there is some mastermind at work coming up with terrible ideas on purpose. I can completely see a world in which someone just says, "Hey, we need to make sure that our warehouses are efficient. We need to make sure that our employees are performing." And that's also completely reasonable to expect in a commercial enterprise. But I think the problem becomes when we only center that world view and that set of values, and don't try to reconcile it with alternative points of view.

[00:10:59] And again, these are the kinds of things that show up in the headlines. They are the more visible things that we see. But things don't have to fail catastrophically for us to pay attention to them, or for them to be bad outcomes for our work. It can also just manifest as a lack of adoption, or lack of engagement, or just creating things that are not quite as useful as they could be, ultimately.

[00:11:26] And again, commonly we talk a lot about these negative scenarios. I'm actually not trying to be a downer about this at all. I also really believe that working with these creative tensions and creating these well-rounded products, that's also a way for us to think about brand new opportunities, to think about things that we maybe hadn't considered possible. So I think there's a lot of upside in all of this as well.

Reducing risk and ambiguity: the feasibility study

[00:11:55] One other area that I think research and design can really help with is in reducing risk and ambiguity. Deploying AI systems is complex. It's risky. It's often a long, long, long journey. And design and research, the tools, the mindsets, the kinds of methods that we can bring to the table, can actually really help reduce some of that risk. And that can happen again, not just at the point of designing the actual experience, but actually all the way from the outset. So doing things for instance like Wizard of Oz prototyping, to actually understand what level of accuracy or detail is even valuable before going in and doing all of the actual work to model and create that desired outcome, is a really easy way to make sure you're conserving your energy or putting your chips in the right places.

[00:12:53] So one very practical way that we do this in our work is called a feasibility study. Quite often, obviously it depends on where customers are in their journey, but they'll come to us with a hunch, a sort of broad hypothesis, let's say, that they've got some data and they want to achieve some sort of outcome, and they believe that if we introduce machine learning into the mix that that will help, it'll make things better, it'll make things faster. And in order to both actually improve our understanding of the problem, but also to make sure that what we're about to embark on is broadly doable and makes sense, we run this sprint of work called a feasibility study.

[00:13:36] And that involves a data scientist or two looking at the data that's actually available, and understanding where it came from, whether it's appropriate for the kind of thing that we're thinking about using it for, and just having a very preliminary exploration on what kinds of things might be gleaned from it. And then on the other hand, researchers and designers exploring both this desired outcome — I'm sure you're all familiar, this tends to be pretty broad and vague at the early stages of something, so making something say faster or better or more effective, so we dig a little deeper into that idea and try to unpick it a bit and understand what that actually means in their context — but also exploring the general problem and opportunity context. So who are the people that have a stake in this system, the literal stakeholders? What are their values, what are they looking for and so on. And also doing that in a way that is always in constant dialogue and collaboration, so that we're making sure that we're creating something that is both possible with the data but also desirable and useful for the actual context.

[00:14:47] And what that means is, and again gross oversimplification obviously of the actual outcome of this kind of process, but what might happen as a result of this is that we might reframe the original hypothesis or hunch that came from the customer, and find that perhaps it was kind of useful, kind of doable, but actually we found things that are maybe more valuable that could be done with that same data. Or maybe things that were in the data that they didn't realize were in the data, that could be used to address the same opportunity. So it helps us essentially to reframe potentially, but also to just align on what it is that we're doing and to reduce any risk before we go into any kind of actual modeling or any kind of complex, more risky activity.

[00:15:39] So in my mind, the role of research and design can exist throughout the life cycle, or different altitudes of the product essentially. So around finding the actual right opportunity and reframing it, around refining the actual product proposition, so what does it actually do and for who, and then eventually actually designing the experience, so how should it be doing the job and how should it be behaving?

Rethinking what "good" means: consistency, trust and friction

[00:16:13] And on this note of designing the experience appropriately, working with AI also requires us to rethink what good means. And I'm saying us, I also mean that for designers. I think there's a lot of, let's call it design gospel, or ideas about what good design is, that get challenged when we work with AI systems, which as I've said can be harder to anticipate or can be more prone to failure.

[00:16:44] So we might have told ourselves historically that good design is consistent and it's reliable. But as you can see in this example, and as I'm sure you've all encountered in your use of digital products, there can be instances where it fails. And in this case it's failed in a vaguely amusing way. You take a screenshot, you stick it on Reddit, we all laugh, great. But if this keeps happening over and over again, or if this keeps happening not in a scenario where it's an email and I can scroll past it, but it's a mission critical exercise or it's very high stakes, then we have a more fundamental problem. So really then good design becomes more about determining the acceptable amount of failure. So is there maybe a context or a scenario in which failure is unacceptable? In which case maybe we shouldn't use it in that setting, or we should figure out the extent to which it might fail and move forward accordingly.

[00:17:48] Another common refrain I think is this idea of trust, and good design building trust. We also think about trust as a net positive, I think. So we're trying to always engender trust from our users and get them to lean on us and rely on our work as much as possible. But actually when it comes to AI systems that can fail, that's not always desirable. Automation bias is a thing. When we're presented with systems, especially ones that are called artificial intelligence, and we don't quite understand what that means, we're much more inclined to actually listen and take on their suggestions and to rely on those systems, than our own reasoning and our own five senses.

[00:18:32] So these are three separate cars that drove into three separate lakes on three separate occasions, because they were following their navigation over and above what they were actually seeing and their own judgment. And again, this probably wasn't funny for the people that own these cars or drove these cars, but it makes for a headline that we chuckle at and we forward to our friends. But when we imagine systems like this in higher stakes contexts, then it becomes less amusing and more problematic.

[00:19:07] So then again good design becomes much more about us communicating the limitations of the system, or potentially the reasoning for what it's saying, so that humans can ultimately decide whether and how and in what context to actually trust that information.

[00:19:26] And finally, again, a very common phrase that we throw around is, design makes things frictionless, makes things easy. I'm sure you've seen lots of stories like this as well. So given again that AI systems can't always understand or contextualize what we're saying, or can make mistakes and can fail, it becomes then less about removing that friction altogether and more about encouraging appropriate control and respecting human goals and values.

[00:19:58] So those are a very high level thematic set of things we're thinking about. This is a short ad break — no, I'm kidding. This is actually really great, I wouldn't be talking about this if I didn't genuinely use it. We have a human-AI experience toolkit at Microsoft which actually has a set of guidelines and patterns and examples that you can work through if you are working on an AI product or feature, that help guide your efforts to make sure that it's basically behaving appropriately and partnering appropriately with humans. And it goes into a lot more granular detail around some of these themes.

Rethinking how we work: structures, representation and language

[00:20:39] And finally, designing AI also requires rethinking how we work. I think implicit in all of the stuff that I've just shared is not that it's just about design and research, but that it's about showing up in a particular way, and thinking about AI products in a particular way, and doing so in a way that cuts across disciplines.

[00:21:04] And when we are trying to reconcile these different perspectives and these different forces, it's really important for us to be mindful of how we're actually showing up as a team, or how we're actually designing our processes as well. So we need to make sure that these concerns can be adequately represented, but also that we to some extent are breaking down some of the barriers between our disciplines as well. So just because we don't have a user researcher in the room, it doesn't mean we get to throw that consideration out the window, because as we can see that's a very easy way to create this kind of skew and start potentially causing harm or creating things that are not really valuable.

[00:21:49] So that requires us to think about a number of areas. One is around structures and around representation. You may be familiar with Conway's law, but ultimately whatever we design, the systems that we create mirror the structures and the teams that we work in. So knowing that, how can we actually almost flip that on its head and show up to work on AI products in a way that reflects how we want those products to be? So say we're saying we want this product to be human-centered. If there's nobody who is responsible for actually learning, understanding that, and advocating for that in your team, then how can you actually be sure that that's going to happen at all? And how do you make sure that you're not introducing any kind of bias into your process?

[00:22:36] A really clear example of this is when we work in areas where there's a lot of embedded expertise and subject matter knowledge, say in healthcare. It's really important for us to make sure that we actually bring those expert voices into our process, regardless of what we're working on, because it can become quite easy, I think, to go down a bit of a slippery slope, that when we understand the specifications of what we're doing, we maybe get a little bit blinkered by that and we just drive to delivering on that thing. But actually it's really important for us to continue to keep those voices involved in the process of developing these products.

[00:23:16] And extending from that, also how we think about our language, our rituals and our artifacts. So how we think about how we've defined success. If that's about how the system performs technically, or about the business value that it generates, then is that really a shared value, or a shared idea of value, a shared idea of success? Or is it going to be a really easy way for us to again set ourselves down a skewed path?

[00:23:41] How do we also create the right communication and the right boundary objects? So yes, we can invite the right people to the table, so to speak, the proverbial table, or have them in the room. But if we have all of that representation, but the conversation that we're having is still a technical one, then is that really a collaborative and participatory process? So how do we create common language, common ways of working that everyone can feel invited to and included in, so that we actually shape these systems together?

[00:24:16] And also the pace and the cycles of work. So quite common in our world, I work in an engineering organization, so we're very used to the dynamic of very quickly getting to a definition of what it is that we're trying to create, breaking that down into small units that we can deliver, and then getting started on that delivery. Which doesn't always create the right kind of space or dynamic for some of the consideration and some of the creative tension that we talked about earlier. So it's really important again that we make sure that the values that we have, in say trying to create something that is human-centered, actually align all the way down through how we show up to do the work, and we're not creating a bit of a square peg, round hole situation.

Anticipating future harm, and the mindsets we bring

[00:25:06] And the other thing that's implicit, I think, in all of this, is that it requires us to anticipate future harm. That sounds a little bit... well, it sounds simple actually, but I don't think it is at all. In a context where we are usually incentivized to just do stuff, to keep moving forward, to deliver things on time and so on, it becomes really easy for us to get used to just suspending our disbelief almost, and just believing that what we're doing is the right thing to do and just working to achieve that.

[00:25:39] But actually it's really important for us to deliberately create space to consider how things might fail, so that then we can design those things out, design out that potential. And that requires, I think, the deliberate creation of those safe spaces. So doing things for instance like premortems, that allow us to safely air some of those concerns, safely share some of those concerns, so we have the right language and also the right kind of safety, actually, in our teams to be able to raise those concerns as and when they come up.

[00:26:16] So I think this last chapter, there isn't an easy fix to much of these things, but it's much more about some of the rituals or processes and things that we put in place, but also just about how we all show up every day to this process, how we all show up as practitioners. It's much more about the mindsets that we bring and being aware of our biases. So the humility to basically say, what are the biases that I maybe bring to this? What's my agenda and what might I be pushing towards? The awareness to actually ask who's not in the room. You know, I'm here, I'm representing my view, but maybe what else should be represented that isn't here? And also the curiosity to explore alternate points of view even when we think we've reached a good decision, to continue to challenge ourselves with, who might disagree, what might be an alternate view, a disagreeing view on this, so that I can maybe learn from that or challenge myself to consider those other perspectives.

What you can do next

[00:27:22] So with that, I want to leave you with some things that you can do next. There are a lot of existing tools and resources, some of which I mentioned already in this talk, that I would encourage you to explore. If you maybe don't work with AI quite as much yet, or haven't had as much exposure, then I really encourage you to explore some of the key concepts and principles, and actually play around with what's out there. I will say it's not necessarily representative of the full world of AI, but it is super cool. And if you do already work with AI, or have worked with AI features in the past, I really encourage you to reflect on some of your own ways of working. So where are maybe areas in which research and design could do more for you, and how can you foster better collaboration in your team and in your work?

[00:28:13] That's it. Thank you so much, and I'm really looking forward to hearing your thoughts and your comments.

[00:28:21] Host: Thank you very much, that was really great. Unfortunately we don't have time for questions, but I have written down about five or six, and I've just seen that Joe has asked if you can put the links that you just had up on that slide into the chat. So what I'll do is I'll connect with you after this and we'll put the questions that I had up on our ask the speakers channel, because there's just so much came to my mind, particularly around, in the last year my definition of feasible flipped, because I thought image generation and ChatGPT were not feasible, whereas now if that's feasible, what else can we do? So there's just one of those kind of really interesting things that I'd love to get more insight from you. But I'm really sorry, unfortunately we are out of time for today, but I will post those in the Slack channel and I'll invite you shortly.

[00:29:13] Esra: Yes, happy to take it offline.

[00:29:16] Host: Great. So thank you very much, Esra.