How to Keep People the Focus in an AI-Centric World

Katie Schmidt at UXDX Community: Enhancing User Experience: Reducing Friction in E-Commerce Checkout and Prioritising People in an AI-Driven World. Video: https://youtu.be/krSY7j3n-bk

Readable transcript: edited from the recording's captions for readability (fillers and false starts removed, punctuation and section headings added). Wording is the speaker's own. Timestamps are positions in the video. Names marked [?] could not be verified against the audio.

Introduction and agenda

[00:00:08] Thank you, Rory. It's a pleasure to be here today with all of you, and I'm so excited to share some personal learnings that I've had while working in an AI-centric world at ServiceNow. So today we'll be talking about how to keep people the focus, something that's very important to all of us.

[00:00:28] Just a quick little background about me. I have over eight years of experience in UX research and UX design, and also as a UX manager. For the past two and a half years or so I've been leading an AI UX research team at ServiceNow, really working with a cross-functional group of folks to help drive ethical, human-centered AI experiences. It's not something that was on my bingo card as a UX research professional, but it's something that I've fallen in love with over the past couple of years.

[00:01:02] Today I'm going to take you through three areas and hopefully share some of my key learnings with you, and have some engagement as well. First we'll talk a little bit about human-centered AI, or HCAI, and the UXer: what it is and why we matter as UX to the world of human-centered AI.

[00:01:23] Next I'm going to share with you some principles for AI experiences. These are six core principles that build trust between humans and AI. These are six key things that we've learned over our time doing research around AI experiences, and I'm excited to share them with you today. And lastly I'm going to share a little bit about how to lead teams through generative AI. This has been a crazy, crazy time, and I'm sure a lot of you have experienced this personally, so I want to share just a couple of key learnings about how to actually lead UX teams through these historic moments.

What human-centered AI is

[00:02:01] So let's get right into it. What is human-centered AI, or HCAI? I borrowed this definition from ServiceNow's responsible AI guidelines, but ServiceNow by no means is the pioneer around HCAI, or the first people or business to bring this into how we operate. There's a lot of great information out there in the world.

[00:02:23] But what human-centered AI really is, is the development and deployment of AI systems that prioritize needs, values and well-being centered on the human. So alarm bells should be going off in your head right now. Of course UX is the perfect discipline to help support a very human-centric approach for AI.

[00:02:46] But this definition is a bit academic, so what does it mean in practice? I pulled a couple of favorite quotes for you about the interaction of people and AI. Andrew Ng is a globally recognized leader in AI, I'm sure you've heard his name if you've done any reading up on AI. He says, AI is a tool, it's up to us to decide how to use it. So just like anything else that you might have at your disposal in your UX toolkit, or when you think about a hobby that you enjoy, let's say cooking or knitting, AI is a tool that you bring to the table to create something great, but it's really up to humans to bring their creativity and inspiration to make the best use of it.

[00:03:34] Geoffrey Hinton, also known as the Godfather of AI, also says that the real power of AI comes from combining human ingenuity with machine learning. So again, bringing forward this theme that AI is just a tool. Machine learning is a tool that's available to us, but it's how we decide to use it as humans that really makes it have great potential.

[00:04:01] If there's only one thing you walk away with from today's talk, this is my message to you: the more human-centered we are, the better AI will work for us. This is my thesis, if you will, my proposal, my belief as a UX research manager and leader in this space. I have truly seen that the more human-centered we lean as we're building these AI experiences, the better that AI technology is going to work for us and for our needs. We'll dive into this a little bit more deeply, but keep that in mind as we're learning today.

What UX's role is in human-centered AI

[00:04:41] I wanted to also take a moment and share my perspective on what UX's role is in human-centered AI, and this is across all UX practitioners, not just researchers. These are a couple of things that I've seen in my time managing teams in this area, as well as working with a number of cross-functional groups.

[00:05:02] The first thing that I've noticed is that UX is very uniquely situated to communicate user needs and translate that to design in an AI-centric world. We are already coming to the table with a deep knowledge of how users interact with experiences and products, and in their context of use, and a lot of the tools that we already employ to do our work can apply in AI experiences.

[00:05:31] The next thing I've learned is that we are really helpful when it comes to determining an appropriate and responsible use of AI. This should make you a little nervous. This is an incredibly important responsibility, and it's not just UX that holds this responsibility, but we are a key player when it comes to working with other teams to determine when AI should be used, when AI can provide a significant benefit to users and humans, and when is the appropriate place to bring that AI in. Because spoiler alert, AI is not the appropriate technological solution for every single problem that humans have.

[00:06:12] Another aspect that UX excels at in building these AI experiences is actually designing experiences with trust building mechanisms and guardrails. We're going to talk a lot about trust today in this talk, and that will help set the stage for how foundational trust is to having any adoption of these AI experiences. But we're really good at that already as UXers. We think ahead about how do we get users to have a good experience, and what makes up a good experience, we know, is not just how it looks but also how it feels and what value the user gets from it.

[00:06:49] We're also experts in making sure that AI, or generative AI, is appropriate for a context of use. You'll hear us, who have been researching in this AI experience field for a while, talk a lot about the importance of context of use, because knowing your personas and knowing how those personas are potentially using AI in their context is something that we can help highlight and bring knowledge to for our cross-functional teams. We're also some of the best people to make sure that AI is the best solution to solve a problem, because we're really good at getting to the core of that problem and what's causing those problems or opportunities for our users.

[00:07:31] And lastly, something else that is so crucial with AI experiences, and particularly generative AI, is how rapidly our users change their beliefs, willingness to trust, interest, all these elements that come together to actually test, use and incorporate AI into their lives. Why that's important from a UX perspective is that it means our design approaches, the way we build in guardrails, the way we assess and understand people's willingness to try AI, is going to change over time. So we're a great, well situated group of practitioners to be able to drive that regular assessment of these experiences.

Four principles for UX research on AI

[00:08:21] Now, because I do come from a UX research background, I wanted to dive just a second deeper on what UX research's role is. I fully acknowledge that a lot of us are playing multiple roles on our teams, and so I hope that this can also apply to others who are often taking on UX research roles but not exclusively doing UX research. There are four principles that we've brought to our UX research AI team at ServiceNow to help anchor our best practices, and we share this with our other UX researchers across the company to help them better tackle and understand how different AI experiences can be from doing traditional types of UX research.

[00:09:08] The first principle is to learn, be curious and ask questions. It's really important that when you are thinking about AI considerations and experiences, you understand how this technology changes things like data readiness, bias, risks, and how the state of the technology can also impact the experience for users. So this is a really key thing. You're going to always be learning in this stage, and not necessarily becoming an expert in AI, but having a good working knowledge.

[00:09:44] Second, be the champion of your user. You have to learn those user mental models around AI specifically, and their expectations with AI, and track that rapid change that I mentioned on the previous slide. Of any experience type I've ever worked on, and with any other seasoned UX practitioners in the field, what we're seeing the most is how rapidly these expectations are changing with our users. So it's our priority to be on top of those things.

[00:10:20] Next, AI experiences are all about building trust and prioritizing those human needs. There are real consequences when AI is inaccurate, bigger consequences than when certain mechanisms you're used to in digital experiences don't work. These consequences can be harmful, sometimes they can have bias, and it can really impact people's willingness to trust, especially dependent on how they're using the AI. If they're using an AI in their daily work, which could impact their performance for example, that's going to have a greater impact on their willingness to trust and re-enter an experience.

[00:11:03] Lastly, can't emphasize enough, ensure AI is solving a real world problem. There's this tendency, and it's a very human tendency, to just put new technology on everything, to get excited about its potential, to bring it in, to want people to try it, to use it, to see the value in it. But quite frankly there are just some things that AI is not good at. It doesn't fix every problem that humans have, and so we want to make sure we're solving for real, appropriate pain points with our users.

Six principles that build trust

[00:11:38] All right, so that is HCAI very quickly, and a little bit about UX's role in ensuring that we really create these human-centered AI experiences. Now I'm excited to transition into a very actionable piece of this talk. I want to share with you some of the principles that we have discovered while working on AI experiences, and these principles are the things that we found to drive true human centeredness as we are thinking about these AI experiences.

[00:12:10] I think it's really also important to emphasize that all these elements that we're going to go through — the whole point of bringing these forward and studying them and emphasizing them in the design of our AI experiences is because they are the things, the mechanisms, that build trust. We've talked about how important trust is in an AI experience for your users to adopt the technology, to get the most value out of it, to maintain this sense of human centeredness in these experiences.

[00:12:45] So through many years of research, both on traditional AI experiences and now generative AI, these six principles are what we found to hold the most true as the key mechanisms that drive trust in these experiences. We'll go through each of these one by one, and I'm going to share some specific things that you can do when you're thinking about designing these experiences to help support these different principles.

Control and human oversight

[00:13:19] All right, so the first principle is control. In short, give the human control over the AI. It's incredibly important that the human always feels in control and feels like the driver, where AI might be considered the passenger, the co-pilot, the backseat driver if you will. We want the human to have control over the AI in an experience setting, especially a digital experience.

[00:13:49] Some of the examples of things that you can do to help the user feel that they're in control, and actually give them tangible control, are to create mechanisms so they can ignore an AI suggestion, or hide AI within the experience, or pause the use of AI. So instead of having it be a constant companion within their experience, they can actually pause it. You want people to be able to choose to use an AI output, especially when we're thinking about generative AI. So if you have a prompt, you're asking it to summarize something, and you end up liking the summary and you want to attach that to your work, that needs to be an action that the human has control over and not necessarily something that happens automatically.

[00:14:37] We also want to be able to give users the option to redirect the AI, so give it feedback — and we'll talk a little more about feedback later — but give it feedback to redirect towards what the human's actually looking for it to do. Also remove AI inputs if they are automatically inserted, but also have that option to insert themselves. So these are just some examples, but the whole point is that you're giving the user control over how the AI is assisting them in the experience.

[00:15:14] The next key principle is human oversight. The humans should oversee the AI and not the other way around. When you think about an AI overseeing a human, that's where you start to get these doomsday scenarios and start to roll into a very un-ideal future with AI. So one of the ways that we maintain trust in these experiences and keep it very human-centric is having that human oversight.

[00:15:43] This includes things like monitoring the value, the health, the impact of the AI, and having very clear, understandable metrics that are associated with the AI's performance. It also has to do with general control, like we talked about in the first pillar, having control over that AI and being able to bring it in and out as needed. And of course value measurement: what is the value that we're getting from the AI? That should be determined and ultimately understood by humans.

Transparency and guardrails

[00:16:17] The third principle is transparency: never hide the use of AI. It can be very tempting to put AI in the background, to have these generations of AI, because you know that it provides value in the experience. But at this stage in our journey to create human-centered AI experiences, we don't want to hide the AI from the human user. We want to make sure that there are clear indicators of when AI is at work, where the sources are that it's pulling the information from.

[00:16:53] If any of you have played around with Gemini or Copilot or ChatGPT, you know exactly what we're talking about here, when it provides sources back to you when you ask a question or ask it to summarize something. But when you're extending that to experiences that you're creating and using AI to enhance, you want to make sure that people are fully aware that AI is there at work.

[00:17:19] The fourth principle is guardrails. We want to assume nothing when it comes to AI ethics and responsibility. We encounter this actually quite a bit working within the world of work or employment, business, corporate scenarios. At ServiceNow our customers are often actual end users working at companies, so they are employee users. What we found in a lot of our research is that they are assuming, if the company is providing them with something to use, that it is safe for them to use, that they do not need to necessarily double check whether or not they're allowed to use something, whether or not it's correct, if it is giving truthful information. So it's really important that we as the UX practitioners working on these experiences have an assumption of responsibility for AI ethics.

[00:18:22] Now, there's a number of things that you can do. Building in guardrails into an experience, having the ability to filter data by things like confidence, or output confidence. You can include bias mitigation in the building of your models, and that's not a place that we typically think of UX supporting, but it's very much an important place for us to be a part of, in building those models, providing feedback to it, considering other aspects of the people that are going to be interacting with those large language models. And also just having user warnings, like, yes, your company has given you this thing to use that's generative AI powered, but it doesn't necessarily mean that it's always accurate or truthful, and it can potentially make up answers when the data is not there for it to pull from.

Explainability and feedback

[00:19:17] Explainability is the next principle. This is all about creating context within the design, and content, good content design, within your experience. So making sure that we're defining concepts and terms and managing expectations. You don't want to assume that your users understand anything about AI. We are of course seeing a growing interest in trying AI and applying it for different things. People have much more comfortability, I would say, with AI even from a year ago, but that doesn't mean that they understand what's happening in the experience.

[00:19:59] So you want to work with your team to understand how much guidance is necessary for that type, whatever that user is, whatever their context of use. What is their mental model about AI or generative AI, and what are some things that we can or should do within the experience to help them use AI most effectively? A lot of us in this field, not just at ServiceNow but working across companies that are employing generative AI, are leaning in more to over-explaining, and working with content designers to find the best way to communicate these principles. I do expect that's going to change over time, we're already seeing a shift in mental models, but that's again why it's so important that we keep up with how our users are thinking and feeling about these experiences.

[00:20:52] The last principle is one of the ones that sounds very easy to do but is actually incredibly challenging in practice to implement effectively in an experience, and this is feedback and contestability. This is all about having human powered feedback and reporting. This is about the human interacting with the AI in an experience and being able to create feedback for that AI and drive that feedback experience. We don't want the AI doing feedback on itself. We don't want it to just keep learning from the same things over and over, so we have to provide corrections to it, and feedback, and ways of improving that model.

[00:21:36] Some experienced designers and AI practitioners drive a more human-centric model input, so they'll take end user feedback from the experience — sometimes that's a thumbs up, thumbs down — and they'll take that in more at a less automated way and will use that feedback to drive model improvements. That's a place where UX research and UX design can really play a significant role.

[00:22:09] Other approaches include more automated types of feedback, where the AI is actively learning from a thumbs up, thumbs down for example, with more granular feedback attached, or having a specific reporting tool that automates those changes. So there's a lot of different ways that people are exploring doing it, but the point is that the human needs to be able to give that feedback and say why an experience is good or not, and what's happening with that AI that might not be ideal. And of course you always want to have some way for users to report unethical transgressions by the AI, concerns about the outputs of an AI, or how it's working within a system.

Leading teams through generative AI

[00:22:59] All right, so that was our six principles that we've learned through several years of research that really help drive trustworthy experiences. But the last thing I want to share with you today are three key learnings that I've had leading teams in this space. Some of these things may sound obvious, but I'm going to share with you why they have been such an impactful learning for me leading these teams.

[00:23:25] The first thing, and probably the most important personal learning I've had, is that when you are going through these historic technology moments, as a leader, as a participant in these moments, this is the time to be the most human you've ever been. It's the time to have the most understanding, the most patience, the most thoughtfulness, the biggest desire to connect with others that you're working with, and outside of those that you're working with to best understand your user. This is the time, of any time, to not assume anything about the people you're working with or the people that you're designing and building experiences for.

[00:24:08] It seems counterintuitive, but what I've seen is that we get so excited by the technology and the pace at which we're moving in this field to keep up with it and provide the right support, and we found this wonderful opportunity as UX practitioners to be front and center in building these great adoptable experiences, that we sometimes let those things overshadow the foundational aspects of creating a great team that can do great work.

[00:24:40] So if you find yourself on one of these teams, leading one of these teams, adjacent to one of these teams, take the extra time to connect as humans and to drive a very thoughtful approach to how we do the work, and not just how we execute on the work. I truly believe that that kind of foundation is what's led us to have more human-centered experiences as an end result. So, not the time to not be a people first leader, and actually the time to be the most people-centric you've ever been.

[00:25:21] The next key learning is that you have to really understand the impact of technology on your users. You cannot effectively understand AI experiences without the knowledge and risks and how they impact trust, like we've talked about today. And those things are things like data security, data bias, inaccurate outputs. You can't underestimate how those things can impact that trust in those experiences, and how that long-term impacts users.

[00:25:48] As I mentioned earlier, we're seeing rapid change, and so sometimes, if we had these longer cycles of building and deploying, we are going to hit this stride where we're always behind. We're always behind our users, what they need, what they're thinking, what they want, their expectations with AI. And so if you have a better understanding and lean in a little bit more, not to necessarily be an AI expert, but to be able to understand the language that your cross-functional teams are using and the risks associated with this type of technology, you're going to be that much more prepared to advocate for your users.

[00:26:29] And lastly, make some sacrifices, but only some. With all this pressure to integrate technology, to be the first in these experiences, to drive adoption, we sometimes go too fast. What I've seen is that having a really solid foundation and understanding of your users, the experience, and how we can grow that and strategize that over time, is the key to actually learning quickly and to moving quickly.

[00:27:07] So if you jump right in head first, if you make too many assumptions about your users, you're going to have challenges with AI experiences. But that's not to say that it shouldn't be an iterative approach, or that it shouldn't be incredibly malleable over time. It just means that this is the time to make sure that you're operating as one true cross-functional team, and not a time to isolate as UXers and push through our work without really understanding that bigger picture and long-term impact.

Three takeaways

[00:27:47] It's been a pleasure so far sharing everything that I've learned with you in a short amount of time. If there's three takeaways for you today, these are it. First, UX plays a significant role in the success or failure of a human-centered AI future. If we want the best possible experience, we have to continue to play this role and be uncomfortable with how important our role is as UX.

[00:28:13] Second, we all need to be smarter about the factors that lead to an unpalatable future with AI. Nobody wants the doomsday scenario, nobody sees that as a good outcome for humans. But the way that we combat that is through learning about the technology and how it applies to the experiences that we're creating and supporting. By being educated in that, you can better see how to mitigate some of those challenges and bring a more human-centric driving approach to these AI experiences.

[00:28:47] And lastly, creating a strong team to work on these experiences is crucial, and everybody plays a role in that. Whether you're a manager, a team lead, in your first job out of university, this is everybody's role, to make sure that we're taking the time to connect as humans and really understand what everyone's point of view is, how we bring that together and how we problem solve as a team.

[00:29:16] So, bringing back that thesis of the day: the more human-centered we are, the better AI will work for us. Thank you so much for your time today, and please do feel free to connect with me on LinkedIn.

[00:29:32] Host: Excellent, thank you very much, Katie. Really informative, because we've all heard the horror stories of AI making too many decisions and being too automated, and it not turning out well for people. It's true, scary. So hopefully some people can take some insights from that and apply it. Unfortunately we don't have time for questions, but if anybody has any questions, please do write them in and we'll share those with Katie, and we can probably share them on the Slack channel or something like that. So please do feel free, because Nicole for example has put in her thanks rather than a question, but saying thanks for the information, Katie. Great, thanks so much.