Shaping AI Agent Behavior: Why UX Must Own AI Evaluation Frameworks

Melissa Wittmayer at UXDX Community: From Design Thinking to Product Decisions and AI Evaluation. Video: https://youtu.be/lFwbEKc65WQ

Readable transcript: edited from the recording's captions for readability (fillers and false starts removed, punctuation and section headings added). Wording is the speaker's own. Timestamps are positions in the video. Names marked [?] could not be verified against the audio.

What this session covers

[00:00:08] Welcome, everyone. Today's goal for this session is pretty simple. You will leave knowing a little bit more about how to evaluate AI agent behavior, and also things that will help you get started tomorrow, or even today if you want to jump in right away.

[00:00:27] How are we going to do this? We're going to cover a little bit of why UX is really positioned to own this framework, and then walk through an evaluation framework I've used myself. And then lastly, my favorite part, we'll really dig in and talk about the how-to, the implementation piece, and that's what will set you up to really start implementing this as soon as you want. Then we'll have some time at the end for questions.

[00:00:55] First, a little bit about me. I am a director of user experience at Ontra, and I've been working in AI products for a number of years, and most recently, the last few years, I've really been focused on AI agents and designing those. I'm also an avid traveler, so you're going to see a couple of my travel photos throughout this presentation. We'll start with this one on the left, which is a bridge in Norway called the Bridge to Nowhere, and it really became a symbol of how I felt when this project started, when I started working with AI agents and also working on evaluations. I'm happy to say I don't feel that way anymore.

[00:01:41] But I really want to emphasize that I'm not coming at this from a theoretical perspective. I actually learned this the fun way, which was building agents into production.

[00:01:55] First, just to cover it: when I say eval or evaluator, I'm really talking about a structured way to measure how an agent is performing. Instead of manually pulling up threads one by one, you're actually building something that surfaces these issues automatically. And at scale, we're typically using LLM-based judges to do that. I won't go too far in depth into what that is, but here's a resource for you. It's also at the end of these slides, which you will get. I highly recommend jumping into that. It's a great way to get familiar with AI evals in general.

Why UX should own evaluation

[00:02:34] Let's talk about why UX. To do that, I really want to rewind to Q4 2024. I get the news that our next big launch is going to be agents. This is a world where AI agents are just coming onto the scene. I'm immediately into design mode. What patterns are out there? Spoiler: none. What foundations can I create? What components do we need? How are we going to onboard these things? What does engaging an agent look like? What is auditing it? What's that human-in-the-loop experience? There was a lot to unpack and figure out in that moment.

[00:03:13] While I went off and did that, my engineering counterpart was really focused on building the scaffolding of these agents and the evaluation for them as well. He was focused on: are they calling the right tools? Is it pulling the right data? And are they performing in the way they really should be performing? It made sense for him to take on the evaluation efforts and focus a lot of that on accuracy, because that was really the first thing we needed to get done.

[00:03:43] Fast forward a little bit. We're a couple of quarters into this, we're close to launch, and he comes to me, and to his credit, about three to four months before I even jumped into this work, said, "Hey, this is something that product and UX really need to own." What he was noticing was that when we had evals centered primarily on accuracy, it set a low bar in terms of experience. He would notice threads that technically passed and were technically correct, but they didn't feel quite right. So he was pulling them into annotation queues and asking product and UX folks to review. But it wasn't really providing the rich feedback that the evaluators needed to surface the issues to us. It was a very manual process.

[00:04:28] Just imagine someone asks you to go make a picture of a landscape. You go, you come back, and they're like, "No, no, no, the sky can't be blue." So you're like, "Okay, I'm going to go back." You draw a picture again, come back: "No, no, no, the sky can't be green." You go back, draw another picture, and on and on and on. Come to find out you were meant to draw the night sky. What we felt we had was that accuracy alone wasn't giving us the full picture, the full insight into the experience.

[00:05:00] And if I wasn't going to do it, then someone else got to define what good looks like. What we know about UX is that we are really positioned to own mental model understanding. We understand the expectations of users, we're in the research, we know when the experience breaks down. So this is something that we're really well positioned to own. And an evaluator at these points is really just a structured way for us to operationalize that knowledge, but do it at scale.

Building the framework: trust, momentum, sentiment

[00:05:30] That's what I set out to build. And it didn't actually start with a framework. I pulled a thread that I knew the agent had handled wrong, and I wrote a prompt for an evaluator to address all of the issues I saw on that thread. And what I got was a mess. It didn't really serve as actionable insights. Everything was thrown in one bucket.

[00:05:57] So I took another step back. I took a look at the issues in aggregate. I categorized them. And then I ended up building evaluators for those categories. This might sound like a familiar process to anyone in UX. It's affinity mapping. Yay! These processes are here to stay. It's just how we use them that is a little bit different.

[00:06:19] By doing that, I was able to set up a framework that worked for us. What I'm about to cover isn't meant to be prescriptive. I want you all to take it, tweak it, share your findings, continue the dialogue. This is a great conversation to be having. But what ended up working for us was a three-layered approach, and that started with trust, then momentum, and then sentiment. I'll walk through each of those right now before we jump into the implementation piece.

[00:06:49] Starting with trust was a very easy place to start, because we already had accuracy covered, and we knew that trust was really the thing that was going to get people in the door. But even when things were accurate, we would still see drop-offs in places where the agent was technically doing the right thing. We knew it wasn't possible for the agent to be 100% accurate. We knew they were going to make mistakes. And the question really became: why would a user forgive a mistake from an agent? That really came down to behavioral alignment.

[00:07:23] If your agent is aligned with your users' expectations, then they know what to expect. It doesn't feel random, and they're more likely to trust the intent of the agent. So accuracy got us in the door in terms of establishing trust, but behavioral alignment was really what made somebody truly trust that agent.

[00:07:45] That brings us to momentum. We were finding situations where we could get the same accuracy and the same alignment with a human interaction. So then the question became: why are we even going to use this new agent experience? What we found is that agents actually have to alleviate more than just a small portion of the mental load for users. We used to be able to get away with, "Oh, if we use this field on a form, it'll make it a little bit easier, and then customers will use it more." What we found is that it had to take a lot more time and effort off our users' plates for them to use these new tools.

[00:08:26] Once we got these things into production, we had customers that were changing their standard operating procedures to work around these new tools and new experiences we were building. If they're going to go to that amount of effort to change their business, we really need to show the benefit, and it has to be clear what agents do.

[00:08:49] For us, what we saw is that a huge benefit of agents was that they were able to work without distractions. They didn't get interrupted by phone calls, texts, notifications. I think we can all feel this, but for our customers it was really true. I've probably gotten three notifications from my mom already today sharing her Bird Buddy updates. So we really had to hone in on how our agents were going to help with that workflow and alleviate that pain.

[00:09:23] Our goal became having them generate this sense of flow, or, if you were in athletics at all, that being in the zone. And that meant if a user hit a brick wall in any sense, then the agent failed. It didn't matter if the response was correct or if it matched their mental model. If it added more work to their plate, then it didn't work.

[00:09:48] Those two things really set the foundation for us, and then we ended up throwing in sentiment as well. We had the proactive understanding of surfacing things, but we still wanted a signal that could ping us and tell us where to look, and that's where we really relied on sentiment. With this one we were looking more for trends versus individual sentiment cases, which would show us if there were any frustrations that were unresolved, if there was any volatility, things like that.

Starting from your level of access

[00:10:21] All right. Now we'll get into how you actually do this. The first question I have is: what tools do you have access to? Your level of access is really going to shape your approach, but I will say it does not stop you from starting.

[00:10:42] Your access levels can vary, and I'm being a little bit dramatic with this "no access," because you'll still, I'm assuming, have some access to whiteboarding tools like Figma, Miro, Lucid, maybe even an actual whiteboard and markers, pen and paper, things like that. Those are the tools that I'm assuming people will have with no access. With limited access, I'm thinking more of data analytics tools and customer feedback.

[00:11:07] And when I talk about full access approaches, I'm looking at people who have access to agent observability tools. Your teams might have access to this, but I'm thinking more in terms of design and product being able to be in there: agent observability tools, developer platforms, actually being able to get into GitHub and LangSmith and look at these things. We'll walk through each of these layers of the framework and then talk about how you get started from each level of access.

Trust in practice

[00:11:42] Starting with trust. The bare minimum for this trust layer is to identify the most critical expectations you need to get right first. Looking back at the mental model of the user, what do they expect this agent to do? Especially if you're starting with an MVP product and you haven't released anything yet, it's really critical to understand the main things users are going to need this agent to accomplish.

[00:12:11] One of the ways you can do that without having access to these things is to start creating examples for your team in terms of shaping that mental model. This is an example for you. The simplest way that I like to do this is to categorize what kind of mental model it's addressing, and then give an example of an input and an output. We'll see here a customer support agent example.

[00:12:44] If someone is coming in with advanced troubleshooting, and they've already tried things like turning it off and turning it back on, we want the output to match that, so that it's not having them start from zero with troubleshooting again and not having them repeat steps. It's acknowledging that they've already taken steps to try and fix this, getting some of those basics out of the way before jumping into troubleshooting.

[00:13:10] You can see something similar with the operations category, where someone is just trying to figure out how to do something. Instead of just giving them a generic help doc, it's actually trying to narrow down what specifically you are trying to do, so it can be very pointed with its support.

[00:13:31] That leads us to limited access. For this, it's really relying on some of those sentiment signals to point you towards parts of the workflow where the agent maybe isn't following the expected behavior, so you can look at places where there's adoption drop-off. The biggest thing for us was repetition. I remember we ended up changing the UI experience of one of our tools because we saw a thread of a user saying the same thing over and over again: email, email, email, email. Sometimes all caps, sometimes with an exclamation point. That was a key to us: we're not quite matching what this user is trying to get done. That actually led to us changing the experience there.

[00:14:24] And then lastly for trust, full access is really to build an evaluator yourself. I was using LangSmith at the time, and they have an evaluator called the Align evaluator. There are a couple of other options they have as well, but I liked this one specifically for matching the mental model, because it was a very nuanced thing to try to get the evaluator attuned to.

[00:14:52] With this Align evaluator, I started by writing a system prompt, which I'll share with you in a minute. Then I went through and pulled the threads that I felt really matched the mental model I was trying to get it to align to. I scored those myself in terms of whether our in-production agent responded in a way that matched the mental model or not. And then I ran that through the Align evaluator to see where we aligned in some of these cases.

[00:15:25] There are a lot of helpful things here. You can see that alignment score, and that was helpful in terms of, okay, we want this at least better than 50%. We generally looked towards 80 to 90%, but we didn't always have a set number, because we could look at the reasoning for why the evaluator scored it a certain way and not another. We were also able to see the risk level in terms of what this evaluator was getting wrong. Often, if the evaluator was saying, "Hey, this is an issue," and it actually wasn't, that was less risky than there being an issue and it not catching it.

[00:16:08] We eventually trained this evaluator, and then what we ended up doing was adjusting some of our team ceremonies. Depending on the stage of production the agent was in, we would have PM and UX meet for about 15 to 30 minutes, either daily or weekly depending on how far we were in that development process. We would go over things that were surfaced to us and then turn them into suggestions to the full team in terms of how we improve this experience.

[00:16:45] Here's an example of that prompt for the Align evaluator that I used. A couple of things I want to highlight here, and again, you'll get these slides, so don't worry about getting all this down. I started with, "You are an evaluator assessing whether an AI agent's responses align with a mental model." We tried this initially with giving it a little bit more domain expertise. You could think of, "Hey, you're a customer support manager and you're analyzing your team's threads." I found that it took a little bit more liberty and assumed knowledge that it didn't necessarily have, and I wasn't asking it to have. So we ended up stripping that back a little bit and making it more generic in terms of the evaluation it was doing and what type of evaluation.

[00:17:38] Another piece I wanted to make sure to give it was that the agent had access to back-end systems and internal APIs. Without this piece, it kept saying that the agent was assuming something was correct when it wasn't. We didn't have all of that wired up into the Align evaluator, so I needed it to know: you're not trying to look at accuracy, you're not trying to look at whether it was correct or not. We're merely looking at this mental model and making sure that you're matching that.

Momentum in practice

[00:18:13] All right, on to momentum. For no access here, the very basics is that your team really needs to feel that brick wall moment. Whether you do this through storyboarding the ideal experience, or identifying potential brick walls or actual brick walls, you really want to make sure your team is grounded in what it means to move through this and feel momentum.

[00:18:39] I'll show an example of what that looks like with workflow mapping. I'm going to jump over here so we can zoom in a little bit. I just mapped out a generic workflow of submission to review to draft to edit and then to send. And then here I have each of the players in the flow: the submitter, the agent, and the responder. This is intentionally a little bit of a wonky flow.

[00:19:04] What I've done is identify brick wall moments that could do things like impact trust or adoption, and then show the team what might be a better experience. Let's assume that I've identified these ahead of time, but this is our MVP approach, and we're down to iterate, but we've got to get something out. This is a great way to say, "Hey, there are these potential brick wall moments, and we're probably going to need to monitor these areas to make sure that we're not causing issues."

[00:19:39] That brings us to the limited access bit. Once you've got that workflow mapped out, you can start identifying signals that would indicate a brick wall moment. Going back here again, you can look at: are people dropping off here because they have to do more of a manual process of reviewing a summary and then generating? Are they taking too much time from this step to this step? Similarly with these other handoff moments, there are ways you can proactively identify this ahead of time for your teams.

[00:20:18] Then for full access, again, we're getting into building those evaluators. This one is a little bit different, though. Instead of looking at an input and then an output, you're actually looking at a full thread-level conversation, or thread-level interaction, with the agent. This gives you the ability to look at the entire workflow that's being accomplished and say, "Where are the areas where we created a little bit more friction that didn't need to be there?"

[00:20:52] Here's an example of the prompt that I used. Again, it gives it that general evaluator role, but really identifies what momentum means. For us, it needed to be helpful, clear, and needed to have forward progress. You'll notice here that I have a callout of assuming accuracy is a prerequisite. I'm telling you, all of these evaluators were very quick to say, "This is inaccurate. This is not right." So we had to intentionally call out accuracy there.

[00:21:28] The other thing we had to call out with that momentum evaluator, if you look down at one of those last lines in the bolded part, was that sentiment or polite tone didn't necessarily mean there was forward progress. Don't look at the sentiment in order to determine that. We're really looking at how the agent has packaged that message and the next step, and whether it is clearly making things feel like they're moving forward.

Sentiment as a flashlight

[00:21:57] That brings us to the last one, which is sentiment. This one is a bit of an odd one out, but it's intentional. We're not going to go through the examples like we did with the last ones, because sentiment isn't really a new metric for UX. We've been using it for a while to determine what the experience is like. But what was a little bit different here was that we could build an evaluator around it, and we could have it do more thread-level evaluations.

[00:22:27] You're not necessarily looking for a single sentiment, like an all-caps "EMAIL!" from a user. You're actually looking from the beginning to the end: how did it change? What happened? Especially with situations where customers tend to come into an interaction frustrated but might leave feeling either neutral or even happy sometimes. We didn't want that initial indicator of frustration to just have us keep pulling in threads. We wanted to see overall: is the sentiment degrading? Is it improving? Was there a break anywhere here? The intention was really to use it as a flashlight, not necessarily a grade for whether the agent was doing something right or wrong.

[00:23:25] All right. In summary, I couldn't resist; I had to throw some sort of layered object in there. When we look at it all together, we have trust and momentum as your proactive layers, and then sentiment is there for the signal that needs attention. That is the whole framework that we worked through: trust, momentum and sentiment. And with that, I will open it for questions.

Q&A

[00:23:56] Host: Thank you very much. I've never actually thought about what it would take to be on the other side of building an agent, so it was great to hear and walk through that process, because it's a very interesting one, for me at least.

[00:24:11] Melissa: Yeah.

[00:24:12] Host: Just to reiterate, if you have any questions, please write them in on whichever platform you're watching on, and we'll put those questions through to Melissa. One of the things I'm always most concerned about with agents and AI and all of this, and you mentioned it at the start, is trust. The whole concept of AI is that it's probabilistic: most of the time it will answer one way, but sometimes it will go the other way. So after doing all of this alignment work that you've talked about, how confident are you that it will actually stay within the guardrails you're setting?

[00:24:57] Melissa: Guardrails is a big one. We didn't even jump into that one. With the agents we were developing, we had about three layers of guardrails. We had a general guardrail set in place, and then the domain-specific teams would institute other guardrails if they needed to. In terms of that, I felt pretty confident in what our team was doing as far as making sure the right procedures were in place. I think as long as you're taking an approach where you are really trying to confine the agent into a space, and not giving it such broad access to things, it makes it so much easier to troubleshoot, so much easier to fix, and also easier to confine it to a space where you feel confident it's not going to mess things up.

[00:25:57] Host: Sounds good. So the guardrails are the key piece.

[00:26:00] Melissa: Huge. Yeah.

[00:26:02] Host: And we have a question coming from Rocky. "This has been fantastic." Excellent, that's me. "How would you handle proposing to your org that UX owns this process, not the PM who owns the agent?"

[00:26:19] Melissa: I've been really fortunate in a lot of the PM-UX relationships I've had, in that we have taken a very shared approach to a lot of the work. My general approach has been to work really hand in hand and be curious if people do have more ownership over things. Like I said in the beginning, it was my engineering counterpart that really brought me into this and said, "You need to own this." So I do feel very lucky that he acknowledged that.

[00:26:54] Melissa: But I've also found, and this is more specific to engineering, in cases where I've walked in without those relationships and engineering has owned it, that it works to just come in and be curious about it: "Oh, this is what I've done before, or this is what I've heard. What do you think about that?" Or almost taking a UX discovery approach to it: "What's the hardest part of evaluation for you? What are your stuck points? What is going on?" And then figuring out how I can support that, how I can help as a designer. I think you could probably take the same approach with product as well. It's just taking more of a business lens to it versus the technical one.

[00:27:34] Host: Excellent. I love the idea. I'm always surprised by people saying it's so confusing. You're like, "Well, you're UX. Talk to your customers." Just look at it from their perspective.

[00:27:44] Melissa: Yeah.

[00:27:45] Host: I think we've got time for one last question. One of the things I was always thinking of is the pace of models changing. It's a big thing, and every time a new model comes out, it almost increases quality and it's much better. Have you had to upgrade your underlying model? And when you do that, does it change all of this work you've already done? Do you have to do it again, or does it carry over?

[00:28:16] Melissa: In a lot of situations I've been in, I haven't had to own the A/B testing of that. That was a certain person's or even a team's job for our agents. The process there was to do the A/B testing with a model, but once we had these evaluators in place, it actually made it a lot better, pointing to that proactive state. Before we had the experience evaluators, we'd say, "Oh, this is more cost-effective and the accuracy is great, so let's ship it." And then we're like, "Whoa, that experience is not great." We'd find ourselves fine-tuning things after the fact. But when you set the evaluators up, they can actually be part of that A/B testing process, where you run them through the momentum or the trust one and just say, "Hey, can you double-check that if we switch this model, it's not going to break these other points?"