From Design Systems to Interaction Systems: Creating Coherent AI Experiences

Connor Joyce at UXDX EMEA. Video: https://youtu.be/LmSR9vfU-qE

Readable transcript: edited from the recording's captions for readability (fillers and false starts removed, punctuation and section headings added). Wording is the speaker's own. Timestamps are positions in the video. Names marked [?] could not be verified against the audio.

Implicit and explicit systems

[00:00:08] Connor: Hi everybody. This is my first time in Berlin, and I have absolutely loved it. I've been here for about five days and this is a wonderful city, so thank you so much for having me here. I'm very excited to talk about something that I'm really passionate about. It has been a good portion of my life for the last year, and this is my first time talking about it on such a large stage, so I'm really excited both to share it with you all and to get your feedback. What I'm going to be talking about is the journey from a design system to an interaction system, something that I think is necessary in the age of AI that we are currently in.

[00:00:41] I've done a lot of traveling. By the time that I get back to Seattle, where I live, I'll have been in 10 cities over the last eight weeks. It's been a pleasure. I went to a wedding in India; I've still got a little bit of henna that you probably can't see. I've been to a bunch of different places. I was in New York last week for UXDX New York, and like I said, now I'm here. At one point I was sitting in an intersection in Kathmandu on the back of a motorcycle, and I was watching the chaos of this intersection. There are no lines on the road. You have motorcycles, you have people walking, you have cars, you have tuk-tuks, you've got a little bit of everything. And I thought to myself, how is everybody not crashing?

[00:01:20] It looks a lot like this. This is a picture from Delhi, actually, but it's a very similar intersection. And I compared it back to my home city, which is Chicago, up here. I was wondering whether there really are many more traffic deaths in a place like Delhi compared to a place like Chicago. The reality is there aren't. Even though this is what daily life looks like there, and this is what daily life looks like in Chicago, there are about the same number of traffic deaths. And it's because they both run on systems. Delhi and Kathmandu run on implicit systems. There are no lines on the road, there are no real lights like you have in Chicago, but people know what to do. On the flip side, in a place like Chicago, yes, people know what to do, but they run more on an explicit system.

[00:02:03] It connected back to what I've been working on for the last year, which is design systems. Design systems are sets of rules that make doing work easier because people can follow those rules, and they are set with both implicit and explicit rules, as we're going to discuss.

Should I add AI to my feature?

[00:02:20] Why am I up here talking about design systems, beyond intersections and travel? It's because I was on a stage not too different from this in October, and I gave what I thought was going to be my pinnacle talk. It was about a book that I've written and about some of the other work that I did before the design system work. It was this perfect formula for how to develop AI features, and I spent 45 minutes going into depth about this. It was recorded, if you ever want to watch it. I got done with it and I was talking to the audience, asking them what they took away from it. Instead of talking about this framework, instead of all the things that I wanted people to walk away with, the biggest thing that I heard was: "I left thinking, should I add AI to my feature?"

[00:03:07] So simple. I was like, I could have just had that up on a slide and talked about that for 45 minutes, if that's what people really needed to take away. And it really showed that everyone is building AI. Not everybody's thinking about whether they should be building AI, whether they should be adding it into their feature. I believe design systems, when done right, lay the rules for when you should develop AI features.

[00:03:33] Through this talk I'm going to talk about the three steps that I've seen through this and some of the work that I've done with Microsoft. Evolve: changing from a design system to an interaction system. Align: getting that team to really focus on a few main questions, and having that be a guiding light for how you go and build AI into your features. And then lead. Because if you do that correctly, you have the opportunity to become a strategic leader on the product team, whether that's a design system team or someone who is operating like a design system team.

From Workplace Analytics to Copilot

[00:04:05] Let's start with a little bit of history. I started at Microsoft about six years ago, on a team called Workplace Analytics. There I had the opportunity to build those frameworks I was up on that stage talking about, and to think about how to appropriately build technology in a way that satisfies outcomes. After a few years of that it launched; it became what's now called Microsoft Viva. I saw that as a good opportunity to leave and go and write a book about some of the philosophy that I had learned through that. So I left, I did consulting, I worked with startups, and I wrote this book, Bridging Intention to Impact.

[00:04:40] While I was doing that, ChatGPT came out, and I was lucky to have happened to study generative AI as part of some of the work I did in my master's program. So when that happened, I knew I had knowledge I could build on in the work I was doing. I wound up working on a team at a company called BetterUp, where we were deploying AI features across an entire product suite. That history gave me the opportunity to ultimately come back to Microsoft at this pinnacle time for the company, when they're heavily investing in the Copilot product.

[00:05:11] What I learned very quickly coming back to Microsoft is that we have built a single brand, but it's really a collection of features. And that's not a Microsoft problem. That's an every-enterprise problem right now for anybody building suites of AI features. They are branding them as singular products: Google Gemini, Salesforce Einstein, Microsoft Copilot. All of them are branded as single products, but they're really a collection of features that don't really talk to each other.

[00:05:44] One of the primary stakeholders that I found through this work was a team called Fluent. Fluent is Microsoft's design system, and within Fluent there is a team that is building out a Copilot version of Fluent. They are the systems team. They are the ones who can be this guiding light for AI and how you build AI into product. Through that process they became my core stakeholder. I'm a researcher on a systems team; my entire domain is thinking about how to create a unified, singular Copilot experience. And the team that was one of the best matches for that ultimately became the design system team.

What a design system is

[00:06:28] So what is a design system? I had to ask myself that. I had heard about them, but again, my background is research. I've been a researcher my entire career. I've never actually worked on a design system; I'd just heard that they exist. You have things like Segment, which was one of the companies I had worked with. I love theirs; I think it's my favorite branding. It's great colors, and I love the typography. There are things like Material Design. If you've ever built an app, when I did, that was one of the best places to start. Material is a Google design system filled with a bunch of different components. It's entire Figma libraries where you can utilize pre-built elements that are the launchpad for whatever you're developing.

[00:07:12] Similarly, in the B2B space, the SaaS space, Shopify has Polaris. These are all systems of pre-developed components that you as a designer are able to leverage in whatever you're building. And if you're at one of those companies, it's a library that's really intended to be used, because that's how you keep brand coherence.

[00:07:32] Now, the power of a design system. This is an image from an article whose author I'm blanking on, and I'm sorry to them for not being able to attribute it, but I love this Medium article this individual wrote because it really simplifies what a design system is. It creates efficiency, but it starts with having to bring in those components. It starts with giving you parameters in the beginning; it's those implicit and explicit rules. So yes, in the beginning it might take a little longer to get things up and going, because you have to learn the system before you can design the features. But in the long tail, if you design something right, it takes a little bit longer, but then your efficiencies explode, because now you're using a preset of components to develop whatever you are creating.

[00:08:18] With my mandate of thinking about Copilot as a unified system, I saw the design system as a great place to lay out those rules that would then allow our teams to go and build right the first time.

Evolve: AI is a paradigm shift

[00:08:32] Design systems have been around for 15, 20 years, but they are facing a challenge, and that is that AI is a paradigm shift. How far that goes is for us to figure out going forward. Are we going to actually shift to full AI? We don't know. Are we going to only have conversations with bots? I personally don't think so, but there are great thinkers who do. What I do know is that there will be more of a conversational format going forward in whatever we ultimately create. So design systems have to adapt.

[00:09:06] One of the big questions is this: when do we create a chatbot versus when do we create an AI-enhanced feature? You may ask, what is an AI-enhanced feature? You've all interacted with one if you've used Gmail. In Gmail you have that button that says summarize this email. That is an AI-enhanced feature. It's utilizing AI to create that summary, but you don't have to prompt it. You just click a single button. What's beautiful about that is it's extraordinarily easy to use. You click the button and it works.

[00:09:42] But you lose the flexibility, because that button is not going to tell you how tall Mount Everest is. That button is not going to be able to draft an email. That button is only going to be able to summarize an email. Extraordinarily easy to use, not at all flexible. On the flip side you have ChatGPT or Copilot or Claude. Extraordinarily flexible, so you can ask it any query, but not as easy to use, because it's up to you to figure out what query you ultimately want.

[00:10:12] That's a foundational question for product development now, and a design system has a great opportunity to start to lay down the ground rules, explicit and implicit, to guide teams on when to do what. There are a lot of these, and I've interacted with all of them in some way. There is input: how much of the prompting do you do for that person? There is feedback and confirmation. These systems talk to people like we talk to each other, so there are different expectations when you're talking to something that's having a conversation with you. If one of you asked me a question after this and I stood here for 30 seconds and blinked and stared at all of you and then gave you a response, you'd probably be like, "What's wrong with this guy?"

[00:10:58] If an AI system does that, you're probably going to think it's broken, that something's wrong with the system. People expect it to talk to them: I'm thinking about this. If you used o1 from ChatGPT when it first came out, it was one of the first to tell you what it was doing: hey, I'm searching this; hey, I heard your request, I'm doing this. Copilot does that really well now too. We confirm your request, then we tell you what we're doing, and then Copilot responds to you. It talks to you. That would be an example of latency. Error recovery, handoffs, which I'm going to talk about: these are all examples of the types of problems that a design system can work with in this new age of AI.

[00:11:40] The takeaway for the first step, evolve, is that in the past design systems have played the primary role of designing interfaces, buttons, layers, components, and they still have that significant role. But now, to really succeed going forward in this age of AI, to build good AI products, they have to think about how the system acts, how it behaves.

The problem of retroactive support

[00:12:08] If you do that, it begins to break a trend that I have seen at Copilot, and that the other people I work with in the design systems space, or thinking about AI systems as a whole, face. That is retroactive support. A lot of companies right now are investing huge sums of money in their AI features. What that means is these teams feel very autonomous to run off and go and build a bunch of features. They're encountering all the problems we just talked about, those eight that I called out plus many others. They are running into those problems, and yet they're steamrolling right through them.

[00:12:43] They're doing that because they just need to build something. They have pressure on them. They have KPIs to hit, they have OKRs to fulfill. So individual teams are saying, "We'll just go and build a latency experience. We're not thinking that much about it; we're just going to make people wait the best way we think." In other words, they're relying on product intuition. As a researcher, whenever people are following product intuition I get a little nervous, because intuition can lead you to great places, but it can also lead you far astray.

[00:13:13] The pattern I constantly see is: product designs a feature scope. Design goes and creates a specific solution to it, making a lot of critical high-level decisions about how they're integrating AI into their systems. And then the design system has to come in and retroactively clean up multiple different teams' work, creating a single pattern, and then go back to each of those teams and say, "Okay, now we've built the pattern. Can we please recreate this feature to fit that pattern?" Yes, this is a way to get work done, but it's much better to lead. And leading requires, first, as a design system, to align.

Align: a workshop around the right questions

[00:13:54] I did that with the design system team. I conducted a research workshop with them to help them pull out what they could become the strategic leaders of. This started with having conversations with members of the team: internal stakeholder interviews. I also have friends that work for Google and friends that work for Salesforce, so I talked to them too about the things they were facing, hearing the questions that their teams were talking about as design systems. I created a big list of questions: what types of questions could we, as a design system, go out and actually answer?

[00:14:33] From there I held a workshop. In all honesty, it was a three-hour workshop, and the first hour and a half was really just empathy. It was just sitting down and sharing: wow, a lot is changing right now. We are facing huge swaths of change, and as a design system, things really are feeling different because we're interacting with a new technology. Then in the other hour and a half we got deep with those questions. We ranked them, we categorized them, we thought through them. What were the biggest problems that we saw many features facing and trying to answer? And we prioritized those. We left with next steps: these questions that we could go and begin to build different material to answer as a systems team.

[00:15:20] Here are a couple of examples of these questions. I didn't categorize them. What I want to say is that I would love to be up here on stage telling you the answers to these questions, but these are just the questions our team faced, and we're still working through them. This workshop was three months ago. We're tackling the handoff question that I'll reference, and also that bigger question of when to do a feature and when to do a chatbot. We're still at that level. These are other questions that came from this that I would love to tackle over the next year.

[00:15:54] Some of these are going to be relevant for your team; some are not. Instead of taking this list and saying these are the questions the system should be worrying about, it's more about the process. It's more about this workshop that we went through. I highly recommend that every team that has a system in place re-evaluate and consider what questions they ultimately should answer as a group.

[00:16:19] On top of asking these questions, the systems team reorganized around a new model, to more effectively deliver on how they could approach creating a system for AI-enhanced features. They chose to have two teams built specifically around, one, conversational AI, and two, AI-enhanced features, because such a foundational question for everything we're building now is how conversational you make it. They approached that, and now they also have a design language team, more of a strategic team, that's focused on the components themselves. So they still have the core elements of what was a design system, and what is a design system today, and they've repositioned themselves to be able to grow with the product as it includes more artificial intelligence.

[00:17:09] Takeaway number two: now is a great time to unplug and replug, and reconsider how you are approaching your design system, how you're approaching the system of your product overall, especially if you're including a whole bunch of new features or you're changing things because of a new tool. In this case, that new tool is very likely AI.

Lead: the four assets of a systems team

[00:17:33] So we've evolved the system. We've aligned as a team. Now the last part is leading. For this, systems teams have four great assets they can create. There are others, but at least on my team, these are the ones we focus on most. There are research briefs, or design briefs. I call them research briefs; I think most people call them design briefs, but for me a good design brief has a lot of research in it too. This is a document that gives guidance. These are the explicit rules, along with some of the implicit rules in how you write the document: this is when to build a chatbot, when to build an AI-enhanced feature, the questions you should ask yourself. This is the guidance behind the components.

[00:18:23] You have your Figma starters. These are the components; this is what you can choose from. This is more traditional design systems. You have your engineering documents, again more traditional, but the components included in these documents have the guidance in the research or design brief to be able to guide teams. So yes, there's that initial cost we saw at the beginning, because they're going to have to read that document to help decide what they're going to build. But if they do it correctly, they'll be able to go to these Figma and engineering documents and accelerate how they build.

[00:18:59] And the last one is amazing, because it's been one of the most profound deliverables I've been able to create from the research side: starting giant threads with anybody and everybody that's building a component. I've seen it work best with latency, the example I gave of how the system talks back to you. That thread has about 60 people, because everybody's building latency experiences right now. Getting them all to talk to each other and say, this is how we're interpreting the guidance, this is how we're using these components: it's a shared community, and it builds a lot of that community aspect around each set of components, so people can accelerate their delivery of a well-built product.

Handoffs and becoming a compass

[00:19:45] Just two weeks ago I was in a call. Right now one of the patterns I'm focusing on the most is handoffs. Imagine you're in Copilot Chat and you want it to start drafting an email. You can draft that email in Copilot, but at some point you go, now I want it to be in Outlook. It needs to hand that context off to Outlook, so it has that draft and knows who you should be sending that email to. That's a handoff. If I'm working on a Page, a draft document with a collection of thoughts, and I want to have it in Word now so I can polish it, that's a handoff. You're handing the document off.

[00:20:20] I was in a call, and there was a team member from Outlook, or from Office AI. There was a team member from another team, and another team, and each of them was talking about their specific group: well, I need a handoff here; I'm thinking about building handoffs here; I'm thinking about building hellos here. And then I said, "Wait, who is thinking about this from the holistic point of view?" They looked at each other, and then a PM said, "I think that's the role of the design system team." And I was like, "Yes, it is." That was in its own right a huge success, because now we are leading. We are the ones being asked to build the strategic documents at the top.

[00:21:03] The third takeaway here is that design systems used to be libraries of components, and now I believe they're compasses. They are the ones that will guide you to where the team is going. If done right, they have that power in them.

Summary

[00:21:22] We've talked about three main things here. We talked about the importance of evolving from a design system to an interaction system. That is what is going to give your team the ability to meet the moment and effectively build AI at scale into your product. We've talked about aligning, and I think a workshop is a great way to do that, though there are many ways to do it. The whole point is to decide on a collection of questions that systems can answer and become a strategic leader for. And then you need to deliver, and that is leading. You lead by deliverables. You lead by creating the collateral, so that even if a design system member isn't in that call, there are documents out there that teams can go to when they need guidance. If you do that correctly, you will begin to be known as the person that has the guidance, and that in its own right makes you a leader.

[00:22:16] I'll leave you with this. I ask that you all ask yourselves: what are the biggest recurring challenges we see our teams face when implementing AI? That right there can be the guiding light for the workshop that you do. Thank you all. This is my book, Bridging Intention to Impact. I have a newsletter that I'm really excited about, because I've been wanting to build a newsletter about product development as a whole. That is the QR code to sign up for it. It comes out every Wednesday. Thank you all for your time.

Q&A

[00:22:54] Host: Thank you. We'll leave that up on the screen for one more second. Thank you very much, Connor, appreciate it. What an interesting story of your journey beyond just the work. Earlier today we talked about how AI empowers people in different roles, user research, design, product, and how the lines are going to start getting blurry. Do you see that at all being the case here, or is it just a question of the project at hand?

[00:23:23] Connor: I see it holistically, definitely. Already as a researcher I've played a product role, I've played a designer role, so I think even before AI that was true. I think it has been accelerated purely because there are a lot of efficiencies to be gained. I used to spend a good amount of time writing outreach communications and other forms of content just to get the research started. All of that is automated now. With a lot of the tools I use for analysis and other pieces like that, I still play a role, but I spend about 50% of the time doing it. So it's equipping me to do more product tasks, like creating that Slack channel, or Teams channel, excuse me, and things like that that may previously have been the role of a PM. I can handle that now and I can bleed into that area. So yes, I think it's happening, and I think it's going to continue to.

[00:24:18] Host: That connects with a question I actually want to ask. Do you have any anecdotes from your own use of AI as a productivity tool during this process that would probably surprise people? You talk about the first half of that workshop being about empathy, and that's been a lot of the theme today, people getting out their worries by sharing. Maybe there's something there you'd like to share.

[00:24:43] Connor: From this specific workshop, the best use was this. Any time you're building content now, I think it's a great question to ask yourself: how am I going to be able to utilize AI in the development of this content? I remember I took the conversations, I took documents, and I built an agent and asked it: from all of the suite of PowerPoints and Teams channels and things like that, what are the highest-order questions that this team is facing? That helped influence the development of the question list I had. So instead of going into that workshop and saying, "Hey, design system team, tell me all the questions you've heard," I was able to come in and say, "Here's a list of 15 questions. Let's triage these and add more." It was great. There were a couple of people who really took it and added a lot of theirs, and then I came in still with the 15 that I had built from my experience and through the use of AI.

[00:25:40] Host: Before we get to the audience questions, I've got to ask this one. Is voice UI the holy grail? Is that where we're headed? The systems are all going to be gone someday, and it's going to be like the old days: you just talk to somebody who does your bidding.

[00:25:51] Connor: This is my theory on it. I've only been here in Berlin since Thursday, but I know where I live, and I know most cities in America that I visit, and I have still not seen people walking around talking to themselves on their AirPods, on their headphones. I actually do it a decent amount, and I generally feel like people look at me like, "Is he crazy, or does he have earpods in?" So until we as a society change, where it's very normal to have conversations with yourself because people recognize that you're talking to an AI system, I don't know how we're going to make that shift. If you're sitting on a subway, you have to have some sort of interface, because you're not ready to talk to yourself in that public environment. So it's going to take a cultural shift before it becomes an actual UI shift, I would imagine.

[00:26:39] Host: Interesting. I remember the first time I got an Apple Watch and I thought, this is so cool, I can talk to my watch, until I did that in front of other people and they were like, "What is wrong with you?" I like this one, because I like these questions that bring us back down to reality, not working at Microsoft: do you think a company can build a good design system without a team dedicated to this? And I'd love for you to focus on your point about how AI tools empower you nowadays.

[00:27:08] Connor: I'll give you my take, but I also want to say I'm not a design systems designer, so I'm not the most equipped to say this. I'm a researcher, so I'm coming at this from a third-party perspective. I think it is possible to build a set of rules that people can follow. There are a lot of startups that have great cultures that are developed before they have an HR department, or before they have a team dedicated to developing culture, because the leaders live the cultural norms and the cultural rules that they then ultimately embody in a set of actual standards. So I think the same thing could be true here. If you have design principles and design rules, and if you create those implicit rules, which I think is really important, by how you act and how you treat the team and how the team is able to work, if you are able to construct an environment where they are following certain standards because of the culture or the norms around it, then yes.

[00:28:10] But you need somebody to update an actual library. The team that I work with is a significantly sized team, and I know other major companies like Google and Salesforce all have full design systems teams. So is it possible to create something like a design system? Absolutely. It could literally be a Google Doc with some principles that people follow, and that could be a design system. But to have a proper Figma-based library with components that are being updated to hold standards and keep uniformity across a product, you could do that, but someone has to do that work. So it's going to be part-time, or whatever that might be.

[00:28:48] Host: We have another one that's similar to the cold start problem that I asked Mikuela [?] about a moment ago. "Love how you are shifting what design system documents are and bringing documentation into a place where people will see it. What were the most pivotal steps to initiate this strategic shift?"

[00:29:06] Connor: That's a good question. I don't want to take more credit than I should, but I do think it's having research there, since I am not actually a part of the design system. I am a part of the research systems team. Fluent is not my only stakeholder; I have stakeholders across the Copilot system. I've been sitting in Office AI as another team, and I'll work with Office AI, and when they encounter a problem I'll be like, "Well, why don't we bring in Fluent?" I think Fluent themselves do that, and they're a much bigger team, so they probably do it better than I do. But I think that in its own right is honestly just reminding people. Because again, there's the narrow tunnel vision. You have two months to build this feature. It's a net new feature. You're adding AI to it. You have your engineers saying, "Okay, we're ready. We have two months, which means we need to start right now." So people just make decisions in those calls.

[00:30:02] Sometimes I'll watch people make decisions and I'm like, that's a huge decision you just made, and you seem like you thought about it for 10 seconds. So it's really important to have people in the call who can say, hey, remember there's guidance that exists, whether it's a design system or whatever that ultimately is. To point them to it and say: remember, we don't have to make this decision today, or we don't have to make this decision with just the information in this call. We can expand out, because there are probably resources out there. But you've got to make sure people know that, and that just comes down to a knowledge thing. So get it out, get it public, wherever you can. It falls back to everything else in the information environment that we live in.

[00:30:42] Host: There was another question that disappeared, but I believe they were asking a more process question. A lot of people are sharing, coming out of the closet a little bit, about how they're using AI. Did you use AI in the design of any of the actual components?

[00:30:58] Connor: Did I use AI in the actual design?

[00:31:00] Host: Of the system itself. Not you, but the team.

[00:31:04] Connor: That's a good question. Honestly, I don't know. I would love to know. Now I have a question to go back and ask the team.

[00:31:09] Host: And you can follow up later. Thank you very much. I appreciate Connor taking all these questions.

[00:31:16] Connor: Thank you all. Cheers.