Motion for AI: Creating Empathy in Technology

Pavithra Ramamurthy at UXDX USA. Video: https://youtu.be/_lRT6ktfOEQ

Readable transcript: edited from the recording's captions for readability (fillers and false starts removed, punctuation and section headings added). Wording is the speaker's own. Timestamps are positions in the video. Names marked [?] could not be verified against the audio.

Introduction: a designer with a varied skill set

[00:00:01] Hello everyone, welcome to this really great, very interesting topic actually: motion for AI, how to create empathy in AI. And I'm here today as a lead product designer from Salesforce. I've had the opportunity to lead a lot of different projects from zero to one. I'm both a strategic as well as a technical designer, and I've had a hand in defining a lot of our critical products.

[00:00:35] So because this is the first time I'm here at UXDX, I'm going to tell you a little bit more about myself. I am a designer who basically designs for 30% of my time, because 70% of my time I actually perform in more blended roles, in design strategy, but also as a product manager, a frontend developer, as a designer, sometimes even a researcher. And that is always the case when you're working at a bigger company.

[00:01:07] And in my case, I've always loved learning. I hated it as a kid, because if you have to sit and listen to someone drone on and on about an ambiguous topic, how much are you going to pay attention? Like the way I'm doing right now. But later in life I found that scribbling and doodling can actually be really fun and very useful for intense topics, and it actually helped that they coined a technical term for it and called it sketch noting.

[00:01:39] For me, I began my career in character animation back in 2010. I worked in character animation for five years, but then I wanted to explore technology and robotics more, and that's when I dived into a human-robot interaction program. And during my capstone study, when I was doing my masters in human-computer interaction design, I explored robotics a lot more deeply, and during this time I also learned how to code and I enhanced my research skills.

[00:02:10] Then I joined a design systems team after that, for Salesforce. That's when I learned how to think of systems at scale and design for components and platform. And I found that at that time my varied skill set was really useful, because most of the time you're fighting for budget, you also don't have that much support, and you kind of need to perform in this multifunctional role where you do your research, you test your components, your use cases, you work very closely with product and end user product teams, sometimes you have to do a little bit of front-end development yourself in order to make sure your design carries forward. And I needed to find creative and strategic ways to push progress. And this was the biggest learning experience for me, moving from research and code-based robotics work to actual design development work for Salesforce.

Objects, meaning and empathy

[00:03:26] So let's step back a little bit. When I was working on my project, the main focus of my work was to observe humans and their most beloved surroundings, basically how humans engage with the objects around them. And what really stood out was that most of the time humans engage with objects with a certain amount of empathy. There is a meaning, they put so much of profound feelings in even the simplest of objects: the touch, the texture, the color, the memories. Every aspect of an object can trigger something very deep within.

[00:04:02] So it was a display of how humans have a very intimate relationship with the environment around them. Simply put, objects have meaning because humans seek the connection as a way to foster psychological safety. The relationship between objects and people and the motivation behind those things, the meaning behind those interactions, is actually explored more deeply by Csikszentmihalyi and Rochberg-Halton in their book The Meaning of Things.

[00:04:33] The second thing that stood out during my capstone research was how the current age is reflective within the engagement that we have with our surroundings. So technology has become so seamlessly integrated that we have come to expect simple and smart features in the objects that we use around us. For example, with Philips Hue, lights can be integrated with a movie experience, and you can adjust and see and visualize things the way you want to experience, in a very immersive quality, by integrating with such a simple technology. Innovation is a very powerful force that stimulates our senses and imagination, and we seek to enhance what is already familiar to us.

[00:05:22] So let's think about that for a second. Why do we do that? It's because it gives us quick gratification. Designing for technology involves understanding the mental states of the humans who are using them, and to think about the mess within which they would be using it. Maybe someone is distracted, maybe they're looking for humor, maybe they're looking for something more, maybe it's support that they're looking for. If you listen to this very insightful interview with Danielle Krettek[?], the woman behind Google's Empathy Lab, by dscout, they talk about how we shouldn't just be designing for the most productive states of our mind but rather for our most messy selves.

Movement, animation and the illusion of life

[00:06:15] So let's talk about movement for a moment. We see and experience the world in movement. Every action leads to another action, leads to another action, and it evolves into a chain of events, and then together they form a structured meaning and decision. Now imagine we don't see movement. Imagine such a dimension doesn't exist. So you'd be locked in time, there would be no change in time, there would be nothing to actually connect two different moments together.

[00:06:55] Motion, or animation, purely started out as a form of entertainment, to just keep us engaged, to exaggerate and engage in our imagination. And it was a way for us to escape and humor ourselves. Early Disney animators used emotionally charged moments within simple shapes and forms to actually create empathy and make them feel alive and real, and we actually could relate to such shapes and forms, like this flour sack over here. I mean, are you convinced that it's throwing a tantrum, that it's joyful? That is because, using movement, they were able to deliberately create that effect.

[00:07:44] The animators actually detailed their process and wrote fundamental principles in their book called The Illusion of Life, and these methodologies were then translated and reused by a lot of animators in the future in order to recreate those moments. If you actually read The Illusion of Life, it breaks into details of not only how a human might move, and how a set of drawings with those movements might make you feel like that person is real, but rather it actually does that for even objects. How many people have actually seen Disney animations? You would have seen teacups animated, and they feel so real.

The blinking cursor: motion as an interface tool

[00:08:32] But let's think about it for a second from a computer standpoint. With the advent of technology and computers, motion and animation found new applications. It became a very important tool to emote and connect to the user. So here is a really great example of one of the earliest and most effective uses of motion: the blinking cursor. You can actually trace its application back to 1960, by electronics engineer Charles Kling[?].

[00:09:05] So the blinking solved for a very key use case. He actually created it because he wanted to catch the coder's attention and he needed it to stand apart in a sea of text. If you don't know where you need to type next, how would you know where you would go? So this example depicts the fact that not only did user experience design exist all the way back in 1960, but also the fact that motion played a very critical role in providing a very effective solution.

[00:09:40] Now it is actually being used even today, 60 years in the future. But what is critical here is, notice how motion here is actually connecting two different moments in time. It provides a reference for where previously you typed something, and that particular action, it connects it to where you would be typing next.

[00:10:07] So let's go back and think about that for a moment. It provides a reference for where the previous action ended, and for the next action a means of anticipation. You provide an anticipation for the user and you communicate nonverbally that the system is waiting for the next action. This interaction actually further evolved over time, and now it can accommodate suggestions for what a user might want to type next. And this is a universally recognizable language of the cursor: blinking means user takes action, maybe if it's not blinking there is something going on in the background. With AI interfaces today, the typeahead is even used to actually show you where the text is going to be streaming, and it gives you a significance for when a user can take the next action.

Motion in human-robot interaction

[00:11:07] In the world of human-robot interaction, motion has been utilized to actually create fundamental robotic movements. Robot builders, prototypers, they actually utilized a lot of Disney's 12 principles of animation that I talked about in order to instill these more realistic movements, even in robot-assisted therapy, especially in robots like Probo, Paro, Keepon.

[00:11:37] My partner and I, Kathy and I, we designed and prototyped Buddy back in 2017 to provide speech therapy for cleft lip and palate children, and we actually utilized the foundations that were written by the Disney animators, the 12 principles of animation, and we utilized that to create the fundamental movements for this robot. What we also did was we utilized a controller in order to mimic the robot movements in order to test it with our end users.

[00:12:07] And what we observed was that when familiar movements, that are also predictable, are paired with visuals that are approachable, it can go a very long way in creating a more comfortable context for our participants. In our case we observed, especially, you can see this fun moment here, when the robot turns and its hat is almost falling off, but you feel that jiggle, there's that secondary action as it turns, the robot is turning and looking at you. It made our participants feel more comfortable and they were ready to engage a little bit more, because it did something very humane.

[00:12:53] And this was true for both participants, adults and children. They would be nervous when they're first approaching the robot, but the moment it would turn and look at them, blink at them, smile at them, do something very normal, or have this quick cute jiggle action that is likely providing an affirmation, that gave a lot of encouragement to the children who participated in the study, and they were very eager and excited to engage with the robot after that.

Motion in AI interfaces: transparency and interruption

[00:13:24] So what we can learn is that use of motion is instrumental in enhancing user satisfaction in both robots and interfaces. When it comes to AI interfaces, motion can be used very powerfully to provide transparency and better context for what is happening in the background. It doesn't just make things feel approachable, but you can take it a step further. And by that what I mean is, take it a step further to instill faith and trust in your AI applications. For example, when a user requests something and your AI jumps into action, you can actively provide a glimpse into the background process and tell the user exactly what the AI is doing before it produces the result.

[00:14:16] Now it is also important to think about how a user will interact with an AI. A conversation is hardly question, answer, question, answer. Think about how you would talk to your friend. Your friend responds blah blah blah, no no no, you know what I mean? Or, blah blah blah, hey, you know what I think? Or, for me this is how I want to express interest in this topic that you're talking about. So there are frequent interruptions, there might be sometimes anger, there might be sometimes humor. Now this is the mess that I referred to earlier. This is the mess that you need to keep in mind as you're actually designing for a context, even when a user might be talking to an AI interface.

[00:15:06] In most AI models today a user has to actually wait for streaming, or they cannot interrupt, they'll have to press a stop button in order to interrupt and actually change context or request. But it's very important to do so, because in a natural environment where someone is requesting something from your model, there's a likelihood that they might want to change context, or they are not getting exactly what they're looking for in the streaming that they are seeing. So how might they go in and do that? Right now, if you actually press the stop button in most models, the context change is not taken into consideration as well. So because interruptions can happen more frequently than what you expect, you would definitely need to actually think about that as a main part of the interaction model that you're designing for your AI applications.

[00:16:08] If anyone has been following the recent updates, with GPT-4o you can actually interrupt, and it is such a game changer, because if you follow those interactions it actually feels very close to a natural language conversation flow.

Speed, easing and pauses

[00:16:30] Another thing to keep in mind is controlling of speed. Controlling the speed of an interaction is a way to mimic natural conversation. A slow-paced start to streaming text provides a way for a user to scan the text first, and then slowly increasing the speed and cushioning it as you bring more and more text in actually helps the user get an understanding of the block of text without feeling overwhelmed.

[00:16:59] Now this is something in traditional animation that we call ease in, and this has been applied to even large chunks of text. You will see that these are a lot more digestible, and when paired with a combination of being able to interrupt, or even providing control over the speed — provide a control to your user to change the speed — that'll actually help them take control a little bit more, and they will feel like they are in control of that conversation.

[00:17:32] Pauses, that brings me to the next part. Pauses are actually a very, very important part of conversational flow, and it also provides a space for you to think and reflect. Now pauses can add more thinking, if a model needs to take a moment in order to process something in the background. Like in this example, we have "finale incoming, drafting description"[?]. It tells you what is going on, but it is also taking its time because a background process is in place. So in that case you would want to provide more context to what is happening, but also you provide that extra pause. And here latency will not be a problem, because you're giving the user a moment to think about what they have requested, and you're creating a natural amount of pause that a user might be used to in a regular conversation.

[00:18:22] You can also provide visual affirmation. So it can be a combination of text and visual, it can just be text. Most of the time, pairing visuals and text does a great deal in providing better context, especially since you're seeing things visually, but the text is going to make it more accessible. So it's always better to pair the text and the visuals, even as you're providing these pauses within your AI model, and it's a crucial part of making motion accessible.

The uncanny valley in AI interfaces

[00:18:53] So where does the treasure lie for the natural, and when does something become creepy? Anthropomorphic robots can feel very unsettling if you actually see them. That's because they look like dolls who have come to life, with their dead eye stare, and this is what horror movies are made of. And they are so successful, and yes, we all enjoy watching Chucky. So this is technically what we call uncanny valley, but it can be very, very hard to detect uncanny valley in an AI interface. But it is also very important to do so, because this can go a long way in making or breaking your users' trust.

[00:19:41] And here's a way to recognize uncanny valley. In all those AI generated images or texts, they may feel and appear hyper real, but there's something lacking in depth, something just doesn't sit right. For example, here is an image that Midjourney created for people eating spaghetti. It's super hyper polished, it looks so well done, but when you take a closer look something is just not right. What is going on with the spaghetti? Who's eating? Who are there, people? So many questions.

[00:20:21] And this can happen to text too. Once I actually requested an AI model to generate an introduction for me, and it nicely gave me an introduction that described my skin tone as wheatish[?] that could rival a summer sunset. I mean, I would never introduce myself like that, but if you want to be poetic, I'll take that. But it's awkward.

[00:20:47] And more specifically, animation has a way to make uncanny valley more emphasized if you're not careful. Think to all those interfaces that overindulge in a particular role. Who here remembers Clippy? Yes, our favorite assistant took the role way too seriously, had to get booted, because even when a user was idle or was probably very busy doing something else, Clippy had these moments of, knock knock knock knock knock, can I help you? It had these creepy characteristics that felt very overbearing.

[00:21:31] And in today's AI interfaces uncanny valley can really manifest in similar ways, like popup interruptions that just offer you insights, or random suggestions, untriggered. Imagine you're typing text and alternate AI generated text just comes up whether you ask for it or not. Maybe you set up a kind of, this is the level that I'd like, polish in my text, but whether you triggered it or not it comes up and it tells you that this is how you should represent yourself. These can be very awkward, because it's trying to mimic you but at the same time it's not you. And it's even worse when it changes the context of your text. That has happened in a lot of ways, like when we feed in research data and it takes that up and starts hallucinating, changing the context of the text. All of these kind of take it into the uncanny valley territory.

Human in the loop, and what motion does for productivity

[00:22:35] So one of the valuable AI related projects that we are doing at Salesforce is assessing how to design and implement AI with human in the loop. Our research has revealed that having humans at the helm is actually very, very important in increasing the quality and for building the trust and accuracy with our users. So the ability to give users control over an AI tool is very crucial.

[00:23:07] When it comes to motion, you might actually do that by providing a way for them to control the speed. So this is something I referred to earlier, but here's an example. So you put the human at the helm, they tell you exactly how they want to see and what kind of text they want to see, how many times should the text check in. Feedback responses and bias, providing biased responses, these are all part of putting the human in the driver's seat. But with motion what you can do is you can take it a step further. Apart from just giving control over the speed, you can also give control over the frequency of certain kinds of responses.

[00:23:55] Our research across the board for both robots and interfaces shows that motion helps increase productivity for users and actually enables a much better, more accessible user interaction, even for differently abled users. We performed A/B testing with end users where we showed them an interaction that did not leverage any motion, and we showed them another one where it did. And the difference was that we studied that the users who utilized the UI that leveraged motion, they were 11.5 times more productive. They could finish their task that much faster, and this was true even for our differently abled users who needed the extra support and accessibility.

Systemizing motion so teams can scale it

[00:24:51] So if that is the case, why do organizations deprioritize motion? Well, that is because motion implementation is not easy. It's a big hurdle. Lack of skill set, lack of developmental capabilities, lack of dependable and implementable tech stacks or libraries, patterns, existing patterns that can be reused. These are all things that are not easily usable or available out there.

[00:25:22] So how can organizations overcome this? Well, by prioritizing development of certain kinds of artifacts, which is basically systemization. Invest in systemization efforts. This involves creating principles, guidelines, artifacts that set up teams to maintain consistency and efficiency, allowing for faster recreation and enablement of your product teams.

[00:25:48] So at Salesforce we actually created a suite of enablement artifacts. We wrote foundational principles, guidelines, patterns, code implementations, prototypes that can be reused, and these were made available across multiple channels. We also provided education modules to actually help scale teams better.

[00:26:10] Over the span of several releases our team established a design to development workflow using a readily available pattern library on Figma, on Storybook and on After Effects, and prototypes on CodePen. These were all referenceable and usable. And we also rolled out knowledge shares, workshops, and we incentivized education modules through our public facing Trailhead knowledge portal. And we published blog posts and case studies, and we informed on how teams can actually scale these patterns into their end user products, which helped not only set up the system but also scale it more effectively and efficiently across to a lot of our critical product areas.

[00:27:01] So creating a motion system was actually only the beginning. We are now quickly innovating and scaling our motion system to our AI pattern library. The AI pattern library is coming in with baked in motion that can be utilized by teams who want to leverage our Einstein AI copilot in all of their end user interactions that they are building.

[00:27:30] Okay, so today I actually shared several insights on how and why motion is very essential for designing for AI technology and for robotics, as well as insights into how you can systemize all of these and scale it for your end users. So I really hope I was able to inspire something. I don't know where all of you are in your motion design journey, but I hope I was able to help you reflect on where you are in that journey, as well as your AI journey, and how you can put those two together and why it is vital to do so.

[00:28:05] And I'd also like to give a big shout out to all of my collaborators at Salesforce, and my partner who worked with me on the robotics projects, and our engineers and AI partners, and thank you to all the UXDX organizers for giving me this opportunity as well. So thank you for being here today, and I hope you were able to take away something.