Data Visualization & Generative AI: Human-Centered Principles for Building GenAI Data Experiences

Kent Eisenhuth at UXDX USA. Video: https://youtu.be/SqailbRWuOM

Readable transcript: edited from the recording's captions for readability (fillers and false starts removed, punctuation and section headings added). Wording is the speaker's own. Timestamps are positions in the video. Names marked [?] could not be verified against the audio.

Why data visualization

[00:00:17] So, who's ready to talk about charts, graphs, data viz? Who likes this? All right, awesome. Well, they say a picture is worth a thousand words. And as generative AI products became popular, I couldn't help but wonder, how might charts, graphs and visualizations enhance these experiences?

[00:00:39] Today I want to share more about my time working with the Google Fitbit team, now known as the Google Health team, as we explored this opportunity space. Now, it's not just about data viz, because I'm going to share why that user-centered design process, the one we all know and love so much, is relevant now more than ever. I'm going to share a few lessons learned as we worked with generative AI models and people. And finally, I want to provide a few thoughts on the importance of attention to detail and design craft. All of this is going to be packaged in a set of eight guiding principles that you can take home and apply to your own work.

[00:01:18] So you might be wondering, why data visualization? Well, it turns out the human brain can process images exponentially faster than text. And I preloaded this slide here so you can all take a look at it. We're starting out with a bit of a thought activity. I want you to tell me, how many times do you see the number 11 appearing on this slide? Now, I made it kind of difficult. It's a lot of text smooshed together. We have a lot of single zeros and ones neighboring each other. I'd have to imagine you're scanning the individual rows, and it's taking you some time to arrive at that total. I think we can all agree it's a laborious task.

[00:01:56] Now, let's look to some graphical treatments, some classic graphic design updates to typography, color. We're going to change some highlights here. And now you'll notice, when we make those design adjustments, it should be almost immediately apparent that there are five instances of the number 11 just jumping off of this slide.

[00:02:18] So why is that? Well, it turns out as human beings we are visual creatures, and we're all born with this innate ability to pre-attentively gather, accumulate and process information that's presented by the environment around us. And with this pre-attentive processing capability, we can almost instantly spot trends, patterns and outliers in all of the images that we see.

[00:02:43] So when we look at an example like the one on screen now, it should be almost immediately apparent that there are three visually distinct patterns jumping off the screen. We have one that's defined by the size of the shape. We have one in the middle that's defined by the use of borders and fills. And we have another out on the right that's defined by the type of shape, circles and triangles.

[00:03:06] So I want to zoom out here for a minute and think about these last two examples. We talked about a lot of different visual characteristics. We talked about font weights, highlights, color shifts, the size of shapes, fills, borders, the types of shapes like circles and triangles. All of these are examples of visual variables, visual characteristics, visual encodings if you will. And they serve as the building blocks for all charts, graphs and visualizations. And when they're arranged in just the right way, we can create a picture of data that's understandable in a way that we might not otherwise be able to comprehend the data set with.

The opportunity in a generative AI experience

[00:03:47] So let's think about how this applies to generative AI now. I think a lot of us have worked with chatbots before. They're fairly verbose. They like to talk. They like to produce a lot of text. And this is an example where I was asking about a recent trend in my sleep. And I think we can all agree that if we were to add a picture of data to this particular example, now it becomes more immediately understandable.

[00:04:14] And arguably we can even see more here. So through this picture of data, I can see that my sleep score is trending down. We have all of these wonderful visual encodings and visual variables arranged here to show this particular picture. Arguably it's a more robust insight, because now I can see individual measurements, I can see my annotated high and low values. There's a lot going on here. So this right here is the opportunity for data visualization in a generative AI experience.

[00:04:41] Now of course we corroborated this, my team, with our foundational research. And for our customers specifically, we knew they were interested in seeing the unseen relationships in their seemingly unrelated health metrics. That triangulation was going to be key for them. So two foundational UX research studies here produced very strong signals for us. So we forged ahead and we thought, how might we include charts in our generative AI conversations?

[00:05:10] Now, I want to make something very clear before we move on. This was not a project that focused on enabling people to go out to a chatbot and ask it to create a picture or an image or a chart or a visualization. This was all about building a product that would let you ask questions about your health and wellness in plain language, and the product would decide if, when and how to provide a picture as part of its response to you in the conversation. And the goal was to drive engagement, to drive comprehension, even to increase the delight factor of using this particular product. We wanted to make it memorable for people.

Starting with the questions people ask

[00:05:49] There was a lot of ambiguity, especially a few years back when we were starting to work with generative AI products and large language models. And I want to talk a bit about how we were able to navigate that ambiguity. What did our UX process look like?

[00:06:02] And really, for us, it all started with the people, our customers. And we simply looked to them and we asked them, if you had access to a fitness-style chatbot, what kind of questions would you ask it? What would you be interested in learning about yourself? What would you trust it to respond with? And of course, you can imagine, we surveyed over 2,000 participants in this study. We received a ton of different responses to that question.

[00:06:28] But there was a group of questions that were very important to us, and they were the questions where people were looking to learn about themselves, health insights. And we started to write those questions down on sticky notes, and we arranged them based on their intent.

[00:06:43] So when people were looking for an insight, first it was important for us to understand, what type of insight are they looking for? We have people who are just interested in understanding, how is my health trending? How do I sleep, is it getting better or is it getting worse? We had active people who are interested in their personal bests and personal worsts. When did I have the best runs? Maybe when was my sleep not so great? And they wanted to see a ranked list of that. We also had folks that were interested in understanding correlations in their health metrics. How did they relate to one another?

[00:07:16] Now, if we unpack this a bit more, each insight type has its own set of characteristics. For example, there's different flavors of correlations. And what I mean by that is, immediate correlations, for example, could look something like this. I had a sudden spike in my physical activity yesterday. I want to know if there's a correlation between that and the quality of sleep I was able to get last night.

[00:07:42] There's long-term correlations. I'm on week six of my new workout plan and I want to know if there's a correlation between this increase in sustained physical activity and my next day resting heart rate. I want to see how those two variables correlate to one another.

[00:07:58] So only until now, until we understand the intent of the questions people are asking of the product, in the case where we're responding with insights, ideally what types of insights and what are their characteristics, only now can we start to potentially think about how to visualize that.

[00:08:14] Now, charts, graphs and visualizations have been around for hundreds of years. And there's a lot of research out there, there's a lot of best practices, there's a lot of vision science backing the decisions we make around when to use specific charts. And what we started to do is think about those best practices, and we started to map questions through this framework to responses where ideally the GenAI product would consider which type of chart to use to visualize that response.

[00:08:43] Now, we could have stopped there with best practices, but for us it was important to understand and learn about which types of visualizations do our audience understand. What is their level of data visualization literacy? What's going to resonate with them and be memorable and effective?

[00:08:59] So we went through this process of testing each of the different insight types, and this framework now provided a roadmap for a lot of our research. So we started with trends, and there's a lot of wonderful visualization techniques out there that help us visualize changes over time. Here's an example set of that. We have our area chart. We have our bar histogram. We have a line chart. These are classics. You've probably seen them in a lot of data visualization books. You see them on a lot of dashboards.

[00:09:28] We A/B tested them, and our team had a few hypotheses here. We thought the line chart was going to test the best, and there were many reasons for that. If you look at a bar histogram, visually it's more difficult to compare the heights of bars across the time series. It's easier to see the slope of a line. Line charts do not have to start at a zero value for their Y axis, so when they start to be reduced down to be visible on a wearable device, they're easier to read and easier to see some of those nuances in those trends.

[00:09:57] And instead of stopping there with assumptions, we actually tested this. We ran a survey, in this case with over 600 participants, and of course the line chart won. In this case our hypotheses were validated. But as we started to run this process over and over again with different insight types, we ran into a few surprises.

[00:10:18] So for the first guiding principle I want to start with today, it's: start with the questions people are asking of the product. Think about the insight first, and then everything else will follow from that.

Choose charts your audience recognizes

[00:10:31] Now I want to talk about some of those other lessons learned as we ran this UX process over and over again with some of these different insight types, considering different visualization techniques in our product. And the first guiding principle, this is maybe arguably one of my favorites, is: always choose charts that your audience recognizes. Think about their level of data visualization literacy, and there's ways to evaluate that.

[00:10:57] So when we started this work a few years ago, we just wanted to know, first of all, do basic charts actually add value? Now, from little on up in grade school, we were all trained to learn how to read your classic bar charts, your line charts, your area charts. And we did a quick study to understand, can a chart like this add some value?

[00:11:19] And it turns out when we added a chart like this bar chart to this particular insight, we ran a study with over 2,100 participants, there was actually a 30% increase in comprehension of this insight when we added the chart. So for all you data viz skeptics out there, even if you're a data viz hater, I think that's a pretty strong metric to consider, that pictures can add a lot of value to conversations.

[00:11:46] So, bar charts for the win in this case. It's like, all right, green light, we're going to forge ahead. Now, we tried other chart types as well. And it turns out our specific consumer audience wasn't used to seeing technical or scientific visualizations. So a scatter chart for instance, like the one on the right, didn't really resonate so much with them. It was hard for them to interpret. They weren't familiar with those visual metaphors.

[00:12:07] But when we started to think about visualizing it in a way that's a bit more relatable, like using calendars or tools that we are used to seeing in our everyday lives, those types of metaphors worked a lot better for our particular customer base. So again, choosing charts that your audience recognizes.

Sometimes the best chart is no chart

[00:12:27] Now, this is probably my favorite. I did love the last guiding principle, but now we're getting into the cool one here, because sometimes the best chart is no chart. And it's important to also consider chartless alternatives.

[00:12:42] So we learned about this a lot when we were thinking about how to visualize correlations. If I think about the framework, how we show how one metric might correlate to another, we had a lot of different types of visualization options to choose from. We tried punch card charts, heat maps, we tried scatter charts. We even thought about how we might visualize correlations and trends over time using multiple line charts.

[00:13:09] And we mocked these up and we tested them. We ran a study with over 2,000 participants, all these different visualization techniques. We even went as far as thinking about, where should the chart be placed in the chatbot's response? Should it be inline like we're seeing out on the right? Should we just kind of stash them at the end of the response like we're seeing in the middle? And in all of these studies we had a control, and that was a text-only response that included no visualization.

[00:13:35] And this bowled us over, because for this particular type of insight, the text-only response consistently tested an order of magnitude better for our customers for comprehension than any of the alternatives that included a visualization. So we thought, oh my gosh, this is it, we blew it. Triangulation was key here. This was our big opportunity in visualizing correlations. It's over.

[00:14:00] But we also knew that the human brain can process images faster than text, just like I had mentioned at the beginning of the talk. And we started to embrace that principle a bit. And we wondered, well, maybe we should move away from traditional charts for showing a correlation, and maybe we should think about how traditional design elements, typography, imagery, iconography, just those classic graphic design elements, could be arranged to spotlight this particular correlation.

[00:14:32] And we came up with this idea for creating an infographic where we just leaned into good typography and iconography. And we came back with this design, and when we retested it again with another 2,100 participants, this approach actually tested an order of magnitude better than that text-only response. So all right, we're back in business, full steam ahead with data visualization.

[00:14:56] Well, we encountered a surprise in working with generative AI models, because in cases where it was creating this infographic to complement its response, we noticed that there were cases where the order of the text in the infographic was the reverse of the order of the insight it was communicating as part of its text response. So now it looked like we were showing two different insights that opposed each other when it was really telling the same story. Now, we could have given up there, but we also knew that, because we had confidence in this design direction, we could use our research to convince our engineering teams and leadership to invest more resources in fixing some of these issues in working with the AI model.

Guide people through their data

[00:15:37] Now, in that same kind of study, we learned about the importance of guiding people through their data. How might we place less onus on people to interpret their own charts? So I'm going to share a bit of an example here. This particular visualization shows an increase in a trend in step counts. It's kind of bumpy.

[00:15:58] And we could have stopped there, but if we start to think about how we might add some of those design elements, highlights, inline labeling, annotations, now inline, this becomes a lot easier to read. We can actually point to the interesting aspects of the data. We can annotate it. We can do the math for you and show you a more robust picture of that data.

[00:16:20] So you might be wondering, well, how do we know what's interesting to our customers? And when we're working with generative AI products, at least in this case, the answer was simple. What's interesting to them is the very questions they're asking of the product. So we already know that information and we can highlight that as part of our response.

Facilitating data exploration

[00:16:41] So we also thought a lot about how to facilitate data exploration. Let people get into the metrics and the data points and play with them and kind of come up with their own conclusions. And this is also another key guiding principle for us, the importance of this.

[00:16:58] Now, to investigate this opportunity a bit, we looked at two famous cartoon characters. We thought about how might Bart Simpson use this experience, and how might his sister Lisa also use this experience. So think about Bart Simpson. He's probably coming into the app, he's going to look at a few metrics, he's going to look at a few summaries and say, okay, I have the insights I need, and just kind of peace out, and that's it, it's over.

[00:17:22] But his sister Lisa is much more inquisitive. She's much more academic in her approach. And Lisa probably wants to see all the individual data points rolling up to each of those metrics. She wants to know, what's the evidence to support the claim made by the LLM?

[00:17:37] So we set up a study. We wanted to sniff out in our audience, what's the breakdown of Barts and people that fit into the Lisa profile? And it turns out it was a 48 to 52% split, almost even. 48% of our audience identified with Bart Simpson, the other 52% identified with Lisa. Now, we didn't just directly ask them, we set up some questions to kind of sniff this out and we were able to do so.

[00:18:01] Now, here is the key signal for us. No matter if you were a Bart or if you were a Lisa, 88% of respondents expressed a need for further data exploration. Strong UX research signal for us. So what we ended up doing was we immediately unlocked the interactivity in our charting library that we were using in this product. So now in generative AI's responses, people could go in and tap individual data points and see each individual measurement, explore it a bit more.

[00:18:29] But at the time, a few years ago, multi-turn interactions were becoming a thing with generative AI products, and we ran a few experiments to see, how complex can some of our follow-up responses be? Can we prompt people to maybe show a more complex chart, and is that going to be useful? And what we started to learn was that if we led with a chart that's relatable, we guided people through that data experience, then we learned that there actually was an appetite for seeing a more complex representation of that response that people could then explore.

Embracing scale

[00:19:01] So another thing that we learned, and this comes to our next guiding principle, is this idea of embracing scale. When we're thinking about UX for consumer products, it's easy for people, especially a lot of visual designers that I've worked with, to reduce a complex data set down to a simple line chart or a simple bar chart, because it's beautiful minimalist design.

[00:19:23] But we have to ask ourselves, is it responsible for us to do that? Because we might not be showing the full picture of people's data. So we ran a few experiments here and I'm going to share one on the screen. This is actually showing 26 weeks of sleep data. We're trying to see some patterns in that data. So each line represents a week, and in this case we can see that this person tends to get more sleep as you go into the weekends.

[00:19:49] Now, something interesting about this: once we hit Sunday, which is at the end of our timeline, this person's probably stressed out in their job, they're getting the Sunday scaries, and you can see their sleep's kind of maybe not as great as it was on Friday and Saturday night.

[00:20:02] Now, instead of just representing this as a line chart and kind of ghosting out some of the other measurements in the background, we can actually highlight some interesting sub-patterns in this data as well. So if I was to follow up and ask, well, what do the outliers look like? Do I get better sleep when I'm on vacation? We can actually start to show that inline and highlight some of those anomalies. So these are all a few lessons that we learned in our different experiments that were turned into these guiding principles as we applied that UX process in working with the generative AI product.

Design craft and AI slop

[00:20:35] So I want to switch gears a bit, and now I want to talk about the importance of design craft. So who considers themselves a maker in this audience? Let's see a show of hands here. Who loves craft and just making great experiences and the attention to detail? Awesome. Okay, so this segment's for you.

[00:20:50] So there's another guiding principle around the importance of exceeding expectations. And when we think about that attention to craft and detail and what it means in this era of AI, that attention to detail, that originality, that authenticity, that's what's going to separate the great products from the AI slop that's inevitably going to be out there and ubiquitous in the future.

[00:21:15] So how do we ensure that our work doesn't result in AI slop? Well, my team took some inspiration from a team in Google DeepMind. And this particular team set out to consider how they might use an AI model to create an animated short film. So the film is actually called Dear Upstairs Neighbors, and it's about a girl who was kept awake at night by her neighbors in the apartment upstairs.

[00:21:42] And as she laid in bed, she was trying to sleep, her mind started to wander off. She was like, how could they be making so much noise up there? What are they possibly doing? And her imagination ran wild until eventually she just explodes in rage.

[00:21:54] Now, a team of former Pixar animators got together to make this film. And they had a very specific design style, a very specific visual aesthetic style, and choreography style in mind that really was original and new, and they were going to use it to properly tell this story and capture the essence of this.

[00:22:19] And it turns out, when they were working with the generative AI model, models are only as good as their training data, and at the time models were trained on live-action motion. So when I talked to Glenn Entis, who is the executive producer on this project, he said we could have done it one or two ways. They could have just pressed the button and let it produce something, and that could have been good enough.

[00:22:38] But instead, because the team wanted to make sure it hit all the right beats and we could capture these scenes, they spent a lot of time fine-tuning the AI model that they were using, the Veo model, to understand what exactly it was that they were going for, to understand that original design style, the attention to all the detail that the animation team put into it.

[00:23:00] So at this point I couldn't help but wonder, well, why use AI? Why not just do this manually? This seems kind of silly. And what ended up happening was the use of AI was really important in this particular project, because the team spent so much time focusing on the design detail and all of the finer elements of the experience, and they spent the time training the model to understand it.

[00:23:23] Together, working with the AI model, they were able to do something truly extraordinary. And the example here is in a lot of the transitional scenes where this girl explodes with rage, it was almost as if Jackson Pollock painted a picture, and the AI model was able to individually animate each blob and mark on the painting in a way that captured the essence of this girl's rage and the story.

Making charts look uniquely Google

[00:23:44] So I think there's something we can all take away from this particular example, and certainly my team at Google Fitbit did. We started to think about, well, what makes our visualizations look Googley? What makes it look like a Fitbit chart, or an Android chart? And at the time, our expressive design language was coming out, and if you look at these examples, there's bubbly typography, there's expressive design elements, there's fresh color palettes. And we wanted to make sure that our visualizations match that aesthetic so they would look uniquely Google, uniquely Google Health and Android.

[00:24:20] So we thought about how to bring those design elements into our visualizations. When can we use curves in a more gestural representation of the data versus maybe something a bit more technical-looking, especially if we're looking at labs or certain health metrics? We thought very carefully about how to apply that.

[00:24:36] We also thought about some of those little big details, those details our users might not even notice. For example, in our heart chart we show different heart zone thresholds, and you'll notice there's dotted lines that delineate those thresholds. Well, the spacing between the dots matters, because the closer the spaces in the dotted lines, the higher the heart rhythm. And there's this kind of visual heart rate that you can see as you scan across the line. Well, a lot of our customers probably are going to miss that. But for the people that kind of got it and noticed it, we wanted to make sure the AI model knew how to use that and properly represent it.

[00:25:15] We thought about details for comprehension. For example, using data markers versus when not to use data markers. Turns out the magic number was 31. We also thought a lot about our accessibility-first approach to design as well. We don't like to use color to convey meaning if possible. How do we provide a colorless solution, especially for people who have low vision or color blindness? And it was important for us that we captured all of those details so the model knew how to work with them, to make sure it could draw charts that looked uniquely Google and Fitbit.

Building for people, not machines

[00:25:47] So I covered a lot today about process, some lessons learned, and the importance of design craft. But there's one principle that I think is more important than all of them. And that's to remember we're building for people and not machines. It's easy to get caught up in the glitz and the glamour of the technology and get intoxicated by the possibilities of AI. But we're here to create meaningful experiences for people at the end of the day.

[00:26:12] And should you find yourself working with data visualization and considering its place in an AI experience, the way that you follow that principle or make progress on it is to consider all of the other principles that I covered today. So thank you very much, I hope you enjoyed the talk.

[00:26:30] I'd like to acknowledge my colleagues that worked very hard on this work. We launched two products that were based on this. One was an experiment and another is the upcoming personal health coach that is launching. A lot of research went into this, a lot of engineers and product partners. And then finally, if you did like the talk, I'm actually going to be publishing a book called Data Visualization in the Age of AI. This topic is discussed in much more detail in that book. It's going to be available next year. And you can follow me on LinkedIn if you'd like updates on that particular project or to continue this conversation. Thank you very much.

Q&A

[00:27:09] Host: All right, thank you, Kent. Sure, I'll take that from you. I'm sure we'll have a lot of really good questions here. Okay, what was the success rate of the data visualization, and how do you handle the failures of the analysis?

[00:27:24] Kent: Yeah, that's a really great question. So in this kind of new age, as we were experimenting a lot with how we wanted to approach data viz, especially in this particular product for consumers, we knew we were going to learn a lot. And with any UX process, there's things you're going to learn in your studies along the way that are completely different than your hypotheses or expectations, that might even nudge the roadmap in a different direction. I think the one example here is the correlation study that I shared. Instead of just giving up on charts, we actually thought more about, how do we just kind of get back to the basics here and think about what are some other techniques we could use to visualize that type of metric?

[00:28:00] Kent: And we wanted to make sure the team had enough room to explore. I think it was 90 different options we came up with for an infographic. And we really put a lot of rigor into that process. So I think one of the big takeaways here is there's still a lot of room for design exploration, not to just start with one idea and roll with it and just hope it does well, but think about all the other different ways you can communicate that information or solve a problem. And I think that's even more relevant now in this age of AI than it was before.

[00:28:28] Host: Did that question come from anybody here, by the way? Anyone in the room? You? We've got a winner. Nice. So Kent wrote a book. There's one book here, and we're going to give it out to somebody from the room who asked a good question. So there you go. It's a great question. See Kent after you see me, we'll get you the book. Congratulations. All right. Can you share a link to the Google DeepMind short film?

[00:28:49] Kent: Yeah, I can share a link after this, or if you just do a quick Google search, it's on our blog, the keyword of Dear Upstairs Neighbors, neighbors plural, it should come up. You can actually watch a preview of it on YouTube. It was also revealed at, I forget which film festival, so I won't name it to be misquoted, but it's out in public now making its rounds in different film festivals.

[00:29:11] Host: Great. And how did you assess your users? They had the alliteration.

[00:29:18] Kent: Yeah, so that's a really great question. It was using the framework that I had shared early on where we looked at the different insight types, the characteristics of them. We started with data viz best practices to get our lineup of charts that we would use to visualize each of those insights, and then we just ran A/B tests over and over with a lot of people. We used mixed methods in our research too, so we had moderated and unmoderated studies. On some of the bigger ticket questions we had, we wanted a much larger sample size to get a better signal, so those studies tended to be a bit more unmoderated. And then there were different success metrics along the way, and we asked questions about how they were able to interpret the insight, what they took away from it. And that was kind of how we approached it, and we learned a lot, as you can see, from that particular process.

[00:30:02] Host: How much do you actually need to train an AI which charts to use for what, versus initially trusting AI to provide the right chart and making the right minor adjustments or iterations every time? That's an easy one there for you.

[00:30:16] Kent: Well, I think we've all seen a lot of bad dashboards out there, right? And as I mentioned, models are only as good as their training data, and we have a lot of models that are trained on some great open-source examples. But as everyday designers, a lot of us weren't formally trained in chart usage or how to select the right type of chart. And even when we ran our Google-wide data viz office hours program, I was part of that from 2021 to 2023, one of the questions, even with our designers, that consistently came up is, which is the right chart to use? And of course, as I've been at different conferences and events like this, I've learned that this is a question that just keeps coming up over and over again. So because of that, it was really important for us to try to really embrace the knowledge that we had and apply it in a way that we could train a model now to make those decisions. Because if there's a lot of questions just with humans and in the design community about this, I wouldn't necessarily look to a model right away to figure this out.

[00:31:11] Host: Right. And I know we're almost at time. I think we have this next one, I think is a really good one, I think you can hit it probably pretty quickly. But how do you measure the success of which chart resonated most with your users? What metrics are you using?

[00:31:20] Kent: Yeah, so there were a few. Some of them were a bit more qualitative than others. But one of the main ones for comprehension was just asking, after someone saw the insight, what did they take away from it after they weren't able to see it anymore? And it could help us assess the memorability of the insight, and if they understood the main takeaway from the chart, to us that meant that the chart was actually working for them, it was pretty effective. We also looked at delight and engagement and just general happiness towards it. I think what we've been finding is a lot of AI models, like I said, like to talk, they produce a lot of text, you have to read a lot. So having some sort of visual organization of that response was also quite desirable within our audience. But we're seeing that in other areas too.