Bridging Human Insight and AI: How Pinterest Builds Visual Experiences That Connect

Larkin Brown at UXDX USA. Video: https://youtu.be/nNNfVyykXVA

Readable transcript: edited from the recording's captions for readability (fillers and false starts removed, punctuation and section headings added). Wording is the speaker's own. Timestamps are positions in the video. Names marked [?] could not be verified against the audio.

Learning from personal stylists

[00:00:13] Hi everyone. All right. I'm wandering through racks of clothing. I'm in a department store and I'm with a personal stylist. She's pulling things out, being like, "No." Pulling out another one. And I'm there to watch how she is figuring out what her client might like.

[00:00:35] And I'm asking her, "Well, how did you come to figure out what her style was?" She explains to me, "Well, first, she said she wanted to look put together. And let me tell you, that's what everyone says and it means nothing." I'm like, "Okay." "And then she said she liked a little bit of edge. But I knew that means something different to everyone. So I started pulling up some pictures and I start showing her stuff. Black? No. Leather? No. Turns out she just likes hardware details." And I'm just like, "Okay."

[00:01:08] And this was not the only interview I was having. I was looking at people whose job it is, professionals, to guide people when they don't quite know what it is that they're looking for. Pinterest is already being described as my cool stylish best friend, my personal stylist. And we wanted to figure out if, by learning how the professionals did this, it's something we could build more into the product. So I focused on people who were helping their clients make aesthetic decisions, visual stuff: personal stylists, interior decorators, and wedding planners.

[00:01:43] And I came back with a framework. My stakeholders wanted the list of questions that they asked. I gave them a list of questions for each type of professional. I gave them backbone questions. It's 2014. So we built a quiz.

[00:02:03] [laughter]

[00:02:05] And we really tried. I did decision trees, and we put a lot of effort into it, but it just wasn't flexible enough. It wasn't responsive enough. And how quaint does that sound now?

[00:02:21] [laughter and gasps]

[00:02:21] And how wonderful that all these years later, that same work we can now use, layering on this incredible new technology that we've been talking about all day. And that's what I'm going to share with you about today: how we are taking our UX research insights and weaving them into the product today, leveraging some of this new technology.

Pinterest and its pinners

[00:02:45] Now, do we have any pinners in the room? Pinners are our users. Yeah? Oh, nice. Okay, I see some pinners. Great. Just to anchor you a little bit, if you haven't used it in a while or if it's new to you, we'll start with the Pinterest mission, which is to bring everyone the inspiration to create a life they love.

[00:03:04] And we've been growing a lot. We're up to 631 million monthly active users. And people are using it to browse that main home feed when you open the app, but they're also searching a lot. So, over 80 billion monthly searches. And then as they're searching, finding things that either inspire them or things they want to do later, they then save those pins, which are ideas which have a big visual component, and they are saving 1.5 billion pins every week.

[00:03:33] And we're also aging down, which is actually quite unusual. We're now at the point where more than half of our monthly active users are Gen Z. And more than half of those think of Pinterest as a place to shop.

[00:03:47] Now, I'll just touch a little bit more on Gen Z, because I was just at my alma mater, Northwestern University, and the alumni folks who had invited me to come to speak to the classes had told me, "We elder millennials were very excited to have someone from Pinterest here, but we weren't really sure if the kids were going to like it." Meanwhile, I go into the communication studies 101 class. It's all freshmen, 18-year-olds, and boy, did they raise their hand to let me know, "I've been using Pinterest since I was in middle school."

[00:04:23] And it makes sense, because those teenage years are this poignant chapter in time that's full of personal development, individuation is what they call it in the research, of trying to figure out who do I want to be. And they're trying on different styles, different aesthetics, different interests. They're building the blueprint of their future self.

[00:04:50] And Pinterest is so good at that. When you don't quite know what you're looking for, it gives you a range of options that you can just skim, beautifully, easily, to be inspired by and save from and curate. And the other piece, and Gen Z really understands this, is that this is a pressure-free environment. You're not going to get mean comments in there. It's a safe space without pressure to post. And they really understand that Pinterest is not a social network.

Rabbit holes and multiple intents

[00:05:20] And the other thing that they like about it is this non-linear discovery. Pinterest is known for rabbit holes, where you might start because you're in your early 20s and you just bought this cool orange vintage jacket and you're trying to figure out, "What can I wear with this?" And you see a cool street style post, but actually what you really like there is the cool sunglasses, and they're clear sunglasses, and then that gets connected to clear jelly sandals, and then you can wear socks with the sandals, and I have sandals, I can put socks with my sandals. The rabbit hole that allows you to go deep or to go wide is part of the beauty and the magic of Pinterest.

[00:06:00] But it also means that people have multiple intents while they use Pinterest, even per session. So that actually makes it quite tricky for us, because we would love to figure out what is the point of what someone is doing in this session so we can give them the best recommendations, the best products. But in reality, in a single session, someone may, I'm thinking of a millennial who's going to redo their bathroom, and they find these green tiles only to find out that color's called wasabi, which leads them to the bag and then to the scarf that they could wear on a beach vacation.

[00:06:37] So then you think, okay, well, if we can't assign intent to the session, let's assign intent to the imagery, to the pins, the content. Well, that doesn't work either, because different people may get something different from the same pin. If you look at this flower pin, maybe there's a design or art student who loves this photo because of the shadows on it. And then someone else might look at the same photo and think, "Oh, this is perfect for my wedding. I wanted all greenery and the flowers anyway."

[00:07:10] So while this makes it tricky for our content understanding, this is really wonderful for our advertisers, because Pinterest brings in people who have an openness. Maybe they know what they want, but they're open to a range of options and willing to jump around and think about different projects or interests in a single session.

Visual first and the taste graph

[00:07:33] And then, as I keep mentioning, it's also very visual first. As you heard with that personal stylist, words can only take you so far, especially if you don't quite know what you're looking for. Maybe you don't have the words to articulate it. But on Pinterest, you can tap on an image and use visual search to zero in on a particular piece of it, whether it's the hat or the sunglasses.

[00:07:58] And behind all of this is our taste graph. Pinterest really understands your style, your aesthetic, and because people use it for multiple domains of their lives, we're able to make some of these unlikely patterns, unlikely connections. Maybe it's how your personal style intersects with your home decor style. Or maybe it's the recipes that you're choosing to cook and save and dream about. Well, maybe that actually is connected to the travel destinations that you might want to go to.

[00:08:36] Because we're able to make these connections, it's a very good place to do trend spotting. Lots of people use Pinterest for that professionally. But there's also this element that makes our data set so special, because it's human-powered. It's based on over 10 years of people curating and saving their dreams. And that helps us both really understand the style of an individual pinner, but also create this deeply, deeply human data set.

[00:09:10] And then you think again about how we're layering on this new technology. It's not a guarantee that we're able to keep that humanness in the way that we build products that are LLM-based. It has to be quite deliberate. And that is what we're doing at Pinterest, and that's what I'm looking forward to sharing with you next.

Evaluating an LLM-powered assistant

[00:09:33] All right. As we're designing LLM-powered experiences, one of the first ways we're experimenting with that is the Pinterest assistant. That's something that's an experiment right now, but we've announced it and we keep tinkering with it. And when you're trying to evaluate how well a product like this is doing, and I've heard other people talk about this today, it's tricky. There are some universal challenges to how you know if the output of an LLM is good or not.

[00:10:04] Because the same person might have the same or similar questions, but if you have multiple turns in the conversation, it could end up in different places, and that might not be bad. That might be okay. And on Pinterest, we have some other unique challenges, where the expectations of our personalization systems are very, very high.

[00:10:27] And the thing that we're really good at, the aesthetics, taste, vibes, that is really hard to operationalize, especially when it's so specific to each person. And then on top of that, we're really visual. A lot of the experiences today that are anchored on LLMs are text-based. So we have this additional challenge where it is critical for us to do this in a way that's Pinteresty and makes sense to do on our platform. It's got to use the visuals. Then there's a whole other element of content understanding that we have to really get right in order to nail this experience.

Researchers pairing with ML engineers

[00:11:08] Now, I can't help it. There have been so many versions of this slide, so I'll go really quickly. This is how we used to work. There's a lot of handoffs. We use different tools. This is how we're starting to work: more blended together. I know so many of us can relate to this now.

[00:11:25] And the element I want to focus on that I've been most proud of is what we're seeing with our researchers and our machine learning engineers. They are really starting to work side by side. And this is what I've been hearing in the room too, in some of these different talks, of having these moments where they're really sitting together. I've learned the word pairing today. ML engineer and research pairing is what I would say we are doing at Pinterest.

[00:11:54] And that is happening both downstream, which is about how you evaluate the outputs of LLM systems, but then also upstream, in how you do some of the highest-level prompting. So we'll start with the downstream side.

[00:12:12] This is roughly what we're doing when it comes to evaluating the output of these. We are having the researchers design judges. And what I mean by judges is it is intended to answer the question, for this output, is this good or not? But if you're looking at a multi-turn conversation that has images and text in there, that is qualitative, that is messy. So you're trying to create a really clear description of what good would look like.

[00:12:45] Then you're going to use that judge against a curated data set. In our case, that's a set of queries that have come from our pinners. And then in order to see if this is working, you also have to operationalize it in a rubric. Creating a rubric is a skill all in its own. It is much more like survey design, because when you have a rubric like this, you need humans, if you're going to use human evaluators, to be able to evaluate consistently. And you also then want to teach an LLM to do that too, and then look at inter-rater reliability.

[00:13:22] From there, as I mentioned, we will have human evaluators and our LLM do it separately, and then see how we're doing with the agreement, and then calibrate and keep going.

[00:13:35] This is a newer skill for us, and it's pretty different for a researcher who's used to being out in the field, or behind a double-sided mirror talking to pinners, to instead take our years of knowledge and distill it into a text prompt. That's usually how it starts. And then work with the ML engineer to create that judge, to really code that quality right into the system.

Replacing a synthetic query set with real queries

[00:14:07] Now, I mentioned curating that query set. While we were building our first judge, we also noticed that the query set that product team was using, and again, these are what a pinner had typed into our chatbot, was actually all synthetic. They were reasonable guesses, but they weren't real.

[00:14:31] So we went through, and we had a researcher pull out a thousand queries from real pinners that were representative of, again, what we know Pinterest is really good at, classic use cases, and also had a range of complexity in the query. And then it would be very expensive if you were trying to do daily evals on a thousand queries. So then we honed it even more, down to a hundred, and that we can use daily to see, when you tinker with the model, is it getting better? That's the point of the evals.

[00:15:06] And thank goodness we did this, because we found out that even within the set of queries around fashion from the synthetic data, it had a lot of gaps. It didn't have any men's fashion in there, didn't have any wedding fashion in there. It was missing some of the Gen Z aesthetic lingo, which is wild.

[00:15:28] [laughter and gasps]

[00:15:29] And that means if you are not training on that, you are going to have whole gaps in your LLMs, which don't understand that part of our data. So this to me is a bit of a nerdy research thing, making sure that you're testing on the right query set, but it's really important.

Upstream: coding research into the prompt

[00:15:50] Now, I want to talk a little bit more about upstream. We just spoke about what I would call downstream: how do you know that the output is good or not? But the work I'm most excited about right now is taking our years of existing Pinterest research and distilling it into something that you can code into the initial prompt, to train the LLM product from the very beginning.

[00:16:18] One of my principal researchers took over 60 studies and did a synthesis across all of them, using AI to help synthesize and pull the themes out, and the output is six endemic user journeys. It looks at both what they are today and what the pain points are, and what a happier path would look like, which then calls out, of course, what the opportunities would be. This is a very classic research idea, to create user journeys.

[00:16:50] But importantly, instead of having the output be a big, beautiful doc, which this researcher is very good at, or even a presentation, she made an interactive HTML file that we then put on GitHub. And the reason that's important is it means that our cross-functional teams can just copy and paste bits of that and put that into the things that they're vibe coding.

[00:17:17] And the other thing she used to make these endemic stories is she had to write out principles around the user experience. And now we can just copy those principles and put those into the prompts of the LLMs that we're building. It also then creates this interactive moment for anyone who's doing vibe coding on the weekend to ask, is this actually going to help the user need? And to be able to look at this as a reference point for some of the unmet needs, the pain points we still have that need some support.

GenAI in the research craft

[00:17:47] All right. Now let's talk a little bit about how we're using GenAI in the research craft. And the short answer is we're doing all the things. This all takes a lot of experimentation as we try and figure out what is the right balance, what are the use cases, what are the guardrails for how our team should use this. This includes AI moderation, heuristic evaluation, synthetic users, agents, synthesis, knowledge management. I'm not going to go into depth on any of these, but if you are interested in this, please come find me afterwards or ask questions in the chat. I'm happy to share more about it.

[00:18:27] Because I also want to touch on, and this has come up so much today, the fact that it's not just us that are using these tools to speed up. Everyone around us is doing that too. And suddenly the amount of high-fidelity prototypes that are coming out of every corner of the Pinterest employee base, it's a lot. That might seem like a good thing for researchers, plenty of things to test.

[00:18:54] But actually, there is so much cognitive load when you get something really high-fidelity that maybe someone didn't spend very much time on. We even saw this in a design workshop we did recently. The outputs were such high fidelity it was actually very hard to evaluate each one, whereas if they had been in wireframes, we could have gotten to the point of whether it solves the user need faster.

From observation to insight to opportunity

[00:19:19] So I also wanted to share this framework. I bet some of you are familiar with this. I've seen it most used at IDEO, around human-centered design, where you can take an observation, something you see, maybe it's a particular pain point, and then come up with a solution. Sure. But true human-centered design is taking an observation and then pulling out what is the insight. What is the bigger meaning behind this? And then turning it into identifying the bigger opportunity at play here. And then you can come up with tons of solutions. And this is also the researcher's job to facilitate, whether it's through a brainstorm or a participatory design workshop.

[00:20:02] Just to give a brief example, I've often heard the one used from Disneyland, which is top of mind because I'm going to take my kids for the first time this summer. If you have any advice, they're four and six, I'd love to hear it. The observation was people are grumpy standing in line. And the solution is FastPass. If you can afford it, you don't have to stand in line.

[00:20:25] Sure, that helps. But with a little more research: people are grumpy standing in line. Well, why? What's the bigger meaning behind it? The experience of standing in line is actually this feeling of lack of control and not knowing what's going to come next. There's this element of uncertainty that is really informing the discomfort there.

[00:20:49] So the opportunity is, okay, then how do we transform the experience of standing in line with an emphasis on control, expectation setting, and maybe a little bit of fun? And from there, what Disneyland did is they did do FastPass. That's great, as one solution. But they also said, "Okay, they need to know how long they're going to be here." So they added wait times. And then they purposely make the wait times longer than they really think it will be, because then people have a feeling of, "Ooh, I got some time back." They feel good about it. And they also then created a bunch more interactive experiences, especially for the kids, having themed ways to wait in line. But you get the idea. There are a lot more solutions here when it's framed this way.

[00:21:36] Now, I will say, in this age of AI, it feels like this to me. You go from one observation and you can make infinite solutions.

[00:21:45] [gasps]

[00:21:45] But then how do you know if they're good or not? I think Rory talked about this in the very beginning: "How do you know you're building the right thing?" And to me, it still comes back to needing to understand the core insight of the opportunity.

Shared understanding comes from shared experience

[00:21:59] So how do you create that shared understanding of the opportunity? I believe strongly that shared understanding comes from shared experience. And this is something that my mentor, Gabe Trantham [?], says all the time, which is that research is a social discipline. And he went as far as to say, "If our cross-functionals are not showing up to the research, we will stop doing it."

[00:22:24] And I think there's something to be said for that, because if someone has to watch a user try and fail on something for 4 minutes straight, they are much more motivated to go in and fight to fund that solution than if someone just heard it from a readout. And we researchers have our tricks of influence and how to make people think that it's their idea. But ultimately, it really is about having the team make memories versus just learning something off of a slide.

[00:22:59] When I talk about these research experiences, I'm talking about the shop-alongs, like I described with the personal stylist, the classic in-home ethnographies. We do something called Pinner Dinners, where we'll invite a group of pinners to have dinner with a set of cross-functionals. It works especially well when you're traveling, because they don't have anywhere else to be. But it ends up with this casual environment where they're showing their phones, our cross-functionals can ask questions, and it is not formal research, but it's a way to get that exposure and momentum and build empathy around it.

[00:23:33] We also use analogous worlds. Sometimes we'll look at extremes. I was lucky enough to get to go to Seoul to study the intersection of technology and beauty, because they're so much further out.

Get in the box, and get your teams out of it

[00:23:46] And the thing I want to leave you with here is the value of this research that is intended more for inspiration. That's more for invention or innovation. AI is so helpful in making these experiences happen, because they are a heavy load, so it's really helpful in that. And it's really helpful that the output of these experiences no longer should be decks. They should be pretty text-heavy, so that they can go back directly into our LLM experiences.

[00:24:24] But AI systems are probabilistic. They're trying to guess the next likely word. What is the next probable thing that would happen, or that someone would say? And in this kind of research, you're looking for the opposite. You're looking for the unexpected connections. You don't want to be in the center of the bell curve. You want to be on the edges, because that's where innovation happens.

[00:24:52] So my recommendation for you all, if we have any researchers, and I saw from the hands earlier there's a fair amount of researcher-designer hybrids, is I want to encourage you both to get in the box. And by that, I mean the prompt box. Learn and build that skill of how to translate, especially that kind of foundational research, into something that your cross-functionals, especially the more technical ones, can help you codify right into the product. Which is really exciting for a researcher. I've often joked, well, we don't really make anything, and also we don't have to make anything. But now we can, especially to be able to put our work directly into the product with no middleman.

[00:25:36] And then the other thing is I want you to get your teams out of the box. Get your teams out of the box into the real world to help find those unlikely patterns, unlikely connections, that are going to lead to the next big design innovation. Thanks, everyone.

[00:25:56] [applause]

Q&A

[00:26:03] Host: Thank you, and great job. All right. Well, we have time for some questions here. I'll start at the top. "How would a patient..." Oh, this is... well, that would be hard for me to answer. Do you want to give it a go? Want to give it a run?

[00:26:17] Larkin: Was the patient, did they prefer a grandma seaside aesthetic? Let's go with the grandma seaside aesthetic. Do we have the right questions?

[00:26:36] Host: This one is? Yeah, it's a completely new list. It says login, but it's still showing from the previous session. Let's scroll down. We'll get this resolved, because we need to answer your questions. Nope, still not there. I don't know.

[00:26:57] Larkin: Helps me appreciate how low the contrast is.

[00:26:59] Host: Let's just roll with this one. How does Pinterest define taste? Got you off the hook with the patient question.

[00:27:07] Larkin: Yeah, I can't tell you.

[00:27:12] [laughter]

[00:27:13] Larkin: But I can tell you we have a beautiful framework around it, and it continues to evolve, because this is a different way, where we need to operationalize it into words versus just connections between content. So it's a bit of a work in progress, and a bit proprietary.

[00:27:35] Host: "When you say codifying..." Where did that one go? Okay. "You mentioned the taste graph, which focuses on color. Were you considering accessibility? Color blindness."

[00:27:48] Larkin: Well, I should clarify. When I say taste graph, I don't just mean color, although color can be a component of taste. The taste graph is more the connections between the visuals or aesthetics that someone might be drawn to. And the question about accessibility is a really good one. This is something Pinterest cares a lot about, especially because we're so visual. We've taken a bunch of steps to increase things like contrast, and we have even been working, for folks who might have visual impairment, to figure out a good experience for them as well. And this is something that continues to come up with every design iteration that we work on, because you designers, you love your gradients and your soft colors.

[00:28:36] Host: "What is Pinterest's stance on AI-generated pins uploaded by users, aka AI slop, and how has it affected user journeys, experiences, and the AI models that you're building?"

[00:28:50] Larkin: This is such a good question. What we're finding is that it depends where you are in your journey. If you are at the beginning of a process or a project, AI is welcome then. If you are just imagining what it could be, people are quite open to it. It is when you are trying to get something done that it becomes incredibly frustrating, especially when you get that bait-and-switch moment where you see this cool rainbow hair color and you're like, "Yeah, I want this." And you bring it to your hairdresser, who's like, "Yeah, that's made up. I can't do that for you." At the same time, I was just chatting with someone backstage about designing a video game. Someone who's at the beginning of thinking about that, well, rainbow hair for them is great, because they're not trying to actually dye any hair right now. It's more for that high-level inspiration.

[00:29:44] Host: "Are there concerns that pins will become too synthetic to be useful to human users, and how would you combat that?" I wonder if they mean AI again by synthetic.

[00:29:57] Larkin: I think so. Yeah. I worry about everything. So do I worry about this? Yes, I totally worry about this. I also think, in the Pinterest ecosystem, we are learning so much from what people are saving. It is powered by human curation. So even if you have content in there that was made by AI, there is still this element of drawing connections between things that brings a uniquely human undertone to the data set.

[00:30:32] Larkin: At the same time, I don't want to look at recipes that were made up just to drive traffic to a site that might have spam on it. So our trust and safety teams have really tall orders to ensure that we get that balance right: when people are creating really beautiful things, or if they're just using Photoshop and it has a little bit of AI editing in it, that that doesn't get hidden away. But to find that right balance, to keep that humanity that's so special and part of Pinterest.

[00:31:05] Host: And how are you measuring the ROI of research?

[00:31:09] Larkin: Oh. That's a good question. We're doing a project right now, and I heard there was a panel in the other room talking about this, how you evaluate UX quality at a high level. We're doing a project right now around end-to-end quality, to ensure that we're capturing the smallest UI fixes, bugs, comprehension. But then you know how this goes. If you have a lot of different teams working on different surfaces, each surface might be optimized, but the flow across them isn't.

[00:31:49] Larkin: And when I think about the ROI of research, we are, at Pinterest, the arbiters of quality and of taste. So I think that our leadership in defining quality for the experience is the best thing I can point to in terms of ROI. And the other thing I would point to is impact. Like so many functions, we keep a really close eye on product changes that were made, or product ideas that were built, spawning from research that we did. And that can be at the really tactical, evaluative level, or at the much grander level, like getting to go to create [?] and learn about skin care.

[00:32:33] Host: Great. Well, we're at time. If anyone has additional questions... Thanks, everyone. Great job.

[00:32:39] [applause]