Bridging the Gap: How Product, UX, and Dev Can Build AI-Native Products Together

Carsten Wierwille, Jacobus Kok at UXDX USA. Video: https://youtu.be/spmY_49Rhl8

Readable transcript: edited from the recording's captions for readability (fillers and false starts removed, punctuation and section headings added). Wording is the speaker's own. Timestamps are positions in the video. Names marked [?] could not be verified against the audio.

AI at Priceline

[00:00:00] Carsten: Sadly, our panel shrank overnight. Kobe and I now have to speak twice as fast and twice as intelligently. We'll see how we do. As you heard, we want to talk about collaboration and how AI might change, particularly, interdisciplinary collaboration. We know UXDX is very much about bringing people together across different disciplines, and with AI now everything gets jumbled a bit, so we thought it would be interesting to explore what that could look like. Let's start with you. Maybe talk a little bit about your role and what you do at Priceline.

[00:00:50] Jacobus: Sure. First of all, apologies to anybody who was really hoping to listen to the other panelists. Hopefully what we have to say is of value to you. This is such an important topic, and as product, design and research leaders we have to grapple with this and figure out where to go. I work for Priceline. Priceline is a travel company. For those of you that may not know, we're part of Booking Holdings, the world's biggest travel company. Priceline's been around, I think, for 27 years. We started during the dot-com boom, and we're one of the few companies that made it through.

[00:01:31] We were very much focused on the North American market, compared to Booking.com, our sister company, which is global. We also have Kayak, OpenTable, Agoda. So a bunch of brands under the same umbrella, but Priceline is the brand I work for. My role is that I lead product for a number of areas. I'm not going to talk about some of those. AI, I think, is the part I'm most focused on at the moment.

[00:02:02] And it's moving extremely fast. It's a really exciting time, because we're a traditional company. We've been around the block in terms of innovations like mobile, the internet, all of those, and in a sense it's history repeating itself, but this time it's just much faster, and the possibilities, you could argue, are endless. What we're really trying to do, and what my focus is on, is to figure out how we use this technology to improve our product for customers. And that's a tall mandate. We've sold travel a certain way for many years: hotel rooms, flights, rental cars, packages. But now AI upends this, and our relationship with our customer really changes, and the interface through which we interact with our customers changes.

Three buckets of AI work

[00:03:01] We're doing a lot of things. I bucket them in three aspects. There's non-conversational products, and how we integrate AI into our traditional OTA booking funnel. That's something that's developed across the industry over many years, and it's the same everywhere. If you go to different online travel agencies, you're pretty much faced with a linear process where you have to go through certain steps. You have to interact with UI to tell us what your intent is, and then you end up with a long list of options and you have to make the choice yourself. Traditionally we've put the burden on the customer to do much of the heavy lifting, and we want to improve that.

[00:03:39] Then there's the conversational part, which I find super exciting. It's essentially starting with chatbots and figuring out what's the next iteration and where that takes us. We launched our AI travel assistant, Penny, two years ago, and we can talk about how that happened, because I think there's an interesting lesson there. But the goal really is to build an AI travel agent that sells you the right product fit for your needs based on your intent, and makes it easier.

[00:04:20] And then there's also a bucket of how we enable AI across the company and make sure we get the benefits of productivity and automating work. What we have found is that we could build a conversational experience that actually reduces costs, and all companies like to do that, and at the same time improves customer satisfaction. For a brand like ours, we're focused on giving customers the best deal at the lowest price. That means you can't expect to have an instant answer on your phone call for one-on-one support. There are some compromises in terms of the cost structure underlying the deal that we offer, and now AI really helps you move quicker.

Roles on an AI team

[00:05:19] At the same time, our product designer, who needs to think through what a refund interaction looks like and what we want to say when, can simulate this interaction in a sandbox. Each team member needs to put different hats on, but we also have to be mindful that people do have different skill sets and different tool sets. It's more about fostering that environment where you have great communication, and where it's not like, this is the line, I'm on this side of the line and you're on that side, and we don't step over it. It's a team sport. It's not easy to do. It depends on the culture of your company and of your team and the right personalities, but that's what we have found helps us build a successful product and team.

[00:06:11] Carsten: Did you assign roles in a typical way, or did you give team members opportunities to take on different tasks?

[00:06:11] Jacobus: We still assign roles in the traditional way. But in the way we solve problems, if there's someone that's like, I really want to go and try and edit this prompt, and I'm going to put it through a smart reasoning model to see how I can optimize it, and I want to test different versions, you can do that. It's not like you need to get my approval. It's all about the outcome, and if you have reason to believe that you can get us to that, by all means step up. It's not like our developers are going into Figma yet to design.

[00:06:53] But I think eventually, if you think about where AI is going, our role as developers or designers or product managers is more going to be to manage AI team members. That's the way we have to think about it. Then the ability to use a certain tool becomes less important, because the AI is going to be able to do whatever you want it to do, because it can work with multiple tools. To a certain extent you still obviously need developers to understand the code, and they need to be able to read it, even though much of it is now being generated by a model. So who knows where this boundary between the disciplines will end. But I think the more fluid you keep it, the less you lock yourselves in, the better the outcome.

[00:07:42] Carsten: Ideally there's this shift from a focus on the process to the outcome. The notion of craft changes, because AI can take a lot of the load away from writing code or designing an application. You said you're also responsible for, or part of, the enablement within the company.

Enabling an AI-first company

[00:08:08] Jacobus: Yes, that's been quite interesting. We want to be an AI-first company. What does that mean? It means we need people from across the company, in every single function, to really learn how to use these tools. And we don't think of it as cost optimization or automating people's roles. It's more like, how can we use our people to do more by not having them do repetitive tasks? It's interesting to see how some people just naturally gravitate towards using AI as a tool, and you have to figure out why, and then help them teach that to other people, which is easier said than done.

[00:08:53] Some people think of AI as a search model, where I have a query that I put in and then I get an answer out, and if I want to change the output, I now have to go and edit my query. I saw someone interact with ChatGPT like this, and I was taken aback. I didn't realize some people had this mental model, because I assumed they had the same mental model as me: that it's a conversation, and you ask follow-up questions, and you treat it like it was a person.

[00:09:18] The point is, I think you really have to find these pockets of people who are advocates, who can use the tools, who also know the limitations. That's super important, because you need to understand where the boundaries are. It's this fuzzy boundary of capabilities that keeps shifting every week, and it's hard for people to keep up with that. You can't expect everybody to be up to date with the latest developments. So it's really to create safe spaces. Give people tools that they can use. Put the right guardrails around those, so that from a governance perspective you have confidence that your data is secure. You have to make sure that your policies are effectively implemented and executed. And then have an environment where people can ask questions and can learn and get excited about it. That's, I think, very helpful.

[00:10:24] Carsten: A lot of companies have started to set targets for code automation, saying a certain percentage of the code needs to be automated. In the world of product management, have you been equally prescriptive and said, from now on the PRD needs to be generated by ChatGPT?

[00:10:35] Jacobus: No, not at all. We're like, these are the tools. Use it or don't use it, it's up to you. We don't force people to use it, because some people are not comfortable with it. Ideally we would have more people use it over time, but it's more to get people to realize that it can actually make your life easier and it can make your job more fun. We all find comfort in these day-to-day repetitive tasks. I do that. I go to a dashboard and I spend 20 minutes looking at a graph, where maybe there's a better way to do that. Maybe I can have the AI just tell me if something is up with the graph and I don't have to spend 20 minutes on it. So it's a cultural change also. But we're not prescribing it by any means.

Getting ready for the new world

[00:11:26] Carsten: What would you say to people who are coming to you and saying, how do I best get ready for this new world? How do I prepare myself? How do I position myself in the right way to be ready?

[00:11:41] Jacobus: The advice that everybody usually gives is just use the tools. Figure out how they can help you, and that depends on what work you do, what your typical day-to-day is. And I would say you really have to challenge yourself and figure out, what am I missing, and why? If it doesn't work for me, why doesn't it work for me? And change your way of working. It's just becoming more comfortable with it, treating it as a, I am tempted to say the word peer, that may be controversial, but you have to realize these models are getting as smart as us. And there's some bias I think we all have, that we don't want to admit that there's now a smarter tool that could potentially get work done faster. The moment you have that acceptance, go through that phase, I think it becomes easier, because then it's all about how you can use this thing for the benefit of your company and your personal development.

[00:12:56] Carsten: We've certainly found at HTEC, and there's a lot of talk about play today, that play needs to be somewhat orchestrated. You can't just tell everybody, oh, play with those tools in your free time. You actually have to give people an environment where they can play with it. You have to give them a sandbox, if you will. Or you have to say, OK, this week we're going to try to do our regular product management and design tasks with AI, and here's the boundary for all of that. You can't just leave it to people on their own. I don't know if that resonates with you.

[00:13:15] Jacobus: That is true, because people do sometimes get stuck. That's why we have found it helps to create a Slack channel where people can just ask questions and can share wins, and create sessions where people can present the GPT that they've built to automate a repetitive task that used to take them two hours a day, and now they can actually spend that time on more value-adding projects. So I agree. It's about fostering this environment where people feel comfortable in not knowing, or want to experiment, but also share wins and then help each other. Again, that's easier said than done, and that's why it does need to come from the top. If as a leadership you're skeptical of the benefits of the technology, it's hard to expect people to be excited about it. Luckily our company's pretty vocal about our desire to use these tools more.

[00:14:22] Carsten: How do you think your personal role will change? What will you do three or five years from now differently than today?

[00:14:22] Jacobus: I have no idea, but I'm excited about it. The thing I'm most excited about is the ability to speed up what you want to create. Now it means you really have to think through what is the thing that you think is going to work, and with AI, hopefully, if all goes well, you'll be able to build that quicker with the same number of people. And the realization is that everybody else is also going to be able to do that. So I don't think it's going to reduce the pressure on us as product companies to innovate, but I think that's going to be a base requirement. Things are going to move quicker. I think we're going to be able to do things we currently can't do because of limited tool skills, and that excites me quite a bit.

Q&A

[00:15:01] Carsten: That's cool. I think we have about 10 minutes for some questions, so maybe we'll turn our attention to some of the questions we have here. The top question is: historically, design-led products outperform engineering-led ones. In the age of AI, does this still hold true?

[00:15:26] Jacobus: That's an interesting question. A design-led product, in my mind, is a product where a lot of thought was put into how this product makes the user feel, why it exists, what its reason to exist is. From that perspective, I agree: engineering-led products could be soulless, for lack of a better word. So I think to some extent AI helps us better engineer design-led products, if that makes sense.

[00:15:59] Carsten: To what extent do you think AI today is engineering-led versus design-led, when you look across the board?

[00:16:15] Jacobus: I do think that the frontier labs are very design focused. It does at least appear to me like they're making very intentional choices. Even though the user interface of a chat box does feel technical in a sense, you do see simplicity as a value. And with personalization, some of these tools are making the product feel like it's right for me, it knows me. So from that perspective I do appreciate that there's a strong design focus. Where that will go, I hope that gets built into the models in a sense, that it becomes more pervasive. But we'll have to see.

[00:17:03] Carsten: I think it's also common that in the early stages of technologies we build something just because we can, and then over time the real meaning and the significant contribution emerges.

[00:17:17] Jacobus: But if you think about it, what we're building is intelligence. It's not just a technical feat. It needs to feel approachable. It needs to feel warm and human, even though you have to separate human versus non-human. But the goal is to have a humanlike interaction with it, so I guess design values are instrumental to achieving that.

[00:17:43] Carsten: Next question. As our roles are evolving, how can you break down silos and bring better cross-collaboration across teams, especially when the skill sets are overlapping more?

[00:17:58] Jacobus: That's hard. I don't know how you do that, because people mostly don't like change, and most companies are also resistant to change. So you have to almost break down the barriers and silos, and sometimes the organizational change, the formal change, may come later. It's again more about communication, and getting a group of people from different disciplines around the same one goal and getting them to celebrate that together. It's a hard thing. I'm not sure how quickly that's going to change.

[00:18:46] Carsten: What happens when design intuition clashes with machine learning predictions? Who wins?

[00:19:00] Jacobus: That's an interesting one. I think eventually, and this may also be controversial, AI is going to play a bigger role in how user interfaces get dynamically generated. Again, if you think about personalization, these models understand me, who I am, what my values are, what I want, what works for me, better than I may realize. Now if you add that to UI, the models are perhaps going to start dynamically generating UI that achieves an objective in a way that resonates with me. So with design intuition, how do you train the model, and the product as a whole, to achieve that? That's going to be, I think, a big challenge for us as product managers and designers, because if it clashes, it's not going to achieve the outcome. So I think that's actually a big focus area in the future.

[00:20:03] Carsten: You mentioned reducing hallucinations. What do you think a typical person's tolerance is for hallucinating AI responses?

[00:20:03] Jacobus: If you look at the data, the data is showing models are hallucinating far less than they used to just a year ago. I think some of the most popular models are in the 2% range. So hallucination turned out to be not this big problem everybody was worried about. With the right guardrails, I personally think it's a totally manageable problem, and it's going to get solved over time. The tolerance depends on what industry you're in. Obviously some industries are much more sensitive to hallucination than others. It's all based on your industry, whether it's regulated, what the impact of a hallucination is on the quality of the product. If our AI assistant confuses New York with Los Angeles, it's less of an issue, and not that this has ever happened, and I don't think it would, than if you're in financial services or healthcare. Obviously that's a much bigger problem. So it depends.

[00:21:12] Carsten: So you haven't sent any of your customers to the wrong destination yet?

[00:21:12] Jacobus: Not that I know of. We obviously need to communicate to customers that they're talking to an AI. Super important. It's still a beta product, so it's not 100% accurate, and I think it's important to make sure customers understand that. But we've luckily not seen any cases of customers being sent to the wrong destination.

[00:21:39] Carsten: More and more, the repetitive and creative tasks I love are being outsourced. How can someone who loves slow creative work get excited about using AI?

[00:21:50] Jacobus: I don't mourn the repetitive tasks, personally. I've found having a brainstorming session with AI quite fun sometimes, because it does come up with ideas that I didn't think of, especially if I push my ideas to the limit and then have it take those ideas and see if it can come up with an even better idea. So I think it's all about context, and trying to put into the model everything you can give it. It's that chain of thought without filtering. I found that approach quite successful. You just put it all out there, everything that's going through your mind, and then you ask the model, OK, help me make sense of this. And sometimes that helps you in turn to come up with an even better idea.

[00:22:50] So in my mind it's not outsourcing it. It's more just speeding up that process. Now, if you like the slow process, you can still slow it down. Just wait longer before you ask that follow-up question. You are still in control. I think the moment you feel like you have to now deliver faster, you don't want anxiety. I don't know if anybody saw this talk with Jony Ive last week, but the thing that resonated most with me is where he said if you're anxious while building your product, you can have an anxious product. So I think there's something there.

[00:23:28] Carsten: How should UX designers and developers collaborate to build services using AI?

[00:23:43] Jacobus: Again, like we've said, it's about having them work together, having them collaborate on one team. In terms of services, I think services are a product. So I don't think there's anything besides the need to better understand the value that each brings to the process, and to see how you can actually bridge that gap, maybe with AI's help. What we haven't tried, but now I'm thinking it through, is maybe having more deliberate design jams, where you have developers in the room and product managers and designers. Because I think the important thing from a design perspective is to have a really good sense of the underlying capabilities. We're drawn to the things that we can do quickest that are going to move the needle most, and I think the developer contribution is really important there, so that you don't come up with a concept that's not actually executable.

[00:25:05] Carsten: I think that's also the realm where there are tremendous opportunities for design, because we now have tools to do things that we couldn't do before.

[00:25:05] Jacobus: Absolutely.

[00:25:16] Carsten: And you've seen that as well. I think we have maybe a few more seconds, so real quick, in 10 seconds: how can AI-driven decisions stay user centered despite data-driven priorities?

[00:25:16] Jacobus: User feedback, I think, is really important. That's something we look at daily, something as simple as upvotes, something as simple as customers telling us, I have feedback, and we really take that seriously. We want to build better feedback loops. Evaluations are such an important part of building AI products, so that you can understand how your product is performing by actually using AI to monitor it. That's something that wasn't really possible before. Now we can be more in tune with how our users are experiencing the product in real time, and I think that's going to be a big differentiating factor between successful and less successful AI products.

[00:26:11] Carsten: Cool. Thank you so much. That's really interesting. Thank you very much.