How to use AI to become a 10x Product Team

Anthony Maggio at UXDX EMEA. Video: https://youtu.be/PWe8YEVbDg4

Readable transcript: edited from the recording's captions for readability (fillers and false starts removed, punctuation and section headings added). Wording is the speaker's own. Timestamps are positions in the video. Names marked [?] could not be verified against the audio.

Introduction

[00:00:10] All right, good morning everyone. Thanks for having me. It's actually my first time in Berlin, so thank you for giving me an excuse to be here. I'm Anthony, I lead the product team at Airtable. I'm based in San Francisco. And if you're not familiar with Airtable already, Airtable is a leader in the no-code space. We make it easy for anyone to build delightful, modern, AI-powered business applications.

[00:00:37] So why am I here? A little bit about me and my role at Airtable. One of the major solution areas that Airtable provides is tools for product teams. We have a solution called Product Central. It's an all-in-one operating system for product in large organizations that is built with AI at the center of the process.

[00:01:00] Because of this solution and because of my role, I end up spending a lot of my time speaking with product leaders and chief product officers within our customer base. We serve customers across every industry, everything from really quickly growing startups and scaleups in Silicon Valley to financial services to major retailers across every industry. So that means I end up getting a very broad view of the market, of what other product leaders are saying, what is top of mind.

[00:01:33] And as it will likely be no surprise to any of you, the conversations that I'm hearing today are AI, AI and AI. Everyone is thinking about, how do we use AI within our products? How do we use AI within our process? Is AI going to come and take my job?

[00:01:53] What I wanted to spend the conversation on today is synthesizing and distilling some of my takeaways from these conversations, as well as my own experience leading product at Airtable, and share some practices and guidance that I've observed over the last few years. From hundreds of conversations, I've really come to the opinion that product teams — all of you practicing in product management and design and UX research, engineering, data science — you're really in a unique position to help your companies unlock transformative value with AI. And so I want to talk about some of the areas where we're seeing this opportunity, and hopefully over the next 20 minutes or so give you some practices that you can bring back into your organizations.

Principle one: operate on the frontier of LLMs

[00:02:42] So I will summarize these findings with three principles, and the first one that I'm going to start with is operating on the frontier of LLMs. I think at this point everyone in this room has had some type of aha moment working with AI tools, whether it was the first time you engaged with something like ChatGPT or using an AI prototyping tool. You've all seen the power of AI within your product development workflows. But one thing that I'm constantly surprised by is how many conversations I have with product development practitioners who don't realize just quite how good and how capable these AI models are getting.

[00:03:29] If we look at this chart, there's a lot going on here, so I'll take a little bit of time to unpack this. This is kind of showing us the speed of AI model progression just over the last few years. The bottom line that you see here on this benchmark test, it's called the GPQA Diamond. It's a test that's used as a benchmark for the progression of AI models and assessing the capabilities of AI models. It's not a perfect test by any means, but it does give us a proxy for understanding just how quickly the models are progressing.

[00:04:03] So what we see is GPT-4 came out in 2024 in May, and at that time it was doing a little bit better than a human PhD using Google outside of their own area of specialty. Fast forward to February of this year, we now have the o1 and o3 reasoning models from OpenAI that are outperforming a human PhD researcher with access to Google in their own area of specialty. And this has happened in just a few months. So pretty enormous amount of progression within a very, very quick span of time.

[00:04:46] The takeaway that I've had from this is that we need to be constantly re-evaluating the capabilities of these models to understand how to best leverage them in our products and our processes. It is not a one-and-done, go play with AI, go learn how to use it, but actually an ongoing practice that all of us as practitioners need to be incorporating into our regular rhythms.

[00:05:09] There's a quote here from Kevin Weil, who's the CPO of OpenAI, that if you're building something right on the edge of the capabilities of these models, you should keep going. You're doing something right, because the models are going to keep getting better and better and unlock new experiences that weren't previously possible.

The Airtable assistant: what a model switch unlocked

[00:05:26] And we've also seen this firsthand at Airtable. So a quick story relevant to that model progression I just shared. In April of last year we started working on an AI assistant at Airtable. The idea for this assistant was that it should be a chat interface that would be available on any Airtable application, on any data set, and allow end users to ask questions about the data within that app or within that workflow.

[00:05:56] When we started working on this last year we were using Claude 3 at that point in time, and the responses that we were getting from questions — this is an example question: review the objectives on my roadmap and tell me where we have some gaps — the responses were pretty high level, basic summaries, not getting a lot of real insights out of the questions that were asked. So much so that I actually continued to push our team on whether or not this was really going to be a viable product, and whether we were going to be able to get this to a state where it would meet the promise that we wanted to bring to market, of being an always-on data analyst assistant that could answer any question.

[00:06:36] So fast forward to February of this year and Anthropic released a new model, 3.7 Sonnet. It was the first hybrid AI reasoning model, and we switched over to 3.7 Sonnet in this assistant experience, and really overnight the entire product experience changed. It went from high-level summary questions to now being able to answer very specific, nuanced questions about any data set that the assistant was presented with.

[00:07:10] And to show you just an example of what this looks like in practice today, this is a demo based off of our own usage of Product Central. Getting a little meta here. But our product teams use this tool. It contains all of our feedback from across the business, so think every recorded phone conversation our sales teams are having, all of our support tickets, all of our community forum threads, emails from customers.

[00:07:37] And to show you what's possible now with a reasoning model like 3.7 Sonnet, you can come in and ask very broad questions and allow this AI agent to come up with a plan of attack to answer them. So here I'm asking, what do customers love about Airtable? You're seeing that the agent is now building a plan to take steps and ask, what fields do I need to look at to go answer this question? What filters do I need to apply? How am I going to interpret the user's intent and come up with a really nuanced and specific answer here? I thought this was just a very powerful example of how continuing to understand and unpack the latest capabilities of these LLMs will unlock new experiences for all of you and the products that you're building.

Principle two: embrace the cycle collapse

[00:08:28] So moving on to theme two, and this has already been discussed a bit this morning: embrace the cycle collapse. I think that in the last year I have witnessed more change in the way that we build products than in the entire 15 years of my career in product.

[00:08:49] Before I jump in, can I get a show of hands quickly? How many people in this room are familiar with the Agile Manifesto? Okay, almost everybody, it seems like. So the Agile Manifesto was written almost 25 years ago, and the intention of this document was to give us a better, more adaptive way of building software at the time. And now, how many people in this room can honestly say that, if you think about the way that your team or your company has built software over the last five years, that you followed agile practices to a tee? Yes, almost nobody, right?

[00:09:30] Because the reality is that there's a lot of things that get in the way of these practices: our organizational bureaucracies, resourcing, tooling, availability, demands and shifts from the business. But I think that we're entering a really important moment now where AI is allowing us to really collapse this cycle and collapse a lot of the constraints that have previously made getting to working software difficult.

[00:09:58] We're seeing that now with the latest AI-powered prototyping tools, we can get from concept to actual prototype and experience in minutes. I think for the majority of the time that I've been working in product we've typically measured the cycle time of major products in weeks, months, quarters. And that is very, very quickly going to change as we now get into a new development practice where we can go from insight to prototype to building products in a very short amount of time.

[00:10:36] This is also changing the way that we think about product development and where we spend our cycles at Airtable. One example here is that a big area of focus for us at Airtable over the past few years has been developing interfaces and visualizations for our users. We have lots of different customer segments, lots of different users who want to be able to present their information in different ways within their applications. And so historically it's taken our product team anywhere from a few weeks to a few months to develop new interface layouts in the product.

[00:11:13] Well, now as we enter this era of vibe coding, and you can really just ask an LLM to come up with any new interface layout or new UI that you might desire, that whole practice kind of goes out the window. We just started working here on a new capability to actually allow our users to develop their own interfaces on top of their data set.

[00:11:37] So this is an example we've been co-developing with a customer in the retail space. They're building an application to track all their new store openings and wanted to visualize them on a map view for their planning purposes. You fire up Cursor, copy in a prompt around what it looks like to visualize all these store locations on a map view, and in a few seconds you've got a new UI with all your locations plotted. And then maybe we decide, oh, actually we don't like this visualization, we want to update it a little bit and give some indication of which stores are new versus existing stores on the map. Simply go back, prompt your IDE to change the visualization, and you get a working visual of this experience in minutes.

[00:12:21] So my takeaway from this is that we've all, for the last 25 years, been trying to figure out how to get to working software as fast as possible, and AI is now removing those bottlenecks, and this should be really energizing for all of us. It's a great opportunity for us to embrace that cycle collapse and to go from idea to validation to derisking new concepts at a speed that's never really been possible before.

Principle three: think like an AI GM

[00:12:52] Okay, so the third area, thinking like an AI GM. What does that mean? So earlier this year Goldman Sachs put out a study that highlighted that while the capabilities of AI have been really transformative, few companies are actually already reaping the gains, and part of the reason for that is that they haven't started to adopt AI into high value recurring business processes.

[00:13:24] The report goes on to mention that while many companies are buying productivity tools like a Copilot or ChatGPT, the gains from those products are actually limited, and really the way that you're going to unlock true productivity and true value is by inserting AI into recurring business processes where the value compounds over time. This is definitely validated by what I hear from many of my peers and many of our customers, that few companies have really figured out those transformative use cases of AI, especially beyond individual productivity or customer support applications.

[00:14:06] And what I think is really exciting here is that product teams are in a really unique position, not only to leverage AI in our own product development processes, in the way that we incorporate LLMs into our products, but actually to start to bring AI value to the rest of our companies. What that's going to require is not just thinking about the bounds of your digital products, where the software starts and ends, but actually what happens before the software is shipped. What is the market that you're going after? How is this product going to be brought to market? What are all the steps required from marketing teams or operational teams that are going to help to make this product successful?

[00:14:55] And what's so exciting about this is that product teams in most companies are really the early adopters of AI. They've been at the forefront of understanding the model capabilities, at the forefront of understanding how far you can push an AI model to gain leverage. And we're now starting to see product teams carry that impact outside of the product development process and really apply it into other parts of the organization.

The shoe company example

[00:15:24] So a fun example of this. I recently met with a product manager at a major shoe company, a company that probably many of you are wearing their shoes today out in the audience. She's responsible for the e-commerce products within this company, so all of the website checkout experiences, etcetera. Her team had actually already adopted AI in many of their core workflows. They were doing a lot of AI prototyping, they were building some AI-powered search experiences into the app.

[00:16:02] But she noticed that the biggest impediment to hitting their revenue goals was not actually the product itself, it was the ability to localize their products and their marketing campaigns for all their international regions. All the content was produced in the US, but each time they needed to bring a new product to market in an international region, of which there were many — I think 90 regions in this example — they actually had to go through all the preparation to understand, do we have the right assets? Are the products that we're showing in these assets actually available in the markets that we're selling into? So a lot of steps in between launching a product and actually getting it distributed across the business.

[00:16:45] And she recognized that the biggest bottlenecks there were outside of her own workflow, but approached us to inquire about whether we'd be able to help the marketing and the content teams reduce their cycle time for getting new products live. And so they came to us as an early design partner for leveraging some of the new image generation capabilities from OpenAI to build a system that could instantly regionalize marketing assets.

[00:17:18] So take an example of an asset that was produced in the US. You want to regionalize it with more European-centric imagery, translate it into a few different languages for the local markets that it's going to be presented in, and then maybe even use it to brainstorm some new campaign concepts that might resonate better in additional local markets. I thought this was such a powerful example of how all of us as product practitioners really have the opportunity to think beyond our mandated scope, and to start bringing our understanding of AI capabilities and really the art of the possible to drive innovation within other teams.

Summary

[00:18:01] So to summarize. Operate on the frontier of LLMs. If you're not using all these products already today, if you're not playing with the latest AI models, if you're not pushing yourself to leverage new feature capabilities as they come out, you've got to find a way to incorporate it into your process. The space is changing so quickly that the only way to stay on top of it and to understand the opportunities is to actively engage with the changes as they're coming out.

[00:18:30] Number two, embrace the cycle collapse. This is the moment that many of us have been waiting for, to really truly be able to take those agile principles and put them into practice, to get to working concepts faster than ever, to derisk them with our users, to validate ideas faster and cheaper than ever before. So push yourself to accelerate the learning cycles within your own company.

[00:18:56] And three, think like an AI GM. Don't hold yourself to the narrow focus of your product area or your product development process, but really look across the business, understand where there are opportunities for you to help other functions outside of your area or your line of business, and really zoom out to those broader organizational goals.

[00:19:18] And just for fun, to illustrate the point around emerging model capabilities, as I was preparing for this talk I asked this same question to both GPT-4o and o3: what are the biggest changes that AI will drive in product development teams one year from now? And you can see that with the o3 model we actually get original, specific ideas about how AI is going to collapse the cycles of research and prototyping and decision making into a much faster, always-on, iterative process. So not coming only from me, the AI overlords validate this message as well.

[00:19:58] And finally, to wrap this up, I wanted to close on a quote from Jensen, the CEO of Nvidia. Many of you have likely seen this quote before: AI won't take your job, but the person who uses AI will take your job. And I think this is very true within product development teams as well. We are seeing so much opportunity for AI. I expect that this will be a major growth factor for the industry as a whole, but it's a critical moment to get on board, to embrace the change, to understand the impact and the potential within your businesses, and to be a leader for leveraging this technology in the best ways within your own business.

[00:20:38] So thank you again, thanks so much for having me. If you want to learn more about Airtable, we've got a booth downstairs and I'll be around for the next two days to chat with many of you as well.

Q&A

[00:20:55] Host: Thank you very much. Another round of applause for Anthony, please. Thanks. It's nice to have like a proactive message. All right, before we get just a couple of questions from the audience, then we get you to go on your break. I've got to ask the elephant in the room. So you mentioned product teams are some of the first groups to move towards this, and you had that great slogan, be an AI GM. Product teams overindex on entrepreneurial spirit, right, and like disrupt if we've got to. I think I may speak for other people in this room that you feel a little weird about the idea of being the AI GM going to people. It sounds all great in theory, but I don't know, when you overindex on empathy it can be kind of difficult to go to someone and say, hey, do this, not knowing how that could actually affect their job. Any thoughts on how you've managed that at Airtable?

[00:21:42] Anthony: Yeah, I think I'll speak to both Airtable but also to some of the customer experiences that I've heard from as well. In the example that I shared, the creative teams that had been responsible for localizing this content for international markets all felt that this was the lowest value portion of their job. No one was excited about making minor modifications and tweaks to existing designs, or translating copy in situ for 90 different regions.

[00:22:18] Anthony: And so I am definitely not suggesting just go into other functions and powerhouse your way through and tell them you're going to automate their entire function. But I think the real takeaway or learning for me from that example was that some of the folks working in these other functions didn't realize this was even possible, and weren't seeing the emerging capabilities of the latest models and able to imagine how they could really be instantly applied into some of their most time-sucking and painful processes. That's really where I see the opportunity for product teams: to help widen that aperture, take a bigger picture look at the entire process, and then be a collaborator and partner to those functions as well.

[00:23:04] Host: Okay, I'm going to make an editorial decision here. I see that we've got 18 votes up for the ethics part of the ride the wave. I have to think, though, I have to call back out to what Pamela said about us not trying to always be the heroes. I think sometimes these ethics questions, they kind of expect us to solve everything. And I think what we're here today to talk about is how we actually use these things. I will give you a shot though, if there's anything you want to say about it, but you don't have to solve the world's problems in one answer.

[00:23:33] Anthony: Yeah, I think there's a lot baked into this question. But there's actually one question I saw on the prior panel that's related to this, that I thought was interesting and wanted to address, about how do we preserve entry-level jobs in this world where organizations are flattening, we're getting kind of more output out of fewer people within each organization?

[00:23:58] Anthony: One theme that I've actually heard really coming up a lot recently — I was just at a dinner last week where this was discussed — is that many product leaders I speak to are actually seeing that the folks who are joining their organizations earliest in their careers have been the best practitioners of leveraging AI. Because they aren't encumbered by the existing ways of doing things, or the existing processes, or always thinking, oh, I'll go start with the PRD, or I'll go start with figuring out the technical risk of these processes, they're operating in a completely different way that's blowing away others in the organization.

[00:24:42] Anthony: And I think that my advice for anyone earlier in their career who may be listening in or may be here today is, I think it's just a really profound opportunity to come in with a blank slate, with fresh eyes, to look at the way that any company or team is operating and not be stuck by what they're doing right now, but to bring your own lens into these processes as well. So, not a direct answer to the ethics part of this, but I think maybe related to some of the job-related questions that we were getting earlier today.

[00:25:14] Host: I wonder if those entry-level people aren't naturally better at compartmentalizing than some of us that have been doing this a long time, and they separate out AI in general, for the world, for movies, for creative arts, from, hey, I don't have to read 18,000 emails.

[00:25:31] Anthony: Exactly.

[00:25:32] Host: I like this one because it's very direct. Which AI tools would you recommend for a PM?

[00:25:37] Anthony: Yeah. So there's a massive landscape of tools for PMs, but I'll leave you with a couple of thoughts on this. One, a lot of what I was referencing in terms of staying on top of the capabilities of the models, you can actually get that experience directly through the chatbots provided by the model providers. So ChatGPT: if your company's on a team or enterprise plan, you can go and switch the model from the default 4o to o3, and as soon as new models come out, push yourself to test the capabilities of those models.

[00:26:13] Anthony: Two, there's been just a large swath of prototyping tools that have launched over the last few months: Lovable, Bolt, v0, Replit Agent. I think that the space is changing so quickly that it's hard to say, oh, just go use this one tool. But rather I think now is a time of experimentation, where you can really be on top of a lot of different tools and experiencing a lot of different nuanced capabilities across them.

[00:26:48] Anthony: And then finally, obviously shameless plug for Airtable, but we've taken a very model agnostic approach in our product. We let our customers create automations with any AI model of their choosing. What can be really powerful about that is that if you have an existing data set in Airtable, like customer feedback, you can run prompts through different models to actually compare the results. So you can take a feedback data set and say, what does it look like if I prompt a question using an Anthropic model versus an OpenAI model, and actually compare, for the same data, what types of answers you get on different model capabilities.

[00:27:27] Host: All right, you heard it here. Yeah, round of applause for Anthony. Thank you very much.