How to Build an AI-Powered Product Development Process

Anthony Maggio at UXDX USA. Video: https://youtu.be/7eou4QDfb54

Readable transcript: edited from the recording's captions for readability (fillers and false starts removed, punctuation and section headings added). Wording is the speaker's own. Timestamps are positions in the video. Names marked [?] could not be verified against the audio.

From AI hype to practical use cases

[00:00:00] Hello, everyone. Nice to see you all, a great crowd for a Friday afternoon. I'm Anthony, and I lead our product management team at Airtable. For anyone who's not familiar with Airtable, Airtable is the fastest platform for building really great applications to run your business. I wanted to spend today talking about AI. I think last year was really a big year for the hype of AI, and many of the conversations I've had with product teams and other product leaders over this past year have really been about how we go from this hype and excitement around AI to actual transformative and practical use cases, where we can bring AI into our organizations and change the way we're working.

[00:00:51] I consistently hear that most companies have not really figured out ways to do that yet, and they get very excited hearing about some of what we've been doing at Airtable and how we've brought AI into many of our digital product development processes. So what I hope to share with you today are some tips and best practices from what we've learned by really prototyping with AI tools over the last year.

A feedback process that worked too well

[00:01:13] I want to get started with a little story, a little backstory, and I hope this resonates. I heard from several of you over the last two days that user feedback management has been a challenge in your organizations. Back in 2022 this was a huge problem for us at Airtable. We had just gone through a really rapid period of growth within the company. We had grown from about 150 to close to 700 people across the team, and our user feedback management process was a total mess. We were getting feedback pouring in through Slack channels and emails, and coming in from our support teams and our go-to-market teams, and it was really disorganized. Nobody could make sense of it. It was hard to tell what was urgent, what was important, what we should take action on, and it was impossible to report on.

[00:02:05] So we really set out to build a better process around how we understood and ran intake with our customers and gathered sources of feedback from many different channels. We put together a team of cross-functional folks from product operations, sales, solutions engineering and customer success, and set out to solve this problem. We really wanted it to feel like feedback was not going into a black box, but that we could actually utilize it to generate insights and take action on it. So we put together this working group, and our goal was pretty simple: let's just make it as easy as possible to submit feedback and to aggregate this feedback, and let our product teams utilize it to inform our strategy and roadmap.

[00:02:55] I remember that in late 2022, after we built this new process and this new application, we launched it at our second-half sales kickoff, and we got a standing ovation from our go-to-market teams. They were so excited about the idea that our customer feedback was now going to drive priorities and change the way we were roadmapping. There was so much excitement and anticipation around that idea. And as you might have guessed, was it a success? Well, kind of, not quite.

[00:03:31] We rolled out this new feedback process and we did a lot of what we set out to do. We did centralize our feedback across many different sources: recorded calls out of Gong, Zendesk tickets, Salesforce, Slack, forms being submitted by our users directly through the product. But we also created a new problem, which was that we went from getting dozens of pieces of product feedback every week to all of a sudden getting hundreds or thousands. We made it so easy that we dramatically increased the volume of submissions we were seeing from our customers and our teams. That created a new problem: our product ops teams and our PMs all of a sudden felt inundated with reviewing, triaging and taking action on this huge volume of data. It was insightful, but it was such a time suck that they just couldn't keep up. So once again, unfortunately, after about a month it started to feel like all this feedback was just going into a black box again.

Mapping the whole product development process

[00:04:41] We could have given up there, but right around this time, in early 2023, ChatGPT came out, and we started to see the power of AI and of these LLMs, these large language models. Our product operations manager came to me one day and said, "You know that feedback process that's kind of broken and still stagnant, even after all this investment we've made? Well, I think AI can actually help us solve this problem. What if we could use AI to automate a lot of the tedious and manual parts of running this process, really make it more useful for our teams, and help them automatically generate insights from all this great data we're collecting?"

[00:05:27] I was very excited about this. I told our product ops manager, "Great, let's go do it." In fact, maybe I added a little scope creep along the way and said, "Why stop at feedback? Let's look at our whole product development process and think about all the places where we can integrate AI and large language models to accelerate some of the more tedious and manual work our teams are spending a lot of time on today." So we went through this process mapping exercise. We looked at every stage of product development: discovery, planning, execution, measurement, launches. For each stage we embarked on a journey to ask how we could really reduce, or just eliminate, all of the administrative work that our PMs, designers and engineering managers are spending so much time on today. What I'm going to share with you are some of the areas where we've actually implemented AI, and hopefully these will spark some inspiration for you.

Smart triage for feedback

[00:06:33] First up was the feedback management process. As I said previously, teams were spending a ton of time manually reviewing every piece of feedback, and we integrated a new AI function into our feedback management application in Airtable that we call Smart Triage. Now, with Smart Triage, the large language model is able to look at each piece of feedback and determine: is it relevant to a product team? What product area is it associated with? Is it related to any themes or existing known product gaps? Is it connected to an item that's already on our product roadmap? All of this happens instantly and automatically across all the different sources we're gathering user feedback from, fully automating what was previously a manual triage process.

[00:07:26] Now, with all this data categorized, tagged and structured in an easy-to-use application, our PMs start getting proactive insights. They get emails sent to them alerting them to new trends or new themes we're seeing in the feedback related to their product area. It becomes much more measurable, actionable and easy to track over time. Pretty amazing. We even see now that as we kick off new features and start developing new products, we can tie them back to specific users who submitted related product feedback, and include them in discovery sessions or show them early designs and prototypes of what we're building.

Prioritization and strategy drift

[00:08:10] Next up was prioritization. With all of this user feedback data centralized and organized, we started to bring it into our prioritization process and use it to inform the next items we might work on in the roadmap: how many people have requested a particular feature item on our backlog, and what revenue is associated with the accounts that have requested that feature. But we also took it a step further. We often see a problem that I think of as strategy drift. You go into the year, you hold your annual planning sessions, and everybody comes out feeling super aligned on what the goals are and what results you want to see across your product org for the year. Then, as you get into the actual day-to-day work, some of your teams start working on other local priorities, and maybe one of the results you set out to achieve doesn't have enough resourcing or enough impactful projects aligned to it.

[00:09:17] So we looked at how we could use AI to start solving this problem as well. Similar to the feedback process, we built an automated matching function that asks, for every item on our roadmap, whether there are any key results the item is related to or expected to inflect. Now, with our roadmap automatically aligned to all the existing results and goals we'd set as a product organization, we could get visibility into where the holes are: the projects that are falling behind, or the key results we set out to achieve that don't have projects associated with them. This provided unprecedented visibility into everything from the top-level goals we'd set as a company down to the projects, and their current statuses, that were underway to help us achieve those goals, resulting in better aligned roadmaps connected to our top priorities.

A weekly digest of blockers

[00:10:16] At that point we were feeling pretty good. We had feedback management under control. AI was providing automation around our prioritization process. We were getting great feedback from our partners that our product roadmaps were really aligned not only with the voice of our customers and users but also with our company priorities. And we started looking at what's next: where else could we integrate AI functionality to help us work better, smarter and more efficiently?

[00:10:48] Our head of engineering, our head of design and I were brainstorming some areas we were struggling with on a regular basis, and we thought, "I wonder if we could actually use AI to help us ship products faster." We have all these bottlenecks across our teams. We constantly see initiatives getting stuck or blocked by a dependency, or needing input from some leader or functional stakeholder. Even at a company our size, we may have several hundred projects ongoing at any given time, so it's pretty challenging to keep up to date on all those blockers, where things are getting stuck, and when something needs action or intervention.

[00:11:36] So we had this idea: what if we could take all of the data we have about all of our projects, the sprint summaries, the status updates, the comments being left on individual tickets related to those items, run all of it through a large language model, and get a digest at the end of each week showing us where there are initiatives that are at risk, falling behind, or maybe blocked in a way we're not aware of? As a result, we found we could really quickly uncover blockers and dependencies we had no idea existed, meet with teams right away within the week, help get them unblocked, and help get projects moving faster.

[00:12:24] This was what I was actually most excited about, because over the last two quarters we saw a measurable increase in the number of initiatives we brought to GA. We saw over a 20% increase in the number of features that went from early conceptual stages or betas into actually shipping, compared to prior quarters. So this has been really transformative for us as leaders, in being able to help the teams move more quickly and eliminate those points of friction across the product development process.

Suggesting launch moments

[00:12:57] Last, up until now the use cases I'd focused on were really about our internal product development teams, but we also wanted to look at areas where we could improve the way product interfaced with the rest of the organization. A big pain point that came up here was launch management. Unlike the other use cases I shared, launch management tends to be very cross-functional. We have to coordinate with our sales teams, sales enablement, support, customer success and marketing, and make sure everybody's ready and aware of launches, the features included in them and what's new, and that we have all the support articles in place.

[00:13:43] This has been a challenge to manage, particularly for smaller launches, the ones you don't have everybody aligned on and rallied around from the get-go at the start of the year. As the dates for projects shift, or as challenges come up through the development process, it was really hard to identify when we wanted to have these tier two and tier three launch moments and what they might look like. So we started prototyping using AI to suggest launch moments to our product marketing team. We feed in all the data about the various initiatives and their planned GA dates, and it comes up with a suggested calendar and even an initial launch narrative that the marketing team can review, to decide where we want to invest in promotion around these launches, what we want to play up, and where we want to invest marketing resources in talking to our customers. This has been really transformative in making our launch moments both easier to coordinate and higher impact and higher quality for our customers and users.

[00:14:59] So we've now integrated AI touchpoints and use cases across the whole product development process. These are just a few select use cases I wanted to share today, and it still feels like we're just getting started. Now that the product teams and product ops folks are really hooked on AI and seeing the transformative power of bringing LLMs into this life cycle, our teams are coming up with new use cases all the time that they're incorporating into our process. All of what I shared has happened over the last year, so for today I wanted to reflect a little on everything we've learned over the last year of building out these processes, and leave you with some advice that I hope will be useful for implementing AI use cases within your own organizations.

Map the process and embed AI in the workflow

[00:16:00] First, starting out with a business process map is a really great way to identify the highest-leverage opportunities for automation within your teams. This is a great place, by the way, to partner with your product ops or design ops teams, if you have them, to look at your end-to-end processes and really identify which steps in the work we do are the most manual and the most tedious, and are impacting the largest number of employees across the company. Which of these are actually a good fit for LLMs and generative AI tools to help solve, because they're not great for everything, though there are many use cases they can be really powerful and transformative for? And what is the impact of actually solving them? Mapping out that process, in those early whiteboarding sessions I shared, was a big unlock for us in building a roadmap of AI product operations initiatives, and in constantly revisiting where we could have the most impact by implementing these features in our apps and tooling.

[00:17:11] The second principle I'll share is to embed AI in your applications and your process workflows. The real value we see from AI and large language models is in utilizing them with your own data as part of recurring processes. A question I get a lot from other product leaders and product teams is, "Couldn't we just do some of this stuff in ChatGPT?" The answer is yes, you can, but you're then relying on every user of ChatGPT or Copilot to become good at writing prompts, to know how to interface with large language models, to remember where they should be using AI as part of their process, and to know when to go out and interact with one of these models. In contrast, when you insert AI steps into your workflows and applications, a very small number of people can create the right prompts, iterate on those prompts, explore the outputs, and scale that impact to your entire organization.

[00:18:19] This use case up here is a legal checklist the team is currently prototyping, where we can feed in any product initiative that's on the roadmap and flag potential legal risks or items to review with our legal team before it goes into a release or GA. I think it's exemplary of the recurring-process power of integrating your own data into these prompts, and of ensuring you're leveraging AI in a consistent way as part of your ongoing, repetitive workflows. To transform your process, you really need to put AI into the process.

Test internally and ride the wave of model progression

[00:19:04] Number three, and this is a big one, is to test and iterate on what you create with your internal teams and partners. I can't tell you how many conversations I've had recently with other product leaders or product teams who had a negative experience testing some AI tools, or whose first impressions weren't great. When I asked them how they tested their model inputs or prompts, or what they were building for their teams, there wasn't really a whole lot of testing. As product teams, we would never build a feature and ship it directly to our customers without any internal testing, QA or iteration, and the same should be true of our internal tools. Any apps or processes we're building and using to scale the way we work as a product organization should go through a similar, maybe smaller, life cycle of building an MVP, testing it and getting feedback from the users who will be interacting with those applications.

[00:20:12] This is something I've really seen our product operations team embrace: this concept of iterative learning and user discovery within our own teams, putting out prototypes to test and refining them. Because these large language models are non-deterministic, the way they work requires some iteration on the prompt, and on exactly what data you're putting in, to get a great output. So it is critically important to do this type of testing and iteration with any AI tools you're using.

[00:20:45] Number four, and I want to spend a little bit of time on this one: riding the wave of model progression. It has been really unbelievable to me to see the speed with which these large language models are acquiring new capabilities. Back in 2022 we had ChatGPT, GPT-3.5, which has widely been characterized as being able to solve five-second human tasks. Fast forward into 2023 and 2024, and we've seen a range of new models come out that are extremely powerful, and I think most people and most companies are still under-leveraging them and would be surprised by exactly how powerful these models have become. It's no longer just simple tasks like categorization or translation that they can power. They're capable of fairly advanced reasoning, identifying trends in data, extracting themes and content from documents or studies, really advanced tasks that would typically take humans a lot of time.

[00:22:10] And they're only getting better. We've already seen another model release from OpenAI just in the past week, and we've seen hints that even stronger and more powerful models are coming out over the course of this year. So the advice I give to any organization considering implementing AI tools is to pick technology vendors who will give you choice and selection of models, and allow you to select the best and latest models for each task or workflow you're looking to automate. At Airtable this has been a really important principle for us from the get-go. We built all of our AI tools to be multi-model from the start, to allow our customers to select which models they'd like to enable or leverage within their workflows. But whether you work with Airtable or any other vendor, I think it'll be really important to ensure that whatever tools you use to power AI in your processes won't leave you stuck with a certain model and that model's progression.

[00:23:15] Like I said, many teams I speak with are really interested in AI but haven't figured out exactly where to start, and I hope these principles give you something tangible to take back to your organizations, and if nothing else that they spark some inspiration for use cases you might be able to prototype and implement. I'll leave you with one slide; this is the one to take a photo of. There are a number of resources here if you'd like to learn more. When you get back to your virtual offices on Monday, you can scan these QR codes and get a live demo of Airtable AI for product teams. We have some resources on Airtable for product and design operations, as well as some case studies of other customers who are starting to implement many of these practices in their day-to-day. Thanks so much.

Q&A

[00:24:20] Host: All right, we've got some questions. Let's see. There you are. Hello, that was great, thanks. A big part of using AI to optimize processes and workflows involves training the models. How did you structure the model inputs and information to train the models?

[00:24:45] Anthony: This is a really great question, and it's actually one of the most common misconceptions I see when interacting with our customers who are implementing AI. The latest models, like GPT-4 and Claude 3 from Anthropic, for most use cases do not actually require much training. There's a big difference between model training and effective prompt engineering, and what we've seen is that the vast majority of use cases can be achieved through prompt engineering, without really any model tuning or training required. These large language models were trained on such enormous sets of data that they have really widespread general knowledge of products and companies.

[00:25:36] Anthony: There are additional techniques that can be utilized in addition to prompt engineering. There's a technique called few-shot prompting, where you give the model, through your prompt, a few examples of how you'd like to see the output. For example, if you were writing product briefs and wanted all your product briefs to come out in a consistent format, using your own company's voice, language and tone, you can provide some of those examples, and it'll improve the quality of the output you get, and it will be much closer to what you would see from an employee writing those briefs. That's something we really encourage our users and customers to experiment with, and we've found that, by and large, for most process management use cases you don't need to train a custom model or do any kind of in-depth AI training.

[00:26:36] Host: Excellent. Concerns with legal: did legal have any concerns about your team using AI to assess the risk?

[00:26:44] Anthony: I think there's probably a question behind this question, which is that a lot of you are probably coming from organizations that are a bit AI-wary, or have concerns about using AI for different use cases. The way we built our AI features at Airtable, we really started with the concerns of enterprise customers in mind. For example, any prompts run through the Airtable platform will never be used for any kind of model training by any of the vendors we use. We've also integrated models that run on our own infrastructure. Amazon has a product called Amazon Bedrock that allows you to run models inside your own AWS instances, so the data that is generated never actually touches the model provider, like Anthropic. That was a major concern for us upfront that we really invested in, to make sure the AI products we built would be the most safe and secure enterprise-grade AI.

[00:27:52] Anthony: As a result of that, no, our legal team did not have any concerns with us using these models to identify legal risk. They actually really appreciate it. Our legal team doesn't have the resources to sit in every product review and every design conversation to review every initiative themselves. So this is a use case that, as I mentioned, we're currently prototyping and excited about implementing, because we think it'll help us be much more proactive about which initiatives require a heavier level of engagement from our legal team, and which ones they don't need to spend as much time on.

[00:28:33] Host: That's excellent. I think we have about 46 seconds, so I want to get you to answer one more. "Great comment, incredible work. Can you please share guidance on how to get leadership buy-in to pursue AI initiatives like this?"

[00:28:48] Anthony: This is a great one. I would suggest the best way to get buy-in is to start with a specific use case. What I've seen consistently is that when you just talk about AI in the abstract, "We want to do cool stuff with AI," nobody gets excited about that. Your leaders are not going to advocate for bringing in another tool, another AI product that's going to add a line item to their budget. But when you can attach it to a concrete use case, like user feedback management, and talk about the time savings, the insight unlocks and the potential impact on your product strategy and roadmap, for something concrete, we see a much higher level of buy-in and interest, not only from your leadership teams but also from the IT teams, the security teams and the AI review boards, because it gives them confidence that they understand exactly how AI will be deployed in the context of these workflows. It's much easier to grapple with than broad and general applications that may be a bit more frightening for them.

[00:29:57] Host: Thank you, Anthony, that was really, really great. I appreciate you. A round of applause for Anthony. We love Airtable. Thanks, great job.