The What & Why of Continuous Discovery

Teresa Torres at UXDX USA. Video: https://youtu.be/3zA-lOCPrEk

Readable transcript: edited from the recording's captions for readability (fillers and false starts removed, punctuation and section headings added). Wording is the speaker's own. Timestamps are positions in the video. Names marked [?] could not be verified against the audio.

Discovery versus delivery, and the shift to continuous

[00:00:01] Hi, everyone. Welcome to The What and Why of Continuous Discovery. I'm Teresa Torres. I work as a product discovery coach. Fortunately, I've been lucky to work with teams all over the world, some at early-stage startups as small as two founders, up to really large multinational companies with hundreds of thousands of employees. What's great about this is that I've been able to collect best practices and really iterate on techniques that have helped teams adopt a continuous discovery cadence. And that's what we're going to talk about today: what is continuous discovery, and why does it matter?

[00:00:31] We're going to start at the very beginning. I like to define discovery, and the easiest way to do this is in contrast to delivery. All product teams have to discover what to build, and they also have to deliver that product. We look at discovery as all the activities that we're doing to decide what to build. Think about all the decisions that we're making. How are we making those decisions? What data are we collecting? Who are we interacting with? Whereas delivery is the work that we're doing to build and ship scalable, production-quality products.

[00:01:01] Now, especially in the last 20 years, we've seen a big evolution in product discovery practices, and most of us are getting to the point where we're starting to adopt modern techniques. We're interviewing customers, we're usability testing our designs, maybe we're A/B testing a feature once we release it. This is great. We're seeing a dramatic improvement in the way that teams make decisions about what to build. But as we see our delivery practices move more towards a continuous cadence, where we're continuously deploying code to customers, we're seeing the same shift happening in discovery.

[00:01:40] We start to shift from a project mindset, where we do a bunch of discovery up front and then we rely on that discovery while we implement that project. We're seeing teams shift to a more continuous mindset with discovery as well. Instead of doing discovery up front, we're doing discovery continuously, from the initiation of the project or product all the way through as we develop our first iterations, and then we continue to iterate from there.

[00:02:02] This shift from a project mindset to a continuous mindset, I think, is particularly important because we're seeing that digital products are never done. We ship a mobile product, we iterate on it, we update it. With our websites, we're constantly evolving our products and services. We see this even in SaaS software; it's continuously evolving. And even in software that is off premise, outside of our building, like hospital software or software in your cars, we're getting to the point where it's easier and easier to update that software over time. So our jobs are never done. With this continuous development of products, we're looking at how we adopt a continuous discovery model to make sure that we're always building the right things.

[00:02:52] And 2020 was a really good lesson in why this is so important. We're probably not going to face global pandemics every year of our lives, hopefully, but we did see pretty big shifts in the market in 2020, where suddenly a lot of people started working from home. Different needs rose to the top. Teams that had a continuous cadence of discovery were able to quickly adapt to these changes in the marketplace. Like I said, hopefully we're not going to see global pandemics every year, but our markets can be disrupted all the time. We see new competitors enter the field. We see technology disrupt what's possible. We see, as we release every new version of our product, our customers' needs, pain points and desires evolve as our product evolves. And this is really the argument for why a continuous cadence matters so much.

A definition: weekly touch points with customers

[00:03:42] I know from working with teams that a lot of teams think they've already adopted a continuous cadence when really what they're doing is project-based discovery. So I want to start with a really clear definition of what I mean by continuous discovery. I define continuous discovery as, at a minimum, weekly touch points with your customers, by the team building the product, where that team is conducting small research activities in pursuit of a desired product outcome. Each line of this definition is pretty critical, so we're going to break it down line by line.

[00:04:15] We're going to start with the first one. Why do we need to engage with customers every week? Most of us on product teams are making product decisions every day. Some of them are big strategic decisions: what goes on our roadmap? What customers are we going to serve? What opportunities are we going to pursue? Others are smaller, more daily decisions, like where do we expose this feature in the interface, or what do we label this button? We're also making medium-sized decisions. How should the data model work? What should be exposed through the API? These are all decisions that can benefit from customer input.

[00:04:51] The more that we engage with our customers, the more that we keep this continuous drip of interaction with them, the more likely we're going to get feedback on most of these decisions. Whereas what we do in a project world is we tend to do a bunch of project research before those big strategic decisions, and then we just do the best we can on those daily little decisions. If we talk with our customers every week, it allows us to engage with our customers continuously and to get feedback on many more of our decisions. And it's not just more of the decisions; it's that we're able to get feedback much earlier in the process.

[00:05:28] A lot of us have adopted what I call a validation mindset, where we do most of the work up front. When we're all done, when we've got the prototype ready to go and we've designed everything, we put it in front of customers and we validate it. We get it right. We do need to do that validation research, but the challenge with the validation mindset is we're often getting feedback too late. At the point that we're getting feedback, our engineers are waiting for it to go into the next sprint. We have very little time to make changes, and we tend to just make the small cosmetic changes. We don't have time to make the big changes that would dramatically improve the product based on our customers' feedback.

[00:06:06] What I like to see teams do is actually get feedback on their ideas when they're still half-baked, when they're still pencil drawings or whiteboard drawings. This can feel a little uncomfortable until you get used to it. But teams that do this start to co-create with customers. They start to adopt what I call a co-creation mindset, where we're inviting our customers to create with us. Now, inevitably, when I share this, somebody brings up that Henry Ford quote: if I had asked customers what they wanted, they would have said a faster horse.

[00:06:38] So I want to be really clear. We're not going to ask our customers what we should build. The way that we're going to co-create with them is we're going to take our knowledge, our expertise about what's possible with technology, and combine it with our customers' knowledge about their own world, about their own needs, about their own pain points. This combined knowledge is what's really going to empower us to create better products. So our customers do need to play a role, and they do need to play a role continuously, because that's what's going to ensure that we stay on track and that we keep building a product that's going to work for our customers over time.

By the team building the product: the product trio

[00:07:11] That's our first line of this definition. The second is "by the team building the product." Let's get into what I mean by this. It starts with this idea of a product trio. This is becoming a more common model. Some people call it a triad, some people call it a three-legged stool. It's this idea of the product manager, the design lead and the tech lead being jointly responsible for those discovery decisions. Remember, discovery is just the work that we're doing to decide what to build.

[00:07:38] Historically, product managers have owned these decisions. They write up requirements, they hand them off to their designers. The designers do all the design work and they hand it off to engineers. The problem is we see a lot of rework in this handoff model. The designer gets the requirements, they run into some problems, the requirements have to evolve. The design is done, it gets handed to engineers, they run into challenges, and now we have to redo design and redo requirements. With a trio-based model, we're saying: let's have these three roles work together from the beginning. Let's have them drive our discovery decisions and jointly decide what to build.

[00:08:10] You probably have more people on your team. Depending on your DevOps strategy, you probably have some QA folks on your team. You definitely have more engineers on your team. You might have some business folks. If you're working on a data-heavy product, you might have data analysts or data scientists on your team. If you have the luxury of a user researcher, they might be embedded on your team. Here's the thing: it can be easy to fall into the trap of "let's include everybody in discovery." The problem with that is the more people involved in every decision, the slower you're going to go.

[00:08:47] So when we think about this idea of a trio, it's not a hard and fast rule. On some teams, the trio is a quad. I want you to think about the idea behind a trio, which is to have the cross-functional roles represented for each decision where they're needed. In most discovery decisions we need at least these three roles represented. But for some of your discovery decisions, for example if they're about your go-to-market strategy and you're trying to learn the best way to bring this product to market, you might invite your product marketing manager to be part of those conversations and part of those decisions. Or if you're working on your data model, you're probably going to invite your data analyst or your data scientist to be part of those decisions.

[00:09:24] So this idea of a trio can flex. Just know that the more people you invite, the slower you're going to go. Think about bringing in the right people for the right decisions rather than having everybody in every decision. But we know that we want to see equal footing. We want true collaboration between the product manager and the designer and the tech lead. That's your starting point: make sure that those three people are driving your discovery decisions.

Small research activities in pursuit of an outcome

[00:09:50] We've covered weekly touch points with customers by the team building the product. Let's talk about these last two lines: conducting small research activities in pursuit of a desired product outcome. This is where we're going to get into the heart of the definition. These two lines work really well together, because when we talk about continuous discovery, if your understanding of an interview is an hour-long interview, and you think when you interview you've got to interview 12 people, you can't do that every week. That's not sustainable. We've got to turn our research activities into smaller, bite-sized activities that we can do continuously over time.

[00:10:23] Same with the outcome. I see a lot of teams that forget that our job as product teams is to create customer value in a way that creates business value. Our job is not just to serve the customer, but to serve the customer in a way that creates value for our business. That's where this fourth line becomes so critical: we're not doing research for research's sake, we're doing research to help us reach our desired outcome.

[00:10:48] Let's talk about how we do this. I like to use this visual I created, which is called an opportunity solution tree. It's designed to help the trio chart out their best path to a desired outcome. I argue that good discovery starts with a clear desired outcome. This is different from what a lot of teams do. A lot of product teams are still in a roadmap world, where they start with a list of fixed outputs. That's an output-led way of doing product management, where they're just delivering outputs. What we've seen over the last 10 to 20 years is teams shifting from an output mindset to an outcome mindset. Instead of saying, what things should we build, we're asking, what impact should those things have?

[00:11:27] I want to be clear: your desired outcome should represent a business need. I do distinguish between business outcomes, which tend to be your financial metrics like increase revenue or grow market share, and product outcomes, which are metrics that measure behavior in the product that, if you drove them, would drive your business. I want to see a product outcome at the top of your tree. Again, a product outcome is a metric that's measuring a behavior in your product, but it needs to be tied directly to a business outcome, one of those financial metrics that's creating value for the business. This is going to ensure that your team is not just creating customer value, but creating customer value that drives business value.

The opportunity solution tree

[00:12:18] Once we have an outcome in place, we need to discover the opportunities that will drive that outcome. Opportunities are customer needs, pain points and desires. This is where we're looking at how we create customer value. Again, there are lots of opportunities that we could address, but we're only going to consider the ones that have the potential to drive our desired outcome. This is a big deal, because it allows us to resolve the tension between business needs and customer needs. We see this all over the place, where a product team is constantly having to decide what's more important, the business need or the customer need. An opportunity solution tree is going to help you resolve that, because you're looking at the customer needs that will drive those business needs.

[00:13:04] Once you discover those opportunities, you of course need to discover the solutions that will deliver on those opportunities, and we run experiments to help us figure out which solutions those should be. We're going to break this down even further. We're going to start at the top. Setting an outcome should be a two-way negotiation between the product leader and the product team. What do I mean by those terms? By product leader I mean your chief product officer, your VP of product. If you're a really large company, it might be the head of your business unit. It's the product leader that has the across-the-business view of what the business needs right now.

[00:13:40] The product leader is saying: given our strategic context, given our company goals, this is what we need your product team to deliver to create value for the business. The product team leading this negotiation is that product trio: the product manager, the designer and the engineer. They're communicating to the leader what they know about that outcome, what impact they think they can have on that outcome, and in what time period. It's really important that this be a two-way negotiation, because the team and the leader need to be aligned on the best way for that team to create value for the business.

Interviewing every week and automating recruiting

[00:14:17] Once that outcome is in place, we're going to start to kick off our continuous cadence. We're going to look at a few core habits that are going to help teams chart their best path to a desired outcome. Reaching an outcome is a really ill-defined problem; it's wide open. One of the best ways that we can add structure to this ill-structured problem is to start to map out the opportunity space. Again, opportunities are customer needs, pain points and desires. I like to see teams interview to discover opportunities. There are other ways to discover opportunities. We hear about them from our sales teams and from our support tickets. If we have the luxury of observing our customers, we can uncover opportunities that way.

[00:15:00] But one of the easiest ways to have a continuous drip, continuously listening for opportunities, is to just interview at least one customer every week. And remember, the goal of that interview is to uncover customer needs, pain points and desires. This is different from what a lot of teams do. A lot of teams use their interviews to get feedback on their solutions. You're going to see in the later part of this talk that we have a better way to get feedback on your solutions. So think about interviewing as a way to discover opportunities.

[00:15:33] As we discover those opportunities, I encourage teams to take the time to map out the opportunity space. Visualize, using this tree structure, all the opportunities you could consider pursuing to reach your desired outcome. This gives the team a big-picture view of all the different paths that they could take, and this is what helps them make more strategic decisions. To be able to take an inventory of all those opportunities, we need to start this critical habit of interviewing every week.

[00:16:05] The reason a lot of teams struggle with developing that habit is because they don't know how to recruit continuously. When we do project-based research, we can just send an email to 500 people, hope 12 respond, and then book 12 interviews over the next two weeks, and we're good. That's not sustainable. We can't do that week over week. The key to making continuous interviewing work is you have to automate the recruiting process. There are a number of ways to do this. It's going to look a little bit different based on your industry, based on your product, based on your target customer. But I will remind you that I have put this into practice in a lot of organizational contexts, so I promise you there is a way to do it for your team. It's just going to take a little bit of experimentation and iteration.

[00:16:46] One of the most common ways, which works for the vast majority of products, is to recruit people while they're visiting your product. What does this mean? If you're a consumer company, or if you're an enterprise SaaS company where people are in your product all day, every day, then you can recruit them while they're visiting your product or service. It's as simple as popping up a message, like you're seeing in the visual here. Or you could do it in a more subtle way: you could include it in your newsletters, you could include it on their account page, anywhere that your customer is going to see the ask while they're using your service.

[00:17:22] What's happening here is we're just saying, "Hey, we'll give you a $20 Amazon gift card in exchange for 20 minutes of your time." The way to make this effective is to make a small ask in exchange for a large reward. The example that you're looking at, Snagajob, is a job board that focuses on retail workers and restaurant workers here in the US. These types of employees don't typically make $20 an hour, so giving them $20 for 20 minutes of their time is a small ask in exchange for a big reward. It's going to take some experimentation to get this offer right. You're not going to just launch this and immediately get this continuous flow of participants. You're going to have to experiment a little bit, but this is one of the most effective ways to automate your recruiting process.

[00:18:08] Let's look at the second way. If you're an enterprise company and your end users or your customers don't spend a lot of time in your product, what you're going to want to do is look at your customer-facing teams: your sales teams, your account management teams, your tech support teams. Start to look at how you can get them to help you recruit for interviews. This can be as simple as defining some triggers for them: if you talk to a customer that exhibits this behavior, ask them if they'll participate in an interview.

[00:18:36] The same criteria exist here. Have your support teams offer a big reward in exchange for a small ask. For enterprise companies, cash is not usually an appropriate reward; a lot of employees can't take gifts. So you want to think about what you can offer that creates value. This could be as simple as a discount on your product, a free month of your service, or access to a premium helpline. You can invite them to a webinar that teaches them a new skill. There are a lot of ways to think about this, but you can use your customer-facing teams to help you recruit.

[00:19:14] I have worked with a number of companies that have teeny tiny markets. When I say teeny tiny markets, I mean they sell to the movie studios in the US and there are six of them, or they work with Canadian medical schools and there are only a few dozen of them. If you're working with a teeny tiny market, what you're going to want to do is set up a customer advisory board. Don't think about this as a focus group. Think about it as building long-term relationships with your customers, where you're engaging with them week over week and working closely with them to develop your product. I still want you to meet with them one-on-one and to think about it as an interview, but you're basically setting up a long-term relationship with a small number of customers.

Asking for specific stories

[00:19:54] Once you have your customers in the room, we've got to look at how to ask the right questions. Remember, we're trying to uncover customer needs, customer pain points, customer desires. The number one thing I want you to avoid is speculation. We tend to ask our customers who, what, why, how questions. These direct questions are challenging, because all humans are pretty bad at answering them accurately. We're really bad at summarizing our own behavior. We're not quite sure what we do, how often, or when. We don't actually know what our needs or pain points are; it's really hard for us to verbalize this.

[00:20:27] What we want to do instead is collect specific stories. We want to ask our customers about specific instances in which they do things. This could be as simple as, if I work at Netflix, I might ask them, tell me about the last time you watched Netflix. If I work in an enterprise context, I might ask somebody, tell me about the last time you had to do this workflow. The key is to keep them grounded in that specific context, so you start to collect reliable feedback. Those stories are what's going to help you identify those opportunities.

Comparing sets of solutions

[00:20:57] Once you've started to map out the opportunity space, you're going to very quickly want to prioritize one target opportunity. This is a strategic decision: what customer needs do we want to address? You want to look at which ones are most likely to drive your desired outcome. Once you have a target opportunity, you're going to generate as many solutions as you can. The reason for this is we want to work with sets of solutions, so we're comparing and contrasting our top solutions against each other. Most product teams are overwhelmed with ideas, but they're not overwhelmed with ideas for the same opportunity. That's the key difference. You're choosing a target opportunity, and then you're working with a set of solutions within that opportunity.

[00:21:41] Why does this matter? When we work with one idea at a time, we tend to set up what's called a whether-or-not decision, where we ask, is this idea good or not? The problem with this type of question is it sets us up for confirmation bias. We're going to see all the evidence that says our idea is good, and we're going to miss all the evidence that says it might be flawed. So instead we're going to set up a compare-and-contrast decision. We're going to ask which of these ideas look best for addressing this target opportunity. Remember, we're going to compare solutions that address the same target opportunity.

[00:22:18] As we experiment, which we'll talk about in a minute, we're looking for a clear front runner. What does that look like? This is a picture of Usain Bolt. I want you to keep this metaphor in mind. Usain Bolt at one point was the world's fastest hundred-meter runner. If you saw him running around a track and I asked you, is he fast, that would be a hard question to answer. Is he fast relative to what? Is he fast relative to a cheetah? Probably not. Is he fast relative to a Tesla in the first hundred meters? I actually want to see that race; I don't know who would win. But if I ask you, is he fast relative to other humans, the answer is clear. Absolutely, he's definitely fast. And this is what we're looking for: a clear front runner when we compare and contrast our solutions against each other.

Testing assumptions, not whole ideas

[00:23:02] So how are we measuring them? That's our last habit. We're going to break our solutions up into their underlying assumptions. That's what allows us to go from big, project-sized experiments down to continuous, bite-sized experiments. Testing assumptions is a lot faster than testing whole ideas. What do I mean by assumptions? Each of our ideas is based on all kinds of assumptions. We have desirability assumptions: we're making assumptions about why our customers might want it, and whether they're willing to do what we need them to do. We're making assumptions about whether this is good for our business. We're making assumptions about whether it's even possible. We're making usability assumptions: can people find it, do they understand it, can they use it? We're making ethical assumptions: is there any potential harm in building the solution? There are a lot of types of assumptions.

[00:24:02] What I recommend teams do is take their top three ideas, break them down into their underlying assumptions, and then rapidly test those assumptions. When I say rapidly, I mean take three ideas on Monday, break them down into their underlying assumptions, take the top two assumptions from each idea, and collect data on all six in the same week, so that by Friday you're able to start to look at which of these might be your front runner. This really is a continuous, fast cadence.

[00:24:30] We went through this really quickly. We started with defining a clear outcome. We talked about interviewing to discover opportunities. We talked about comparing and contrasting solutions by testing assumptions. There's a lot to get into here. If you really want to learn more, I have a book out called Continuous Discovery Habits. It's designed to be a product trio's guide to a structured and sustainable approach to continuous discovery. It's going to teach you how to adopt simple habits that you can do week over week, that will help you run this entire process in a way that's sustainable, that you can do alongside your busy delivery schedule.

[00:25:07] If you want to learn more about this book, go to continuousdiscoveryhabits.com and you can get a nice overview. All right, thank you, everybody. I would love to keep the conversation going. Feel free to learn more about me at producttalk.org, or reach out on Twitter, @TTorres. Thanks.