Product Discovery: How to Innovate with Quick Experiments

Jim Morris at UXDX Community: USA Series: Design Process. Video: https://www.youtube.com/watch?v=FpY81l7WOqM

Readable transcript: edited from the recording's captions for readability (fillers and false starts removed, punctuation and section headings added). Wording is the speaker's own. Timestamps are positions in the video. Names marked [?] could not be verified against the audio.

Build, build, build and the product discovery valley of death

[00:00:00] Jim: Today we're going to focus on three simple words: innovate, quick and experiment. It should be a lot of fun. Just background on me: I'm a Stanford computer science graduate, a long time ago. I started my career selling sporting goods online back when Amazon only sold books. That company went IPO. Then I started a company that was a product reviews platform, and we sold to our competitor, and then the federal government considered us a monopoly, since we were one and two in the market, and so we were spun out. After that 10-year experience in product reviews, I went out on my own as a product management coach. So though I started in engineering, I'm here in the product world, because this is where we're defining what's going to be the next great product.

[00:00:58] I also teach at Berkeley. Right now I'm coaching consumer organizations, a lot of digital health organizations, and then a grouping of developer tools, fintech and other companies, at the organization, the squad and the individual level.

[00:01:16] So what do I see out there in the world? I see a lot of building. Build, build, build. And in that process, people are really falling into what I call the product discovery valley of death. This is where you have great ideas, but you don't test them, and when you don't test them, there's the risk of people not adopting them, not seeing value. You may have great ideas, you may have interviewed folks, but you haven't taken the solutions back to those folks to really see if they would use them. So today it's all about these quick experiments for innovation, so that you do this solution testing step and you avoid the valley of death.

The $200,000 website nobody went to

[00:02:01] Let's start with a story of an entrepreneur who spent $200,000 building a website that nobody went to. He'd found an outsourced company to build his dream idea. But the issue with the dream idea is that he wasn't able to market it, and didn't quite understand what people would see value in. It was valuable to him, but he hadn't done the research with others. So when he came to me, we started from scratch. We built some prototypes to test what people would see in this website.

[00:02:37] I'll tell you what this website does. Imagine that you have a home project and you need a plumber or an electrician. You have to go find somebody. You might be doing it online; a lot of people ask their friends or their family. The premise of this site was that you would ask your neighbors. So how can you facilitate the asking of neighbors? You have to find them. What neighborhood are you in? We tested the interest in this idea, and we tested how people could find out about it, and what we learned led us to create a very barebones, no-code website.

[00:03:19] We started with fake prototypes, quick experiments, and moved to the next set of quick experiments, which was to use a Squarespace site with Google spreadsheets and to drive traffic to that site from Nextdoor and Facebook groups. We'd learned about Nextdoor and Facebook as a large source of these plumber and electrician referrals from our customer interviews. Then this site got too overloaded, as we started to actually drive traffic. And you can imagine Squarespace plus Google Sheets is much cheaper than a $200,000 custom site.

[00:03:54] He then went on to a site using Webflow, another, more advanced no-code tool. Then he also decided that, for this market need, he needed to pivot his idea into a home services subscription service. For a plumber and electrician, whatever you needed, what about a monthly fee to have that taken care of? So through these quick experiments, this entrepreneur was able to pivot and then grow his understanding of his market, and this was in a fraction of the time it took to build the other site that no traffic was going to. So again, quick experiments can help you innovate to get to the right idea.

[00:04:35] This is really what we're looking for. Instead of building all the time, it's learning, building and then measuring. Of course: did you get that traffic? Did the outcome happen? Did you make money? So it's really redefining product development here. Again, people are familiar with this: instead of ideas going to delivery, we're going to add discovery in. Today we'll talk about discovery as innovating with quick experiments.

Innovate: cross-functional teams and engineers

[00:04:59] The first step, the first word, is innovate. You really need your cross-functional team. This is important. What Rory said matters: as you get more folks on the team earlier in the process and you solicit their ideas, you're going to get a much richer set of ideas to start from. That might be in the problem space, that might be in the solution space, but you need this cross-functional team. I have clients that have a data scientist on every cross-functional team. I have clients that have doctors on every cross-functional team. So sometimes you need subject matter experts, sometimes data experts, somebody from customer support maybe. Again, Rory mentioned that team size of up to eight. I think that's great; you've got to keep it low. There are T-shaped personalities that can help you if you have multiple specialties that need to be represented.

[00:05:53] You need engineers. The most common problem I see is that folks create cross-functional teams but don't drive engineer participation. What they're losing is what's just now possible. Engineers are one of our greatest sources of innovation. They can hear the problem and they can apply perhaps a new solution they've been thinking about or reading about, or one that might just come out of their current work stream. So the concept of just now possible is really important for teams, and you might be missing it if you don't have engineers. That innovate word is really important.

[00:06:32] So what are popular engineer-started companies? Well, Google, of course, Facebook, Microsoft, Apple. A lot of the most commonly used companies today had either all or more than half engineer founders. It's really important to make sure that you don't lose touch with your engineers. Keep them involved. Amazon took some just-now-possible technology, secure websites and widespread internet access, to launch in the 90s. My company, PowerReviews, was a product reviews widget inside of a site. We grew to 1,200 clients in 20 countries, all powered by iframes and Ajax. Tesla really started with this concept of a long-lasting battery. So there are enabling technologies that allow certain companies to get over the hump.

[00:07:29] So what are we doing with engineers? We're doing collaborative solutioning, with those data scientists, with those doctors, creating that team and really working up front together. What does each individual do? Certainly you're contributing ideas. One of the best ways is to do sketching. So on that innovation side, I want you to innovate by drawing it. Not just the designer; I want all of you drawing. These are examples of sketches. These are things that drive innovation, when you have people write down their ideas and then you take these ideas to your users in lightweight forms. The Sprint book is a great resource for this type of activity, so I recommend checking it out. You don't need to do these things in five days. You can just take these exercises and apply them every sprint, every discovery cycle.

Quick: Rorschach tests, cardboard tables and micro prototypes

[00:08:22] Okay, let's move on to quick. Quick is a lot of fun. We talked about innovation; well, what is quick? Here's a thought experiment. It's very quick. In 1921, a Swiss psychologist invented the Rorschach test. This test got very popular in the 1960s as a way to understand aspects of patients' personalities. You show somebody something and then they give you a reaction. This is the simplest form of a quick experiment.

[00:09:02] What's a great quick usability experiment? Again, we're trying to figure out: are we confusing the user? I'm going to slightly move you from psychology into the physical world and into technology. Can they accomplish their task? It's usually testing one flow. So what's quick here in usability? Well, one flow was this dining room table that I was planning in my house. How wide would it be? What was the length of it? How far from the ground? Everything was built with cardboard. I had family members sit around the table to test this out. This quick experiment of building it in cardboard rather than in wood, where it was really expensive, led us to understand all the parameters, the usability of this item, and then we were able to build this table. So usability experiments are really valuable. If you can do them quickly, you can learn before you spend all the time in engineering and building.

[00:10:03] What's another form of experiment that's quick? I'm going to show you a backlog prioritization experiment. When we talk about the theory and the practicality, this is a lot of fun. Let's say you've got a lot of items in your backlog. I can tell you that the various frameworks for prioritization aren't that useful, because you're often making up numbers for what's the impact, what's the confidence, what's the effort. If you do some discovery about the ideas and do some user testing, you can learn about the priority with respect to your customer's value. Do they want this thing? That's really what we're interested in. So let's go talk to users. Backlog prioritization: you've got multiple items. How can we do rapid discovery to find out which one will produce the most value?

[00:10:54] This is a thought experiment in itself. I want you to pretend that you work for Zoom and that you have thought of a bunch of ideas for improving breakout rooms. I teach these classes at Berkeley. They have 30 students in them, and each team is three people, so often in my classes I have 10 breakout rooms for five to ten minutes, and I'd love to have more tools to help me manage and improve them. Oops.

[00:11:32] One experiment is, let's say I'm a Zoom user and I've got Zoom on my phone, and Zoom starts sending me push messages to gauge my interest. I call these micro prototypes. Let's read this text here: "Zoom now offers advanced features to manage the success and health of breakout rooms." I can gauge whether somebody's interested in breakout room improvements right away. Would they swipe to learn more? Would they dismiss it? Have them talk about it in front of you. This is a moderated interview. Either I'm with them and this is a prototype, or it's on a computer screen and I'm over Zoom, actually talking about Zoom.

[00:12:19] I show a different idea: "Zoom now offers the ability to listen in to a breakout room without having to enter it." "A new feature: see who has video turned on or off in breakout rooms without having to enter them." "New feature for breakout rooms: Zoom shows you which rooms are the quietest and which rooms have the most people talking." Again, you may not be the product manager for Zoom breakout rooms or Zoom itself. What I'm encouraging you to do is to think of the ideas in your backlog and quick ways to explain them to users and get their interest. Let the users help you, with quick experiments, decide what to work on. Backlog prioritization.

[00:13:05] I did this with a client. They worked in educational technology. We went into a classroom and we showed a variety of ideas, and the team was able to hear the excitement and understand the response as to which of these ideas was most valuable. So you too can go in to your target customers with something simple, almost like a marketing prompt, just to gauge value. Then you can start to build bigger prototypes on it. So try the micro prototype idea.

Test a piece, not the whole experience

[00:13:37] Okay, so again, we were really small with these micro prototypes. What do I see in the market? Lots of stuff. People testing too much, creating these end-to-end experiences. What I want you to do is not test an entire experience. I want you to pick a piece of that experience to innovate. This is an e-commerce flow. How about just picking add to cart and really testing something there? Again, we can be quicker if we can focus. You can certainly test each of these other aspects, but to be quick, you want to test them one at a time, not in these mega testing sessions.

[00:14:19] I also want you to start with a vertical slice. What is really valuable about that add to cart area? Can you launch something? Can you deliver a piece of value end to end, quickly? When I say quick, you may have a vision of that entire product that is reflected in that full pyramid on the left. What I really want you to do is iterate with these vertical slices.

[00:14:47] When I say quick, what does that mean? What time frame am I talking about? Here's a one-quarter timeline where you learn over a four to six week period by testing these four prototypes, then you build it and you launch it, and then you do some measurement. This is not an exact science. Everyone's going to have to tailor this. This is not one-size-fits-all, like Rory said. But what this is, is a way to think about what quick means. Some people in gaming companies or in Google Search might be testing multiple things a day. But here, in that learn phase, you're running through quick experiments to figure out what is working and what's valuable.

Experiment: multiple options and compare-and-contrast moments

[00:15:29] Let's move on to experiments. You're doing great. We've got innovate, we've got quick, now let's do experiments. Again: solution first versus experiment first. In the solution-first world, you've got that one long flow and you just build it, no testing. Experiment first is this respect for the market, which may be able to help you figure out what's actually working: multiple flows, evaluating, maybe building.

[00:15:59] So let's take something where we've got multiple flows. We compare. We make a compare-and-contrast moment that reveals the user's choice. If I pull this all together, when I have multiple flows with experiments, I'm actually bringing more of those sketched ideas, more of those ideas from my team, to the user. So there's more room for innovation if I'm putting more ideas into the market more often. Innovation can often be a quantity game.

[00:16:30] So here we go. I had you focus down to the add to cart. Well, what does experiment mean? Experiment means give me three add to cart experiences. Could be four, could be five. I've tested up to seven experiences in half an hour; it depends on what you're testing. What is valuable to the users? Is it a cross-sell? Is it an upsell? Is it a discount coupon? Is it a service offering? You're going to add these multiple options in. And what experiments do is help you avoid confirmation bias. When you offer no choices to users, effectively you're saying, hey, choose this thing. When you offer choices, you're being open-minded.

[00:17:12] And as you're open-minded, the idea from that data scientist might really come through as the winner, or from the doctor that you put on the team. It doesn't have to be the product manager. It doesn't have to be the designer. In fact, the engineer might come up with the idea that the user selects.

[00:17:31] As we go from big to small: you have this big flow you want to test. This is an analytics experience. Let's just try that dashboard. You still have a lot going on in that dashboard, so let's pick one of these graphics. Is this visualization driving value? Some of you might be analytics PMs out there. Going to the next level, I want you to take that and create multiple options, and that's what brings the experiment aspect in. To recap: you start with that one little module there and you do some experimentation. This is where you're going to build an incredible product one piece at a time, rather than this big-bang testing where you are doing long experiments that really never get off the ground.

Pulling it together: the Zoom breakout rooms prototype

[00:18:21] So let's pull it all together: innovate, quick and experiments. I'm going to show you something and then we'll be out of here. This is great. Zoom breakout rooms: this is a link, and you can go look at this prototype and play with it yourself. This is just an example of giving three options and a compare and contrast, which I built myself in Figma. This is how easy it is with Figma. It's a great tool; there are lots of other tools you can prototype with.

[00:18:43] Here's a scenario of a bunch of folks. I'm going to put them in the breakout rooms, then I have to figure out which one to visit. I'm just giving you screenshots because we're in a presentation situation, but this would be a flow that someone would play with. Option one shows these items. Option two shows some videos that are on and off. Option three has no videos, but it does offer me the ability to listen in. And listening in to Zoom breakout rooms without people knowing sounds creepy, but it's also one of the most highly selected options when I give this prototype to folks. And then at the end there's a compare-and-contrast moment. So again, how you innovate with quick experiments is to make these things that aren't pixel perfect but get your point across.

[00:19:35] Closing out here: don't settle for blah feedback. I like this. That's interesting. This is all meh feedback. This is not what you want; people are being nice to you. I want to hear "ooh." I want to hear, "Oh wow, I want one. Is this available?" You want that emotional reaction from users. A mindset, content, prototypes: these make great experiments. I really encourage you to get out there and innovate with quick experiments. I've done a lot of writing on solution test interviews, so really check this out. And again, hey, stay in touch. Thanks for listening. Here's my LinkedIn and my Twitter, and you can reach out to me through Rory. Thanks for having me.

Q&A

[00:20:31] Rory: Great, thank you very much, Jim. Just as you were going through your experiments with Zoom, I realized we were getting spammed a bit, and I tried to block it, and it popped up on the screen. So that's a feature of this system: when you try to block a chat message, it ends up publishing it instead. Apologies for that brief interruption. But no, I really liked that presentation. Some great examples there. There were two things that jumped out to me that I wanted to ask about. The first is with those fake door types of things, like those push notifications. The feedback I've had when I've tried to push those in organizations is that it will damage our reputation. If we do that, we have to go out to market with a perfectly polished offering, and we can't go out with anything small or micro. How do you deal with that kind of reaction from people?

[00:21:34] Jim: Yeah. I find the world in general, in consumer and in business, to be more accepting of works in progress, just in general, over the seven years I've been on my own, plus before that. I also think that you can run those types of push messages just in a fake prototype interview. You don't have to actually put them out in the app. But for companies that really want to get more of a statistical form of evidence, rather than the evidence that I was gathering in prototypes, the option would be to actually send those push messages out.

[00:22:13] My sense is that people understand that companies are doing market research, and that really what you're trying to do is involve them as a partner. So that may be an extreme, if you were to actually publish those push messages. But what I encourage you to do is to involve your customers upstream to help you make choices, and to fight against that branding conflict, because people actually want you to get better as a company. And if you don't listen to folks, which means actually talking to them and sharing ideas, it's going to be really hard.

[00:22:55] Rory: That's great. I do know some marketing folk that will take a little bit of convincing, but it is a great point that you want that conversation, that two-way communication. The other one, I think Kevin asked online: how do you push for better communication? You said don't settle for meh, but what happens when you're getting meh? What do you do to try and get to that better feedback?

[00:23:26] Jim: Yeah, I'm trying to stretch my teams. My teams are putting out compare-and-contrast moments that are more like usability tweaks, like should the buttons be blue or should the buttons be red. My sense, when you're getting blah feedback, is that you're not trying hard enough. You're not stretching. You're not encouraging or desiring failure. Ideally you'd get to a point where people are like, oh no, I don't want this. You need to probe, in the design phase, with design prototypes, how far your users' interest and value goes.

[00:24:04] My sense is that you may need to add more people to the team. You may need to add more exercises, such as visual brainstorming like the sketching. There are lots of creativity exercises to get teams to put up more ideas. But with quantity of ideas and quantity of testing, you're going to get to those points. And I see it with business software as well: people say "ooh." People want their business software to work well, not just their consumer software.

[00:24:33] Rory: Excellent. All right, with that, that takes us to time. I really enjoyed the talk. Thanks for sharing all of your insights, and I love the examples as well; that really brought it home for me. So thank you very much, Jim.

[00:24:46] Jim: Thanks, Rory.