Product Discovery: How to Innovate with Quick Experiments

So today we're going to focus on three simple words, innovate, Quick and experiments, and should be a lot of fun. So just background on me, I'm a Stanford computer science major graduate a long time ago. Started my career selling sporting goods online back when Amazon only sold books. That company went IPO, then started a company that was a product reviews platform. And we sold to our competitor. And then the federal government considered us a monopoly since we were one in two in the market. And so we were spun out.

And after that 10 year experience in product reviews, I went out on my own as a product management coach. And so though I started in engineering, I'm here in the product world because this is where we're defining what's going to be the next great product. So I also teach at Berkeley. Right now I'm coaching consumer organizations, a lot of digital health organizations, and then a grouping of tech development tools, financial tech, and then other companies. So the organization, the squad and the individual level. So what do I see out there in the world? I see a lot of building, build, build, build.

And in that process, people are really falling into what I call the product discovery valley of death. And this is where you have great ideas, but you don't test them. And when you don't test them, there's the risk of people not adopting them, not seeing value. You may have great ideas. You may have interviewed folks, but you haven't taken the solutions back to those folks to really see if they would use them. And today, today it's all about these Quick experiments for innovation so that you do this solution testing step and you avoid the valley of death.

Okay. So let's start with a story of an entrepreneur who spent $200,000 building a website that nobody went to. And he had found an outsourced company to build his dream idea. But the issue with the dream idea is that he wasn't able to market it and didn't quite understand what people would see value in it. It was valuable to him, but he hadn't done the research with others.

And so when he came to me, we started from scratch. We built some prototypes to test what would people see in this website. And I'll tell you what this website does. Imagine that you have a home project and you need a plumber or an electrician. While you have to go find somebody, you might be doing it online. A lot of people ask their friends or their family. The premise of this site was that you would ask your neighbors. And so how can you facilitate the asking neighbors? Will you have to find them? What neighborhood are you in? And we tested the interest in this idea and we tested how people could find out about it. And what we learned led us to create a very bare bones, no code website.

So we started with fake prototypes, Quick experiments, moved to the next set of Quick experiments, which was to use a Squarespace site with Google spreadsheets and to drive traffic to that site from Nextdoor and Facebook groups. We'd learned about Nextdoor and Facebook as a large source of these plumber and electrician referrals from our customer interviews. When this site got too overloaded and we started to actually drive traffic, and you can imagine Squarespace plus Google Sheets is much cheaper than a $200,000 site custom. You then went on to a site using Webflow, another sort of more advanced no code.

Then he also decided that this market needed to pivot his idea into a home service subscription service. So for plumber and electrician, whatever you needed, what about a monthly fee to have that taken care of? And so through these Quick experiments, this entrepreneur was able to pivot and then grow his understanding of his market. And this was in a fraction of the time it took to build this other site that no traffic was going to. So again, Quick experiments can help you innovate to get to the right idea. And so this is really what we're looking for. Instead of building all the time, it's learning, building, and then measuring. Of course, did you get that traffic? Did the outcome? Did you make money? So really redefining product development here. Again, people are familiar with this is instead of ideas going to delivery, we're going to add discovery in.

So today we'll talk about discovery as innovating with Quick experiments. The first step, the first word is innovate. You really need your cross-functional team. This is important. What Rory said matters. As you get more folks on the team earlier in the process and you solicit their ideas, you're going to get a much richer set of ideas to start from. That might be in the problem space. That might be in the solution space, but you need this cross-functional team. I have clients that have a data scientist on every cross-functional team. I have clients that have doctors on every cross-functional team. So sometimes you need subject matter experts, sometimes data experts, somebody from customer support maybe.

Again, Rory mentioned that team size of up to eight. I think that's great. You've got to keep it low. There are T-shaped personalities that can help you if you have multiple specialties that need to be represented. So you need engineers. Most common problem I see is that folks create cross-functional teams, but don't drive engineer participation. And so what they're losing is what's just now possible. Really engineers are one of our greatest sources of innovation. They can hear the problem and they can apply perhaps a new solution they've been thinking about or reading about or one that might just come out of what their current workstream is. So the concept of just now possible is really important for teams. You might be missing this if you don't have engineers. So that innovate work's really important. So what are popular engineer-started companies? Well, Google, of course, Facebook, Microsoft, Apple.

So a lot of the most commonly used companies today had either all or more than half engineer founders. Okay. It's really important to make sure that you don't lose touch with your engineers. Keep them involved. Amazon took some just now possible technology, secure websites, widespread internet access to launch in the 90s. My company Power Reviews, we were a product reviews widget inside of the site. We grew to 1,200 clients in 20 countries, all powered by iFrames and Ajax. Tesla really started with this concept of long-lasting battery.

So there are enabling technologies that allow certain companies to get over the hump. So what are we doing with engineers? We're doing collaborative solutioning with those data scientists, with those doctors, creating that team, really working up front together. So what does each individual do? Well, certainly they're contributing ideas. One of the best ways is to do sketching.

So on that innovation side, I want you to innovate by drawing it, not just a designer on what all of you draw. These are examples of sketches. These are things that drive innovation when you have people write down their ideas and then you take these ideas to your users in lightweight forms. The Sprint Bug is a great resource for this type of activity. So I recommend checking this out. You don't need to do these things in five days. You can just take these exercises and apply them every sprint, every discovery cycle.

Okay. So let's move on to Quick. Quick is a lot of fun. We talked about innovation. Well, Quick. What does Quick mean? Well, here's a thought experiment. It's very Quick. In 1921, a Swiss psychologist invented the Rorschach test. This test got very popular in the 1960s as a way to understand aspects of patients' personalities. So you show somebody something and then they give you a reaction. This is sort of the simplest form of a Quick experiment. Well, what's a great Quick usability experiment?

You know, again, we're trying to figure out, are we confusing the user? So I'm going to slightly move you from psychology into the physical world and into technology. Can they accomplish their task? It's usually testing one flow. So what's Quick here in usability? Well, one flow is this dining room table that I was creating, that I was planning in my house. How wide would it be? What was the length of it? How far from the ground? Everything was built with cardboard. I had family members sit around the table to test this out.

So again, this Quick experiment of building it in cardboard rather than in wood, where it was really expensive, led us to understand all the parameters, the usability of this item. And then we were able to build this table. So usability experiments are really valuable. If you can do them Quickly, you can learn before you spend all the time in engineering and building. So what's another form of experiment that's Quick?

Well, I'm going to show you a backlog prioritization experiment. So when we talk about the theory and the practicality, this is a lot of fun here. Let's say you've got a lot of items in your backlog. I can tell you that the various frameworks for prioritization aren't that useful because you're often making up numbers for what's the impact, what's the confidence, what's the effort. If you do some discovery about the ideas, do some user testing, you can learn about the priority with respect to your customer's value. Do they want this thing? That's really what we're interested in.

So let's go talk to users, backlog prioritization. So you've got multiple items. How can we do rapid discovery to find out which one will produce the most value? So we have to put ourselves, this is a thought experiment in itself. I want you to pretend that you work for Zoom and that you have thought of a bunch of ideas for improving breakout rooms. I teach these classes at Berkeley, they have 30 students in them. Each team is three people. So often in my classes, I have 10 breakout rooms for five to 10 minutes and I need to decide I'd love to have more tools to help me manage and improve. So one experiment is let's say I'm a Zoom user and I've got Zoom on my phone and Zoom starts sending me push messages to gauge my interest.

So I call these micro prototypes. So Zoom, let's read this text here. Zoom now offers advanced features to manage the success and health of breakout rooms. I can gauge whether somebody's interested in breakout room improvements right away. Would they swipe to learn? Would they dismiss? Have them talk about it in front of you? So this is a moderated interview. I show this either I'm with them and this is a prototype or that's on a computer screen and over Zoom actually talking about Zoom. I show a different idea. Zoom now offers the ability to listen into a breakout room without having to enter it. A new feature, see who has video turned on or off in breakout rooms without having to enter them. New feature, for breakout rooms Zoom shows you which rooms are the quietest and which rooms have the most people talking. Again, you may not be the product manager for Zoom breakout rooms or Zoom itself.

What I'm encouraging you to do is to think of the ideas in your backlog and Quick ways to explain them to users and to get their interest. Let the users help you with Quick experiments, decide what to work on. Start a prioritization. I did this with a client. They worked with educational technology. We went into a classroom and we showed a variety of ideas and the team was able to hear the excitement and understand the response as to which of these ideas was most valuable. So you too can go in to your target customers with something simple, almost like a marketing prompt, just a gauge value. And you can start to build bigger prototypes on it. Try the microprototype idea. Okay. So again, we were really small with these microprototypes. What do I see in the market? Lots of stuff. People testing too much, creating these end-to-end experiences. So what I want you to do, I want you to not test an entire experience. I want you to pick a piece of that experience to innovate. This is an e-commerce flow. How about just picking add to cart? And really testing something next.

So again, we can be Quicker if we can focus. You can certainly test each of these other aspects, but to be Quick, you want to test them one at a time, not in these mega testing sessions. I also want you to start with a vertical slice. So what is really valuable about that add to cart area? Can you launch something? Can you from end to end deliver a piece of value Quickly? So when I say Quick, you may have a vision of that entire product that is reflected in that left pyramid that's full. What I really want you to do is iterate these vertical slices.

When I say Quick, what does that mean? What timeframe am I talking about? Here's a one quarter timeline where you learn over a four to six week period by testing these four prototypes. You build it and you launch it and then you do some measurement. This is not an example. In science, everyone's going to have to tailor the serents. It's not one size fits all, like Rory said. But what this is is a way to think about what Quick means. Some people in the gaming companies or in Google search might be testing multiple things a day. But here in that learn phase, you're running through Quick experiments to figure out what is working and what's valuable. So let's move on to experiments. You're doing great.

We've got to innovate. We've got Quick. Now let's do experiments. So again, solution first versus experiment first. In the solution first world, you've got that one long flow and you just build it. No testing. Experiment first is this respect for the market that may be able to help you figure out what's actually working well. Multiple flows, evaluating, maybe building. So let's take something where we've got multiple flows. We may make a compare and contrast moment that reveals the user's choice. And then if I were to pull this all together, when I have multiple flows with experiments, I'm actually bringing more of those sketched ideas, more of those ideas from my team to the user. So there's more room for innovation if I'm putting more ideas into the market more often.

So innovation can often be a quantity game. So here we go. I had you focus down to the add to cart. Well, what does experiment mean? Experiment means give me three add to cart experiences. It could be four, it could be five. I've tested up to seven experiences in a half an hour. It depends on what you're testing. What is valuable to the users? Is it a cross-sell? Is it an upsell? Is it a discount coupon? Is it a service offering?

But you're going to add these multiple options in. And what experiments do is they help you avoid confirmation bias. When you offer no choices to users, effectively you're saying, hey, choose this thing. When you offer choices, you're being open-minded. And as you're open-minded, the idea from that data scientist might really come through as the winner or the doctor that you put on the team. Doesn't have to be the product manager. Doesn't have to be the designer. In fact, the engineer might come up with the idea that the user selects. So as we go from big to small, you have this big flow you want to test. This is an analytics experience. Let's just try that dashboard. You still have a lot going on in that dashboard. Let's pick one of these graphics. Is this graphic? Is this visualization driving value? Some of you might be analytics PMs out there. Go to the next level is now I want to take that and I want you to create multiple options. And that's what brings the experiment aspect in.

So to recap this, you start with that one little module there and you do some experimentation. This is where you're going to build an incredible product one piece at a time rather than this big bang testing where you are doing long experiments that really never get off the ground. So let's pull it all together. So let's do innovate, Quick and experiments. I'm going to show you something and then we'll be out of here. This is great. Zoom breakout rooms. This is a link. You can go look at this prototype and play with it yourself. This is just an example of giving three options that are compared contrast that I built myself in Figma. This is how easy it is with Figma. It's a great tool. There are lots of other tools you can prototype with.

So here's a scenario of a bunch of folks. I'm going to put them in the breakout rooms. Then I have to figure out which one to visit. And I'm just giving you a screenshot because we're in a presentation situation, but this would be an option. This would be a flow that someone would play with. Option one shows these items. Option two shows some videos that are on and off. Option three has no videos, but it does offer me the ability to listen in. And listening into Zoom breakout rooms without people knowing it sounds creepy. It's also one of the most highly selected options when I give this prototype to folks. And then at the end, there's a compare and contrast moment. So again, this is how you innovate with Quick experiments is to make these things that aren't pixel perfect but get your point across.

Okay, so closing out here, don't settle for blah feedback. I like this. That's interesting. This is all kind of meh feedback. This is not what you want. People are being nice to you. I want to hear, ooh. I want to hear, oh, wow. I want to hear, is this available? I want that emotional reaction. Users, mindset, content, prototypes, these make great experiments. I really encourage you to get out there and innovate with Quick experiments. I've done a lot of writing on solution test interviews, so really check this out. And again, stay in touch. Thanks for listening. And here's my LinkedIn, my Twitter, and you can reach out to me through Rory. Thanks for having me.

Rory
Great. Thank you very much, Jim. And just as you were going through your kind of experiments with Zoom, I've realized we were getting spams a bit and I tried to block it and I realized it popped up on the screen. So that's a feature for this system that when you try to block a chat message, it ends up publishing it instead. Apologies for that brief interruption.

I really, really like that presentation. Some great examples there. And there was two kind of things that jumped out to me that I wanted to ask about. The first being with those kind of fake door type of or like those push notifications, the feedback and when I've tried to push those in organizations is that will damage our reputation. So if we do that, we have to go out to market with a perfectly polished offering and we can't go out with anything small or micro. And how do you deal with that kind of reaction from people?

Jim
Yeah. I find the world in general in consumer and in business to be more accepting of works in progress, just in general over the seven years I've been on my own plus before that. So I also think that you can run those types of push messages just in a fake prototype interview. You don't necessarily have to actually put them out in the app.

But for companies that really want to get more of a statistical form of evidence rather than the evidence that I was gathering, maybe in prototypes would be to actually send those push messages out. And my sense is that people understand that people are doing market research and that really what you're trying to do is involve them as a partner.

So that may be the extreme version if you were to actually publish those four push messages. But what I encourage you to do is to involve your customers upstream to help you make choices and to fight against that branding conflict because people actually want you to get better as a company. And if you don't listen to folks, which means actually talking to them and sharing ideas, it's going to be really hard.

Rory
That's great. I do know some marketing folk that will take a little bit of convincing, but no, it is a great point that you want that conversation, that communication, two-way communication. And the other one that I think was Kevin asked online, how do you push for better communication? So you said don't settle for meh, but what happens when you're getting meh? What do you do to try and get to that better feedback?

Jim
Yeah, I'm trying to stretch my teams. My teams are putting out compare and contrast moments that are more like usability tweaks, like should the buttons be blue or should the buttons be red? My sense when you're getting blah feedback is that you're not trying hard enough. You're not stretching. You're not encouraging or desiring failure. And so ideally, you'd get to a point where people are like, oh no, I don't want this. You need to probe in the design phase, the design prototypes, probe how far your user's interest and value goes.

And my sense is that you may need to add more people to the team. You may need to add more exercises such as visual brainstorming like the sketching. There are lots of creativity exercises to get teams to put up more ideas. But quantity of ideas, quantity of testing, you're going to get to points. And I see it with business software as well. People say, ooh, people want their business software to work well, not just their consumer software.