Building a Lean Assessment Team for Upfront Validation

Penelope Barr at UXDX APAC. Video: https://youtu.be/qPxKvxgBooQ

Readable transcript: edited from the recording's captions for readability (fillers and false starts removed, punctuation and section headings added). Wording is the speaker's own. Timestamps are positions in the video. Names marked [?] could not be verified against the audio.

Finding a life partner, and finding good ideas

[00:00:00] Hello everyone. As Ryan said, my name is Penelope Barr. I'm just going to share my screen in a second. I'm going to talk to you today about a framework that I've created, which is called Kiss Kick Kill, and the underpinning thesis for that is building a lean assessment team for upfront validation. This approach has been developed over several years. First, I'm going to start with a question. Let me just share my screen.

[00:00:46] Are you able to see? I'm just going to check if you can see that.

[00:00:52] Ryan: Yeah, I can see it. It's not in present mode at the moment. It's okay.

[00:00:59] Penelope: Okay, you can see that now, present? Yeah, beautiful. So first I'm going to start with the question: how many of you have used some form of decision-making process to source and find your ideal life partner? And if so, what was the outcome? I can't see any hands, but I'm going to assume that some of you have.

[00:01:20] I'm going to take you through a story of my journey of doing just that. About 19 years ago I created a 12-point criteria and husband-shopped my way across a few countries where I was working, and it was happily ever after. Sorry, I'll just take you to the end of that. Along the way I used my kill [?] framework for early idea assessment. And yes, there are similarities between managing the search for an ideal life partner and managing idea assessment in a portfolio, in our quest for great products or innovation.

[00:02:03] Over many years leading enterprise innovation and product development, I've become keenly attuned to the love language of product, and I've observed teams following really the same patterns. Lots of teams not looking at enough ideas: you need lots and lots of ideas to get to the gold. Doing too much discovery work: working on early ideas when quicker decisions were needed, before any time, energy, people or dollars were expended. Discovery is work. Many teams not stopping ideas because they didn't feel safe to do so. It takes courage to stop ideas. And taking too long to assess if any idea is potentially a good one, one worth pursuing, and they did that often because they didn't have a defined criteria for idea assessment.

[00:02:56] These observations prompted me to create my framework, which I've now used and modified in eight organizations over 13 years. In thinking about how to select the best ideas to shape into products customers will love, and more importantly buy, I've naturally turned my mind to relationships. Well, don't you?

[00:03:22] I don't have to tell you, but one generally has to kiss a lot of frogs to find a sustaining prince, princess or non-binary life partner. In fact, a study of 2,000 adults who admit [to having found] the one found that women go on 11 dates with different people, and men 14. Women have five relationships, two lasting more than a year, and men will have six. So there's a deceptive amount of hard work that goes into this process. Dating is expensive too. Australians spend an average of $170 per first date, and $500 on three dates per month, for a total of 70 billion dollars per annum. This is a huge amount of money looking for love, so it makes sense to try to optimize our decision making around this, and any process.

[00:04:15] You may know Alberto Savoia and his law of market failure, which states that 80% of new products fail, and you may have experienced this yourselves. This percentage means that product development is expensive too, and optimizing for good decision making will help improve this process.

[00:04:34] When I was looking for my husband-to-be, I set my strategic objective. I was meeting lots of people, which was great fun, and for a while this was great; I was having really a wonderful time. But it was also really costly, and it wasn't making me feel great, and it was really unsuccessful when measured against my strategic goal. Which quite often happens when you're creating products as well: when you're thinking about whether this product you're creating is going to achieve your strategic goals, oftentimes it's not going to.

[00:05:11] When we run a product or innovation portfolio, we usually think about ideation as a funnel, because we need lots of ideas to come through the funnel to get to the gold, the good things that will enable successful products to come out at the end. And we know this is really difficult, as is finding a husband or a life partner. Sure, sometimes we can get lucky, and we all know those people who have met the one first off, or lucked out on a market-shaping idea that was just meant to be. However, history, the stock market, product stats, love in all forms and our own experience tell us these examples are rare.

Theories of how teams go wrong

[00:05:55] Let me posit a few theories from what I've observed of innovation and product teams working over time, and we'll work through these. Most of these really rely on key leadership, and empathetic leadership, over time. The first is the Field of Dreams theory. This is "build it and they will come." We've probably all been in the situations where we get really excited about an idea and we just want to start building, because it's really exciting and it's easier to just start. There's lots and lots of theory around just start doing something.

[00:06:32] This could be okay if we've got all of the lean methodology behind us, but quite often we just start doing a lot of heavy work. Don't we all just want to get to a market-leading product? Aren't we sure of our market, and don't we know our space? And effectively, don't we all really want to get to that happily ever after as quickly as possible? But skipping the time and the validation steps in between could be problematic. Meeting and settling into that happily-ever-after routine with someone may expedite our path to coupledom, but we might miss warning signs, or opportunities, along the way.

[00:07:17] The "when it's right, it's right" theory. It's hard doing the work to make sure that what you're doing is right and is a good bet. Getting to know an idea or a person takes time, but it's hopefully worth the investment. We all have our own processes for doing so. I employ the kill [?] framework, which is basically saying yes, no, maybe to strategically aligned objectives, as a way to help support effective decision making on the opportunities worth pursuing. And I do this in a structured, time-effective way, to place the right bets on what we might pursue.

[00:07:59] We all know this one: "whoops, we've spent too much again." Making the wrong bets can result in additional spend and wasted time, and can also really impact our confidence or momentum. Choosing any idea comes with inbuilt risk. Ninety percent of mobile apps don't make any money, four out of five startups lose money for investors, and sixty percent of new restaurants fail within a year. So starting anything new is expensive and scary. Starting a new relationship is the same.

[00:08:32] Sunk cost fallacy. Again, anyone who's done any initiative or any project is familiar with this: "Well, we may as well keep this going, we've come this far." We know it doesn't work with the wrong partner, and it certainly doesn't work with the wrong idea. A standard part of the product development process is seeking buying signals and feedback that we're aligned with the strategic direction, that we're creating something that people want to buy. So we need to go out and be seeking those signals. If we've spent a lot of time and money, it can be really tempting to just keep barreling along on what we're doing without seeking feedback, and ignoring any feedback that doesn't fit our narrative. No one wants to feel like we've wasted time on an initiative or a person.

[00:09:21] It's a really good thing to kill an initiative. It's cheaper, it's less emotionally damaging in the short term and the long term, and it saves time. "No, no, no eventually gets to yes" and "failure is the path to progress" are two good phrases to keep in mind when you find yourself in a sunk cost phase.

[00:09:48] The "I don't know what came over me" theory. You may have encountered this if you've got a particularly strong CEO, or if you're in a space where people get very excited about new trends, people who are very passionate about what's the latest thing in the market. Or if you've been on a first date and you've come home and you've already picked your twin girls' names, Daisy and Maisie, because you think that everything's going to work out. You need a little perspective to evaluate people and ideas effectively. NFTs, crypto, anyone? There are a lot of things to consider when you're thinking about ideas and which ones are going to be best, and there are a lot of things that can go awry.

The early assessment approach

[00:10:42] On the screen here there's an outline of an early assessment approach, which leads into discovery. If we just focus on the early assessment: first, ideas can come from lots of different places. You might source them externally or internally, you might generate them within your own team, or you might do a range of market research, internal research or generative research by talking to internal and external customers.

[00:11:14] The approach that I've outlined here is basically the Kiss Kick Kill approach. I'm going to go through each of these steps, but basically, once you've created some ideas, every idea gets incorporated into a standardized idea assessment form, so they can be evaluated effectively in the same way. Then there's an initial kill decision, which basically is: are we going to keep it, are we going to get more information, or are we going to say no to it? Then we have a LAT assessment. LAT is the lean assessment team, which I'll talk you through in a minute, and that team together makes another decision. Then together the team will prioritize, if the idea is going to continue, where it sits, and then we'll move into prototyping and then the discovery work.

[00:12:13] The key thing is that the early assessment work is very, very light touch. We don't want to be spending too much time or too much money, because we want to be churning through ideas, so that we're getting through lots and lots of things while spending as little time and as little money as we can, to give ourselves the best opportunity to be successful.

[00:12:37] We also want to be making good, quick decisions. We know that the length of time it takes for a company to make decisions is one of the key factors that can be correlated with an initiative's success or failure. If you're breaking it down in terms of time and decisions, if teams and leaders take on average an hour to make any decision, decision success rates increase by 58%, whereas delaying any decisions, such as which ideas we'll be pursuing, by more than five hours decreases success rates to eight percent. So to optimize decision making, that's why I've created a really structured approach that now gets used in organizations I've worked in, and that I've left this [?] through.

[00:13:33] What we're looking at here is that at each stage of any kind of idea assessment, we're moving through and we're deciding which ideas are going to continue on, and which ideas we're not going to pursue any more. This final assessment is taken again from Alberto Savoia's work. If you don't know him, I'd really recommend that you get onto YouTube especially and follow along with his really easy product maths lessons. Basically, he works from saying that there's a number of ideas we start with, and at least 70% of those will result in nothing. So then, working through, we've got so little chance of success with really strong product ideas that we need to apply a light touch with all of the others, so that we can get to the best ideas as quickly and as effectively as possible. And those ideas, even though they're only in the up to 10% range, can still make us a lot of money. But we need to know where the best bets are, because there's a lot against us, as we know, in terms of working through to the best ideas.

The idea assessment form and the first decision

[00:15:05] As I mentioned, each person who has an idea, and that might be from your own teams, or from outside of your organization or your community, basically uses the same form. If anyone's really interested, I can send you the fields for the forms that I advocate. Basically, this categorizes key thinking, the background to the idea, who the sponsors are, what the value of the idea is, how it aligns to the strategy, et cetera. We want to capture as much information in this form as we can, to be able to assess the idea.

[00:15:48] This form is then assessed initially by a product lead, an innovation lead, someone who has experience in assessing ideas. So they've got enough experience to say "this looks like the kernel of a good idea" or not. They've got enough experience to be able to take the idea forward or not, and to be able to help the idea generator champion the idea in a cross-functional team meeting. The person who's assessing it makes the first decision about whether or not this idea should progress.

[00:16:29] If they give it a kiss, if they say yes, it's going to proceed, that means it's just going to proceed to the next step. It doesn't mean this idea is a great one for the rest of time; it just means they're giving it the seal of approval to proceed. If they're going to kick it back, it means there's something there, but some fields need a bit more intensity, they need a bit more information, or it's not quite clear enough in terms of how this is actually going to deliver the value.

[00:16:58] And if it's killed, this is the really important step. From the beginning, we want courageous decisions and quick decisions to be made. If it's killed, it's killed, and we want to be able to keep a list. In all the teams I've worked in, there's been a list that's been kept that says, okay, these are the ideas that were presented. We don't want things coming back to life under new guises, new names, et cetera, later, because if we take a good decision around why something shouldn't be there, we don't want it to come back. Again, we've all worked with portfolios where ideas stay on portfolios for years and there's no movement, and they just drag everybody down, because of their weight and the time that holds you back, and they're blockers. We want portfolios that move and build momentum. So that's the first Kiss Kick Kill approach.

The lean assessment team

[00:17:56] That brings us to the LAT, the lean assessment team group. This is a really key meeting that I created and then I've used in several teams. Think of this cross-functional team as all of the people that you would ordinarily individually go to to get buy-in for your idea. I've listed some of the names of the groups of people on the slide there, but in your organization there would potentially be different people.

[00:18:33] What you're wanting to do is bring everybody together at the one time, so that you're having a cross-functional meeting, so that everybody is having the chance to hear the idea from the idea initiator, and is having the chance to input into that idea at the same time. They're teasing out any blockers, teasing out any areas where they see there could be risk, teasing out any budget concerns, et cetera. But they're not solving any of the real issues at the moment. All you want them to say is, again: is this a good idea, or something that they're not going to be able to support? So again, kiss, kick, kill. That's where that group comes together.

[00:19:21] This is the overall approach. The initiative is brought in, as we've talked about. The LAT comes together. There's a weighted decision criteria, which I'll go through in a second. Once the idea initiator has presented the idea, everybody has the opportunity to individually assess the idea, and then collectively the group goes through that, and then you arrive at a decision. If the LAT says yes, if it's a kiss, then it goes into the portfolio and the team prioritizes that. If it's a kick, it goes back, we get more information, and it's re-presented. And again, if it's a kill, we don't want to know about it. Then the idea follows your standard discovery and delivery processes, and the LAT will just be informed at key milestones.

[00:20:21] In terms of how the LAT runs, the overall agenda is similar to this. Each of the teams that I've run has something similar. But the key thing for success is that nobody wants more meetings, or long meetings, so it's a quick thing. The idea is sent out before the meeting. What I've advocated for teams to do in the past is that they set up a weekly or a monthly standing meeting, depending on how many ideas come in, and then confirm the meeting a couple of days ahead. The idea assessment form is sent out so that people are expected to absorb that before the meeting, and so the idea initiator is just hitting the highlights in the meeting.

[00:21:12] Then there's a quick discussion in terms of any questions, any blockers, et cetera. People in the meeting can then confirm their decision criteria, if they haven't already done that before the meeting. Then the LAT makes the decision, then the prioritization happens, and then the decision happens. In most of the teams that I've worked in, the teams have then set up an automated process for this.

Decision criteria, killing and prioritizing

[00:21:35] The decision criteria, again, can be quite detailed, but it has to be related to your strategic objectives. They're largely around the opportunity that the idea presents; client impact, which might be internal or external impact; the strategic value the idea is going to deliver; how able the team and the organization are to deliver this; what the timing is, so if it's an urgent idea, or if it's not, how it's going to be developed over time; what the cost is going to be; and then what it needs to make it successful. And then there are more detailed requirements. Each of these decision criteria is created either in a manual form or an automated form, and these are the criteria upon which the decision is made.

[00:22:34] As I've said, I've outlined what Kiss Kick Kill is, but it's really important, especially the kill. Again, we've all been in places where ideas hang around for a long time. They really impede momentum, they bring teams down, and for stakeholders who don't see progress, they can actually really hurt the internal and external brand for your team.

[00:23:03] Once you've got a decision about an idea that you're going to kiss, then as a team you work through where it's going to sit from a prioritization perspective, and you can use a range of prioritization approaches. Quick decision making, again, is the name of the game, or I've often advocated using RICE prioritization. It depends on where your team is, because you've got a cross-functional team. Sometimes you have to start with a very simple prioritization approach and then work into more complex ones. So that's the Kiss Kick Kill approach that I use, which is structured, time-bound, and gets you to the right decision in terms of giving the best ideas the best chance to be successful.

[00:23:56] To end my story on a happy note, my husband and I have now been married for 16 years. We have a beautiful daughter who actually just this morning broke her arm in New Zealand snowboarding. And I'm much clearer about creating good decisions around ideas for portfolios, because every organization I go into has pretty similar challenges, and I'm able to modify this process each time. I think one of the key things is having those really courageous conversations within your team, and then outside, about killing ideas right from the beginning, and exercising that muscle to say: we know what we're trying to do. If something doesn't align with our strategic objectives, there's no point in us doing it, because we really want to start with something that's aligned to our success criteria and give ourselves the best chance at success. So thank you.

Q&A

[00:25:04] Ryan: Thank you, Penelope, that was a really interesting approach. I'm not sure, if I said to my wife I used a checklist for trying to find her, I don't think we'd still be married.

[00:25:20] Penelope: I didn't tell him straight up.

[00:25:24] Ryan: Excellent. I have a couple of questions coming out of that that it'd be great to know. One of them is the idea. That lean assessment team, they're looking at lots of ideas, so they'll need to get a lot of the context. Is there a format, like the way Amazon publishes their six-pagers, their approach for pitching ideas? Is there a particular format that you use for communicating the idea to the team?

[00:25:50] Penelope: Yes, that idea assessment form pretty much has that format on it. There's an induction approach for each of those LAT team members, and at the beginning of that as well there's a social contract that the team undertakes, and if there are new members coming to the team, they undertake that as well. With each idea, the same form is used, so the members get used to looking for the value, looking for what the blockers might be, looking for what it is that the idea is going to tell them. But also, because they're coming from their own area of expertise, each of them is interested in their particular area.

[00:26:47] The key thing with that team is that it saves you, as the person who's suggested the idea from a product perspective, so much time, instead of going to every single person. It doesn't mean it's one and done; you're obviously going to have to talk to them again if the idea is successful. But it just cuts down the time. In one organization I was in, it was taking two months to get around to all of the stakeholders for every single idea, and it moved it to a half-hour meeting.

[00:27:24] Ryan: Excellent, nice. The other one, I guess, because you mentioned that you keep a kill board, or the list of everything that's been killed. One of the things that jumped to my mind was, sometimes an idea isn't quite right, but particularly if you try it and then you get some feedback, you're able to then evolve it to the right idea. So how do you handle that, where there's probably something good in the idea but it's not quite right yet, so that you don't discourage people from continuing down and checking that route, because it could be the right thing to do? How do you manage that, with the kill board that stops ideas coming back, when maybe an evolution could be good?

[00:28:16] Penelope: The key thing with the kill board is to be really clear about why an idea was killed. Absolutely. One of the key things with all of this is that it does take a lot of empathetic leadership to manage this kind of process, because in order to be managing any kind of portfolio, you need more ideas coming in, so you don't want to stop people bringing the ideas in. But equally, you have to create the framework for what makes a good idea. So in saying no to something, there has to be an explicit understanding of why that idea is not being pursued, and that's why the decision criteria, and also the structure of that idea assessment form, are so important.

[00:29:06] What we don't want is that same idea coming back with a different name. That particular idea is killed. If somebody, though, has done some other work which substantially changes the idea, then that's okay. What I've discouraged over time is people tinkering around with their pet ideas, those passions. When I talked before about that obsession theory, that's what we've really got to guard against, because there are some people who are really focused on doing something they really believe in, but it's actually not aligned to what the strategic objectives of the organization are. So you've got to be saying, okay, if this is something that just hasn't got enough yet, let's take a bit more time to pursue that, and then that would be in the kick category. But if it's in the kill, it has to be killed, and then if you want to revisit it, it comes as a new idea.