How to come up with the next 'Needle moving’ feature for any product

Shambhavi Pandey at UXDX Community: Empowering the Teams and Product Impact. Video: https://youtu.be/3GasaZF_gTo

Readable transcript: edited from the recording's captions for readability (fillers and false starts removed, punctuation and section headings added). Wording is the speaker's own. Timestamps are positions in the video. Names marked [?] could not be verified against the audio.

What needle-moving features are

[00:00:00] Shambhavi: Hi everyone. Hi Rory. Thanks for the introduction, and thanks everyone for joining. I'm very excited today. This is Shambhavi, and I'll be talking about how to come up with the next needle-moving feature for your product area. Specifically, I'll be starting with what these needle-moving features are and also why you should care about them, and then I will be talking about a five-step framework that you can follow to nail down the next needle-moving feature for your area.

[00:00:36] Before we get into it, just a little bit about me. I am an engineer turned product manager turned entrepreneur. I have spent about two years in engineering and 10 years in product management, in companies like Amazon, Groupon and Home Depot. I have worked a lot in e-commerce, and within that personalization and marketing. Later I also got into the B2B space, so I did some enterprise product management for Amazon Business customers. And currently I'm running my own startup called Mentor Me Good, where I help people land their dream roles in big tech and other tech companies. So yeah, let's get into it.

[00:01:28] What are needle-moving features? Why don't we look at some features, and maybe you can write your responses in the chat box, and we'll see if some of them are needle moving or not. Starting with the first one, this is Microsoft Bing, and here the use case is that the user wants to plan a Disneyland trip with a five-year-old. On the left-hand side you're looking at an old experience, where someone would go to Microsoft Bing and just search for a Disneyland trip with a five-year-old, hoping to get a ton of blogs and personal experiences from people to see how they should go about it.

[00:02:15] But on the right-hand side, you'll see the new chat feature from Microsoft Bing, where instead of searching, the user asks, "Help me plan a Disneyland trip with a five-year-old," and a bunch more questions. And in the result you'll see that they get a structured response to each of their questions. So do you think this is a needle-moving feature? I think it is, because it fundamentally changed how the user goes about their use case of planning a Disneyland trip.

[00:02:52] Let's look at one more. On the left-hand side you have the good old pressure cooker, a slow cooker, but on the right-hand side you have Instant Pot. For those of you who don't know, Instant Pot came in as a product in the Amazon marketplace without any big launch or standalone launch, but it still was a hit, because it fundamentally changed how people cook. And they built a community around it to share recipes. This was certainly a needle-moving feature.

[00:03:31] Let's look at one more. Here you have two landing pages. On the left-hand side you have one image. On the right-hand side the image is changed, but the rest of the things are the same. So is this a needle-moving feature? Probably not. You may get a lot of incremental benefit from this. If this website has a lot of users, then even a small incremental impact on an individual basis can add up to a big impact for the product. But this is not really a needle-moving feature.

[00:04:12] So having looked at all this, what are the attributes of needle-moving features? The first is that it improves the outcome of users significantly on a per-user basis. As a user, your behavior can change, how you interact with the product can change, and it's not just a small change on the website. The second attribute is that these product features are often first in their market. They might be a distinguishing feature, something that people have not seen before. And the third one is that these features have a major impact on the product's overall trajectory, its growth, and they can also impact the other features of that product.

Why you should care

[00:05:07] Having looked at what needle-moving features are, you must be thinking, why should you care? If, like me, you are into product management, you know that you have to juggle a ton of things as a product manager. You might not have to do everything that's on the slide, but you definitely do a lot of stakeholder management, coming up with road maps, looking at data and so many things. But if I were to ask you what the main deliverable that you have is, what would you say? Is it building road maps, or is it planning meetings, or is it doing the user design, working with the UX team? What is it really?

[00:05:59] I think as a product manager, delivering business impact is the number one priority. You may ship a lot of features. But someone who has shipped 10 features versus someone who has shipped one feature: do you think you can tell who is the better product manager? Probably not, because it's always about delivering the business impact. And when you deliver a needle-moving feature, you deliver on the business impact.

Steps one to three: output metrics, input metrics and a goal metric

[00:06:34] Now that we know why delivering needle-moving features is important, let me share with you a five-step framework that I think will help you have a structure to your thought process and come up with a needle-moving feature for your product area. Step one of this five-step process is to understand your product's output metrics and the actions that drive them.

[00:07:01] I will be using an example throughout this framework, just so that you can resonate with it better. Let's take the example of LinkedIn, and let's look at how you think about the output metrics. But first, what are output metrics? These are the metrics that you don't control directly, but they are the outcomes that you want for a product. There could be many output metrics. It could be revenue. It could also be monthly active users, because when you come to think about it, you can't really do something to increase monthly active users directly. You launch features, you influence the input drivers, and those in turn impact the output metrics.

[00:07:56] In this case, for LinkedIn, I have chosen subscription revenue and ad revenue as two output metrics, just for this example. And within ad revenue, one of the things, again an output, is the time spent on the platform. Now, what are the actions that you want the user to take to drive these output metrics?

[00:08:20] If you look at subscription revenue, you want the users, specifically job seekers, to find jobs, and the recruiters or hiring managers to find leads. If they do that, people will subscribe to LinkedIn Premium, which in turn will drive revenue. The other option is to have courses, so that job seekers can upskill, and that would again contribute to subscription revenue. If you look at ad revenue and the time spent on the platform, if you enable creators, it will definitely drive time spent on the platform and ad revenue. And the last one here is enabling conversations between users, which is LinkedIn messaging. If you add features that help users communicate with each other better and get the results that they want, the ad revenue will increase.

[00:09:28] So that's the first step, where you have to understand what the output metrics of your product are and what actions you want to drive to achieve those output metrics. Step two is to connect those actions with the input metrics for your product area. Input metrics are the metrics that you can directly influence. Whenever you launch a feature or define a feature, you have an input metric in your success criteria.

[00:10:00] These are the actions from the previous slide. Let's say that your product area is LinkedIn messaging, and the action that you want to drive is healthy conversations between users. Now think of the input metrics that are relevant to this. For example, one could be number of sends. If people send more messages, then that's good for your product area. The second could be reply rate. If a user is sending messages to other people, what percentage of those messages get a response back? That's reply rate. And then there is conversation depth, which measures how many back-and-forth messages are exchanged on average. And if this number is higher, it means that users are having meaningful conversations among themselves.

[00:11:00] For the sake of example here, let's say you have data and you see that you have a reasonable number of sends, but your reply rate is really low. And that's where, with the same action, you have selected reply rate as your goal metric. So this is step number three, where you select a goal metric among the input metrics that you have. And then you have to build a hypothesis, or multiple hypotheses, that would drive the reply rate. So we moved from the output metrics to relevant actions, to the input metrics, and we chose a goal metric.

[00:11:39] Now, with reply rate as a goal metric, you have to think about what the hypotheses are, or what things you can do to drive it. You start by asking yourself how to increase the reply rate of LinkedIn messages. You dive further into data and find out why the reply rate is low. There could be multiple reasons. Maybe users are not seeing the messages. Or maybe they see them but they don't reply, because there is a lot of friction in replying. And it could be a combination of both. By using data and user feedback, you'll understand which of them to prioritize, or you may come up with hypotheses that would help solve both issues.

[00:12:30] In this case, for example, one of the hypotheses you can have is that if we send email reminders to users for unseen messages, then it will lead to higher reply rates because of increased awareness, because probably people are just missing these notifications. That sounds reasonable. There could be another hypothesis: if we give InMail credits to users who reply to messages, then it will lead to higher reply rates because of incentivization. So introducing a gamification element here. And one more hypothesis we can throw in is that if we autosuggest replies in messaging, then it would lead to higher reply rates because of ease of use.

Steps four and five: prioritize and launch

[00:13:18] Once you have the hypotheses laid down, you go to the next step, step number four, which is to prioritize the hypotheses to validate. Usually, as product managers, when we are prioritizing, simply put, we use two criteria generally. One is the estimated benefit, and the second is the effort that it'll take to do it. In this case I recommend you also use a third criterion when you look at these hypotheses, to see whether these hypotheses or features have the potential of being a needle-moving feature or not.

[00:14:03] For example, if you look at our hypothesis from the previous slide, if we send email reminders to users for unseen messages, it may help us. But we may have data from our previous emails, like click rate and open rate, and with all that data we can see that maybe the estimated benefit is not that high. We can also work with our engineering team to figure out what the effort is. Similarly, we can do it for all these hypotheses.

[00:14:37] But when you are trying to evaluate whether the feature has the potential of being a needle-moving feature, you go back to the attributes of a needle-moving feature. Remember, in the beginning we saw that it should have a high individual impact. There should be a shift in how the user is interacting with the product. Hypothesis number one doesn't satisfy that criterion. Similarly, you should also look at whether this is a feature that we don't offer, or that no one in our market is offering. And you should also look at whether it has the potential to change how people in general interact with LinkedIn, LinkedIn messaging and other LinkedIn products as a whole. So with these three criteria in mind, you should prioritize the hypotheses.

[00:15:33] Which brings me to the last step, which is to launch the needle-moving feature. Usually the first two steps are the same as how you would go about any other feature. The first one is to define the feature. You convert your hypothesis into a well-defined feature. Include the end-to-end customer journey, the mocks, the assumptions and everything. The second step is to have a launch plan, which we usually do when we launch something. When you're launching something as an experiment, include the experiment details. What would be your success criteria? What is the dial-up schedule and rollback plan?

[00:16:21] The third point here is the difference: you need to track the attributes of needle-moving features. When you launch this as an experiment, you should measure the impact per user, because that's the key to a needle-moving feature. You should also see how it impacts the other product areas. For example, in LinkedIn messaging, if you have a high reply rate with this feature, are people using more of the other LinkedIn features because of it?

[00:16:53] That takes me to the end of the five-step framework. This is a quick recap. The first step is to understand your product's output metrics and the actions that drive them. From the actions, you come to the input metrics, so connect the actions with the relevant input metrics. The third step is to select the goal metric out of all the input metrics and then come up with multiple hypotheses. You will also need to drill into data for this. The next step is to prioritize the hypotheses. Make sure that you are looking for those needle-moving attributes when you prioritize, along with the estimated benefit and the effort. And the last step is to launch the feature and measure the right metrics to make sure that it indeed turned out to be a needle-moving feature. With that, I will come to questions.

Q&A

[00:18:04] Rory: Excellent. Thank you very much, Shambhavi. That was a really nice, structured way of going from zero to good features. I have a few questions, but if anybody has any questions out there, please do post them on whichever platform you're watching on. It could be on UXDX or YouTube, LinkedIn, wherever, and we'll collate those. One thing that I was thinking of was the inputs to your output metrics. How did you track those, or what kind of tools or structures do you use? Because I can imagine this could almost be fairly static in this first step, that it's quite common which things impact your core metrics.

[00:18:52] Shambhavi: So your question is about how to go from output metrics to input metrics?

[00:18:57] Rory: Yeah, sorry. And what tools do you use to track that?

[00:19:03] Shambhavi: Firstly, to come up with the output metrics, you should know your business model really well. And then from output to input, the best way to go is to track those actions, because it is hard to come, for example, from ad revenue to enabling conversations. But if you look at the actions that you need to increase the time spent on the LinkedIn website, and you think through it, you will come up with a lot of actions that we want the user to take.

[00:19:44] Taking another example, there is a popular use case of onboarding users. A lot of sites onboard users, and they want to make sure onboarding is a really frictionless experience, but they still want to get the relevant details that they want. So if you want to increase the signups, that would be the input metric, and you want to grow your business, that would be the output metric. The action that you want to take is to make sure your signup flow is really easy and visually very intuitive. When you think about the action, then you can relate it: okay, if it is frictionless, then I would increase my signups. So it's really going from output to the action and then to the input metric.

[00:20:42] Rory: And do you track these in Excel, or an opportunity solution tree, like a Miro board or something like that? How do you track these?

[00:20:52] Shambhavi: I think this kind of activity is a good collaborative effort, so any kind of whiteboarding tool would be great for it. If you are in person, then an actual whiteboard would be good as well. But other than that, I don't think there is a specific tool that you have to go for. It's more of an exercise, a collaboration, where you get ideas from different stakeholders, taking into consideration whatever you know about the product, because you will have some data about the users and the feedback.

[00:21:32] Rory: Great. And a question just came in from Gary on taking it to the hypothesis stage. Who do you get involved in that? Which team members, and how do you pull people in to try and get you from "this is our metric" to "these are our ideas"?

[00:21:51] Shambhavi: I think the number one is your customer. If you are a consumer company, you would have a lot of data on where the customer is spending their time. You may also have a website feedback mechanism where they'll send you feedback. That will help you identify that this is a real problem. And sometimes the customers give you their version of the solution: "Oh, I wish this was there." And if you see that bubbling up from a lot of users, then that will be a good hypothesis to have.

[00:22:30] If you are in the B2B space, you are talking with customers, sales and customer success managers. I think those people are of really great help when you are building your hypotheses. And I've often also seen, and it depends on how the team is structured, but if there is a UX team that is a central team, they would know a lot of issues that you, as the PM of a siloed product, don't know. So it'll be good to involve them as well.

[00:23:09] Rory: And then there's the age-old prioritization. I like the way you added an extra step into the prioritization: I guess the volume, or the needle-moving nature, of the item. I've seen the benefit and cost factors be a bit challenging, because historically we're very poor at estimating how much of a benefit something has and how much of a cost. So what do you do to try to improve your tactics for figuring out roughly how much we think this can improve it and roughly how much it will cost?

[00:23:51] Shambhavi: Definitely there's no one answer for all types of features. But going back to that example where you want to estimate the impact of sending emails: you are sending other types of emails. Maybe you are sending transactional emails or some other emails, so you know what the engagement on those is. And you use that number and try to estimate how successful this will be. It really depends on the feature, but you should always have some estimation instead of just being blank. So estimated benefit, I feel, is definitely an important prioritization criterion. It just cannot go away. Benefit and cost are important.

[00:24:43] And the reason I added a third one is because as a product manager, when you are building your road map, you want the road map to be balanced. It should not be that you are only doing KTLO features, or you are only doing small incremental features or compliance features. You should have think-big or needle-moving features in there as well, so that you have both short-term and long-term value getting delivered. Thinking about this third dimension will help you make sure that the road map is balanced.

[00:25:20] Rory: Excellent. It makes sense, and I think it's a very good point. And then the final question that I have, but please do keep sending in your questions, is around the execution step. Do you put any kind of iteration or validation early on in there? Or how do you spec out how you should go about delivering this?

[00:25:44] Shambhavi: If we talk about this example where you're trying to introduce, let's say, auto-replies, this would require a significant investment in a machine learning solution, where you would use your data and it would automatically predict different responses that the user can choose from. So you don't make that huge investment right away. There are ways that you can iterate on it by building something small. For example, in this case you can have a rule-based system, where as LinkedIn you know what the popular question types are that people are getting. Job seekers, a lot of the time, want to connect with people for a one-on-one chat to explore opportunities, or they want someone to review their resume.

[00:26:40] So there are these popular types of things, and a rule-based system is a quick and less costly way to implement this solution. From that you can easily understand if people are really responding positively to it, if it is indeed increasing the reply rate. And if that is happening, then you can invest in a full-fledged machine learning solution. This was relevant for this example, but similarly for any other thing, if you have a way to test something on a smaller control group with a cheaper solution, you should go for that first. And if you get evidence that this would work, then you can invest in a full solution.

[00:27:25] Rory: Excellent. That's a great approach to it. I think that brings us to time, but thank you very much, Shambhavi, for sharing that process, and thank you for joining us tonight.

[00:27:37] Shambhavi: Thank you, Rory. I do cover the execution piece in a video, so if you want, I can share the link to that with you so that we can share it with the audience.

[00:27:48] Rory: That'd be great. Thank you.