The power of customer feedback for a successful product roadmap

Itamar Rogel at UXDX Community: Rolling Research and Customer Feedback. Video: https://youtu.be/4XgbdzAErjg

Readable transcript: edited from the recording's captions for readability (fillers and false starts removed, punctuation and section headings added). Wording is the speaker's own. Timestamps are positions in the video. Names marked [?] could not be verified against the audio.

Why look at user feedback directly

[00:00:00] Thank you so much for having me. I'm from Great People; our product, our platform, is known as Africa [?], and we deal with user feedback, of course. I'm not going to talk about that. I'm going to talk a bit more generally about how, as product managers, we can use user and customer feedback, and how we can integrate it into our product decision making in a really viable way. I think it's an interesting topic, because we want to get a few things done, and we don't have a long time for this talk. So my goal for this short talk is really just to plant the idea of how you can use user feedback, how it could be meaningful and can move the needle, and how you can actually work with it, because there are definitely challenges involved.

[00:00:49] We're going to look at a few hurdles and a few issues that are maybe stopping you from doing that today, and maybe some ways that you could overcome them. And really the goal is to be actionable: to get some ideas rolling, so anyone on any budget can really start incorporating that into their process in a way that works for them. Hopefully we manage to get all that done in the short time that we have. I thank everyone for joining in. If you have questions, of course, feel free to post them. I would love to address any questions; I think it would be very helpful to have a discussion. Let's get started.

[00:01:29] I think a good place to start thinking about this is, first of all, I'm sure every product manager on this webinar has tools that they use already. Probably product analytics; I'm guessing most everyone uses that. There's a lot of other stuff that may be in place. Maybe you have a customer advisory board, which is really great to have. Some people do focus groups; maybe they're becoming a bit more rare, but they're still definitely done in many, many good companies. If you have a brand monitoring tool, then it's actually pretty unique. I don't think a lot of product managers use that, but some companies do use it to some extent, maybe directly, or maybe indirectly, where they have other stakeholders that provide them with reports, for example.

[00:02:17] And I will mention personas. I don't want to talk too much about that, mainly because I think personas are a model, and there are a lot of weaknesses. But we have all these things that give us some kind of view, some kind of reflection of user behavior, of course, but also maybe user sentiment, or how people perceive our product, or things like that. So the question is really, I think, why would you want to look at user feedback directly?

[00:02:46] If we really look at all these things that we have, they're really great tools and they are important, but they're not reflecting to us the feedback of our own users who actually use the product directly. Maybe they're reflecting a model. Personas are a model. Maybe it's based off of actual users or actual people, but it's still a model. The map is not the territory; it's not reality. Focus groups can give us a great sample; they can give us very useful in-depth feedback. But still, they're not necessarily our users who are actually experiencing the product day to day, and it's not in real time. It's something you can do only very occasionally, so it's not a replacement for direct user feedback.

[00:03:26] A customer advisory board is great, but again, that's probably your most engaged customers, very thoughtful customers. And lastly, product analytics gives us the behavior; it gives us the what. Super important. I don't think any product manager would want to, or should, go without it. But it reflects the behavior, not necessarily the reasoning or the thinking behind it. So that's the idea: even when we have all these tools, and of course there are other things that I didn't mention here, there is a lot of value in getting direct user feedback.

Three reasons not to use customer feedback

[00:03:57] Now, before we talk about how to do that, I want to talk about why not to use customer or user feedback. There are actually great reasons. I don't know who we have on this call and what their practices are, obviously, and I would love to hear that. But I talk to a lot of product managers in various organizations who are top-notch professionals at really great world-class companies, and they don't really use customer feedback directly. Obviously they have very good reasons, and not only because it's tricky.

[00:04:28] The first is that sometimes it's just a vocal minority. I'm sure anyone here has at least heard someone else say it, if they did not think it themselves: "This thing that we hear is just a bunch of people who are super vocal. They're very online or whatever. They're not really reflecting user behavior, because look, I'm looking at the analytics, and it doesn't reflect what all these complaints are about." So that's one major thing.

[00:04:57] Secondly, there's the idea that user feedback can't really teach you something new, because people complain about the same thing. I'm just giving examples, but let's say you have an ad-monetized app, or a free-to-play game, or anything that involves ads or payments. People will always complain about them. So a lot of people have the view, "I read reviews, and people are complaining about those things. Of course they complain about them. It's not really signal. I understand why they're saying that, but it's nothing that should drive our decision making, or that we can do much about."

[00:05:37] And the third big reason is the idea that it's all really just fluff. "I try to read some reviews, but if you can't really quantify it, someone said something, maybe it even sounds intelligent, it's a good point, but what does it really mean? It reflects one person's opinion." And in order to get even those good opinions, those valid feedbacks, you have to sift through so much, even garbage in certain cases. There's definitely a lot of noise. So again, it's a very good reason why perhaps user feedback is not incorporated into your processes, or into the processes of other really great companies or product managers that you know.

Feedback fits into every part of the methodology

[00:06:18] Now, if we talk about actually using it, it's important to think about how user feedback fits into the overall methodology. I didn't want to go with a specific methodology; there are several good product management methodologies for how you plan your roadmap, what drives decisions and all of that, so I don't want to be too specific here. My point is really that user feedback is not one thing. It's not just, "Let's do our strategy and product the way that we do it, that's our methodology, but let's see if users are complaining about bugs, or are talking about our competitors." Or only looking at it in the context of a user acquisition strategy, or only in the context of user satisfaction: we're going to measure CSAT, and we want to see that people are happy in general.

[00:07:11] All these things are valid, but really, all these other things, including the strategy and even the mission of what you're doing: the mission that you as a product manager, as a project, as a company have is ultimately about serving people in one way or another. That's what we all do. So probably what people have to say about the way that you're serving them is meaningful. User feedback can fit into really any of those contexts, any of those stages, any of those drivers for your strategy. It's not one thing, and that's really the point. User feedback can definitely affect strategy, and even affect the mission, or show you that perhaps we have this mission but we're not in complete alignment with it yet. That's why it can be quite a good thing to integrate into our process, in a way that can affect things across the board. It's not just one corner; it's really all around.

The vocal minority: design your feedback channels

[00:08:05] If I've convinced you, or if you're already convinced, and you think, "Maybe if I incorporate some user feedback into my process, it could be useful," what about those issues that we just saw? The fluff, the noise, the vocal minorities, all that stuff. What do you do about them? Let's try to look at them and see if we can find together some useful ways to address them, and to deal with user feedback in a constructive way despite the challenges it presents, which are definitely real challenges.

[00:08:35] Let's start with the first one, the vocal minority issue. Hopefully people know this meme: someone for whom everything is because of the aliens. It's the same people. "Those users are always saying that. I'm ignoring that. It's not meaningful; it's nothing." And to be very, very clear, it's a very real concern. For many products you would have specific audiences, specific groups of users that may or may not even represent the users that you're actually meant to serve, or that are your best users, or that you have product-market fit for. And those users may have feedback that is not really something you want to act on. It's not in line with your strategy. Maybe they shouldn't even be there, quote unquote, because your product is not meant for them, or maybe they're just a small group that doesn't really reflect your audience at large.

[00:09:35] I'm sure people here have examples of what they've experienced. I can give you an example: in my previous company we had apps where you had a discussion function. It wasn't the main goal of the apps; it was something that was added to create engagement, and it worked really well. The problem is that because it was present in a lot of different communities, in some cases you would have those people who just really enjoy a very abrasive style of conversation. I don't know if anyone has dealt with online discussions, but it can be really hard. There are certain people who really enjoy that kind of interaction, but most people, especially the people who are not easily participating, don't like to observe that. They're not seeking that necessarily.

[00:10:25] So the vocal minority is very real. How would you give those people an outlet, if you want to give them an outlet at all? Maybe the people who want to have abrasive discussions, you want to just drive them off completely; you don't want them there. That's a separate question. But the point is, there are these vocal minorities, and you need to take into account whether the feedback is coming from them or not.

[00:10:46] The first thing that I would say, and I think we don't always see that kind of thinking, is that the feedback channels in your community, whether in the product or outside the product, are actually product too. They're part of how people experience your product. Sometimes we manage a product and we think, "Okay, I'm building whatever it is: this service, this app, this SaaS application. This is my product; this is what I'm making decisions about." It's true. But your users or your customers are experiencing it from the first moment that they're considering using something like that. They have a question, they go online, they seek an answer, whatever it is. So those things, in a way, are part of your overall experience of the product.

[00:11:28] So if you don't have a supportive community, for example, maybe you should have one. Of course it depends exactly what your product is, but this is something to think about. Maybe you want to build a community that you own. Maybe you want to have a Discord server, which is a great solution, and not only, classically, for gaming or things like that. Definitely not only for that. Maybe you want to have a Discord server for your users. Maybe you want to build those feedback channels in the app, or in the product, wherever it may be, in a way that is contextual and really low friction, and would get you useful feedback that you can act on.

[00:12:01] The way people provide feedback, the way people experience your product and other users, and can ask questions, provide feedback and share their pains: whether, like I said, a Discord community, or maybe there's a subreddit of users. Maybe a subreddit that your users created, and you don't even have a presence there. It sounds weird, but it's actually the case for a lot of companies, a lot of products. There are communities online and they're not even participating there necessarily. And this is a shame, because even though you can't really control that, you can participate. And you can sometimes develop communities that you own. That experience of people sharing feedback, asking questions, being in a constructive environment, is part of the product experience at large.

[00:12:45] So definitely something to think about, and I think it can actually be done at a pretty low cost. Very interesting things can be done. So that's the first thing: if you design this thing as a whole, if you design your community, you can also get more useful conversations, more useful feedback, beyond that vocal minority that will always complain about something, whatever it is in your product's case. That's the first way to address it, and it's very important.

Stratify the data and correlate with observed behavior

[00:13:12] The second thing is to really have, and we'll talk a bit about that, and it's not about having a specific tool, a way to get the data, whatever it may be. Maybe it's App Store reviews, maybe it's incoming messages in your ticketing system, whatever it is. You can use very primitive things. You can use a Google spreadsheet, whatever it is that you want to use. You don't have to buy a specialized tool. But it's important to stratify and really apply rigor to your analysis. Maybe you want to get all those feedbacks where it's the same people, and it depends on really what the case is, but maybe it could be according to a specific user trait, a specific type of complaint, whatever it is, and really get that out of the way.

[00:13:56] Use simple tooling, or complicated tooling, depending on what you have in place, and really be able to get it out of the way and look at the other stuff. There are a lot of ways to do that, but the point is, even if you have a lot of noise, get it out of the way so you can do the analysis. That may require different levels of sophistication in getting to that end result, but you can do that. And then you'll find that after you remove the noise, there's actually really useful stuff.

[00:14:23] The third way to address this, and this will be something that will serve us really across the board, is correlating it with the observed behavior. Like I said, probably everyone here has some kind of in-product or in-app analytics, whether it's Google Analytics or Mixpanel or Amplitude [?] and so forth; there are so many great solutions out there. And correlating between that and the feedback is something that, I would say, is not at all often done. It's pretty rare actually that product managers do that, and if you're doing that, way to go, because I talk to a lot of product managers and it's not common at all.

[00:15:00] When you look at the analytics, obviously you see the observed truth; you see what people are actually doing. But look at the kind of feedback that you're getting and the points in time that you're getting it. If you see spikes in specific things: people complain about this thing at a specific point in time, or after this release that we made. Or, let's say you're in gaming, where there's live ops, or different types of products where maybe you've had a special release, or a change that you made for a period of time, whatever it is. You can correlate what you observe in your analytics with what people are saying, and it can give you the reasons behind the behavior. You see the behavior, you don't know what happened, but look at the feedback and you can understand more. So this is also a way to see through all this data, all this noise, and get to something that is beyond what perhaps a small group of users might be saying.

"It's nothing new": filtering, volume and trends

[00:15:57] Now for the second big reason why user feedback is too easy to ignore. That is pretty much the idea that, "Hey, it's nothing new. We know. The pricing? Yeah, you don't like paying. Nobody likes paying. I don't like paying. Nobody likes ads; I don't like ads. That's not signal, and you can ignore it." And yes, that's true: people will always complain about certain things. Here the way to address it is threefold. Some of these continue with what we just discussed, but some of them are a bit different.

[00:16:35] One, you want to have some tooling in place. You see here a screenshot of our product, but it's really not about that. Like I said, you can use Google Sheets, you can use Airtable, you can use any kind of product that works for you, and today there are really great ones. If you can filter the types of things that are always repeating: "Okay, people are always complaining about the ads. People are always complaining about in-app purchases. People are always complaining about our price points," whatever it is, filter that out. You want to have a tool that allows you to filter things out in an intelligent way. Maybe it's some semantic, some linguistic analysis. Or maybe just keywords: you're doing it in Google Sheets and you're filtering those keywords out, and you're getting 80% of the value and you're getting it for free. That's great too. Just get it out of the way and look at the other stuff.

[00:17:23] Because a lot of the time, and I know it sounds simplistic, but this is reality, a lot of people are like, "Yeah, I'm going to read the app reviews, and that's what I get." Okay, it's very nice. But a lot of people don't actually read all the reviews, especially for products that have hundreds and thousands and more. For a big consumer product you would have huge amounts of reviews, and reading through them is actually not easy. That's why you can get that data, put it in whatever works for you, and filter out the things that are repetitive, or even the things that you expect: "I don't want to hear about the ads; I know that we have annoying ads. Let's see what other things people are saying." So that's the first point.

[00:18:01] The second point is really about the volume. Yes, people complain about, as I mentioned, the ads; let's take that as an example. Sure, they will always complain about it, but the way they complain about it, and especially the volume, which is an easy one, can change. Maybe you switch to a different ad provider, or your ad provider added a new type of ad unit to the mix, or some kind of new advertiser. Or you have filtering in place for specific categories of advertising, but, of course, for many companies you're working with some providers, so maybe a new type of ad or a new advertiser somehow got around the filtering. They actually have something that should not be shown to kids, but somehow they got over the filtering, and your product also has kids as users, which is of course very, very sensitive.

[00:19:01] Those things are actually very important to track. You want to see the volume; you want to see if something new is coming up within that same thing. That is actually the danger of ignoring things because "people are always complaining about that," because there could be changes to it. Things change, and you want to see: "Sure, people always comment about the ads, but after this release, after such-and-such date, suddenly the volume spiked, and it's three times as much. Let's look into it." It's worth looking into, because probably something happened. The ads are way more annoying, whatever it may be.

[00:19:32] So really looking at the trend, even if it's an obvious thing that people always say, can really teach you a lot. And again, it's very easy to do. Like I said, take all the data, put it on a Google spreadsheet or whatever you want to use, graph it out, and you'll see: "Oh, look at that, this weird spike. What happened here? We didn't do anything, but something happened." There's a new advertiser, there's a new ad unit, whatever it may be. So that can be very, very useful.

[00:20:01] And the third thing, and this is repeating what we said earlier, but here too: correlating it with observed behavior. You want to see if those things that people complain about, especially where there are ebbs and flows, trends and things like that, have an effect. For example, look at your cohort analysis, your retention: is it being affected? Do you see some correlation between the volume of these repetitive things that people are always complaining about and engagement, with cohorts that are retaining more or not? Again, it's simple, but it's not so often done, and it's not hard to do. You just need to have the data, have the right tooling in place, and have it as part of your process. So it's definitely something that can be done, and looking at the observed behavior and the feedback can really teach you a lot.

"It's all fluff": getting rigor from unstructured feedback

[00:20:49] The third big reason is, "It's all fluff. Blah, blah, blah. So much noise, so many things that are pretty much useless or meaningless. You have to go through so much stuff to get something that's really meaningful, and even if it's meaningful, it doesn't necessarily represent anyone." It's very hard to get statistical validity with that sort of data. That is also, like I said, a very real pain point. So how do you address that?

[00:21:18] First, and this is also about having some kind of tooling in place, some kind of process in place, but here you want to be a bit more rigorous. You want to have the statistics. You want to be able to look at things and see if specific patterns, specific things, are repeating in certain ways. Go beyond the fluff, beyond the blobs of text that people are maybe posting, and try to look for specific semantic themes, specific keywords, specific things that are repeating, something that maybe goes above a certain threshold. It's very hard if you have millions of reviews, which obviously some consumer products have. It's not like you're going to read it all. That's why it seems like fluff.

[00:22:03] So you want to have some sort of way, and here it can get sophisticated. Maybe you do some clustering, maybe you use some semantic analysis. But maybe you just do something super simple: you look at a few keywords. Sure, some people may talk about the same thing but use a different keyword, and that's okay; you have enough data. So even if you use a very simple keyword search in a Google spreadsheet, like a COUNTIF [?], and graph it out, see what's coming up, you can really identify those things that are beyond the fluff, that really indicate some kind of trend, that indicate something that you may want to pay attention to.

What to do next

[00:22:43] In this short conversation we've managed to show how you can take user feedback, customer feedback, and incorporate it into what you do. But I want to make it very real. I want it to be useful; I want to really have something actionable that we can do. So what can we do as a follow-up to that? The first thing I'd say is: incorporate some kind of user feedback processing into your iteration planning. I intentionally say iteration planning; it doesn't have to be that. But I think what we'd want to avoid is saying something which actually happens a lot: "Okay, when we do our quarterly planning or something, we're going to also incorporate user feedback and see if there's something meaningful there."

[00:23:30] That goes back to what I was discussing earlier: user feedback does not belong to one single bucket of what a product manager takes into account. It really takes part in everything it can affect, from optimizing that certain thing that causes people to churn, to aligning with the company's mission. So it's the really small-grain stuff, and also the very big stones that we want to make sure we have in the right place. And I think the way to incorporate it, and the way to make it into a real utility, is to do it frequently enough. Now, it needs to be easy. It can't be too heavy.

[00:24:07] That's the problem: sometimes companies have really heavy processes. They have some textual analysis tools and you have to feed them tons of data, and you get some kind of report, or you have five analysts working on it, and you do it once in a blue moon, and it doesn't really affect much. So the idea is to do something easy that you can do frequently enough in order to actually affect your decisions. And more than that, if you are exposed to the themes, to the insights from user feedback frequently enough, your mind will also be shaped by that. You'll have a better, more intuitive understanding of your users and what might be bothering them, and you'll see things repeating. I can't stress this enough; hopefully I get this message across. Do something that is maybe easy, maybe simple, maybe even stupid, but that you can do frequently enough. That's the first thing that I would say.

[00:24:54] The second thing is that the rubber needs to meet the road. Take a couple of things that we discussed and apply them. It doesn't have to be perfect. Maybe you're not going to listen to the community that you have; you're only going to look at reviews. Or maybe you only take your community. It doesn't have to be perfect. Even if it's one source, maybe you're not looking at everything, maybe you're looking at only a specific subset of things that are accessible to you. It's all good. It's something you can start with. It doesn't have to be perfect, and it's a great place to be.

[00:25:26] And the third thing is that we need to think about how to shape the feedback channels, how to shape the entire experience around the product of people hearing, conversing, helping other users perhaps, and sharing feedback with me, so it's more useful. And I use the word wholesome. Why wholesome? We could talk about that a lot, and I'm not going to talk too much about it, but I think it's very important. Nowadays a lot of smart companies are very much thinking about how you create a user community that is wholesome. Avoiding toxicity, fighting toxicity, is very important. Anyone who spends some time online can see the online space is so toxic. And probably, if you have a product with a meaningful user base, it may be present there as well.

[00:26:12] I think it's on us as product managers to also think about how we structure engagement and interaction, and who gets a voice, in a way that creates a welcoming community. Because how people are experiencing your community and their fellow users is also part of the product. So these are three things that I hope are useful and actionable enough that you can do. Building a community can be very easy; maybe you want to start a Discord server, or incorporate a feedback channel. There are a lot of ways that you can really act on that. And I think this is it. If there are any questions, happy to answer, and I thank everyone for joining.

Q&A

[00:26:49] Host: Great, thank you very much. That was really interesting, and I think it's something that a lot of people can relate to, because I know all too often how you get people just dismissing things with the law of small numbers, where you've only got 10 people saying something, or six people. So that was really helpful, thank you. We're actually out of time. There was one question that came in: the one objection they continuously get is, "Well, I know what's needed." So it's not even the problems with the research; it's that we don't need it at all in the first place, because we already know. We are out of time, so what I'll do is invite you to the Slack channel. I'll post that question in the Slack channel, and if you could come and answer it there, that would be fantastic.

[00:27:44] Itamar: Of course, we'd love to.

[00:27:46] Host: Great. Well, thank you very much.