Keeping It Fresh: How to Evaluate In-Market Products

Lan Allison, Jen O'Brien, Danielle Krusemark at UXDX Community: Evaluating Performance The Right Way. Video: https://youtu.be/EieX9EXKCmI

Readable transcript: edited from the recording's captions for readability (fillers and false starts removed, punctuation and section headings added). Wording is the speaker's own. Timestamps are positions in the video. Names marked [?] could not be verified against the audio.

Why in-market products need evaluating

[00:00:07] Jen: Hi everyone. My team and I from the American Marketing Association are excited to share a step-by-step process for evaluating in-market products.

[00:00:19] There are a lot of great resources out there on the earliest phase of the product life cycle, developing new products. You probably know and love some of these books. I know the Sprint book lives right up on my bookshelf. It's something that I go back to time and time again.

[00:00:37] But when it comes to continuing to evolve and innovate products after launch, we didn't find as many tried and true approaches to help guide us. Especially for established companies, there are a lot of opportunities in the later phases of the product life cycle that we want to take advantage of, thinking about technology, competitors and customer expectations that are shifting quickly. As product teams, we need to constantly evaluate and adapt our products to stay relevant.

[00:01:06] Especially with some of the recent advances in AI, a company's foothold in a specific market can quickly be disrupted. How can we make sure that we are staying on top of it? This type of continuous evaluation and innovation may be standard practice at some companies, but our team did not have a process in place to do this effectively. A few years ago, our products were on autopilot. We feared making changes to our long-standing customer offerings, and we didn't have a good way to determine when it was a good time to sunset a project or product. A lot of times we just kept on doing what we were doing without making changes.

[00:01:54] To put this challenge into the language we use as researchers and product people, we were trying to figure out: how might we evaluate our in-market products so that we stay relevant?

Who we are at the American Marketing Association

[00:02:08] Before I share the solution that we developed for this problem, I want to provide a little context about who we are at the American Marketing Association. We are a professional association. We offer training, events and certification for marketing professionals and marketing researchers. We are a nonprofit that's been around for a long time, 85-plus years. On the one hand, we have the benefit of being a really credible, recognizable brand. We're really established, but on the other hand, things can be pretty slow to change within our organization.

[00:02:46] Product is a function that has only been around for a few years, so our way of working hasn't been around very long in comparison to our company's long history.

[00:02:59] In addition to being new, our team, the product development team, isn't embedded in cross-functional product teams that are the gold standard for this type of work. We work mostly as consultants and facilitators with non-product people. Our challenge is creating a product culture, and a culture of using customer insights to drive decision-making, amongst folks who might not have that kind of experience or that kind of skill.

[00:03:28] Next we're going to share a process for evaluating in-market products with our team at the American Marketing Association. We like to call this our business review process. That's our internal lingo.

[00:03:46] Here's a bird's-eye view of the product evaluation process that our teams have developed for our particular circumstances. We start with addressing biases, because that's a place where we found our team has been most likely to get tripped up. Then we go into mapping the current state. We pause to frame good questions before diving into the data, and then sharing out our findings.

[00:04:08] Our hope in sharing this process is that there might be some tools or ideas you can adapt for your own use. We'll be sharing a Miro board template at the end to make it easy for you to see how we structure these evaluations and how we do the activities in detail, and you can take what works for you. I'm going to pass it on to Lan, who's going to go into each one of the steps.

Assumption island and the business model canvas

[00:04:32] Lan: Great. Thanks, Jen. Zooming in on one of the very first things that we do on day one, after a brief orientation to the overall process, is addressing our biases and assumptions head-on. Some of our reviewers who are evaluating a product are evaluating something that they work on in their everyday jobs, but other times our reviewers are pulled from other departments to try and balance some objectivity.

[00:04:55] Regardless of who is reviewing, everyone goes through this exercise we like to call Assumption Island. We give them a short refresher on the different types of biases that are out there: status quo, confirmation bias, loss aversion. I'm sure you're familiar with some of these things. The reviewers are asked to write down everything that they've heard about the product they're reviewing, both good and bad. Maybe they've heard that their product is the biggest money maker for the company, or that customers really love its adaptability.

[00:05:27] Whatever the case may be, the objective is to get all of their assumptions down and laid out on paper.

[00:05:34] And here's the catch: assumptions are not allowed to leave the island throughout the review unless the reviewer finds substantial data throughout their process that proves the assumption to be true.

[00:05:48] To put this into context a bit, I'm going to show you a review that just went through for our daily newsletter, called myAMA Daily. This was reviewed by one of our colleagues from the marketing team. You can see here some of the assumptions that Jonathan wrote down that he has heard about myAMA Daily. Things like: people want a daily newsletter, the daily newsletter increases engagement for the AMA, people care about AI-powered content. Just things that he has heard by working on the product, or things he has heard from different colleagues of ours surrounding the newsletter itself.

[00:06:31] Now that we've addressed our biases, we're aware of their existence and they're locked away on Assumption Island. We need to focus now on the landscape in which the current product stands. We really like to use the business model canvas framework to help us out. It's a nice way to visually see how your business model works, the customers you're targeting, the value you're creating, and then the resources it takes to actually deliver that value. Remember, it's all about mapping the current state at this point in the process. Save your ideas of changes or adaptations until a little further into the process. But the business model canvas will be your first building block for that type of evaluation.

[00:07:15] Going back to our newsletter example, here's a business model canvas that Jonathan created to tell the story of how myAMA Daily is currently running at its current state. It looks at the customer segments of our newsletter subscribers, but also our sponsors, and then really maps out all it takes as a company to run this product.

Framing questions before the SWOT

[00:07:40] Okay, we have the current state of the product laid out, and now we can start evaluating it through a SWOT analysis. But data pulling can be intimidating without some guiding questions. Before we have our reviewers jump straight into pulling and collecting the data behind their product, we set them up by asking some framing questions tied back to the business model canvas. This allows them to prioritize and create direct questions that they can answer with data, rather than pulling a bunch of things and then running into the issue of having to synthesize it accurately.

[00:08:12] Strategyzer has a really great framework for asking these types of questions in the context of conducting a SWOT, and we like to utilize it to ensure that the reviewers are bridging that gap between their business model canvas and their SWOT analysis properly.

[00:08:28] Here you can see some of the example questions that Strategyzer has laid out when you're looking at, for example, the value proposition of your business model canvas.

[00:08:41] Here you can see some of the questions that Jonathan created to set himself up for his data dive, all within the scope of feasibility to answer. Why do people unsubscribe? When are people subscribing? What are the job levels of those who are subscribed within our audience? Things that can really guide where he's finding his information and how he's finding it, rather than trying to take it head-on and looking at everything without a lens.

The data toolbox and pairing with reviewers

[00:09:14] Now we're finally ready to start diving into the data and getting our hands dirty. We have found a few common barriers across most of our reviewers that hold them up from beginning that race. It's the where and the how. Where do I find the answers to my questions? What resources do we have? And then how do I actually find it? How do I navigate that specific platform to pull this data?

[00:09:39] Because data literacy ranges and people have different levels of comfort with finding and interpreting data, we really want to teach, equip and empower those who might not work with data directly in their everyday work to be able to learn these skills on their own. To address this, we do a couple of different things. First, we put together a data toolbox, which you'll see on the right-hand side. That is simply all of the platforms that we at the American Marketing Association use to collect data. This ranges from platforms like Mailchimp to Salesforce, Qualtrics, a lot of other things. We fill our data toolbox with the platforms we use, along with a brief description of the type of data each collects and who the go-to person would be for access. This allows a clear landscape for the reviewer to understand what is possible to uncover and where to go.

[00:10:33] Then we address the how. How do I find these things? It's one thing to know that we collect surveys and qual. It's another thing to know how to access the platform, how to navigate the back end, how to pull the information you're looking for. To address this, we use the "teamwork is the dreamwork" framework, pairing ourselves with our reviewers during this process as dedicated helpers to ensure that they're getting one-on-one support and guidance throughout.

[00:11:10] Here you can see an example of a snippet of the information and data that Jonathan was able to collect when looking at our daily newsletter. For this specific example, he was looking at the key question of whether customers are engaging with the newsletter. He answered this by benchmarking it against some other newsletter performances, both internal and external. He was able to do three different benchmarks with six different data sources from our data toolkit, looking at three different metrics. You can really see the breadth and depth of data when we're able to unlock that platform accessibility and make sure people understand all the tools that they have access to.

Sharing the findings

[00:11:58] Once the data has been gathered and the SWOTs have been completed, we are almost done with our process. It's time to wrap it all up in a pretty bow and hand it off to whoever needs to hear this review. This could be stakeholders, upper-level management, internal teams, you name it, but the information has to be shared. I'm sure you're all familiar with the struggle of presenting information. There are a lot of great talks out there about how to present the information that you found, because it's a hard thing. It's hard to collect research and interpret research, but it's also really hard to articulate that research accurately and effectively.

[00:12:36] One way we try to facilitate this is by creating a very simple Google document template that the reviewers use to help guide them in sharing their discoveries in a way that is both succinct and impactful. I'm sure you're familiar with these types of models. I know Amazon has their six-pagers. Other companies have slide decks. Whatever you feel is the best method for your team, I encourage you to create a template that people can use, so that all the information is being presented equally, and it's also very easy to go back and reread and review all the work that has been done.

[00:13:19] Within Jonathan's final review analysis, he went over his current-state business model canvas, all of his SWOT findings and high-level information. All of our reviewers end with this type of next-step focus: what is our next-step recommendation, where he thinks our time, energy and resources should be spent after conducting his review. He just spent the last four weeks gathering data, looking at it, evaluating it. Our reviewers are in a great position to connect the listener to where we think we should go from now. It's really great for us to not only have a review, but then have some action items tied to it.

Implementation and results at AMA

[00:14:06] I'm going to pass it to Danielle to go over implementation and what that's looked like for our organization.

[00:14:12] Danielle: Thanks. I'm going to talk about how business reviews have ended up resulting in different implementations at AMA. Across different business reviews that participants have conducted, a variety of actions have been suggested for our team and organization to take. Most commonly, participants have ended up suggesting that we conduct additional research into the product or process, probing further. Next most commonly, they've suggested increasing the product or process, expanding it into a different area, utilizing it more. Less commonly, participants have suggested replacing the product or process with something else, or revising it in some manner, or, least commonly, sunsetting that product or process.

[00:14:56] Of these actions suggested, the vast majority were agreed upon and implemented by senior leadership. Rarely were they not implemented, and even more rarely was something different entirely suggested by senior leadership. At AMA, business reviews have largely been successful in enabling our team to generate actions that have been successfully implemented.

[00:15:18] In addition to this, business reviews have been largely successful on a personnel level. The people who participated in these business reviews held largely favorable reviews. Most of them agreed or strongly agreed that these business reviews helped improve their expertise and experience in data-based decision-making, strategic thinking and innovation frameworks. Moreover, everyone agreed or strongly agreed that these business reviews should be continued, that there is something valuable and beneficial in them. Overall, business reviews have been a great success at AMA.

Key takeaways and the template

[00:15:57] To sum it up, some key takeaways from our presentation. First, when starting this process, take a little bit of time to assess and address your assumptions and biases about your target product or process head-on, what may be coloring your research and your end result. Use the business model canvas as an organizational foundation so that you stay on track and keep things organized. Make sure that your SWOT analysis is framed with the right questions, so that you're remaining on target and not getting bogged down in the weeds or anything that's off topic.

[00:16:36] While doing this business review, make sure that everyone who's participating has strong data literacy, by providing toolkits, tutorials and help, and pairing them with experts in the different data areas, because your participants will likely vary in their data expertise and comfort. And last, create a template that you can use over and over again that is easily accessible and consumable for your different participants and audiences, who will have differing levels of expertise and comfort with the process.

[00:17:07] Next, Lan is going to share the template that we have for you.

[00:17:15] Lan: We didn't want to leave you empty-handed. The QR code to the right will lead you to a Miro board of our business review template that we currently use within our organization. Our hope is that you can take inspiration from our talk today and implement this process into your everyday work, so that you too can start effectively evaluating things that are in market at your company.

[00:17:42] Jen: On this slide we have our contact information. Please feel free to reach out. We'd love to hear your questions and ideas if we don't get to them today, and we'd love to learn more from you about what you have found works when it comes to evaluating in-market products. From here, I think we have about 10 minutes to take questions.

Q&A

[00:18:04] Host: Excellent. Thank you very much. I found that personally very interesting, because I love the way you went into the detail of showing the outcome of it. Often that kind of process can feel very stale, but I love the way that you brought it through a real example. Thank you for sharing that. We do have a question in from Rebecca. She's asking, "How do you decide or prioritize which products to review?"

[00:18:34] Does anybody want to take that one?

[00:18:38] Jen: I'm happy to take this one. Initially, when we first started the process, we had some products that were marked as things that our team had decided should go through this process. They were products that people had been feeling unsure about for some time and wanting to make changes to, but that decision point had never come up. Also some of our bigger revenue-related products.

[00:19:10] After that happened, I think there was some success with this process, and everyone was like, "Everything needs to go through a business review. This needs to go through a business review." We went through a period of taking a lot of requests for maybe smaller products that didn't have such a big impact revenue-wise on our business. Going forward, we are returning to this idea of what are the big products that have the biggest revenue implications and customer impacts for us. That's the way that we are taking things going forward. We're also adapting some of these tools for different processes we have, taking more of a portfolio review approach with some of our products, which is pretty exciting, but we didn't want to bog everyone down with the details about that.

[00:20:11] Host: Brilliant. And as a follow-up to that, how much time do you invest in doing one of these reviews?

[00:20:19] Lan: I don't think we mentioned this at the beginning, but our company is completely remote. We are doing all of this online, hence why we utilize Miro boards. Because of that, this takes about four weeks. When I say four weeks, I don't mean four weeks of having a meeting every day. It's typically one or two meetings a week, because it's not only doing the process, it's learning the process. Typically those meetings are an hour. We're going over, okay, here's what a business model canvas is, here's how to conduct one, here's why it's important.

[00:20:52] Then we give people at least a week to go and conduct things. We want to make sure that people are given enough time and space to go find the information they need and do it. They obviously are also doing other things for their day-to-day jobs. Typically it takes about four weeks, but a lot of that is asynchronous, time for them to work independently.

[00:21:16] Host: Great. One thing, and I guess it's the controversial topic: when you picked up SWOT analysis and business model canvas. An example that I did, I typed into ChatGPT, "Give me a SWOT analysis for UXDX," and that was all of the context I gave it, and it gave me a very good SWOT analysis. What's your opinion on using tools like that? Do you think the value is in people going to the effort of really thinking about their own products, or do you think it's the end output that's important?

[00:21:56] Jen: I'd love to take that one. I think ChatGPT is a really helpful tool for doing the reviews, especially in terms of external data and things that exist on the internet. A lot of the data about our products is not out there on the internet, so ChatGPT doesn't have access to it. It can't see what registration has looked like over time, who exactly our customers are, demographic information about them, what their job titles are. It doesn't have access to all the data, if you will, but it's great in terms of things that aren't in our secured data sets.

[00:22:43] Host: Brilliant. And have you found that some of the people that you're teaching and sharing it with are using those tools?

[00:22:54] Lan: I'm not sure. I'll jump off of what Jen was saying too: I think tools like that are great starting points, because again, we want to go back to addressing biases. We found throughout the process, and we're still iterating the process of course, that it's really hard when you work on a product every single day to see a full picture of it. Something that we're currently working on is making sure, like Jen said, that people are getting those macro views, and not just that internal view of, well, we have one survey and we want to stick to the survey, and this is what people said about our webinar, when there are other things that we can benchmark against too. I'm not sure if anyone's using ChatGPT with their business reviews, but I would say it's a great place to jump-start and see what other platforms, software, humans, I don't know, ChatGPT, are saying about some of our things.

[00:23:49] Jen: I will say, just today I saw there is a CSV GPT, or I think it's called ChatCSV, and it's really interesting. You can upload a CSV of your own data, so data that wouldn't exist on the internet, and ask it non-expert questions about the data, and it can even visualize it. That's just something that I was playing around with today, but tools like that could be really helpful, especially for the kind of people that we're working with, who maybe don't have a ton of expertise in terms of visualizing data or doing analysis. It can shortcut some of that for them so that they're able to more easily navigate through data sets.

[00:24:37] Host: One of the other things that came up when I was reading the outcome of the review was: what is our business goal here? Are we looking for revenue? Are we looking for value add? I've had those conversations internally, and so many people don't want to say definitively we're this or we're that, because, oh well, we're doing all of those things. Did you encounter that resistance to being strict on what the goals were, and how did you overcome that?

[00:25:15] Jen: I don't know who would have the best experience in answering that one.

[00:25:21] Lan: I'll take a crack at it. If I'm hearing it correctly, it's how do we overcome some of those barriers of, this is what the product should be, or these are the goals that it should have, and some resistance within that when you're doing a review.

[00:25:37] Host: Yeah. In my experience, let's take your newsletter: revenue generation, and it's value add. People are like, "Well actually, you know what, it's both." It's hard for you then to prioritize, because it's trying to be a product for everybody instead of asking where your focus is.

[00:25:56] Lan: That's a great question, and it's something that I don't know if we completely have the answer for, because that comes up a lot. Something that we try to make sure our reviewers are taking into account is our capacity and our limits. As Jen was saying earlier, we are a very small team, and it's really easy to benchmark us against bigger corporations with a lot more resources. Something that we noticed, especially with the business model canvas, is that when people look at the cost section, it's so overlooked. No one accounts for staff time, and staff time is a huge cost that's not monetary for the product.

[00:26:38] When we come to situations like that, of this is a product for everybody, how do we make sure that we're touching everything while prioritizing what we're capable of? We don't want to do everything poorly; we would rather do a couple of things well. What does our staff time have the capability to do? And then really looking at it in a portfolio sense: where does this product lie, and where do we want it to sit within our portfolio? If we have three things that are all trying to do the same thing, can we maybe pull back on one and really invest in something else?

[00:27:12] Host: Brilliant. All right, that brings us to time. Thank you very much, Jen, Lan and Danielle, for sharing that fantastic case study.

[00:27:21] Jen: Thank you.

[00:27:22] Lan: Thanks.

[00:27:23] Host: Comment in the section below, and let's keep the conversation going. Finally, if this video resonated with you, please do us a favor and hit that like button and subscribe to our channel. This not only keeps you up to date with all of our latest content, but it also helps us with the YouTube algorithm. Last, if you want to keep learning, we have a couple of videos suggested here. Otherwise, until next time, take care and enjoy the rest of your day.