Taking Action on Data: How to Set up a Product Health Program

Lilia Royanova, Julia Yager-Reisen at UXDX Community: Taking Action on Data, Career and Work Life. Video: https://youtu.be/H9sc2VqZ8EE

Readable transcript: edited from the recording's captions for readability (fillers and false starts removed, punctuation and section headings added). Wording is the speaker's own. Timestamps are positions in the video. Names marked [?] could not be verified against the audio.

Introductions and why product health matters

[00:00:00] Julia: Awesome. All right, hello everyone. We're really excited to be here with you today. As Rory said, I'm Julia Yager-Reisen, and I'm a UX program manager working on Google Cloud Platform. I've been at Google for nine years, and I'm here with my co-presenter, Lilia.

[00:00:18] Lilia: Hey everyone, my name is Lilia Royanova. I've also been at Google for about nine years, also a UX program manager, currently in the Google Cloud organization.

[00:00:30] Julia: Awesome. By the end of this talk we're going to answer the following questions. Why is product health important? How can you get buy-in from partner teams to implement a product health program? Why should you align on a single measurement framework? And how can you integrate product health best practices into your product health life cycle?

[00:00:55] We want to hear from all of you. We're curious: why should we care about product health? What does product health mean to you? Please go ahead and type some answers into the chat. Maybe a little quiet today. All right, we can keep going, and we'll tell you a little bit about what we're thinking.

[00:01:24] There are a lot of dimensions to consider when it comes to determining if your product is healthy. What's working in your product, what's not, are you growing well, are users being retained, and so on. When we talk about product health in this context, we're most interested in finding a consistent approach to evaluate the impact of what we launch, in order to track progress against key outcomes. To put it simply, the problem we're trying to solve is being able to answer the question: are we achieving our product goals, our business goals?

The streetlight effect

[00:02:07] Let's consider a type of observational bias called the streetlight effect, which is when people only search for something where it's easiest to look. As the anecdote goes, a policeman sees a man searching for something under a streetlight and asks what the man has lost. He says he lost his keys, and they both look under the streetlight together. After a few minutes, the policeman asks if he is sure he lost them there, and the man replies no, he lost them in the park. The policeman asks why he is searching here, and the man replies, this is where the light is. In essence, it's harder to find something on the part of the floor that is not well lit.

[00:02:49] We all collect a great deal of data from a wide array of sources, but each of these individual pieces of information is like a piece of a puzzle, and each of the teams that collect them is looking under their own individual streetlamp. It's really critical to think through the best way to bring the pieces together and shine a light on the full picture of how the product is doing across the entire experience and stack.

[00:03:23] Using a consistent framework across your product helps make sure you're bringing all the data together for that big picture view of product health, and then enables you to take action on the product via data-backed decisions in order to move the needle on your specific goals.

The product health life cycle

[00:03:42] Let's take a look at how to apply this type of thinking within our version of the product health life cycle. Initially, it's really critical to define what good looks like. What do you think is important to the business? What is going to be really important to the users to get right?

[00:04:04] Then we need to make sure we can measure whether users can actually achieve the goals that we outlined. Pre-launch, this can typically happen via benchmarking surveys and A/B testing, but it's really critical to instrument your product so that post-launch the user data can be properly collected and then analyzed. By instrumentation here we mean implementing tagging within your product to enable metrics and user tracking, which is a great process that sets up a good foundation for easy and insightful analytics.

[00:04:40] Next up, we'll monitor and assess, to have a sense of what the key things are that we need to work on to improve, as well as how we're tracking over time. And finally, we'll determine where to focus our efforts and always finite resources, to make updates and continually evolve the product. All right, and with that I'll pass it over to Lilia.

Factor one: articulating the value of a product health program

[00:05:11] Lilia: Awesome, thank you, Julia. We know why it's important to set up a product health program. Julia and I have actually done this with our teams at Google Cloud, and so we'd like to share with you some key factors for setting up a successful program that will last and really take root in your organizations. For us, obviously, we are working on an enterprise product in the context of large teams, but we think that these are broadly applicable, whether you're working on a consumer product or you're in a smaller organization. Hopefully the lessons carry forward regardless.

[00:05:50] The first factor is articulating the value of the product health program to teams, almost like what we focused on at the beginning of this talk. Why is this important? It's really about getting back to that question that Julia mentioned: are we meeting our business goals? If you have each team looking at siloed data and not really aware of the importance of the data between the different teams, or understanding what it actually means, then you may end up with team-based solutions, piecemeal fixes to things. Whereas if you have a more holistic approach and you take a step back, you'll be looking at the big picture, you'll help your leadership and your stakeholders be more informed in their decision making, and ultimately have better, more integrated solutions.

[00:06:48] As we mentioned, there are a lot of different teams that may be involved here. Your support team is collecting support tickets, and they'll probably have a priority of which topic we're getting the most support questions about. Your finance team will be thinking about revenue, usage, adoption, retention, churn; they'll probably have a lot of data regarding that. UX research will be thinking about surveys and customer feedback, verbatim requests for features. Account management and sales will probably be listening to customer decision makers. All of these different teams have such an incredible wealth of information, but it may live in its own world, in its own pockets, if we don't bring it together.

[00:07:34] Let's walk through an example. This is a very simplified version, but let's say that your product is focusing on adoption and usage, and you can see in this chart it's up and to the right. It's going well. However, if you take a step back and superimpose satisfaction, or CSAT, you can see that it's actually going down in this case. By being able to look at more data in a framework that makes sense, you can identify and address issues that you may not be able to see individually. An example of this is some enterprise products, where users might have to use them because they were purchased by their organization, but they might not actually be so easy to use. You can think about expense tracking software, for example.

[00:08:26] Some of the key lessons that we want to share from our own experience implementing this kind of thing. Being able to avoid blind spots by bringing data together, and having aha moments like bringing in that CSAT. We've actually seen that happen with some of our engineering leaders and some of our product leaders, and that truly shows us how important it is to bring this data together. Every team's work is important: support, sales, UX research, it doesn't matter who it is. It's all important information that can help us make better business decisions together. And getting cross-functional buy-in from all the leaders of those organizations, or, if you have a smaller org, maybe just individuals, getting that buy-in at the outset that you want to work together is going to be very helpful for making this successful in the long run. And I'll pass it back to Julia.

Factor two: rallying around a single measurement framework

[00:09:24] Julia: All right, the second key factor is focused on getting everyone to rally around a single measurement framework. Overall, the most important piece is that the framework must be consistent. It's critical that teams across the organization begin speaking the same language, but also that they begin standardizing the way that they measure and track their metrics. If each team is measuring and tracking differently, it decreases the ability to make clear decisions and it makes prioritization difficult. The same framework enables consistency, and eventually it makes it much, much easier to get to comparability, which really decreases the effort needed to make decisions.

[00:10:10] Next we'll take you through some examples of frameworks across the industry that you can use. We worked with our research team to come up with a framework which is an amalgamation of a few of these, and which works best for our particular org's needs.

[00:10:27] One of these that we pulled some inspiration from, obviously, was Google's HEART framework, which measures the quality of the user experience and identifies the goals of a project. Another one that was really interesting is the pirate framework. Abbreviated, it spells AARRR. These metrics are intended to capture each stage of the customer life cycle, and then each product will have unique metrics within each bucket that then track up to key performance indicators of the business.

[00:11:00] Another one that was really interesting to look at was Sequoia's framework, which focuses on how consumer companies can measure specific aspects to evaluate product health, in order to continue to grow well, penetrate their market, and retain and engage users.

[00:11:17] The benefit of a framework is its ability to flex and scale. Let's say you have two products: one internal with a small number of users, the other external with a larger set of users and a higher revenue stream. The way that revenue interplays with CSAT is going to be the same no matter if it's an internal product or an external one. You can't compare revenue in this case, but you could compare adoption. As a result of having this holistic framework, you can say that the product on the left is still a healthy product, even though these are two very different contexts. You might use a stand-in for revenue, such as time saved, but this framework enables you to compare the relationships between and across the products.

[00:12:03] The big lessons learned on this one were, one, education around the health of your product is really important, to make sure everybody understands the baseline. This can include everything from basic sentiment metrics to education around where the data is coming from. Remember all those different streetlamps, and everyone's been focused specifically on their own space. But we need to make sure everyone gets familiar with the various data sources that exist, as well as their current benchmarks.

[00:12:38] The second lesson we really discovered was around starting with the framework first and not trying to force comparability at the outset. It will come later; it's a longer term north star. Don't force something so stringent initially. Flexibility is really key to getting everybody ramped up and going.

[00:13:01] Last but not least, in a similar vein, lean into the gaps. You're never going to have everything perfectly in place from the jump. If you don't have any data for one dimension of the framework, acknowledge this, surface it within the holistic picture you're showcasing, and then make it an input into prioritization and roadmapping discussions. Don't make it a blocker. With that I'll pass it back to Lilia.

Factor three: integrating product health into the product life cycle

[00:13:32] Lilia: Awesome. We have aligned on the value of the product health program, we have aligned on a single framework that we're all going to use, and now it's really about making it real. The third factor we want to focus on is integrating product health practices into your product life cycle. Why is this important? Again, a very similar trend here. Instead of focusing on individual standards and team-based approaches, team-based fixes for things, you can supercharge your organization by having templates, consistent processes and playbooks to make sure that everyone is on the same page and speaking the same language. Of course each team is collecting different types of data, but as long as at the high level we are aligned on the types of things that we want to measure and what they mean for our product and our business goals, that is really helpful for us to be a successful business.

[00:14:31] Let's walk through a couple of examples. Remember the product health cycle that Julia walked us through. We use this as the basis for each of the examples we're going to share, but in your organization you may have your own product life cycle defined with your cross-functional partners, you may have a design process that you stick to, and so these will probably slot into those frameworks as well.

[00:15:00] In the very beginning, when we define what good looks like, our product managers use product requirement documents. This is a doc that contains all the information about a new feature or a new product that we're going to work on, including things like who the target users are, what the features are going to be, and what the technical requirements are. As part of that list of information, you want to include success criteria based on the framework for product health that you align on. Let's say you use the Google HEART framework: you actually want to walk through and think about what each one of those buckets is going to be for this particular thing. And you can imagine that if all products or all features within the org use that consistent planning method, it's going to help bring consistency overall.

[00:15:56] As program managers, we want to make sure folks are getting started quickly. We found that getting started guides have been super helpful for both of the teams that we've worked with, and others as well. It helps to standardize and decrease the ramp up time. So things like explainers and playbooks for how to set up a dashboard, or explainers of how to think about each metric and what it means.

[00:16:23] Once you get into the instrumentation stage, being consistent in how you evaluate and instrument your user journeys is another great way to ensure that you have product health baked into your process throughout your life cycle. For us, this has meant making sure that critical user journeys, or core user journeys, are tracked consistently.

Dashboards, communication and forums

[00:16:54] Once you've defined what's important and you're tracking it consistently, how do you make sure that you're monitoring and regularly getting insights from the information that you've now collected? For us, dashboards are a pretty great tool. We found that they resonate with both our high level leadership and with individuals on the team. They provide a way to standardize, again to the same framework, so that everyone's seeing that same consistent visual. But they also allow folks who are closer to the product to drill into the data if they would like to, whereas our leaders can stay at the high level and just take a look through very quickly.

[00:17:41] Just a little tip here: at Google, internally, we use Looker Studio, and this is just one of the many templates that they provide directly. It can seem a little daunting to try to put together a big giant dashboard, but there are tools out there, whether from Google or other providers, that you can use.

[00:18:03] The next two items are applicable throughout the life cycle, so you can see we've lit up the whole thing here. Central communications are really important for having a standard language, making sure everyone's on the same page, and maintaining a clear set of owners for what you're doing, whether it's owners of the instrumentation pieces or owners of particular product improvement working groups.

[00:18:31] And lastly, communication can come in many forms, like email or posts in your chat spaces, but we also recommend having a regular forum where everyone gets together, all your cross-functional stakeholders and product owners, and you discuss all the information that you have gathered. It's a great place where people can bring their questions, they can learn more, and they can even identify what working groups can get started to improve on any issues that you might be seeing.

[00:19:08] To summarize lessons learned: we talked about a lot of different examples of things you could be doing across different teams and across different products, so it can feel a little bit overwhelming. We highly recommend starting with a pilot with just one product or just one area of your organization, and then scaling up from there.

[00:19:31] Definitely allow for some deviation. Just like Julia mentioned with the framework that you choose, with the program that you develop it may be different for different teams: how they prefer to communicate, how they prefer to work, how they define success criteria. So allow for some deviation, as long as at the high level you're aligned.

[00:19:54] And then lastly, it's really important to keep channels of communication open. Whatever makes sense for your org, whether it's a regular forum, a chat channel, an email that you're regularly sending, however you all get together and talk, make sure that you are cultivating a culture of sharing and asking questions, so that everyone's on the same page. And I'll pass it back to Julia.

Summary: illuminating the whole

[00:20:19] Julia: All right. In summary, having a standardized framework to evaluate product health enables you to know if you're achieving your business goals. In order to make this effort successful, we recommend articulating the value of product health, getting buy-in from all those partner teams across your space to implement a product health program, getting all of those folks to then align on a single measurement framework, and then integrating product health best practices that make sense for your product health life cycle.

[00:21:01] If we revisit our streetlight again: if it's harder to find something on the floor where it's not well lit, we need to make sure we are illuminating the whole, not just focusing on one area. This takes a bit more time and effort, but it will set your product up for healthy growth in the long run. And with that, I think we'd like to open it up for questions. Thank you all for your time today.

Q&A

[00:21:31] Rory: Thank you very much. That's a really interesting insight into how you've structured it. I'm always intrigued at how other companies are tackling the same problems that we face, so thank you very much for sharing that. If anybody has any questions, please do write them in on whichever platform you're on and we'll pass them through to Julia and Lilia.

[00:21:55] But I wanted to get started, because as I was hearing that, what kept jumping into my mind, and it's a bit of a trendy term now, is product ops. It's a team that is responsible for the advocacy of better ways of working, of streamlining, of writing guides and things like that. What's your opinion of this kind of collaboration that you've just described versus the product ops pieces that are spinning up in companies around the world now?

[00:22:28] Lilia: Yeah, I can start. We work very closely with... we do have product operations folks. We call ourselves UX program managers, and we also have engineering program managers as well. So there's a whole set of people working across teams to make sure that we are running smoothly and that things are going successfully. I would say it's another team that we have to interface with. From the UX side, we're really bringing a focus on the user experience. But, for example, the finance team will have a lot of product folks working on initiatives to address financial goals. So it's about everyone coming together, having a conversation and getting on the same page, as opposed to us running something, product running something else, and engineering program managers doing a different thing. It's best to just align.

[00:23:32] Rory: Great. And is there any authority there, or are you doing alignment just through influence?

[00:23:41] Julia: I can take that one. I would say it's about alignment, it's about influence, absolutely. But it's also about highlighting and echoing back the problems that leadership is facing, and making sure that those problems are real and that they resonate. I think of that streetlamp of everybody looking individually, product being one of those streetlamps, where they're looking at things through a very specific lens. Gaining that alignment, everyone has the benefit of having the holistic picture.

[00:24:19] It was around some experiments with pilot teams that we really helped showcase that. By bringing that third metric in, like we did see with the CSAT example, all of a sudden, instead of indexing on two KPIs and letting some other things slide, we were able to look at the whole picture and make sure that we were improving the product holistically and not over-indexing on one or two things. So it is definitely through influencing, but I think if it's a real problem, people will listen.

[00:24:55] Rory: Yeah, you're helping them solve a pain point. Okay, great. I think this is more a comment than a question, from Natalia. She says, thank you for the presentation, interesting points. In my previous work I was always pushing and cultivating a culture of sharing and asking, which brings the team together. So I think she's agreeing with your comments there.

[00:25:19] One thing, actually, just jumping on the point you mentioned there about the third metric that you introduced, and that it highlighted new and interesting things. The North Star metric is popular, trendy in some companies, and YouTube was actually one of the companies that popularized that, with minutes watched. What's your opinion, after your example there, that when you start layering your metrics is when you get the real value? I don't know who is best; maybe we go back to Lilia.

[00:25:54] Lilia: Sure. You can't look at a million different metrics and make decisions based solely on that. For sure you want to prioritize what you're going to focus on, and I think there are north star metrics within the different components of the framework that we're talking about. To make it more concrete, if we're talking about user sentiment, there's actually a ton of different user sentiment metrics that we have available: satisfaction numbers that we have from surveys, satisfaction numbers that we have from in-product feedback, satisfaction that we hear about from support. Bringing all of that together is the first step for us to then decide, okay, which one of these is the most important, and which one of these is really going to move the needle for us.

[00:26:54] Rory: Friends, you mentioned two things that led to my last question, I guess. If anybody has any others, please do send them in. It was the mixed methodologies. You said start with one team, but if everybody's starting, I might choose pirate, you might choose HEART, somebody else has gone North Star. How do you then try to massage and move them back into a single one?

[00:27:21] Julia: I would definitely figure out the framework first. I wouldn't say that a small pilot should be empowered to choose a different framework to pilot. That would be a case where you need to align initially on what might make sense for your org, and I definitely recommend working with the research team to assess what is going to make the most sense given the business goals of your organization. That will help lead you to what are the things we're going to care about the most, and what do we need to keep reviewing at all times, and it'll help you define your framework. Or you can leverage, I think Google's HEART framework is a great one, off the shelf, to just start with, and then tweak from there if it makes sense.

[00:28:11] But I wouldn't suggest the pilots be around changing the framework that gets used. It's more around test driving the implementation: getting teams ramped up on using the framework, and making sure that you vet the processes of how they're implementing each aspect of it throughout that product health life cycle. What worked really well, and then iterating on where maybe you hit some snags with that team, so that once you expand those pilots and scale them to other teams, you've done it once within your org and you know what's going to work really well.

[00:28:51] Rory: That clarified it a lot better for me, so thank you for sharing. And I think that actually brings us to time, so I just want to thank Julia and Lilia for sharing. I really enjoyed that.

[00:29:03] Julia: Thank you, thanks so much for having us.