Coming Together to Build Exceptional Products Through Product Metrics Map

Jason Zhou, Jason Zhou at UXDX Community: Testing and Scaling the User Journey. Video: https://youtu.be/rInjYsO1XWw

Readable transcript: edited from the recording's captions for readability (fillers and false starts removed, punctuation and section headings added). Wording is the speaker's own. Timestamps are positions in the video. Names marked [?] could not be verified against the audio.

Product development as a relay race and a road trip

[00:00:01] Jason: Okay, good afternoon, everybody. My name is Jason, so welcome to "Coming Together to Build Exceptional Products Through Product Metrics Maps." A little bit about me: I'm a product designer, a UX designer at PayPal. I'm focused more on authentication and the digital ID experience at PayPal. Besides working at PayPal, I'm also working for the User Experience Professionals Association here in Austin. I'm also a mentor, really interested in and passionate about teaching experience design [?], and I also really like to cook.

[00:00:37] Right. Today my main question is: how do different types of people actually come together to work? I saw a problem in a big tech, fast-paced environment as a designer. As a designer, I used to believe that the responsibility for coming up with brilliant ideas and making the best decisions was my own responsibility. But as I gained more experience, I came to realize that product design is more like a relay race. Collaboration and strategy planning play a really important role in achieving success. By working together as a team and focusing on smart product strategy, we can create a better solution that genuinely meets the needs of the user.

[00:01:22] As I mentioned, product design is more like a relay race. Let's picture a relay race. This is what I experience every single week. My product partner and my engineering partner provide all these ideas to my design team and my content team. We all start working on these ideas, and after working on them, we pass them back to the product and engineering team. We do this every single week, but this kind of creates problems.

[00:01:52] Sometimes we also think that in product we're actually on a road trip together. Product, design, engineering, leadership, we're all coming together and working together to drive success and customer and user experience goals. So how I define product development is really as a journey for everyone to come and work together.

Different perspectives, hard agreements

[00:02:19] The problem is, as designers, we always think about what is the best way, the most efficient and user-friendly way, to get to the destination. We care about the customer experience. My product partners, I always feel, care about whether this can be delivered on time. They care more about how we can better deliver these products on time, and how we can utilize all the resources to get to the destination. And my engineering partners think more about whether this product can be achieved, whether it's doable. They're more realistic, and they ask more questions related to whether this can be achieved, not just through a design but also from a technical perspective.

[00:03:05] This brings a problem. We get a lot of ideas proposed from the product side, and sometimes we get into meetings that can be really inefficient. Sometimes I'm in meetings with 30 or 40 engineers and product managers. We talk about one thing, and after that we have no idea what is going on, and we don't have any agreement, and it's less efficient to work together. And sometimes we don't have any data or any reason to support the idea. So basically it's really hard to reach agreement together.

[00:03:38] So the current problem is: how can we achieve our goals collaboratively when everyone has a different perspective and ideas? More important is how we can work more efficiently to make decisions more easily and work toward a great direction for our products.

The product metrics map

[00:03:57] Then I started working, and I came up with some tools to help our team work together more efficiently. One of the tools is the product metrics map. For a product manager, the goal is to create a product that meets the needs of the customer and makes a successful business. To achieve this goal, you need to focus on the metrics that matter most for the product. That's where the product metrics map comes in. The product metrics map is a tool that helps us identify and prioritize the most critical metrics for our products. It provides a visual representation of the most important metrics, to help the team prioritize effort based on the impact each metric has on the product.

[00:04:50] The idea I came up with is not just from a design and engineering perspective; it's also coming from product and the customer experience. What you see here is that we as designers care more about whether the features we design are nice to have or definitely have to meet a need, need to have. This can be different based on your business need. For example, when we're designing features for phase one, we probably just want to set a goal that the designed features are need to have and take a lot of effort, to serve quadrant 2; probably the best way is we put a goal into that. But sometimes, when we're moving on and improving the idea, improving the design, optimizing the design, we want to design a feature more toward nice to have and also take less effort. So this can be customized based on your business goal and the product you're actually working on.

[00:05:50] For the customer experience, we definitely want to find the metrics that focus on the customer experience: designing a feature that has to be necessary for the customer, and also making sure that our users of the product aren't spending a lot of time, that they use low effort to accomplish their goal. And for product, of course, the product partners care about outcomes. They care about high KPIs, and they also want to spend less time achieving the deliverable. That's the goal they're striving for. So we always bring these three dimensions to the table to discuss all together, which is really important.

[00:06:33] There are five steps you can do. First, identify the problems and the key metrics: what is the problem we are trying to solve, and what goal do we actually want to achieve? Step two is evaluate the impact, from the design perspective, product perspective and engineering perspective. We want to make sure what the product outcome is and what impact this product can make. Step three is assess the effort for the requirement. Design effort, engineering effort and product effort may differ, so we want to make sure the metrics can be clear and easily represent all this effort together. Step four is you actually start putting these ideas into the scatter plot, this matrix, and prioritize all the work using the metrics.

[00:07:17] Here's an example I used with the team. For example, this is the goal I have for the product part. We put all this together from a product perspective: what features we think can be really high impact and actually take less time for everyone. This is an example from when we worked on a verification flow. We wanted to make sure users can verify their ID easily, so we were thinking about a lot of ideas for how we could improve the experience. This is one where a lot of product partners said let's probably add a progress bar to that. But from a design perspective, it's maybe a little bit time consuming, and it takes so much time for engineers to work on. So that's why we use this to keep products aware of ideas, in order to find the best goal we actually want to achieve.

[00:08:18] From the customer perspective, we also always think about whether the feature we design is essential to the customer's need, whether the customer feels like this is something they actually need. We always bring the user testing data and rapid research data to actually help us prioritize the metrics from the customer's perspective.

A design roadmap

[00:08:41] Oops. Okay. Another one I want to bring in is the design roadmap. When we get a product metrics map, and we have this exercise together, it's really important we get an outcome about how we can track this data together. I always believe every designer needs a roadmap to keep tracking their tasks. This is what I created for myself to use. I bring these goals together into a roadmap and I separate them into different phases. If an idea is more at the discovery phase, I put it in the discovery phase. If I keep working on the idea into a design phase or a prototype testing phase, I'm moving the idea forward, so I can keep tracking my progress and also keep updating my product manager on what is going on for this project. So we are not losing ideas, but we also keep bringing new ideas to the table.

[00:09:44] So why do we have a roadmap for design? Like what I said, we need better processes to test together. We need to find a way to enhance communication and keep tracking progress, in order to help designers achieve their success and bring this to product and engineering as well.

Communication strategy

[00:10:04] The last one I want to talk about is the communication strategy. We've got a product metrics map, and as a designer we've got design roadmaps. The most important thing in working together is how we communicate together, because we all have different perspectives. How we actually work together is really, really important. Communication takes more than 80% of my workflow daily. I communicate with my different stakeholders every day, every week. I think it's really important to focus not just on having tools to use, but on how you can actually present your idea and share ideas with your stakeholders, which involves some technique as well.

[00:10:48] The first tip I want to share is that you definitely have to have trust with your product partners, trust with your engineering partners, with all the other stakeholders. Trust is really important, because we know everyone has a different perspective. People share their ideas, and people can disagree, but you must trust your design team, you must trust your product team and engineering team that they can do the work in the domain they're best at.

[00:11:20] The next is make sure you clarify the problem you're trying to solve. Sometimes, when we work on these ideas, some people are afraid to ask questions in the team. They feel like this is something the product partner set, or something engineers set, that we have to follow. But sometimes that's without understanding what core problem we are solving, and this is actually a problem. From my perspective, if I don't know what the core problem is, it's hard for me to design a feature that really solves the problem. So clarifying the problem is really important in the whole product process, and make sure it's part of your product communication as well.

[00:12:01] Here's how you can get started. This is basically what I used to do. I'd have a lot of conversations with my product partners, and I'd get an idea: "We want to share this with you, can you do this?" I used to say, "Okay, yes, let's do it." I started working on design, I started working on a low-fidelity prototype, started doing testing, but eventually found out, okay, this is not something that will really help us achieve our business goal. The new way is you can actually talk to the product partner and say, "Okay, how about we do a quick exercise session to work on this kind of product metrics exercise, so we can better understand what goal we're trying to achieve and what problem we're trying to solve?" And you can involve your engineering partner as well. In this process, the engineers might give you a lot of feedback and say, "Okay, can you do more of that? Can you iterate more on this idea?" Obviously this is pretty normal, because you may not be able to get to a goal easily. That's why continuing to iterate on the idea is really, really important when you come to work with other people at the table.

[00:13:10] Here is the strategy I'm trying to summarize for this conversation. First, make sure you have everything to support decision making. No matter whether you use product management tools or other tools you work with together, make sure you have data to support the idea. You have the research data, you have the engineering data, and it can really help you make valuable decisions. Second is to utilize these tools in different directions together. I used to have more than ten meetings a week just to talk about one problem. Right now, if I use one of those kinds of tools, I can probably solve 10 problems in one meeting. So it's really important for all people to try it and experience it.

[00:13:57] Another one is to make sure you keep iterating your ideas. Ideas definitely take time to iterate. You need time to think. You may have this idea, but later on you may have more ideas coming through, or you may keep improving the idea. That's why I think the better experience is that you have to keep involving these techniques with your team.

[00:14:20] Finally, I would say a good strategy plus good collaboration tools really equals efficient work. As I mentioned, I used to have more than 10 meetings a day, but when I started using this kind of tool, I really could increase my efficiency working with my product partners and my engineering partners. I think it's really great. I feel like my goals get clearer, I have data support, and I also make sure my ideas can be efficiently presented to my engineering and product partners at the table as well.

[00:14:57] That's the key piece for this topic: not just focusing on working with those tools, but also focusing on how you actually use them to keep track of progress and keep your team motivated toward a good business strategy. So this is about the product metrics map, and a couple of tips I'd love to share with you all to help you increase your working efficiency with your product partners and your other stakeholders. Yeah, thank you very much. Waiting to get any questions if you have them.

Q&A

[00:15:37] Host: Great, thank you very much for sharing. It's definitely a lot better than the graphical kind of presentation that I could do, so it shows that you're a designer. I loved the transitions and the whales. It also communicated the message that you were trying to get across.

[00:15:55] Just a reminder, if you have any questions. I saw Carlos messaged in just before, asking for the slides from the last talk, so we'll chase those up and share them with you once we can. But if you have any questions for Jason, please do write them in whichever platform you're in and we'll put those through to Jason.

[00:16:16] I'm going to kick it off, though, because there was one question, particularly with those maps at the start that you were sharing, the cost-benefit weighing up. One of the challenges I've always found is that it's quite easy to convince ourselves that there is a benefit somewhere where there might not be, or to convince ourselves that this could be quite easy to build, and then it turns out not to be. Do you do anything where you go back and validate, to see whether we are actually being accurate with our upfront estimates?

[00:16:54] Jason: Yes. I think the tool is really important, but sometimes we still come to the idea from our own perspective: this is something I think is really going to improve our user experience a lot. But eventually we feel like it has less data support. So I always bring the user testing data and research into these conversations as well, because they have the data to support my design decisions. Then I can bring the idea to my product partner and my engineering partners, and they know, okay, the idea I'm giving you is not just based on my own perspective. It's based on validated data, user testing, rapid research. So they accept the idea: okay, this is something that's already been validated, so we can consider it more, we can take it more seriously.

[00:17:42] That also involves so much more communication between different cross-functional teams. But that's what I like about using these tools, because they help you push your idea forward. So it's not just a single idea anymore. It makes sure your ideas make sense, and also makes other people better able to hear from you.

[00:18:08] Host: That's great. I'm going to put in the question that I've experienced many times, because I love that answer of going with the data, going with what the customer is saying. But have you experienced the position where somebody just comes with an idea, and, speaking to a lot of designers, they would love some type of research: can I just go validate that, can I do whatever? As you were saying, get that clarity about what it is that we're trying to build. But the answer is, "Oh, this is a sure thing. It's definitely right. Just go ahead and build it." Have you had that experience, and how do you handle it?

[00:18:49] Jason: Yeah, I mean, that basically happens every week. We get an idea, not just from product but from others as well, saying, "Okay, this is the data we have. I think this is a really good way we can do it." I think it's always a balance. As a product designer, you get ideas from all these people: your team, other people, other stakeholders as well. I think the most important thing is that you as a designer have to understand what core problem you are solving for, and use data to support the idea.

[00:19:23] Sometimes we do user testing and we use rapid research, and we get data, but we feel like we want to test more. We feel like this data cannot really support moving forward with all of this idea together. So let's maybe just do phase one. Let's do it one by one. Phase one, let's design a little bit for this, and then do another round of testing to make sure this is something we really want to go for. That's what I always like to do: I agree with you, yes, of course it's a good idea, but let's not take it all the way forward at once. Let's do it one by one. Let's split this idea into phase one, phase two, phase three, and test it. Maybe in phase two or three you find the problem is different from what you found in phase one, so you have to keep iterating on the idea. I think that's always the question for everybody: how you can balance the idea and make sure it fits what you need and also the business goal. So I think that's really, really important as well.

[00:20:25] Host: Great. I'm going to keep throwing the tough questions at you, because I'm fully on board with exactly what you said: let's iterate, let's try something small, let's validate, and then invest. Have you ever encountered, again back to that product, UX, dev, or product, design, dev, some people saying, "Let's stop with this. It's inefficient, it's really slow, it costs too much. Just tell me what we need to build. I don't want that uber-expensive, or perceived expensive, testing. Just give me the full spec, because it's too long and inefficient to go through these steps." And then what did you do?

[00:21:07] Jason: Yeah, I would say this happens a lot: really inefficient meetings, inefficient communication. Every single time I receive a lot of PowerPoint decks from product, I can't understand what is actually going on, because they put in so much data, so many numbers, that it's so hard to comprehend and really understand what the core problem is. So I think that's the skill I would encourage all designers to build more. Not only are you a designer, but also think through the data. Analyze the data, analyze all these conversations, to prove yourself. And sometimes you have to be a product manager. You have to help your product manager get this perspective going.

[00:22:05] So focus on the things that matter the most. Sometimes you do get a lot of inefficient meetings. Ask yourself: do you actually need to go to this meeting? Do you actually need to cover this conversation with a crowd? Or can you use a better tool, like what I shared, to iterate on this idea and work with other people? I think that's super, super important. Don't just focus on "I'm a designer, I'm a UX designer, I just focus on user experience." If you want to make it really inefficient, you just work in your own lane. But for me, I have to keep thinking: okay, if I'm the product manager, how do I fix this problem, or how do I make sure I deliver this efficiently to my team? So switching perspective and understanding what you actually need to achieve is the key, I would say, especially in this fast-paced tech environment.

[00:23:02] Host: No, I think that's a fantastic answer. I love the way your presentation continued, in my mind, from Swapna's [?], where it's: is it product, is it marketing, is it sales? But no, it's collaboration. I always find there's never really one soul running, and that's the same with product, design and dev. It's the collaboration that really counts.

[00:23:25] Jason: Exactly.

[00:23:34] Host: Great. Well, unless anybody has other questions, I see Carlos is just saying thanks. But if there are no other questions, I might just say thank you very much for sharing your insights.

[00:23:45] Jason: Of course. I really loved sharing this with UXDX. Thank you very much. We appreciate everybody's time.