Connecting the Dots in Product Design

Alkistis Mavroeidi at UXDX USA. Video: https://youtu.be/gSCuyl7kTcU

Readable transcript: edited from the recording's captions for readability (fillers and false starts removed, punctuation and section headings added). Wording is the speaker's own. Timestamps are positions in the video. Names marked [?] could not be verified against the audio.

KAYAK, and why problem definition is the hard part

[00:00:00] Very excited to be back on the UXDX stage. My name is Alkistis M. I'm a director of product design at KAYAK. I've been at KAYAK for seven years and a day, actually yesterday was my work anniversary. And today I'm going to talk to you about what I think is one of the most difficult parts of the product development process, which is connecting the dots in problem definition.

[00:00:23] But before, let me tell you a little bit about our wonderful product and our wonderful team. KAYAK started in 2004 with a group of technologists and travelers who wanted to solve one of the most complicated things about traveling, which is how to search and compare flights. If you think about it, before KAYAK existed, if you wanted to find a flight you just had to check every airline or every OTA, online travel agent, like our friends from Priceline. You had to go through all of them to find the one that actually works for you. And they wanted to solve that for themselves and for the people who use our product. So 20 years later we are a global brand in more than 60 markets and seven travel brands.

[00:01:12] The travel industry I would say is one of the most complex industries out there. There are so many diverse stakeholders, there are so many changing parameters, from environmental, regulatory, economical. And of course there's the increasing high standards that our users have for personalized and for seamless experiences that we need to offer. But despite that, despite being in a complex environment and in a company that's been a mature product and team for 20 years, we're still very passionate about innovation and we're a team that's lean and fast.

[00:01:56] This is just a shameless plug of some of our awesome features. Check out our most recent product release that we did earlier this year, and how we're using AI to help people solve some of these problems. If you test those out, please find us, we have a booth right outside, talk to us, we would love to hear what you think.

[00:02:13] But I'm not here today to talk about these things. I'm here today to talk to you about what I think is one of the most critical parts of product development. So the question that I'm asking here is, how do we, as a company of 20 years with a mature product, still retain speed and flexibility? Part of the answer to that question is that we're always relooking at our processes, and we're always trying to see the parts of our processes that slow us down and cause a lot of disruption, and trying to optimize on those. So looking at what's happening right now in our teams developing product is very important for us to ensure that we retain speed.

Signals that your problem definition is broken

[00:02:57] This is our product development process. This should be familiar to all of you, and you should also notice that it's portrayed as a linear product development process, but it's actually never linear. There's always back and forth, and there's always disruptions and things that happen along the way that make us either have to pivot or restart some of the parts of the development process.

[00:03:20] Where have we identified that most of these issues come from? It's the problem definition. The problem definition usually ends up creating some of the reasons why the project might succeed or might fail. One of the things that we'll have to note here is that it's critical not just for the development of the project, but also for the success of the team, because the more disruptions you have along the way, the more likely you are to cause time to be wasted, teams to be whiplashed, and also burnout through the team. So always you have to make sure you put appropriate time on your problem definition.

[00:04:06] So how might you know that maybe you, or your team, or your organization is struggling with some parts of the product definition? Here are some signals to look out for.

[00:04:19] One, you might be spending excessive time in discovery mode, sifting through a lot of insights and data trying to make sure that you have your problem definition defined.

[00:04:30] Two, you might be struggling to distill your projects into a concise and compelling narrative, making it very challenging to communicate your project's goal effectively.

[00:04:42] And third one, you might have difficulty prioritizing your list of opportunities out of a big list of identified issues that you want to fix.

[00:04:52] So if you are facing any of these signals, the good news is that you're probably not alone, a lot of teams struggle with this. So today we are going to share with you some of the learnings that we had out of trying to resolve this, under the theme of connecting the dots. We're going to have three areas of focus: our research, our data, and our communications.

Case one: research that ends in a thorough deck and no answer

[00:05:24] So let me start with one of our first cases for KAYAK. At KAYAK our product design team does a lot of our UX research, and we love talking to our users, we love getting as many insights as we can from those sessions. But in the past year or so we found ourselves in this situation where we're being presented a very thorough deck with a lot of data, and at the end of it we're kind of left with the question: okay, what does this actually mean, what is the main learning, and what does it actually mean for the project that we're currently working on? And it's unclear what the answer to that is.

[00:06:04] So what might be some reasons why this is happening? Here's some things to ask yourself. Do you have a scope of your research that's way too wide? Some of the reasons why this would happen is that you're just excited to talk to users and you want to get as much out of it as you can from each session. Or you might have stakeholders who are curious about something and they want to throw in a few questions just because it doesn't hurt. But the problem is that then you're getting into difficulty in actually getting the answer to the problem that you set out to answer.

[00:06:42] Another issue that might come up is that you might be focusing on gathering data rather than insights and patterns. So we're going to talk about some learnings that we have to make sure that that doesn't happen.

[00:06:54] Learning number one. This is about making sure that you're disciplined when you're running research. Always write down your goals and hypotheses. And while that seems basic, we often actually skip that step and we go into the research plan, what questions we're going to ask and what we're trying to get, the script, making sure all the technical parts are fixed. But the goals and hypotheses are a huge indicator of whether your research plan is actually set up correctly.

[00:07:23] So if you have more than two goals, or a maximum of five hypotheses, you should consider breaking down your research into different sessions. That is because your goals might actually be contradicting in terms of what you're actually trying to learn. Is it a usability question, or is it a behavioral question, or market research? Which means that you can't select the right methodology, or you might have to choose a different number of participants. So it's very important to write down what you're actually trying to get out of this test, to ensure that you're running it the best way.

[00:08:01] This is also important when it comes to qualitative research. You should try to reduce the number of participants to make sure that you get to the point where you're getting insights and not necessarily data. So your research is valuable even without big numbers, because what you're getting out of it is not statistical significance, it's insights. If you then need to go and validate in different ways you can do that with different methodologies, you can do that with surveys, you can do that with testing, you can do that with data. But your insights can still come out of a small number of participants.

[00:08:35] One of the tools that we use to ensure that we're running the right amount of users per session is the Pareto principle, which some of you might know as the 80/20 rule. That says that 20% of your efforts actually have 80% of the impact. So if you start going above the curve of the 80% of impact, then you're just using a lot of your time without getting as much impact from your research as possible.

[00:09:05] This one is a little bit more about the methodology of making sure that you have insights and not data when you're presenting your results. We call this processing vertically and horizontally, which means that you're processing each of your sessions as one whole thing. You summarize it, which means that you summarize the user's behaviors, characteristics, things that are notable about that specific session, and then you go across and compare the answers for all your sessions. That is a great methodology for creating patterns. Because for example, if you have the answer to a question that says two out of five people said that the site is easy to use, that doesn't really tell you anything. But if then you start connecting the patterns and say, oh, those two users actually have a common characteristic, let's say they're very budget conscious, then you start getting an insight that you can chase and see what it actually means.

[00:10:06] And finally, this is also a thing about discipline in research, is to avoid distractions. Again, we love talking to our users and every session is so valuable, so it might contain tidbits that you want to share with the team or you want to use, but they're not actually answering the question that you set out to answer. It's important to remove those distractions from your test, and ideally you have a system for saving them so you can go back to them when you're investigating that specific subject. So that system could be tags, could be a repository of insights, could be something that you can go and look back to. But don't fill up the rest of your deck just picking up all those little insights that I agree are very, very valuable and enchanting to go through, but you have to focus on the thing that you set out to do, which is your specific research question.

[00:11:01] So if I want you to take one thing about this, it's that it's very important to practice discipline when you're running your research projects.

Case two: the hunt for the one perfect metric

[00:11:15] All right, let's move on to KAYAK case number two. At KAYAK we love numbers, we love to ensure that we have an A/B testing culture, so we like to ensure that we A/B test most or all of our projects. We have internal and external tools that we use to track our metrics, so it's a conversation that we always have.

[00:11:36] This is an example of a conversation about a new metric, which is: how do we measure the quality of our UX? We started discussing different ways of doing that. One of them was the number of UX bugs that were reported, so we can track that, but that doesn't actually tell us if they've been resolved. So then we add onto it, okay, what if we track the number of bugs that were resolved? Which is great, but also creates a dependency on another team, and it still doesn't show us where we're at if we've made the site so great that we don't have a lot of UX bugs to resolve. So then we talked about, what if we do heuristic audits? That actually shows us the state of the site and that way we can benchmark it the next time. And there were other ideas as well. The point I'm trying to make is that there isn't really this one perfect metric that you can chase and that can always tell you the full story.

[00:12:39] So what I would love for you to ask yourself when you're looking at data and trying to connect the dots is: are you trying to find the one perfect metric to tell you if your goal is achieved? And are you torturing data to tell you a story?

[00:12:57] That's a famous quote by Ronald Coase that says that if you torture data long enough, it will tell you any story that you want. So you have to make sure that you don't start from what you're trying to prove and kind of switch through data, does that tell that story, does that tell that story. And that has happened, we've seen it happen, where we're pulling the data to try to prove the hypothesis that we have. But you have to have multiple indicators to tell you where you're going.

[00:13:29] So, some learnings for us. Continuously ask yourself what you're trying to solve for and focus on that first, and try to figure out what the gap is. The problem you're trying to solve means that you're trying to get from one place to another, so that is the gap that you're trying to measure. And that gap, already just by defining it like that, you can think of it as indicators that show you that you're in one place versus the other. So those are the indicators that you should look for and start measuring in order to see where you're going.

[00:14:01] And the second one is what I already mentioned, that those are your indicators and it's not your end goal. So one other story that I can say here in terms of where you are and where you're trying to go is that we had a team at KAYAK that was very metrics focused, and there were issues in the UX that we were trying to solve that were coming in qualitatively. But often we got the answer, we don't see that in the metrics, so we don't have a problem. However, people could see it, even internally, unofficially, that feedback was coming in that there's a lot of friction in that. So that's the problem with having metrics that are your only north star, and following those: they can blind you to what the outcome that you're chasing actually is.

[00:14:52] So the main learning that I want to leave you with on this is another famous quote, Goodhart's law: when a metric becomes a goal, it stops being a good metric. And that doesn't mean that we don't love metrics, we love them, but they're not the goal, they're the indicators that help you get where you want to go.

Case three: communication, and the one pager

[00:15:12] All right, let's go to KAYAK case number three, and the final example is about communication, which I think is very, very critical. I would love to know how many of you have been in this case where you shared your very, very thorough decks and the solutions and briefs, no one gave feedback, and then when you start working on something further down the line people are like, why are you working on this, you shouldn't be, change this, change that. Is that a familiar struggle with anyone here? That's great.

[00:15:48] So let's see when that's happening. And I'll tell you another example before I go into this. Another example that we had happen is we're having leadership come and ask for one pagers of long and complex briefs. When I took upon that task to summarize some of our briefs and some of the problem definitions that we had, I went into it with a mindset that I was catering for the needs of people who didn't have enough time to go through long briefs. So I thought I was catering for the lack of time. But actually, after spending a lot of time on long briefs in projects that I was familiar with, that I knew were a lot more straightforward, I realized that the more details we were adding, the more vulnerable we are to actually introducing contradictions, or losing sight of what we're actually trying to achieve. So the one pager wasn't really just a tool for executive leadership, for example, to go through it fast. It was actually a tool for us to make sure that we haven't lost sight of what the goal is.

[00:16:59] So, questions to ask yourself when it comes to communications: are you hindering communications, and are you getting lost in the details?

[00:17:09] Here's a couple of learnings from us when it comes to these kinds of indicators. The first one is to cater for your audience upwards, downwards and laterally. That means, with the example that I said, that upwards you might have to have a summary of the project that you're working on, and that's very helpful to you also to make sure that you keep sight of the big picture. Downwards it might mean that you get into the details, you ensure you have all the edge cases. And laterally could mean that you're handling all of the dependencies with the other teams and making sure you expose them.

[00:17:44] As an individual contributor this is very important for you to do, because then you are enabling people to give you feedback at the right time. So if you have an indication maybe that your manager is spending a lot of time trying to understand what your project is about, that means that you should do the exercise of summarizing it. And as a manager you should also feel comfortable asking for that. You're not being difficult by asking for the executive summary, you're just protecting your perspective, because if you go in the weeds then you have the same perspective as the team that's working on it. You are the one that can be able to connect the dots across that level of strategy or the cross functional teams that you're working with. So it's important to ask for it if you need it.

[00:18:33] And the final learning for this part is to keep the big picture in mind when it comes to prioritization. I'm going to share another story on this, which is a project that we're working on, and I'm going to use a metaphor here because I think it's going to be easy to understand.

[00:18:49] Let's say we were fixing a house, and we're in the house and it's a mess, we're trying to clean it up and there's just so many things going on. And then someone from our leadership team comes in, sees us cleaning the floor, washing the dishes, and the roof is falling. Have you noticed that there's no roof? We're like, yeah, but there's all these things that we need to fix, we've done this audit, it's so much friction. And it doesn't matter, because if you don't fix the roof then the flood's going to come, it's going to wash out everything, so it doesn't matter.

[00:19:22] But this is an actual case where we had to convince the team to drop what they were doing and fix a roof, because it just felt very important. You're very close to it, you've seen the user interviews, you see how people struggle with it, you want to fix it. However, having the big picture inside can help you prioritize some of these tasks that need to be fixed first before you move into those details.

[00:19:48] So what I would love to leave you with on communications is that empowering communication is one of your strongest tools to make sure that your project direction is correct. And also, when you enable feedback you create quality checks for your product. So feedback will come, so it's to your advantage that you make sure that the feedback comes in timely.

Summary

[00:20:15] All right, so to summarize. Connecting the dots is one of the most important things that happen in the product problem definition. Problem definition is very, very hard, because when you're working in the weeds it's really hard to zoom out and see the big picture, and you have to ensure that you're handling the right scope and that you're taking into consideration the right data.

[00:20:45] So connecting the dots is a collection of strategies to make sure that the designers are able to see the big picture while working on their problem definition. And also, especially for mature products, it's a critical lever to make sure that you protect your speed, your flexibility and the quality of your product.

[00:21:11] We shared some of our learnings from problem definition. On research, which is that it's very important to practice research with discipline. On data, which is to focus on the outcome and not just the metric. And on communication, which is that empowering communications helps you keep sight of the big picture.

[00:21:33] So how can you apply this? Always check that your problem is well defined. Scope your research wisely. Identify the gap that you're trying to solve for, and find your indicators that help you understand that you're moving forward. And communicate concisely and effectively. So, hope that was helpful. Stay in touch, and thank you very much.

Q&A

[00:22:07] Host: Alkistis, that was fantastic. All right, you know, maybe I'm a cheap date, but you had me at "how might we." Just a great way to frame the presentation. There always has to be a "how might we" there.

[00:22:23] Alkistis: Yeah, you know your audience, right.

[00:22:26] Host: I also really enjoyed the vertical, horizontal that you had laid out there as well.

[00:22:35] Alkistis: Great, yeah, I think that's a very effective strategy, especially when you can actually display it visually. So when you have a tool that you can put your insights in and kind of go through them in that manner, it really helps connect what your data is with the patterns that you're seeing. So you're not just presenting three out of five people said this, you're actually picking up the insights that you need.

[00:22:59] Host: Yeah, it's a bit like root cause identification, right, which I think is somewhat in line with your connect the dots there. You could have taken that metaphor in a lot of different directions, right. So it looks like we have a question that starts with "love the presentation," so we're off to the races. What would you say are the key components of a one pager?

[00:23:23] Alkistis: So the one pager, usually the first thing that you have is a problem definition. If you have that part, that's already a lot of work that you have to pull together, and it should be simple and straightforward, but you have to make sure that you're actually hitting the problem definition. But there's more things that you could add there. Some of it is the context that helps you give a little bit of the bigger picture of what you're working with. This might be a little controversial, but perhaps some people have proposed concepts, and this is a very dangerous part to go into, solutionized examples, but this is more about identifying the gap, where you are and where you're trying to get.

[00:24:13] Alkistis: And then that one pager could also include, it depends on how technical your project is, it could include your insights from a user perspective, it could include your dependencies, the things that you need to check next, and of course what is your expected impact. So after you have your problem definition and your context that shows where you are and where you want to go, then you need to ensure and talk about why you need to work on this, both from a user perspective and a business perspective.

[00:24:45] Host: Love it. Alkistis, I want to take us out of theory and into application here. There's an interesting question which I think we could all benefit from an answer on, which is, how do you deal with conflict when it comes to that problem definition? How might we deal with scenarios whereby product has a different point of view than development and/or design? How do you address that?

[00:25:16] Alkistis: I would ask why there is a conflict to begin with, because this could have different methods that you apply to it. For example, if you have a conflict because those teams are working in silos and they're not bringing their insights together to actually create that problem definition, I would start there. Why is your organization not collaborating in making sure those insights come from a combined effort?

[00:25:49] Alkistis: Then, for us, because we work very integrated with design and product, I think it becomes just a matter of making sure that if those teams are not convinced about it, maybe the problem is not validated enough. So then you need to take your problem definition into more validation. If your one pager is just not convincing, then you need to relook at it. So I'm coming at it from a perspective of those teams working together to actually develop some of these problem definitions, so I assume that you have experts from your product validating that this is actually a problem definition that makes sense. If there's disagreement there, you go back to it as a team and you resolve where these disagreements come from. So if that process is not happening, again, maybe it's an organizational issue where your siloed stakeholders should come together at an earlier stage.

[00:26:51] Host: So you talk it out.

[00:26:54] Alkistis: Well, if you can't talk it out, then you need to go and research it. You need to bring the data that actually shows that this is a real problem. I'm not here standing to just defend a problem that came out of just my imagination, right. Usually problem definition comes with data that will say that this is important to solve, or this is actually happening to the users of our product. So if you don't have that, where is the disagreement happening? Is it on the existence of this data, the validation of this data, or is it on whether this data actually means that that's the problem, right. So it depends on what state the conflict is in.

[00:27:38] Host: I love it. Unfortunately we only have like 27 seconds left and I don't think we have a question on here that's easily answered in 27 seconds. So, Alkistis, I just want to thank you again for your presentation and everything that you put together for us and for sharing your knowledge. Thank you very much.

[00:27:57] Alkistis: Thank you.