Keeping Your Eyes On The Business Goal

Raquel Damas at UXDX Community: Amsterdam. Video: https://youtu.be/87811b_dP1g

Readable transcript: edited from the recording's captions for readability (fillers and false starts removed, punctuation and section headings added). Wording is the speaker's own. Timestamps are positions in the video. Names marked [?] could not be verified against the audio.

What does better mean

[00:00:11] I'll talk a little bit about how research and monitoring can help us get products to market. But if we don't scope it well, it can be more of a roadblock, and just leave us not knowing what to do with the amount of data. I'm going to start with this word, better. Because as designers and developers and product owners, we often say that we want to develop a better product. It's like the Miss Universe type of answer: "Yeah, we just want to do things better." But that's a trap that we set for ourselves. That's a subjective value that brings nothing to the table or to a team when you're trying to conceptualize what to do.

[00:00:50] You need to create alignment and you need to create purpose, and you need to define a set of shared values by which you can create something that's meaningful and impactful. A better product brings a benefit both for the user and also for the company that defines it. Design is a function within a company. We often tend to go, "Oh, we're designing user experiences, we are user-centric," and sure, users need to be at the heart of what you're doing. But if what the user says is better for them is not aligned with what your company believes to be better for its own growth, it has no value.

[00:01:30] So before you take any decision about a product, and what you're going to do, and how you're going to go about it, and what research you want to do, you have to define what your outcome is. What is that better? And you have to say it in a tangible way so people understand. The reason why you need to do that is so you can evaluate it and know whether you're succeeding with it or not. So how the heck, then, are you going to know whether you're making something better?

Make it tangible: the account sign-up example

[00:01:58] I remember when I joined Shopify, two and a half months ago, so I'm fairly recent at Shopify. I joined the team that is working on our wholesale storefront. There was a sprint going and design work going, and I asked them, "What are we working on?" "Oh, we're making account sign-up better." Oh. I don't even know what account sign-up is, and I don't know how you want to make it better. Why is it better than it is now, and what are we improving? This doesn't say anything. So I had to ask the team to just write it for me. Make it tangible. Give me a word, a concept, whatever it is that is driving your design, and the effort and the money that Shopify is putting into us as a team to make it better.

[00:02:46] Because the moment you make it tangible, you can define your metrics and your key performance indicators, and those are essential for whatever experience it is that you're trying to build. Without those, you don't know what you're doing, and you cannot show the company that employs you that you are bringing value and impact. At the end of the day, that's what we are hired for: to help them grow. And the moment you have those, you know exactly what to study once you launch, to understand: am I really empowering the user? Am I really setting myself up for success, or am I failing the user and creating something super complicated that they don't need?

[00:03:29] That's how research and monitoring work for you in an agile process. Research will bring you the truth that you need to stand by to build your hypothesis, a successful one, because there's money involved. You can say whatever based on your beliefs, but research will inform you. And monitoring, once you launch, is going to tell you, "Dude, you're doing great," or, guess what, you're not. So change: revise your vision, revise your concept, readjust your strategy, so you can make a customer happier and help your company to grow. Because better companies make money out of happy customers. At the end of the day, that's what we work for.

Context is more than the user

[00:04:14] This sounds fairly easy. You think, "Okay, I have my goal. It's well written, it's very clear for everyone in the team. I have my KPIs, I know what I want to do, I know this is a loop process going on and on." Now, success is not easy or straightforward if you don't have this: you need to gather context. I think about myself when I joined Shopify. "Oh yeah, sure, let's make account sign-ups easier, allow customization, all those things." I learned about the user. I talked to our merchant success managers, who told me everything they're asking for. I understood the product a little bit. But I actually didn't know anything about Shopify.

[00:05:06] Context is way beyond what your user is doing and needs and wants, and what your product does and can do, and what the competitors do. Context is also about the company that you work for. This is part of your research, and this is how you can scope your research to be useful to you. This context gives you power. It means that you're not replicating work, you're not duplicating the efforts of different teams. You're tuned in to any component that might be being developed in another product that can be of use to you, and you gather insights from other work streams so you can be more informed.

[00:05:48] It gives you control, and control of time, and time is everything that you don't have in the digital market. Because there are new companies starting every day, because there are tons of heads thinking about a way to be disruptive. And you need that time. Digital products are like a speed train, a bullet train, and either you're trying to run to catch that train, or you are at the same pace. So what do you think context is? How do you gather context? How would you define it?

A framework for shared knowledge

[00:06:23] At Shopify we call this a framework for shared knowledge. We have a vault that has all the information on everything happening in the company, and we are 3,000 employees, and there are tons of products running at the same time. It's a framework that helps everyone developing products to start to research them. And it involves everyone that at some point will make a decision about your product. We think, as product developers, that it's only the designers, the developers, the engineers, the product managers, but it's not. You have marketing, you have sales, you have support, and at some point they are going to make a decision about your product, because they are in between the product and the user. They're going to define the product for you as they talk to users.

[00:07:15] So you need to involve them when you're trying to learn and to scope your research, and to define what you're going to do and learn about, and what type of research you're going to do, even. They need to be there at the table with you, scoping the intention of your product and this first release that you're going to have, or the second or third, it doesn't matter. You're going to be releasing forever, as long as you have a product in the market, because you're going to make it better all the time. It never stops.

[00:07:44] Involving the team brings higher levels of engagement. When you have a developer, even a junior developer, being part of the decision of what we are doing and why we are doing it, it means that when the work gets to him, he's not going to be changing it, tailoring it, making little tweaks, because he was part of the decision on why we went down that way. Because you have so much knowledge from different areas. People in support: if you ever have the chance to listen in and shadow some support calls, where they're helping clarify any doubts that your users have about your product, it's so interesting, the things that you can learn. Support people have a lot of knowledge about your product in a way that you don't as the designer, as the creator, because they're there every day dealing with the users. So do merchant success managers, or account managers, however you call them.

[00:08:51] All the diverse knowledge that you can gather by having them at the table from day one also means that the time that you're going to spend doing research is way smaller, because there are so many things being shared that inform you from moment zero. And time is, again, what we don't have, and time for a company is highly costly. Even from the idea that if you take too long doing research, by the moment you build a product, it might not be relevant anymore.

The year and a half that killed a product

[00:09:26] We have those failure cases at Shopify. We were basing things on information like, "Yeah, that would be a nice-to-have. Oh, I would like that." But doing the research and building something took a year and a half. And when we launched, the complexity of the product, of that nice-to-have, was so big that with our merchants, feature adoption didn't happen. They didn't buy the product. And we killed, in terms of cost to the company, a year and a half. That's a lot, in researchers and in the things that you didn't do because you were doing that.

[00:10:04] So sharing all the knowledge from day one helps you define what the priorities are. It also means that research is not done only by researchers. Because everyone is there learning what has to be uncovered and what has to be learned, it means that every single one of you, including me, can do research. I'm doing research tomorrow. I'm not a researcher, I'm a UX person, a design person, but I'm here in Amsterdam and we have partners in Amsterdam and we have merchants in Amsterdam. So I'm going to sit with them and listen to them, and I brought some prototypes to show them, just to gather some feedback. We all can do that work. We don't need to say, "Hey, you, researcher, just go and do that, I need to know more about this." No. They help us build the scripts, because they know the right tools and the right methods. They guide our research. But at the end of the day, it's on every one of us to be part of it.

[00:11:01] And because you are part of it, it means that you don't have to spend time then selling whatever value you're trying to build and what the learnings are. There's an easy buy-in; everyone's on the same page. I'm not spending my time writing emails to the stakeholders and to the other teams, like, "Hey, this is what I want to do," and then with the VP, I need to wait five weeks, he's going to read it at some point, and nothing happens during five weeks. Everyone's on the same page. It's easy. And the time that I'm not spending doing that is time that I have to speed my product to market, and to scope things that we need to do next, and to understand how to improve them. And a market that I'm most likely to fit and be ready for, because all those other teams and team members told me whether I needed to learn the why or the what.

Learning the why or the what

[00:11:59] These are different things, and this shapes the strategy of your research very differently. Either you're trying to learn about the user's life, creating empathy for all that context, understanding the problems they go through, how they live, identifying gaps and opportunities. Or you focus on facts: what is actually happening, what are the friction points, what are they doing, clicking, going through. This is quality, that's quantity. Throughout product development you're going to do both, and at different moments in time you're going to learn both. But the thing is, you need to know what you're trying to learn to ask a researcher to help you build a script and put that strategy together. Otherwise, you can be researching for a year. There are so many things to learn all the time, but you need to learn in stages.

[00:13:00] At some point... Sorry, technical problems.

[00:13:14] Host: You're still working on the laptop screen, if that helps.

[00:13:19] There you go. I must have clicked too many times. So we came from context, and context is everything in the digital world. Context is what allows you to achieve success faster, steadily, in a managed way, because it includes what a full team believes to be true as your scope of work, and a way to quantify those assumptions. It's a shared definition by everyone that's going to be part of the project, and that's context. If you do that, you're probably 80% on the right track, and that's good. There are fewer unknowns. There are still unknowns, there are assumptions, and you're going to have them, as has been said already today.

Assumptions, risk and where to start

[00:14:05] The problem with assumptions is they represent risk. They represent the possibility of failure. I'm not happy with the idea, three months down the line in my new job, of failing at something, because they're going to fire me, I don't know, maybe I'm not good for the job. No one likes this idea. Our legs shake a little bit at the idea that somehow we're going down a path that might not be as successful as expected. But be okay with it. Because every assumption, or every failure, is an opportunity for you to learn. And if you try to control everything, if you try to back up every single decision, you won't get anything done. If you try to be too perfect, too assured, it won't happen. And again, you're missing the train of disruption, and your products will take really long to get to market.

[00:15:01] How we control this at Shopify is we look at the things that we don't know and we put them in a quadrant between risk and likelihood of happening. This was also said here today already, and it's quite true, because it helps you a lot. If it's something that's highly risky, like it can completely kill my brand, users stop using it and it totally damages what I'm trying to do, and it's very, very likely to happen, then, oh yeah, you need research. You need to validate that. You need to be fully assured that you're ready for that thing. And it doesn't matter if it's from a technology standpoint or a user standpoint, whatever it is that can happen that you don't know, but you believe will take time to know as well, then you will need to put effort into that.

[00:15:57] But is that the way you should start, and where you should start? There are other things that are less likely to happen that can still hurt you, but where you can easily do damage control. It's something you can easily iterate on, launch a fix, whatever that is. And if they're low risk, in the sense of, "Okay, if it happens, it happens, and we deal with it in an easy way," then that's a safe area to start. Because you're giving yourself the opportunity to learn about something that you didn't know, and that is power, again, to you.

[00:16:31] Another way of looking at this is that whatever is safe probably represents low cost for your company. And again, cost in technology, research, time. If you can learn a lot about it just by deploying a simple, small outcome, a simple, small feature, if it can bring you the most learning, then that's where you should start, because there's a lot of value there. Because in the digital world, at the fast pace that we change, and everything goes crazy and everyone is asking you when is the next feature. "There's this competitor that is doing this, and now Adobe bought Magento, what is Shopify going to do?" There's so much pressure from all the angles. You want to start where you can learn the most, so your next iteration brings a lot of value. It's about less guessing and more knowledge. That's how research can help you, because you want to build for learning. You want to embrace failure. You need to see failure as a way to feed more research. And the more that you learn, the more powerful you are, because you know how to help your users and make them happy.

Scoping the research: user or product, generative or evaluative

[00:17:45] So you have to be intentional in every step, and that also includes scoping research and defining what type of research I am doing and what I am researching. There are two big pillars. We could go over tons of methods to do research, but you can also go on Medium and online and on Google and they will show up. The value of research is not necessarily applying all the tools all the time. It's understanding what methods and tools I need to apply to help me deploy faster now, on this scope of work. Am I trying to learn about the user, or am I trying to learn about the product, in this specific iteration and for this specific outcome? What is the biggest value that I'm going to take out of these two? And it might be that I need to learn about both. That's okay. And you might even use the same tool for both. I might be doing interviews and observations, but the script I'm writing for each is completely different, and my conversations during research sessions with the users are completely different depending on what I'm trying to learn.

[00:18:53] It's not about quantity and it's not about the amount of data that we collect. It's about relevance for this specific moment in time where your product is, and how it can help you grow, and again make your company grow and your users happy. In the same way, you need to be aware of this when you're putting a research strategy in place. You need to think: what is the purpose of the research? What are you trying to get out of it?

[00:19:22] There are two bigger streams of research, or bigger approaches to it. Either you're trying to be generative: you're proactively looking for opportunities to generate new ideas, identify gaps in the market where you can fit your product. So you're likely to be less confined to the product that you're trying to develop, and understand the broader spectrum around your product. Or you're being more evaluative. It's more of a reactive position to be in, where you're more or less risk-averse, and you're afraid of failures and the impact that a failure might have on you. So you're just listening to what the merchants are saying, doing some observations, but you're picking up on their words and the things they're complaining about to do your next increment.

[00:20:15] It's an interesting thing to think about when we're doing research, the value of these two during product development, because one is clearly more innovative and the other one is clearly more incremental. But we need to think sometimes that users will never ask you for what they don't know. iPads and Spotifys and iPods don't come from someone saying, "I so want a service where I can just pay a fee and have access to all the music." It comes from, "I have this many CDs and I don't know what to do with them." Because I don't know something, I'll never frame it in a way that is going to drive a new product. Sometimes we need to be more innovative and less incremental if we really want to be disruptive. And again, this goes back to taking risks, which is okay.

Small cycles and launching fast

[00:21:11] The better way to achieve this, to control your success and use research and monitoring for your growth, is to do this in small cycles. Try to research only a little thing. It gives you focused learning and you will have faster insights, which means that in a more steady, managed way you're likely to get to the bigger picture of your company. Shopify wants to make commerce better for everyone, but it's down to each product team to define what better is and how they can help achieve that better in small increments, so they keep growing and keep employing people and keep building a product that's lovable, and the merchants and customers that pay a fee for that product are continuously happy with it and don't consider going to competitors. So if you do this in small bites of work, you're controlling your likelihood to succeed. That's highly important. That's not just a random thing. That's planned, that's staged and that's scoped.

[00:22:23] And this is the right path for you to learn. You need to launch as fast as possible, because the faster you launch, the faster you'll know that you're succeeding and you're on the right track. The other team at Shopify took a year and a half to launch, and it didn't go well. It was too long. They tried to create a minimum viable product, and that takes way more time than starting to ship or deploy little features, little outcomes that allow a product to work. But you start to bring adoption and conversion and attract users just with the minimum things that you're doing. So if you launch, you have all these contacts and learnings in your hands.

Four tools for monitoring after launch

[00:23:15] And the best learnings come during this monitoring process, because you need to prepare for that as well. It's not only about, "Let's launch and let's do some things." Once I launch, what am I learning and what do I want to learn? That's why you have your metrics and your KPIs defined, because you need to pay attention to something. These four tools are highly helpful for you to learn once you launch. There are many more, but when you layer all these together and you connect different dots, it's super powerful.

[00:23:49] Of course, customer feedback: listen to them. Just let them talk. Believe me, it's really hard to be on a phone call with someone for an hour and all they do is complain about your product. You breathe, and it's bad. "I know about that. I know. Tell me something good." And it's like there's nothing good. It's really hard to be listening to this, and there's a tendency for us to go and say, "Well, you don't get what that's doing, and you didn't see that button, or you didn't see that link." You can't do this. Literally sit back and listen. Let them talk. Also, they love it. They love the opportunity to talk about the product. It's really good, and you make them happy by allowing them to do so. So just let them.

[00:24:34] Run some usability tests, for sure, for many reasons: how your layout is performing, your interface, your colors, your typography, the placement on the page. All that influences how they interact with the product. We tend to believe that we know what the best solution is, but we're often wrong, for many reasons. Humans are complex and we design for humans. I'm super complex; I assume any user will be, and we can't control how they're going to relate to and read and perceive a page. So renders [?], they're important. And for all the grey areas [?] where you're not sure, because there are many possible solutions to be built, there's no right or wrong. Often a team of 15 is not going to agree on the best way of doing something. Run some A/B testing if you have enough traffic and enough users, because A/B testing is an easy way to untie a knot at the table and just say, "Well, the users go for this, so let's do this."

[00:25:33] And analytics. Build ways to get reports where you can see what they're doing. Often what users do is very different from what they say they do, and what they say they want to do. When you look into how they click, what the funnel is, what the journey is, and you then connect all these dots, it can be quite surprising. Either you can be doing pretty well or you can be doing pretty badly. And then you start all over again. This is a cycle.

The paint never dries

[00:26:08] Monitoring here is the bridge to the next set of research, either within your product and how to build it, or feeding other opportunities for other products, because you've just learned something that told you there's more to it than you knew. Basically, your context just grew bigger. It's not stagnant. The context you work within, in your company, for users and for your products, is like a living organism. It's like a child; it grows with your product as well. So you need to go back and do all this again. Context is everything. And if you think that once you've learned, you can sit back and just drive from there, you're totally wrong. It's not going to happen. I think I'm doomed, because I have another three years to learn about Shopify, and the amount of products that we do all the time. There is so much happening around you, and with the people working with you, and in other streams of work, that you can't just assume that it has been stagnant. We can't get complacent with this. You need to be actively doing this.

[00:27:26] Naval Ravikant [?] is a graphic designer. He usually says that digital design is like a painting, but the paint never dries. And it's exactly that. You're going to iterate on it always. Unless you've made enough money and you don't care about it anymore, and you're just like, "Yeah, whatever, I'm just going to sell this to someone."

Balancing speed, refinement and evidence

[00:27:48] How we manage at Shopify, in Shopify teams, to solve for this, and to understand how much time we're putting into research and what things we're doing, is at the team level. We understand these three factors. Depending on the outcome that I want to build, I need to balance and define my roadmap, my strategic goal, the sprint, based on speed, refinement and evidence. These three need to exist for you to achieve that 80% and manage sustainable growth within your product and your company, so they don't fire me and they keep doing more great things for commerce, for everyone.

[00:28:36] The reason why these are important, and we often see these failing, is because if you ignore evidence, so research and monitoring, data, insights, learnings, you're just throwing spaghetti at a wall. You're going with your gut, with a hunch: "I think users would like this because I like this. I think this is a cool idea." But you don't know. You didn't validate that in any way. You can try it and be successful, but that's pure luck. It's not based on anything.

[00:29:06] If you ignore speed, though, you're missing the train. If you're so focused on collecting as much as possible, you're just not going to learn fast enough, and you're not going to be able to be disruptive or speed your product to market at all. It's just going to be slow. That is a big, big, big cost. But if you ignore refinement, and that's where design plays a role and front end plays a role, if you're not looking at the evidence, if you're just deploying and deploying and deploying and keep incrementing your product and adding more features, "Well, we do all these things," if you're not refining based on what you're learning... Often it's not about adding more features. It's improving the features that you already have, to make them better, to make them more effective, to allow the user to be happier.