Many Faces of Data: How To Begin With UX Success Metrics?

Justyna Belkevic at UXDX EMEA. Video: https://youtu.be/VSloctMMQhw

Readable transcript: edited from the recording's captions for readability (fillers and false starts removed, punctuation and section headings added). Wording is the speaker's own. Timestamps are positions in the video. Names marked [?] could not be verified against the audio.

Heroic Hugo and the question of measurement

[00:00:00] UX designers are famous for wearing many hats. We can be artists, business consultants, workshop facilitators, researchers, et cetera. The hat of the UX success metrics is a new addition to our collection that can sit just as well. As designers, we want the products we build to be satisfying and easy to use for a long period of time after we build them. But how can we know that's the case? Let's explore how to start with UX success metrics and prove that our design work is worth the investment.

[00:00:42] Let's start with our main hero of the day. Let's look at our Heroic Hugo. Hugo is a UX designer. He is good at what he's doing, and he loves it too. For the past year he managed to build a successful new product, meeting all the user needs through design thinking methodologies. So everyone in the company started to refer to him as Heroic Hugo, performing heroic UX deeds, and he believes that UX outcomes are so good and clear, why would anyone want to measure them? The project is done. The success itself is self explanatory. Everything works.

[00:01:28] Well, our Hugo poses a great question. Why should he measure UX, especially if it seems that it is a success? I would throw a curve ball and ask Hugo, what does he mean when he says that the product is successful? How does he know? Is it going to be successful after one year or two? Crossing the finish line is a great feeling, but seasoned UX designers know that there's always more to be done. To build a product based on user insights isn't enough nowadays. We need the data that shows our continuous success. We need to start tracking UX metrics. So let's find out how we can do it.

[00:02:15] Hello everyone. My name is Justyna Belkevic, I'm a senior UX designer. I have worked with many fantastic clients through the years and I can tell you that UX metrics is still a new kid on the block. So that gives us UX designers a fantastic opportunity to be trailblazers in this new field and wear these hats before anyone else.

What UX success metrics are

[00:02:41] So let's start from the beginning. I have been using this UX success metrics term quite a bit now, so let's unpack it. UX metrics are a set of key quantitative data points used to measure, compare and track the user experience of a website or an app over time. And most importantly, they need to be aligned to business goals, so we can measure behavioral and attitudinal data points. Behavioral metrics focus on what users are doing, and attitudinal is how they're feeling when they are using the service or a product.

[00:03:18] So let's look at the example. Let's take Superman as our main hero, and the business goal for Superman is to make sure people of Metropolis don't die in accidents. So the success metrics reflecting that goal could be rescued people per hour, flight time to arrive at the crime scene, or maybe even outfit change time. The key metrics are usually different for each product and should always be determined individually. As you can imagine now, the metrics that would fit to measure the success of our Superman might not fit in the call center environment.

[00:04:02] Okay, let's move to the big question, the why question. The use of success metrics reduces opinion decisions, helps prioritize problems and improves our products. It's always a good idea to have UX data with us when discussing any changes to the product. And an even better idea is to always start with the questions you want to get answers for, and only then think about which data to use. So for example, to our Superman, we can ask questions like: which skills make our Superman most useful for the citizens of Metropolis, or maybe which outfit leads to the most social media engagement. Interesting, isn't it?

[00:04:53] The question "when" is something that I haven't seen being answered frequently in the internet space, but I believe it is just as important. From my professional experience, I want to share that it is never too late or never too early to start working on your success metrics. I personally like to fire such discussions to the stakeholders as soon as I complete the initial research and discovery. The reason being is that, as we saw in the previous slide, those discussions don't need to begin with data, but with goal-oriented questions first.

[00:05:33] You don't need to have a wireframe to already start imagining how the product's success might look. And in between the research and the final user testing activities there's plenty of time to change the goal and the tracking methods. Most importantly, speaking about UX success metrics early will help you to manage the expectations early on too. So if during your research you discover that your users will probably use your product once or twice a month, you can already manage the expectations of the stakeholders and advise that you shouldn't choose the daily visit rate as the metric to track your success.

Step one: start with the goal

[00:06:17] That all sounds great, so how do we start? And I'm glad you've asked, because I have a five step guide on how to start with UX success metrics. So, step number one, and the first step is always the most important one. One way of creating success metrics: we need to start with the goal.

[00:06:42] Okay, amazing. But how do we choose it? Is there anything that we can do to help us? Absolutely, yes. When I am thinking about the goals, I like to ask myself this question: what will happen if our design is successful? And the answer might be, we will increase orders, or maybe our call centers will get fewer calls about the log-in issues, or a decrease in the crime rate in Metropolis if our question is targeted to our Superman.

[00:07:17] So to help with choosing a goal, I want to share with you the most frequent goal types. Those are revenue, acquisition, and engagement. You see, spending time thinking about our goal is really important before we jump into the nitty-gritty of choosing a specific metric to track it. It is like building a product itself. We don't jump into UI before doing UX, right?

Step two: prioritize your success events

[00:07:49] So, once you have one clear goal that your team and stakeholders agree on, prioritize your success events. Success events are actions that you want your users to take towards your goal. I advise teams to have micro-conversions and macro-conversions. Your micro-conversion is basically your goal, for example registering for your newsletter, maybe downloading a software package, et cetera. And your macro-conversions are all the necessary steps that users can take which all help this micro-conversion to happen.

[00:08:34] For example, if you are an online course company and your micro-conversion is to have users registering for a course, your macro-conversions might be number one, signing up for a newsletter; number two, downloading a brochure; and number three, maybe requesting a session[?] with an instructor. So knowing your success events will help you to prioritize them and understand where to give your all.

Step three: choose the right metrics

[00:09:04] Once you have the strategy and prioritization in place, it's time to think about the metrics. Here we'll not be discussing necessarily all the pros and cons of every single UX metric. I strongly believe that the go.com[?] or many other presenters at this conference will do it better justice, but we will look at the metrics from one level higher up.

[00:09:33] We are going to answer the question of how you can choose metrics that are right for you and your project. Good news, it is actually pretty straightforward, like 1, 2, 3. So number one, make sure that the metrics are aligned with your goals and success events. I know, I know, you might be thinking, why am I mentioning such an obvious statement, but trust me, sometimes in the weeds of stakeholder discussion we needed to agree on metrics that had nothing to add to the success of the project but just sounded rational.

[00:10:12] Number two is to choose actionable metrics. Ask yourself, what would I do if the metric is out of line? Do I have the levers that can impact it? Measures that track final outcomes like revenue or total customers don't give you much time to react or guidance about what to do next. And number three, focus on three to five key metrics rather than trying to capture and analyze everything. As Yuval Noah Harari in his book Homo Deus said, "In ancient times, having power meant having access to data. Today, having power means knowing what to ignore."

Step four: turn the numbers into a story

[00:11:09] Great. Everything that happens after this point, you need to have a working product. All the previous steps, you don't need to have it just yet, but now we have it. We have either an app or a website. Your dashboards or reports should be easy to understand and provide enough data to give your audience context. I'm sure many of us have seen dashboards that stubbornly show some numerical values which we can't decipher on our own, and which then have the potential to be ignored even if they are super important.

[00:11:51] So when you create your own artifacts, remember to share insights, not just data. Tell a story about the numbers, make it easier for the reviewers to remember your point. For example, start with setting the scene by explaining the origin of the data, then introduce your characters and the plot, making the viewers care about those numbers. And don't forget to add the conflict through a story, and a happy ending or a resolution.

[00:12:24] So let's do it now together for our usual client Superman. Many citizens of Metropolis are just like you and me. We want our day to go crisis free. Recently we noticed that 66% of younger citizens of Metropolis are curious, and they want to know more about Superman, not just that he solves the crimes. But our Superman is so busy solving approximately 10 crimes a day and saving the city $1 million every half a year that we can't have him hanging out with the kids. So in order to solve this challenge, we decided to experiment with social media and created an account for him. Now Superman has 15 million subscribers, and we measured that it positively impacts Metropolis citizens, who are now 10% happier and 14% feel safer by seeing what Superman does on his social media. So stories like this will help your stakeholders remember the data and care about the user.

Step five: read between the lines

[00:13:36] Slowly but surely, most companies have arrived in the 21st century and are collecting large amounts of data every day. But having a lot of data, or tracking everything you can think of, will not equate to successful UX metrics. As UX success metric enthusiasts, we need to make sure that we can all read between the lines of what our data is showing us, because even simple metrics can be easily misinterpreted, and I will give you an example.

[00:14:06] Think about time on site, which people often equate with engagement. More time could be positive, but it could be also negative. For example, time that the user spent feeling confused, distracted or even frustrated. Even if you track engagement by looking at time spent on your site and pages per visit together, it's not clear how that equates to engagement all the time. The skill is to read between the lines, and I'm sure that will come with practice, or with the use of more and more sophisticated tools. So be always aware of the many faces of data.

[00:14:57] Great. So let's summarize the five points. In order to start with UX metrics, we shall follow these five steps. We will start with a goal. We'll prioritize our success events. Then we will choose what is right for us and the project at hand. Then we will turn numbers into a story, and we will always be aware of the many faces of data. As UX success metrics enthusiasts, we will now be equipped to make sure that those ones and zeros are tailored to our goals and help us to achieve our design outcomes.

Bonus: triangulation

[00:15:42] But wait, a bonus: how to be one step ahead of the game. I'm sure you'll be thinking about it. Well, organizations across the board are seeking more meaningful metrics that go beyond the defaults that off-the-shelf analytics can provide. The newest development in metrics is to leverage different methods to collect data about the same phenomenon, rather than relying on single signals, for example using the number of page views as a proxy for engagement. The trend is now towards setting metrics that draw from multiple sources or event streams. We call it triangulation: having multiple points around the behavior we want to study, which enables us to make better decisions.

[00:16:34] So when you are measuring your UX success metrics with the help of analytics, and you sprinkle in some moderated usability tests or user interviews, it will enrich your understanding of how users use the product, and that will help you minimize the limitations and blind spots of single methods.

[00:16:58] So if you remember one thing from this presentation, it is that successful UX metrics start with a goal. Once the UX metric goal is defined, then we can talk about the tools and the data itself. This practice, the goal setting, will be the strongest building block in your new journey with success metrics. So it's time for UX success metrics to have the status it deserves in your organization. And the sooner you start measuring, the sooner you can slip into your Superman outfit. Thank you so much. My name is Justyna Belkevic and it was my pleasure to share my knowledge with you.