How UX Measurement Wins Over Executives

Rima Campbell at UXDX Community: Driving Success with UX, Generalist Design skills, and an Engineering Culture. Video: https://youtu.be/SJBzvh9j_dU

Readable transcript: edited from the recording's captions for readability (fillers and false starts removed, punctuation and section headings added). Wording is the speaker's own. Timestamps are positions in the video. Names marked [?] could not be verified against the audio.

Why executives struggle to see the value of UX

[00:00:00] Excited to be here today and looking forward to having a great discussion. Quickly: how UX measurement wins over executives. Just a little bit about me. I have 20 years of experience in UX and CX, and CX of course, UX research. I now work for UserTesting as a consultant, but prior to that I worked for Citi and established the UX research center of excellence. Citi started with a $35,000 budget and ended up with three million. I started as the only researcher on the team and then built a team of 13 researchers, and then we started as a North America only region for research and we ended up going global, and that's within five years. You will see a little bit later how we ended up doing that at Citi.

[00:01:04] Some of the challenges UX is facing in enterprise, as you all know, is really UX getting a seat at the table and getting executive buy-in, and doing research across the product development life cycle. There are many reasons for why we don't have executive buy-in, or why executives struggle to see the value of UX. They focus on immediate ROI, short-term focus, cost concerns, especially these days everybody is concerned a little bit about budget and tightening up the belt a little bit. Lack of awareness, and perhaps ineffective communication from UX professionals about the value that they bring to the organization and how they impact the business metrics.

[00:01:53] So how do we actually bridge the gap? UX wants a seat at the table and executives want KPIs to measure against. We all know that what can't be measured cannot be managed. So how do we confidently say that the design is performing as designed and it's actually impacting the business KPIs? I believe that's how we bridge the gap.

What digital leaders do differently

[00:02:28] McKinsey ran a massive analysis that made a convincing case that digital leaders are creating much more shareholder value than the laggards. You can see that in digital services the gap between digital leaders and laggards has remained constant. The reason is that it's relatively straightforward to create an app, a new feature is quickly seen and copied. However, that mobile app must perform well, it's actually table stakes these days. Digital sales, however, is a much more revealing indicator here. Leaders have almost doubled their advantage over laggards over the five year period. The analysis shows that digital leaders grew digital sales from 40% to 70%, while digital laggards grew from 8 to 17%[?].

[00:03:30] So digital leaders drive growth and profitability. The reason for the large differential is that to drive digital sales, leading banks for example go well beyond the mobile app to digital transformation, what's hard to see and hard to copy: the end-to-end process from originating to fulfillment services. To do this they must orchestrate hundreds of teams capable of developing digital and AI innovation these days, day in and day out, across all their customer journeys and core business processes. Knowing what to do is important, but executing on how to do it, and having confidence that it's actually working and performing and engaging customers as well.

[00:04:25] So that's really what digital leaders drive: growth. Everyone wants to be a software company these days, if you notice. If you're not online you're not going to survive. So that means launching software features faster, improving code quality while increasing developer velocity, building world-class user experiences, and freeing development time for work that has business impact.

[00:04:54] What are these digital leaders' secret sauce? According to McKinsey there are three hallmarks of high performing organizations in the digital space. They continually test and iterate. They back research through the product development life cycle and make data-driven decisions with confidence, statistical significance. And they measure the design performance and its impact on the business goal. They conduct user research, develop user personas and user journeys, they build a user-centered product road map and then measure the impact and make data-driven decisions.

The Citi case study

[00:05:45] So setting a north star and combining qualitative data with quantitative data measures and demonstrates the value and effectiveness of the design. Here's a quick case study, back in the days when I was at Citi, that was the first to start measuring the experience and demonstrating business impact. The business wanted to increase the credit card online application submission by 25% and reduce the call volume related to it. It's all about shifting to digital. And the idea was that hey, the website has been redesigned, the online application for card has been redesigned, so we should be able to achieve that goal.

[00:06:35] Well, we ran a baseline study. One task, basically we measured how participants are able to select a credit card, are they able to complete the task, what was the ease of use. And we realized that those metrics are a little low. We tried to look at analytics, tried to look at voice of the customer, didn't know. We ran a qualitative, exactly the same test, qualitative study where we listened to participants as they completed the task, and we found out that comparing the credit cards is really what's not performing very well, and that's why they're not going to submit the online application.

[00:07:20] And so with that we were able to go back to the executives and say that this is why we're not going to be able to meet that business goal, and we need to redesign that particular experience. The question was, how do you know that those metrics are good metrics or not? So we ran a competitive study with the exact same task, and we found out that amongst all of our competitors we performed the lowest in this. So that convinced executives even more, because they have metrics to tie it to, they have competitive metrics to tie it to.

[00:08:00] And with that we were able to get their commitment to redesign it. We ran the benchmark again and we realized that we increased each one of those metrics by 10% at least. And then, tracking towards the business goal, we were able to contribute to increasing the online credit card applications submitted by 22%, and we reduced the call volume by more than 30%. So that was the initial start, and then from there the CTO will not receive any request for any work unless it meets the threshold of those usability metrics.

Meet Alex, the chief digital officer

[00:08:44] So across the product development life cycle, companies deliver differentiated experiences by informing the product at every stage: discovery, design, launch and measure. Measuring the quality of the experience, mitigating risk, aligning with customer needs and improving time to market is really crucial to really become a true software company. Let me show you what that may look like.

[00:09:17] So let's meet Alex. Imagine that Alex is your chief digital officer at Verie Bog[?] and you're the UX design leader. Verie Bog[?] is a digital native brand renowned for its sustainability, luxury footwear and accessories. Verie Bog[?] is on a mission to make sustainable fashion accessible to their target customers across all channels: web, mobile, in store, social and other omnichannel. However, the road to success is paved with challenges. The industry market changes very quickly. To stay ahead, Alex's product team must be agile, customer focused and innovative. We're all facing that, and I hear that from our customers every day, so that's a big challenge.

[00:10:24] So Alex has two primary goals: deliver differentiated experiences and positively impact revenue, and build those experiences better, faster and cheaper than the competition. To do this, Alex has approached customer insight in a number of ways, by investing in focus groups, market research, A/B testing, session replay, analytics tools, online reviews, and the list goes on.

[00:10:56] But on top of that, we all know that those traditional approaches like analytics platforms, A/B testing, they're really not giving an indication of true customer centricity. Why is it working, why is it not working, how does that differentiate me from the competition? So the only solution that allows Alex to even start to think about managing that experience, and making sure that it's aligned with users' needs and it's innovative and fast, is to actually look at every resource available and realize there's missing information. There are tons of gaps in analytics, it doesn't tell you the why. You look at the survey, there is not much depth of information about customers' needs and behavior and attitude, and the list goes on.

[00:11:59] So what really matters the most here is waste. How do we know that the experience that we're running is a good experience, that it's not going to cost a lot of rework and wasted effort or resources, which as you can see here from the graph is massive? And that's not what she wanted to create.

Measuring experience quality with a QX score

[00:12:30] So measuring the experience quality, the quality of the experience, with a QX score. Of course you can use any quality experience score, but this is a QX score that is actually plotted as a 100 point index, and it's used to benchmark your experiences as well as those of your competitors over time. So for Alex, she has a standard that before the experience goes live, or even moving forward to the next step in the product development life cycle, she makes sure to use that QX score that her researchers and design leaders agree to at every stage of the development.

[00:13:21] As you can see, this score is 100, but it's also 50% behavioral, how did they do, and the other 50% is attitudinal, what they think about it, how they react to it, what they think about the experience overall from an ease of use, look and feel, appearance, as well as will they recommend that experience to friends and family.

[00:13:54] So before she goes live she wanted to make sure she hit that 80 plus score before we go into development. And with that she'll be able to save on rework and make sure that the experience is relevant relative to its competitors. It helps her identify experience strengths and weaknesses within and outside of her industry to ultimately inform her road map. So as she looks at the experience and measures the quality, whether it's a prototype or the current experience, we wanted to make sure that we know what to prioritize in the road map to improve those weaknesses.

[00:14:41] As you can see from the scorecard here, when you run an unmoderated quant study to measure the quality of the experience, you choose tasks that have direct impact on the business KPI. And with that you'll be able to track with confidence if they completed the task successfully, and then you also look at the score for each of the attitudinal measures. So it will tell you right away which task needs more work and which attitudinal measure needs improvement.

[00:15:25] So to recap, measuring quality experience improves the quality with stage gates in the product development life cycle, saves development cost by validating before development, and improves the customer experience by identifying strengths and weaknesses with confidence. And then stay ahead via competitive benchmarking as a differentiator. With that you build experiences that customers love, boost productivity and drive more business impact. You get a seat at the table.

A framework to get started: activate, ignite, drive

[00:16:06] So now you say, well, how do I start? A quick framework: activate, ignite and then drive. So how do you activate? You really want to identify what matters most to the executive. Talking to executives is pretty clear, all that matters to them is their business KPIs. And then you identify a pilot project that focuses on a user journey, and you connect the UX metric for that project to the business metric. And then you run that quality score study, qual and quant. You always want to balance the two because you need to know the why.

[00:16:52] And also know your competitors' QX score. As you sit at the table and talk to product and talk to your chief marketing officer, chief product officer, you need to know the weaknesses and strengths, you need to be able to educate them and build trust with that QX score. It's a measure for them, so you will have their ear because you're talking about KPIs and how that KPI is impacting their KPI.

[00:17:21] And then you prioritize the improvements with stakeholders, test and iterate and benchmark until you reach that threshold of 80 plus. And then you visualize and track and share progress regularly, as you have QX scores for all the digital experiences that you're responsible for, and you continually build on that scorecard and make it visually available and accessible to all across the organization. And then you can tell stories and continue measuring experiences and managing experiences successfully, and have a seat at the table.

[00:18:01] So why connect UX metrics to business KPIs? I talked about that a little bit, but there's a quote here: the one thing that business leaders understand is KPIs. So the QX score provides that consistent and comparable way to measure UX across studies, products and business units, and build consensus across the organization.

[00:18:29] If you would think about a model of why the experience score has impact on the revenue: it definitely has a direct impact. If you look, it's like a level four, it's the lowest, like increased ease of use, increased task success, reduced click path, comprehension, improved user satisfaction. In turn it has a direct impact on the product specific goals: increased conversion rate, task success rate, increased volume and transactions, reduced abandonment rate.

[00:19:05] And then with that you go up to level two where those metrics are actually impacting divisional goal metrics, in terms of number of new customers, increasing new customers, retaining existing customers, improving brand awareness, increasing NPS and improving customer satisfaction. And with that, as you all know, those divisional goals have direct impact on the enterprise financial goals, where revenue of the organization and decreasing costs matter the most. So if you think about it from that perspective it makes more sense, and we call that the 21st century UX metrics framework.

American Airlines and other customer stories

[00:19:49] I want to play a quick video from one of our best customers, American Airlines, that stood up a benchmarking program. We're really happy for them. Our colleagues Lucas and Kimberly.

[00:20:08] Video: Here at American Airlines our purpose is really to care for people on life's journey, and that's exactly what we're doing. We're coming up on our 100th birthday at American Airlines in a few years, and with that just becomes a deep rooted history of lots of industry firsts and lots of innovation. Even though we're not a digital native company, we're not setting the expectations, the leaders in technology are setting the expectations, and as a really large brand we have to perform at that level. It's what's expected of us by our customers. We've got folks who are shopping and looking around, more than the US population visits our website annually.

[00:20:49] Video: We adopted UserTesting in the first place to be able to talk with our customers, to get their feedback on the different prototypes and products that we build. UserTesting essentially is the medium between us and our customers. The four key areas for us are shopping and booking, managing and being able to change that booking, the day of travel experience holistically, and then the thread across all of those is the AAdvantage. We use UserTesting to ensure that we're keeping up with the experiences.

[00:21:19] Video: QX score essentially takes what customers are doing on the page and pairs that with their attitudes, and what QX score then does is combine those two to create a composite score that we could then take to leadership and show the quality of an experience. For the digital experiences we've measured with QX score, after going back and making changes we were able to achieve on average a 15% increase in QX score. When we went back and changed the concepts we were able to achieve a 37% increase across the board for all of those tasks.

[00:21:51] Video: We've been able to test our early prototypes, or the website or the app, or even real life experiences that our customers go through, for example interacting with a kiosk at the airport. This is taking a human on a literal customer journey, and being able to talk to humans and talk to customers along the way as we design this experience, whether that's digital or hard product, is critical. Travel can be complicated and there's lots of emotion that goes along with travel. What we can do at American Airlines is make it the easiest for our customers to engage with. UserTesting gives us the human insight that we need to ensure that we're truly meeting our customer needs.

[00:22:34] All right, so now you got a seat at the table. Quickly, some stories from other customers as well, like Kimberly-Clark. When they stood up the UX measurement program they were able to drive prioritization and demonstrate project progress. And also Kroger, same thing, they were able to know that they're building the right solution because now they have a seat at the table and they're able to conduct UX research at the discovery stage. Don't we all want to do that? And then Adobe, they actually embedded the UX metrics in the performance reviews of the UX team and the product team. I won't play those videos, they are available on Vimeo. I will open up to questions, and thank you so much.

Q&A

[00:23:30] Host: Excellent, thank you very much Rima, that was really interesting, particularly how you can convert that qualitative aspect into a more quantitative metric that a lot of senior executives are looking for.

[00:23:44] Rima: Absolutely, thank you.

[00:23:47] Host: So if anybody has any questions out there, please do post them in. Jose has asked about the slides in the recording, so yes we will be publishing the recording, and Rima, if you're happy to share the slides we can post that with them as well.

[00:23:57] Rima: Sure.

[00:23:59] Host: And if anybody has any other questions please do ask them. I'm going to jump in with one. I think that's incredibly powerful to have that QX score, but it feels to me like there's a step before that where you need to get the buy-in before you even have the data to show how important the buy-in is. And particularly in some of those examples that you gave, American Airlines, Kroger, they're what you would call traditional businesses, they're not digital native businesses. So how do you go about convincing them on step one, how to invest in the first place before you can get your QX score?

[00:24:40] Rima: Great question. I think it really requires someone to take this on as their own initiative and take one for the team, take a risk, as what I did at Citi. It's not easy, it's like taking a side project and taking it to the next level without asking for permission. We're all in an enterprise organization, we have access to tools like UserTesting and UserZoom and Validately[?] and others, even other tools that will be able to help you run those baselines to find out what is going on and measure the quality of that experience.

[00:25:22] Rima: So that you are more informed and you have the quant data that matters to the executive, and you have the why data to show them that you know the root cause of the problem, and how, if you really attach those metrics to their KPI, to show them how that directly impacts their KPI, they will listen. Because they want to know what is going to impact their business KPIs. And if you have numbers and you have quant data and the why, you'll be able to convince them.

[00:25:56] Host: And that's a great, it's always the challenge, but take that first initiative and use the tools that you currently have to try to build the case.

[00:26:04] Rima: Yeah, it requires tenacity and passion for sure.

[00:26:08] Host: Excellent. That actually brings us to time, but if anybody has any questions please do post them in and we'll get those questions to Rima. We have a Slack channel as well, so please feel free to keep posting. But thank you once again to Rima for sharing that case study. I found it really insightful and I hope everybody out there did as well.

[00:26:29] Rima: Thank you.