Measure What Matters: Aligning UX Research to the Customer Journey

Eric Karofsky at UXDX Community: Product Growth: Data, CX, and Research Strategies. Video: https://youtu.be/-rUc91dOLEM

Readable transcript: edited from the recording's captions for readability (fillers and false starts removed, punctuation and section headings added). Wording is the speaker's own. Timestamps are positions in the video. Names marked [?] could not be verified against the audio.

What it means to measure CX

[00:00:00] I lead VectorHX, which is a human experience agency. We focus on customer experiences that build loyalty and user experiences that delight. We've led digital programs for companies like Royal Caribbean, Reebok, Michelin, Fidelity and many others. It's a pleasure to speak with you all today.

[00:00:18] It's a total understatement to say that CX is a fundamental business priority. A great quote by Jeff Bezos: "The most important single thing is to focus obsessively on the customer. Our goal is to be Earth's most customer-centric company." You can find all sorts of quotes along those lines. But it's not just grandiose goals; there are real stats. The majority of interactions are multi-channel and over time, and this is increasing. CX enables 60% more profitability than for those with poor experiences. And people are willing to switch brands due to poor experiences, and I'm sure you all have experienced that and have switched at different times.

[00:01:10] What does it mean to measure CX? Let's level set. First of all, let's start with the definition, to make sure we're all on the same page. I really like McKinsey's quote; I'll read it. "A customer experience encapsulates everything a business or an organization does to put customers first, managing their journeys and serving their needs." Now, when you think about all of these different items that encapsulate CX, there are certainly some overlapping Venn diagrams here. But I do want to call out one specific thing: CX is not UX. There can be some debate here, but as I see it, UX is more about digital interactions with a product or service, and key areas are usability, design, interface and the journey. CX, though, is much broader. It encompasses all attributes of the business, and you need to work in concert with multiple departments, and one of the key areas that you address is change management. Getting back to that quote about everything an organization does: it's about organizational change and collaboration across departmental boundaries.

[00:02:26] Here's a high-level overview of the process, and the goal is to develop actionable insights through measuring touch points. We start with understanding all of the different channels, from web to social to email to in-person and events. Then you obtain the data and segment it to understand behavioral personas. You want to see how analytics have changed over time, and that's the longitudinal measure. And you also want to check specific UX changes, whether there are specific items that have changed or specific functionality that has been developed and deployed.

[00:03:10] Then you want to start tracking metrics. There are two main types of metrics, and we heard this in one of the earlier sessions: leading versus lagging indicators, and then those that are contextual in nature. A leading indicator is predictive in nature and suggests future outcomes based on current events. For example, site usability indicates how well people can accomplish their tasks, and it can imply future success. Versus a lagging indicator, and an example of that is churn, which reflects satisfaction and the effectiveness of retention efforts. Then there are those that are in the middle, like CSAT, customer satisfaction, and net promoter score, and those are heavily contextually dependent. CSAT could indicate the likelihood of a future purchase, or it could also indicate the satisfaction with a specific interaction.

[00:04:08] But ultimately what you want to do is get to actionable insights, and I'm going to read an actionable insight. "Customer satisfaction scores have decreased over the past quarter, with a notable increase in negative feedback related to slow response times in customer support. To address this, we should do X, Y and Z." Creating these actionable insights is really important. It keeps you grounded in making sure that you're providing insights that somebody can do something with, but it also shows that you understand the business. Getting to these actionable insights, and really thinking about how things can change, becomes really important.

Why measuring CX is hard

[00:04:55] The issue is that measuring CX is really hard. Just 49% of US consumers believe that businesses deliver a good customer experience. It's not because they don't want to; it's because it's really hard to do. You really need a good amount of alignment and organizational maturity to understand the experience and improve it. I'm not going to hit on all of these different items, but overall, some of the big challenges: there's siloed data, legacy systems, inconsistent data formats, and data isn't aligned. Even simple things like a demographic of male versus female: it could be coded in the database as male versus female, or it could be a zero or one, it could be an M or an F, or it could have multi-gender options. Then there's departmental fragmentation, and I'm going to talk about this a little bit in the case study in a few minutes. There are different technologies and vendors. And I'm just going to hit on this last one, about inconsistencies. It's a big one, because often qualitative research will ferret out specific issues that get hidden in quantitative research, and so there can be inconsistencies in the results, and that's really hard to understand and rationalize.

[00:06:20] To measure effectively requires alignment across diverse entities. This first group is different departments. Some of these are much more aligned with the customer experience, such as product, UX and research. But then there are others, like security and legal, which are all about minimizing risk. That is their goal. But when you think about minimizing risk, managing permissions and consent, which are both legal and security issues, becomes an absolute usability issue.

[00:06:55] In this second group, the way the organization works is an enormous factor in figuring out norms and how to collectively improve the customer experience. And then this third group is tools, capabilities and partners. There's overlap between what all the different divisions are using, and it becomes really hard to measure these effectively. But when you think about what is the central, number one issue, it really all comes down to leadership: whether there is a comfort with creating the change across divisions that's necessary to improve the customer experience. The problem is we're not all leaders. In a bit, I'll talk to you about how you can affect leadership.

Case study: a large government health research initiative

[00:07:50] Now I'm going to jump into a case study, and first just some overview for context. We're going to talk about a very large government initiative. The initiative is collecting healthcare data from individuals to support research. There are two types of users, individuals and researchers, and they go through a pretty similar journey. The individuals first discover this initiative, then they register, they go through and provide their consent to permissions, and then they provide healthcare information, whether that's EHR, specific survey information, or measurements such as weight. And then there are researchers, who analyze the information that is supplied by the individuals. They discover, they register, then they do a pretty in-depth ID verification, because they're dealing with very sensitive data. They research, and then ultimately they publish.

[00:08:52] We have three key measures, and we back them up with a lot of detailed information. The three key measures are trust, satisfaction and engagement. To define those a little bit more: for trust, we use the HX trust score, which is something that was developed by Deloitte and Harvard Business Review. It was important to us to use published information, because trust can be so subjective, so we wanted to leverage existing research, and we measure a lot of that through different surveys. Satisfaction we measure through customer satisfaction, CSAT, as well as NPS. And for engagement, there are many different factors, such as interaction, response time and abandonment, that all roll into it.

[00:09:41] If you take a look at all of the specific information that backs that up: we look at the different channels, whether it's digital channels, phone or in person. We look at key analytics, such as the ones that I just spoke about, but also feedback, usability and accessibility. We segment by different personas, and those are primarily behavioral personas that we use. We look longitudinally, over time, as well as at where they are in the journey. And then, as a matter of frequency, we do one-off reports, such as what was the CSAT of a specific event. We also do periodic measures, like monthly or quarterly reports, such as trust at different stages of the journey. And we also do real-time dashboards, and there are many different ones, and we provide input into the organization's dashboards.

[00:10:43] There's massive organizational complexity, which creates challenges, and I'm going to hit on a few of these. Just to explain this chart a little bit: you have the overall government organization and all of these different areas. This is just an order of magnitude of the number of different departments that we deal with in the government organization. Then on the left-hand side there are health institutions, special interest groups and academic institutions, and these are all involved with supplying data and organizing and understanding the data. The way I have this organized, these are multiple different institutions, again just an order of magnitude of how many different ones there are, but within each one of these institutions there are multiple different divisions, and you need to work with all of those different divisions. In this middle group there are consultancies, agencies and service providers. These are the people and groups that are understanding this information and making this whole program work. Again, there are multiple different consultancies, agencies and service providers, and different departments within them. And then there are technology vendors.

[00:12:07] Here's where the complexity comes in. Not only is this fairly complex, but doing work with the government can be a little bit of a different animal if you haven't worked with them beforehand. There are a lot of procedural items that you need to understand and work with. But then there are things that you don't necessarily think about, and that's things like those looming government shutdowns. When there are those government shutdowns, about a week or two beforehand you know that there's one expected, and everything goes into planning mode for that shutdown, which totally starts to mess with all of your planning. That can add a lot of complexity to your ability to meet obligations. PII, personally identifiable information: we're dealing with healthcare data, and there's nothing more sensitive than that, so being able to access data becomes really hard.

[00:13:08] And then the other one is coopetition. All of these different groups right here are constantly vying to get a bigger piece of the pie, but at the same time they're working with each other to provide the best possible solution for the government. There become some political dynamics that you need to navigate, where you always want to support the ultimate goal, but you also want to support the company that you're working for, so that becomes really challenging.

[00:13:47] There are multiple organizations involved in the journey, and that adds complexity. If you think about it, these individuals providing healthcare information: it's a pretty simplistic journey. But when you look at the depth, again, they're doing it in different ways. They're providing it digitally, they're gaining their understanding of the program in physical ways, they interact in all sorts of different ways. And the big challenge here is that all of those groups I mentioned beforehand own different pieces of this puzzle. There's a big matrix here in trying to understand where the data is and what we do with that data.

[00:14:33] And there are a lot of competing priorities, which makes it difficult to know where to start. Initially, when I came into the program, there weren't any defined processes or procedures to systematically analyze data across the initiative. There wasn't an understanding of how often individuals are asked to fill out certain surveys, and I'll show you the ramifications of that. It's hard to find the right people because, as I showed you beforehand in this chart, there are a lot of people involved. And there are a lot of meetings, and you want to be conscious of the time that you're spending.

Tactics: relationships, doing the work before the tools, and a data catalog

[00:15:12] There are a few tactics that I want to address. If you look up how to align metrics and research to the customer journey, the textbook responses are: understand the customer journey, identify the touch points, define metrics, etc. These are all really good points, and they're absolutely accurate. What I wanted to do, though, is show a few tactics that have really helped me in this initiative.

[00:15:41] Number one: building relationships across the organizations becomes really important. You want to find out who the people responsible for reporting analytics are in all of the different groups, and who their leaders are. You want to define roles and responsibilities and plan meetings. Just simply come up with a really simple spreadsheet of who the different people involved are, how supportive they are of this initiative of understanding analytics, and what sort of analytics they own. Keep it up to date. It's something that's going to become really helpful, especially if you're dealing with a large decentralized organization.

[00:16:31] You want to do the work in advance of buying the tools. There are a lot of different tools out there, specifically voice-of-customer tools and CRM tools, that can help, but you need to do work in advance. You want to find out what your key metrics are, what technologies are used, where the data is, who has access to it and who needs it, what the compliance issues are, and what dashboards you need. If you don't gather this information before these projects start, what happens is you do it during the project, and then these vendors and consultants end up being the central focus. A way to keep yourself central is to constantly collaborate and try to find out this information before you hire all of these companies to support you.

[00:17:29] You want to develop a catalog of all the data being captured. Here's just one sample screen of it. Purposely, you can't read it, but I just want to show you the complexity and the depth of it. I'm going to go through some of the items that are in these columns. You want to find out the instrument information. There are a lot of different surveys out there, or we have a lot of different surveys; there's different qualitative research, different quant research and site analytics. You want to capture all of the different items and instruments that are being used, and then which constituent is providing that insight: is it an individual, a researcher, staff or other? If it's a survey, and that was one of the main focus areas of the work that we're doing right now: how many times can they take it? What's the average length of time to take the survey? You want to ask: is it a good survey? Is it written well, or is it one that needs to get updated?

[00:18:30] Then there's other information in there. You want to find out details about the instrument. What journey phase are they in? Is there a specific trigger? Are they driving leading or lagging indicators? Do they support NPS, CSAT or any specific yearly goals? Then, what do you do with the metric? If it goes up, what do you do? What do you do if it goes down? Who's the point of contact for that instrument? Who owns the data repository, what's the access, and who's the recipient of this information?

[00:19:01] Just to back up for a second: this is a lot of information. It's hard to do, especially with a program that's already running. It will become, though, the absolute most referenced document in the entire process, and it becomes so important to get all of this information. Again, the more that you can own this, the more you become central to this whole initiative.

Mapping metrics to the journey and telling the story

[00:19:30] You want to map the metrics to the customer journey. Again, our focus has been on surveys lately. Here's the simplistic customer journey, and here's a little bit more detailed version, just as a visual. For surveys, we launch persistent surveys that are always available, and then there are surveys at different triggering events in the journey. This is just showing when we ask NPS or CSAT. Not until we aligned it to the journey did we find out that within just two days, there were four times that we asked net promoter score and two times that we asked CSAT. When you're asking it that often, not only is that a burden to the user, you start to question the results, if they're getting asked so many times.

[00:20:23] You want to create insights working groups, and this is something that we did. What this is representing is people from all of these different organizations, and we actually have many more people than are referenced right here. These insights working groups bring together people across the whole organization and initiative to discuss research and define new projects that need to be executed. We develop a learning agenda, and we have access to an insights repository. We meet about quarterly for these different ones. It's really important, because not only is it sharing information, but it's about collaboration, and you're building relationships.

[00:21:13] Ultimately, and this is the final slide here, you want to tell the story, retell it, and tell it again. Constantly message what you're doing. The way that we've done this is, first, we came up with a North Star vision of what analytics looks like in the future. You want to show how these metrics have led to different decisions, and then, most importantly, and this is a really key one, show how those decisions have led to some form of return on investment. You want to highlight the different people involved and empower the insights working groups, and once you do that, other people are telling that story as well.

[00:22:01] Overall, this is just a great way to impact culture. As I mentioned beforehand, we're not all leaders, so it's a way to influence the culture of the organization and influence the most senior levels of the organization. As a result of doing this, we've made it into the weekly status reports to the CEO, which becomes really important, because it highlights all of the different analytics work that is driving the initiative.

[00:22:30] To sum up some key takeaways: you really want to build relationships across the organization. In an organization like this there are so many different people; you want to talk with a lot of folks and understand what they're doing. Create a catalog of the different instruments. Map the data to the journey. As often as possible, try to find ways to map analytics to return on investment. And finally, tell the story, retell it, and then recruit others to tell the story as well. And with that, open to questions.

Q&A

[00:23:07] Host: Excellent, thank you very much, Eric. I really like that. It's not the pure "oh, it's amazing, this is a perfect process". It's the nitty-gritty, real-world example of what people are facing out there, so I hope people found that interesting and useful for their context as well. As always, please do ask your questions on whichever channel you're watching this on, and I'll put those questions to Eric. But my first one, because I like the way you ended there by saying we're not all leaders; a lot of us have to influence. If I was a person in one of those big bureaucratic organizations, like you drew out in that diagram, and I'm just down in one of the wings of that, how much influence do I have, or what can I do? Because I don't have control over some of those bigger decisions about what to measure and what metrics. What are the steps that, as a person in one of those organizations, I can start to take?

[00:24:07] Eric: Yeah, great question. The more that you can start asking questions and understanding the metrics that the different departments use to measure their success, and start to collate that together, becomes really important. It helps put you at the center of the conversation. Once you understand all of those different metrics, you can start defining how we measure them in the most appropriate way, and then checkpointing back with those leaders and reporting back on those. And then again, as much as you can, start asking the questions of how these metrics affect ROI. That's when leadership will really start to notice.

[00:24:58] Host: That's a great answer. I like that: just take it step by step and work it in. It ties in really well with a question that Kesha[?] asked in the last talk, which was: how do you convince people? Because that costs money. Yes, you have your own time, where you can go and ask questions, but actually building it into the applications to start collecting that data costs money. Are there any ways, before you have the nice result to highlight how well it's working and all these insights, that you can pitch to make the investment to start tracking them in the first place?

[00:25:36] Eric: Yeah. If it's a large organization like this, these questions about metrics should be being asked by multiple different groups and leaders, and if they're not, there may be some other issues at stake. But these sorts of questions about metrics should be floating around, and each department has metrics to gauge its specific success. Understanding what all of those people are doing to measure their success, and collating that together, helps put you right in the center, to say: we need to find out answers to these. And if we don't find out answers to these, you can work with those people and find out what the ramifications are. It's almost like saying: if we don't understand it, we don't know the ROI of any specific initiative, or we don't understand the ROI of any specific success. So I think working with the different groups and finding out their different metrics of success becomes really important.

[00:26:59] Host: Great. Yeah, that's a great idea: what are they using today to answer those questions, and if they can't, well, then that's obviously a good place to start.

[00:27:06] Eric: Exactly.

[00:27:06] Host: And do you think that this kind of initiative needs to be centrally organized, or do you think you can have just pockets of people doing lots of things around the organization? And then how do you collate it and bring it up, as you mentioned, bringing it to the CEO?

[00:27:23] Eric: Yeah. One thing that I didn't get into is governance, and governance becomes important, because when you think about customer experience, it's about crossing divisions, crossing departments. Creating these initiatives that cross different departments needs executive approval, or executive oversight, or permission, because otherwise each department has its own goals, and those goals may or may not align with customer experience goals. That's what I was talking about beforehand with security. Security is all about minimizing risk, but if you make it too onerous, it becomes an awful experience and people can't do their work. At an executive level, you need to make sure that all the different departments are working together, and that's where this governance model comes into play. There are centralized governance models and decentralized governance models, and it has to do with the maturity of the organization as to which way is most appropriate for the organization.

[00:28:38] Host: Yeah, it's a great example that you brought up about the departments, because I often find the biggest problems are the bits that go across. The departments tend to be quite good at looking after what they need, but it's the experience that moves across. So do you need to start with governance, or is it bottom-up? I know you said there are two different models, but do you see it more growing up organically, with people saying, look, we need to measure this? Or do you think it's only from the top down, because if you're only focusing on your area, you're missing the cross piece?

[00:29:18] Eric: I think it's both. It's bottom-up as well as top-down. If leadership is not asking for it, then it has to become bottom-up. But ideally you want to be able to pitch to the executive level why this is so important, and if you can align it with ROI, it becomes very apparent why it's important, and more than likely they will listen. And if they're not listening, then it needs to be a bottom-up approach: get the different departments involved in understanding the metrics that are supporting them, and try to find specific case studies or specific examples where you can work together to impact change.

[00:30:11] Host: Brilliant. Well, thanks very much, Eric, for all of that. I think we're just at time now, so just one more time, I want to thank you for sharing your insights.

[00:30:20] Eric: Thank you.