Insights from Liberty Mutual’s Corporate UX Team: Mastering Measurement Practices

Jeremy Rook, Ketan Galhotra at UXDX Community: Mastering UX Metrics and Creating Products for Tomorrow. Video: https://youtu.be/6K03j87JAdo

Readable transcript: edited from the recording's captions for readability (fillers and false starts removed, punctuation and section headings added). Wording is the speaker's own. Timestamps are positions in the video. Names marked [?] could not be verified against the audio.

Introductions and agenda

[00:00:00] Jeremy: Thank you so much, Rory, I appreciate the introduction. Thank you all for joining us for a brief session here to talk a little bit about our efforts in mastering measurement. I'm excited to share with you how our internally facing corporate functions team started to level up our ability to apply UX metrics to our work. Like Rory mentioned, we've refined this presentation down, so we should have plenty of time at the end to answer questions. Feel free to drop questions into the chat and we'll answer them at the end. For now, let's jump right into our time together.

[00:00:35] Ketan: Thanks, Jeremy. We'll quickly go through our agenda. We're going to begin with quick introductions, then we'll give a little bit of background on HEART metrics at Liberty Mutual and how we got involved. Then we'll talk about the workshop prep and the structure, and then Jeremy will talk about the outcomes and the takeaways.

[00:01:02] A quick introduction about me: my name is Ketan Galhotra. I'm a senior product designer at Liberty Mutual, and I live in Boston, Massachusetts.

[00:01:10] Jeremy: And again, my name is Jeremy Rook. I'm a principal user researcher located in Lansing, Michigan.

Why the HEART framework at Liberty Mutual

[00:01:18] Ketan: Before we begin, we wanted to check with all of you: how many of you have tried creating a measurement plan before? Unfortunately, we cannot see the responses here, but if you are saying yes, then the next question is: how many of you felt overwhelmed or not sure where to begin? Maybe you have done this before or maybe you haven't, but creating a measurement plan can be an overwhelming experience, and trust me, you are not alone in this. There are many UX professionals who struggle with creating a measurement plan or feel overwhelmed. Hopefully the knowledge we share in this presentation today is helpful for you to get started.

[00:02:06] We wanted to begin with a quick intro to the HEART framework. For those of you who don't know, Google came up with the HEART framework, and HEART stands for happiness, engagement, adoption, retention and task success. By aligning goals, signals and metrics along these dimensions, this framework can help you make a clear plan to measure a website, a tool or a service. That's a very basic intro to the HEART framework.

[00:02:43] Now the question is, why the HEART framework at Liberty Mutual? Liberty Mutual has three main areas: corporate functions, investments and insurance, as you can see in the image on the right of the slide. The HEART framework is flexible enough to be used in different domains or in specialised applications. The HEART framework also helps product teams align their goals with their business objectives, and it also helps them prioritise the most important metrics for their product. For these reasons, the leadership decided to use this framework to meet the organisation's data measurement needs.

[00:03:26] We also wanted to give a little bit of background on our UX team. Our UX team supports all internal tools that are under corporate functions and investments. Our team has grown a lot in size in the past couple of years. I believe we went from a team of 6, and at this point we are at 22. We have also made broad impact across the organisation, with various levels of UX maturity. Although our team has really talented designers and researchers, we also learned that our team has limited UX measurement experience.

How we got involved

[00:04:09] How did Jeremy and I get involved? First of all, we are not experts when it comes to data measurement. Jeremy and I both have experience of creating measurement plans at our previous jobs. Last year, one of our team goals was to learn more about UX measurement plans and frameworks. So we volunteered to help the team learn about measurement frameworks, including HEART. Jeremy and I decided to run a half-day workshop with the team to get them introduced to various frameworks and how to set up a measurement plan.

[00:04:53] This was the biggest question we had: how can we make it easier for the UX team to learn about UX measurement? I will talk about our approach in the next two slides.

Preparing the workshop

[00:05:08] We wanted the workshop to be engaging and useful, but before we decided the workshop content, we also wanted to understand how comfortable our team is with data. So we decided to run a short survey. You can see a screenshot of one of the questions we asked in the survey. This survey really helped us in painting a better picture of where our team stands when it comes to understanding and measuring data.

[00:05:39] Then we also wanted to see if anyone in Liberty had tried something similar before, and luckily we found someone who had tried running a data measurement training workshop with their team. They shared their experience of running the workshop, what worked well and what didn't work well. All of this information was really helpful for us.

[00:06:03] Along with it, we also wanted to give the team a real problem to work on rather than a made-up one. We contacted various teams across the organisation, and one UX director agreed to partner with us. That UX director's team had researched and tested a new product, and they wanted to measure its impact once it goes into production. Jeremy and I felt that this was the perfect problem for our team to work on during the workshop.

[00:06:36] And lastly, we wanted to work on the resources. We wanted to train the team about understanding and capturing data, and also to inform them not to just rely on the framework. There are many data measurement frameworks available. Our main intention was to tell them: learn what you want to capture first, and it's okay if a certain metric doesn't fit the HEART framework or whatever framework you are planning to use.

The workshop structure

[00:07:17] Along with the prep, we set up the structure of the workshop. First of all, we went through the basics. We went through the benefits of the HEART framework and its limitations, and we also guided the team on how to set up a basic measurement plan using the HEART framework.

[00:07:41] Secondly, we invited our partners, as you can see over here. A data analyst talked about how UX and data can partner up when it comes to creating a measurement plan, and the UX director, along with other members of their team, shared the real problem with us. Once the team was introduced to the problem, the team was asked to workshop solutions, and a bunch of instructions were shared with them, as you can see in the screenshot on the right of the slide. The last instruction for them was: once you are done creating a measurement map, please share your prototype measurement plan at the end of the workshop. And now Jeremy will talk about the outcomes.

What we learned from running it

[00:08:33] Jeremy: Right, thank you again, Ketan. We were glad to have such good collaborative participation and involvement from our partners across the organisation, but at the end of the day, what mattered most is what our team got out of the session itself and how it's propelled us to be more successful within our work. The outcomes of this are still unfolding, but there are some clear successes and opportunities for improvement based on our experience running the workshop.

[00:09:01] As is sometimes true, we learn more from teaching a subject than we do from the learning itself, and we definitely found that to be true in this case. In reflection, some things to be ready for if you do a training like this are as follows. Be aware of other conferences, courses, trainings or team events that might be going on around you. We had a deadline of October to complete this training and share it with the team. We got a little too close to that deadline and ended up having to put this back to back with a different team event. Everyone felt burnt out and like they were playing catch-up for the rest of the quarter. If you're trying to get people to try new things, that's not a good place to start.

[00:09:45] Due to time constraints, we had to condense the training down from what really should have been eight hours to a four-hour time frame. This posed challenges, especially with our participants being spread across four different time zones in the United States. Don't compromise on how much time you think it will take to introduce this concept to your team.

[00:10:09] One of the more predictable responses from heavily worked designers was them feeling overwhelmed with the expectation of taking on more work. We emphasised that our goal was to provide templates and support for initiating conversations rather than burdening them with additional tasks. Ketan and I also wanted to communicate that we're here to assist in any way that we can.

[00:10:34] The final unfortunate reality of working in a large enterprise is that there are varying levels of maturity, so not all teams are equally prepared for implementing measurement plans. Some have data analysts, so they're very ready to start down this path. Others are unsure of how even to get started. We've been very fortunate to have a supportive data analyst named Mike, who's helped us advocate for user metrics organisation-wide. Despite reporting to a single workstream, Mike has been collaborating on data measurement efforts all across different workstreams for the improvement of the greater organisation.

The wins so far

[00:11:21] One big win coming out of this session is that we're starting to see an uptick in the belief in the value of user experience work, beyond what it has been historically positioned as. In a short time, we've been involved in creating five different measurement plans across different workstreams that focus on human-centred success metrics. This has elevated our team's role and made us leaders in some areas. We're proud of this growth over the last eight months.

[00:11:50] In the data technology space, we've guided the product team in identifying the most important user metrics while steering them away from vanity metrics. We have created a roadmap and aligned on primary and secondary measurements for a new Temple [?] product that the product organisation as well as engineers are working to roll out. Through this work, we've built on an existing relationship by showing what UX can bring to increase overall product maturity.

[00:12:21] We've also established ongoing connections with partners who are interested in experience measurement across the org. While maturity in measurement is not equal across workstreams, we are beginning to make inroads into integrating into the work via our data analyst partners like Mike. We also see that there are other parts of the organisation that are beginning to soften or warm up to the idea of engaging with us on creating these measurement plans, as you can see from this email.

[00:12:53] And finally, the trainings provided us with a platform to objectively track the impact of our experiences on the organisation that we work within. We're creating accountability and objectives that focus on impact, regardless of what framework is used. We've also developed resources like templates and measurement briefs to empower designers, researchers and other partners like product and dev to take their first steps towards measurement maturity. All of this adds up to an increased focus on the impact of measurement thinking rather than just the implementation of metrics.

Moving forward

[00:13:33] We continue to move forward from this training by socialising the resources that we have available, such as this measurement brief that we created for our designers to use in conversations with product owners. Resources like this are a great jump-start to conversations that eventually product owners will typically take up the cause for.

[00:13:53] Our hope is that we can move from focusing on frameworks onto the impact that measurement can provide. By focusing on outcomes first, we believe we'll create better systems for our employees to use and overall make a higher impact on our company's success. You can see from this screenshot there are others in the company thinking the same way, combining frameworks to fit the needs of their specific use case. From there, we hope to create an expectation of baseline metrics across our entire, very wide-ranging product footprint. This includes creating accountability within our own team for advocating for user metrics.

[00:14:36] We also continue to build cross-discipline relationships, like expanding efforts within our data organisation and moving them into other spots in the company. This strong relationship with our data org leads to a stronger desire for increased involvement of UX in defining early work: what an engagement funnel will look like for an internal tool, what the correct user behaviours to track are, these sorts of things. The work that we've done in this data organisation has led to a positive relationship that, again, we hope to see replicated across the other workstreams.

[00:15:14] As you can see from this brief chat excerpt, one of the data product owners that we built a measurement plan for is just a little excited to have us on board and figuring this out with them. We see developing these measurement plans as a clear path to create stronger partnerships with product and engineering teams.

Takeaways

[00:15:36] If you're going to take a screenshot of one page, make it this one, but wait until all the points are up on the screen. The first thing to think of as you take something away from this: there are dozens of reasons why people don't get started on flexing their measurement muscles. Ketan already said that neither of us are data experts. The first thing that you should take away from this is to get started. It can sometimes be easiest to start from where your comfort level is, along with your team's. Are they excited by the prospect of measurement? Concerned? Do you have people on your team who have done this or something like this before?

[00:16:18] I think my team is probably tired of me saying this, but I really cannot stress this enough: frameworks are made to help you, not to constrain you. The framework, in other words, is made for the work, rather than the other way around. Maybe HEART is the right fit for you and your use case. Maybe it's Nielsen Norman Group's CASTLE framework. Perhaps it's VMware's user framework, or some combination of the three, or none of those at all.

[00:16:46] Next, find other people who want to help. I promise that they're out there, so start asking around. I bet there are folks in whatever organisation you're in who are willing to dive into the deep end with you if it means furthering the practice of making better-informed, data-backed decisions. UXers may not always have the skill set for implementing measurements, but we all have the skill sets to set up conversations and influence like-minded partners.

[00:17:19] As you think about how to best serve in your context, start lowering barriers to entry. Are there documents you can create for people to reference? Can you set up meetings to help product folks understand the key goals and behaviours that they should be measuring? Whatever you do to decrease the cost to designers and other partners is going to return its value.

[00:17:43] And finally, as a data-curious person, one of the best things that you can do is be available. Answer questions. Find answers to questions that you don't have an answer to. Walk through the resources that you have available. Help people see through the fog that can so quickly cover things and prevent you from addressing the real needs that measurement can assist with.

[00:18:11] That does it for our time here presenting. I want to thank you all for your attention. Please feel free to reach out to Ketan and me for more information. At this point we'll have plenty of time for questions, so again, feel free to submit those through chat if there are any, and I'll pause now to see if we have any come in. Thank you.

Q&A

[00:18:35] Rory: Thank you very much, Jeremy and Ketan. That was a really nice dive into the practicalities of how you put these in place, because often people will just say, oh, just use HEART or just use pirate metrics, but it's harder to actually go and think about how to do that in practice.

[00:18:56] Jeremy: Absolutely.

[00:18:56] Rory: As Jeremy and Ketan said, please do post your questions. We've been having some comments, and some people were engaging with your questions at the start. But we do have a question from Alex, and he's just looking for a bit more context. He said: if you're able and willing to share, what is your UX team composition? Designers, research, other roles?

[00:19:19] Jeremy: Yeah, great question. More broadly, without sharing too much, since I'm not sure what I should and shouldn't share, we have a really diverse set of UX teams across the organisation. As mentioned, our team works exclusively on internally facing tools. There are other teams that are focusing on the insurance experience: purchasing insurance online, purchasing it through a broker, whatever it might be. So our context is really exclusively internal tooling, to improve the employee experience. In terms of our team's composition, the vast majority are designers. We have one content strategist, four researchers and two UX dev positions.

[00:20:04] Rory: And sorry, when you say UX dev, are they actually front-end developers, or what do you mean by the dev part of UX dev?

[00:20:10] Jeremy: Unfortunately, I don't have a great answer to that. It's a relatively new position; we've recently filled the second one. Ketan, do you have any additional visibility into that?

[00:20:24] Ketan: Yes. I don't have visibility on the new person we hired, but the other person we have is a front-end developer.

[00:20:34] Rory: Okay. The term design engineer, I know, is getting a lot more popularity these days, so maybe it's in a similar vein. It's often front end rather than back end.

[00:20:46] Brilliant. You did mention workstreams. Is that what you were talking about in terms of your internal facing, online, broker, or do you then subdivide those into deeper workstreams?

[00:21:01] Jeremy: It is substantially more divided than that. When you think about the functions of an insurance company, it's easy to associate that with: I'm going out, I'm getting a quote online, I'm submitting a claim, maybe I'm having to make a change to my policy. If you own a business, you may be familiar with the process of acquiring business insurance, which is a separate category of work. But if you think about all of the underlying work that happens to get you to that point, there have to be legal processes in order for claims to be submitted, reviewed and completed. There have to be contract reviews done. There's an entire side of finance that is the unknown side of insurance, that all has to be done for insurance to exist. On top of that, there's things like human resources, technology, data as an organisation, and these are the organisation workstreams that I'm referring to.

[00:21:59] Rory: And the last question on this: are the UX people embedded in those workstreams, or do you have UX as a separate function and then the workstream developers separate?

[00:22:08] Jeremy: Yeah, great question. We operate on a matrixed model. Designers are often embedded into a specific workstream but report up through a centralised structure. All of us will report up to a couple of different UX directors, but we'll be ostensibly embedded into a team for a given amount of time. On top of that, the researchers are a centralised shared service across all of the different workstreams that we support. So it's a mixed structure we have over here.

[00:22:44] Rory: Excellent. We have a couple more questions coming in. "I love the idea of socialised resources and all the preliminary work. What is the biggest learning in the preliminary work before bringing people together for measurement thinking?"

[00:23:05] Jeremy: You know it's a really good question when you have to pause to answer. So, Ketan, take some time to think about it, and I'll take a stab at it first. Preliminarily, as Ketan and I set out to create a workshop to bring others up to speed and start to upskill in this area, the big thing that I recognised in terms of resourcing was that a lot of times the assumptions that UXers were coming into this work with were: well, of course we're going to track user frustration or time on task.

[00:23:44] What we really wanted to do was give our designers space to back up a step from that and say: okay, before we go into the specific measurements, what are the specific outcomes, or what are the missions that these tools are supposed to be doing? All of our work is internal, which means we don't actually want people to spend time on certain tasks. There are some things that have to be done internally that should be efficient over everything else. They shouldn't necessarily be tracked on whether they make someone happy. No one is happy if they have to fill out a specific legal form internally; that doesn't drive joy. But we can track efficiency. So the metrics that we then create from that downstream are going to be impacted by understanding the context of the measurement that you're deciding on. Do you have anything to add, Ketan?

[00:24:36] Ketan: Yeah. I think another big learning was, along with the resources, how do we equip the designers in the team to be comfortable having a conversation with our stakeholders about creating a data measurement plan, even if our designers have not done it? How do we give them enough confidence? Having these kinds of resources that we put together, along with the workshop, I believe really helped our team. As Jeremy showed in the presentation, from zero we went to five measurement plans in a year, so that was a big win for us.

[00:25:18] Rory: Awesome. I guess we have time for maybe one or two more. Is there an easy way for the teams to share ideas outside of workshops, given that a lot of teams are now working in an async fashion? Or do you still think that it's worth just bringing everybody together for that workshop?

[00:25:39] Jeremy: Yeah, I think a lot of that's pretty heavily dependent on the overall level of measurement maturity within your team. Like Ketan said, we went from six people to over 20 in the span of a couple of years. We were collecting people faster than we were understanding their skill sets, and so what we ended up with was a very wide range of comfort with measurement. As a result of that, we wanted to lower the barriers for everyone, and if you're trying to do that, I think some sort of synchronous activity is probably the best way to go about it. If there is a larger mix, or people are further along in terms of maturity, there may be opportunities to do something asynchronously, but I would not have been confident in the success of that for our specific context.

[00:26:30] Ketan: And along with that, I would say it also depends on the team culture. For us, we knew that a workshop would work better, but if you can think of some other activity that will work based on your team culture, go for it. Again, we are UXers. We like to iterate, experiment and see what happens. If the outcome is not as expected, try a different way of sharing the measurement stuff.

[00:26:57] Rory: I love it: using your own iterative process in your work as well. Excellent. Well, that actually brings us to time, so I just want to thank you once again, Jeremy and Ketan, for sharing your process of how you've helped roll out measurement across Liberty Insurance, Liberty Mutual Insurance.

[00:27:18] Jeremy: That's right. Well, thank you very much. Thank you for having us.