Bridging Perspectives- a Framework for Seamless Collaboration Between UX Researchers and Product Analysts

Subhasree Chatterjee, Archana J. Shah at UXDX USA. Video: https://youtu.be/ndP3SnlJy-o

Readable transcript: edited from the recording's captions for readability (fillers and false starts removed, punctuation and section headings added). Wording is the speaker's own. Timestamps are positions in the video. Names marked [?] could not be verified against the audio.

Peanut butter and chocolate

[00:00:12] Archana: Hi everybody. Hi everyone. Thank you so much for your time today, and thank you UXDX for this amazing conference and for giving us a platform to talk about something we're both very passionate about: research, analytics, triangulation and all the magic that comes from it. He did such an amazing job introducing us, there's very little I can add to that, but my name is Archana Shah. I work at LexisNexis as a principal UX researcher. I've been a researcher for over a decade now, just with a plethora of different titles.

[00:00:45] Subhasree: I am Subhasree. I'm a data analytics manager working at LexisNexis as well. I was initially hired to help the UX research team with their data needs, and that's where I met Archana. But I quickly realized that it's not really about me helping them with data needs. Working together and collaborating together can help us create a much more holistic picture of the problem that we are trying to solve. And that created this framework that we are going to talk to you all about.

[00:01:14] Archana: Before we go any further, we wanted to quickly introduce all of you to our protagonists, who we are going to use today to tell the story of us going from strangers to best friends to now this framework. Peanut butter here is the data analyst, and chocolate represents research, because chocolate, and we wanted to prime you for lunch.

[00:01:36] Subhasree: As a data analyst working with product, the questions that I'm generally trying to answer are how big is the problem, how much is the problem, or how many users are feeling that kind of problem, and generally trying to quantify the effect or the problem space. That's why I generally use different kinds of code to create different kinds of narratives around graphs and charts and trend analysis, to be able to provide answers for those questions.

[00:02:06] Archana: And on the flip side, traditionally research has been well known for going to the source of the problem, the users. What are the problems? What are the problem spaces? What are the boundaries? Why do those pain points exist? And I might be preaching a little bit to the choir over here, but the usual slew of methods that apply over here are contextual inquiries and surveys and so on and so forth. And that's what my toolkit tended to consist of.

Two separate processes

[00:02:33] Archana: A little bit about the processes when we were both new to the organization. Everything that you're going to look at today is oversimplified for the purposes of this talk, but this will give you a general sense of how it broke out. You start with a business question. Any business question is never as simple. There are lots of facets to it, lots of research questions it breaks down into. Once you have that, there's the research planning: what method is going to be best for answering those questions most effectively and efficiently? Once you've collected all of that data, what you're working on then is compiling all of that information to then put together your recommendations. So it's advice, answers or directions for what the business should do.

[00:03:18] Subhasree: The analytics process looks pretty similar as well. When we get the business questions, we are generally trying to figure out what part of the question can be answered by the data and analytics that we have access to. We then generally go off to figure out what are the data sources that we can get some of these answers from. We have those findings, create a narrative around it, create some data and insights based on the graphs and dashboards. And then we go back to our stakeholder with the recommendations from those data and insights.

A tale of two strangers

[00:03:48] Archana: And that brings us to the fun story part of the talk today. This is a tale of two strangers, and it's a story from 2017, so basically a million years ago. But this was a story where peanut butter and chocolate were standing on two opposing ends of a cliff and did not know anything about each other at all. And something that you should know: any good story starts with some background and some context. So here it is. The world that Subhasree and I work in is basically one that caters to legal professionals. For any of you who have watched any kind of courtroom drama, Suits or any one of those legal dramas, you may already have an inkling of how complicated law is. Now, if law is going to be complicated, the legal tech that supports that world is just as complicated, if not more.

[00:04:34] Archana: And the problem that we were looking at was basically catered to solving for legal professionals. A cornerstone of what legal professionals do is legal research: looking at all of that history, all of that case law, all of the nuances and subtleties, to be able to build a solid case that you can present, and you cannot afford to look like a fool in court. And that is exactly what our flagship product was catering to. It was meant to support that research.

[00:05:02] Archana: Now, that said, customer support was overwhelmed and inundated with lots of people calling in asking for help in using that system. And this was a problem because we actually did have a mechanism to self-serve, look at the capabilities and figure out how to use the system, in the system itself. So what the business came to us with was a simple question: maybe we should rebuild help, add in all those bells and whistles, add in more content that people can self-serve and use the system for. And like most projects these days, you've got one month to do it. And that of course meant that Subhasree and I were scrambling, and we were running off on our own individual paths to find as many answers as we could.

[00:05:45] Archana: Breaking down that entire process, the focus of our talk today is more the collaboration and the framework piece of it, so I'm not going to bore you with the methods or the details. But what I can tell you is, from the business question of would rebuilding help help, what I broke that out into as research questions is: why do users need help, and if they need help, where are they seeking it, where do they want to seek it? For this I ran interviews. And the biggest takeaway that came from those interviews with customers who were calling infrequently was: there's a lot of data in your head. You're thinking about your client, you're thinking about that case issue, you're thinking about the state you're in when you're researching that issue. There's so much in your head, and putting all that into the system to get everything that's relevant is extremely difficult. So it was clear to me that users wanted help with figuring out how to craft an effective search.

[00:06:41] Archana: And the second piece of this was the survey, the piece of where do users need help. This was fairly straightforward. All of the data has been falsified to be respectful of our confidentiality needs, but the takeaway was the same. Yes, people wanted to go to customer support, but if you look at that big red box, they wanted to find help online. So they wanted help online, but no one seemed to be using it.

[00:07:08] Subhasree: When I looked at the business question of whether we should improve the in-product help, the data analytics question that I came up with was basically to figure out: are people using the current help system, and if they are, how frequently do they use it, and when in their user journey within our product are they trying to use help? When I looked at the data, what I found was a very low percentage of current users are actually interacting with help. Even among those, only 20% are coming back and using it multiple times. And the sessions where they are interacting with help are not successful, so they are probably not able to find what they are looking for. And they are generally using help right after running a search. So the conclusion from this analysis was that people are not really interacting with the in-product help.

[00:07:56] Subhasree: Remember, in this scenario we don't know each other. So we went separately to our stakeholders and provided our insights and findings, and based on the time crunch that we were running through, product actually decided to improve the in-product help that we already have: add more content to it, add more flexibility and options for the users to be able to interact with the help system and help them with their searches. So we ended up creating this new and improved in-product help, but nobody really came. We didn't really have any increased engagement with the in-product help. People were still calling up customer support to get the help that they needed.

What went wrong

[00:08:42] Subhasree: So what went wrong here? I did provide the insights that people are not really interacting with our in-product help. This one over here suggested, after talking with five users, that we should build new and improved help.

[00:08:56] Archana: To be clear, I made no such recommendation. You can get the problem right and the solution wrong, and that happens a fair bit. At least I shed a lot of light on the subtlety, the nuances and so much of the complexity that goes into that search system that clearly needed help. What did you do? Boil all of that into a number on a bar chart?

[00:09:12] Subhasree: Well, my bar chart at least talked about the whole user base that we had, and people who were using or not using help. But of course, we should all listen to your qual insights. So basically vibes.

[00:09:27] Archana: Those vibes are why the product is even half as successful as it is today. You try launching a successful product without understanding the user or the problem space more deeply.

[00:09:36] Subhasree: What are we going to do with understanding our users deeply? We are not doing therapy sessions here. We are just trying to solve business problems.

[00:09:45] Archana: No, you're right. Let's just keep solving the wrong problems and A/B test ourselves off a cliff.

[00:09:53] Archana: We hope this isn't the case, but for those of you that it was, I hope it wasn't triggering. But a lot of this, if it wasn't text, may have been subtext for all of you.

The barriers between research and analytics

[00:10:00] Archana: Now, in all seriousness and jokes apart, what we've found over the years that we've been working together with our teams, growing our teams, and also in the conversations we've had the privilege of having across the industry, is there are some very real barriers that exist between our functions. One of them is lack of clarity in roles. If you don't know what the other person does, how do you even know what to go and ask of them? And then, who owns which piece of the question? And it's only natural to butt heads in the age of AI, when everything is accelerated. Everyone wants those insights yesterday. That tension becomes a little bit stronger.

[00:10:35] Archana: The second is lack of understanding of each other's disciplines. No, research is not just talking to five users, and analytics is not just bar charts. There's far more depth to what each of us is capable of. But again, without that understanding, and with all of the jargon that can come with each of these, that barrier gets stronger. Time constraints. At this point, we're all working with stakeholders who wanted those insights yesterday, and those bring a very real pressure, forcing us to resort to a toolkit we already know and are familiar with.

[00:11:06] Archana: And the last of this, of course, is stakeholder biases. I don't know if this is also relatable, but over my career I've worked with stakeholders who built features off of that one thing that one user said that one time. But at the same time, I've also had stakeholders who will only budge if there is a statistically significant number attached to anything I present. And both of these biases are real. They're just as real as the impact it can have on the business. If you continue down that path and you refuse to change, you could be solving the wrong problems. And when you're solving those wrong problems, it leads to a lot of repercussions as well. You're less efficient and it's a lot more expensive.

[00:11:47] Subhasree: At LexisNexis, an A/B test can take about 3 weeks. And if you're trying to recruit from a very niche market audience, then even one single qualitative study that you want to get really strong, reliable results on can take very long, and time is money. So are we all doomed here? Don't we have any way to get out of this spiral? We actually might have a solution for you.

The parallel universe where they are best friends

[00:12:10] Subhasree: Let us now imagine this parallel universe where UX research and analytics are best friends. We know each other's strengths. We talk to each other on a regular basis. And in this scenario, the process actually looks pretty different. The moment that we get business questions, we sit together and we figure out what part of the question can be answered best by the research team versus the part which can be best answered by the analytics team. Then we go off, we do our research, we do our analysis, and with the findings that we get, we then come together and share a recommendation based on our holistic understanding of our users' problem and the problem that we are trying to solve. And this is of course iterative as well, because the insights that you are getting can lead to more questions that we need to answer.

[00:13:00] Subhasree: In this scenario, the questions that I needed to solve for were still the same. Are people using help? How are they using help? How frequently do they use help? And my findings were exactly the same as well: that they are not using help, and in the sessions that they use it, they are not finding what they're looking for, and they are using help right after running a search.

[00:13:20] Archana: And the nice thing about having a BFF is you know exactly what they're capable of, what they're going to find, and how it might drive your next steps. So it became a lot easier for me to plan sequential research and parallel research. The piece of this that I knew I could run in parallel, with that one month still in place, was what do users want? So that same survey was run, the same results were found: people wanted help online.

[00:13:44] Archana: So when you put it all together, people wanted help online, but the success rate was very low even when they used it. So maybe online help wasn't offering things in the right manner. Customer support was clearly doing something strongly different from whatever was available in the system. And the second piece of this: we knew that they wanted help with search, because Subhasree's data pointed out that whenever they used help, it was typically after they ran a search. But when that was available in the system with all those bells and whistles, they still weren't clicking into it. So maybe people are just having a hard time finding it.

[00:14:19] Archana: I took the first piece of the puzzle, which was basically what is customer support doing that's distinctly different from our online system. This time around my research question was a little bit more pointed. It wasn't just why do you need help. It was why do you need help with search? And therefore my findings were also far more pointed. It was the usual stuff: there's too much in my head, lots of variables that I need to plug in, and I need comprehensive coverage. But it was also: the system isn't hard to use. I can figure out the bells and whistles. I don't know what to put in the terms, the right kinds of sources, what libraries I should be looking in. All of that is my biggest problem. And that's why it's wonderful to have someone on the phone brainstorming with me, telling me how to put those terms and connectors together, craft that search, and then I have far more confidence in the answers that I got.

[00:15:13] Subhasree: And when I was trying to answer the question of whether they're not able to find what we have in the product, we conducted some A/B tests and made help much more visible in the product. But even after doing that, people were still not engaging with the help that we have in the system. People were still calling up customer support, and they were still doing that right after running a search.

[00:15:33] Subhasree: So this time around, because we were talking to each other, and based on all of these additional insights that we have, we actually ended up creating a more comprehensive narrative of the problem space. We realized that even if we create much more enhanced capabilities in the product, people are probably still not going to use it, because they didn't really need help with just writing a search. They needed that brainstorming partner, that sounding board, going back and forth to figure out answers for those very specific and complex questions that they had.

[00:16:08] Subhasree: So our recommendation for the product this time around was different. We suggested that enhancing the capabilities is probably not going to increase engagement or reduce the customer support volume that we were seeing, but maybe conversational help can help. So this time we actually ended up shelving the project, and that ended up saving a lot of time, money and resources, as you can imagine, for the business. Bringing perspectives together this way can really help us become more effective analysts and researchers, because we can mitigate some of the blind spots that we might have if we are working in our silos, and it can create much more effective solutions for our users and for our business.

Find your counterpart and share findings

[00:16:52] Archana: Yeah, sounds simple, right? But we've taken many a stumble along our journey to building this framework, and we'll walk you through what we've learned along the way as well. First portion of it: find your counterpart. For that, if screaming Marco and waiting for the Polo doesn't work, typically what you can do is reach out to product. More often than not, product naturally aligns all of these different pieces of the puzzle together, but it's on you to take that next step and continue building more on top of that relationship that is established.

[00:17:24] Archana: If that doesn't work, one of the things that our leadership has been supportive of, and what I would recommend, is reaching out to them and setting up monthly reviews where you're able to talk about the kinds of insights you learn, what you bring to the table. The first step to anything is awareness, and from there on, you can lead to further connections. The third piece of this, and I know this sounds mildly stalkerish, but look them up on the Active Directory and ask if they're willing to have an informal coffee conversation with you. Talk about your priorities, run them against theirs, see where you might be able to supplement or complement each other's data.

[00:17:57] Archana: Second piece of this, and this is actually where we got started. Typically, after you run your research is where you have findings, so there is a typical starting point for a conversation. Otherwise, it's hard to know what to talk about when you meet them for the first time. So share your findings and then ask for analytics. If this is what I found, what are you seeing in the product? Are people doing what we think they are doing? Are they doing something completely different? If they are doing what you think they are doing, then you've found a stronger story to tell your product or your stakeholders about what they should build, which of the problems to focus on, and you have a quantified priority you're able to attach to all of the issues that you have found.

[00:18:39] Archana: Now, if they don't agree, that's actually the fun part, because what you've accidentally discovered is a third hidden door. In what world can both the qualitative and the quantitative insights be true? And what would those hypotheses be? That becomes your new research project.

Plan research together and bring your counterpart in

[00:18:54] Archana: And the next step, assuming that this has become second nature and we've built that habit of reaching out to each other, is to start planning your research together. The same things that we demonstrated earlier: if you know them and you know what they're capable of, you also know what parts of the answers they'll be able to inform, and that can also inform what you do as a next step within your research. So essentially, you start with the business question, break it out into all of those different facets, different research questions, and sit down and have a conversation about what can you bring to the table? What should I own? And if the same question can be answered by both, which subparts of these questions can I own, and how will we put that together? You can have that conversation up front, and all of that friction goes away.

[00:19:42] Archana: And brownie points for this one. As you're conducting that research, if you're able to tag or bring your counterpart in, what I've seen over the years is that the analytics counterpart's research and their findings become a lot better informed, because they hear it directly from the source, and you're building a level of empathy that lasts very, very long. And at the same time, if you're talking about a particular study that you've already run, this was a case very recently where my counterpart was able to look over my shoulder and say, "Oh, are you trying to measure willingness to use for the AI? I can add some behavioral data to that, and we can make that model a little bit stronger." And we were able to go to the business with: here are the levers you have to pull, this is the strongest lever, that's where we need to go and have the biggest impact.

Be proactive, share success metrics, build it into the process

[00:20:31] Subhasree: Now, when that becomes very normal, when you are best friends and you are already working on projects together, the next step to take is becoming a little bit more proactive and starting to solve the biggest problems for the product: figuring out what those biggest pain points are, and who are the users who are feeling those pain points that we should solve for. It also helps if we can design some of the success metrics together, because oftentimes, as a supportive function within the product, it can become difficult for us to figure out how much impact we are really creating. So having these success metrics together can really guide us: are we solving the right problems the right way, or are there things that we still need to do?

[00:21:13] Subhasree: So look at those success metrics together, from a dashboard, from a periodic report, and figure out if we are moving in the right direction or not. And if we are not, that kind of insight can give you access to a prioritized backlog that you can work off of, to make sure that we are solving the most important problems for our users.

[00:21:35] Subhasree: And all of that sounds great in theory, of course, but we are always dealing with the real constraints of time and resources. In our experience, what has worked best is to include collaboration as part of the process, so it's not something that we are ignoring or neglecting when we have these real challenges. Whenever we are talking about a business question, the part where we are trying to figure out what the problems are and what we are trying to solve for, it helps to have a cross-functional conversation with the key stakeholder in place as well, so that we understand the context and the problem that we are trying to solve, and it helps us create better research and analytics questions.

[00:22:18] Subhasree: And then we can go off and do our own research and analytics and figure out what the findings are. But in these phases, it helps to have regularly scheduled working sessions, so you are coming together on a regular basis and talking through your findings or your challenges. And it can also help to include some templates and checklists as part of the process. In our case, we have some templates around how we can share the findings together with our stakeholders. For checklists, we have some around what are the data sources that we should look into, who are the people that we need to invite to our next meeting, who should we reach out to to triangulate and synthesize the findings that we have. And of course, looking at the dashboards or reports together to make sure that we are moving in the right direction of solving the right user problems.

Happy users and a happy business

[00:23:10] Subhasree: This is our full framework in action. There's a lot going on here, of course, but this is just for future reference. Hopefully all of it makes sense, because we have tried to build up to this total framework. The point that we are really trying to make, if it's not obvious by now, is that working together and bridging perspectives can really help us get to happy users but also a happy business. Because analytics on its own can answer questions like how big is the problem, how many people might be facing that problem, but adding research insights to it can also help us understand the why, or how much of a pain point it is.

[00:23:52] Archana: And on the flip side of that same coin, while research can easily answer the what and the why, it can also go that extra step of informing which one we should focus on and which one we prioritize, especially in an age of limited resources. And that's why bringing both together and collaborating can really help us get to this holistic idea of who our users are and what their biggest pain points are. And while that benefits the business, what will benefit you personally is that you would become a lot more effective and efficient, because you're able to squeeze a lot more out of the same kinds of studies that you'd be running, and it would lead you to an effective product. Join your partner in crime.

Q&A

[00:24:38] Host: That was wonderful, right? All the zooming in and zooming out of the frameworks. And of course the metaphor of peanut butter and jelly and all things seamless collaboration. What we're going to do now is transition into some online Q&A. And of course you can stalk our wonderful speakers on LinkedIn. But in terms of the substantive nature of all things seamless collaboration, there are a couple of really good questions up here that I want to lead off with, if you're open to it. LexisNexis, in terms of size, what's the number of employees?

[00:25:18] Archana: About 10,000, if you're looking at just our company.

[00:25:22] Host: Yes. Sure. So there are people in the audience that are like, "10,000, that's so cute," and there are others that are thinking that's a pretty sizable organization. But the question here is around large orgs: you're getting a lot of data coming in, a lot of opinions coming in as well. What would your recommendations be for navigating conflicting data points?

[00:25:46] Archana: More often than not, the conflict can arise because you haven't defined what you're actually measuring. And depending on whom you ask, how big the stick is depends on who's looking at it. So more often than not, what it takes is for all of these different research functions to get together and define how you're going to be measuring and what you're going to be measuring. From there on, it becomes a little bit easier. Not saying that all of those will agree perfectly, because a lot of it depends on instrumentation, what the tools' capabilities are, etc. But that is usually a great starting point to start avoiding those conflicts.

[00:26:24] Subhasree: Yeah. And just to quickly add to that, the metrics that I was talking about. Defining what good looks like, because in many cases people have different ideas, as Archana said, based on their understanding, their expertise. But having that shared understanding of "this is the problem that we are trying to solve, and this is what good looks like, this is how we know that we have solved the problem" helps.

[00:26:48] Host: That's great. I'm going to ping-pong this back to Archana here. Hearkening back to an earlier presentation from today, what we heard about is the dual-stage approach of delivering tactically as well as thinking big picture and strategically. And there's a question here around navigating and/or prioritizing different research initiatives. We have the big strategy-level initiatives of trying to answer larger questions, maybe some things that are less defined, versus at the lower level, product-level and click-level interactions. How do you manage those tradeoffs?

[00:27:33] Archana: As much as we proposed a framework over here, I want to be clear that our roles aren't always 50/50. Especially when you're starting to enter into areas that are unknowns, where we have no data to begin with, that's where I think research is maybe sometimes working in isolation. You're going off and running based off of market research data, customer support data, whatever you're able to get your hands on, and you're answering the big strategic, big picture questions with the usual methodologies. Jobs to be Done is my favorite. But that said, the tactical end of things is typically where she probably spends a lot more of her time than I do, in terms of making sure that things are working right, feeding things back into the prioritized backlog, because there is something to look at.

[00:28:21] Archana: So in terms of time split, I think research ends up being a little bit more front-heavy. That said, in terms of prioritizing, and this is more a personal opinion than anything else, if you have only a limited amount of time, big picture strategy questions have to take priority. You get those wrong, it doesn't matter what you build.

[00:28:39] Host: Okay. Anything?

[00:28:41] Subhasree: No. She said it perfectly.

[00:28:43] Host: Right. She nailed it. Yeah. As far as our speakers are concerned, they have a wealth of professional experience that extends beyond LexisNexis, that includes Twilio, T-Mobile, and a shared experience consulting at Infosys. I'm curious because, as a common thread, as a researcher, something that I've run into, and something that's coming in from the questions here, is people discounting the value of research because it's more qualitative than quantitative. I'm just curious, from your perspectives, how you handle those sorts of objections.

[00:29:24] Subhasree: That was one of the reasons that we built this framework, because we feel like collaborating together and getting these insights together can make us a much stronger case, not really against our stakeholders, but for our users, let's put it that way. That's why working together and combining all of these different data points that we have access to is the best bet for us to create that comprehensive narrative and be like, if you don't listen to this... we have such a strong case here for that problem.

[00:29:59] Archana: Okay. And we've also gotten to a point where we punt it to the right person. We say that requires strong confidence. You're not asking a deep question, you're asking a broad question. We need to go across the market, and we need to have higher confidence that this is a solution that will work for everyone else. We need to work together on this one. It's not just qualitative. And they do the same thing when it's, "We don't know the why behind this, we need to go to research." So building that function, building that partnership is important.

[00:30:25] Archana: But in addition to that, in complete honesty, what has worked somewhat successfully with stakeholders has also been: if we don't do the qualitative research, here are the number of things that could go wrong, and I just want to highlight these as risks. Positioning those as risks can be convincing. Worst case scenario, if you run that research up front and you learn that you were fine, you spent a couple hundred dollars. But if you find out that you were wrong and you were solving the wrong why, it could have potentially cost you millions of dollars.

[00:30:52] Host: Love that. The opportunity cost. Yes. Y'all, thank you so much. That was a great presentation.

[00:31:06] Archana: Thank you.