Conducting objective research - Mitigating your hidden biases

Yael Gutman at UXDX USA. Video: https://www.youtube.com/watch?v=9-wUcPZCG04

Readable transcript: edited from the recording's captions for readability (fillers and false starts removed, punctuation and section headings added). Wording is the speaker's own. Timestamps are positions in the video. Names marked [?] could not be verified against the audio.

The customer service calls

[00:00:00] Hello, everyone. Thank you so much for joining me today at UXDX. My name is Yael Gutman and I've been in the product space for over 20 years, leading projects, running user research and advocating for product. Today I want to share with you a use case that prompted me to identify practical strategies to help brand advocates such as myself collect and interpret data objectively, so that we can continue to provide our users with the most meaningful products.

[00:00:27] Inspiration for this talk came from the internal customer service team at the organization I was at. This team reported monthly to management, capturing the main reasons users were calling into the call center. It was clear that on a monthly basis, very consistently, there was a specific area on the website that drove most of the calls. Customer service had shared that this was making it difficult for them to serve their audience overall, and so management requested that we investigate the area.

[00:01:01] This was a familiar area to the product team. There was a previous project around a year earlier, managed by the head of the product team, that really focused on increasing the end-user conversion rates. This project was deemed successful by the heads of the product team, and so we came into the investigation with this pre-existing narrative.

[00:01:26] We started looking at the user experience, and as always, we found areas for optimization. We also pulled reports. Some of those reports included conversion rates of the end-user experience, and those were found to be high, very well within the industry standards. So we concluded that while the calls were coming in, they more or less were part of just making the business work, and there wasn't any concern with the end-user experience. While we did find a few optimizations, and we did recommend making some updates to the experience.

[00:02:02] While we did our investigation, another team had suggested making updates to the navigation to the area. So there were two solutions put forward: one on the user experience in the area and one on the navigation to that specific area. But both of them shared the same measure of success, which was increasing the end-user conversion rates. Just by the definition of the measure of success, it's clear that both teams were focusing on the end user and not on the reporting team, which was customer service. There was something that inhibited both teams from really listening to the reporting team, and we'll touch upon that as we proceed.

Why we missed the real problem

[00:02:53] Both solutions were put forward to development and both launched. We did a post-launch report for both, and both were successful in achieving the defined KPI of increasing the end-user conversion rates. So we reported this out to management, and as the weeks went by, it was clear that while the end-user conversion rates went up, we were unsuccessful in actually making any change to the reported issue. There was no change in the ratio of calls that were coming in, and there was no real change in the types of calls that were coming in for this area.

[00:03:34] In retrospect, there were three main reasons that contributed to our inability to look at the challenge customer service was bringing forward. First, the product team was suffering from arrogance. We had a preexisting belief that the end-user experience was satisfactory. We used the quantitative data to validate that, and for that reason we thought that the customer service team was reading the situation wrong.

[00:04:01] The second, and this is really hard to say: we didn't treat the customer service team as a user group, although they had reported the challenge to their own workflow. Our discovery, our KPI and our solution were really focused on the end-user experience, really minimizing and ignoring customer service. And the third thing is that between the two teams that proposed the two solutions, there was a lot of tension, which we didn't realize at the time, and that didn't allow us to collaborate and really leverage the experience and the knowledge between the teams.

[00:04:44] All these are in clear violation of the cornerstone of user research, empathy. The key verbs in the definition of empathy are understanding and sharing. We clearly did not take the time to understand the impact that these calls were having on customer service workflow and day-to-day operations. We didn't investigate and we didn't understand the pain points. Only when you understand those pain points and the experience that the user group is actually experiencing, only if you share those, can you actually come up with a solution to help resolve their issues, and that is where we completely failed.

[00:05:22] This realization was a huge blow for me and others on the team. Being empathetic is part of our professional persona. So to understand that it was our own internal emotions and unchecked biases that led us to a solution that was completely off the mark was a very hard pill to swallow. We were lacking the objectivity that would have led us to a more meaningful solution. Many times we underestimate how our emotions, general drivers and biases influence our attitudes and behaviors, and the direct impact this can have on doing user research and interpreting data. So with that in mind, I want to share some biases and some strategies to avoid them.

Confirmation bias and a wider range of views

[00:06:09] The first bias I want to talk about is confirmation bias, and this rears its head in user research when you gravitate towards specific responses and specific qualitative data that fit into your preexisting narrative. That is exactly what we did in this experience. We had the idea that this specific area was already dealt with in the previous project, and we did not believe there was an end-user experience issue. So we used the quantitative data to validate that the end-user experience was good, and this allowed us to minimize and ignore the customer service team.

[00:06:51] Avoiding confirmation bias is made easier when you have a wide range of points of view within your own team, as opposed to the narrow perspective we exhibited in the example I shared. We have since expanded the user research ideation sessions to include representatives of other teams. We now have representatives of customer service, of the technical team and of QA. This has given us a greater understanding of our business in general and of our users specifically. Another benefit is that we have increased the breadth of testing ideas, as we have much more to learn from.

[00:07:29] Another thing that we do is leverage the analysis partners we have outside of the organization. When we have an area of interest, we share that with these partners and we solicit from them, from their experiences with other clients across a wide range of industries, what testing ideas were had and what they learned. This not only improves our research capabilities but continually puts us in a place where we have to listen to others and learn from them, which is key for being in research.

Active listening and marrying the data

[00:08:05] Very important also is to employ techniques that force objectivity. Active listening is one of those. The goal of active listening is to capture the feedback that others have given you without adding any commentary. To do so, we listen more and talk less. Allow for this negative space where the users you're speaking with need the time to gather their thoughts. Don't feel the need to fill in this space with words.

[00:08:37] One thing that I do constantly is mirror what I'm hearing. After I think I've heard the point that the user is sharing with me, I summarize it and then I mirror that. Many times the user will say, "Yeah, that's exactly what I meant." But many times they will also say, "Well, that's sort of what I meant, but this is what I'm really trying to say." This is really important in order to make sure that you have captured the essence of what the user is telling you.

[00:09:07] And the last thing is that active listening involves more than just listening; it involves all your senses. It's also observing how the user is interacting with you during the session. There are a lot of behavioral cues that could give you a lot of context about what is being said versus what they really mean and how they actually behave.

[00:09:31] Using quantitative and qualitative data is key in order to have actual insights. The quantitative data is what the users are doing and the qualitative is why they're doing it. Only when you really marry the two do you have the full picture. In addition, a discrepancy between the qualitative and the quantitative can many times give you a clue that you actually have a bias. In our example, there was a discrepancy between what customer service was telling us and the quantitative data that we decided to use to prove that the end-user experience was satisfactory. Had we taken a closer look, we could have understood that we were gravitating towards that piece of data and ignoring what customer service was telling us.

[00:10:31] Communicating impartially is key for research, so that whenever you collect information, you do so without influencing the subjects that you are speaking with. We spend a lot of time on the script for user tests, for user interviews, for surveys. We do so so that we are asking the questions and managing the interactions in a way that doesn't influence the responses that our subjects are giving us. Once we are happy with the script, we give it to peer review, to colleagues that are outside of the specific project. Many times those peers, who have not been as close to the script and have not seen it for as long as you have, will be able to find even more areas you should fix to make sure that you are presenting yourself and communicating objectively.

Implicit bias and the customer service team

[00:11:33] The next bias that I want to talk about is implicit bias. In popular culture, this is stereotyping. In user research, this means that you are coming to the table with preconceived notions and generalizations about the users you are speaking with or the groups you are engaging with. In our example, we definitely had an implicit bias towards the customer service team. While we did understand their role in the organization, we overall underestimated their understanding of the website and didn't value their perspective, and for that reason we were dismissive of them.

[00:12:17] In order to avoid that from happening again, we have since instituted periodic sync-ups with the customer service team. This is not with the management of the team but with the actual reps, and this has proven invaluable. They are on the front lines with our customers and have a lot of information that is key for us in order to do our jobs much better. We share with them what our ideas are for projects as well as for testing, and they have been able to share back feedback regarding what's a really good idea, and some enhancements for ideas. We've also been able to collect from them additional challenges beyond what we heard when we were just speaking with their management. So I very much recommend identifying any user groups within your organization that have that direct contact with your users and leveraging them for your needs.

Self-awareness

[00:13:22] In order to avoid your own biases, you first need to be aware of them. Only if you are aware can you put in place tools and methodologies to neutralize them. So I suggest that whenever you start a project, whether it be discovery or research, first check in with yourself and be honest. Figure out if you have any preconceptions, any defensiveness, any ego that's related either to the project you're looking into or to the user groups you're going to engage with.

[00:13:53] It's very helpful to write these things down. There's something about putting pen to paper that makes it real; it makes you accountable for it. It's also useful to do this as a group. If your whole team writes down what their own perceptions are and you share them, it's then easier, with the buddy system, to be accountable and help each other out and point out: are you going down a road that's based on your bias? I believe that had we done this in the example that I shared, we would have noticed that we definitely had these preconceptions, and hopefully we would have been able to take a different path.

[00:14:36] Being self-aware requires intent on a regular basis, so whatever cadence works for you, you should do that. I do it at the beginning of each project, discovery or research, and throughout the project itself I go back to my notes to make sure that I'm continually aware of what my initial beliefs were and continually avoiding them. I know some people do this on a daily basis. Some do it as a meditation, five minutes a day; whatever it is for you, I suggest you start that practice.

Humility and celebrating learnings

[00:15:11] Humility is key to being open and receptive to others and what they are sharing with us. It's important that we remember that our opinions are just opinions. They're not facts, and they are not more valid than anybody else's just because we hold them. Once you are experienced, it's easy sometimes to fall into the trap of believing your own opinions and the notions that you already have in your head, and so it's even more important to practice humility.

[00:15:40] A fun way to do so is to experience being wrong from time to time. One thing that we do in our organization is bet on which testing variant will win and by how much. This ensures that all the team members will be wrong from time to time. It normalizes being wrong, and wrong is not a bad word. It just means your hypothesis was incorrect and it was proved otherwise. It's really important that we continually put in place the ability to remind ourselves that it's not about being right. It's about proving hypotheses and learning from them.

[00:16:24] Another thing that we do is celebrate learnings, formerly known as failures. Whenever we have a post-launch report or other results of a test that we did, we share and socialize this within the team as well as with other teams that are related to that specific area. When we share the results of something that didn't go as expected, we make sure that we emphasize what we did. For example, the costs the test avoided that we would have incurred had we moved to development without making sure that the impact would be what we expected. And when we share things that didn't go as expected, we also acknowledge the importance of the hypothesis itself, and how refreshing and how humbling and how important it is for a researcher, for a product person, to get down to earth and understand that we don't hold the answers. We're actually looking for the answers.

Continued learning and summary

[00:17:32] The strategies that I spoke of until now were really related to the workspace. Now I want to talk about something that's more holistic, which is continued learning. Continued learning lends itself to research because you put yourself out there; you must learn from others. You must collect information, and you must be open to what others' experiences are. There are multiple ways of doing this. You can learn a new skill that's related to your work or unrelated to your work, and just by doing so you put yourself in the student's seat, and that is super helpful to force yourself to listen and to learn from others.

[00:18:15] I've recently learned a new skill, which is furniture refurbishing, something I've never done. I've had some failures, but I'm learning new skills and new techniques and I'm very happy with what I have been able to achieve. Another thing is reading and traveling. This by default opens you up to a new world of concepts, environments, people and cultures, and this lends itself so naturally to then being able to engage with people in the user research world, and to listen to them and gauge what they're saying to you.

[00:18:58] To summarize, it's important that we're able to identify our internal drivers and our biases in order to keep them at bay and make sure that they don't influence us and don't blind us. For this, we have to put in place self-awareness techniques to identify what those drivers are, so that we are able to put in place tools to neutralize them. It's important that we continue to practice humility so that we remember that our opinions are just opinions, and we are here to actually learn and validate hypotheses and not prove them right.

[00:19:36] To help with this, it's important that we create an inclusive mindset where we have access to a wide range of points of view. It's important that we put in place tool sets that force objectivity: active listening, to make sure that we collect information without interpreting it with our own preconceptions, and using qualitative and quantitative data in order to see the full picture and to understand if we have any preconceptions and biases. We should communicate impartially with our users, so that we don't influence them with our perceptions and we allow them to respond in the way that is natural to them, without any impact of our own beliefs.

[00:20:31] And lastly, continued learning. This is so that we as people are in a place where we are continually open and receptive to others, and this naturally will help us with our day-to-day roles. Thank you so much for joining me today, and enjoy the rest of UXDX.