Iterating Towards Success

Yael Gutman, Erica Jorgensen, Guppy Ahluwalia, Martin Woodward at UXDX USA. Video: https://www.youtube.com/watch?v=vNEVlCBCYY8

Readable transcript: edited from the recording's captions for readability (fillers and false starts removed, punctuation and section headings added). Wording is the speaker's own. Timestamps are positions in the video. Names marked [?] could not be verified against the audio.

Panel introductions

[00:00:00] Martin: As Rory said, I'm Martin Woodward. In this morning's panel we're going to be joined by the same group of panelists who were all speaking yesterday. If you haven't had time to watch their sessions yet, then I strongly encourage you to go back and take a look. Don't worry, you can stay here and enjoy this panel first; all the conversations are going to be relevant to you. But definitely go back and take a look at those sessions; they're excellent. I'm going to let the panelists introduce themselves by just quickly saying their names, where they work and what they work on. First of all, let's talk to Yael Gutman.

[00:00:30] Yael: Hi, thanks for having me. I've been in the product space for around 20 years. I started in marketing and moved to project and then product management, and I've really been focusing on user research for, I'd say, the last eight years. I'm very excited to be here.

[00:00:47] Martin: Fantastic, thank you very much. We're excited to have you. Next up we have Erica Jorgensen from Microsoft.

[00:00:54] Erica: Hello, everyone, how are you doing? I'm Erica Jorgensen, and I'm a senior content designer at Microsoft, also a manager. I've been in the product space for about 15 years. I started off in journalism, but I've been at Microsoft about four years. Happy to be here, thanks for having me.

[00:01:12] Martin: Fantastic, thank you. And last but very much not least, we have Guppy Ahluwalia. Guppy, fantastic to have you on. Do you want to say hello to everybody?

[00:01:22] Guppy: Hi, thanks for the introduction, Martin. I'm Guppy Ahluwalia, senior design program manager at Dropbox. I've been in the operations space for about 15 years. I started in theater operations, I've worked for non-profits, with engagements at Google, and I'm very excited to be here and talk to you folks.

Confronting bias in research

[00:01:41] Martin: Well, thanks very much. First of all, just me personally, I wanted to say how much I enjoyed all of your sessions around user research. In particular, I loved how you were really honest and told stories about mistakes you've all made. This was a common theme across all your sessions: mistakes you've made, mistakes the organizations have made. But what was great was how you learned quickly from them and then improved, trying to continuously improve.

[00:02:12] Yael, in particular, you had a very, very open story about biases that your team had taken into a research study, and how you found out you were using data to validate your research, to prove points you wanted to go in there and prove, rather than actively confronting your biases and continuously testing those biases, which is really good. Do you want to explain a little bit about that example, and then how you and your teams actively challenge your biases when you're looking at data, especially around user research?

[00:02:51] Yael: I think being open and objective is part of the persona that any research person definitely has to have, and product too. But we're all human, and I think we're sometimes blind to what's actually going on internally, so we don't realize that we're actually being influenced by our biases. We don't even realize we have biases; that's the definition of it. So it's really important to have strategies so that we can continually make sure that when we come to any discovery engagement or any research, we are really open to receiving the information and analyzing it as it comes in, without interpreting it based on what we're coming from home with, so to speak.

[00:03:52] In our example, there were so many things that just happened at the same time. There was a project we were asked to look into, and the head of the product team had already done a project there. The story in the team was, "We've already dealt with this issue. We've solved it. We have to look into it, but there isn't really a story there." And it just happened that with the reporting team, we didn't realize it at the time, but we weren't really considering them knowledgeable enough about the website. It was the customer service team, and I don't think we really understood how well they knew the website and the users. We sort of thought, "They're on the side, people are calling them, so they're not really sure what's going on on the website."

[00:04:50] These two things happened together, and it was a terrible storm. We used data to validate the story we already had in our heads. I think that's really important: when you have data, that doesn't mean you know how to interpret it, because you can make a story go each and every way. It's really important to be true to what the data is telling us. In our example, we took from the data what we needed in order to validate the story we already had, and that culminated in a solution that didn't solve the actual problem.

[00:05:43] Sometimes you can do that and it's okay; you don't really realize. But in our case it was clear. We even set up the wrong KPI, because we weren't focusing on the right user group. It was just so shocking for everybody. I'll speak for myself: part of my persona, and I believe it is, is that I listen to users, that I'm open and receptive and I can follow the trail. And there was something in this story, after so many years of working, that was such a blow. But as you said, we took those learnings and put in place strategies to make sure that whenever we approach something, we are aware of where we're coming from. We are aware of our biases, we are aware of our internal drivers, and once you're aware you can then put tools in place to make sure you don't fall into those pitfalls.

[00:06:45] Martin: Definitely. I think it's good to go into every situation with a hefty dose of humility. I quite often think about some of the amazing mistakes I've made along the way, and try to remember those as I'm going in, just to keep me honest with things. Erica, how do you make sure your team isn't taking biases into experiments, and that they're testing hypotheses rather than trying to prove assumptions?

[00:07:17] Erica: Absolutely. I think in our team, and at Microsoft in general, the culture is to acknowledge that you are biased, and that that's part of it. You can't erase all your biases, but you should be aware of them, and you don't know how many you have. One thing we've been digging into a lot is David Dylan Thomas's book Design for Cognitive Bias. I think that's a wonderful resource that I would recommend to everyone. It's chock-full of biases that I didn't even know about. So it's about being humble every step of the way, like you said, and checking yourself every step of the way. It's really important.

[00:07:56] It is humbling, and I think in that way it's great. But I'm thinking of teams in the past that were very ego-driven: "I'm right, I'm going to prove I'm right." That's garbage; you don't want to work that way. It's all about the customer, and you are not your customer. Even if you use the products that your company makes, you're not everyone. So you have to think of the customer all the time, and we always check ourselves. Our team's chats are chock-full of "Hold on a minute, wait a minute, stop, stop, stop and think."

[00:08:31] If you're moving really, really fast, which we like to do at big companies like Microsoft (we want to get things done, we want to accomplish things and make business impact), we have to slow ourselves down a little bit and not move too quickly, so we can check those boxes to make sure we're not making mistakes. When you're doing usability testing, when you're talking to customers, every minute is an opportunity to slow down a little bit, think, and be more careful and cognizant about bias. It's everywhere, every minute of the day.

[00:09:06] Martin: Definitely. It's interesting as well, when you're talking to customers, one of the things I very much enjoy about customer research is actually trying to read through their implicit biases and the way they see the world, and trying to understand the world from their viewpoint. That's what I find fascinating about the whole role. Guppy, what about you? With bias, how do you make sure biases aren't coming into the data you collect? You've obviously done a lot of data analytics over the years.

Privacy and responsible data handling

[00:09:33] Guppy: From my perspective as an operations professional, I think about who you are including in your studies, in terms of diversity in recruiting. One of the learnings that came out of the exercise of putting together this customer research privacy training was coming together and figuring out who we want to participate in our studies, who we were talking to, and then how we increase that diverse pool of folks.

[00:10:11] But when you ask that question, there's a lot of responsibility we have around the kinds of questions we might want to ask folks in research, and especially who you're including, and that may be personally identifiable information or it might be sensitive information. So you want to increase as much choice as possible for your participants in being able to answer those questions: whether or not they want to be included in your study, and how they want their data handled.

[00:10:44] There was this process we went through, with some learnings around what kinds of questions we were asking before participants entered into studies, and whether we needed any additional practices in place around handling data more responsibly. That was an exercise we did. We broke up the questions around the purpose of the study, as well as why we were asking these questions, whether we needed to be asking them, and how we were giving the participants choice, especially around sensitive data and answering those questions: giving them options and letting them know how we're treating data. I would say that actually led to us creating trainings and making sure we have platforms and tools in place so that we can ethically and responsibly deal with data.

[00:11:39] Martin: Definitely. If people haven't seen it, your talk was all about baking privacy into customer research experiments. I know I shouted it out at the beginning, but yet another shout-out: make sure you go check out that talk in the recordings if you can. It was very interesting as well. Yael, what about you in terms of privacy? We obviously hold on to lots of customer data when we're doing research. Do you have processes in place and training in place for your groups in that area?

[00:12:14] Yael: We do. Our organization has a big legal team that advises us on many issues, and part of that is how we collect data, what we can keep, and what type of analysis we can actually do. For example, we've used ClickTale in the past, so we had stored recordings. We do our best not to collect any PII, but if something slipped through, then of course we would erase it. When we do surveys, or when we do outreach for user research, it also goes through our legal team.

[00:13:10] There's a lot of back and forth regarding what we ask and how we ask it, to make sure that we collect enough information for it to be meaningful, but we don't want to collect too much information. People don't want to give just any information, and we don't want to collect information that we can't hold. So there's definitely a lot of teamwork across different departments. Not everything can be as fast as we want it to be, but that's in order to make sure we comply with everything we need to comply with.

[00:13:48] Martin: Fantastic. And Erica, I used to be at Microsoft, so I've seen all the videos about Nelson[?] and "Microsoft builds on trust," so I know for a fact that privacy is baked into the Microsoft standards of business conduct training and all that sort of stuff.

[00:14:02] Erica: Absolutely. I think that's top of mind for everyone. We don't want to lose our customers' trust, and I think that's one thing Microsoft stands for, front and center: customer privacy.

Moving fast without statistical significance

[00:14:15] Martin: Yes, super important. Now, the title of this panel is all about iterating towards success. Erica, I was really taken by the ways in which you were using customer research to learn more quickly without having to do full A/B tests in production. I'd be interested in the message you had about not getting too hung up on statistical significance. That was really interesting. Do you want to recap that a little bit? Then we've got a question from the audience that I want to take as well. Can you recap that first?

[00:14:49] Erica: I get on a bit of a soapbox about it, because I think A/B testing, any experimentation, done to be more precise, can really slow down an organization. It's great to be confident in your results, but I've seen that the content research or content testing that I dive into in my presentation can bring velocity to your work. With A/B experiments, I joke that we're not creating the next COVID vaccine; we're working on content. It's important to be customer-centric, but we don't necessarily need to be super, super precise and get to statistical significance with our work in some cases. If you have a home page and you're worried about making a change to a really high-value piece of content, go for it.

[00:15:35] I think people like to bandy about the term, though. It makes them sound important, it makes them feel scientific. My brother's a statistician, so maybe I have an interesting point of view about that. He is a professor; he teaches statistics for a psychology department. With content, if you're getting directional data, sometimes that's enough. Hurry up, make your content better and get it in front of the customer faster.

[00:15:57] I've seen A/B experiments slow down teams and just get people all bent out of shape: "Well, we're almost at statistical significance, we're not there yet," and they make experiments cook for longer, and then another month goes by. Just hurry it up, is how I feel, if you are confident you're heading in a direction that's right. I think content research, which we do with UserZoom and usertesting.com, can get you there faster. There is a time and place for statistical significance in A/B experimentation or multivariate experimentation. I did it all day long when I worked at Amazon; it's very valuable. But in some cases it will turn into this molasses and slow down your work. So use your judgment, use your judgment, and don't use the term without knowing what you're doing.

[00:16:56] Martin: A question from the audience is: how do you strike that balance between slowing down, as we said before, to check biases, but then, like you say, ensuring you go fast as well and don't get caught in analysis paralysis? How do we strike that balance?

[00:17:16] Erica: Know your customer really well, know the customers you're focusing on, so you can tune out the noise. For example, if I am not sure of the right word for a new product, I will test a few examples with UserTesting or UserZoom so I'm confident. If I had five contenders, if you will, I can put those examples in front of my audience and get feedback very quickly with a tool like UserTesting or UserZoom, to get the two or three that bubble up to the top, and then put those into A/B experimentation. Or retest: if you had 10 options, boil it down to five, and then boil it down to three, and figure out what the winners are.

[00:17:56] That's by testing the words themselves. I think if you can decouple words from prototypes or visuals, you can get a lot of information very quickly. Not to say that prototype testing isn't valuable; it certainly is, and our teams do it all the time. But if you are looking for the right words, you can do research on the words only. That is so simple-sounding, but it can be revolutionary in the way that you approach your work.

[00:18:28] Martin: Right, sort of constrain the scope.

[00:18:29] Erica: Yes, yes, and you can chunk it down. These tools, like UserTesting and UserZoom, are super helpful, but I also want to throw out there that sometimes they can help you move too fast. You want to make sure your tests are structured in a way that reduces bias. Again, we're trying to run, run, run, but I'm actually working on a book about this very topic, so maybe this time next year I can throw a book at y'all. UserTesting has a lot of resources on its website that I find super valuable for checking bias while at the same time speeding up the process and making things easier. And I think the more you do that, the more confident you get in your judgment, in your ability to say, "Ah, this is the right direction to go in, or not."

[00:19:22] Martin: Makes sense. Guppy, what do you think about this as well? Are there any tips for the audience on iterating and speeding up as well as doing it carefully?

[00:19:31] Guppy: I think it's the last point you made there, Erica, around increasing confidence: just being able to build in trainings. We all wear different hats, and sometimes we're getting asked to do research, or a little bit of moderation, or note-taking, or these extra things. So increasing the confidence of the people doing these new tasks by building in training, building that education piece up front, builds that confidence and that muscle to then be able to iterate faster, so there's not this fear of, "I'm doing something new, taking on this lightweight feedback session or taking notes in this session. What data considerations should I have in mind?" When you have that training, you have that knowledge bank to go to: "Okay, I have that information now. I'm not floating in a sea of not knowing what I should be mindful of." I think that builds confidence, and that then increases speed and productivity as well.

Simple habits to catch your own bias

[00:20:40] Martin: Yes. And what about you, Yael? Any tips on iterating and speeding things up, and making sure you don't get caught up in analysis paralysis when you're checking your biases?

[00:20:52] Yael: What was said earlier about reaching statistical significance definitely resonated. Sometimes you can see the direction it's going; you have to have judgment. And sometimes you just have to let it go. We also do tests that are just copy, decoupling it so you're really sure, and we've found that changing the copy itself is really significant. Just a word change can be really impactful.

[00:21:27] I do want to talk about how to quickly figure out whether you have bias and how to mitigate it. There are simple things people can do, like creating new habits, that can help you with your day-to-day, so it doesn't become "Oh, how do I do this, how do I figure it out?" It just becomes a routine, and I think for someone in research that's really important. For example, when I start a project, I check in with myself and I write down: do I have any preconceptions about the user group I'm now speaking to? Had I done this in my example, I would have realized, but I didn't. Do I have any preconceptions about the area I'm testing? Do I already think I know what the problem is and what the solution is?

[00:22:16] Writing it down proves to me whether I have them or not. It's just a tool, and then you keep going back to it. It's not a big, dramatic thing that you have to take time over. Just make sure you take the five to ten minutes to listen to yourself and write down what you really already have as your existing narratives, and then throughout the project go back to it. Every week, just go, "Oh, now I remember. This is what I need to make sure I avoid." Of course there are other techniques that I shared in the presentation, but I think it's about raising awareness that we're all human. As Erica said, we all have biases.

[00:23:08] Someone once told me, "Well, if I write it down, doesn't that give it power?" And my response was, "Well, not acknowledging it doesn't make it go away." Raising awareness is the only way you can then figure out how to avoid it. So just take a few minutes every time before you engage with the user group and think: do I have any defensiveness, any ego that I'm bringing in? Just by acknowledging it, you can then start releasing it and being more objective.

[00:23:45] Martin: That's fantastic advice, and that's definitely something I learned from my days doing DevOps: if you're not good at something, if you find it painful, then do it again. Just keep doing it until it stops hurting, and then you get better at it. You build the muscles to be able to do this over and over.

Small steps to take next

[00:24:02] Martin: Gosh, this has really flown by. We probably need to start wrapping up, but we'll go around one more time before we do. I just want to thank everybody again for their time, and everyone who's been following along. As I said at the start, I really appreciated the real stories of lessons you've all learned. There were some great sessions, and you've all been open and sharing about them, even when they were painful, so thanks on behalf of everybody watching at home. As we are wrapping up, I was wondering if you had any last thoughts or tips on how people can iterate more quickly themselves. What small steps can the people at home take tomorrow, or next week, or next month, to improve how they do user research and how they handle data? Guppy, we'll go to you first.

[00:24:56] Guppy: On the question of how to handle data, I'm really excited about knocking on my cross-functional partners' doors and trying to get out of the silos we sometimes get into working on our specific teams. Knock on your privacy office's door and ask for help: what are they working on, how can you collaborate? Just that simple knock can build a partnership, potentially into something that might help your customers in bigger and better ways. So I think being collaborative, taking a step and saying, "Hey, what are you working on? How can we partner and make our product development process even better?" is just a simple step. You never know where you can build a partnership or a connection in your own company.

[00:25:51] Martin: Fantastic. This week has definitely told us that we're not on our own, so it's been a great week for that. Erica, what about you? Any tips for the folks at home on what we should do tomorrow, next week, next month?

[00:26:02] Erica: For content research, my favorite tool is visualthesaurus.com. If you're wondering what words to throw into a test, it's, I think, $20 a year for a subscription, and it has verbs, adjectives, adverbs. If you don't know what words to come up with for a content test, Visual Thesaurus is a great place to start. And I'd echo what Guppy said about making friends across your company. I think about my colleagues around the world. I am in Seattle, I am a white woman, I am 50 years old, I am biased. The more I can get input from my colleagues around the world, in different countries and different cultures, the better the customer experience will be. That's another humbling moment of working in UX: not solving for yourself, but getting input from all over, being transparent and welcoming that input all the time is important.

[00:26:56] Martin: Fantastic. And last but not least, Yael, what about you? Any tips for the audience in the last minute we've got?

[00:27:03] Yael: I'm going to echo what Guppy and Erica said, just from a different point of view. I think there are many times researchers work in silos, and there's a lack of leveraging the information already available in the organization. One thing we've really emphasized this year is bringing in these other perspectives: not just from where people are situated and what their background is, but from other teams. My example was us not leveraging the customer service team. We've now moved to the complete opposite: they're involved in our research planning, and we share everything, not just with management, of course with management, but also with the actual reps.

[00:28:03] We've brought in the technical team and QA, and that's widened our perspective so much, not just on how the organization works, but in understanding our users in much more depth than we did before. I think the research has really benefited from it: the breadth of ideas, and also the granularity of what we're actually testing and what the problem is. So: you're not on your own. There's so much knowledge in the organization; just tap into it.

[00:28:37] Martin: Fabulous. Well, we'll leave it at that: you're not on your own. Everybody, thank you very much for your time, and I hope you've been having a great UXDX so far. Enjoy your next session. We'll hand you back to Rory. Thank you very much.