Improving UX Research Quality with Cross-Department Collaboration

Kasey Canlas at UXDX USA. Video: https://youtu.be/FQCUUfz85Bk

Readable transcript: edited from the recording's captions for readability (fillers and false starts removed, punctuation and section headings added). Wording is the speaker's own. Timestamps are positions in the video. Names marked [?] could not be verified against the audio.

You're never actually alone

[00:00:00] Hello, I'm Kasey Canlas. I am a research operations manager at Genesys, and I'm going to be talking today about improving UX research quality with cross-department collaboration. And I will never have a title this long ever again, because it's terrifying. It will be three words at most. I have a picture here so you can see what I look like if you're far away. So there you go, I'm Kasey Canlas. I've been in research operations for five years, and I'm going to talk about some collaboration, some tips on how you can convince people using evidence that they want to see, and also just getting straight to the point, which hopefully I do well, because otherwise you'll all judge me very severely and I'll be super sad.

[00:00:51] Where I came up with this from was that oftentimes I am at conferences and I hear people say, I have so much to do. I'm on a team that's really small, or it's just me. I'm the only person doing this job on my team, and they give me a million things to do, and I feel overwhelmed. I can't do it. I'm underwater. And then they'll say, I don't know how to do this all by myself. And that's where I say you're not all by yourself. You are at a company. You have a team, you have an organization, and guess what, you're all working towards the same goals. So you're never actually alone.

[00:01:25] And yes, it feels that way, but there are a lot of people doing stuff that can really help you. So I just wanted to touch on that and give some of my previous experiences, and hopefully that'll help. It doesn't matter whatever the size of your team is. Look past that and think about who you have in your organization that you can collaborate with.

Cross-functional structures and operations teams

[00:01:46] Look at that, I am really good at this, I hope. What I'm showing here is a cross-functional structure that I've worked with before. Essentially you have your PMs, you have your devs, it's divided by products, and then you also have your UX researchers and your UX designers. Often there's a ratio where it's like we have 5 million devs and we have 1,000 PMs and we have three researchers and we have, I don't know, 10 to 50 designers. But still, it's not great. So then you're like, I guess I'll just work on a few projects. That's cool, it's totally neat. And there's a reason for that; you don't want to have a crazy amount of everybody.

[00:02:29] Oh sorry, I missed one. But I wanted to tell you it's basically divided by products wherever I've worked. And then we talk about operations teams. I'm very fortunate that at this large company I work at, I've got PM Ops, I've got DevOps, I've got research Ops, there's design Ops. You can collaborate with those people, because they're the interconnective tissue that know a little bit of everything, and they're often trying to fix things that you're trying to fix too. You just don't realize it, because you just don't have that conversation. So part of it is just getting to talk to them and know them.

[00:03:04] Up here, I don't think I have to go through this, but I will. It's a process, and you're probably familiar with it, but I can go through it. I'm not going to read it off of that thing because it's teeny tiny, so I apologize, I'm going to look over there for a split second. Basically, PM and UX Ops, we collaborate on generative research. We develop wireframes and prototypes. We test concepts and usability. If everything goes well, which we know it always goes well, there's never anything that goes wrong ever in its life, then we'll get dev resources scoped and we'll build. Final reviews, beta if needed, which maybe, maybe not, product release, and then we're going to track in analytics and make sure we did the right thing.

[00:03:42] But what I want to talk to you about is these things: less is more, follow that research, break it down, and find the right people. These are our topics. I'm sure you're like, I totally know what she's going to talk about now, those are very descriptive, thank you so much.

Less is more

[00:04:00] It's hard for research to make an impact when you're not saying the right words, the right things, for the people that want to know. For example, if I was like, hey, are you going to die of boredom at my presentation? Do you think that you've learned anything from me at all? Do you think you were paying attention? Do you think you were maybe sleeping after lunch? I don't know. But it's quality over quantity.

[00:04:27] So to make your research have an impact, here's what you should do. I hope you're reading this. I hope you can read it all. I mean, it's definitely 8.5 point and it's very, very visible, there's no problem. I'm kidding. How many of you actually started reading this? How many of you know what this actually means? Yes, I'm very impressed by you. I would not have; I have already turned off from this slide. Please, no, don't do this. This is terrible. Here you go, it's little pigs. That's just little pigs, that's all it is.

[00:04:57] But a lot of you were probably like, yeah, I'm not reading that. I just want you to tell me the important part. Just tell me what matters and let's move on. That's awful, I can't believe you did that. And a lot of presentations end up being that way, where they throw everything into it because you're trying to prove here's how well I did at the research, here's how great I did at everything. And maybe not everybody cares about that. It's not that it's not good, and it's not that you shouldn't have it there in the appendix and say, here are all of the things in the background that I did to validate this. But you want to make sure that what they see first is what actually matters to them.

[00:05:33] I'm going to talk about that a little bit more, and I have some quick tips here. First thing: make it easy to find and understand. Don't give them everything. Chunk it out. Think about what this actually is, and write an executive summary. Make sure you tell them why we did this project. Give them context, and give them three to five important bullet points, because that's where you'll get them interested. That's the moment where they go, yeah, I do want to learn more about this. Maybe I will go to the details, maybe I'll go to the appendix. If you don't give them this, they're going to look at that report and go, I skimmed it, it was nice, you did great. So just cut out the busy work, figure out what they want and put it down there.

[00:06:13] And then make sure that you put everything somewhere that they can access, because you shouldn't just be thinking just the researchers need to see this, just the designers need to see this, or only PMs. Anybody in the company that works with product should probably have a chance to have access to those insights, access to your findings, and you're actually doing them a disservice by not putting everything there. And it's not just putting a whole bunch of reports in one place. It's going, oh, I actually have a presentation, here's my deck, because maybe they're visual learners. It's going, hey, here's a recording of me talking, because maybe they want to hear you talk and see you present.

[00:06:54] If I handed you this deck, you're not going to get all the funny pieces and parts of me that are what make my presentation good. You'd be like, I don't know what she was talking about. That deck's okay. It's not great, but I'm great, and the deck's okay. So make sure that if you do have a presentation, if you do record it, you put it in there with everything. Throw the prototype in too, because otherwise you're asking them to get context of something that may be clickable with no clicking. They're just going to look at it and go, I kind of see what they did there.

[00:07:22] So it's really thinking about curating to your audience, not going crazy and throwing everything at them, because they're not going to pay attention. And that's why I say if everything's important, nothing's important, and nobody cares at all. That's a rough thing to say, but it's true. We want to get stuff fast, we want to keep moving, and we want to make sure that we don't slow anybody down. So you have to really think about who your audience is, because maybe you're going to give that same presentation to three different people, but you don't want to have the same stuff in it. And that's where you throw it all in the appendix: here's everything I could possibly ever want to share with people, but I only need these three things, because that's all they care about.

[00:08:01] Sound good? Maybe, maybe not? Is this making sense? Am I helping anyone? All right, yes, I like it. I like that we have a little bit of interaction here. Sometimes it's silent and I think that you've all just gone away. Luckily the lights aren't bright enough, so I can actually see you sitting here, so I'm very excited.

[00:08:21] Anyways, one of the things I also want to make sure that I say, before I get into everything that stakeholders care about: as you are talking to them about UX, if they seem to be hesitant about anything, or maybe they don't understand things, take the time to educate them. You cannot expect everybody to know everything about UX like you do. You need to be able to say, you know what, hey, can I share something with you? Did you have any questions? And really get into it and help explain to them what the UX things are, what the principles are. If they have any questions about things, help educate them, because otherwise you're just expecting them to be an expert when you're the expert. And the whole point of being an expert is being able to share your knowledge, so you have to give them the context. Don't just expect them to figure it out. They will not figure it out, and I mean that lovingly, but I also mean that seriously.

What product managers care about

[00:09:15] Next. I don't have to do this, but I like doing this; it makes it feel like I'm actually doing something, so you're just going to see this continuously. Anyways, here's PM and product management, and if there are any PMs in the crowd, you're probably like, this ain't true, she did a terrible job, and that's okay. But basically, PMs, product management, what do you need to know? You want to know, okay, what are the user's needs? What do we need here? Are we going to be able to make this project? Do we have enough dev people? Is it prioritized? And then what findings and recommendations are there? And well, is it usable? Do we actually want this thing? And if you don't have that, then why are we even making this?

[00:09:58] What do you want to do if you really want to communicate to them? Yes, you can hear me. If you want to communicate to them or you want to convince them, what you want to do is make sure that you talk about the efficiency for the user as they're going through the product, as they're doing things, because that's one of the things they care about. They want to know that their product is able to be used and it's working well. And then you also want to make sure that you have empathy for users. There are quotes and videos and stories, but if you use those, make sure you get permission from your participants, because you can't just show that to everyone. If you don't ask permission, you shouldn't do that.

[00:10:36] So try to get permission. Say, hey, would you mind if we showed your videos or quotes or whatnot? Anonymized quotes. You can't anonymize videos unless you're getting really fancy and shady. I don't know how it works. I didn't say shady, I didn't mean that; that was just a random word that came out of my mouth. But anyways, I say all this because you're like, eh, efficiency for the user in the project is not exactly the same as empathy. And it depends on what the ideal is versus what the reality is.

[00:11:02] The ideal is they want to talk about here's the empathy for the user, we put the user at the center of everything, and we really thought about them as we were creating this feature. But in reality, they're thinking about how many features can I get adopted? What is going on? That's what matters to people, because you hear that so many times when you're in all-hands and they're like, we released 5 million features, are we great? And then you're like, were they the right features? Just wondering, just curious. So it depends on where you're at with your PMs. Maybe they're thinking ideals. They're thinking, yes, we want to do quotes, we want to support it that way. But there might be people that are like, we just want to get this moving. So it's knowing what they're thinking about and trying to appeal to them.

What developers care about

[00:11:51] Once again, I can't stop this. Yeah. This is dev, and there should be devs in the audience, maybe, I don't know. Do we have devs in the audience? You want to disagree with me? It's quiet. No? It's very important for devs, if they want to build something, if they want to implement something, that you're specific. You tell them what exactly you need from them. Anything that you don't put down in the brief is open to interpretation, and guess what, I'm going to interpret something different than you are. We don't have the same context. We don't have the same backgrounds.

[00:12:24] So as a designer, if you are giving them things, or as somebody who is a researcher, make sure that whatever you give to dev is very clear. It's very much talking about: here's what we expect, here are the specifications that need to happen, here are the requirements we need to complete this thing. And we're going to look at it and we're going to say, okay, these are actually the important things out of these 50 things, and these are the 40 nice-to-haves. And that's what dev cares about too. It's like, tell me what I absolutely need to have in there. Don't tell me about the other 40 things, and then we'll put that in V2. That'll be our next one. It doesn't need to be in the MVP; it's a nice-to-have. I'm not wasting my precious time building something for you when I could just put that in the next iteration after we've learned more about it. So make sure that you are clear about that.

[00:13:17] And then also, I want to make sure I didn't miss anything here. No, we're good. You want to make sure that you have quantitative data. Devs love quantitative data. They like facts. Empathy is really lovely, but it can also be fuzzy if you're someone who deals with logic and you just want to know, straight to the point, how many people use this, how many people like this. Give your devs more quantitative data, not as much qualitative data, because they're just trying to say, can I build this? Do we have the resources? How much are we going to be looking at? And is this something where I'm trying to build Disneyland on a shoestring budget? If it is, they are going to tell you, no, that's not feasible. And they'll know that faster if they don't have 50 things to look through. They'll know if they have 10, and then an appendix with the other 40.

What executives care about

[00:14:04] Next thing. I'm never going to stop this. VPs, executives, anybody above the VPs. I'm going to get water at some point. What do VPs want to know? What does anybody that's an executive want to know? They don't want to waste their precious time, because time is money and they make a lot of money. So it's: tell me the effect, show it to me, make it easy to understand, and then I'll take whatever I need to know from it and I'll go do my stuff. It's very important to know how to relate it to the business. They want to know, is this going to impact anything? They want to know, all right, what is the big advantage here? How are we going to beat the competition? That's what they care about, because that's what they're being asked to deliver on.

[00:14:52] And then, oh hey guys, I'm so awesome, I brought a mint. This way I don't have to keep doing this with my mouth. I was hoping somebody would bring me a water, but it's been very silent and very sad. And I'm really sad about AJ, because I thought he would bring me a water by way of supporting me, but instead he just gave me his belt so my pants wouldn't fall down. Hold on a second. Literally, this is his belt. I don't know if you saw it, but it's fantastic. We're the same size. I even used him as a dummy. Not as a dummy, I mean that nicely, but I used him as a mannequin so I could make sure I got the sleeves right. It was really good. Yes, thank you.

[00:15:30] There we go. I have eight minutes and now I can almost talk. All right. I talked about visualizing. I mentioned executive summaries. Just tell them the basic facts, the context, here's why we did this, and three to five important things. Don't go more than that. Don't you dare give them a whole page, because guess what: skimming. And then they're going to go and inform everything from that, and you're going to wish that you had said, that wasn't the important part, here was the important part. And it gets way more expensive to fix after the fact. So yeah, learn that.

Follow that research

[00:16:03] And then we've got follow that research. Where am I at on slides? Oh boy, we've got to speed up. Okay. Don't just wave goodbye and let it go off on its own. You did not raise that research to just let it go and live its own life separate from you. You want to check in every now and then and be like, you okay? All right, do you want a meal or something? I don't know. But you need to follow it and see where it's going. There's a Mitch Hedberg [?] thing where it's like, I'm just going to ask it where it's going, I'll catch up with it later. And that's what I want you to do.

[00:16:36] Okay, so what you're going to see here is a basic standard. I wrote a report. I presented my findings. The stakeholder's like, I love that, that's really cool, great. And then you close the door on that project and you're like, to the next project, let's go. And then that project just disappears into thin air because you haven't looked at it, and now it's imaginary. And that's one thing that will affect your research quality, because if you don't continue to find out where your research goes, you're not going to know the impact you have on your entire company and what the actual... I really need water. How many times do I need to say this? Guys, I love you, but somebody bring me water. Save me. I'm not joking. There's no one. It's so sad.

[00:17:25] Yes, thank you, lovely, I appreciate you. I was like, man, this mint isn't doing it. Guys, I was going to carry water on stage, but I was like, no, we're going to be very classy. Thank you, I appreciate that.

[00:17:45] So now, what I think you should actually do, ta-da: talk to your stakeholders. Don't let them go away. Be like, hey, after I give you that presentation, do you have any questions? And just make sure that they let you know if they have any questions so you can explain it, because you don't want them having the wrong idea and going off and doing something when you weren't clear about what you were delivering and what you were telling them.

[00:18:16] And then you want to make sure that you confirm the next steps. With research, we know it's a lagging indicator, so we know that we're not going to find out the impact of it for weeks, months, years, I don't know, decades. I hope not decades; that would never go well. But anyways, if you have that, you're going to want to schedule those future check-ins. You're going to want to make sure that you say, did research lead to any important decisions? Did you follow my recommendations? And then later on you're going to want to follow up and go, hey, you told me that this had informed some decisions, what ended up happening from that? Did you cancel some projects from that because you were like, oh, this makes no sense? Did you add new projects? Guess what, that's impact. Make sure that doesn't get lost in the ether.

Break it down

[00:19:00] And then we are going to go here again, ta-da. Five minutes, we're doing great. We're going to break it down, and not in a dancing kind of way. I'm not that cool. I'm not a great dancer when it comes to cool dancing. What I mean is, when you are a researcher, and I say researcher meaning you're doing research, this doesn't have to be a researcher, this can be a designer as well. So when I say that, think of whoever it is that does research. That's what I mean; it's all-inclusive. Make sure that you are able to break it down after you build it up.

[00:19:31] Because when you have research, you're doing interviews, you're talking to people, and you have all these pieces and parts. Then you're building this story and you're making it work. Great, here's all this evidence. But then if you don't break it back down into little pieces, that one big chunk of everything will get applied to things and no one will actually look at it. They won't realize what pieces and parts you're trying to talk about. So it's important to break it down, and what I mean by that is atomic UX research. This is not mine; I linked it there, and you can see it when you get the deck.

[00:20:02] I want you to think about it as product versus project. Don't go project to project. How is this impacting the product? And then make sure that you keep those findings in rotation. Does this need to go somewhere else? Should this be a finding that provides evidence for something else? It shouldn't just end with the project. He breaks it down into experiments, facts, insights and opportunities. You can read it there, because I only have three minutes and 45 seconds, so we're just going to roll through this. But essentially what it is is we have facts, insights and conclusions, and there are so many different ways that they can be put together.

[00:20:38] But if you don't break it apart, you're not going to put it together. No one's going to sift through all that information after the fact. It's important, once you finish that project, that you take that time and you break it down so it can be applied to everything else. Because otherwise you are leaving so much evidence, so many learnings, findings, recommendations, insights, you name it, all on the cutting room floor, and we're better than that. So please, please break down your research in a good way. It's also going to be able to make more connections across the board, and you're not going to have as much duplicative work, because you'll be able to look and go, oh, we did that already, so no, we don't have to do this one. We already have it. We already figured that out.

Research repositories

[00:21:19] And then where do you store it? In a research repository. I'm sure you're like, oh cool, what tool do you use? It doesn't matter about the tool. I'll talk about that in a second. But essentially it's making sure everything that you have is in a shared knowledge network. It's easy to find. The company has access to it. Preferably it's anonymous research data; you're not putting your raw data in there. But it's everything that they need to go and look at and be able to find their research themselves, so they're not relying on a researcher to pull that out.

[00:21:52] We know that researchers are often the contributors. You're often the ones that are in there curating it and stuff, but everybody from the company can go in there and look. And it's really nice if they're like, hey, remember that project that you did on XYZ? Guess what, I want to know more about that. You don't have to take all that time. You just have to go into wherever you put it, SharePoint, you name it, and say, oh, here it is, here's the link, you can go look at it. And then they'll be able to see what else it matters to. Preferably it's linked to something else, so they can go see what pieces and parts there are.

[00:22:20] It also just allows everyone to see everything, because it's really helping them to inform their decisions through research. Otherwise, as other speakers have talked about, it's just going to be off of their best guess, their perspective, and it's not really going to be informed by research. And do you want that to be your business goals? Right. Love it.

[00:22:44] So now we have what matters most. Structure it. Make sure you think about it with your end user in mind. I know we're really good at that, because we are user experience. It's thinking, what are they going to need to do, and what are they going to need to see, in order to find the information that they need and then move on. And then make sure that you have a taxonomy. Tags are going to categorize it. It's going to make it so they can search easily and they can find, oh, here's a project. We saw Richard have his whole taxonomy with all the beautiful colors and everything you could think of, and that would help people find stuff. So it's making sure that it's logically done in a tagging way. I can't wait to find a librarian at some point so I can have them do this for me, because I'm not great at it. I need someone. And then it'll make sense and it'll be easier for people.

[00:23:29] And then I've said company-wide access, limit the raw data, and maintenance is required. We saw AJ, who gave me the belt, lovely. Anyways, he said that you can't just create something and then automate it and leave it, let it go: it'll be fine, it'll keep doing its own thing, it'll maintain itself. You have to have somebody there to go through and be like, are we following the processes? Are we doing this right? Or is everything going a little bit haywire? Because otherwise, if you just set it and forget it, it will have gone haywire at some point in time. And if you catch it sooner, it's better than catching it later, believe me.

Find the right people

[00:24:06] Find the right people. How many of us love recruitment? We all love recruitment, right? It's our favorite thing of all time. I thought so. Quality participants means quality research. How many times do you go, yeah, I'm looking for elementary science teachers. I got 20 of them, I got so many people, and you reach out to them and you're like, this is great. And then you also have criteria, like, I need to see if you have five years of experience. This is additional criteria, and your prospects start to dwindle, because then they're like, no, I don't have five years of experience. You're like, that's okay, we've still got plenty of people. And then it's, oh yeah, they need to specialize in STEM integration. You're like, cool, that's fine, we're good.

[00:24:46] But then now you only have five people, and maybe some people didn't respond back and they're not going to, because they're just like, no chance, I don't want to be in a study, I don't want to do it. Now you have enough people to actually get your findings, but let's say maybe someone cancels. Let's say maybe someone doesn't schedule. Let's say someone just ghosts you, rudely. This happens. I mean, once upon a time it was a beautiful, perfect world and every participant showed up. They scheduled, they had great things to say, and did not lie about who they were, and it was just so easy. You could say, I have five people and they're going to be here and it's all going to go great. I'm never going to have to change my criteria because I couldn't find enough people, and then have my research affected because I don't have the right people.

[00:25:36] And so I'm going to tell you that this is where you're going to look at your company, because your company has people that talk to your customers. And the great thing about that is it's everybody that's customer-facing. That's sales. That's if they run a message board for customers. That's if you have people that are working with partners, or somebody that's onboarding your customers, anybody like that running anything. And especially PMs, as funny as this sounds. You ask PMs, have you ever interviewed anybody? They'll be like, no, definitely not, I wouldn't do that without any processes. Like, cool.

[00:26:14] And then you say, oh, where did you get that data? And they're like, oh, well, I was talking to so-and-so and they told me this, and so that's how I'm going to inform everything. And it turns out they have a really long list that they've just kept in their back pocket and never shared with you. So it's great to ask them if they can share those people with you, because your PMs often know somebody who is using that feature or who is doing this task or something like that. It's helping democratize that and saying, oh, if you know somebody for this project, tell us about it, and maybe you can provide the introduction to whoever it is who can provide the participants.

[00:26:46] Because that's how we get a lot of participants. You are literally talking to sales. They know everything about this customer. They're like, this is our best customer, they are power users, and we also know a couple of people that aren't power users; they would be great for this study. But if you don't set up the one-on-one and you don't talk to whoever it is that seems to interact with customers, you're not going to know about it. So finding those people is very important in order to find the right participants.

Wrapping up

[00:27:13] I'm getting real close. I'm wrapping it up, it's okay. All right, yeah, she just did this and it's getting intense. Basically: quality over quantity. Don't let your research go unsupervised. Make sure it goes where it should and people are following recommendations, and if they're not, make sure you prioritized it and you've made it clear and actionable, because otherwise you're just giving them a list of here's everything. And then breaking down your research: break down your research so you can put it into all kinds of pieces and parts. It's great. And the last one is finding the right people: who knows customers and who's willing to help you. If you contact somebody and they're like, yeah, I don't want to, then go to the next person. You don't need to deal with him, it's fine, or her, or them. It's fine.

[00:27:57] And then lastly, I know this seems like I just told you everything to do, and now I'm going to say, no, don't believe me. You do need to talk to whoever your stakeholders are, and you do need to find out what they care about. I'm giving you a broad generalization, and everybody's an individual, so don't take this as the be-all and end-all. I said that backwards, but yeah: all teams, stakeholders and projects are unique and different across companies. I'm done, with 23 seconds to spare.