Democratizing Research

Caylie Panuccio, Anna Nguyen, Vineet Kumar at UXDX APAC. Video: https://www.youtube.com/watch?v=DQZ0I1XhnFw

Readable transcript: edited from the recording's captions for readability (fillers and false starts removed, punctuation and section headings added). Wording is the speaker's own. Timestamps are positions in the video. Names marked [?] could not be verified against the audio.

Introductions

[00:00:00] Anna: Thanks so much, Rory. Welcome, everyone. My name is Anna Nguyen and I am a researcher at Dovetail. We're a platform that allows you to store your research and analyze all that customer data. Today I'm super excited to welcome you both, Caylie and Vineet. Welcome.

[00:00:17] Vineet: Thanks for having us.

[00:00:19] Anna: Of course. Maybe, Caylie, do you want to kick us off by telling us about your role at Atlassian?

[00:00:26] Caylie: Yeah, sure. I should mention I'm getting over a cold, so my voice is quite croaky at the moment. Excuse me, so apologies to those who are watching. I'm a researcher at Atlassian. I've been with Atlassian for about four months. I work in our Open DevOps business, which helps our customers do DevOps: shipping their code and getting it out the door into people's hands. Previously I was at Coles for about nine months, which is one of Australia's largest supermarkets, heading up the research team there, and prior to that I was at SEEK, which is one of the [?] job boards. That's all for me.

[00:01:05] Anna: Thanks, Caylie, and feel free to take any water breaks if you need. I totally understand the feeling of being live and needing to take a water break, so totally fine. And Vineet, tell us about you and your world at Postman.

[00:01:17] Vineet: Yeah, I think I just did a quick introduction with the talk, but just to reiterate, I work as the director of UX at Postman. I've been there a while, so I have seen it scale. I've had the luxury of having a lot of failed experiments in that time, so hopefully I have learned a few things from that, and maybe I can share a few of those.

How research is structured at Postman and Atlassian

[00:01:43] Anna: Awesome, thank you both for your introductions. Speaking of research, maybe, Vineet, I'll start with you. I'd love to hear how research is structured at your organization.

[00:01:54] Vineet: Absolutely. We started off with research being centralized, but always working within specific domains for specific initiatives, so at a time you would always be focusing on a certain part of the product at the end of the day. But over time it became very, very apparent that there were two kinds of research that needed to be done. There was a certain set of high-level research on our users and the user experience which couldn't be done by any team other than a centralized team. They needed to have that global perspective. But there were also smaller research initiatives at a feature level or at a product level, which were much more democratized, in the sense that product managers were doing it, UX generalists were doing it, et cetera.

[00:03:05] Over time we've gotten to a point where we now have a dedicated research and strategy team that looks at centralized research and to some extent governs the research requirements of other product teams. But our product teams are built out in a way where they're completely free and independent to do any research they want on their own. In practice it's mostly secondary research, but yeah.

[00:03:33] Anna: That's awesome. At Postman it sounds like most of the research is centralized, but it sounds like the researchers also have the freedom to conduct research they're interested in, which is really great to hear. Caylie, does it differ much for you at Atlassian?

[00:03:48] Caylie: Yeah. Excuse me, so sorry. I would consider our research function to be a hybrid. We have qualitative researchers embedded in product teams. For example, I work with two product squads, and they are effectively my product team. I go to all their meetings, and I attend any decision-making session with the product manager, the engineering manager and the designers. However, the researchers report up into a centralized research group. I report to a research manager for my product space, and we all ladder up into the head of research, which sits within our customer experience organization.

[00:04:25] Within research, while our qualitative researchers are embedded, we have a centralized quantitative research team and a centralized ops team, and they help all of us get stuff done. I guess our closest friends in the organization, apart from other researchers, are designers. I think that structure works really well, because as an embedded researcher you understand the product a lot better and you work with your stakeholders a lot more closely. As a result, the research you conduct is ultimately more fit for purpose, and it gets used because you understand the product and the team understands what you do.

Giving context without overwhelming people

[00:05:01] Anna: Yeah, that's awesome. I totally relate with that. Being embedded in a team, you can really dive into your product team's goals and the pain points of your customers. It sounds like both of you have this common theme of having a centralized UX or research team. I'd love to hear from either of you how you balance giving non-research-facing team members enough context on research without overwhelming them. Vineet, I don't know if you want to go first?

[00:05:35] Vineet: I'll let you have this one.

[00:05:38] Caylie: Yeah, this is the tricky one, and it's something I've seen happen in all the places I've worked. I think it's a balance of pushing the information out but also bringing the information in, depending on context. Excuse me. When I say pushing the information out, I'm talking about running showcases with your stakeholders in the broader organization, disseminating research insights through newsletters or updates, either via email or Slack or whatever it is, running workshops to share insights when they're really fresh, and creating a research repository, even if it's rudimentary, so people can jump in and soak in the data if they want to. Just don't get too precious about who has access to what. You ultimately want your research to be used.

[00:06:25] That's one side of it, pushing the information out there, but there's also bringing information in. I think as a researcher you have a responsibility, once the research is complete, to keep bringing up that research, to keep sharing and keep reminding people it's there. Get yourself into the right forums, get yourself into the places where decisions are being made, and repeat that research information, repeat that data. Also rely on triangulation: look at research data you might have collected and pair that with maybe some quant or analytics to really strengthen your argument. Rely on other people to share it and help you advocate for your research and for the data. You've got to be everywhere. To summarize, it's a balance of pushing it out but also bringing it into the places where you are.

[00:07:14] Anna: Yeah. Vineet, anything to add to it?

[00:07:16] Vineet: Yes, I think Caylie pretty much covered all of it, but I just wanted to reiterate leaning heavily towards insights at Postman, because that's the kind of information that a centralized research team can provide which individual research functions in product teams might not be able to. I think when you're working in a product team and listening to your users on a day-to-day basis, you're getting on calls with them, you're getting a lot of passive feedback through feedback channels, and you're doing your own secondary research on top of that. The hardest part here is to be able to figure out which feedback is relevant, how much feedback is noise, and how much feedback you can actually work upon.

[00:08:08] The danger there lies in relying on it anecdotally. At the end of the day, the centralized research team is building out stuff at a higher level, so they are the best people to gauge what's noise versus what's not. That input coming from the centralized teams, that form of dissemination, is invaluable in my opinion.

Pros and cons of decentralized research

[00:08:37] Anna: Oh yeah, totally agree with that one. We've spoken about centralization. I'm keen to hear your experiences, perhaps being in a decentralized research team. What are some of your thoughts on pros and cons? Maybe, Caylie, you can kick off this one.

[00:08:52] Caylie: Yeah, sure. Sorry, you're asking about pros and cons of centralized or decentralized research?

[00:09:00] Anna: Decentralized.

[00:09:02] Caylie: Yeah. Because I was at SEEK for a very long time and we had quite a decentralized research function, what I'm going to talk to will be reflective of those experiences at SEEK. I think the pro of decentralized research is that designers can really take charge of their own research for their own product. It's really beneficial to have a designer who understands their customers, who sees customers interact with their designs, because it builds empathy for what the customer goes through. If a designer sees it firsthand, they're more likely to believe that's true, and the same goes for product managers too. I think empathy is the biggest pro, actually. It makes it easier to really push for the change that customers want if people are seeing it with their own eyes.

[00:09:47] It also means that the researchers can focus on tackling some juicier problems, so not just testing designs or concepts, but looking into future-focused research or into the strategic problems the business wants to tackle. The cons, though, and excuse me, I'm speaking from experience here: if you're not careful, your researchers can turn into a bit of an ops function, which is not something that all researchers want. If you are going to turn towards decentralized research, I'd recommend really carving out what research your designers and product folk do and what research your researchers do, so your researchers have that purpose.

[00:10:29] The research can also become quite disconnected from the products if researchers are not working directly on the products themselves. As a result they may not have the context they need to deliver really fit-for-purpose insights. So I guess it's important to give your researchers... Sorry, phones going off. It's really important. Sorry. I might end it there.

[00:10:56] Anna: Vineet, I'd love to hear from you. I was spying on your talk just before this, and it was really cool to see that the research team or UX team was quite small in 2019 and 2020, and today it's growing quite a bit. Have you seen any of these pros and cons that Caylie has mentioned around decentralized research?

[00:11:17] Vineet: Yeah, absolutely. The way I like to think of it, you have a linear scale between centralized and decentralized, and what you're trying to find is the balance on it. If there's too much centralization, you're probably not building enough empathy, because you're not working deep enough in the product to actually understand user pain points at some level. If there's too little centralization, you're always focusing on the immediate problem. Going back to what Caylie mentioned about having the freedom to think longer term and do research that maybe affects the roadmap two years down the line rather than now, that is just not possible when you are doing decentralized research, in my opinion.

[00:12:12] That's the linear scale. It completely depends on what scale of a team you are and what's the right kind of balance you're trying to find, and depending on that you might want to choose a level of centralized or decentralized.

[00:12:26] Caylie: Yeah. I just remembered what I was going to say before. In terms of another couple of cons of having decentralized research: when researchers are working on those bigger, more strategic things, that's really good, but it also means they have less context on the product teams and what they're working on, which means the research can become less useful. You might also end up with researchers getting swamped with requests to review research work from other designers, and it can get a bit too much. You really want to have that balance between the researchers doing some strategic work but also being close enough to the product that they can execute on those research projects and really understand them when necessary.

Training non-researchers to run research

[00:13:11] Anna: Thanks, Caylie. I think you both mentioned a theme of balance, which is super relatable and super interesting. I can imagine you're both super busy in your world of UX and research. I'd love to hear how you balance giving training to non-UX-facing roles and letting them feel empowered to run their own research. Caylie, happy for you to take this one.

[00:13:40] Caylie: Yeah, sure. Excuse me. This is really about making sure you're getting maximum impact from minimal effort, because training UX designers and product folk and whoever else to run research is a lot of work. When I was at SEEK, the first thing I did when decentralization was getting talked about was set up a wiki of how-to guides and templates, so designers and product managers could self-serve as much as they needed to. I find that typically designers and PMs are conducting things like user interviews or concept testing, so maybe start off with some guides around how to run user interviews and how to run concept testing, and set up some templates that people can just take and edit to suit their own purpose.

[00:14:28] You could also set up some, excuse me, office hours. This cough, my God. Rather than having people coming at you with lots of requests to review things, getting them all to come to you in one hour will mean it's not going to take you as much time to review the different pieces of research that are going on. Even just basics, like how to ask questions that aren't leading or how to recruit for a particular segment, all those things are really useful. That's all I'll say there.

[00:15:04] Anna: Awesome. Vineet, do you have anything to add to this, like how you keep the team [?] on track and having enough training to feel confident to run research?

[00:15:15] Vineet: Yeah, and this is where I just want to bring back the idea of the centralized team working with product teams on certain low-level initiatives, just because the centralized team gets to show everyone how it's done. You have the product managers and the designers working in the product team, who are actively learning from centralized UX researchers how they do research, and to some extent they are replicating it during the initiative. Once that initiative is done, they come out of it a lot more confident and independent to do it themselves.

[00:15:58] Obviously there's a lot of educational material and wikis and guides around this, but I feel like that one interaction of active listening is just super helpful. It also builds a certain relationship between centralized and decentralized teams. Once you've done a run-through of how a centralized team would do research, the designers and the product managers in those product teams know who to reach out to if they need certain questions answered. They know who can help them and guide them. It's almost like having a mentor available whenever they need some help with research. At the scale we're at so far, that's really helped, and that's worked pretty well.

Getting research used

[00:16:52] Anna: Thanks, Vineet. It's really cool to hear from both of you. It sounds like at Atlassian, and in your past roles, you have templates and a kind of theory there, but it also sounds like it's really important to have that listening ear, that team that people go to when they do need that extra support. That's awesome to hear. I might actually sidetrack a little bit. Caylie, I was listening to your talk yesterday around building an organizational user research culture. Super interesting, super cool. Something you mentioned is that democratized research is really hot right now, and we totally agree with that. But the other thing you said that was really interesting was that doing research is the easy part, but it's actually the research getting used where things get a little bit tricky. I'd love to hear how you let the research live and breathe and have impact, especially research conducted by designers and product managers, for example.

[00:17:54] Caylie: Yeah. I'll start with the designers and PMs. I've found in the past, and this is just speaking from my own experience, that research conducted by designers and PMs is typically more likely to be used. I say that because it's usually more tactical. It's usually conducted on the product for that quarter's initiatives, so it might be things like improving conversion or doing a bit of discovery for a particular initiative. Because it's in the here and now, it's very topical [?] and the team are ready to deliver it.

[00:18:24] When you're a researcher, maybe you're not doing as much of that tactical stuff and you're doing more of the strategic stuff. That's where you've got to get a little bit sneaky, or at least a little bit smart with how you think about where your insights are going to go. I've heard this referred to as the research long tail: what happens to the research after it's complete. When you complete a piece of research and it is really strategic, you might do an initial share-back with the stakeholders and go, right, this is what we found out, and people go, oh, that's really interesting. And you're like, okay, great, but that doesn't mean it's going to be used necessarily.

[00:19:02] This is where you start to ask your stakeholders, people who might be in the strategy team or people who are higher up in the product chain: what document does this research live in? Does it live in a strategy deck? Does it live in our plans for the next financial year or something? Try to find out what those longer-term artifacts are, so you can insert your research findings into those. It's tricky because it's different for every organization, but an example to share from SEEK: we were looking at our strategy for the higher-level product spaces for the next couple of years. I just happened to hear from one of the strategy managers that this workshop was going on, and I said, ah, it sounds like you could use some longer-term research insights, invite me to that workshop. So I just invited myself.

[00:19:55] I guess the takeaway here is to be a bit cheeky. Don't be afraid to invite yourself to those forums where decisions are getting made if you hear about them. And also be really upfront and ask these key people where they think these insights should live, what they are contributing to, and where that is documented, so you can get that research in there.

[00:20:16] Anna: Well, that's awesome. Love that little cheeky tip there, Caylie. And Vineet, what about you? Any thoughts on this, anything you'd like to add?

[00:20:23] Vineet: Yeah, I totally love that part about just getting there in those meetings and getting stakeholders to listen to it. It's especially effective, I think, with leadership. The other part to this is probably more bottom-up research adoption, in the sense that what we've found is, if research and product and design are all at some level talking the same language, or using a framework that effectively communicates things that the product managers are interested in, things that the designers are interested in and things that researchers are interested in, there's some amount of communication happening.

[00:21:09] I've talked about this before: for us that's the Jobs to be Done framework. If everything in Postman is being framed around a user job, it just makes it easier to consume, to some extent. Product managers start off their problem space exploration with jobs, designers are designing workflows based on those jobs, and researchers at the end of the day are validating or fleshing out those jobs more. It sounds very pretty, and it is super idealistic when I say it that way. In reality it's a lot more dirty and messy, but in principle, just having that common layer of communication makes collaboration between research and other teams much more effective.

Who owns democratization

[00:22:00] Anna: Thanks, Vineet, and yeah, it does sound pretty when you outline it like that. Vineet, I actually have a question for you. In your talk just now about foundation teams for UX, you mentioned that the UX team sits in two pillars: product design, and then your UX foundations team. A bit of a juicy question, but who do you think owns democratization? Is that both of those teams, one of those teams? Thoughts on this?

[00:22:28] Vineet: I don't think democratization is necessarily a vested interest for foundation teams, just because the entire problem they're trying to solve is centralizing stuff. There's stuff we've seen go wrong to a point where we absolutely need to centralize it, and that's why some of those foundation teams exist. For the product design teams, a lot more so. But to some extent it's an org-wide responsibility. I don't think you can take a decision on the structure of the design org without getting feedback from engineering and product, because everyone's going to get affected by it at the end of the day. At some level, democratization of research, or of whatever it is, is going to be an org-level decision that needs to have the go-ahead from everyone.

[00:23:33] Anna: Interesting. And Caylie, what about you? What are your thoughts on whose job it is to democratize research?

[00:23:39] Caylie: Yeah, I'm smiling because it's quite a funny question. While a lot of researchers might say it's researchers' job to democratize research, because you want to put in those guardrails around how research gets conducted and make sure it's of a high quality, I do think it's everyone's job. I say that because there is no point in a research team putting together a bunch of resources and training if people aren't going to take them up on it. The other folks in the org who might want to be doing research, A, need to want to do it, and B, need to want to do it well. Research isn't just having a conversation with a customer. You need to think about what it is you want to get out of that conversation, and you need to think about where you're storing that data so you can use it again.

[00:24:25] So I think it's everyone's job, and it's everyone's job because there needs to be a level of respect for the research practice and what the research discipline means. With that comes a bit of accountability from other people in the organization. They need to come to the table and be willing to learn, I guess.

How long research stays valid

[00:24:52] Anna: Yeah, that totally makes sense. Caylie, you touched on something interesting. You mentioned high-quality research. I'm keen to hear how you keep people aware of the time sensitivity of the research data that you have.

[00:25:08] Caylie: Sure. A rough rule of thumb that I normally tell stakeholders is that for strategic research, or research where we're looking at people's behaviors, needs and wants, those things don't change very quickly, so you can trust that a piece of research looking into that sort of stuff is valid for a few years at least. Things like the results of concept testing or usability testing are not going to last as long, because those designs change, and they might change with little bits of iteration or they might change in a bit more detail.

[00:25:50] Normally in my past organizations I've given those rules of thumb to people and said, strategic data you can use for a bit longer. But in terms of policing how that data gets used and where it gets used, you really just can't. It's one thing to do amongst many things, particularly if you don't have a dedicated research ops team. The best thing you can do is just try and make sure people aren't using data that's too old when you can.

[00:26:20] Anna: Yeah, I totally agree with that. I'm keen to hear from you, Vineet, if you have any rules of thumb for how long research should be used for.

[00:26:29] Vineet: Yeah, I think I'm a bit more flexible when it comes to this, in the sense that I feel as long as the data is kept in the context of the timeline, there is value in looking back at historical data to derive insights. We've faced the problem where, because you don't necessarily have context on historical decision making or research, you keep going back into mistakes that you shouldn't have. You made a mistake two years ago, you learned from it, you iterated, and then two versions down the line you're iterating and going back to the same mistake, just because whoever is working on it didn't have context of the earlier research or the earlier decision making. As long as the data is presented in the context of the timeline, in a format where every designer and every product manager can take the best decisions there by themselves, it's fine. The real risk is just not having it in that context.

What excites them about democratization

[00:27:36] Anna: Yeah, totally. Context definitely makes a huge difference. I do have one last question before I let you go. I'm keen to hear from you both what excites you the most about the democratization of research.

[00:27:52] Vineet: Maybe I can take a stab at this first. I think both Caylie and I touched upon this. For me the biggest draw is just building empathy. The more you centralize something like research, the more I feel you take away certain expectations from every single person in the product team around knowing your users, knowing what affects them the most, knowing what job they're out there to do, knowing where your product fails and where it's great. At the end of the day that is going to hinder your ability to build better products.

[00:28:36] It's a lot more visible for us at Postman because we're a team of designers who are building products for developers. We are not even the target persona, so the need for empathizing with developer workflows is just a lot more evident to us. We've seen situations where not having that empathy can lead to really terrible decisions sometimes. So I think the biggest draw is just building empathy at the end of the day.

[00:29:09] Anna: Thanks, Vineet. Any quick thoughts from you, Caylie?

[00:29:13] Caylie: Yeah, I'd reiterate the point about building empathy. That's amazing. But also, from a research perspective, it means that you can do so much more stuff. If you've got designers and product managers taking care of the more tactical pieces of research, and everyone in the team has empathy for who you're building for, it means the researchers can focus on the future and what the organization could and should be doing for its customers, which means solving more problems, and that's really exciting.

[00:29:41] Anna: Yes, totally agree. Well, thank you so much. I'm really inspired by all your thoughts today. Caylie, I hope you feel better. And everyone, thank you for all your questions. We will be posting them on Slack, and Vineet and Caylie will be jumping on there to answer them. Thank you so much.

[00:29:58] Caylie: Thanks.