Design Specialist vs Generalist: How to Manage handovers and bottlenecks on workload

Jennifer Cardello, Kelsey Kingman, Patrick Mooney at UXDX Europe. Video: https://www.youtube.com/watch?v=cGbBG5OuYk0

Readable transcript: edited from the recording's captions for readability (fillers and false starts removed, punctuation and section headings added). Wording is the speaker's own. Timestamps are positions in the video. Names marked [?] could not be verified against the audio.

Setting the stage: a very large Agile transformation

[00:00:00] Patrick: We're getting well used to these types of conferences and presentations. I'm working from home, so thank you so much for that very warm welcome. It's my honor today to host this incredible panel. Hi to all the attendees here. I've been watching all the wonderful content, and the content that we have that's giving us research from Jen and Kelsey, for me was simply eye-opening and I've got so much to discuss. We're going to talk today a little bit about Agile, Agile transformation, about the role of UX research, and also how generalists and specialists fit into all of that. We're not going to do introductions, you hopefully know me from UX and you've already heard Jen and Kelsey speak.

[00:00:41] Patrick: So I'm going to go straight into it, because time is pressing. Jen, you're a VP of UX Research at Fidelity. I really, really enjoyed your talk, hearing how you've scaled up that capability in the operation itself. You really support a very large Agile operation, and I think the ratio, you said you've got one to 15 ratios of UX researchers to scrum teams. That's a very large operation, and Agile transformation takes time. Can you talk to me briefly, just to set the stage, about that Agile transformation that you've actually gone through?

[00:01:10] Jennifer: Sure, thank you so much for having us. This is a very large Agile transformation, moving in rolling waves through four business units. It started in late 2017, before I appeared on the scene, so I didn't join Fidelity until mid-2018, and they were six months into their learning and testing and growing. It's truly a very meta or recursive exercise, because they're learning how to work a different way, and we're helping with that, primarily focused on enablement and empowerment.

[00:01:51] Jennifer: When you have 500 scrum teams that are all charged with having some level of autonomy, we need to figure out how to multiply the value that we can offer. There's only 35 of us, so we're not going to be embedded on every scrum team, but we know that they want to, as all Fidelity associates want to be customer obsessed and really, truly deliver on that and connect with customers. So we've had to separate ourselves from the idea of being focused on our craft, and how do we actually package that up and spread that capability, that goodness, across the organization.

[00:02:29] Jennifer: Every initiative we take on, including Democratization, which Kelsey has led and has talked about in the presentation and we'll talk more about, is really about building and enabling these teams, a teach a squad to fish type of mentality versus ownership and governance. That takes a lot of carefully walking a line. As researchers, we want to make sure we have high quality in the things that we learn that inform our next steps, but we also don't want to get in the way of teams learning fast, which we call learning velocity.

Where UX research sits in the organization

[00:03:05] Patrick: That's exactly it. I want to definitely touch on learning velocity later on. Can I ask you, Jen, again in relation to this Agile transformation, where did and where does the UX research capability sit within that Agile transformation?

[00:03:25] Jennifer: Organizationally, we don't sit in UX design, which is different than many orgs. We sit in a larger research organization which sits within the data organization. So analytics, AI, measurement. But my partners, my peers, are strategic market research, behavioral economics, customer loyalty, and brand and advertising research. So it's a really multidisciplinary, and in some sense specialists in certain types of research, situation, where we get to lean on each other when it comes to the different techniques that should be used.

[00:04:03] Jennifer: And so helping the teams, it's hard. Scrum teams will say, who do I even talk to about learning this thing, this question I have, or this hypothesis? What we've learned over time is how to intake those requests and have a collaborative discussion about how we can partner or hand it off to the disciplines that make the most sense. So that specialist-generalist conversation isn't just about designers, it's about researchers as well, because we have a very well-developed, mature research organization.

[00:04:37] Patrick: That's fantastic. And you are joined today with Kelsey. Kelsey, thank you so much. I really enjoyed hearing about the Democratization of research, and great job on that, I'm very impressed. I really want to focus in on that Agile transformation. Does the Democratization approach that you talked about and you have executed upon naturally fit into the Agile process?

[00:05:02] Kelsey: I would say yes, but it's like cheating, because we built the Democratization program after the Agile transformation. We looked at it, we said, oh my goodness, there are 500 squads. So it fits into the Agile process because it was designed to do so.

How the research process had to adjust

[00:05:16] Patrick: That's good. Can you talk to me a little bit about how did the UX research process have to adjust? It's great that the Agile process was there, so you had something to lean into. Can you talk to me about how it had to adjust, just give me some examples of that?

[00:05:42] Kelsey: Yeah. Part of the adjustment for our team was, rather than being the quarterbacks for every single project and working with teams in sprints, was really being people who are guiding the process of deciding and talking to designers, hey, should this be democratized or should they use a researcher ticket? So rather than leading every single thing, working with designers to help assess what could be democratized. Now we're a mix of guidance people, Sherpas, and also quarterbacks.

[00:06:16] Patrick: That makes complete sense to me, and it is that collaboration to figure out how can we enable you, how can we enable you to ship faster, build faster and learn faster and do the right thing. That's great. Jen, I'd like to go back to your early days working with Fidelity. You mentioned in your talk that you did a 90-day listening tour, and that always excites me, there are always great insights you can get from that. Can you talk to me about some of those key insights that you actually learned, which led to the creation of, one, your playbook, which you spoke about, and then also the Democratization?

[00:06:49] Jennifer: Sure. It's interesting because you talk about a 90-day period, and I did bookend it, but every day is a learning day for me. It's a very large organization and we have business units that have different ways of working and different pathologies, as some would say, in the way that they're built and the things that they're going after. So I'm still learning.

[00:07:09] Jennifer: But what I heard when I first got there is people were nervous about having been put into a new, better way of working, we call it, which was the Agile transformation, but still had some very strong connection to the idea of research as a mechanism to generate proof points and evidence. Which I think makes sense, given where some companies have come from, but there's two problems there. First of all, it weaponizes research. You've got people saying, go do some research that proves that we're building the right solution. So they're zeroing in on one solution too early, and then they're looking for someone to tell leadership that that's a good choice, and it's tough to do that. It's the wrong tool for that exercise.

[00:08:02] Jennifer: We would like to have that type of proof point generation in market, but we can't always do that. And the other thing is that it doesn't optimize research for what it's really good at, which is discovery. We know we have to do some validation work, of course, but we want to make sure that we're utilizing research to help teams discover the right problem to solve, and discover and harvest many ideas that can then go through a proper process of vetting to find things that will resonate best with the market. We call that problem solution fit. You want to get them to that really healthy behavior. I think it leads to high functioning teams and better outcomes for our users.

Socializing the change, and the resistance to it

[00:08:50] Patrick: I think that's wonderful. And you've got a great reference to talking about the altitude at which you were operating, which I think resonates very nicely. Can we ask you to talk about how you got buy-in? You mentioned that you had to socialize it and you had a slide up that mentioned, we went to socializing, we got buy-in and a bit of refinement afterwards. Was the socializing easy? Was there pushback? Was there resistance?

[00:09:14] Jennifer: Absolutely, yeah. There's always going to be resistance. My coach likes to say that influence equals persuasion divided by resistance, so my focus is on helping avoid or reduce that resistance. And this, again, is something I'm still learning as we move through. Including people in the process of developing that plan is one piece of it, so they don't feel like it's being imposed upon them. And then also trying to bring people along in the journey of seeing things maybe differently than the way it's been practiced previously.

[00:09:46] Jennifer: One concept there is the idea of research being output centric. So the result of research is a series of bespoke, discrete research studies. What if research was about discovery and learning and it was a continuous process? What would be the infrastructure we would need to set up to live in that reality? And that's a tough sell for people who are like, well, we purchase research studies, or we commit funds to conducting bookended research. We're still working on that through the business units, because we need to bring them along and we need to have examples of where it works really well. So we have projects that have gone through this process, using discovery more generously, and have had good outcomes.

Specialists, generalists and who you hire

[00:10:42] Patrick: I think that's so true, having those proof points to say, look, we've done it, we've tried it, it does work, trust us, you're going to get these outcomes or outputs. I think that's exceptional. Kelsey, I want to touch on one of the core topics that we're talking about today, which is specialist versus generalist. You are a principal UX researcher, so would you consider yourself a specialist or generalist?

[00:11:01] Kelsey: Definitely a specialist at this point in my career, yeah.

[00:11:11] Patrick: Okay. And are there other types of specialists? You're principal. Can you talk to me about the type of makeup that you have on your team, what type of roles you have, and do you have generalists and specialists on that team?

[00:11:21] Jennifer: Are you asking me or Kelsey?

[00:11:24] Patrick: Oh, Kelsey.

[00:11:27] Kelsey: My internet keeps freezing, so I may pause. I think your question, though, was, are there other specialists on our team?

[00:11:33] Patrick: Yes, exactly.

[00:11:37] Kelsey: Perfect, if I pause just bear with me. There are a lot of specialists on our team, and I would say that the directors, we have directors who lead these pods, our presentation talks about it, they are more of the generalists. I'd say that as people grow in their careers, they start to grow their skills and become more of a generalist rather than a specialist.

[00:11:58] Patrick: That's quite interesting. And I think that is one of the learnings from your Democratization, that you are teaching people to become generalists. Is that right?

[00:12:15] Kelsey: Ideally, that's the idea, is to help people expand those skills, and expanding your skills is really what then makes you a generalist.

[00:12:25] Patrick: That's fantastic. I'd like to look at scaling this out. Jen, you talked about having to make a case to increase headcount, and I am always interested, when you open up those job reqs and you hire people in, do you hire in specialists or generalists into your team?

[00:12:44] Jennifer: We will hire both. Like Kelsey was saying, it depends on the role that someone's playing in the span of influence. If someone's having to look out over multiple domains, tribes and squads and manage and oversee research activities, and very importantly partner with our UXD team and our other research specialists, then they need to have a really broad T-shape in order to function in that space. Because the point isn't about trying to just focus solely on UX research, it's about helping the organization gain better outcomes. And you can't do that if you don't really understand how the business works, how those functions work, how they can partner together to create better things.

[00:13:29] Jennifer: So sometimes we will look for someone who actually might be part UX research and part market research as well, has that type of awareness and experience. And then sometimes we might hire people that are really very focused on quant, or very focused on design research in a very qualitative sense. It depends on where the needs are. For example, in one of our groups right now we're spinning up a very involved co-creation program, and our UXD partners want us to lead that out of UX research. So we need a different profile to run a program like that, that's different than what we would expect of someone leading right problem or done right research.

Learning velocity and the impact of Democratization

[00:14:16] Patrick: That's great. I'd like to go back to learning velocity, and Kelsey, if I may. That was a new term for me, I have to say, with every Agile term. When I hear the word velocity, can you tell me more about learning velocity?

[00:14:31] Kelsey: Yes, absolutely. I will own up to this. Jen first told me about this learning velocity, so if I pause I'll let her take over. Learning velocity is really what we're trying to increase, where teams are getting learnings from users, and what we're trying to do is decrease the time it takes teams to get user feedback. That's where really Democratization comes in, by teaching teams how to conduct certain types of research themselves and enabling them to do so. So the learning velocity is what we're trying to increase. It's how quickly are you hearing from users?

[00:15:05] Patrick: That's incredible. Can you talk to me about some of the impact you've made? There are really stunning numbers about taking the evaluative research down. Talk to us about some of the success stories and those numbers that you have.

[00:15:16] Kelsey: Yeah. Part of the success story, especially last year, we ended up training 130-some people just last year alone, and that had a huge impact on the organization. We had teams who were able to conduct a lot more research. We had teams who typically didn't have a lot of research support who suddenly could do this research themselves. So we had a lot more user feedback and, I feel like, better designs, and in the end less rework on those designs, because so many designers, over 130 designers, were able to get user feedback.

[00:15:52] Patrick: That's great.

[00:15:56] Kelsey: One of the interesting things about it is, someone asked me the other day, we actually have increased the number of overall studies in the done right category. So evaluative studies have exploded in quantity, it's just that we're not doing all of it. If we look at the past 12 months, from September to September, only 44% of this very large number of evaluative studies was conducted by UX research. So 56% of those done right studies were conducted by people who went through the certification program. It doesn't mean that done right goes away, it just means that now we can share that skill and get more of it done for the organization.

Who volunteers, and why

[00:16:40] Patrick: I think that that's really important, because when you're looking at the very tactical approach to research, yes, that's exploded, but the person who's doing it has changed. Can I ask what type of person is naturally gravitating to take part in the education, in the learning to empower themselves to do this? Are we looking at designers? Are we looking at product owners? Are we looking at the developers? Who within the scrum team do you find is actually saying, yeah, I want to do this?

[00:17:10] Kelsey: I would say it's mostly towards designers, because they're the ones who can actually create the things they want to test. But we do allow other people who aren't designers to join and to test live sites, as long as it's not post login. So we have a whole bunch of people who are accepted to join. And I would say it's not only just a role, but it's people who are willing to be vulnerable, honestly, and who are willing to get that user feedback and put themselves on the line and actually hear directly from users. So I don't think it's a matter of the role, I think it's your willingness to listen and learn and work off of that.

[00:17:47] Jennifer: We talk about mindset being, fall in love with the problem, don't fall in love with the solution. That's why in the right solution phase we really focused on this very rapid iteration, test and learn, because we want to get people not to grow too attached to the solutions that they're creating. And so, like Kelsey is saying, people who fundamentally get that idea gravitate toward this type of program, where they feel like getting feedback makes for better solutions, versus feedback is critical of their thing they love so much.

[00:18:22] Patrick: That's great.

[00:18:22] Jennifer: Also, we very much focused this on designers, so again that's by design. That said, our content strategy team was like arms up in the air, like, how do we enroll? We have squad leads that have been very like, can I get in line, can I queue up? So we have had a backlog of people in other disciplines who are very eager to be part of the program, which was very exciting for us. It brought us great joy to know that people wanted to learn this new skill and have this capability.

[00:18:53] Kelsey: Yeah, and it was honestly unexpected. I was thinking, oh, only designers would want to do this. But it's a lot of people, and that's so exciting.

Strategic research inside Agile

[00:19:01] Patrick: That's pretty incredible. It's an empowerment, isn't it, really? And when they realize, hey, we can validate quickly, we can get to a better solution faster, we're empowered. Empowerment is important. I'd like to look at maybe these altitudes, it was a great part of the discussion, when you're talking about altitude, Jen. When you spoke about the thirty thousand feet versus the tactical, and the Democratization is very much in that tactical, and what you've been able to do is do more of that strategic research by shifting your research team's focus up into that sphere. I think that's really empowering to both the research team and also to the product teams.

[00:19:42] Patrick: What I'd like to touch on, when we look at strategic engagement, that's going to take longer. It's not going to fit in nicely into a two-week sprint window. So how do you perform research, and strategic research, that research that's focusing on the bigger game and the long rounds? Where does that fit inside of Agile? Can it fit inside of Agile?

[00:20:06] Jennifer: Yeah, it absolutely can. The whole idea of Agile isn't that the research happens faster and, like you're saying, it gets condensed into two-week sprints, because if you're limited by that timeframe you can only do certain types of research. But we do have larger strategic initiatives, the initiatives that everything really rolls up under, that we can attach ourselves to early on to help support the team, along with our research partners, in getting them through right problem and right solution.

[00:20:35] Jennifer: And sometimes right solution will take three months. It looks very similar to a design thinking engagement, an IDEO-esque engagement. It could be 12 to 14 weeks. They come into the process having a good idea of what the problem is, they're harvesting ideas, they're screening many concepts down to something smaller, doing resonance testing and then going into market assessment potential. That's definitely a three month or more process, but that sets the teams up for better delivery velocity once they get into done right. It's definitely a more involved and longer process, but it ensures that the teams that are executing on that aren't executing on something that no one wants.

Right problem, right solution, done right

[00:21:31] Patrick: I want to talk a little bit, if I can, about the framework of right problem, right solution and done right. When I looked at it, it was clearly inspired by double and triple diamond, the converging and divergent thinking in there. Can you tell me a little bit more about that? You've mentioned it a few times, so can you talk to me a little bit more about that framework?

[00:21:51] Jennifer: Sure. It is not invented by me, a lot of this work was done by — it's standing on the shoulders of giants. My partners at Athena[?] work tirelessly on trying to figure out a product development life cycle that made sense. It's built on work that Jesse James Garrett has done, and Peter Merholz, the Double Diamond. But basically it was just turning it into something that rolls off your tongue, it's really easy for people to get behind.

[00:22:24] Jennifer: And it really highlighted, again, what I learned in the listening tour, which is listening to people being very concerned, squad leads in particular, that they weren't building the right things. That those things could be very usable but not necessarily useful. So how do we set teams up for success? And if we can actually call it something, it actually facilitates that conversation. Someone could show up and say, I'd like a card sort, or a usability test. And you can be like, awesome, let's roll out the framework. Where are you guys? Can you tell me what the business problem is? Can you tell me the user problem? Can you tell me the hypotheses you have?

[00:23:02] Jennifer: And a lot of times it's like a therapy session. People will be like, oh my gosh, we don't even have right problem. Thank you for making that clear to me. We were handed a solution and told to go build this thing, can you help us? It makes it so we can have a stronger partnership. In some instances, the can you help us is, sorry, we don't have right problem but we're being forced to move through right solution, and we can empathize with them and say, we'll help you out. In some cases people actually have the latitude to back up and make sure that they were on the right track with the right problem, and they'll do some discovery research in there. It was facilitating the conversation and giving people, giving squad leads in particular, some leverage, so that they could talk to leaders in the company and say, can we regroup on this? We want to make sure we're building the right thing.

A learning organization

[00:23:51] Patrick: Let's talk about education for a moment. Kelsey, you talked about sharing that process and Democratization through an educational formalism. And Jen, you summarized it nicely, you said that we really shouldn't be hoarding our talents, and I thought that was beautiful, and we really should be trying to build out our skills to other teams. Can I ask, when you went to educate others, of course there's a convincing aspect to that, that they're going to have to upskill, they're going to have to maybe stretch into their area. You mentioned, hey yeah, everyone wanted to do it. Can you talk to me about whether there was resistance? Was there resistance in groups that maybe started up and thought, hey, this is too much for me to do, you're the ones hired into this, I don't want to do it? What's the resistance to sharing that knowledge?

[00:24:44] Jennifer: Can I just say one thing? I know this is aimed at Kelsey, but I just want to make sure that, the company we work for is a learning organization. They talk about it obsessively. There is no shame in it, it's actually encouraged, and actually a third of our goals for every year, fully 30%, if not more, is focused on the things that you learned this year. We have full frameworks around this, and in fact understanding user experience is a core skill in one of our business units. It's expected of everybody.

[00:25:20] Jennifer: So where there is resistance, it may be more about time availability stuff, but this is a place that is fully on board, a hundred percent, with making sure people have carved out time to learn. We actually have something called Learning Days. Every Tuesday is learning day. So I'll hand it over to Kelsey to talk about the specific details of that, but it's a very important cultural understanding of the pathology of the organization, and why it's one of the core ingredients here that makes this work. And we understand that that's not the situation in a lot of companies.

[00:26:01] Patrick: Well, I think that's the question answered. I think it's the culture that enabled it.

[00:26:03] Jennifer: Sorry.

[00:26:05] Patrick: No, I think this is it, because it's the culture. I heard, dedicated to continuous learning. I think that's exceptional and it does make it a lot easier. So maybe, Kelsey, did it make it easy? Was that culture really working towards your advantage?

[00:26:22] Kelsey: Absolutely, yeah, it was so easy to get people on board. And I would say the only piece of resistance is people wanting to go faster and do more. Part of the Democratization team is working on, okay, where can we streamline versus where can we not streamline? You have to get this approved by legal because we don't want to get in trouble, you can't just not do that. So I think the resistance is people wanting to go faster and wanting to have less oversight, and that's where the friction is, getting them to work with us.

Audience questions

[00:26:53] Patrick: I get that, that's great. We're almost out of time. I just want to look at the questions coming from our watchers and attendees today. I've got one question which touches on, what's the distance from research to delivery at the Democratization level? How close is that research to the actual shipping and delivering of that particular product or service?

[00:27:12] Kelsey: Incredibly close. The researchers are working with designers all the time and the research is absolutely key to the delivery. I don't know if that quite answers the question, but we're linked arm-in-arm all the time. Well, as best as we can with supporting 15 other squads. It's more like a red rover thing, but whatever.

[00:27:40] Patrick: That's great.

[00:27:40] Jennifer: There's another connection point in there too, which is this amazing team we have in CTSI, the parent org we live in, that's specialized in A/B testing and multi-channel A/B testing. They are great partners, because once something moves into market it's still being tested. And there's a lot of whys that will happen, like we see behavior and we want to understand the why, so it will get kicked back over for us to understand the why. That partnership is also key. So as far as being tied very closely to execution and delivery, it's there, very close.

[00:28:14] Patrick: That's great. There's one final question which I've seen here, which was about, are designers doing this? I think we've answered that. And what I will say is that if you do have a question about how Democratization is working, go and watch the talk. The talk is exceptional, it's half an hour, you will learn so, so much. Kelsey, Jen, thank you so much for talking to me today, I really, really appreciate it. It's a shame we're not all together in Dublin, but we're here virtually, and at least this wonderful UXDX conference is bringing us together in this way. I can't wait to learn more, and hopefully you'll be back in a year's time to tell us about Democratization 2.0, and where you scale that to, has it gone through research, has it gone into other areas. It's really an exciting space for me and I can't wait to learn more. So thank you so much, really appreciate it. I'm going to hand it back to Frank.

[00:29:03] Jennifer: Thank you, Patrick.

[00:29:03] Kelsey: Thank you so much.