Tackling the Feature Factor: Scaling UX Research for Platform-Level Impact

Eric Olive, Joseph Mauri Mauriello at UXDX Community: Validated Delivery: Platform Research and Automated User Feedback. Video: https://youtu.be/2D0SXWg2jV8

Readable transcript: edited from the recording's captions for readability (fillers and false starts removed, punctuation and section headings added). Wording is the speaker's own. Timestamps are positions in the video. Names marked [?] could not be verified against the audio.

A long-standing problem at DocuSign

[00:00:07] Joe: Thank you so much for the opportunity to speak today. We're going to be talking about a problem that has plagued DocuSign, and actually one that Rod[?] touched on as well. This has been a long-standing issue, one that has really been part of the cycle here at DocuSign since I joined about eight years ago, but we are tackling it today, and one of the important components of that is research.

[00:00:34] Before we begin, as Rory, I assume, mentioned (I couldn't hear him), my name is Joe Mauriello. I'm a UX research manager at DocuSign, and I'm the creator of the audience research program that we're about to present. My co-presenter today is Eric Olive, and he's a senior UX researcher who's led a lot of the research that we've been conducting. Before we begin, a little fun fact about me: my 10-year-old daughter got our family into ranchu goldfish wrangling, and it's become a huge part of my life, mostly for water changes, but they are adorable. Eric, do you want to talk about you?

[00:01:12] Eric: Sure. My last name is Olive, but I happen to not like the food olives, so my friends find that mildly amusing. So that's a fun fact.

[00:01:21] Joe: Indeed, indeed. Great last name, not a great snack for Eric. Here's a quick tour of the topics we're going to be speaking about, just for your information.

Shipping flawless features to an empty room

[00:01:32] This is not a problem unique to DocuSign. Anytime you have different product teams focused narrowly on features that are meant to contribute to a larger, holistic experience for users, without some way of stepping back and seeing the customer's point of view, you're likely to run into this, and that's what we're going to be talking about today. I will talk more in a moment about exactly what this problem is and how it manifested for us, and it's going to be a really familiar problem. But if you have similar issues, or if you've experienced this frustration, please enter it into the chat. We can compare notes later.

[00:02:21] For DocuSign customers, this pain was the result of how teams were coordinating. The way we talked about customer value was to frame it around this idea of an agreement life cycle, or the system of agreements. This was something that was so clear to us at DocuSign, right? We were known early in this transition as an e-signature company, and we were trying to move the perception of what we do to be the entire agreement process, the life cycle of the agreement. It's an expansive brief.

[00:03:04] But this proved to be too abstract when we actually went to solve problems for customers. Believe it or not, we didn't have a consistent, unified set of example scenarios tailored to our customers' use cases. Individual teams might, right? This team is focused on sales quote to cash. This other team is building forms for a wealth management intake process. But as far as looking across in a way that had some structure and allowed us to narrow in, it was lacking.

[00:03:44] In short, we were shipping our org structure. There's the tagline that y'all might be familiar with. This seemingly obvious mistake was masked by flawless execution. The echo chamber here refers to shipping flawless features to an empty room. So what? Why does this matter? Well, because solving for an abstract vision poses a serious risk. Building products that were disconnected and not grounded in the customer's day-to-day needs, or at least not grounded in the same consistent set of customer day-to-day needs, meant we were all aiming for different targets. In other words, we were shipping these flawless features to an empty room.

Speed or quality is a false choice

[00:04:30] I'll get into the solution in a minute, and Eric's going to really talk at length. But first I want to dispel this false choice, this idea that you can either go fast or you can ship quality. What we're advocating for is: take a step back and look at the end-to-end. It sounds like we're saying, "Hey, slow down, everybody. Let's stop and do a bunch of research before we ship anything." That may be true, but with the right structures in place, and the right methodologies selected and implemented at the right times, you can move fast and you can be scrappy. You can get insights out the door quickly and deliver value at every step.

[00:05:12] Here's how we did it. First of all, we created a thread that connected all of our different customer value props through different products with end-to-end mapping. We mapped those use cases, the end-to-end process, and we looked for the pain points that surfaced that we knew our product could address. And then this is the anchor of the whole thing: we had to start with strategic audiences that would act as the starting point for all this. Eric is going to walk us through exactly how this worked and some of the mechanics of the program.

Choosing the audiences

[00:05:58] Eric: Thank you, Joe. As Joe just mentioned, there were three key components to this audience research program: the audience, wow moments, and end-to-end business processes. Let's take each of these in turn. For audience, we are a large organization, so we have good, solid data for the most part. With any initiative of this size, of course, we recognized that we needed to choose the right audiences. When I say audience, what I mean is a pretty well-defined group of users in our customer base. For example, quote to cash would be sales, so it's a sales audience. We had a procurement audience. We had a legal in-house counsel audience. We had bankers. Each audience was its own research track.

[00:06:48] In all of that, we used the data that we see here, or some of it, to make our decisions. One of the reasons we were able to move quickly was because Joe had worked tirelessly not only to build this entire program, but to establish and continue to establish relationships with our cross-functional partners, and also with leadership, so we had buy-in. That's how we were able to quickly arrive at the sales audience, the procurement audience, the different audiences that we wanted to study. Because of course, as I wrote earlier, you have to build the right thing, and you want to build it quickly and be efficient. You can't do that if you don't know for whom you're building. So audience was number one.

Wow moments

[00:07:27] The second key component was wow moments. At DocuSign, when we say wow moment, what we mean is a severe customer pain point and a DocuSign solution, and that DocuSign solution must exceed customer expectations. When we eventually show that solution to customers, they will actually say, "Wow, I love this. This is amazing. This is a game-changer." Wow moments are powerful because they provide a focus for chunking DocuSign's platform capabilities into meaningful moments of interaction that we can test in research sessions.

[00:08:03] For example, I mentioned the legal use case. Specifically, that audience was in-house legal counsel at larger organizations focused on commercial agreements, often sales agreements. It's well defined, not too narrow: in-house counsel. For that audience, finding contract deviations takes hours. They have their standard terms, and here's something coming in: they're negotiating with another party, and the other party wants something different. The attorney wants to figure out, is that non-standard? Is that a deviation? I realize this is not enthralling, but it's really important in the legal space. It takes the attorney a long time to find out, "Hey, did we do that before? Is this non-standard?" It's a huge pain point for in-house counsel at large organizations, or really organizations of any size.

[00:08:56] Our proposed solution to that severe pain point was to automatically flag those deviations and surface them right where the attorneys were working, usually in a Microsoft Word document, and have those deviations right there. That was our solution, and the goal, of course, was to get attorneys to go, "Wow, that's amazing."

Testing wow moments with outcome-driven innovation

[00:09:21] But let's take a step back and discuss how we figured out which wow moments would even work. In a moment I'm going to talk about the end-to-end process map and why that's so important. When we had these wow moments, of course we worked with in-house SMEs and with the partners that we wanted to develop them with, because we have a lot of knowledge and existing research to identify wow moments. But as with anything, and I really emphasize this point, we still have to talk to our users. So how do we figure this out?

[00:09:50] In the legal space, for example, we had about seven wow moments, seven key moments that we thought were critical to solving customer pain and making that whole end-to-end journey more successful for our customers. We began early on with an exploratory phase, which I'll mention again in a second, and we used the outcome-driven innovation framework, which is based on jobs to be done. We used that framework to ask standard ODI, or outcome-driven innovation, questions, and we asked an importance and a satisfaction question. The importance question was: minimize the time it takes to find a deviation from standard terms, how important is that to you? Do you, attorney, even care about this? The second question was: how satisfied are you with the way that works today, in the current state?

[00:10:40] Then we took the difference, which is the formula. Ultimately we wanted to plot that information on a graph, because we were trying to identify gaps and see if what we thought were wow moments were in fact differentiators. In the ODI landscape, it depends on what you're doing. There's a matrix, and you have to figure out what you're looking for, but in our case we were looking for differentiators. In this framework, differentiators fall under underserved, if your users rate them that way.

[00:11:18] What we found was that nearly all of what we thought were wow moments were, from our customers' point of view, gaps, and therefore differentiators. The reason that's important is that we then felt fairly confident that that's where we would put our energy. Those would be our best bets. If they fell outside of that, coupled with other information and research, then we would not. Again, the data here showed our product team clearly where we should focus, so it was a good illustration.

The end-to-end business process map

[00:11:52] Moving forward, as I close on my time today, let's talk about the end-to-end business process map. We've talked about the audience, we've talked about wow moments. Now, for this end-to-end business process map, you'll see some color. This is an end-to-end business process map that's focused on jobs, jobs to be done. A job could be something like reviewing and negotiating a contract, for in-house counsel. That was our use case, in-house counsel, and the end-to-end business process is everything from the time a quote comes in the door all the way to signing, and even managing those agreements. It's a whole process.

[00:12:29] As Joe is hovering over, you'll see some purplish, reddish squares and rectangles. Those are wow moments. The reason those wow moments were important on this business process map, beyond what we've already talked about, the customer's pain and solving for it, was to add some nuance and make the jobs concrete: oh, this one's not too bad; oh, here's where they're really suffering. In addition, or closely related, these wow moments instilled customer empathy, because our product partners could see, look, in this part of the whole end-to-end process, they're suffering with this job when they're trying to do it and finish it. They're suffering with this one. Oh, this one's not so bad.

[00:13:16] In just a moment I'm going to talk about the whole process, but I want to emphasize that this is a map we built because we had a lot of in-house expertise and knowledge and research. But of course we needed to validate this end-to-end business process map, in this case for legal, with our customers. That's why we started with an exploratory round of research. The first research was exploratory, and that's exactly what we did. We focused on the end-to-end process map, and then we iterated, revised and tested it again. We were trying to nail that end-to-end process, in this case for in-house legal counsel. And did we have the right wow moments?

Scrappy rolling research

[00:13:54] Subsequently, we shifted to rolling research. We did not invent rolling research, of course, but our innovation, we'd like to think, was that instead of focusing on features, in approximately three rounds of rolling research we focused on, again, the end-to-end process. While those individual wow moments were important, in some cases they were a feature, maybe a little bigger than a feature, but we really wanted to see: do we have the whole process right?

[00:14:22] By starting with exploratory, getting that stakeholder commitment up front, getting their time, and then shifting to rolling research, we were able to be scrappy, because we could timebox our stakeholders' commitment and have them be involved with light check-ins and summaries of sessions and summaries of whole studies. It allowed us to be scrappy not only in the sense of limiting our stakeholders' commitment, but also pausing, pivoting, changing, extending, condensing.

[00:15:00] This whole process, starting with exploratory and quickly moving to rolling research, testing those concepts and those higher-fidelity prototypes, and really focusing on the wow moments as well as the end-to-end process: the scrappiness allowed us to add value at every moment. Our stakeholders were already bought in, and we were checking in with them frequently, but again keeping that commitment light. For that reason we got good results, and we did feel that we had impact. However, as with any initiative, we always want to assess afterward, and Joe is going to talk about how we assessed the success of the program and the impact on the product roadmap.

Measuring impact on the roadmap

[00:15:46] Joe: Thanks, Eric. We measured success based on what DocuSign shipped: the new features we found through identifying those gaps, the gaps we exposed and the opportunities that came as a result, and also the ideas that we validated that came out of product teams in the normal course of work. These three phases flow into one another. As Eric mentioned, we started with the discovery phase. This is where we were looking at the pain points of the end-to-end. For some of these, you can see we found zero opportunities. These were for audiences that were a little further along in their release cycle.

[00:16:30] But down here you can see we really worked with product teams together to say, "Well, where are these pain points? How can we address them and bring these concepts to customers and then finally validate?" These are all the ones that we tested, and this is the count of what actually ended up on the roadmap. We had a pretty good hit rate. The iteration helped with that. But some items just weren't ready. We got feedback from customers: "Not the way you did it here. I like the idea, you're over the target, but you're missing." We were able to figure out, before we went and did a whole long development cycle, which ideas actually should move forward and land on the roadmap.

What to do on Monday morning

[00:17:12] I know it's Tuesday and may not be the morning, but when you are ready to start this, that Monday morning sometime in the future where you want to start looking at your customers' end-to-end processes and bringing teams together, here's what you can do. First of all, look across your product team for allies, people who see potential in looking at the end-to-end customer experience. In our case, we saw the writing on the wall. There was definitely energy, and already some focus on scenarios, starting in the marketing department. It was questionable whether it would come down to product, but we were well poised. We saw that writing and we started to plan. We started to find our allies, and we started to find SMEs that we could work with, so when the starting gun went off, we were there and ready.

[00:18:07] We were lucky enough to have PMs that were designated for the audiences that we ended up focusing on. But align with marketing, the field, a pro serve team if you have one, a customer success team. These are folks that can come in and be your SMEs. Of course research is going to have tons of information about customer end-to-end processes that you can start to piece together, and these folks can help you really sanity check what your map looks like.

[00:18:35] Start sketching out that map early. Don't wait until you've talked to customers. Go with what your organization already knows. If you go in cold, you really run the risk of delivering learnings that your team already knows, and you don't want to be up in front of everybody telling folks stuff they've heard before. You want to take their assumptions and pressure test them, find where the weak points are, and tell them the story that they need to hear to make that unified product vision.

[00:19:07] And then finally, identify the opportunities that fill the gaps. You have this map that you came up with internally. You'll validate it later, but then look at those opportunities. Those are the teams that you'll want to seek out and start to rally towards your vision of this research program. Through several cycles, you can validate what goes on the roadmap. All right, that is our presentation. We are very open to questions, comments and feedback.

Q&A

[00:19:47] Rory: The question is with the ODI matrix. One of the challenges that I've seen with some companies and teams when they do this is that it's a little bit of opinion where you place those items on the ODI matrix. How do you avoid people influencing it and ranking their pet projects a little bit?

[00:20:06] Eric: Oh, I see. I don't think I quite got that at first. Okay, now I've got it. The product partners were not involved in any of the plotting. What I did is I did in fact leverage AI, as we all are, using meta prompting to refine my prompt, and I actually just used the data from customer responses. No cross-functional partner or product partner was involved. All I did was, obviously, run it and check it and look at the scores, and Joe and I and colleagues talked it through to make sure I didn't make a mistake. But no, there was no one involved in that plotting. That's straight from the data. All that is is a visual representation of the data. Does that answer it?

[00:20:48] Rory: Yeah. And because with user research it often isn't statistically significant...

[00:20:54] Eric: That's true. That's true.

[00:20:55] Rory: How are you ranking one above another?

[00:20:59] Eric: Yeah, that's a fair point. I would say we would look at it and say that anything that fell within what we're calling differentiator, based on the ODI framework, would be worth pursuing, and then we would encourage our PMs to at least consider it and put it on the product map. However, in terms of what they ultimately decide, you're right, you might get some bias. I think the best we can do, and it depends on all kinds of factors, including our relationship with the PM and how long they've been there... PM incentives aren't always the same as researcher incentives, and that misalignment is the problem. That's a SaaS issue, not an Erica choice[?] issue, but it's a good point.

[00:21:42] I will say that we did our best to be neutral. If the question is, oh, some are a little more in the green than others: as long as it was in there, we said, look, based on that and what we learned qualitatively, it fell in line. Now, if it happened to fall in that, but all of the other qualitative findings were like, nah, then I would report that and try to be neutral.

[00:22:03] Joe: I can piggyback on that and say the qualitative findings were probably where we really leaned for our insights, because that's where we could say, well, yeah, we got this rating, these are important problems, but how you solve the problem is really where the rubber meets the road. It's like, yeah, we're over the target, but we had examples where we had a certain solution cooked up, the pet project. There were a few pet projects that didn't make it, and they didn't make it because they weren't the right way to approach the problem. They were the right problem, not the correct approach. What we ended up doing was sending those back to the kitchen, so to speak, and they reformulated. They didn't make it onto the roadmap that cycle, but some of the stuff in the latest round that Eric ran on legal got a refresh of some of those ideas, and they were much stronger as a result.

[00:23:02] Rory: Brilliant. I love that. I always think people are close to their ideas, their first idea, but it might not be just right, and it just needs a little bit of massaging to get the idea right.

[00:23:14] Eric: And one thing I should say that I didn't mention, Rory, that I should have, is that in addition to all this qualitative work, one of my colleagues did run a MaxDiff survey, so we did have some stat sig. It was an ambitious survey with a lot in it, but we were able to see if the stat sig study supported these, and it did in some cases, but not in all instances. Of course we factored that in, and I know that everyone's familiar with that tricky balance, but we did want to add some statistical rigor. We do believe in that. We do quantitative work too when we can, to find out if it supports... I was going to say bolster, but that's not the right word. To find out. And sometimes it doesn't, and then we just have to go back to our stakeholders and say, this is where we are.

[00:24:02] Rory: That's where the scrappy cycle comes in, that you're able to iterate quickly and just discount. Like Irv[?] said, it's better to know within a week if you're building the wrong thing versus waiting a month or...

[00:24:14] Eric: 100% agree. And I should say, to be transparent, we did have one little pet project on one of mine that did sneak through. I told them, "Nobody cares about this," and they're like, "We're going to do it anyway." I don't want to make it sound like we have 100% influence. I told them again and again, "Our customers don't care about this at all," and they said, "We're going to do it anyway," and I said, "Okay, but it's not going to be a game changer, based on what we've learned so far."

[00:24:43] Rory: And the example you gave of legal, having it automatically figure out if there was a standard or non-standard...

[00:24:52] Eric: Non-standard, right.

[00:24:53] Rory: That sounds to me like a machine learning problem, and it sounds like there could be a serious feasibility challenge there. When Arad[?] was talking about getting engineers involved in that research, were you getting feedback early that this is a solvable problem versus one that might not be solvable?

[00:25:12] Eric: That's a good question. I would say we had some preliminary feedback that these kinds of things could be done based on our AI offerings, but it's not rock solid yet. I would say the signal is yes, but I can't give you a yes for sure we can do it. But it's a great question.

[00:25:32] Rory: And did they get... Sorry, Joe, can you hear me yet, or still no? No. Okay. The question then is, were they involved in that scrappy iteration, to help direct it a little bit as well?

[00:25:44] Eric: Our engineering teams unfortunately were not involved in this process. It is a challenge that we've had. We want to involve them, and I think it's the classic issue of whether they have the bandwidth. We on the research side welcome it, and when I have been in other organizations I have really tried to encourage that, but in full transparency we haven't had quite as much success there as we would like. We would like to include them more, because I certainly don't have that background. Joe does have some engineering background, so that helps. He's not currently working as an engineer, however. Bottom line is we do need more. I 100% agree with that. That's something we are still working on.

[00:26:26] Rory: It's a very common problem, so I wouldn't feel bad, and it's what we're trying to encourage. It was great to hear that Arad[?] is moving in that direction, but...

[00:26:35] Eric: Yes, he's doing great, and we should do more of what he's doing. I did listen to that. Yeah, I know, 100% agree. But we do appreciate everyone's time today. We hope that this framework was helpful.

[00:26:47] Rory: Brilliant. Well, thank you very much. That brings us to time. I really enjoyed learning about how you were taking that flow and taking that process through, particularly talking about your scrappy cycle of continuous research. I think that's a really important and useful thing that people can take away. Thank you very much.

[00:27:07] Eric: Thank you. Bye-bye. Have a good afternoon, evening, everyone.

[00:27:10] Joe: Thanks, everyone.