Designing a Single Experience for Different Users

Raina Mehta at UXDX USA. Video: https://youtu.be/dAMGs2cci80

Readable transcript: edited from the recording's captions for readability (fillers and false starts removed, punctuation and section headings added). Wording is the speaker's own. Timestamps are positions in the video. Names marked [?] could not be verified against the audio.

How I approach product and design

[00:00:00] Thanks for joining me today. I'm here to talk about how to keep UI simple and cost down by designing a single experience for multiple types of users. By way of introduction, my name is Raina Mehta. I'm the Head of Product and Design at Kaplan. I've spent the last 15 plus years designing digital experiences at companies like Amazon, NBC Universal, Prudential and Kaplan. I ended up in product largely because I'm a maker at heart and a collaborator who enjoys seeing ideas come to life in the digital landscape. And those skills have served me really well in product for these 15 plus years.

[00:00:36] A little bit about my approach. I thought it was important to take a minute and talk about the ways that I operate within product and design, which I think have led to some of the success. At the heart of it is really that I consider product management a team sport. It involves many different functional areas and constituents. Once organizations understand what the product function is all about, they have tons of ideas to share, and then there are expectations to go along with it. I think that the joy and the challenge of being a product manager is being able to manage all of those ideas and have those collaborative conversations, understand leadership barometers for success, and then respond to those in some way, shape or form. So collaboration is a key to success as a product and design manager.

[00:01:33] Secondly, customers are critical inputs to the product roadmap. The success of an organization fully depends on your ability to ingest customer feedback through qualitative or quantitative means, and having the ability to know when each of those is appropriate is both an art and a science. So qual and quant research come together to inform and paint the picture of who your customer is. And once you understand your customer, you can get a product that serves their needs.

[00:02:09] Finally, product management definitely requires humbleness and a willingness to be proven wrong, as well as the ability to pivot and operate in what is an ambiguous environment. The desire to be led by the customers, as I just mentioned, instead of your own intuition and gut instinct, is a huge factor in the success of a product manager. And it's also something that takes years to really adopt and understand and hone. I'm speaking from my own experience.

The challenge at Kaplan: 15 to 20 courses, 15 to 20 personas

[00:02:38] Having said that, I wanted to talk a little bit more about Kaplan and some of the experience that I've had there. You may have heard of Kaplan. Most people know about Kaplan from their test prep business. They offer courses in person and online for preparation for all types of exams, including the ones you see here, the SAT all the way to the UNCAT[?], the GRE, the bar, med, nursing.

[00:03:06] When I started at Kaplan about a year ago, my job was to redesign the 15 to 20 different types of online test prep courses into one singular experience. It was a daunting task, because basically students of each course were considered unique and deserving of their own custom experience, their own curriculum, their own methodology for learning. It made it really tricky to think about how we would reduce costs by building something singular.

[00:03:39] So that was the challenge, as you'll see here. Every test had a persona that looked very different. We had essentially too many custom experiences, too many personas to design around, and as a result also a multitude of backends and operational differences. Being able to figure out how we build something more cohesive and singular for all of these different students was really going to help us with margins, with our overhead and with processes. Everybody was really excited about it. It was a real challenge to figure out how to do it.

Moving from personas to mindsets

[00:04:16] One of the early decisions that I made was to move us away from personas and really thinking about every student as so unique. I think personas, in some instances, will trap us as product managers and design managers in the weeds. We think about idiosyncrasies that are very related to a person, and so we needed to extract ourselves, bring our thinking one level up.

[00:04:45] Instead I turned to mindset segmentation. You can Google mindset segmentation if you want to learn more about it, but essentially mindsets help you cast a wider net to build for a broader range of students. In our case mindsets focused more on the goals, the motivations, the ambitions, emotional needs and beliefs of the users.

[00:05:04] Instead of having a persona such as, Joe Smith is a 35-year-old male who's studying for his CFA because he wants to take his career to the next level, you're really thinking about Joe as a self-directed learner. He has a high level of confidence in his ability to learn material on his own. He needs flexibility in his schedule. He's impatient with group classes because he wants to learn at his own pace. You can see the difference in how you think about a design or a product and the features that are going to guide the design, as you have a little bit more of a broader categorization around mindsets. And so we found a lot of commonalities across these 15 to 20 different student personas as we looked at mindsets, and that enabled us to build something that was more common across all of these students.

Guidance Seekers and self-directed learners

[00:06:04] We took our legions of research that we have on our test takers. Kaplan is great about having a lot of research and anecdotal qualitative research available. We organized them into mindsets, and what we found were two distinct and clear delineations in the ways that people learn and choose to engage in test prep. The first is that many students are Guidance Seekers. They want to be led through their learning. They want a teacher who can create a roadmap for them and keep them on track, keep them accountable. And the second is the self-directed learner. They are the mindset I was talking about earlier. They're more confident in their abilities and they want to have the autonomy to do things their way and create their own roadmap.

[00:06:56] What was very interesting about this exercise is that for so many years, most students were automatically categorized as Guidance Seekers. But nowadays we're finding more self-directed learners, because there's so much self-direction inherent in so many online services, and with personalization and gamification students, digital natives, are really learning how to navigate through digital assets on their own. Whereas Kaplan was originally building primarily for Guidance Seekers, we realized that actually a lot of the population is now skewing towards these self-directed learners, and that helped us plan and strategize how we wanted to grow our business.

One journey, widgets turned on and off

[00:07:44] Once we were able to draw these delineations, we were able to then very clearly create one experience. In the most robust and comprehensive journey, you'll have a student who takes the diagnostic test. It tells us a little bit about their skills and aptitudes, and from there we customize their curriculum. So they'll take a live class. It'll be followed with videos where they can dive deeper into topics. Then we'll test them through questions. And finally they'll take an actual practice exam that is akin to the actual SAT, for example. As they get closer to test day, then we have another live test prep, like a final review.

[00:08:37] This works fine, and this was our experience for the Guidance Seeker. But what became interesting then, since we had broken this into widgets for the Guidance Seeker, is we knew then, based on what we discovered for the self-directed user, that we could just present only a subset of these widgets to the self-directed student. And so the whole linear path actually stayed the same, but we just turned on and off different widgets based on the mindset map about particular students.

[00:09:09] So a self-directed student would come in and take a diagnostic, and then based on that we would present them with practice. They just wanted to practice: based on what you know about my strengths and weaknesses, just hit me with questions and I'm going to hone that muscle memory. And then once I feel like I'm getting a score that's high enough, I'm going to be ready for the practice test and I'm good, that's all I need.

[00:09:38] What ended up happening here is that, once we defined the Guidance Seeker path and the self-directed path, then we went to the personas. So for the diagnostic test and the content of the questions and the content of the practice tests, those are all going to be different, and that's where custom configuration comes into play. You still can build customization based on those personas and tests, but the overall journey and the experience across all of the tests was going to be the same, and it was also going to be largely the same for the two different mindsets. And we were able to curate something very cohesive, consistent, and it reduced so much of our overhead.

Defining success first, then working backwards

[00:10:24] The next thing that we did is that we had to define success first, in order to really understand how we were going to design and build, and we worked backwards from that success metric. So now we're moving away from understanding customer goals to understanding business goals. We first aligned with our leadership on how we would measure success and then, again, worked backwards from there.

[00:10:54] This was the set of organizational goals that we were looking at. At every stage of the customer journey we were defining KPIs to help us stay focused. At the acquire stage, which is where we started, we would measure something like the free to trial conversion. What's the percentage converted from free to trial? What's the percentage converted from trial to paid? Then we cared about engagement next. Once we've acquired them, how are we engaging them? We decided on repeat visitation as our primary measure of engagement, as well as the time it takes for them to engage in that first activity.

[00:11:38] When we talk about value next, we're talking about how do we make sure that the student is learning the right things and is actually going to get the score that they want on the test? And that's where we work with our data science team. They're called psychometricians. That's really important, because if we're putting people through this journey and they get from start to finish and they're happy, but then they get a score that has nothing to do with where they wanted to be, then we've failed. So that was another KPI that we needed to measure.

[00:12:16] And then finally, the delight factor for us was really around NPS. We don't have a subscription model. Our model is a transactional model. For people who do have a subscription model these metrics will look very different. Perhaps you're looking at annual revenue per user, retention rate, cancellation rate, that kind of thing.

Top-level goals, team-level goals, and autonomy

[00:12:41] But from these top-level goals, what we then did after aligning with leadership is that each team would decide how they could impact the top-level goal, and then they would assign their own team-level goals around that. Defining an OKR system, where you're defining the objective at the top level and then the key results that each team is responsible for, so that each of them will be driving towards that top level, allows each team to have autonomy.

[00:13:13] It really helps with the autonomy that product and design really want, where leadership can define that goal and then say, you're closest to the customer, you're closest to the research, the data, the product, the technology, the costs, so you figure out how we get there. We're just going to tell you where you need to get and what our expectations are of you. And that works really nicely. I think this is a really critical step for creating empowered product and design organizations that also are focused on business goals.

[00:13:47] In our case, the top-level goal was to increase repeat visitation by 10%, let's say. Product may decide that they're going to use mobile notifications to drive repeat visits, and that's what they focus this quarter on. They go off and build notifications that tell a student, "Hey, it's time to practice now, so come back to your coursework and do your 10 minutes of daily practice." The UX team may decide that gamification with streaks and levels will help motivate the user to keep coming back. And marketing may decide that it's a new weekly email that shows the user their course progress that helps them come back.

Validating designs against the KPIs

[00:14:24] That's a really important step. And then finally, once you've figured this out at a team level, what you're focused on now, you can really design around making sure that you're hitting these particular KPIs. You start with open-ended discovery, but at the end, once you've got your hypothesis and you've designed something, what we did was we validated against the KPIs.

[00:14:53] For example, if we think that streaks are going to motivate, then we're going to put this dashboard, this gamification approach, in front of users, and we're going to have task-based ways of really understanding, did it motivate them to come back or did it demoralize them? Does the mobile notification incentivize them to come back, or is it just something where they're like, stop annoying me, I'm going to turn you off? That's where the validation and the testing need to marry up with the objective. And that's how we did it, to make sure that everything was quantitatively driven. There has been a history of preference-based design testing, where you put designs in front of students and then ask them, do you like this, or what do you think of this color? So we tried to move away from that and really focus our design around KPIs.

Launching is just the beginning

[00:15:45] Finally, we haven't launched this new experience yet, but what I do know is that launching is just the beginning for us. Many people think of launching as the culmination of months of work and team effort, but I tell my team that it's just the beginning, because it's the data that tells us whether we're on the right track. We make a huge number of assumptions. Even the tests that we run to validate are just proxies for how users might behave. The only true test is putting it in the wild, and so at this point we learn: did we get it right?

[00:16:23] And so we have this structure which is very much built around the sense and respond methodology, that some may know, which is you build, you measure, you learn, and then you optimize. You have a continuous optimization cycle. And when you're learning, you're really learning against the KPIs, because remember, you had identified repeat visitation needing to go up 10%. So if after, let's say, six months you're not seeing either a trend in that direction or having met it, then you know that something, whether it's the mobile notifications or the emails, something needs to be modified and you need to continue to refine this. And that's where you go into build.

[00:17:07] We have top-level KPIs, but then, as most people will have, story metrics. Those are the actual metrics that help you decipher what's working and what's not. So we are moving towards having a regular weekly or bi-weekly readout of these story metrics to the team, so that we can keep on top of what we think is working and what isn't, and have group discussions about where we think some of the friction might be that we could unlock and optimize.

[00:17:42] And this is where the humbleness of product and design becomes important, because we get really personally invested in our solutions. And the problem is, if we do it, it binds us to doing the right thing by our customer. So if we launch and the usage data tells us that self-directed students are abandoning at the diagnostic test, for example, then yes, that's going to be a huge surprise to me, but I'm going to run an A/B test and say, okay, if I remove the diagnostic for 50% of students, is this group more likely to get to the next step of the journey? It's okay to be wrong, it's part of the job. It's not okay to ignore the signals from your customers because you think your intuition is more right.

Takeaways

[00:18:33] So that is the recap of where we are and where we've been. I think it's a good place to recap, and the takeaways I would share with you are: segmenting by mindset is a different way of approaching design, and I think it's been really successful for us. It definitely helps you find more commonalities if you have different types of users that you're really struggling with. It really helps you focus on jobs to be done for the customer.

[00:19:03] And then designing against data-driven goals is something that also has been really new for us. It's not easy. It's really not easy to align all these different areas of the organization around particular KPIs. But if you can help your leadership understand the benefit of it, and it can start there, it really does help with the organization. You have less friction among teams, and it's just much more efficient in creating a customer centric experience.

[00:19:37] And then finally, we're going to launch and learn. That is the exciting part of it all. This is also something that is hard for some organizations to orient around, but continuous innovation, or continuous optimization, is definitely something that I've had to have those conversations with leadership about, before we started the build, to say, let's understand that what we build is going to be flawed, because again, we're making assumptions, and we need to have the conversation now, before we launch, to agree that we're going to reserve capacity to do the continuous optimization. And if we don't, then we will be leaving money on the table, essentially doing a disservice to our customers. That's often a harder one to get other folks outside of product and design wrapped around, but it's a really important one that I would encourage you to advocate for.

[00:20:30] So I hope this was helpful to you as you find and define your own approach to building productive experiences for your customers. And if you have any questions, you can feel free to message me through LinkedIn or other means.