Continuous Research: The Qualitative Approach Of Preply

Kate Martynova at UXDX Community: Eastern Europe. Video: https://youtu.be/eYm1E0ur7xI

Readable transcript: edited from the recording's captions for readability (fillers and false starts removed, punctuation and section headings added). Wording is the speaker's own. Timestamps are positions in the video. Names marked [?] could not be verified against the audio.

Why user research, and where Preply started

[00:00:03] Hello everyone, my name is Katya and let me tell you today about user research that makes a difference. And why do we even talk about this today? User research is on the rise. From 1983 to 2017, the UX profession went from 1,000 professionals to 1 million. Can you imagine? According to Nielsen Norman Group, from 2017 to 2050 they forecast that we will have a strong 100 million representatives. So therefore today I'm going to tell you why this profession is booming, why insights are so important, and in what situations do we need to apply what type of research. And I will give you the real examples of the research that we do at our company.

[00:00:45] So who am I again? My name is Kate and I'm head of the user research department at Preply, and Preply is a global marketplace for online tutoring. In case you want to learn any language in the world one-to-one with some professional, and you want to improve your accent or whatever, you go to our website, book a lesson and enjoy the ride.

[00:01:06] So let me give you some perception of what was going on in Preply when I joined three years ago. When I joined, our product teams were developing functionalities for a really long time. We were deciding what product squads will own what area of our product, and usually they would kind of work on it for three months. So respectively the prices of such functionalities were really high. And the ideas that the teams had in their mind for these functions and features, they were brought out by the team members, they were not validated with users, and in the end the adoption of these features was sometimes low, well, not sometimes, I mean the majority of the times, and our North Star metric wasn't even impacted. On the contrary, sometimes we had situations when the metric was even lower than before.

[00:02:02] So we were kind of reflecting and looking back at what we're doing with the leadership team, and thinking how we can improve, to not spend so much time for development, so much money, and in the end to not receive negative feedback from our customers, and to move the needle in terms of North Star metric. So what we did is we had a lot of strategic sessions with our external consultants, with the whole leadership board, and we decided to adopt two different frameworks.

[00:02:31] The first one is A/B testing framework, the one that is targeting time. Our benchmark for now is to move from three months of development to one day, so now we're trying to launch one A/B test a day. Sounds crazy, but we're on the way to achieve this goal. And human centered design framework says that you need to involve users throughout the whole process of feature development.

Segmentation and getting PMs to talk to users

[00:02:56] And we started off with learning who is our user base and with doing segmentation. Because before, when we were developing features, we were thinking that we are building this functionality for these people, they like this, they like that, they would love these features. But in reality it turned out that we have 22 different micro personas. They have different behaviors, they have different understanding of what's the ideal product for them. So for example we have parents that are looking for teachers for their kids. We're having career builders, they want to improve their language to succeed in their meetings, cooperation with customers. We're having people who relocate, they want to assimilate in their environment, refugees that need language to assimilate as well, and other different segments. So this was really interesting.

[00:03:42] However, when we started doing research and receiving insights, it wasn't that easy to make sure all of them are validated in the product and people take action on them. So together with the leadership team again, what we did is we've kind of executed a partnership program. We decided on the company level that every PM in the team will have an OKR that is called improve customer development in X team. In this particular example, this is a screenshot of our software that we use to track OKRs. CRO team, conversion rate optimisation, and the manager that works there as a PM committed to key results of meetings with five customers, in his case students, per month.

[00:04:26] And it actually showed really good results, because in that way PMs and all stakeholders are super engaged. They talk to users, they hear their voice, they hear their pains, and they're like, okay, let's execute on this, that's a really nice insight, and stuff like this. So I strongly recommend to try this out.

[00:04:42] So after this, as speaking to users became part of our routine, we've focused on structuring our research methods, and I'm going to talk to you about the top examples that we use at Preply and in what situations. This matrix shows that we use qualitative and quantitative types of research. When we want to discover new things, we want to understand the whys behind something, we use user interviews and field studies, aka shadowing or contextual inquiries. To understand behaviors, because what people do is the best predictor of the future, we do formative usability tests, so we observe how users interact with our products and we ask questions to know more.

[00:05:30] We also do summative usability testing. This is also usability testing but with statistical significance, so we do it without moderation via some software. Session recordings and heat maps, really nice methods, and I will tell you more about it later. Search logs, also not so frequently met in my personal experience, but also interesting. And surveys, a quantitative type of research that serves you as a validator of all your findings, as a very good strategic tool to measure product market fit, to measure brand awareness and NPS, and make some actions.

User interviews: the flashcards hypothesis

[00:06:15] So let's start off with user interviews, a really frequent method, probably the one we apply most in our product development. The problem that we were trying to solve at Preply was, we wanted to understand how we can target our mobile application, because it's a really young one and we just had the fundamental functions but nothing special that can differentiate it among competitors. So we had a hypothesis that if we add flashcards, it will give a valuable feature in the way of flashcards, so our users would kind of see the value there and they will install the application more, they will be more engaged because they will use it all the time, and it will be positive for our brand and for all our metrics.

[00:07:02] So while researching this question we've been conducting user interviews, and we've been asking whether they have this problem of learning new words, how they do it, what are the substitutes they use to solve this pain. And what we found out is that the pain actually exists, the tutors do share new words with students, so it was like a green light, okay, let's develop this feature. However, we learned that they shared these words all over the place, either in Preply chat, or in Google Documents, or they attach documents and share them in chat during the conversation, during exercises. This shows that there should be a centralized place for that, and it would be a workable feature in our app.

[00:07:47] But the most interesting finding here was that, as our initial hypothesis was to add flashcards, we've heard that flashcards, according to our users, are not a very useful type of practice for new words. They were saying that the lifehack of learning new words is if there is a context. If you use it in a joke, at work, with your friends, in real life, remembering words is very easy, but just memorizing them using a flashcard doesn't work. We were super surprised. So instead of the original flashcard hypothesis, we decided to add a feature of adding words manually, which really encourages the student to build their own library of words. And in addition we've added exercises with examples of real context to help them learn them all.

[00:08:37] So it was really interesting to learn what kind of substitutes they use to solve this pain at the moment. You see them here: deliver[?], Quizlet, Anki and other applications. And we learned about pros and cons of each of these applications, and that's how we kind of have already some plans on how to improve our current functionality in the app. And we've also learned that people do pay for these applications, so the demand exists, and the demand is backed up with money, which is a really good sign. So this is it about user interviews.

Field studies on the search page

[00:09:11] Another type of really insightful research is field studies. Why did I insert this image here? Whenever our research team conducts field studies, I always tell them, imagine yourself as a production crew of a documentary, because you need to shadow customers without extra interruptions. You need to note their goals, their interactions, actions, everything that surprised them, everything that causes some difficulties.

[00:09:41] So our recent experience conducting field studies was related to our search page, you see it here depicted on the right side of the slide. And what problem we were solving using this research is that we wanted to understand how our potential customers will interact with our filters, here you see them, and with the tutor cards. This was done for the conversion rate optimisation team, so they would know how to build a roadmap of maybe tests for the next quarter. And we also had a hypothesis that, as our target audience is just learning the language, probably the UX copy might be complicated, so we also wanted to test this hypothesis.

[00:10:23] So this is the affinity diagramming, the type of analysis of field studies, how it's done. We've been noting an observation, each observation for each sticky note, and by observation again I mean quotations, something that you noticed, surprised or caused pain for the user, their goals, actions and everything, but not your personal interpretations, to avoid biases. And affinity diagramming asks the team to cluster all these observations into different groups, and in our case we've clustered according to the time, so to then have a chronological order. And in the end we depicted everything in a CJM, customer journey map.

[00:11:04] On this stage the team was super engaged, and they've used the dot voting technique to choose what observation to take further into action. So each team member had five dots, they put them next to each sticky if they thought that it's a very useful and impactful insight, and then they would kind of have a roadmap of insights that they would need to brainstorm how to act on further. So the whole company had this summary of this research, and let me guide you through the observations and how they were used in the product.

[00:11:42] First of all, you see that the chronological order is here, when it started from need awareness, other ways of satisfying the need, trigger, discovery, prep, which was an intro trial lesson, and next steps. Different teams found something useful for their particular user journey. As we've been working for the conversion rate optimization squad, they found lots of interesting insights in the column choosing a tutor, this one.

[00:12:04] So let's start with the first thing that we see here. Check a few reviews means that we've realized that almost all respondents were paying attention to reviews in the first place, so it gives us an understanding that reviews is a very important, strategically, part of the search that we need to work on more. And observations sounded like, I also read not only good reviews, I also like to read bad reviews, and for me when I see that the website does not provide bad reviews it's a bit scammy. So what we did, we added filters to reviews so users will be able to read different types of reviews, to increase kind of credibility of our website.

[00:12:47] We've also heard some feedback that when students were kind of going through reviews, they were surprised that reviews were written in different languages. But because the website is global and students come from all over the world and they type in their own language, we've added an auto-translate button to allow students to read all the reviews in their native language. And this experiment was statistically significant in terms of our North Star metric, so it showed us the importance of using this kind of insights.

[00:13:19] We've also noticed here, we've highlighted in red, that users were not able to find specialty filters. For example, I want to find a teacher who will prepare me for IELTS, TOEFL and this kind of exams. And it was really interesting because, you see here, we actually do have this filter, it's called specialties. But as we received this feedback we've realised that they do not simply notice this filter, and our hypothesis was that it's because it's located not in the main line. So what we did, we've moved it to the first line of filters to increase its usage and to create the behavior we want to have on the website.

[00:14:01] But also research revealed that the variety makes it hard to choose, meaning that we have a lot of different tutors on search and it's really hard to choose amongst such a good number of tutors. So what we did is we launched an A/B test to test how the limitation to 10 tutors in search will impact our North Star metric, and we're currently observing the behavior of students. There were also different types of observations, like they were saying that this tutor card was super wordy and it was really messy, so they were not able to kind of see the main information from it. So what we are doing now is we're trying to kind of brush it up a bit and to make it minimalistic.

[00:14:41] This was my favorite, a very interesting insight, that students didn't even expect that this tutor card can be clickable and open a pop-up, so they were just clicking on some weird places. And in the end what we did is we've made this whole area clickable, to give an opportunity to students to see a lot more information about our tutor and make a good decision. So this is it about the field studies.

Formative usability testing on the video platform

[00:15:08] Let's move to usability testing. Usability testing is also really important to make the product easy to use. One of our core values is simplicity. However, as we're trying to experiment a lot and to launch many, many tests, it's really hard to test everything. So this is how we filter what we need to test. We try to test everything that should be really critical, that it should be right, but on the other hand it's complex to design, right? So we need to test this kind of functionality.

[00:15:41] So let's start with formative usability testing. It's a qualitative type of testing and we use it just as a discovery tool, to discover if there are any pains or anything that we need to develop more. So the recent example at Preply was when we had a problem that we were trying to solve. We have a video platform, you see it here, we call it Preply Space, and we wanted to know how we can improve its adoption rate by watching what people do there. So our hypothesis was that some functionality might not be clear and maybe something is missing that we need to develop.

[00:16:21] So here are the findings. We made sure that there are no issues with some functions like mute and unmute, permission request, and turn camera on and off and others. However, there were some issues with other functions, and they are really important, like minimize video and floating video. So when we were giving tasks, can you minimize the video for example, and can you return back to the full screen video resolution, there were some issues. Users didn't know what to click, how to return back, and they found it really frustrating. The same was with the floating window.

[00:17:04] So what we did, we first of all improved the onboarding of this feature. We've added labels and icons, and before the lesson starts now we have a separate window that says, this is minimize video feature, this is floating window, this is how you can use them and this is where you can find them. So for sure people will see the screen and they will understand how to use this, and we're really hearing really positive feedback from our customers.

[00:17:27] We've also discovered a really interesting thing when we've been probing the entry point to our video platform. We wanted to make sure there are no issues to even enter the video platform, but apparently there were. So the way users can enter our video platform, they need to go to my lessons page, so the page for managing lessons, and there is a CTA called enter classroom, and not everyone was noticing the CTA, which is really crucial. So now we're working on improving copy of this banner and on improving contrast, to create kind of primary area, to give a perception of the primary action we want them to take here.

[00:18:09] And there was also an interesting insight, that some respondents, when we asked them to return to my lessons page from video, were clicking the Preply logo, and it was incorrect because the Preply logo terminates the session. So we've added tracking to the Preply logo to validate this issue of termination of sessions in the middle of the class, accidental ones. This is how we're using insights from our formative usability testing.

Summative usability testing and prototype comparison

[00:18:36] And now let's move to summative usability testing, really a great tool to receive statistically significant results if your company is data driven and likes to hear these words. So what problem we've been trying to solve in this case at Preply. Sometimes if we have time we're developing a couple of designs, as the human centered design framework says the more prototypes you have the better solution you will eventually suggest to a user. So in this case our designer from the Supply team, that is working to improve the experience for our tutors, developed two different prototypes to suggest them a weekly lesson booking option. So for example if I'm a tutor and I want to book a lesson for my students every Tuesday at 3 p.m., I can use this function.

[00:19:26] So they were different in terms of UI, and we wanted to test which one is better in terms of performance metrics, usability performance metrics. And we also had a hypothesis about one particular field that sounds like time to schedule, and we show them the amount of paid hours left with the particular students, so they would know how many lessons they can schedule upfront.

[00:19:52] So while conducting the research, what we found out is that prototype B performed better compared to prototype A in terms of different metrics. It had a higher completion rate of every mission, it had a lower misclick rate of every mission, and less testers dropped off from the expected path, and it was easier to use according to the feedback, you see here. So the percentage of satisfaction is higher for prototype B. And as for the hypothesis that we had about understanding time to schedule, we validated this hypothesis, so the text was not really clear to tutors, so now we're trying to rephrase it a bit to make it easier for them to understand.

Session recordings, heat maps and search logs

[00:20:36] Another type of research is session recordings and heat maps, not really frequently used in other companies as far as I know, but we like to use it at Preply. Now, how you can use this kind of research: you just record what users do on some specific page without any qualitative feedback. We do it with the help of Hotjar software.

[00:21:00] So what problem we were solving at Preply with the help of this type of research. We have calendar functionality for tutors where they can set their availability and where they can set their calendars and everything. Our hypothesis was that the UI of this functionality is not really comfortable for tutors, because we've been receiving quite a lot of complaints from our customer support about this page. So we were trying to understand how tutors are setting their availability, how they manage time slots, to make proper UI changes.

[00:21:37] And what we've found out is that tutors do have different patterns when they set up their availability. Because currently, as you see here, the calendar works in a way of dragging time slots usually inside the calendar view. So based on Hotjar recordings we can say that tutors have different patterns when they set up their availability. So we've noticed that they were either dragging slots, or either clicking or trying to manually add one by one, or copy pasting. And of course it causes some issues because initially it was developed just for the dragging pattern.

[00:22:12] How did we act on this? By changing availability settings from calendar view to fields with time slots, as we do during the signup process. We simplified availability settings and improved the speed of operating with it. Plus it made the UX more aligned, because they see the same patterns throughout their whole journey. And there was also a tiny kind of observation, but still, we've noticed some frustration about colors. Sometimes slots in the calendar, some are colored in gray, some in white depending on availability, and they didn't understand why, so we've added an explanation there.

[00:22:53] And another type of research, before, I've mentioned that it's pretty rare because not all the websites have search engines, but at Preply we do have one and we have it in our product called Preply Library. Preply Library is the Preply learning plans with different units, and our methodology team is working on these units, so they are creating different materials for English learners to improve their language. So we've added this search functionality so users would be able to find materials based on topics they're interested in. So we've been just kind of finalizing what were the keywords in this search, to make a decision about what topics gather the most interest.

[00:23:38] The most interesting result here was that future perfect tense didn't produce any results. That meant that the first priority for the methodology team would be to add this topic to this learning plan. Other topics were like famous people, celebrities, actors, slang, level, idioms, placement tests. So now our methodology team has a clear roadmap of topics that they need to add to our learning plan, with the priorities.

Surveys, results and takeaways

[00:24:02] And surveys. I think it's a very frequently used type of research at Preply, because surveys are applied to a really broad range of users and it makes results statistically significant. So there is an example of applying surveys at Preply. We redesigned my messages page for tutors, where tutors can chat with their students, schedule everything and stuff like this, and we wanted to gather feedback regarding the redesigned experience. Because when we've launched this redesign as an A/B test, the North Star metric showed flat results, however supporting metrics were a bit positive, so we just wanted to understand how to move forward, either to kill it or to iterate or just to scale. So what to do.

[00:24:52] So we launched the survey asking how tutors would evaluate their experience with the new my messages page. There was a five-point scale and as a result, you see here, we received 4.2, which is pretty okay, however it's not 5. And we've been wondering why it's not 5, so we asked an open-ended question to elaborate what was the reason for their rating. And tutors were saying, there was a really obvious pattern, that they all were asking for search functionality inside my messages page. So they were saying, wow, the page is lovely, everything's great, but how can I find the specific student and how can I kind of make sure I'm not scrolling endlessly among the huge list of my students just to text some person? So what we did based on these insights, we've decided to scale to 100% of users this experiment, and currently we're designing the search functionality for users.

[00:25:53] So this is pretty much it on the types of research and the real examples from Preply. I just wanted to say that it's really impacted our current product releases, so now we're kind of more learning and iterating on these learnings. So usability objectives are met and users are happy and the North Star metric is impacted positively. So I can say that we've grown 2x year over year, and we have an uplift in conversion to paid of 19.7%, which says that it really works, so I can't add anything more to follow up.

[00:26:34] This is the matrix, the types of research that we use, with main questions that we try to answer. Hope it helps you just to kind of memorize. For me it works well because I have a visual memory. And key takeaways from all my speech are that I encourage you to involve users in product development, because you can save time, money and increase the probability of moving the needle. You need to involve stakeholders in the process, because our practice shows that it really helps to kind of boost engagement inside your team. And to choose the right type of research depending on the main problem you want to solve, because in that way your insights will be more reliable and you will have more actionable items. So thank you so much for listening, hope it was useful, and I'm really glad that you guys have a chance to ask me lots of questions.