Continuous Research: The Qualitative Approach Of Preply

Kate Martynova at UXDX Europe. Video: https://youtu.be/yBLiEXUXtg8

Readable transcript: edited from the recording's captions for readability (fillers and false starts removed, punctuation and section headings added). Wording is the speaker's own. Timestamps are positions in the video. Names marked [?] could not be verified against the audio.

Why UX research matters now

[00:00:00] Today, the name of my session is "UX research that makes a difference." Let's start by talking about how topical this direction is at the moment. From the 1980s to 2017, the UX profession grew from 1,000 specialists to 1 million in the world. That's a bold statement, huh? But not really, because the Nielsen Norman Group, the group that follows our UX research direction, says that their projection to the 2050s is that this profession will grow even to 100 million.

[00:00:31] I know it's hard to believe, and probably it's not true, but still it shows the linear growth, and it shows the importance of learning about our customers, talking to them and receiving constant feedback, especially in the period of COVID-19, because we cannot just trust the quantitative data. We need to gather user opinion and to understand their behavior.

[00:00:55] So, who am I? Again, my name is Kateryna and I'm the Head of User Research at Preply.com. Preply is a global marketplace of online tutoring. In case you want to learn any language in the world, you can go there, choose a tutor that you like and have classes with them on our video platform.

Building features without users

[00:01:10] Let me give you a brief introduction to what was going on when I joined Preply three years ago, to understand how we got to having different research methods working in our company. What was happening is that our product squads were developing different types of features, and we were calling these features key rocks [?]. A key rock, you understand, is a huge feature that will be developed for around one quarter. It will be a less solid functionality, so it was really time consuming to develop those, and also, irrespective of the fact that it took lots of time to develop them, it was pure pricing [?].

[00:01:54] The ideation process was the following. All our product teams were brainstorming inside the team about what they thought would be topical for our customers. So they would not solve user pains, but rather go with satisfying the ideas that they had in their mind. And why did I pick this emoji? Because in the end the product became more or less Frankenstein-ish. There were some functionalities that had low adoption rates, some functionalities that were not easy to use. In the end, we were reflecting back and thinking, "Okay, what's going on? We're spending lots of time, lots of money and in the end having a not very likable product. What can we change?"

A/B testing, human-centered design and OKRs for PMs

[00:02:32] What we did is we first adopted an A/B testing framework. It's a continuous experimentation framework that is targeting speed of launch. Our goal benchmark is having one A/B test launch a day. We're not near that, but we are going in that direction. We also added the human-centered design framework, which means that we need to involve users throughout the whole process of product development. We started with researching who our customers are, what needs they have, how we can satisfy them. We were showing them our early prototypes, reflecting on the feedback we were gathering, changing the product.

[00:03:19] This showed pretty interesting results. Since the A/B testing and human-centered design framework was adopted, our conversion rate from leads to payment has grown by 17.9%, and we in the company grew two times [?] year over year before the lockdowns started.

[00:03:42] Of course, it sounds pretty fascinating: after just adopting two different frameworks, everything changed. No, of course it was a complicated process, and what helped is that we launched, on the high level, on the company level, an OKR for all PMs, and they committed to own customer development, to take part in different types of research. For example, here's a screenshot from the software that we use to track our OKRs that says "improve customer development in the CRO team," which is the conversion rate optimization team, and the key result was to speak to at least five customers at the very beginning. It actually improved the engagement in our company, and our PMs started talking to our users more, and they were more engaged in acting on the insights that we were sharing.

Choosing the right research method

[00:04:37] As we reflected on how we had adopted all these frameworks in the company, we started working on structuring all the research methods that we can use for solving different types of problems. Here I've tried to show all the common problems that we solve with the help of research at Preply. These are validation of product ideas, identification of market opportunities, uncovering new product ideas, understanding quantitative data, detecting design and copy issues, and development of a user-friendly experience. Below the statements, I've tried to list the different types of research that we use to solve each particular problem.

[00:05:18] In order for you to choose the best type of research to solve these problems, I suggest you answer these questions. They will help you to understand which one is the best one. What we usually do is discuss with our stakeholders: what purpose do we chase? What do we want to solve with this research? Maybe we can move even without the research. What expectations do they have? What type of data do they want to receive in the very end? What known data do we already have? And the main questions, which are really important so as not to make our study very broad, because not having a focus is not a good idea, and we lose all the assumptions and hypotheses.

[00:05:59] In this slide, I've tried to show different types of research depending on whether they are qualitative or quantitative, and whether we are observing what people do versus what people say. In this session, what I will do is give you a specific example from our experience of using each of these research methods.

User interviews: the vocabulary feature

[00:06:24] To start off with, I will give you some details and an example of using user interviews. User interviews can be used in the cases displayed on the left side of the slide. The first example was related to our vocabulary feature. Our team had a hypothesis that we could develop a vocabulary feature in our mobile app in the form of flashcards. We were 100% sure that it would be successful, but as we understand, it's not a one-day A/B test. We need to minimize all our risks and to define whether this pain actually exists and whether flashcards would be the best solution.

[00:07:04] So we talked to our users, both students and tutors, and realized that the pain actually exists, but flashcards were not useful according to their previous experience. They shared that the most efficient way of learning new words was to complete tasks with a real context. So this type of research was ideal for this purpose, which is validation of a product idea, because it provides lots of insights about whether the pain exists or not, and how the current pieces satisfy this pain. Maybe they will name competitors that you can refer to in your benchmarking studies. And in the end, it allows you to minimize all the risks and move forward with lots of ideas. So this is our experience of using in-depth interviews.

Field studies: the search page

[00:07:55] Let's move to the next type of research, which is field studies. There are lots of different names for this kind of research: contextual inquiry, observation, shadowing and many more. But let's not focus on this part. Let's focus on the real examples, so you understand the value of using this type of research. Our recent example was using field studies on our search page. You see it here. We have filters over here, we have tutor cards over here, and what we wanted was just ideas for our next quarter on what A/B tests we could launch to improve the North Star metric of the team that is working on the search page.

[00:08:38] What we did is we observed how our potential customers were interacting with the search page, and our main goal was to note their problems, if they faced them, some workarounds, or maybe some interesting user behavior. In the very end, what we noticed is that indeed there were different problems and different interesting user behavior. For example, I will go back and show you. We have the main line of the filters here, and a subline, and students were not noticing this one at all. Also, they were clicking on the tutor cards here, and when you click on the tutor cards, you are transferred to the tutor page, and they were not expecting to see this.

[00:09:20] In the very end it was pretty useful to launch a field study for this purpose, for ideation, because it led us to a minimum of seven already released A/B tests, and we have even more in our backlog. The benefits are the following. It's very insightful: you don't need to have lots of observation sessions to have these insights. And it provides more viable results. Why so? Because in research, data perceived from observations is considered more reliable than data from interviews, because people provide you more viable insights when they do something rather than when they say something. That's it about field studies.

Formative usability testing: the video platform

[00:10:09] Let's move to usability testing. I don't know if you know, maybe you know, maybe you don't: there are mainly two types of usability testing, qualitative and quantitative. They're called formative and summative. We don't usually launch usability testing for everything that we produce at Preply because, as I mentioned before, we are trying to move very fast and to grow fast. In this picture I've tried to show you how we filter, how we try to understand whether to launch the test or not. We do usability testing only in the cases where it's critical to get something right, or it's complex to make well. Simple A/B tests are not checked with the help of usability testing in our case.

[00:11:00] Now, let's move to formative usability testing. Formative usability testing is a qualitative type of testing where you give predefined tasks and observe how a person interacts with the product, and try to define some problems or issues that should be fixed before you launch something in production. On the right part of this slide you can see screenshots of our video platform and what we were trying to do with this formative usability testing. Our goal was to improve the adoption rate of our video platform. It was pretty low. Our target was an adoption rate of 80%, and it was 30 at the time.

[00:11:38] Our hypothesis was that there were some problems preventing users from constantly using this video platform. What we did is we conducted this formative usability testing, and we indeed explored the main inconveniences. For example, students didn't know how to use the minimized window option, and for them it was really crucial. What are the benefits of using this type of method for this purpose? It's very fast. You need only five respondents to have a viable result about the main problems that your respondents have. And as I mentioned before, when you observe what people do rather than what they say, these insights are more reliable.

Summative usability testing: weekly lesson booking

[00:12:27] Now let's move to summative usability testing, the quantitative usability testing. In our case, what we wanted to do with this type of research is choose between two different prototypes, which one would perform better in terms of usability, and release it to production, because the human-centered design framework says the more prototypes you have, the better the user experience you will build. The real example at Preply was related to our product called the weekly lesson booking schedule option, and we wanted to launch it in our mobile application.

[00:13:05] As I mentioned before, the goal was to choose the best prototype of the two. Our hypothesis was that one of the two prototypes would perform better. We launched this summative usability testing, and we usually use Maze software for this purpose, and it's very easy. You just upload the prototype, you run it with your sample, and in the end this software generates an automatic report, and it's really easy to read and to make the respective decisions.

[00:13:38] The results we received in the end were that prototype B indeed performed better compared to prototype A in terms of the main usability metrics, which are effectiveness and satisfaction. Only the efficiency, which is the time taken to perform a task, was a bit bigger than we expected, but it wasn't crucial for this feature, so we made a decision and released prototype B. What are the benefits of launching summative usability tests? They're very easy to launch. You don't need to moderate the test and to create user scenarios and ask respondents to do this or that. The software will do everything for you and generate reports for you, and you will also know how to move forward, because they give some tips and tricks on how to improve the software.

Session recordings, heat maps and search logs

[00:14:30] The next type of research is session recordings and heat maps, which we often do at Preply with the help of software called Hotjar. I think most of you are familiar with it, but just to reiterate why it's useful. The recent case of using session recordings and heat maps at Preply was related to, as you see on the right side of this slide, the calendar. We wanted to understand whether it's easy for tutors to set their availability there, because we first started to receive lots of complaints in our customer support that our calendar was not easy to use. We wanted to check whether all these complaints were just complaints from the complainers, or whether the problem indeed exists.

[00:15:11] With this goal, we launched a Hotjar session on the availability page, and the results were the following: the hypothesis that the UI is not too comfortable was confirmed. We spotted different patterns of setting the availability. Our initial calendar worked by dragging and dropping different timeslots, and tutors wanted to either drag and drop, or click and add manually, or copy-paste time slots. It was a nightmare. They didn't know how to use the functionality properly, and so we knew what to do next, how to improve this UI.

[00:15:49] What are the benefits of using Hotjar? It's fast to check whether these complaints are just complaints or whether something stands behind them. But the disadvantage here is that you have some questions unanswered, because when you observe some user behavior, you might not be sure why the user performs this or that action. You cannot ask questions, because you are just observing their screen. So for that kind of purpose, if you want to ask more questions, it's better to use formative usability testing.

[00:16:23] There's also search log analysis, a very fast type of research. In case you have a search engine incorporated into your website, you can just analyze the search queries and make sure you have the content on your website that is requested by your users. How we use it at Preply is we have a product called Preply Library. Our methodologists create different types of content for our students to learn the language, and we wanted to make sure we have everything our customers are searching for. We just started analyzing the search logs.

[00:16:59] The results were the following. Our hypothesis was that there were some gaps that we hadn't filled, and we noticed that there were some topics that didn't produce any results. These were some grammar topics, or topics about accent reduction, and we just added these topics to the backlog. Now our methodologists' department is working on creating this piece of content. The benefits of search logs are obvious. It's a quick, cheap and easy method to use. And also, why it's useful: it provides the user jargon, because you literally see what they type in, and you know how to promote these topics further in their own language.

Surveys: the redesigned messages page

[00:17:40] Now let's move to the final section, which is the quantitative part of user research. We can start with surveys, because of course this is the most frequently used type of research. Everyone knows it; you've either created one or participated in one. Our recent example of using surveys at Preply was when we wanted to make a decision about how to move forward with an A/B test. We redesigned our My Messages page on the back side, and it showed okay-ish results, but we wanted to make sure that we could expose it to 100% of users and that they would not have any complaints or suggestions on how to improve it.

[00:18:16] We just followed up with a satisfaction survey with a five-point scale question: "Please evaluate your experience with the new My Messages page." What we observed is that our respondents chose 4.2 on average as a satisfaction rate on this page, and it was a very good estimation, so we decided to scale this experiment. However, there was an open-ended question that followed this question, asking, "What can we do to make it five for you?" There was an obvious pattern: we needed to add a search function, so they would be able to search their students by name and other attributes. So we decided just to write it all down and launch it as a separate A/B test.

[00:19:05] Why was a survey the best type of research for solving this kind of problem? The problem, to remind you, was to make a decision on how to act on our A/B test. The benefits are the following: it's quick to launch, and you have statistically significant data, so you are backed up with numbers when you make any decision about how to proceed with your A/B test. However, in case you add qualitative open-ended questions at the end, you have a qualitative research part there, so you need to spend some time to analyze it and to detect some patterns. In case you have this time, do this research and it will point you in the direction.

Comprehension surveys and highlighter tests

[00:19:55] The next type of research is like a subdivision of surveys: the comprehension survey. It's a very cool and very fast method to check your hypothesis regarding UX copy or content issues. Our recent example of using a comprehension survey was fun. We had the idea of creating, over here, and this is a screenshot of our video platform, in this section, a message to the tutor: "You have X hours scheduled with this particular student," not just "X hours scheduled with your student."

[00:20:29] What was the hypothesis behind this idea? That the tutor would notice this sentence and would message the student, in case there were zero hours scheduled, to purchase more lessons with them or to schedule more lessons with them. But we wanted to make sure they actually understood what we wanted them to do, so that the behavior of our users would be the one that we actually wanted. So we decided to launch the comprehension survey to make sure they understood our UX copy. What did our comprehension survey look like? There was only one question, asking how you understand this specific phrase: "X hours scheduled in the next seven days."

[00:21:14] The results showed that the majority of tutors did understand what the UX copy meant, but they didn't understand which student it relates to. What we did is we just added the avatar of the student, and the tutor knew what to do. The benefits of launching UX comprehension surveys: it's very fast. It's very easy for copywriters and researchers to analyze any issues with UX copy and to speak the user's language in the very end. However, sometimes it's better to incorporate this type of question in your formative usability testing, because it's hard to predict whether they will be on the same page and whether they will be in the context of using this flow, so there might be some issues with this. Just keep in mind.

[00:22:03] Our next type of research, related more to the UX copy and content part of the product, is the highlighter test. It's a very interesting type of research where you simply ask your respondents to highlight, in green and red, those parts of the content that make them feel confident or not confident. How did we use this type of research at Preply? We have a German version of the website, and our localization team just recently started working on this, and we wanted to make sure we were moving in the right direction. Our hypothesis was that, since we had just started working on this, there might be some issues. Maybe we could miss some cultural things, or we might not be very aligned with our brand identity. So we needed to make sure that we were moving in the right direction.

[00:22:58] We launched this highlighter test, and the results were the following. Everything was pretty okay, but just some part of the content on the page caused uncertainty in German customers. We also asked one qualitative question: "Why are your highlights in this part in red?" And they were commenting, so we also knew what the users were thinking and how to rewrite this part of our product. The benefits are obvious. You know how to make your content speak the user's language, how to create the user behavior you actually want to have on your website. And in case you don't want to add additional software for conducting different types of research, it can be done really simply. For example, in a Google Doc you can just paste the tasks, ask them to share the screen and do the highlighting there. It's really easy.

Cloze tests for localization

[00:23:53] The final type of research, also for working with content and UX copy, is called the cloze test. When can it be used? When you want to work with your UX content, as I said. In our particular case, we were using it for evaluating our localization of words. On this screen you see part of the Russian version of our website, and our hypothesis was that the main words that participate in our key messaging may have some different connotations and might also not be culturally fit. So I wanted to make sure our Russian localization managers [?] were working in the right direction.

[00:24:35] We launched this test, and the results were the following. We learned that some words have a negative connotation in our students' minds. For example, the word "teacher" was associated with the strict teacher at school, whereas when we say "tutor," it's more flexible; it's a friend that provides a positive emotion. After that we understood that we needed to change this word in our key messaging framework for them.

[00:25:08] The benefits are the following: it's a quick and insightful tool for localization assessment. The main disadvantage here is that you need to actually prepare this task with the cards, and for someone it might be complicated to do on the fly, because you need to get the respondents and you need to do it online. So if you have time to organize all of this, it's a good tool to use for assessing the effectiveness of your localization of words.

Summing up

[00:25:35] To sum up the different types of research that I just introduced to you, here they are on this matrix. Whenever you want to decide what type of research to use, just use this matrix and think: what do I need to have? Some opinions of people, or some observations of what they do, maybe what they say? Do I need something backed up with data, or do I just need to cover the main problems? This will help you choose the right type of user research method for solving your problems.

[00:26:10] How did all this change our product? It's really easy. Now all our hypotheses are tested really fast. We have knowledge, we have insights to iterate, and I want to think that our product became more likable and easier to use for our respondents. The key takeaways from this session: involve users in your product development. In case you don't have lots of resources, do it at least from time to time. Involve stakeholders in the process; it will boost engagement. And choose the right type of research to make sure you have answers to your main questions. It was a pleasure to share all of this, and in case you have some questions, I'll answer you.