A New and Scalable Approach to Usability Testing

Kristian Vybiral at UXDX Community: EMEA Online Series. Video: https://youtu.be/n8xIquUZ-LI

Readable transcript: edited from the recording's captions for readability (fillers and false starts removed, punctuation and section headings added). Wording is the speaker's own. Timestamps are positions in the video. Names marked [?] could not be verified against the audio.

Introduction

[00:00:00] Kristian: Everybody should see the slides very well now. My name is Kristian, and I'm the founder of Uitrial. We're a company that does a lot of qualitative testing. We have different solutions and products, primarily for the gaming industry: game development, mobile games. It's an industry very focused on UX, because it's a very competitive market. The companies there really want to do everything they possibly can, at the lowest cost and in the fastest time frames, to improve their product and stand out from their competition at every step.

[00:00:42] There are quite a lot of things I've noticed over several years working in this field, and I wanted to share them with you. This is intended as more of a Q&A talk, so I'm not sure if I'll be able to get questions in real time, or if there will be any, but it's really aimed at answering some questions from the audience about processes and how individuals in the audience go about doing their testing. That really guides the talk to a certain extent.

[00:01:19] Anyway, I will start. The aim of this talk is to give a basic overview of what's stopping companies from performing very well in their product development, the barriers there are, how they can actually be overcome, and why this is not really a major thing. Pretty much anybody can start improving their products at any time, even though it seems like such a daunting task.

Experience, expertise and information

[00:01:51] Designers, primarily, or producers, depending on the market and what somebody's doing, have a triangle of factors they can influence, which come together in their own professional image as a professional working on a particular product. Most have experience and expertise. That's how you're hired; that's how you're put into a position to lead a certain product or develop a new product or an update. But that's really an old way of approaching things, and it only gets us so far.

[00:02:34] Information is something that is possible to obtain only in the present context. It's not something you can take with you, most of the time, from your previous place of work or another industry. It's very context-specific. In some markets or industries this effect is more pronounced and information is more important, for example in more competitive markets where everybody's looking for that edge, than in less competitive markets, like some very cutting-edge technology that's coming out where there really isn't anybody trying to do the same thing. There you have a lot of opportunity to make a product, and even if the UX is not so good, it still performs well.

[00:03:29] What I mean by information here is the data and the information from the audience on what is required from the product and how the product is perceived by the audience. This element is usually not obtained straightforwardly through experience or expertise or your hard skills or something like that. It's obtained because you use a certain method, or you have a certain process set up with your audience, maybe your users, perhaps even the target leads you run ads to.

[00:04:08] There are several ways to obtain this information, to make a better product, to guide your development, to get insights. But a lot of designers, even though they understand they need this information, often find themselves lost. It's a vicious cycle: you come to a barrier of some sort, you face it, you try to experiment, nothing really works out, and then you fall back to just relying on your experience, reading something on the internet, maybe connecting some studies and stuff like that, looking at your competitors, and working with that.

[00:04:58] So the primary question here for the audience, and it'll be great if some people could put some questions together and send them while I'm talking, is this. Most of you would answer that you lack actionable information when you develop your product. Because of that, you feel lost, and you really cannot understand what process you're supposed to set up to not feel lost, to be able to get a product to achieve the goals set for it, and to have the audience respond positively from the start.

UX debt

[00:05:42] This leads to a problem, and it's kind of an organizational problem. It also leads to responsibility. When the production team is working on a product and they can't get enough insights to build great UX for the product, to set it out in a way that provides the most value to its primary user target, this leads to something similar to the technical debt developers accrue when they develop a product very fast. They leave a lot of unpolished code, and then they have to return to it and refactor all of that stuff, and they have to remember that they have a lot of time set aside in the future to dedicate to refactoring.

[00:06:38] It's similar here. There is debt collected, and it's because there is an ignorance of the need for information. As more time passes, more is invested, and more things need to be changed each time. With each update you ship with an unpolished process, with a method that does not involve information from your audience, you accrue debt that becomes exponentially bigger each time. Suddenly there are no changes you can make to a product other than replacing it, or maybe reworking it, or having more people in your team working on it.

[00:07:27] If we think about startups, startups have this process where they iterate, and by that they get information. They release their first version and see how it performs. Usually it doesn't perform well, because they don't know what the market wants and they don't have a really good go-to-market strategy. That fails, and then they iterate again and again and again. Obviously not everybody can do this, because not every company is a startup, and a lot of products have been there for a long time and require work. So the question is: how do we simulate the iterations that happen in startups, to avoid this debt, to have good information, to build great products and impress your users?

Why teams stop at quantitative data

[00:08:19] From my experience, designers and producers start to look into quantitative things for this. They subscribe to services like Smartlook, or various other analytics services out there that show you product performance from a quantitative perspective: how many users went to a certain page, the sequences of their interactions on every page, where they churned, where they didn't, where they stayed most, and stuff like that. But a lot of them actually ignore the qualitative itself.

[00:09:06] This is a big issue, because qualitative data is actually significantly more impactful on product development. I don't know how to stress this enough, because a lot of producers and designers just stick to quantitative, and they end up with quantitative data, which is super hard to attribute. It's really hard to understand. They have these cohorts of users, some of them performing well, some not, some with these needs, some churning on this page or not. You get all this data.

[00:09:56] Obviously you understand the symptoms of what's wrong with each segment of your product and its functionality, but it's really hard to attribute what these symptoms actually lead to. What is the underlying cause? What is the value users are missing? Who are those users? It's really hard to constitute. At this point, a lot of product designers just stop. They think, "Okay, this is good enough, I've done my job. The marketing department should provide better leads, or do better targeting, or stuff like that."

[00:10:31] So they start to outsource the responsibility to the organization, and then the organization also outsources this problem throughout itself. This is what usually happens in most companies, especially startups and those that are not too experienced in how to build a product. That is because qualitative is a step forward: qualitative research and everything you need to do for it is a big step, and it's a big step not everybody is willing to take. Somebody has to take responsibility for it, and somebody has to have the incentive to do it as well, and sometimes it's just too much. But actually it's a lot easier, sometimes even easier than setting up quantitative systems.

[00:11:23] So this problem remains. To symbolize that this is a big step and why this problem remains, it's really easy to look at this graph here, for example. The majority of companies throughout the world that participated in various studies and surveys say they're not happy with their UX research process. So something is not working right. But the upsides are really, really huge. If they've set up a great product, if they position it well, if they have the right audience coming to it, they can obviously win the market, sell their product and get a lot of revenue.

[00:12:11] So there's this huge gap: everybody's having a problem, and then there's this upside. How do we get from the problem to the upside? A lot of the people I've interviewed and a lot of companies make baby steps, which are really unstructured or chaotic. They don't really have a process set up, and then the organizational heads, the bosses, really don't want to finance or invest in all of that that much.

[00:12:46] What happens is that this gap remains, and all the attempts that are made just solidify the stance of the company that this isn't necessary: this is not good enough, this is not structured enough. But there are some ways you can really start to build a method, a process, that can avoid this, and you can actually show impact and opportunity to your management even at the start, so they can appreciate the possibilities of the upsides that can be taken.

The barriers designers face

[00:13:27] This is another thing: the problems for designers, or producers if we're talking about gaming, when they start to talk about qualitative data. They want to find out more about their users and get more information that way, to build better products for them and drive more value, basically what anybody would want to do in their position. But they also have individual problems. They don't have enough free time. They have a lot of tasks. They have this technical debt, or designer debt, or whatever we want to call it.

[00:14:06] There's too much overhead to set up a qualitative research center or facility, or whatever you might call it. There's a lack of understanding of what methods can be used and which are most applicable in their particular case. Stakeholders don't want to accept it, because it's additional investment, and qualitative studies are usually quite expensive compared to quantitative. And a lot of designers trust their intuition. They think they know what they're doing, whereas in reality there's always more information out there that can drive you to build better products than your experience or expertise, or even your intuition, which is an aggregate of those two.

Remote, in-person and outsourced studies

[00:15:00] So what are the different modes of qualitative studies we can understand before we actually go forward? A lot of companies stick to in-person, because remote, despite its upside, has a very big problem of coordinating the participants of a particular study to get feedback on the product or insights. A lot of people are in different places in the world, it's hard to reach out to them, and they don't really trust you.

[00:15:41] But remote allows you to build a very good representation of your audience. If you're doing a remote qualitative study, you're sitting in your office looking for people from your user base, or maybe from your target audience, who can join you in a moderated or unmoderated session. That can result in a panel that is really representative of what you're looking for.

[00:16:10] In-person is super unrepresentative. I don't recommend it at all. I don't know why people and companies actually do in-person testing. It's super old-school stuff that's absolutely irrelevant, in my opinion, unless you have a cafe or something very localized. There's absolutely no reason to do in-person testing, and it's usually a lot more expensive. Of course, there are contexts where it's the only viable option, like doing UX for Ministry of Defense software or something like that, where you have to have very high control over who's actually participating. But it's a very localized and expensive route to go.

[00:16:57] Then there's outsourcing, which is the other end of the spectrum, and unfortunately it's also very popular. A lot of the companies I've worked and consulted with were either doing in-person or outsourcing. They used to outsource to consultancies, which charged a lot of money, and it's very non-transparent. The outsourced company does a lot of research for you, but if you break the contract, suddenly you don't know the processes they actually used to get that information. It's completely opaque. You don't retain any of the value you paid for. You just retain the information, the output, which is really bad as well, I think, because all knowledge has to be retained in the company. That's what drives its value, and that's what drives the value of the product eventually.

Starting small with testing services

[00:17:51] Here is a brief overview with a hassle meter. Actually, the graphics are not really representative; they're really awful, hard to set up. But I would say that, to start, there are many services available right now, online software-as-a-service programs and stuff like that. You have UserTesting, and I don't know, dozens of services that can help you with this.

[00:18:31] I would suggest designers start getting qualitative feedback on their products before they're actually developed, by going to some of these services and experimenting with very small budgets, and actually asking them questions. Just come to them and say, "We have this product, we're working on it, it's not performing very well, we have these metrics, and we want to find out how to improve it and make it better for the target audience." A lot of them will consult you. They will tell you what type of method you should stick with and what kind of process you need to set up.

[00:19:13] But be wary, of course, because a lot of them want to upsell their services. They want to sell you something, just like the consultants, so that you stick around. They also want to retain you, so they won't tell you all the secrets.

[00:19:30] A question to the audience, because this is a very problematic section: you have a lot of quantitative data, and then you start to understand that you can't really do anything with that alone to build a product. You go into this whole new area of expertise you need to learn, which is qualitative research and moderated testing. You go into in-person studies or try to set some of those up, you go to outsourced companies. It would be great if some of you shared your personal experiences, because I know all of you have gone through this at some point in your career, and I'll be glad to answer some of your questions.

Organizational and individual barriers

[00:20:22] Moving on to the solution: finding the right tool. Actually, this is far from all of it. This is not the solution; it's part of the solution. Finding the right tool is the last thing you're supposed to do. Before that, it's really important inside your organization to understand the barriers there are. Those barriers are usually split into two groups. You have organizational barriers: information silos, bad processes that are kind of eluded, very bad communication between departments, stakeholder relations and all that.

[00:21:09] And then you have individual barriers: the universal need for the status quo, reluctance from product designers and producers, and perhaps you can put some individual stakeholders here as well. In my experience, those two segments are really important to overcome in order to build a great product overall. This also influences how you build the UX and the UI, and the quality of your product is directly correlated to it.

[00:21:48] Sometimes you will get lucky and a sales guy will come in, be very good, and sell you a service that will provide qualitative insights to your company and product, convince the stakeholders, share the processes with you and provide you reports on how they do it. But this is super rare. I wouldn't depend on it.

[00:22:13] For this to work, it's really important for designers to understand that they need information, and for that information to be obtained, they need to go from the bottom up and actually start to restructure their organization, be proactive, and promote this idea that this is the way to go forward. Because without it, you will spend a lot more money, you will do a lot more iterations, and you will release products users will not like. Eventually this will either cause your entire company to crumble, and you'll end up releasing a new product or starting a new thing again, or you will overcome this challenge, build a great product and have a process, and you'll be super happy with yourself because you've set something up, it works, and you've accomplished what you wanted with this particular company.

Choosing a tool and starting with your questions

[00:23:17] Once those are dealt with, we actually come to the tool, and there are many tools out there. I would suggest UserTesting is one of the standards. Let's say you have a system set up where you have sprints, and at the end of each sprint you make a build of your product, of your particular app, and you run that build on your target audience. You would go to UserTesting with that build and say, "Can we have some users matching this particular audience criteria go through and play this game, or use this app, for an hour or two?" And you will get some information there.

[00:24:02] There are many alternatives, but remember that it's good to experiment, because doing everything in-house is a pain, and not even the biggest companies out there can do it effectively. It's an entirely self-contained area of expertise that a lot of companies are missing. It involves a lot of organizational skill on the part of the team that organizes these qualitative studies, even if they're remote. It even requires HR skills: you have to work with hundreds of testers.

[00:24:43] For one of our products, we primarily provide value just because we're capable of communicating with hundreds of people from all around the world, in different time zones, and having them do the same thing to a certain quality. Big companies, some of the biggest in certain industries, can't even do that. So doing everything in-house is a pain; I wouldn't suggest it. Finding a particular suite of instruments, or a consultancy mixed with instruments, or something transparent, something you can get experience from and actually retain in your company, is the way to go.

[00:25:28] But it all really comes down to you. What are your goals right now? It would be very helpful if you could sit down with your team and look at the questions you have. "We have this product; why is nobody using it? We have this feature that is not being used. We have a particular audience that's not performing well: we target them, they come to our app, they don't perform well. We have some features that are misused." Just get a list of questions with your team and try to understand what is missing: how to get the information to actually answer those questions, going from the most basic ones to the more complex ones.

[00:26:21] You will start to understand where you're lacking. You're lacking in variety of information: you have a very good quantitative setup, but you're really lacking in understanding what each particular user or cohort is actually doing or thinking at a certain point in time. So you'll understand the requirement to get those people and actually talk to them, in a moderated test, for example, and try to understand your audience. That's something a business development manager would do in a B2B business, but if you're B2C, this would be the only way to actually get feedback. Or you might set up surveys in your app, or email your users to answer several questions. By the way, that almost never works, because people don't read their emails most of the time.

[00:27:16] But you will start to get answers, and once you get those answers and try to challenge every single point, you'll understand the actual instruments you're missing. With those missing instruments, and the anticipation of an upside if you actually answer those questions, you can go to your management and your stakeholders and say, "Look, we're performing under the model. We can perform better than the model next month, or maybe next quarter, if we take these certain actions, and we have them outlined here."

[00:27:58] Once you start to work in this very simple way, and dedicate a certain portion of your time each day to thinking about how your users really perceive your app or your product or whatever you're working on, what their motivations are, what the underlying reasons for their behavior in your product are, why they churn on certain pages and all that, it will make working with the UX and the product in general significantly easier. It sounds quite vague and maybe not super straightforward, but really it kind of is. As soon as you start to dedicate some of your time and appreciate the importance this matter has during product development, when you truly understand how important it is, you prioritize it, you start to focus on it more, you dedicate yourself, and you start to see returns on it.

[00:28:58] That's pretty much the main thing I wanted to pass on to the audience here. It's never enough to just stay in one place, setting up quantitative analytics and product analytics and stuff like that, and then just sticking with that. You have to go beyond that, always evolving, because the market is evolving. There are people out there constantly trying to improve their products and find new processes and methods, and if you're not doing that, you're falling behind. And falling behind is not usually a good thing to happen in any market.

Participants and screen recordings with voiceover

[00:29:36] This is the last slide, basically. It's kind of redundant. It just shows some of the important features, techniques and factors you have to look out for when you set up a qualitative study framework and start obtaining qualitative information. The most important here, I would say, are three things. The first is doing very good due diligence on the participants somebody is providing you for qualitative studies. If you're using your own users for such studies, that's fine, that's great, that's the best thing there is.

[00:30:21] If you're using a third party [?], always be super vigilant about who the participants are. A lot of those platforms really lie. They don't lie because they want to; they lie because it's super hard to understand who a particular user is at a certain point in time. There are people who game the system, who use VPNs and stuff like that. So you might end up spending a lot of time, money and effort on a study with an audience that appears to be yours but is actually a completely irrelevant audience.

[00:30:52] Another thing: I would suggest doing unmoderated studies, not moderated, because unmoderated [?] provides you the ability to see how the user performs without personal guidance in an app or product, interacting with it just as they would in daily life. A screen and video recording is really good for capturing that. Ideally, you would set up a system where a participant from your target audience uses your application for an hour or so, or, if it's a diary study, for several days.

[00:31:32] They provide you a screen recording of themselves using it, with a voiceover recording, so if they have any emotions or opinions about a certain thing, or they're surprised, they voice them. You tell them to voice their feelings and emotions and provide that recording at the same time. This is super helpful. Universally, from my experience, what I found was that screen recordings of devices or computers with voiceovers are the most helpful for designers and producers to create a great product.

[00:32:12] Host: Sorry, I just have to interrupt you, because we've gone a bit over time. But thank you very much, Kristian. I really enjoyed that; I thought it was really interesting.