Unmoderated usability testing: finding clarity in the chaos of speed and scale

Alessandro Caliandro at UXDX Community: Scaling UX: Clear Content, Smart Testing. Video: https://youtu.be/X3vBz3YqBNA

Readable transcript: edited from the recording's captions for readability (fillers and false starts removed, punctuation and section headings added). Wording is the speaker's own. Timestamps are positions in the video. Names marked [?] could not be verified against the audio.

Introduction: clarity in the chaos of speed and scale

[00:00:08] So today I will speak about unmoderated usability testing. I called it "finding clarity in the chaos of speed and scale". The chaos is the chaos of product teams. Today every product team moves very fast, with multiple releases, and it's very important to have evidence on which to base their activities. So unmoderated usability testing, and usability testing in general, is very important. And clarity is probably the most important word, because it's what good research provides. So it's very important to understand how to find this clarity.

[00:00:47] So let's move on. A little bit about myself, very, very fast. I've been working at UNGUESS for some years right now. Before, I was, I am also a designer and researcher. I do a lot of events, I like very much participating in events as a speaker but also participating in general at events.

[00:01:10] What we do at UNGUESS, in two lines: we are a crowd testing platform. So we do research and testing with a lot of companies. We started in Italy and now we are expanding in all of Europe and all the world. What we do is with three pillars: so we have our technology, our crowd, so it's a crowd testing technology, so we have like Tryber[?], that is our tribe of humans, and we work with a lot of certified experts, that are the ones that provide this kind of research. And we do it in four areas. Today we'll speak only about the first one, that is my main pillar area, that is the experience one, but we also work in accessibility, in quality and in security sectors.

[00:02:07] It's important to say that what I speak about today is something that is not only theory, but it's a lot of practice, because we do a lot of unmoderated usability testing and we also do a lot of moderated usability testing. So today I will speak to you about what I found in the last year. We have done more than 70 unmoderated tests for different clients, different sectors. So these are my ideas about this topic.

A little history of usability testing

[00:02:36] And okay, we can go on speaking about usability testing in general. Just a little bit of history and a little bit of context about usability testing. I don't think that in this context we need a lot of theory, but I use many times this kind of content because it's very important to understand the differences between usability and UX and other topics.

[00:03:02] So just a little bit about usability. Okay, we have a definition, and it is defined as the mix of effectiveness, efficiency and satisfaction. And it's very important to think about usability, where we measure everything in our product or service. And UX is broader. So it's not only the usability but also the perception and also the responses of the user to the anticipated use. So beyond, okay, can they do it, we care also about the trust, the emotion and the expectation. So we can say that usability is the foundation but UX is the whole building.

[00:03:47] So, starting from how we started to test usability, we can refer to the '80s, that is probably when the field took shape around that. So we started with usability principles, then also we have Nielsen and Norman in the '90s, and they start speaking about usability testing, user centered design. So everything we talk about every day today is based on those years. And also there start the topics about how to measure and also how to test the usability, in the 2000s, because there start the internet-based platforms that could help people also to test this thing.

[00:04:36] And the essence of usability testing in the first years was based on these three pillars. Okay, this is important, for testing usability it's having real users, so to test with real people. It's important to have real tasks, so having tasks where you analyze and observe what they do with your product or service. And also it's important to observe, the observation. It's very important also to analyze what we are observing. So usability testing in the first years was everything about these three main pillars. And the main quote that we use in this sector is like, we don't trust what users say. It's very important to look at how users behave and what users do.

[00:05:31] So usability testing in the first years was like this. We had, traditionally, in-person labs. We love a lot this kind of environment. So every usability test was performed in presence with dedicated labs. So we were recording equipment, and sometimes also a one-way mirror was inviting to look at. And the role of the moderator was very central in everything, because the moderator has to welcome, to explain everything, to probe, and also to rescue users during the task. Stakeholders observed the session in real time, directly beyond the mirror usually.

[00:06:22] And so this was the standard, and we continue also to do this kind of usability testing nowadays. But for a technological reason but also for a methodological reason, we are going on with new methodologies and new ways of testing the user. So around the 2000s there also starts the topic of unmoderated testing, so a new way of testing the usability of products.

The rise of unmoderated testing

[00:06:52] And so this is about the little history of the unmoderated. So in the 2000s, I say, we start also having some platforms that could help us to do this kind of tests. And in the 2010s they start to have a wider adoption in digital first companies, of this methodology of usability testing, and also start to test not only websites but also complex websites or mobile applications.

[00:07:28] And for the last years, the pandemic was a big accelerator of remote work and also global collaboration, and also all the platforms of testing have had very massive enhancement in the direction of speed and scale of the projects and of the research topics. So everything became faster and everything became easier also for users, so researchers could easily set up their studies.

[00:08:03] We saw a shift in our normal use of usability testing. So we have not only the moderated usability test but also a new way of presenting this kind of methodology, which is about the unmoderated, that is no more in a lab but is remote, it's asynchronous, that is one of the most important things, we will discuss it right now, and there is no moderator in real time, but everything is done before and after. So sometimes it's also called a post-moderated usability test.

[00:08:43] I have to confess I was skeptical the first time, because I thought that the ability, the experience of the moderator, the role of the moderator was very important for extracting the real insights, to understand how people behave, and also to understand, with the right question at the right time, what people are thinking, or the motivation behind their behaviors. And so I approached in the first years the unmoderated test a little bit shy. So I was not so confident about this, but then I understood that there are very big reasons to approach it with an unmoderated way.

The four advantages: speed, scale, real context, metrics

[00:09:30] So there are some advantages, very big advantages, of using the unmoderated approach, and also there are some opportunities of this kind of approach. So the four, sorry, the principal advantages of an unmoderated approach are the speed, the scale, the real context, and the possibility of using metrics.

[00:09:59] So the speed is the first one. The results are already there in hours instead of weeks. Because for creating a protocol of moderated, you have to do in person or remote directly all the interviews. You can have bigger teams but the work is so long. It's very important also that you don't have to coordinate or schedule the calendar. So all the research ops part, that in a moderated approach is very, very consistent, becomes very easy. And also it's possible to align with agile cadences or rapid prototyping. So it's very, very fast, and so it can be put everywhere, also in a product team. So the speed is the first very big advantage of the unmoderated approach.

[00:10:52] Also the scale, it's very important because you can do it at the same time with dozens or many, many participants. So we know that Nielsen told us, teaches us, that like five or six for a qualitative test is enough, but sometimes it's important to have more tests done, and so using an unmoderated approach can help you to have a broader number of participants.

[00:11:31] And also with this broader number of participants you can also create user groups, have different devices, different environments. So you can test also different approaches to the same product and you can compare them, and it's very important to arrive at something.

[00:11:52] The real context is another very important thing. So users interact in their own natural environment. You don't have the problems that you have in a lab, where people do not behave naturally. And also they use their real device, and this is very important to reveal usability issues, and also to have like contextual inquiry that could help you to understand the real problems and also the real strengths of your digital product.

[00:12:27] And the last one is that having a more solid number of people could help you also to start measuring things. So we know that with five or six users it's not enough to have quantitative numbers, but when you start having 20, usually 20 is the first one, you can start talking about a summative test. You can have more important metrics and you can start to track these metrics, also to compare between different versions, or to compare with competitors, or to compare with whatever you want.

[00:13:10] And the metrics is a thing that only with the unmoderated way you can use, because many metrics you cannot track during the moderated way, like the completion time, a thing that you cannot do, ask people to do directly things and then check that time, or other topics that are not usable in the moderated way that are possible in the unmoderated. So this was the speed, the scale, the real context and the metrics, the main four advantages of using an unmoderated approach.

[00:13:50] So, two simple takeaways, then we go a little bit into how it works, so how we can do efficiently an unmoderated test. But it's very important to start extracting some takeaways also for this topic. And the first one is, unmoderated testing is not a shortcut. So we don't choose unmoderated when we have no time to do a moderated approach. And also that unmoderated and moderated are not competitors. Usually the best way is trying to combine them in different moments of the evolution of the product, to enrich the deepness with the moderated and also the possibility of adding metrics with the unmoderated approach.

When to use it, and how the process works

[00:14:40] So just a little bit about how it works. The applicability, it's very important to understand how we can choose. It's better to choose an unmoderated approach definitely when there are live products or high fidelity prototypes. It's a better way to test them in an unmoderated way. For the reason I said before, it doesn't work so well with low fidelity prototypes, because sometimes you need to guide the user a little bit into the process, and so it's better to use it with products.

[00:15:14] It's the best choice for repeatable flows, because it's very, very easy to identify patterns or friction points when the flows are very repeatable. And as I said before, it's the best choice when you have to benchmark something, so when you have to compare, using an unmoderated approach is the right choice. And it can also be used in complex industries. So also in finance or healthcare, having metrics at scale can reveal trust and comprehension issues much faster than with a moderated approach.

[00:15:58] The process of an unmoderated test is quite the same as moderated, with some little changes. So the objectives are the first part, also for the unmoderated way, and it's very important to specify valuable goals. It's very important to recruit participants in a specific way, and so being sure that the participants are in the target, because we cannot go back. So when we start an unmoderated activity, we cannot stop 5 minutes after, like we say in the moderated way, because if something goes wrong, everything is done.

[00:16:50] Then it's important to design tasks and instructions in a very clear, realistic and unambiguous way. Then there is the part of the field activity, which is very fast. So you have to collect all the sessions, all the videos. It's important also to have videos not only of the screen but also, it's possible, also videos of the face of the user. So we can also track and observe the expressions on the face during the navigation. And then there is the part of analyzing and synthesizing and creating the report of everything we need for our research.

[00:17:35] The tools: there are so many tools, so I don't want to suggest a tool in general. There is a very interesting page from the Nielsen Norman Group that compares 11 usability testing tools, and they have also a small survey to choose what our need is and try to define what is the best tool for this need.

Four lessons learned in the field

[00:18:03] And the lessons learned, four lessons that I learned in the last years on the field. The first one is: clarity beats creativity. So it's important to be creative, but the tasks must be simple and direct, because we cannot correct after. So everything, it's important to be more clear.

[00:18:29] The importance of the pilot. It's important also in a survey to have a pilot, to see if everything is going well, but in an unmoderated it's more and more important, because a 10-minute run with someone from other teams, or someone, can save many days and much money. So understanding if the tasks are easy to understand, if the tasks are also in the right order in the session. So the pilot is very, very important.

[00:19:05] It's very important also not to use only the stats or the metrics that you track during the observation or with the survey that you usually combine with the unmoderated, but also having real recordings. Of course they are very important, the observational recordings are the most important thing.

[00:19:30] And the last one, that unmoderated testing alone is not enough. So you must mix, having an unmoderated session, having surveys before and after the unmoderated session, to try to understand what are the words, what are the opinions before the confrontation with the product and also after, and also the possibility to compare the before and the after.

Challenges and how to handle them

[00:19:57] Last two or three slides about the challenges. So the things that can go wrong during an unmoderated. So there are some quotes, some conversational quotes, that sometimes we face. So the lack of motivation could be a problem. So people that say, I don't really run the task, so I don't know what to do. And for this the solution is to be very, very careful with the screening, to choose the right people to participate in your unmoderated session, and also use fair incentives. So it's very important to pay, and to pay well, the users, because we ask them to be professional, not professional users but real users, but professional in the way they present and in the way they register themselves.

[00:20:55] The other challenge and the other problem sometimes is about the clarity of the tasks. So we don't have a moderator directly to clarify the tests and the tasks, so it's very important to be more clear, to be easier to understand.

[00:21:14] And another challenge is about the difference between the different registrations and different users. So the data quality is very important, and for this it's very important to screen, also in this, to screen the right users but also to combine metrics and what is observable in the video, and also to check a little bit the standard of the responses. So to be sure that everything was good in the first time.

Final takeaways

[00:21:43] And so, three final takeaways before the Q&A, if you have some questions. I already said it, but for me it's very important: unmoderated is not a shortcut, but it's another opportunity, another alternative to usability testing done by a moderator. It's a valid method to generate evidence. Moderated and unmoderated are not competitors, so it's not a fight between different methodologies, but the best thing is trying to combine them. And the real strength of unmoderated is bringing speed and scale without losing the clarity that, as I said before, is the objective of every product team.

[00:22:25] So if you have questions, doubts or curiosities, I'm here right now, and also you can find myself on LinkedIn probably. Yeah, I think this QR goes to my LinkedIn page. So just write to me, I will answer very fast.

Q&A

[00:22:46] Host: Thank you. Awesome. That is my cue to join in again. I'll just leave the QR code a little bit longer so people can connect with you. There are some questions, also from early on in the talk. First of all, really clear talk, really thank you for sharing it. One thing that came across, because one of the first things you said as well is like, can we trust the data without knowing external circumstances, as in what the people are doing, like maybe someone is driving in their car or something like that. How can you still make sure that the data is, do we trust it? What are some experiences there as well?

[00:23:21] Alessandro: Yeah, it's important, if we need data about effectiveness or efficiency, it's important to have them in a real environment. So if we have a transport app, it's okay that they are in mobility, because they will use it generally, also in daily life they use it in mobility. Otherwise, if it's something like a mortgage request, probably they will do it in the tranquility of their room. So we ask people usually to do it in the normal environment, and so in this environment we can easily track things about efficiency or effectiveness. And also satisfaction, it's very important to track. So we use a lot of different KPIs or different metrics to track satisfaction, and it's quite easy with a survey post the unmoderated activity.

[00:24:20] Host: Okay. And in the process as well of recruiting, how do you guarantee that you have the right audience with the right intentions? Because one consideration can be money, but also other considerations that are usual.

[00:24:34] Alessandro: Yeah, money is the standard compensation. Sometimes you cannot pay, for a legal reason, or because maybe you have a user group that doesn't want money, like high spending people, they don't want money sometimes. In that case usually we will speak with clients and try to give them something else, like discounts or participation in exclusive events or something like this. And there are right ways to incentivize people. And I think the most important thing is to try to screen out the people before, and also trying to be sure that people are not like a professional tester. So we run a community of professional testers for the quality assurance activities, so that must be professional, but for everything about experience we try to have people who don't do too many tests, like only one test a year, because we want people that are real users and not professional testers that speak in a technical way. They must be closer to real users.

[00:25:49] Host: Okay. And in terms of the task clarity, would you also recommend testing it on colleagues, or how do you make sure you get the task clarity properly there?

[00:26:03] Alessandro: Yeah, usually we pilot them, or we try the tasks with someone else. Usually we have some parts of the company that are very, very interested to do this, like the legal part or the administration, that are not so deep with the technical parts but they are very interested to help the company also in this part. So asking our legal sector or HR to understand if the tasks are good, it's a fast way to test.