UX research for cybersecurity products

Łukasz Ogrodniczak at UXDX Community: Research Impact and Improving Digital Products. Video: https://youtu.be/yI0wpoG9XmQ

Readable transcript: edited from the recording's captions for readability (fillers and false starts removed, punctuation and section headings added). Wording is the speaker's own. Timestamps are positions in the video. Names marked [?] could not be verified against the audio.

From marketplaces to cyber protection

[00:00:00] Łukasz: Thank you very much once again. Thank you also for explaining the mission, which stands for UXDX, because actually I think this is the goal of all UX teams, and DX as well, and all the people obsessed with the product. Today I would like to give you more understanding about how I'm conducting UX research at my current company, because I think this is quite interesting. We'll be talking about the UX research process, about impact, about methods, and also a little bit about cyber security products, because I find it quite interesting.

[00:00:32] I will introduce you to the process I usually follow. I will also introduce you to the area of research in cyber security products. I will also be talking a little bit about research methods and how to choose the best one to support the decision-making process. And at the end, of course, I will summarize, just to explain why I said all of that to you and why I think this has an impact on the product.

[00:01:04] But before I jump into the next section, let me introduce myself. I come from Poland and I'm a UX researcher, of course. I think I'm quite obsessed with user experience and conducting research, probably because I'm a psychologist. I mean, psychology was my major, and I was always very much into combining technology and psychology: all of the cognition, emotions, motivations of people. That's why I started to work as a UX researcher. But before that I was working for a marketing research agency, which was Hunter [?]. Then I jumped into a marketplace platform. Probably you don't know it, because Allegro is mostly focused on the Polish market, but it is one of the biggest marketplace platforms in Europe currently, because they acquired some different platforms, and I think you can check this website, because it's very interesting.

[00:02:03] Why am I saying that to you? Maybe it's not very relevant for now, but I decided to change jobs about eight months ago and start working for Acronis, which is a cyber protection company. And now I will have a bridge, because once I communicated that to the stakeholders and my colleagues from my previous job, they started to ask me what kind of research I would actually do at a cyber protection company. For their understanding, a cyber protection company is very deeply connected to development and back end and all of those technical solutions, and they were worried that I would have nothing to do there.

[00:02:51] At the beginning I was like, "Oh my god, guys, of course I will have plenty of work." But their question raised different doubts in me. If we are talking about marketplaces, most of us understand that product, because we are buying different things online, sometimes we are even selling online, so we have a general understanding of how marketplace platforms and products work. But what about a cyber protection company? I don't know about you, but before joining this company and having the first interviews during the hiring process, I wasn't very much interested in that. And by the way, I think cyber protection is very important nowadays, because we all jumped into digital environments, and we have a lot of smart devices connected to the Wi-Fi. So I started wondering, okay, do I really understand what I will be doing there?

Research across the product development cycle

[00:03:58] But then I started thinking that I was supposed to follow the research process which I know, and follow the steps which were already introduced in my previous jobs. What you are seeing right now is the product development cycle, and this is the framework we introduced at Allegro. By the way, you can read about it in an article, because my previous boss, Alina Magoska [?], recently wrote an article about Allegro and about collecting insights, and she introduced that cycle as well.

[00:04:35] Briefly, what I would like to show you here is that we are supposed to conduct research at very different steps of the product development cycle. Starting from strategy, when we are deciding why we should do something and what we are supposed to do. Then we go to the exploring phase, when we need to explore what we are supposed to do and in what way, just to fill the gaps in the market or to attract more people to our product. Then of course we have the design step, which I believe most people associate with UX research activities, because if we think about UX research, most people think about conducting some usability testing sessions, evaluating an interface and how people interact with it. Then, after developing, we can also have some A/B tests and conduct UX research by conducting some testing.

[00:05:31] But what I would like to say to you here is that we are supposed to cover not only those two steps, designing and developing, but also very different steps like exploring and what's happening after releasing features, because this is very crucial for us if we would like to have an impact. As a UX researcher at Acronis, we try to follow all of those steps. It's not always possible, because we don't always have resources, we don't always have people to cover all of those steps. But we are supposed to be in those steps of exploring and delivering, because even after releasing features we are supposed to monitor people's experience, just to check if they are satisfied enough with our product, or whether we're supposed to change something.

[00:06:20] That's to give you a bigger understanding of how I work and which framework I follow. This is the product development cycle, and I really recommend anyone, if you are not doing that yet, because of course I don't know all of you, to start thinking about implementing research at very different steps of the product development cycle.

Three types of users, and the MSP quiz

[00:06:44] Jumping into cyber security products, I would like to tell you a little bit more about the users which we need to study, because we have three main types of users. First of all, we have individuals: people like me, who have a lot of devices like computers, laptops, mobile phones, etc. We have solutions for protecting these devices for private households. Then we also have more complicated products for business. These users are completely different from individuals, because they are protecting not only themselves but also their workstations, their employees. So this is a different group of users. Of course it's a more complicated product, and also more, how to say, complex. Complex is a better word than complicated.

[00:07:43] And then we have MSPs, and MSPs are a very specific group of users. To be honest, I hadn't heard about them before joining Acronis. Just to try to engage you a little in my speech: what do you think, guys? This is a very simple quiz for you. What does MSP mean? We have three answers here: A, managed service provider; B, Medicare secondary payer; or C, managed service program. Try to think about that. Maybe some of you know the answer, so let's try to guess. I will give you a few seconds, and I will count up to five and show you the answer. One, two, three, four, five.

[00:08:30] MSP means managed service providers. This is actually very interesting, because I don't know if you had that knowledge before, but I didn't, for example. An MSP is a third-party company that remotely manages a customer's information technology infrastructure and end-user systems. That means installing things; they manage all of the IT structure at the company. So this is even more complex than business. These people are very smart when we talk about technologies and understanding cyber protection solutions, backups, etc. So this is also very interesting.

[00:09:13] So at Acronis we have three different types of users. Why am I saying that? Because if we are conducting research, we need to understand who our audience is, and we cannot combine the research for these three different types of users, because these groups are completely different in terms of needs, problems, and also experience and their challenges in daily activities using our products. So first of all, of course, we can conduct research about the three different types of users.

Tracking the user life cycle

[00:09:46] Then I would like to introduce you to a different idea. I don't know how you work at your company, but if you are an in-house product team, a good thing to do is monitoring the life cycle of users. We are supposed to split them into at least three different groups. For example, we are supposed to track the experiences of new users. Let's say this can be a group of users who logged into our product and started using it in the last six months. Then we have current users, who have been using it for more than six months. And then we have churned users. I don't know if you are familiar with the term churn, but churned users are the people who decided to leave our product, to not use it anymore.

[00:10:34] This is also a very important group to track the experience of, because we really need to understand the reasons for leaving our company. Maybe we didn't match their expectations in terms of product. Maybe it was more related to the service, or maybe the pricing, or maybe the reason is completely unconnected with our product. Maybe they just decided to close their company. So once again, these are three very different groups of users, and we are supposed to follow their motivations, drivers, blockers, needs, problems.

[00:11:14] Because what is the main focus of most products? Maybe not all of them, but we would like to have a bigger base of users. We would like to increase the number of users, and at the same time we would like to decrease the number of churned users. So it is very interesting to understand them deeply at different steps of the life cycle. I'm also adding one more group here, because we shouldn't only be focused on current users who are already engaged with our platform or product or service. We are supposed to also track possible users. Even if they haven't decided yet to start using, let's say, the trial version of our products, we are supposed to also understand their motivations or their obstacles, why they haven't decided yet, whether we can do something to serve them better, just to engage them with our product and make it more useful, so they switch into production with us.

Testing scenarios before implementation

[00:12:18] Okay, but of course it's not only about different types of users, because as UX or product designers or product researchers or product managers, we are also supposed to follow different scenarios and track them. For example, at Acronis we can test the checkout process on the website: how people are buying the product for individuals, for private people. We can test the registration flow, the trial flow, how people are learning our product and our features, so this is basically onboarding to our product. We can also of course test very different features, like creating backups, etc.

[00:13:12] What's very important here is to conduct tests before implementing changes or implementing features or new functions. I think it's quite easy to understand that if we are too late with our test, there will be no time to implement changes, to implement improvements. So usually we are supposed to do it before implementing changes. If we have an exact time to implement some new feature, we are supposed to have at least one month before that implementation to conduct the tests. Because once something is implemented, it's usually quite hard to change, especially if you have quite a major company with a lot of employees and very different roles.

[00:14:03] What I would like to tell you here is that of course in a cyber protection company we have a lot of interfaces, a lot of different features, a lot of products. So I can assure you that we have a lot of tests, meaning usability testing sessions, in the more visual and interface phases.

Choosing the best research method

[00:14:22] Then we have methods, and now I would like to jump a little to this topic of choosing the best method, because of how I understand that process. It's not only being on the step of, for example, setting the direction of our product strategy or exploring what we are supposed to do next. It's also about choosing the best possible methods. I think there's a lot of description of the very different techniques which we can use as product people, but there's not a lot of good material on how we are supposed to choose. So I would like to give you some tips here which you can follow. I believe it will be interesting for you.

[00:15:12] In my example, I don't start the project if I haven't answered some very simple questions. What is the purpose of the project? For example, what KPIs would we like to follow, what metrics do we have? What kind of questions or hypotheses or assumptions do we have? What decision would we like to make based on that research? And how much time do we have, of course, to deliver the results? Basically, these questions are a scenario for a stakeholder interview, and again, you will find a lot of templates on Google. I think the most important thing here is just to start doing that stakeholder interview, because without answers to those questions we will not be ready to choose the best method. So we need to have that stakeholder interview.

[00:16:08] After that, we need to understand the method limitations and choose the best one for the project we are currently facing. This is very important. Of course it's easy to say, "stakeholder interview, we're supposed to choose the best method," but how to do it? What I usually use is this table. You can of course use it later, or just find another source for it. What are we supposed to do? We are supposed to check what limitations each of the methods has. For example, we cannot generalize conclusions to populations if we are talking about qualitative research, like observation or interviews. Of course it's easy to understand that we will not conduct 100. I mean, we can conduct 100 interviews, but most often we will not have time for that.

[00:17:08] Sometimes, if we are time sensitive, we will not use those methods, because we need to follow a specific timeline. Or if you would like, let's say, to make some very strategic decision, it's also better to use more quantitative methods like surveys or analytics, because we can generalize those conclusions to populations. Why am I showing you that table? Just to show you that we have a lot of different methods, but we really need to understand when to use them, what the best usage of those methods is, and what their limitations are. For example, sample size is also one of the factors which has an impact on our methods, so we're supposed to take that into consideration.

[00:18:06] Probably most of you know the NN Group, the Nielsen Norman Group, and they have an article about which UX research method to use. They also show that we can use very different methods depending on the product development stage we are currently in. Let's say we implemented something, we released some new feature. Then we are supposed to use usability benchmarking, or unmoderated UX testing, or surveys. Why? Just to monitor how people interact with our product, what they think about the product after release, and whether they are facing any problems we are supposed to solve, just to keep them engaged.

A second quiz: property insurance

[00:18:53] Okay, I have a second question for you here, because we don't have so much time, but probably we will have time for one more question. Let's imagine that you work for Amazon or any other marketplace platform, for example Allegro, and they are considering starting to sell property insurance offers. This is a new project for them, and they do not deeply understand the specifics of that market. That's why the project team wants to understand how people buy insurance and what they are guided by: what the drivers and motivations are, what is important to them in property insurance offers, and what concerns they have about buying insurance. Try to think for a moment what the best-suited research method is.

[00:19:45] First of all, what you are supposed to think about here is the product development stage: whether they are at the strategic step, the exploratory step, or delivering or designing. This is the first factor you're supposed to think about. The second is what kind of questions they have: whether they are very specific, or quite open. And what will be the best method to answer these questions? Again, I will give you the answer quite quickly. In my understanding, this is interviews. Why? Because we can deep dive into people's motivations to understand their reasoning. And also property insurance is quite sensitive, because we are talking about money. So it would be good to do that in person, by individual interviews, not by focus groups, let's say, because it will feel safer for people, and we will still have the possibility to ask additional questions.

Tips for making research count

[00:20:50] Okay, actually this is the last slide, and I think I'm on time, so I would like to give you a few tips, because UX research is all about impact. We are not delivering results only for us, we are not delivering them only for the design department; we are delivering results for the whole company. So we are supposed to follow the product development cycle, as I said at the beginning. We need to feel the heartbeat of the organization, and we are supposed to support the different stages of the product development cycle, from strategy up to monitoring released features.

[00:21:28] We are also supposed to deliver conclusions, and conclusion here is the key word, because we're supposed to explain the consequences and impact of the described results. We cannot only describe them; we need to talk to people about the consequences of those results. We are also supposed to share the results with stakeholders, not only with our internal design department or product department. We are supposed to evangelize, and share the results of that research by, I don't know, sending a newsletter or having monthly calls to present the most important results, etc.

[00:22:08] Also, to make an impact, it's very important to have some kind of UX metric. We can use different surveys or metrics, like the System Usability Scale or the customer satisfaction score, or Net Promoter Score if someone likes it, because I do not. But what I'm trying to say is that you're supposed to introduce some kind of UX metrics, and if they are quantitative it will be better, because usually, if we are talking about some strategic decision, it's better to follow some bigger numbers than insights from only one qualitative [?] research study.

[00:22:44] As UX researchers and people involved in product, we're supposed to work together on recommendations. We're supposed to cooperate with different roles on improving the product experience. Our work is not done by only delivering the results. We're supposed to also be part of the project team that is working on the recommendations.

[00:23:06] And the last tip, which I feel is very, very important: don't be late. We need to be really on time with delivering the results and giving the insights about our users and our product, because if we bring them too late, it will be too late to incorporate those results when we are making important decisions. So guys, remember that we need planning. And I'm not talking only about planning what to do, to conduct a usability testing session or a survey, but the planning process in terms of designing a roadmap. Conducting that research on time is very crucial here, because if we are late, we will not have an impact on the decision-making process. Guys, thank you. I think that's all, and I'm ready to answer your questions if you have any.

Q&A

[00:24:08] Host: Thank you very much. I thought that was a really interesting example of the best ways of moving through the methods, because it can be overwhelming when there are just so many different methods.

[00:24:20] Łukasz: And it's only a few examples, because there are a lot of different variations of that, so we need to follow very different rules. Sometimes it's very hard to choose the best one, but as I said, we can follow some rules, set up some process, and it shouldn't be so complicated once we start to follow that framework.

[00:24:47] Host: Great. If anybody else out there has any questions, please do post them in the comments on whichever platform you're watching this on, and we will post those questions through to Łukasz. But I'm going to start off the questions. You talked in a lot of detail about picking the method, but with the scenarios that you mentioned up front, how do you help teams avoid getting overwhelmed by too many scenarios, and help them to pick the best ones that they should start focusing on?

[00:25:20] Łukasz: So you are asking about introducing more features than we need, more scenarios. Of course we can do that in very different ways. For example, we can conduct some card sorting, to give that choice to our users. We can ask them what is the most important for them in terms of functionality. We also need to think about the cognitive effort of people who are interacting with our product, because basically, if we show them too much, they will not understand anything. So it is very important to somehow limit those scenarios, and of course as UX researchers we are supposed to give that recommendation to the product team.

[00:26:09] For example, we can use card sorting and ask people what their needs are. Or even better, sometimes we can check in analytics which scenarios are the most popular on our websites, what people use most often, just to check not only opinions and what people are guessing, but how people actually behave. So this is between UX research and analytics here.

[00:26:40] Host: Brilliant. Your point there about opinions leads perfectly into my next question. In your second example you gave the difference between interviews or a survey for trying to gather information from people. Some of the pushback that I've seen is, "Let's just do a survey, because we can get a bigger amount of data." That kind of pushback: interviews will take too long, and it's too little data. How do you try to negotiate or explain to people that, yes, interviews will be a little bit smaller, but it'll be much more valuable data?

[00:27:19] Łukasz: Okay, it's quite a complex question; I can answer it in very different ways. Let's start with this: usually the golden rule and North Star here is to have some kind of triangulation, to use very different sources of data. We can have both interviews and surveys, and that would be the best, because we would have a bigger understanding. Let's say interviews, surveys and analytics. But most often it's impossible, and we do not have enough time for that, and of course we cannot follow that protocol of triangulation using three different methods.

[00:27:55] Sometimes people think that surveys are much quicker, but it depends, because you need to prepare that survey, you need to send it by an email campaign, let's say, or we need to show it as a popup in our interface. So it's not always faster. But of course the reasoning that we will collect much more feedback is very true. So it depends on the decision we need to make. If we are deciding about redesigning the platform and the homepage, let's say, of some product, it will be better to ask more people if they have any problems with it. But if we would like to understand exactly what the problem is, we are supposed to check that in in-person meetings.

[00:28:52] So it depends on the product team we have, and it depends on the UX maturity level of our company. But I would suppose we need to start with understanding what decision we need to make. Sometimes we need to explain that this method is not better, because it will give us only a general understanding. We can check it in our data, and to have a bigger and deeper understanding we are supposed to ask at least five people, and it won't take us more than, let's say, three days. So we need to negotiate that somehow, and play with the cards which they offer us.

[00:29:38] Host: Excellent. Well, thank you very much, Łukasz. That was really informative, and I think it was Ara [?] who said as well, "Very informative, thank you," in the comments. So it seems like your talk resonated with people there. Thank you for your time.

[00:29:54] Łukasz: Yeah, thank you very much for having me once again. And if anyone has follow-up questions, do not hesitate to reach out to me later. I'm here for you guys. Thank you.