How Useberry Enables Continuous UX Research for Ongoing Product Improvement

Harry Karanatsios at UXDX Community: Luck, Trust & Design: Building Resilient Teams & Secure Interfaces. Video: https://youtu.be/HyuhwiVAI2M

Readable transcript: edited from the recording's captions for readability (fillers and false starts removed, punctuation and section headings added). Wording is the speaker's own. Timestamps are positions in the video. Names marked [?] could not be verified against the audio.

Why continuous research matters

[00:00:07] Harry: Hey Rory. Hi everyone. Thanks so much for joining today. I'm Harry Karanatsios, senior UX researcher at Useberry, a UX research platform that helps customers conduct remote and moderated research. Today I'll be walking you through how we conduct ongoing UX research to support our continuous product development and inform our roadmap. What we will be seeing today is the UX process that we have set, why continuous research matters, how we collect insights, how we share the insights with different teams, and at the end we'll bring a real-life example of how research led to and shaped a product decision.

[00:00:53] So let's jump into it. User expectations and behaviors are always changing. If we only research once, at launch, or when something is broken, we risk missing out on huge opportunities or critical problems. Ongoing research helps us stay aligned with real user problems and needs, validate our assumptions, and deliver a useful product that's also viable for the business. And when I say useful product, I mean a product that is usable but also offers the features that customers need.

[00:01:32] Through our process, we gather data from different sources in a continuous manner. It's like always having a conversation with users, not just a one-time interview. That way we know what's working, what's not, and what should come next.

The process and its loops

[00:01:50] As you can see on the slide, we start with planning, going to discovery, then we explore solutions and test them during the design phase. Then we go to development, and we set up our listening loop. Listening helps us decide what problems we should focus on and informs the roadmap items. Based on the roadmap planning, we conduct upfront research, upfront discovery, for the items on the roadmap. And when we get to the design phase, we explore solutions and test them in an iterative manner before we get to the listening loop, once the product has been developed and it's live. Many times we are also undertaking UX activities in the development phase.

[00:02:39] Now, these loops ensure that research isn't just a side project. It's deeply considered throughout the product development phase, and it's also considered by the business before the business decides on the roadmap items that will serve the company's strategic vision. A small note about the slide: we also have other loops that might happen throughout our process, for example going from design to discovery. But such loops are not shown in this diagram, as we will not be talking about them during this presentation.

Listening: channels and methods

[00:03:17] As I mentioned, we start with listening. We listen to our customers for everything new that we develop. We might capture metrics per quarter for a new feature to measure success or benchmark it, or we might capture improvement ideas via different channels and methods. At the same time, we have set ways to gather customer feedback around the current service offering, which results more in needs, usability issues, friction points and pain points with our service.

[00:03:57] We constantly learn from users via the following channels and methods. We do, of course, interviews and conversations with users. We look at customer support logs. We do follow-up Q&A calls, demo calls, onboarding calls, surveys, analytics, email comms, expert reviews. Sometimes we even get into ad hoc calls with customers to help them create a test with us. This has a tweak. The tweak is that we get closer to usability testing. We ask people to do what they asked us to help them with, and then we intervene when we think it's appropriate. We do this in collaboration with SCS [?].

[00:04:44] Some of these methods and channels we are listening to are automated, while in some others we collaborate with different departments, or we even take Q&A calls, or we participate in demo calls. So it's a team effort. We also do some things quarterly, for example competitor reviews or benchmarking.

[00:05:12] We do all these methods, and we are trying to listen to current and prospective customers to help direct future efforts, by understanding who our target group is. What problems or pain points do they face? What pain points and problems affect them? This can be with our service, a competitor's, or with their current process. But we also like to learn what they need and what they value from a solution. Throughout the listening, sometimes we might also capture the why and when a problem occurs and the effect it has on the customers. I think it's equally important to try to surface the effect that a problem introduces to the business as well.

[00:06:01] And throughout listening we sometimes also capture why a solution or a feature is needed, or its perceived value to customers, and the effect it will have on the customers if offered or if not offered, while at the same time, again, surfacing the effect that it has on the business is also important.

From data to roadmap to discovery

[00:06:23] We capture loads of data, and this data comes in different forms. It can be questions asked by customers, or notes made during an interview. Really, based on these methods, you can understand that we receive a plethora of data stated in different forms. Once the data has been captured, we store it, we categorize it and we analyze it. We organize the data that we have heard to generate ideas, and we think these ideas might satisfy the captured customer needs and/or solve the problems they are facing. These ideas are prioritized before being placed into the roadmap. And of course, each item that is selected to get into the roadmap contributes to the company vision and strategy. Essentially, we are always trying to create a win-win situation between the customer and the business.

[00:07:25] At this stage we might have a more detailed understanding of a problem, or a more generic overview of a problem. Personally, I think it's always best to continue with an upfront discovery for each item on the roadmap, ideally one or two months before any design activities take place. Throughout this discovery phase, our goal is to better understand the problem space by surfacing what we don't know and better understanding the customer's needs. At the same time, this is an opportunity for us to validate and discard assumptions before delivering insights to the team.

[00:08:08] There are a couple of activities that we do here. Predominantly we do interviews, but we are also analyzing sales, customer support, onboarding and training calls, and logged tickets, depending on what we have. We might be conducting competitor reviews or competitive testing, and we might also be doing usability testing in this phase.

Explore and test, then development

[00:08:35] During the explore and testing phase, we create solutions and concepts to test based on the knowledge that we have gathered. Usually this is done via usability testing in an iterative manner. Throughout this phase, new knowledge might be gathered that informs, or can inform, the original requirements and thus the design. Not always, but when necessary, we start with card sorting and tree testing before creating prototypes. And many times we also do expert reviews before any usability testing takes place.

[00:09:19] Going from design to development, we also conduct usability testing during the development phase, predominantly when we have a staging environment, or in a closed beta environment. There are other activities done by the UX design team here, which I'm not going to be expanding on.

A real example: the randomization feature

[00:09:45] Which brings us to the live example that we have. We will be speaking about the randomization feature. At Useberry, we noticed in data from customer support that customers are asking how to apply randomization to their usability testing tasks and tree testing tasks. By looking at the ongoing monthly interviews that we do, we noticed in the data that there is a need for question randomization as well. We went further and we examined competitors, from the competitor review that we do quarterly, to see what the competition does, whether they offer this feature or not, really high level. And we looked into more reports that we had, and this particular feature was proposed by experts, and there were other sources indicating the need for such a feature. So we thought it was worth allocating time to conduct a discovery.

[00:10:50] Throughout this discovery, we conducted interviews and desk research to understand the benefits and the use cases of randomization, and the problems introduced by not offering this feature. We found out that many customers would like to randomize the way tasks are shown to participants, to reduce the learning effect. Or we heard customers would like to randomize the questions that follow up a task alongside the task. We also conducted another competitor review here, from scratch, particularly for this feature, to get a more in-depth understanding of what use cases the competitors' randomization features cover and what problems they solve. This was done before we moved to the explore and testing phase.

[00:11:43] Throughout the explore and testing phase, we initially conducted usability testing with some concepts. At this stage we heard that some customers would like to limit the number of tasks shown to participants, to reduce participant fatigue. We also captured some other use cases that were not discussed earlier. For example, participants would like to randomize only part of the test.

[00:12:14] Finally, we had some participants mentioning that with one of the concepts that we presented to them, they could also do A/B testing. That was very interesting, because through our listening loop we had already heard the need to offer an A/B testing feature that would allow customers to conduct within- and between-subject studies. So based on this data, we brainstormed a new solution that could offer A/B testing and randomization, and we placed it in a design and test iteration before continuing with our dev handoff. This way we killed two birds with one stone.

[00:13:03] So initially we heard about this feature via our listening loop, and before the feature was released, we defined how to monitor it, how to measure its success, and how to capture future improvements. For this particular project we didn't carry out any tests throughout development. We could have, or wouldn't have had enough time, and my suggestion would be to reduce the risk as early as possible. So doing usability testing when it comes to the development phase is equally important.

Key insights: triangulate and adapt how you share

[00:13:47] So, some key insights about what we've heard today. My suggestion is to always try to triangulate. What do I mean by triangulation? Triangulation means using different methods, data sources and researchers to cross-check and validate your findings. This helps make your results more reliable and trustworthy. What we do at Useberry is we normally triangulate by gathering different sets of data, gathered by different methods and sources, and, when possible, involving more than one person in the analysis. Either just one person does the analysis, or we use two people, and they can do the analysis in a collaborative manner from the start, so get together and do the analysis together to deliver insights, or do the analysis separately and then meet to discuss the findings and connect the insights. The approach is up to you.

[00:15:03] Another interesting part is that you should adapt how you share your research and your findings, so people can actually listen. Depending on the phase of the process that you are at, or the project, you might have to speak to the business, the product team or the design team, or even the marketing team or another stakeholder. So explaining the effect a problem has on the customer and the business, and how it will benefit both, is always a win-win situation for me. And connecting findings, problems and solutions with metrics that matter for the business or the stakeholders is equally important and helps a lot. Trying to create easy-to-digest insights is also important. Depending on which phase of the project you are at, you might want to bring something more detailed or something less detailed, but try to make it easily digestible.

[00:16:05] And if possible, involve the team members in the research sessions. I've found this very useful, especially when I'm collaborating with the design team to iterate on design solutions in a qualitative usability testing manner. The team is aligned, everything moves quickly, and there is less need for documentation and time-consuming back and forth.

Useful, viable, feasible

[00:16:36] One last thing that I would like to pinpoint here is that at Useberry, at least, we try to make a useful product, a viable product and a feasible product. So useful, viable, feasible. Useful refers to a product that provides the features that the target group needs and allows them to use these features in an easy and pleasant way, minimizing errors and friction points across touchpoints. Viable refers to the product being connected with the business value, strategy, vision and pillars. While feasibility refers to a product that can be built the way it was designed, in time and of course in budget.

[00:17:22] The most important thing is: keep doing research, keep listening. Don't just wait for a design or a problem. Make it part of how the business grows. By continuously listening to our customers, to our users, we're not only catching problems and needs early, we uncover new opportunities for growth and reduce risk. Ongoing research isn't a luxury. It's essential to building products that succeed. That's all for me. Thank you, and I'd love to hear your questions.

Q&A

[00:17:58] Rory: Excellent. Thank you very much, Harry. That was a really interesting overview of, I guess, how you use research to build a research product.

[00:18:06] Harry: Yeah, it was quite high level, I'll have to agree. But I didn't want to get into the nitty-gritty details of how to do discovery and iterate. I think the focus was mainly on continuously listening to customers, by collaborating with different departments, or many times by offering yourself, for example, to customer support, to see how they do their work, what questions they get, how we can even improve the experience that customer support offers. Because the experience is not just digital. It goes across many different channels. You might be offering all the features and everything might be very well placed, but someone might be talking to support or searching for something online and having a really bad experience, or maybe trying to cancel the account and getting a dark pattern.

[00:19:03] Rory: Right. Exactly. Just as a reminder to everybody out there, please do write your questions in on whichever platform you're watching this on, and I'll put those questions through to Harry. But one thing that I was just thinking as you were talking about that randomization feature, and I guess this is probably going into the weeds, but how do you ensure that... As a person, I might unintentionally create linkages between my questions, so that when they're randomized it breaks things. Is there some way that you can solve for that, or do you just try to educate people not to do it?

[00:19:36] Harry: I'd say not always. If one question leads to the other and one question has to be answered first, you have to do it this way. The same goes for tasks in a usability test. Before registering you might have to do KYC. It's an important step, or it's a step of the registration, so you cannot just ignore it. But if the questions are not connected and there is no particular reason to have them in sequence, I would advise randomizing them, to avoid sequence effects.

[00:20:13] When it comes to usability testing: when I'm seeing the same interface again and again, I learn the interface. So if I go from task one to task two and task three, by task three I would have already learned a lot about the interface, which will make it easier for me to use it. That might lead to me, as a researcher, not evaluating this particular task objectively, because I don't actually know which of the tasks participants will do first when they get into my service. So randomizing them makes sense for me, to reduce my bias and make my results and metrics more sound.

[00:20:59] Rory: Excellent. And I guess, just given Meline's [?] talk earlier: when you're embarking on a new initiative, do you create a detailed research plan, or how do you go about kicking off your research initiatives?

[00:21:16] Harry: In discovery and design iteration, many times we do a detailed plan, not as detailed as we saw earlier, but I'm inspired by the idea. I'll see how this could work for us. When we go to the listening phase, I think most things reside in a repository. So it depends on us to go ahead and look at it and find insights and findings that can help us create ideas that solve customers' problems.

[00:21:53] Rory: And because you mentioned that continuous nature, do you find that you end up pivoting a little bit, because you're learning something from customers and it's invalidating some previous assumptions that you had, so you end up going in a slightly different direction?

[00:22:06] Harry: Yeah, sure. And this is part of the process: validating and discarding assumptions. We might have an idea that we think is great, but in the very end it needs fixing, or it needs updating, or adding more features. So we do the listening to get the high-level idea, the discovery to get into more detail, and throughout the design and testing iteration we might be hearing a lot of things. It's qualitative testing; people are placing their thoughts on the table and we listen to them. So we might jump back and rediscover, or update our vision for this particular feature, or even bring something back to the planning phase, so inform the business about something new.

[00:22:54] This is actually what happened with the randomization feature. We started by focusing on randomization, but through our testing, apart from new use cases, we understood that, hey, participants said to us that this could also be used for A/B testing, and we knew that we needed A/B testing. We just couldn't make the connection. So these sessions helped us connect the two insights, and, you know, two birds with one stone, just one feature: Groups and Random. That's what we call it.

[00:23:33] Rory: Excellent. That's always what you're looking for. I guess one question that always comes up with researchers is how difficult it is getting in front of real customers. You mentioned, again, that continuous nature. Do you find that difficult, or are you blessed because your customers are researchers and therefore know your pain and are willing to help out?

[00:23:57] Harry: To be honest, we have some initiatives here. I take a lot of the demo calls or the Q&A calls myself, and of course I know the platform very well, so I can help our customers. But when I undertake the sessions, I ask particular questions. So in a demo call I might ask, "Hey, what are the things that you are looking for within a UX platform?" This helps me understand what this persona, or this set of people, are looking for within a platform, and it also helps me showcase the platform more accurately to them. So it's a win-win situation. I might ask them, "Hey, what difficulties do you face?" or "Are you switching from another provider, a different service, a competitor? Why?" So I'm trying to capture bits of info throughout the sessions. It's not a lot, but it's happening in context, as the customer has the need and they're willing to speak. It's not an interview. It's more natural. It flows more naturally.

[00:25:02] The same goes for Q&A sessions, or the sessions where we help customers achieve something. We might just ask them, "Hey, how did you try doing this?" And we observe, and when they get stuck we intervene and say, "Okay, you could click this part." But we have gathered our insight. And it doesn't stop there. We encourage them to continue until the end, so we can explore what further could be discovered in this particular journey. So that's also another interesting approach. We do have very engaged customers, but customers reach out to you via different channels every day. If you can work with colleagues to leverage the data, or expose yourself to these processes and try to make the process better, but also capture your data while helping the customers, that's a win-win situation for everyone.

[00:26:01] Rory: Brilliant. And do you volunteer your time to other teams, like sales or support? How do you get that collaboration?

[00:26:10] Harry: We have things set up. Not all the calls, not all the demo calls, go through me, but a plethora of them do go through me. We have some days when we do interviews or usability testing, and we know that I cannot take any calls that day, or a couple of minutes before or after these calls. Any other day, I'm happy to get on a call with customers, because I'm learning without having planned, giving incentives or doing anything. I'm here, I'm sitting, and customer support says, "Hey, demo call, would you be available tomorrow?" Yes. Or they might be looking at my calendar and see, okay, Harry's available, so I might pass this to Harry as well. So recruitment many times is a pain point, but everyone has customers reaching out to them. Try to engage with them and collaborate with other departments.

[00:27:08] And again, collaborating with other departments helps you refine processes that go outside of the UI, service processes. If I order an item and everything is perfect on the website, but I receive a broken item, that's a bad experience. The wrapping might not be good. There might be an issue in the factory while manufacturing it, or anything in between. But this is a bad process, so we have to hear about that experience and try to improve it. It might be just one out of a million, and we might take this risk. But if we don't hear it, we cannot evaluate whether it occurs more often or not. If it doesn't occur often, that's a risk we might be willing to accept. But if it occurs often, we have to ask: are we willing to take this risk and have customers buying and receiving a bad product or not? What should we refine to improve this process?

[00:28:12] Rory: Yeah. No, it's a great example of just taking the initiative, which a lot of people sometimes don't even think of, getting involved.

[00:28:20] Harry: There's also a lot of jargon there. Many people say continuous discovery. The design sprint might have different words. I've used listening as a loop. Many people might call it different words. That's absolutely fine. My main point here is to set a process, set a vision, set a strategy to listen to customers one way or another.

[00:28:50] Rory: Brilliant. And you also mentioned getting the rest of the team involved, because, as you said, and I fully agree with this, if somebody can see the pain points or the challenges directly from a customer, it really hits home a lot more than something written in a report. But have you had pushback from the team going, "Oh, we're too busy," or different things where they don't see it as a proper part of their job?

[00:29:14] Harry: I've seen both sides of the coin. I've seen teams that don't want to get involved as much, and teams that really want to get involved. I recall back in the days at a gambling provider, we had 20 people in the back room, all the designers, all the product team. And there have been cases where there was just the agile team, or other cases where I might just have a designer. I'm referring to the design evaluation phase at the moment.

[00:29:49] So I think having people in the back room helps a lot. It helps a lot because everyone is there. You can do quick debriefs after each session, have a journey map projected onto a board, a Miro board, whiteboard or digital board, it doesn't really matter, and start building a common knowledge, a common agreement of what happened. Which means that at the end of eight sessions or six sessions, we all know what happened. We have all agreed and placed it into a journey map, and no one will come back to me and say, "Hey, where did you observe this? I think it's something else." Everything would have been discussed earlier on.

[00:30:35] And many times we have this loud voice, a PM or a PO or a stakeholder, where it's "my way or the highway." But if you have three or four team members, and the PO or that stakeholder gets defensive, let's say defending their point of view, then via conversation they will understand that everyone else observed something different. So if we go their way, they will be the ones taking the risk, because with all these processes we are trying to remove the risk, or make it smaller compared to what it is.

[00:31:15] If we go back to the listening and discovery phase: in the discovery phase, getting people into the sessions is also important. It's normally more time consuming, but throughout the conversation and the interviews, unstructured or structured, more things and questions might come from the back room. So it's a great opportunity to ask questions of customers. And in the listening stage, I think we have everything tidied up in our repository, so anyone can go ahead and look. And this is a collaborative effort, so we allocate time to synthesize what we found into problems and ideate on potential solutions. That's also a team effort most of the time.

[00:32:03] Rory: Brilliant. That brings us to time, but that sounds fantastic: get everybody involved earlier, and you'll save time on the back end by not having to go around in loops fixing things.

[00:32:14] Harry: Documenting is good, but if you need to work quicker sometimes, you might document less, or at least take a picture of what you have on your board, especially in the design iteration phase. It works wonders for me. It has worked wonders in the past. And I'd suggest and propose you should try it out. If you have any questions or you want to chat with me more about how to do any of this, or get any extra advice, let's talk. You can find me on LinkedIn, or you can see my email on the screen. It's h.caronisbury.com [?].

[00:32:53] Rory: Excellent. All right. Well, thank you very much, Harry. I hope everybody out there enjoyed it as well.

[00:32:58] Harry: Thank you. Thank you as well. Have a great evening.