UX Researcher: Team of one

Swati Sachdeva at UXDX Community: From Conversation to Code and Research to Impact. Video: https://youtu.be/2VRDvHuOEgM

Readable transcript: edited from the recording's captions for readability (fillers and false starts removed, punctuation and section headings added). Wording is the speaker's own. Timestamps are positions in the video. Names marked [?] could not be verified against the audio.

Being a team of one feels like the fire department

[00:00:07] Thank you so much, Rory. I'm sure it's evening time for you, but in CR [?] it's a great morning, and Amit the case of the city [?]. I would like to start this presentation and this conversation by talking about how being a solo researcher is often fantasized about, and then over time you become an indispensable part of the team, the only member who understands the users and can talk from their perspective.

[00:00:40] In reality, being a team of one feels less like being the research department and more like being the fire department. You have one truck, and the building keeps catching fire again and again. Before I go further, I want to give a disclaimer that whatever I am sharing is my opinions, my learnings and my experience, and does not represent anything from my company, my client, my customers or my users.

[00:01:13] I talk to a lot of researchers, whether it is in meetups, in conferences or in work settings, and I ask them about their work. Here and there we hear a researcher talking about making a terrible mistake at work. Those mistakes come not from lack of skill, but from misreading the kind of problem that was in front of the researcher. By the time they realize it, the moment to prepare for it is already gone.

[00:01:41] In this presentation I want to talk about a way to see it coming, through the lens of something that we can all relate to: going from a junior researcher to a senior, as a team of one. I say team of one because whether you're working in a startup, in a small company or in a big tech company, you're more likely than not working as a solo researcher.

From junior to senior: diagnosing before acting

[00:02:06] As a junior, you treat every request that comes onto your desk the same way, usually with whatever method you are best at. I am guilty of that: using usability studies and surveys for everything that was coming my way, because at that time I did not have a way to tell the requests apart. That works for a while, and then it doesn't. You'll burn a week on something that needs just 10 minutes, or you'll give a 10-minute answer to something that really needed deep analysis and a week's worth of work.

[00:02:46] Somewhere in between all of that, you will have a request come your way that catches you completely off guard, because you have never seen such a request before, and then you panic. But when you become a senior, you start diagnosing before you start acting. When things are normal, you start building the infrastructure for all of those problems that might show up in your work.

[00:03:11] The gap between reacting well just once and being ready every time a request is thrown your way is based on recognizing patterns and understanding what system you are in. Systems thinking helps you segregate the problem in a way that is easier for you to act on. There's an old firefighting idea that I really love: a good firefighter doesn't just fight the fire that is in front of them.

[00:03:46] When they arrive at a situation, they read the smoke, because the color and the shape of the smoke can tell them exactly what's burning inside, even before they open a single door. We're going to borrow that instinct. Not because research is firefighting; most days it isn't. But reading the situation before you act is a skill that really moves the needle, and it's a skill that really comes with time.

Four patterns, and the clear problem

[00:04:17] Every problem that you will face, you can put in one of these four patterns, and these patterns are defined by Cynefin. There's a clear pattern, a complicated pattern, a complex pattern and a cataclysmic one. Why is it important? Because once you name a pattern, two things fall out of it for free. Firstly, what you need to do with it. And most importantly, what you could have done previously to be ready before the problem showed up, so that next time this problem shows up, you are well prepared for it.

[00:05:02] We will talk about these problems one by one, starting with a clear problem, or a clear pattern. Think of a time when a request came in for you and in a beat you had an answer pop into your head; you know what the answer is. For example, a designer comes to you and asks whether the navigation bar should sit at the top or the bottom of the screen. Sounds reasonable, except for the fact that it depends on the context of the screen.

[00:05:31] The UX industry has years and years of research on exactly this. Where should navigation information go? How do we make it accessible? How do we make it easy and distraction-free? For example, if you are in a car, all navigation information should be closer to the driver rather than away from the driver, because it is difficult to reach otherwise. The answer already exists and can be retrieved, and that's how you know you are in a clear problem.

[00:06:01] Either it's settled knowledge, like established best practices, conventions or prior research, or the request is so low-stakes that formal research is overkill. With a clear problem, the cause and effect is obvious to anyone who looks. Your job as a researcher isn't to be very clever in these situations. Your job is to go and find the answer that already exists. You need to make sure that you aren't overspending your time reinventing the wheel. The decision that it feeds is small, it's reversible, and it's cheap even if you get it wrong. That's how we tackle a clear problem.

[00:06:51] But there is a trap here. The trap you might fall into is treating it like a real project: running the study anyway, because saying no to it feels like doing less research and makes stakeholders feel that you are not prioritizing their work. When you over-staff such small fires again and again, that's how solo researchers drown in a lot of work which is not meaningful at the end of the day, or which won't give you as much of an outcome as you would get if you prioritized better.

[00:07:28] What do you do when you're in the clear pattern? You don't get faster at studying it. You get faster at finding out that you don't need to study it. The only reason you can catch that in 10 seconds is if you have built things that let you check, for example a searchable library of past findings and standards.

[00:07:48] Going back to the firefighting example, the fastest fire to put out is the one that you never had to fight, and that is what your research repository is. Without a repository, every clear question looks like a fresh study, and there is no way for you to know otherwise. Any time things are normal and you are spending your time organizing your files and wondering whether it's worth it, that is for when you really face a problem.

The complicated problem

[00:08:30] Coming to the next one, which is a complicated problem. What's the difference between a clear and a complicated problem? When we are dealing with clear patterns, the cause and the effect are very easy to identify. In a complicated problem, the cause is there, the effect is there, and there is a link between them, but it's a bit hidden. You need to do deep analysis, or you need to bring in an expert who can figure it out.

[00:08:59] Imagine that a PM comes to you and asks why retention has been 8% lower in the last month. There can be 10 different reasons for it, which you can guess. But if you do deep analysis, maybe you study the user data analytics, maybe you study the past complaints, maybe you figure out what really happened in the last month, you will find that of the 10 guesses you were making, only one or two of them are true. But you need to do the analysis for that, and you need expertise working on it.

[00:09:40] Another example is your leadership asking you, "Do users find our product premium?" It feels like a question you can answer in a day or two: you can run a survey asking people, "Do you find our product premium?" But it is not until you define what the abstract term premium means that you can really act on it. People would say yes or no, but what do you do with that data?

[00:10:08] It is the same as asking if your pizza is tasty and using the same measure to see if your ice cream is tasty or not. What makes a pizza tasty is very different from what makes an ice cream tasty. To answer that question, you need an expert who understands the difference and can help you get an answer that really means something: that people find the pizza less tasty and if you increase the cheesiness, that would really move the needle. Or people find the ice cream really tasty but the sweetness is a little bit too much, and if we reduce it we can get to a better outcome.

[00:10:49] How do you know that you are dealing with a complicated problem? There are multiple plausible causes, and the arguments get settled by evidence, not just guesses and not just opinions. People would agree if they had the right data in their hands, and your job as a researcher is to bring that right data. Show your rigor in your methods and show your rigor during your analysis. It helps if you follow a research plan and have a checklist in hand so you do not miss things. And if nothing works, and the situation is complicated but outside your expertise, call in an expert.

[00:11:30] This is also where I personally find AI helpful: in learning the right methods, their right execution and the analysis structure, while I have the final call on how we read the data.

[00:11:44] What could be the trap here? There could be two of them. Firstly, answering from your gut because there is no time to do the research properly. And in the opposite direction, running rigorous research for six weeks when your business wanted just a directionally right answer in one week. You're overspending on it; you're overbuilding your research.

[00:12:09] What can you do to deal with a complicated problem? Firstly, match your method to the decision, not to your comfort and the methods you are comfortable with. Match it to what decision needs to be taken and how you can give the data to move ahead in that direction. Start asking how reversible it is. If it's a high-stakes, irreversible decision, it earns heavier methods; otherwise you can bite-size it.

[00:12:44] Use AI as your second analyst. It really helps: let it code for you, let it synthesize your draft for you, and then you as a researcher validate a subset of your data and see if the output is accurate or not. And lastly, in these problems people are looking for quick answers. They want to move in a certain direction, and they are looking for that direction from you. Apply your rigor in the methods and the analysis you are doing, but not so much in packaging the findings. Once you have an answer, ship that answer as quickly as the decision needs it, and help the team gain momentum sooner.

The complex problem

[00:13:32] As we move on, next are the complex problems: the ones where there is no right answer, and in fact the answers keep shifting the more you study them. Let me start with an example. Suppose a company rolls out an AI assistant across the whole org, which might be happening in some of the orgs that you are working with. That AI assistant is applicable to each team and each workflow, and the leadership comes to you wanting to know whether employees will actually adopt it or not.

[00:14:11] You as a researcher might think, "Oh, I can just go to the employees and ask them." You can run a survey, and people will tell you what they imagine using it for. But if you really think about it, it's almost useless, because in six weeks nobody is using it the way they said they would. Some people will have adopted it. Other people will have it open in a tab and won't have touched it since day one.

[00:14:41] What people said they would be using the tool for in week one has completely shifted by week six. Employees in week six understand the tool differently, have a different sense of how they might use it, use it differently and trust the tool differently. At this point no one knows what successful adoption looks like.

[00:15:09] As a researcher in this situation, you might have this heavy feeling that you don't even know what you don't know, and that's a signal that you are dealing with a complex problem. When facing a complex problem, instead of trying to find one precise answer, we probe it. A rolling study every few weeks gets you a pulse check of what user sentiment is. A couple of sessions per week with power users and non-adopters will give you a sense of where things are headed.

[00:15:47] When you're dealing with a complex problem, having a roadmap and hiring an expert might not really help you, because of the nature of the problem. You need to conduct small experiments over time and course-correct whenever it's required.

The cataclysmic problem

[00:16:06] Then there are problems where none of your roadmaps, none of the experts and none of the experiments will work, most likely because you won't have the time to do them. For example, a newly deployed piece of software fails unexpectedly. A metric that you've been tracking over time falls off a cliff overnight. Or maybe you launched a new marketing campaign, and the reviews for your product turn hostile overnight; people are very angry at you.

[00:16:44] These situations, these problems, are chaotic ones. They're the ones that keep you awake at 3:00 a.m. asking for a resolution, where things are really getting out of your hands. In this situation, the leadership wants to know what's happening right now, not even waiting for the next sprint. You don't have the time for that.

[00:17:07] You know you're in a cataclysmic problem when your gut is screaming that you don't have the time to do research, and for once your gut is completely right. There is no direct link between cause and effect. You don't have the time to do analysis, go through your checklist and ask the experts. The mistake you might make in such a situation is analysis paralysis: wanting to know the full story before acting. But you can't afford it, because of the urgency of the situation.

[00:17:40] In such a situation, the first step is to roll back and stabilize. Act first, and then when the ground stops moving, you can stop to analyze and understand why it happened and what you could have done.

The four patterns side by side

[00:17:56] Those are the four different problems. In a clear problem, you can see the cause and effect. The answer exists; you just need to fetch it. You just need to prioritize what you are researching for, and most of the time you can say no and delegate the task. In a complicated situation, the cause and effect are hidden. You need to do deep analysis and get an expert's help to figure out what's really happening.

[00:18:25] In a complex problem, the cause and effect are there, but you can't read them in real time. You can see the relation between the cause and effect only in hindsight. So for this, keep probing multiple times from different angles, but go in the same direction. Do quick experiments to get a pulse check every now and then. And in the chaotic one, there is no cause and there is no effect. You need to stabilize first, and then understand what happened and what you could have done.

You were never a team of one

[00:19:03] Lastly, I've been saying "team of one" during this talk, and I want to take that back. Think about what actually showed up for you in each of these four problems. The clear patterns: who answers that question? You go to your research repository, search for something, find it, and give it back to the designers and the PM and whoever is asking you for it. That's the past; you did today's work in the past.

[00:19:38] In the complicated pattern, you have to build a pipeline under some deadline. The frameworks, the templates and the toolkits you make, the participant pools you collect, the analysis and the codebooks: those exist before this problem even arose. In a complex problem, too, you have that rhythm of how you run small experiments over time with your team. Who would you be doing it with? Who would your audience be? If you have any insights, how are you going to communicate them to your stakeholders, and who would your stakeholders be?

[00:20:17] Everything that you're doing has to be done beforehand. The groundwork has to be done before, and none of what you are doing in the moment when you are facing the problem is alone. In this, AI is also very helpful, because you can take all your previous data and feed it to the AI, and it can help guide you on what we know and what we can do next.

[00:20:44] There is one more thing: naming the fire won't put it out. Remember we were talking about firefighting, reading the smoke? Thick black smoke might mean one thing, and gray smoke might mean something else. But if the hose isn't connected to the water line, it's really difficult to save the building, and a lot of time is wasted figuring out what should have been in place already.

[00:21:12] If there is a fire, you can't work on your pipelines. You can't work on your water lines. You need to have something built, and you need to make sure that you're focusing just on the burning building. That is what UX research ops is for: the researchers, the templates, the participant pool, the outreach list, the repository that saves everything for you. This is the boring underlying layer that you invest in over time when things are normal, because when things really go south, you don't have the time to redo your work. And if you think of your research ops as working with you, then you are not working alone. You are not a team of one.

Credibility is the currency

[00:22:02] Again, at the end of this conversation: most researchers do not fail because they are bad at their methods or bad at executing those methods. They might fail because they are applying the wrong confidence level to the wrong situation. The cost here is not just wasted time. It's also credibility. If you are a solo researcher, a team of one, you know how important that credibility is, because it's like a currency for you, which you can spend when you are working on bigger projects, because people trust you.

[00:22:42] A team of five or a bigger team can actually afford to misjudge a problem once. Someone else would catch it, there would be some discussion, and you can rely on each other. There is some slack to redo the work or fill in for someone else. But when you are working alone, you don't have that slack. Every misclassification is a full week that you will not get back, and someone else will keep waiting for your request, and then the request gets delayed.

[00:23:17] Knowing these four different types of problems is really a way to know how much certainty you are entitled to before you claim it. These four patterns teach you judgment, and research ops is what makes that judgment affordable to act on. With this, I end my talk here, and if you have any questions, please let me know. Thank you very much.

Q&A

[00:23:53] Host: (The question is largely inaudible in the captions. It asks about junior and senior people, and how people work out what level of problem they are facing.)

[00:24:33] Swati: I have to answer this in my mind in a couple of ways, and one is just assigning what problem it is, which you can understand based on what kind of request it is and how much clarity you already have. Do you have any previous work done on it? Do you have to start on a new basis? Is it something that is in your domain or not? But this also ties a little bit into stakeholder alignment, because I was talking about what kind of certainty you have, and it's also about how clear you are on your goals and what you need to do.

[00:25:19] What I mean by that is: do you have an understanding of what the business goal is? Based on that business goal, do you understand what the drivers of success are? Let's talk about the premium problem and the tasty problem. What we're talking about is improving that premiumness and tastiness; that is my business goal. What really defines the success metrics? Is it creamy enough? Is it sweet enough? Is it crunchy enough?

[00:25:54] Based on these success metrics, I would define what UX interventions I can do, and based on these UX interventions, I will match my incoming requests. Is it matching what I really wanted to do? Is it moving my research goal forward or not? If not, I will prioritize. I will figure out what kind of fire you are bringing to my table, and I will figure out what kind of research I need to provide to you. Once you have done your interventions, based on that you can make your research plan, do your research studies and come back with an answer.

[00:26:30] Host: OK, understood. Then I think, critically, junior people are missing some kind of... (The rest of the question is largely inaudible in the captions.)

[00:27:17] Swati: Absolutely, I understand. When I was writing this, I felt that most of the research coming to me is either complicated or complex, because I need to either do deep analysis, or figure out what can be done in the situation, or I am running small experiments over time. So when it's with the users, yes, it is running small experiments over time so that you can get a pulse of it.

[00:27:44] But as a researcher, I am not just understanding my user behavior as a whole. There are other requests coming to me too. My designer asks me for something basic, which I can either do for them, or delegate, or maybe delay, or teach them how to do. So yes, it is complex, but you need to figure out what exactly the problem is that you are dealing with. If it's with the users, I'll take it, and if it's a complicated problem, I'll deal with it. But if it's a complicated problem that the designer or the PM is asking me about, maybe I'll have a different approach.

[00:28:21] That's why understanding these four different fires, these four different patterns, is important: you have a better mental model of what you need to prioritize as a researcher and how you need to deal with all of these situations.

[00:28:40] Host: One more question coming in; it's a little bit left field. Someone is asking if you have any advice on job search.

[00:28:46] Swati: I understand the market is really bad, but over time I think it's improving. Keep learning new technologies, and keep improving your methods and your rigor, so that whenever you land a chance, you're really able to deal with the problem head-on. One thing I'm hopeful about, I guess, is that it's early days, but they're going to grow.

[00:29:18] Host: Jobs actually... (partly inaudible) ...a turning point. Keep your head up. Thank you very much.

[00:29:54] Swati: Thank you so much, and I love being a part of the UXDX community.