Is User Experience Research A Science?

Piotr Sliwa at UXDX Community: Copenhagen. Video: https://youtu.be/1iwqTxrK_44

Readable transcript: edited from the recording's captions for readability (fillers and false starts removed, punctuation and section headings added). Wording is the speaker's own. Timestamps are positions in the video. Names marked [?] could not be verified against the audio.

Introduction and why UX research matters now

[00:00:13] I have to introduce myself a bit. My name is Piotr Śliwa. I've spent 15-plus years in the industry, eight-plus of them purely in UX. My sociology education led me to a career in UX, and I also have university teaching experience of my own, which you probably know if you google the speakers, but if you translate that, fine, thanks to my students for giving me that nickname [?].

[00:00:51] I come from Netguru, where we deliver digital products for our clients across the world, and the UK is a major part of that. We started 10 years ago, and more than three years ago we started building the design team, which went quite well. At the moment we're also building and developing our UX competence. We're witnessing UX in general growing in popularity, and finally, in the perception of stakeholders, company owners and clients, it's moving away from judging UX by the interface, by the visual aspect, into every single thing we do on an everyday basis: all of product design, the usability things, and predominantly research. That's what I would love to talk about.

[00:02:05] The job boards are full of UX research roles at the moment, and where there is demand there is supply, and people are trying to get into that role without the proper knowledge of how to get in there, what they have to know and what kind of skills they need to prepare. There is one source of truth, which is Google, and Google says that a UX researcher is a person who applies various techniques in order to add context and insight to the design process. It sounds pretty serious, and there's no blank on Google this time. If we say that research is needed to reach new conclusions, establish facts [?] and find problems, that sounds really serious. So you can ask the question: is user experience research a science?

The case for treating UX research as a science

[00:03:09] What I can say is that since I started my UX career, I've tried to find how to get in there, and I've been on the same path. What I found very useful, and the kind of benefits UX research as a science or discipline would bring us: first of all, unified sources of learning. Hopefully everyone here has never worked with some tutorial channel [?], but it's quite popular: someone just finds a tutorial or a not very verified source of truth, and they try to follow its ideas. Also, if it were scientific, it would change the perception of UX from wireframes and prototypes into the more serious things that UX is about. And finally, it would help us defend different research results, make them more serious in the eyes of stakeholders and clients, and definitely convince them that it is worth doing the research.

[00:04:27] It started with how we can measure something that is not measurable, which is feelings or behaviors, things that are not easy to quantify, and that's what experience is about. And also how to get the most out of the research. That kind of academic thinking, which I mentioned at the very beginning, led to the conclusion that UX research really is a scientific discipline as defined by Karl Popper. In other words, that means the measure of science is a method of posing bold hypotheses and making ingenious and severe attempts to overthrow them. Does it sound familiar? That's something we do almost every day: trying to set a hypothesis and trying to validate it.

Where UX research methods come from

[00:05:27] Digging deeper into whether UX research is a science took us to the finding that UX research methods are just taken from other disciplines. For example, simple methods such as a field study, or the Irish study [?], when we try to describe dependencies between objects, between organisms, even animals, are purely taken from biophysics and biology. You can even imagine a field study report read by David Attenborough. The user interview and stakeholder interview are taken from psychology and the social sciences. Product discovery, very popular and already mentioned in terms of workshops, in different words we can call empirical investigation using scientific methods.

[00:06:38] Brainstorming, very popular, and probably most projects start with brainstorming, is taken straight from social psychology. Lateral thinking, which we try to apply when we think about creativity in general, is taken from psychology. And surprise economics [?]: the economist is a pretty famous guy, who was nominated for a Nobel Prize in 1967. Also, on an everyday basis we use heuristics, which are taken from the cognitive sciences. When we talk about heuristics, everyone in UX probably knows the Nielsen heuristics, which come from psychology. And according to heuristics, we have the logic to validate a hypothesis, and that logic is taken from mathematics [?].

[00:07:43] Also, if we look at the functions around a hypothesis, we witness cognitive bias, and we still do. Daniel Kahneman was awarded the Nobel Prize in economics in 2002 for prospect theory. Even Google Analytics, which we know quite well: we see how the interface of the dashboard has evolved over the years, mostly because the data has to be easier for users to consume. That's why it's not only statistics, but we also apply sociology and demography [?]. So do you think it's a science?

Does it meet the scientific method?

[00:08:36] To call any discipline a science, it has to follow the scientific method. What makes any theory scientific is that it is resistant to falsification: it has to be publicly announced and possible to verify in an experiment. That's pretty much what we're doing on an everyday basis. It's also worth mentioning that a theory, which is also a scientific term, has to be logically consistent, conceptually simple and creative. It's worth mentioning that creativity makes theory in science distinct from religion, where proof is not possible, whereas the sense of research, the sense of validating a hypothesis, is proof. We can prove products by validating hypotheses immediately. So, social science, right?

[00:09:54] Theoretically, we can't call it a science right now, but there is big potential for it to be a science, because of the theory, because of the methodology, and because of methodology similar to the other scientific disciplines. But there's also a big role for scientists to create the right sources for it to work as a discipline, and to bring it into the universities, as with informatics, robotics and other new human sciences [?].

[00:10:36] But we have to remember that even the best research is nothing without analysis, synthesis and interpretation of the data: how to collect the data and integrate it into high-level strategy and design decisions. We also have to think about what you can measure; not everything is easy to measure. And we have to define the problem and ask the right question. Context is king. Only then does it make sense. So hopefully that helps answer the question, if you asked it. If not, I'll just add how we're applying all that theory at Netguru.

How Netguru applies it

[00:11:32] The answer is a bit more practical. First of all, it's as simple as possible, to make it easy to understand and easy to follow for our clients and stakeholders, bearing in mind that amazing design rule: keep it simple, stupid. We also have to answer why we're really doing UX research. First of all, obviously, to deliver the best product possible, to fulfill the client's and the client's users' needs. But also to validate the stakeholder's or client's vision, which is very essential. It can be very different, and you would be surprised what ideas people bring. And finally, to find the intersections within the research [?].

[00:12:24] As we work in agile, which is a very popular methodology of delivery, we can find four different phases: discover, explore, test and listen. We apply those phases in sprints, and it's worth mentioning that a sprint doesn't need to be defined as one week or two weeks. If you have to prepare a big field study, or you need to do user interviews or the PDS or whatever else, sometimes it's not possible to fit that in a one-week sprint. We always discuss and define the scope beforehand and how it should be implemented.

[00:13:16] Within the discover phase we're using the methods listed here. The explore phase, or sprint, is basically to inform and optimize the design using different methods. I'd say prototyping, feedback and testing are the most essential, because that's where we can learn as much as possible about the project in its initial version, at the initial stage. We test that with different methods. And finally we listen to the users, trying to validate the personas, and also do the analytics review, search log analysis [?] and all that usability part.

[00:14:13] Hopefully there's less chance to fail with that, but right off, clients come to us with the idea of building a new Tinder for pizza, or a new something for anything else, and that's quite difficult. We try to get the most from the science as possible, and we try to invest that 20% to get 80% of the results, which makes sense, especially at the idea validation stage: starting with small but meaningful steps, which are PDSs.

[00:14:59] We do love PDS as a workshop, and our clients do as well, and there's a reason behind that. You've probably all seen this kind of iteration [?] during the five-day design sprint announced by Google. But it's not always five days: in the workshop we can validate, define and refine the ideas. The understand phase is the longest one, because we have to collect all the different data. Sometimes it's possible to do it in one day, sometimes not. But when we get all the stakeholders together to diverge and converge and build the prototype together, we can finish with only that little step, because we know what the customer really needs, and we try to identify all the needs of the customer and the customer's users.

[00:16:05] Something that has worked from my experience as one of the best methods of user testing, and is also relatively cheap if you have the right users to do the test, is think-aloud usability testing. You sit behind the person and ask them to think aloud, and you'll be surprised by their motivations, the choices they make, and all the different questions and ideas you never, ever thought about. That's very insightful and very valuable. In terms of testing, we also use our own methods, like heuristics, cognitive walkthroughs and reviews. We're also building our own UX checklist, especially for existing products that we have to improve or redesign. That's probably the primary tool, but obviously that's not a closed list, because every project is different, every client is different, every market is different.

Handling disagreement with research results

[00:17:24] Finally, when we're talking about UX research, we have to think about disagreements, because there are two groups of people who will definitely disagree with research results. The first is people who have already done research, and that research shows something completely different. We can probably witness cognitive bias there: they believe their old research study, not the second one, and that's quite often the case. The second, and popular, is people who already have their personal opinions and preferences. And definitely, if UX research were a science, that would be a strong argument for us as researchers to convince them.

[00:18:21] When we talk about disagreements, we should treat research results as a learning opportunity, and highlight that design is not religion. We also have to remember that design is the business, baby. That's pretty much my talk; I hope you enjoyed it. If you want to find more insights and publications, feel free to sign up to our design newsletter. Open to questions.