Efficient Research And Monitoring

Agnieszka Balcer-Thinlay at UXDX Community: Warsaw. Video: https://youtu.be/rqjMPKsPMsg

Readable transcript: edited from the recording's captions for readability (fillers and false starts removed, punctuation and section headings added). Wording is the speaker's own. Timestamps are positions in the video. Names marked [?] could not be verified against the audio.

Research without a researcher

[00:00:00] Just to be very clear, the things I'm going to show you are not my grand discoveries. Probably everybody does them or tries to do them, especially researchers. And yet you still come across situations where they would be extremely useful for multiple projects and companies, but for some reason nobody does them. And the other thing you will discover very soon after seeing my first slide... Is that better? Another thing you will discover is that I'm not a designer. I'm very sorry about that, but that's how good I get with my presentations.

[00:00:51] So, research. It's great, right? Everybody agrees. It's very useful. Everybody wants to do it. Hooray, yay! And the real problems start when you actually get from this enthusiasm to conducting a research process. It's great if you're a well-established company: you can use the help of an experienced researcher, and then you can run quantitative tests perhaps and fight for statistical significance. It's marvelous. Just the reality is the majority of us don't have those tools at hand.

[00:01:24] So we start doing research. Now I can't hear myself in the hall. And then we hear, "We are OK, but we don't really have a researcher." Some companies try to push this research role to designers, who are already very much overwhelmed with their everyday challenges and providing good design for their company. Or they don't have the money, because research costs money. Or they don't have time.

[00:01:52] Out of curiosity, just to get the feel, can you tell me how much time is too much time for research? Not maybe the discovery phase, because this can go for some time, but in the life cycle of the development, how much time is too much time for your companies? When I switched from a big enterprise company that I worked for to a startup, I faced a challenge of getting from "Well, we have a month to learn" to "If you don't give it to me within two days, we will just go without research." It was a pretty rapid change of pace.

[00:02:30] I hope to share with you three examples of how we managed at Codility to introduce research projects that push the development cycle forward and hopefully bring useful information. The first thing I would like to say is that having no researcher is not really an obstacle to doing research in a company. If you want to do very sophisticated experimental research designs, maybe this would be a challenge, but basic usability studies can obviously be done by anyone. Do-it-yourself research is actually possible for anyone who wants to listen.

Questions to ask before you start

[00:03:05] The only thing you have to do to run a successful research process is to ask yourself a few questions before you start it. If you have them, you have a framework to work with. The first thing you need to tell yourself is what you don't know, and what kind of questions you need to answer before or after your research process is completed. When you know it, it's great to figure out whether the answers to the questions you're asking are already hidden somewhere in your company as a byproduct of whatever you do, and you just need to look for them as a good detective. Or whether you need to explore and find them and get an external perspective to collect them, for example, to involve users of a product or service.

[00:03:49] Then a very important thing that is sometimes missed, as I've noticed, is that the majority of developments or products or features we are trying to create have certain assumptions behind them. They are supposed to do something, as was mentioned. Unless we can name those assumptions clearly, it's very difficult to run good research. It's a great platform to figure out whether we are actually going in the right direction, and then, at the end, whether we kept the promise we made to ourselves.

[00:04:19] The only other thing I would like to mention: if you already feel you know the answers to your research, just don't do the research. Either you actually know the answers, and that's maybe a waste of time, or, more often, you just don't want to hear the answers, and once you do, it's difficult to take ownership of the information that you're going to get from whatever process you're running. This kind of childlike curiosity is a massive approach, and from that you can start. You're a researcher, at least for basic usability studies, so let's go for it.

One day: the UX Lab

[00:04:49] So, limited time and limited money. We are already empowered to do our own interviews, or whatever we need to do; the question is how to do it with those resources. Usability interviews are something you have all seen, but very often the challenge is how to get people in the room and how to incorporate it fully in the development process. So what we did at Codility... there is a very invisible icon saying "one day" here in the corner, which is apparently very poorly designed, as it's not showing on the slide. If we have a day, and that's all we get for research, we do UX Labs [?] in our company.

[00:05:40] What it looks like is we run approximately four usability interviews. We put in one room a user of our service, of our interface, usually a developer, because our product is targeted to developers, and a researcher, and then they run a regular usability interview. In the other room we have all the authors, all the developers and designers that were involved in creating this particular part of the interface or the project, trying to look at the screen and learn what the user is doing. The connection between the rooms is created by some software that enables you to share the screen.

[00:06:22] This is how it looks in real life. I've never seen grown-up men being so excited about dots and commas and indents and clicks and whatever. We have developers here, we have a CTO, we have designers, so everybody's in one room, and this is actually crucial for any success in research. You need all those people together to understand the feedback that is coming, and also to close the feedback loop quickly, because then it will make your life much, much better. Anyway, great experience.

[00:06:59] It goes on for approximately four or five hours, because we have approximately four interviews, and then we end up with, again, a very invisible post-it, which is here, saying "idea", "solution" and "issue". What you might see from all these presentations is that post-its are a crucial part of UX design. If you have no post-its, you don't have UX. And it's true for Codility as well.

[00:07:25] What we do to learn quickly from the interviews is we try to write down every issue we have noticed, every idea we have accidentally come up with, even if it's not related to the particular feature we are currently testing, and every solution that might already be there. We put each of them on a separate post-it, and we try to estimate very roughly and very quickly the impact this particular idea, or solving this particular issue, could have on the end user, and estimate the cost of creating it. Is it five days, is it one hour, is it five developers or a single person?

[00:08:04] Once we manage to do it, we try to put it on a framework of impact and cost estimation. If you see something that is very impactful but at the same time doesn't take much money to develop, you've already done a lot of the thinking through, right? You know what to do, and you can do it quickly. And if you have really great ideas where, unfortunately, you have to put all your money and resources and all your people into it, well, you probably can't pull it off at that time. You can always put it in the backlog and it will simply die there, probably.

[00:08:39] That means that when the day is completed, after four hours of interviews and approximately two hours of working on post-its, we already have feedback, we have ideas how to utilize this feedback, and we have a ready plan for how to progress with our work from 9am the next morning. And everybody is already on board: the designer, the developers, and even the CTO, who is currently trying to be involved in the process more.

What the UX Lab costs

[00:09:15] OK, it sounds great, it's just one day, but it costs money, right? Again, that's not necessarily true. These are your expenses. You need to get the respondents, and yes, you can use the service of a professional recruiting agency. It costs a little money, but not as much as you would think. Then, when the respondents come, you need to say thank you, so there is either a thank-you gift or a little bit of money involved. But in case you can't afford even that, you always have friends of your friends of your friends, if they fit the target group.

[00:09:48] The only requirement here is to remember who you are designing for, who you are creating for. If you are creating a product for mothers, just get your friends who have children. If you are creating a product for developers, just make sure that developers are the ones who come. And yes, please feed the respondents. It's a human being working for you for an hour, so they need a cookie or a tea or water, that's it.

[00:10:15] Then you need something to connect the two spaces, the respondents' space and the conference room. You can use Hangouts, Skype, whatever screen sharing, which is basically free of charge. And then you have the team that is already there in the company, the authors of the idea who will get the feedback. And that is where the main expense comes: feed the team. This is crucial; it will never work unless you do it properly.

[00:10:39] The reality check shows that if you put good food on the table, first of all, people feel happy: "Well, I haven't had a croissant in a while, how about enjoying it?" And when there is a lot of it on the table, they don't feel the need to leave the room every time they want a coffee or a snack, so they keep their attention focused. And they are also working very hard throughout the day, so they need to draw their energy from something. So actually the main cost is probably food, if you estimate it. You can close it all with a really manageable price [?].

An example: the Angular task animation

[00:11:14] How do we do it? We test the interface very often. This is an example of a real thing we tested, simplified slightly. What my company does is create an interface and tests for testing developers in the recruitment process. We write tasks on different tools, and then when developers solve the task, you can say, well, they are great developers, or average, or we really don't want them in the company. And we sell the services.

[00:11:43] Recently we created these Angular tasks. Angular, for those who don't know, is a framework for developing how things look on the internet. I was told it's hip. We were very proud of it. It was a completely different approach. We made the task very real-life: it was really about creating a piece of real web interface. There was a description, some conditions, and what we were most proud of is there was an animation at the bottom of it. If you just scrolled a little bit, there was an animation showing you how your end result should look.

[00:12:22] We thought to ourselves that it's just as realistic as it gets, and it makes your job so much easier as a candidate, as the person who takes the test. What we discovered, among other things, during the UX Lab was that, unfortunately, this was the greatest feature but nobody discovered it in our task. You had to scroll down, and people were so tense because of the time limit incorporated in this test environment that instead of scrolling down, reading and discovering the animation, they stopped reading around here, tried to develop something, read a little bit more, made sure it fit what they already did, and then, well, it would have been so much easier.

[00:13:09] The very simple thing of putting this at the beginning made all the difference in the world. It seems obvious, but somehow, for us, sitting on this particular task for a long time, it wasn't that easy to notice in the first place.

Data recycling

[00:13:24] OK, so one day, again, is all I get. Sometimes you can run these labs, but sometimes you already have the answers in your company. What you can do is data recycling. Recycling is a great thing: you can change a plastic bottle into a tent or a carpet or a polar sweatshirt, and you can do the same with data, which is very often just a simple byproduct of what your company does. If you have an offline product or a service, you have newsletters, you have mailing lists, you have loyalty programs, you very often have some information about users. If you have software, even better: all the clicks are logged. All the logs and all the actions are already hidden there. They're just not used on a daily basis.

[00:14:09] So when you come across a research question, you should ask yourself whether there is a chance you already know the answer. This is what we did with this particular development, for example. This is a part of our interface where a recruiter who wants to recruit a developer can pick a test. We call them test boxes; different tasks are hidden here. We want to make life easy for the recruiters so they can do multiple things. They can send the test to a candidate. They can send it to a bunch of candidates. They can test-drive the test and see if they like it. And they can do many, many other things.

[00:14:42] The issue was our designer had a great idea of updating our look and feel into a new product. We wanted to put a sidebar on the right. As a side effect, the test box had to get smaller, and as a side effect of that, we needed to limit the number of buttons we had there. It was really great to hear how the product owner, the designer and the developer sat in one room and had absolutely no way of agreeing which buttons were going. "This is the most important." "Not at all, this is what people do." "No, let's do the usability interviews," says the designer, "let's see how people use the interface."

[00:15:29] After approximately 15 minutes, I asked the question: how about actually seeing what people actually click on in this interface? We have to know it somewhere. It's our interface. We have it written somewhere. I have no idea where, I'm just a researcher, but I'm sure you do, right, guys? And so they did. They created a very simple tally of what was clicked most often and what wasn't. The features that were not used, the buttons that were not used, were simply hidden. They are a little bit less discoverable at the moment. But the design was completed, and it didn't take much time and it didn't take much money, which is crucial.

[00:16:06] For data recycling, your crucial person who can help you is the doer, whether it's an analyst or a developer. And your crucial cost is, I would say, the soft skills you have to invest to communicate what you need, to get them on board, to make them understand that what they could do for you is important, why it's important, and how it could work for the business. And then you have to spend the rest of your day possibly spoiling your doer or your analyst. Whether it's a hot stone spa or a foot rub or beverages, whatever makes their life easier. It really works. They can do a lot of stuff for you, so if you can do anything to help them in return, it just works great.

One hour: the pizza check

[00:16:51] These are the luxurious situations when I have a full day to do something. Sometimes I don't get a full day. Sometimes all I get is one hour. I want to show you what I get within one hour with my team at Codility. It's one of my personal favorites: the pizza disaster check [?]. The main cost here is the pizza. We had two great ideas and we wanted to choose between them. We invited senior developers who are friends of our friends, we put the pizza on the table, asked them to discuss the ideas, and they helped us choose. That's it.

[00:17:29] What were the ideas about? I'm not sure if you realize, and some of you, I'm sure, do, but when developers code, they do it in their own environment. It simply makes it easier for them. Forget even the dark background: you get syntax highlighting, you get syntax completion. You start writing and it completes for you, just like my mobile does for me when I write messages. And if you forget a dot, it will very often complete the dot, et cetera. Our environment, which is unfortunately quite in the background, so not very visible here, does not provide these features.

[00:17:54] What we thought is it would be so good, so real-life, if we could allow developers to take our tests by writing them in their own environments. We thought it would be so user-friendly, and we came up with two ideas for how to do it. We got the people, we put out the pizza and said, OK, which of these ideas should we go with? They asked questions and tried to understand better. What we showed them were plain drawings, so there was no investment in brand designs and mock-ups. This would all come later, we decided. At this time, it was just describing the idea and showing one or two drawings.

[00:18:47] At the end of this group, we were shocked, because we actually received feedback which was idea number three, and it was not on the table at all at the beginning of the conversation. They said, "Well, it's not great. The issue is we don't really need it, because if we did it in our own environment, we would lose lots of the tools that your interface already gives us, and that makes taking the test easier. We want to use those features, and for that we have to stay in your interface. Why don't you just upgrade it and put all these auto-completions and syntax highlighting inside your own interface, instead of redirecting us to our own environment?"

[00:19:26] It was ingenious, and for some reason not considered before, because the time to develop it was far shorter and the cost of making it was much lower. On top of that, it allowed the candidates to continue using our own interface, which we had already invested so much time in; we have been creating it for years. Now it would make them stay and make their experience a bit more comfortable. So after one hour, we already knew which direction to go, and we put in the box two ideas that were not really optimal and would have cost our organization a lot.

Why bringing everyone together matters

[00:20:03] In this case, we also had in the room a designer, developers from our own company, and the product owner of this particular idea, to close the feedback loop really quickly. Whatever research you do, if you put all those people together, what you get is definitely much better decision making. It's much closer to the optimal result. It limits the risk behind the decisions you take, also business-wise. Therefore it actually saves money instead of being an unnecessary investment.

[00:20:36] Making those people work together, at least for a day, creates ownership of the feedback that is received and closes the loop on the feedback quickly. It also has a great effect on engagement. If people work on a particular product or service for two months, it's very difficult to still be surprised at all by what you do. But with these kinds of reality checks, engagement spikes, and the flow is again much faster and much better, and so far I've seen only good results. I think that's it for me. If you have any questions...