Travelling with your users using Diary Studies

Liam O'Connor at UXDX Community: Travel UX Research and Pair Programming. Video: https://youtu.be/usE-zXaiD4w

Readable transcript: edited from the recording's captions for readability (fillers and false starts removed, punctuation and section headings added). Wording is the speaker's own. Timestamps are positions in the video. Names marked [?] could not be verified against the audio.

Introduction and the plan for the talk

[00:00:01] Liam: Okay, so we're going to take a look at traveling with your users using diary studies. This is a case study at Deem, and we'll find out a little bit about Deem in a moment, where we delved deeply to understand the in-context findings from our users using the diary study format. So let's take a look. I have to... yeah, okay. Just struggling with the tools there.

[00:00:34] This is what we're going to take a look at. We're going to examine the research question and how it came about. We're going to take a moment to step into the shoes of our users, who are corporate travelers returning to travel post-pandemic, so we'll try to get ourselves in that headspace. We'll take a look at some of the artifacts that came out of the study, some of the outcomes and some future research. Then we'll take a step back to the users and understand some of the challenges that they encountered in their destinations, after they had completed their trip and were at their travel destination. Then we'll summarize and wrap up, and there'll probably be some time for questions at the end, I hope.

[00:01:23] Okay, so let's move forward. This is me, an updated photo. Rory had an old one there from when I looked much younger. Some contacts, in case you want to connect or ask any further questions afterwards. Just a little bit about me: I'm very much focused on qualitative research, but quant is super important as an approach to triangulate data, so that you understand your users in as much detail as possible. So we cover that.

[00:01:56] We have two core cohorts at a high level at Deem. We've got the customers, the buyers of the platform, the B2B platform, and the end users, and they have very different needs and goals and issues that we need to attend to. Buyers, for example, are more interested in tracking, reporting and rules within the system, and end users want to successfully complete their task, which in this case is purchasing tickets for traveling. We'll cover that in a moment.

[00:02:29] I myself am generally extremely interested in the strategic part of research. Anil talked about the user side, really the developers in his case, where the users are super important: this tactical stuff. We cover a lot of that, but I also like to help product teams and leadership teams understand where the product is going, how to anticipate that, and where the business decisions should move in the future. And of course, building a research team, a team: this is super important stuff that I really, really enjoy doing.

Deem and the research question

[00:03:07] Oh yeah, I have to... okay. So, Deem. Nope, I skipped ahead. I'm trying to go back. It's important to understand the context of the diary study. Our travelers are corporate travelers. It's not loading that slide. One more... here we go. So Deem makes this tool. It's a corporate booking tool, where a customer will give it to their employees to book flights, hotels and cars when they're traveling. This ensures that the end users, those travelers, their employees, are booking within the rules, not buying extremely expensive flights, and it also helps with tracking, reporting, call centers, et cetera.

[00:03:53] This is a screen mockup of a really nice app, and I must say our design is cutting-edge in the corporate travel space. We really push the design element quite hard, and we have a really, really excellent design team, which covers quite a few different countries, predominantly the US, India, Ireland and Poland.

[00:04:23] So let's continue and find out more about this research project. We saw a preview of this slide earlier. This is the research question: how are COVID restrictions affecting the experiences of our users while they're traveling? This was last year. There was a lot of return to travel last year, return to corporate travel after quite a break. We already knew a lot about our users, these travelers, from previous research, from interviews, from surveys, from customer feedback. We have many channels to gather this information, and we already knew how they use our tool and how the experience of traveling was for them: what the key pain points were and what the delighters were for them.

[00:05:19] Now, as a booking tool where you get your tickets, we don't have a huge influence on the actual experience of traveling. It's more the airlines and the airports and the Ubers and the hotels. But these are our end users, and we're very interested in how the experience goes for them. If we could do anything to improve their experience while they were traveling, then this is an opportunity to delight that user. Maybe there were opportunities. So we really wanted to delve deeper into this in-travel experience context. And this was a unique time. People had been stopped from traveling for two years in the middle of a pandemic, and now they're coming back to travel while the pandemic is over, let's say, or some people thought.

Running a diary study with the tools you have

[00:06:09] Okay, so let's take a deeper look at this, at the methods, when it loads. The method was a diary study, meaning that we followed the travelers in the process of their traveling. We were traveling with the travelers; that's the title of the case study. I'm not sure if you can see the slide. It still says... oh, okay, Rory tells me it's loading, great.

[00:06:52] We really aimed to gather rich data here. This is data from self-recorded videos, using your phone, in context. You can imagine a traveler going through an airport, passing security, and then they have a task, which is to record a one-to-three-minute video of going through security. We also used surveys, and there were debriefing interviews as well in this project. So a lot of really rich and really fantastic data.

[00:07:20] What's really cool about the diary study format is that you can get a lot of unexpected data. When Anil was talking, they had really, really strict research goals. They were focused on the developer portal. But in this case, you could ask your traveler to record a three-minute video on something related to their travel day [?], and they could record anything. It could be about the Uber, it could be about buying dinner at their connecting airport, and this is what's important to them. So this is participant-driven research goals. You can get some stuff that's not directly related to your own research goals, but then you can get some absolute gold. We got a lot of it: some of it gold, and some not-so-interesting stuff.

[00:08:07] What else? This was a proof of concept of this diary study format at Deem. We hadn't run one up to that point, and we decided to go in without onboarding any new tools. We just wanted to use what we had, so we used your spreadsheets and your surveys, and Zoom for the debriefing. For me, it showed me that you don't need a big team. When I thought of diary studies before, it was like, oh, enterprise-level research teams of 15 people with access to all the tools they want. But that's not the case. We had a lead researcher and support when needed. There was not a huge team, and we didn't onboard any new dedicated tools for this. We just used what we had.

[00:08:57] Organization and preparation were key, and these are key research skills anyway. If you're interested in research and you've done a bit of research, like Anil, organization and preparation before you actually start your project is super important.

[00:09:12] Some of the goals that we had were data triangulation. I mentioned that we had a lot of information already, so we wanted to add to this information. We wanted to prove or disprove some of the hypotheses that we had around the experience of traveling for corporate users, especially around the time of return after the COVID pandemic. And traveler sentiment was a really, really, really important goal. Did I say really three times? Travel is a really emotional experience. Everyone listening has probably had the experience of a delayed flight or a canceled flight. That's the bad experience. But there's also the really happy experience of going on vacation, your first trip to New York City, when you're really excited and looking forward to it. So the emotional side for corporate travelers: how does that affect how they work, how they deal with the solutions, how they use our product, and what kind of opportunities are there for us to help support them? So the emotional side and the traveler sentiment were really important.

Stepping into the travelers' shoes

[00:10:27] Okay, so let's try to step into the shoes of our users around this time. You're planning a work trip. It's spring 2022, so it's March or April. Lockdowns are being lifted in some places but not others. This might seem like the far distant past, but only recently countries like China and Japan have been lifting complete restrictions. In Germany there was a mask mandate on public transport until very recently, a few weeks ago. So some of these things aren't completely in the distant past, and some of them are still with us.

[00:11:14] So why were you traveling when COVID was around and lockdowns were happening? Well, maybe you had a team workshop that you wanted to attend. If there was one benefit that came out of the pandemic, it was the flexibility around working styles for us office workers: hybrid or fully remote. But you still want to connect with those people. People connection is one of the key elements in corporate travel. You go to meet people. So this was a really big driving force. Sales teams: any salespeople that I speak to use Zoom as a tool, but they say they hate it. They can't wait to get back and meet their client, sit down, have dinner with them and agree a contract.

[00:12:03] So these are some of the reasons why you're doing it, but how do you feel about the trip? Well, with the COVID lockdowns, most definitely anxious. But returning to travel after two years, definitely excited. And probably an overlap of the two. You definitely want to get out of your house. You've been working there for two years, at home with your kids, who were also locked down from school. Time to travel. So this was a bit of a crazy time. And when you're returning to this state, there's a lot of preparation to do. I think we can take a look at some of the preparation that the users went through.

[00:12:51] I'm going to assume it's loaded; it's not loaded on my screen. There was a lot of preparation and planning. Oh, that's because there's a different slide. I'm not sure what slide you're looking at, because it didn't load on my screen. I think it's the one about the traveler sentiment. Okay, thanks, Rory.

[00:13:16] These were some of the reactions to this planning. We have a participant here; this is just a short quote from them. They're doing their preparation and their planning, and they're wondering about the restrictions in the destination, which is Nashville, wondering if there's a big COVID outbreak. She really encapsulates the feeling of a lot of participants here at the end: "I'm really excited about this travel, but nervous as well." She's just not sure anymore. This is how a lot of those workers who hadn't traveled for a long time were feeling this time last year.

[00:13:56] Okay, the next slide is probably the preparation slide. Yeah, here we go. You've planned and you've prepared, but there is a lot of uncertainty. If you cast your mind back to this time again, some of the rules were lifting and some of them weren't. Each airline had its own rules, each airport had its own rules. A lot of our travelers are in the United States, and each state had its own rules. Some states were more strict about the rules and others were less strict. If you were traveling internationally, you had to have your vaccine certification to get into the country. You had to understand where to get a rapid COVID test in order to leave the country, to get on the plane. And then you had Uber and public transport restrictions, different again in each city. There was a lot of preparation to do, and this was before you'd even stepped out of your front door to go on your trip.

[00:15:00] That's the preparation up front, and we know a lot about that already. A lot of that preparation with the COVID restrictions was new, but the preparation is part of the experience for our users before they buy their tickets. Then when you step outside your front door, after you've bought your tickets, that's typically the end of the journey for us. But there's so much uncertainty left. You're going to the airport, and you're at the mercy of these restrictions and the organization at the airport and the airlines. And if you had a traveler that was normally anxious about COVID, these fears become heightened in such situations, and they can become overwhelming. Are you nervous about the number of people who are going to be wearing masks at the airport or on the plane? These are a lot of the things that we were getting back in some of the diary entries. Before they started traveling, there was a lot of nervousness.

[00:16:07] Let's see how some of them handled these anxious times. Okay, so there should be another traveler sentiment slide for you, and this is about the security lines. This was a traveler going through Atlanta, a really, really busy airport in the United States. TSA PreCheck: TSA means Transport Security Authority in the United States. They handle all of the security check-in through the airports. This traveler, as is very common with corporate travelers, had pre-clearance, where you register online and go through a special queue.

[00:16:56] I have an audio clip there, but it's not working at the moment. She basically gives a little bit of extra context. This was her return flight. She had left her meeting a little bit earlier, wrapped things up, felt a little bit of stress, but the stress level was totally normal. It was on par with what she expected. Then she got to the airport, and the stress levels went straight up, almost vertically, because this is the horrendous thing that she didn't expect. She was expecting a special advantage with her pre-clearance, but she didn't get it.

From diary entries to an emotional journey map

[00:17:30] Okay, so we've taken a look at some quotes, at some of the preparation, and at the traveling issues that some of the participants encountered. Let's take a look at what we did with all of that data, all of that rich qualitative data. Here we have an example of an emotional journey map. It's obviously a journey map, and we added some emotional elements to it. This is a proto version, an early one that we used. We're using this to help the stakeholders, the product managers, the designers and the engineers understand the journey, and understand where the sticking points are: where the rise in emotions is, as they achieve some cool milestone, getting out of the airport, for example, or where the fall is, as in this case, when they get an Uber ride, which they were very, very upset with.

[00:18:31] Let's just take a moment to take a look at this quote. Again, I had a video, but I changed it at the last moment. She had a really heavy bag, and the driver didn't help her put it in. She didn't like the smell of the car, and she didn't feel safe in the car at all. Now, this is Uber's responsibility, but these are our travelers, as I mentioned before. How can we help them in the future? What are the opportunities for Deem to make sure that the journey end to end, from booking to return home, is as good as possible, and to remove some of these sticking points? That's what we're using this journey map for.

[00:19:11] We also use a lot of those clips that I showed you... no, I didn't show you, because the audio doesn't work. But we use the video clips and audio clips to really put the user in the room with the stakeholder. Anil talked about bringing his stakeholders into the interviews. It's such a great technique. If you ever get the chance to bring your stakeholders into the room, it works a treat. They watch people struggle with your system, or having a delightful time with your system; hopefully that would be the case. But these are diary studies, so you can't really bring them into the interview. So you bring the clips into the research readout, and this helps the team to pinpoint where they're going to exploit opportunities for the product.

Diary studies beyond travel

[00:20:07] Okay, so future research. You might be thinking, yeah, this is all well and good for travel. It's an intense multi-day experience; that's why you're using diary studies. The uncertainty of travel certainly hasn't gone away, and we'll be using diary studies again. But uncertainty is kind of constant in the modern world. Every generation believes that theirs is the most uncertain time, that their time is the end of days. We see this in things like the news reports of impending economic slowdowns, tech companies laying off lots of workers, and there's even a war in Europe. So there's a lot of uncertainty in the world.

[00:20:57] I'm going to get to the point here. The diary study format allows you to delve into those people, those participants, your user base, the people who use your product, and delve into the emotional aspect of the uncertainty. If that's where you think you can help them, then you can really drill in in a lot of detail. If it's a banking app that you're working on, money is super emotive; they definitely have some strong opinions about that. E-commerce, even: when you're buying your next product, a new pair of trainers or a big TV, you've got a lot of anticipation there for the joy that you're going to feel when you get your new product. The diary study element can really help with that, after they've taken the final action and they're waiting for the product to arrive, for example.

At the destination, findings and an ode to diary studies

[00:21:46] Okay, let's have a look at the time. A few minutes left. Let's take a step back into the user's shoes, try them on and see if they fit. We've traveled through the airport, we've taken our masks, we've got our COVID documentation. Let's see how it is at the destination, when you actually get where you're going. In this slide, which I hope you're seeing, there are these lovely spring flowers, because it was spring, and the relief was palpable in the diary entries that the participants gave us. They were really, really super pleased to be out of the airport, to be at their destination. It was just like old times. Airport transfers were easy; they were at the hotels.

[00:22:39] And then the juxtaposition of this flower with the challenges. They started to encounter max capacity of 50% at office locations. Maybe they were expecting it, maybe they weren't. Office buildings that required vaccination certificates and COVID testing every other day. Lots and lots of challenges for the participants. But again, the joy. I mentioned earlier the people connection in corporate travel: the joy of meeting your colleagues, the joy of having a joke and a laugh and gossip at the coffee machine. These all came through really, really clearly in the diary study project.

[00:23:24] Okay, so I think we've got one more sentiment slide. Oh, that one's kind of dark. "The world was temporarily closed." This was a participant who had to take those tests every other day. So let's skip on through this one.

[00:23:46] We had a couple of findings. These are broad strokes, the high-level categories: hybrid and remote work leading to new travel needs, and a whole bunch of other things inside of that bucket. We looked at the excitement and anxiety of returning to travel. And we didn't talk about the staffing issues, but this was another one of those challenges that came in during the return to travel after the pandemic.

[00:24:19] Let's just skip right to the last slide, which is actually just the next one, and I'm going to give you a little ode to diary studies. They are so awesome. They're easy to do with the tools that you have; you don't need any new tools. You get a lot of rich, in-context qualitative data. We used it as a standalone piece, but mixed it with other information, other data we had already. And we really, really loved those unexpected findings that we got. I highly encourage anyone who has a little bit of interest in research... Anil, maybe you're going to try a diary study next. Let me know if you need any tips with the organization and the preparation. I'm going to write a blog post on that particular organization. You can check it out on the Deem blog. Right this way. Okay, that's me. Any time for questions?

Q&A

[00:25:12] Rory: Yeah, thank you very much. That was really interesting. I love the richness of the feelings, the emotions that you can convey there. It's more than just, "Oh, this is what people have said." By sharing that Tuesday conference [?], that sounds really terrible.

[00:25:29] Liam: It certainly was.

[00:25:33] Rory: If anybody has any questions, please do write them in. We might not get to them, but we can always move them on to the Slack channel. I do have one question, because you mentioned it at the start, but I didn't really gather how you got through it. You said, "Oh, I think this is for big corporates." How did you manage your time combing through all the data? Even though, as you were saying, it wasn't that difficult to set up, what about comparing the data and combing through it to pull out the insights?

[00:26:03] Liam: Yeah, it's a great question. We had a couple of ways of gathering the data. We used surveys to have structured data. This was easily handled because the questions were all prepared, and the participant answered that direct question, so you could prepare how you were going to handle that data. As for the qualitative data, in this type of research we have a couple of tools. We use Dovetail. We import the video in there, it pulls out a transcript automatically, and then you basically just read through it and tag it and code it, in a very university, academic-style way of tagging qualitative data. This helps you to see the themes that are emerging. You have to read it a number of times, you have to fix your tags, you have to go back over it. You might pull in the other person who's working with you at that time and ask for their support: "What do you think about this tagging group? It doesn't make sense." It's a lot of effort, it's true, but the reward is most definitely there with the richness of the data.

[00:27:13] Rory: Excellent. And what's your timing? Is this a continuous thing where you're always drip-feeding data in, or do you do very much targeted studies?

[00:27:23] Liam: This was quite a long process, because what we had to do was recruit people who were planning a trip. Let's say we recruited them at the start of March and they fulfilled our recruitment criteria. They might not be traveling until the end of March, or they might be traveling mid-April. So we had a few people traveling; there were 17 or 18 people in total, and we had a few people traveling each week. So again, the key thing is to do all of the organization and prep up front: all of the communication, create all of the surveys. Then as you go through that time period, which in this case was probably something like seven weeks, you just continue to drip-feed in the information, and by the time you get to the end, you've got a good understanding of what the themes are. In this case, there was a day, let's say, set aside each week to go over the results. It wasn't a huge mammoth task where you had to sit down and read for weeks and weeks.

[00:28:29] Rory: Excellent. That was really interesting. If anybody has any questions, do send them in, but I think that's all we have time for this evening. So thank you very much, Liam.

[00:28:40] Liam: Thanks very much, Rory. Thanks, everyone.