Q&A With Macdara Butler

at UXDX Community: Ireland & UK. Video: https://www.youtube.com/watch?v=1pW-L5noSfc

Readable transcript: edited from the recording's captions for readability (fillers and false starts removed, punctuation and section headings added). Wording is the speaker's own. Timestamps are positions in the video. Names marked [?] could not be verified against the audio.

Blending digital into the real-world experience

[00:00:00] Rory: That was in my summary. But welcome back.

[00:00:05] Macdara: Thanks, Rory, good to be here. Thanks for having me.

[00:00:08] Rory: Excellent. What I loved about your talk was that it's a real-world, tangible example of how you can't plan for everything. If I take that Wi-Fi example, there was one station that had bad Wi-Fi, and that was causing lots of problems. That's just not something you're going to whiteboard. Maybe eventually you'd come up with it, but it's those kinds of things in the real world that go wrong.

[00:00:34] Macdara: It was a good anecdote, I think, for the kind of random stuff that can happen. Like I said, I spent most of my career doing more of what we call e-commerce, where you're booking something online. For example, I worked in the travel business for a long time, booking flights and hotels. You're not actually going to be there for usually at least two or three weeks, if not months, especially nowadays, so there's lots of time for mistakes to be corrected. But as I said in the talk, with a product like food or fuel, you can't really give it back as such. There's no returns policy for some of the stuff we do. You're looking at a digital experience that really blends into an overall customer experience. It's nice in that you get to work really closely with customer care, and you've got a real-world sense of buildings and funny things that can happen. Traffic jams can build up, and even our field testers can get on the wrong side of either the customers or the actual staff in the stations.

Building the field testing unit

[00:01:53] Rory: Let's jump into field testing. You said you set up this field testing unit. Did you staff that with people and just tell them, you're in the office most days of the week but one day of the week you're out? Or how did you build that team?

[00:02:07] Macdara: We're a big company. Vicero [?] has about 45,000 employees, but in the main, most of the staff are in the larger countries, places like the UK, Germany and the US, so we wouldn't have people everywhere. We had to look at this, and it was tricky. We eventually figured out that using generic IT support small providers has been the way we've solved it. Typically the IT guys, the people who go out and fix your printers and get you new laptops, those small IT support providers: we've worked with them across different regions. As you can imagine, it's a bit like user testing. You want somebody who speaks the language, who is actually a real consumer, who understands what's going on locally, and who can report issues. If you fly me over to Germany, I might miss [?] a lot of the issues that are happening that don't seem right or seem a bit off. So we've used local providers in different markets and set up an internal function to manage that centrally out of Ireland, which is nice.

What field testing catches

[00:03:34] Rory: Excellent. How successful has it been? How many items have they picked up? You finished by saying you're making this a permanent thing.

[00:03:43] Macdara: Talking to the success, it's great. It's critical, because, again, we all complain about test environments all our lives, but you can't really represent what can happen in the field with these kinds of products and channels in a test environment. It's just not possible. We have finds all the time, literally on a weekly basis we find things in our field testing, and then we refine the field testing program and the release strategy around that. Even involving myself: you can do a certain degree of field testing remotely from other locations as well, which we call desk testing in production. But we would allow one to two weeks of field testing before each app release.

[00:04:41] Like I said, we do find subtle issues. Maybe your receipt isn't displaying properly for a certain person, or a certain user who onboarded a year or two ago is experiencing an issue with their account when they upgrade the app. This is turning into a QA talk, but that typically wouldn't be found in testing, because mostly when people are testing they're using relatively new accounts. If you've got a legacy account with lots of baggage and history, you won't really find those kinds of issues in test environments, because they typically get wiped quite frequently. It really helps you avoid those big production incidents, and we find it quite effective.

[00:05:32] It's a key part of our product strategy, because we weren't doing this a couple of years back, and it was really biting us. You were going into the unknown and, as my slide says, getting punched in the face by things you never thought would happen, and then that finds you on the back foot. At least this way you find out these issues, you maybe create a new build and resolve them, and investigate them. We've really had to up our game on debugging and logging, especially since the field has to have special debug builds that generate more logs and stuff like that.

Shifting left, and why field testing stays

[00:06:12] Rory: And are you looking at everything that you spot in the field and asking how it slipped through your other processes?

[00:06:22] Macdara: Yeah, exactly. Again, it's more of a techie focus in this talk, and I'm not an engineer like one of the earlier speakers. My background is in economics, and in UX. But in terms of the software life cycle, you're right. You're always thinking, classically, how do I get this back down to the left of the delivery cycle, as people say? How do we automate these tests? That's probably the next part of our journey: looking into what we can do. But I don't think it'll ever be a situation where you say, we're okay, we've got so much coverage back downstream that we don't need to do field testing. We'll have electric vehicles coming through and charging, and lots of different new products. You're always going to need to field test with something like this, because it's quite complex. The current product has about 50 vendors involved, across multiple countries, with thousands of people working on it, so there are lots of low-level problems that can happen.

[00:07:39] I think we'll have to look at, like you said, how do you identify them up front. You don't want to find those at the last minute; they're expensive to fix when you're already in production. But it's much more expensive, as I said, when you release these things to consumers and then you have customer care lighting up with complaints. So that's going to be the next horizon, I think. The next challenge for us is to try and automate some of this.

[00:08:07] Rory: Great. Thanks for your time. It was great seeing that example you gave of travel, where you've got time to fix it if something goes wrong, versus somebody standing there trying to pay for their petrol. You don't have any time; they're causing traffic to build up behind them.

[00:08:23] Macdara: Yeah, and you can end up driving away without paying if things go the wrong way, so it has risks.

[00:08:32] Rory: Excellent. Well, thanks very much for your time. I really enjoyed that, and I hope you had a good time as well. A reminder again to everybody to ask the speakers your questions.