UAT Validation In Production

Sarah Davies at UXDX Europe. Video: https://youtu.be/OUL10CKa3uY

Readable transcript: edited from the recording's captions for readability (fillers and false starts removed, punctuation and section headings added). Wording is the speaker's own. Timestamps are positions in the video. Names marked [?] could not be verified against the audio.

Introduction: UAT at Vodafone

[00:00:00] User acceptance testing. I know there are a lot of developers around, and you'd be glad to hear that I have a very good relationship with our developers in Vodafone. It's the best way, quite frankly, when I have to go back to them with defects and politely say to them that it's not quite working as we're expecting, even though it's got through various other stages.

[00:00:24] As was explained, I've been in Vodafone nearly eighteen months or so now. I've been in testing probably ten, eleven years all told. And Vodafone — I've been at lots of other companies, and you say the name and people kind of look at you, but you say Vodafone and everyone's heard of Vodafone. Or hopefully they have, and hopefully you've had good experiences as well.

[00:00:51] Now, Vodafone is a very complicated company. Believe me, I have not got my head around it at all. There's the Vodafone Group, which is worldwide[?], and Vodafone UK. In Vodafone Group I'm in Vodafone Business, and we don't deal with the individual customers. It's the big companies, the big corporates, they're our customers. I actually look after the UAT team, and we have internal customers who then obviously service the big clients.

[00:01:24] Even in my own team, and various other people say, I don't really understand UAT, I don't really get it. So I always start by saying: testing is challenging. A bit like today, when I decided, stupidly I thought, to have the chicken noodles for lunch and thought I'll end up wearing it. But anyway, that's where I usually start. Testing is a challenge: the data, getting the people to actually do the testing with you.

Growing up in a village shop

[00:02:07] I thought I'd start by way of introduction to give you a bit of an idea of my background. This is what you won't find on LinkedIn, the first bit anyway. I started out as the top-right corner. I was actually brought up — I came from England, the southern part of England, a very small village — and my parents actually ran and owned the village shop. So you can imagine it was the hub, the central part of village life, and everyone came in and shared their problems and asked my parents for advice.

[00:02:50] It always seems to me that my parents never minded whether someone just came in for a loaf of bread a month, or whether they actually got all their shopping from them. It was all about the customer experience, or, as we would look at it, the user experience, and the customer was really at the heart for my parents. It was a small village and they'd even offer this free delivery service, so they'd load up the family car with pre-ordered groceries. This is not like Amazon, where you're online and so on. This was people phoning up, placing their order, and my parents at the end of a busy day, about six o'clock in the evening, they'd shut the shop, load up the car and off they'd go.

[00:03:39] Even in the winter, occasionally we had harsh winters, they'd load up the sledge and then they would traipse along in the snow, still delivering, because they thought, regardless of the weather, people relied on them. That was the key to them, the customer. That was what really made it sort of embracing me. It was the core of how we were brought up. You always look for others, how you can help them, and actually give people that good experience.

[00:04:07] That kind of lived with me and my career throughout, in customer-facing roles, whether it was in customer service and a pensions department for many years, right through to now, as we stand here today, as UAT lead in Vodafone. I must say the last ten years or so I've actually been in testing, from being a test analyst right through to test manager, and now solely UAT lead, so I only worry about user acceptance testing. That's probably just as well: I am not technical at all. I've never written a line of code. I can just about run a SQL query if someone's written it for me, but that's about the size of it. Dealing with users, that's my bag. I can do that.

[00:04:58] Recently, a few years ago, a new neighbor I had met asked me what I did, and I said, oh, I'm just the tester. Now, you might be thinking, Sarah, you've just been very modest, why just the tester? Bearing in mind, he'd just told me he was a scrum master. How the heck am I going to beat that? That was my idea, I thought it was very topical, where I'd be woke up, we won't mention Run Kweilyn[?]. It made me think: if that's the scrum master, what do people think of testers? Well, I don't do that. You'd be very glad to know, no test tubes or anything at all.

Business change, and what UAT is for

[00:05:45] From our point of view, I work in the business, in Vodafone Business, and I actually work in a business change team. So I'm not part of the IT side of it, not in the IT team at all. I'm in the business change team, and what we do is we deliver, we transform our business. What we hopefully do is, when they're transitioning and we're trying to simplify the business for them, we kind of take them on that journey. I hate using that term, but it is what we do.

[00:06:19] In this business change team I'm part of, we have business change specialists and analysts. What they do is they start off with business impact assessments and they go right through to adoption. They literally do take it from sort of cradle to grave, as it were, and I'm part of that team. What we tend to try to do in business change is to increase, as it says there, the end user engagement, and increase the likelihood of adoption. Because at the end of the day, there is no point in trying to deliver anything if no one's using it.

[00:06:58] Particularly you developers and designers, you know what you want to be able to do with all that good work you spend. If you think, if it's waterfall, you've spent a considerable amount of time and effort and money on it. As it is, we do more agile, and that's UAT in agile, which is definitely for another occasion. But you want to make sure that people want to use that, that they're not going to immediately slip back into what they've normally done and what they're used to.

[00:07:31] How we do it is by building relationships with the key stakeholders. I think earlier in the day Tobias mentioned the personas, and that's what we do, and there's a lot of collaboration. It is literally right from start to finish, just keeping that communication going. At the end of it, what we want to achieve is to realize the benefits and improve the stakeholder appetite for future transformation, because you don't want to turn off your operations teams from actually doing anything with you again.

[00:08:05] So, UAT. What I see — and as I say, I do take it very seriously, I'm passionate about UAT — is that we're the gatekeeper. We get right through the different test phases and we get to UAT, and we literally do stop, try to stop anything going through into production without finding hopefully all the bugs and the defects. Before we let it go free, we try to capture those.

UX and CX inside UAT

[00:08:34] So this is UXDX, and UX in UAT. In Vodafone the user experience and customer experience is baked into everything we do. It really is. There are even customer experience design thinking courses, and we really do put a lot of thought into the customer experience, and testing's no exception.

[00:09:06] From the UAT side of things, I don't use, I think occasionally but I tend to try not to use, any professional testers. It's all people from the business, because ultimately they will have that insight and they will see things and look at it through a lens that you just don't get from the tester, because they'd just follow the scripts, they just don't know that side of it. What they'll do is formal and exploratory testing. What I also encourage, which does slightly irritate the management, is for them to look more at the UX, thinking outside the test case. That's why I encourage the exploratory testing, saying, look, is there any way we can improve this before it goes in? Let's capture that now.

[00:10:00] What I will never do — this is one of my quotes — is I will never compromise the quality of testing. As you'll no doubt be well aware, testing timelines, quite often, no matter what happens and what we do, it often seems to be the way that testing is always squeezed right at the very, very end and we end up with a teeny tiny timeline, and we're supposed to suddenly do all the testing that we need, that we've planned out, in the tiniest thing. And you think, well, this is unrealistic. We don't want to compromise the quality of what we've built by not testing it properly, even though we might have done various other kinds of testing, integration, functional and so on. We want to make sure that we still have that proper user experience.

[00:10:46] So what do we do? Well, if I get really, really pushed, then I'll just say, okay, what, let's not bother, just put it into production. If nothing else, believe me, that is one way to get your program manager's attention, because they often sort of look at it, really? And of course I don't really mean that, but as I say, it does get their attention and make them realize that actually maybe we do need to be a bit more considerate to the testing side of it.

What could possibly go wrong

[00:11:23] But what could possibly go wrong? Now, you're probably thinking, what on earth has the bus got to do with any of this? This bus is a little story a friend of mine told me recently. It was in a town, and what had happened, they actually built this new bridge, and it specifically was supposed to be for buses and bus lanes in it. But for some reason, the actual buses — they kind of put these guideline tram line things down the sides, and the bus literally just fitted, the wheels were virtually touching if they literally just drove in a straight line. They kind of tried to deny this, but as ever, you can guarantee people always have a mobile phone, people are always taking pictures, and there were these pictures splashed around the paper about this test that they were doing. So they obviously had to go back to the drawing board and rethink that idea. Hence the bus.

[00:12:28] But what else could go wrong if you just actually put something into production without any testing? Well, of course you could find the code doesn't work as expected. That's quite a big thing. You might actually find that you have a data security breach somehow. So again, quite a big one. The reputation side of it: it's not just the actual overall company reputation you might damage, it might well be that you damage the team, your own. There's lots of different reputations. And there could be a costly cleanup exercise at the end of the day, if you've got to unpick things and so on and so forth.

[00:13:15] So looking at the risks. Obviously, once you did decide that you wanted to go down that route of just popping it into production, then once the code is put into production, surely it's released out to all customers, all consumers, unless you somehow manage to ring-fence them. So isn't there a risk that someone could actually inadvertently use it before the testers get to it? That was more my thought, of how you're going to stop doing that, and how is this different to deploying it without any testing at all?

Example one: using production data in financial services

[00:13:54] What I want to look at now is to give you a couple of examples. One is actually using production data, and that was before I joined Vodafone, and then a more recent example with production. In the first example we were looking at using production data. To give you the context, it was within financial services. I was the UAT lead at that point, or test manager, but I was asked to actually go and help as a test lead on a client site. So we weren't even delivering to ourselves, as our company. It was actually to another client, and it was in their BAU, their business as usual, program.

[00:14:40] Typically when you're doing business as usual you might have two or three programs on the go at any one point, and so you think, wow, it's quite a small change. It might be just due to the finance, some change where the company wanted to pay a different bonus, or it was to actually test correspondence because they'd changed branding or something. So you kind of think, well, it's only a small change, how difficult could it be?

[00:15:08] Well, this particular company was risk-averse, if nothing else[?]. As I said, it's financial services, highly regulated, so externally by the regulators, but also the company itself had their own compliance department, security department, really tight regulations. It was a system that, being not a very technical person, I referred to as a clunky old system, the ones from years back. It was a pensions system, it was a closed book of business, and, to add more complexity, it was impossible, so I was led to believe, to manually create data. You couldn't create the test data.

[00:15:56] So I said, well, how are you expecting us to test this then? What do we do? And it was, well, you're going to have to use production data. Okay, fine. So how do we do that then? Oh, there's a couple of forms you need to fill out first. Okay. Just explaining what you want to do, why you want to do it. So we did those forms, and then it was, okay, now what do we do with these? Right, you need agreement from the project sponsor. Fair enough, we did that. So, could we get ready to start testing then? No. You've got to pull together this security plan.

[00:16:46] Right. So the security plan gave details of what testing you wanted to do, who was going to be able to see the data, what you wanted to use it exactly for, how much data you wanted to use. How was I supposed to know? I don't know the book of business. So all of it. Yeah, just have all of it. It was a smallish book. And more importantly, once you finish with it, basically your test environment, you had to make sure it was cleared down. Fair enough. So we filled it out, completed it.

[00:17:24] Okay, so who signs it off? A security forum. Right, and they would meet once a month. How useful is that? We managed to get over that hurdle and thought, yeah, great, we must be ready to start testing now, because we'd filled out the first couple of forms and then this great big other document, and off the wall a small bit of testing[?]. But no, we weren't finished yet. The next thing was, because we needed to test with this albeit smallish book of business, all of it, they had to line me up with the business security, the actual information security manager, and present the pack and go through it. And it was absolutely a nightmare. You kind of left feeling literally tied up in knots. And do you remember the first quote I said, about testing being challenging? Wow.

[00:18:21] But at that point we were able to start testing, although, let's be honest, you felt a bit exhausted by that point. Thankfully we were able to successfully test, because we had all the data in there, whatever we needed to do, all the customer journeys we were able to do. We obviously had to make sure that the right people then cleared down the data. But it was a bit of a headache, by no stretch of the imagination.

[00:18:55] I've always got to be honest, I'd been in financial services at that stage — those first ten years of testing — and we'd never, ever, ever even looked at testing in production, mostly because we were an outsourcing company that I worked for at the time and so it wasn't our data anyway, it was the client's data. The clients were often averse to doing that, and you could understand it. Hence the reason, for this particular company, we had to go through all those hoops to actually just use production data in the environment. Most of the time they were able to either use handcrafted data or just manipulate it. So that was just one example, where, to say the least, it was very, very taxing.

Example two: validating in the production environment

[00:19:53] The next one is a more recent example, and this is actually using a production environment. This one is actually testing reports, so the data itself, and it's an internal customer. The internal customer is basically within our operations team, so it's one of our business teams, our personas, as we refer to them. Thankfully we have a very good rapport with them, we built up a very good engagement. They're really keen to have the solution. It's a solution they've been wanting for a long time. As I say, with the business change team, my colleagues have actually managed to get them on board, bringing them on this journey, and thankfully it's actually paying off for ourselves.

[00:20:55] It's an analytics solution. The way we've actually been able to use a production environment is because we can limit it to the type of system we're actually testing. The users are on licenses, so we can ring-fence them to those licenses, and actually it's almost like a soft launch, really. We've come in and run it. It's a pilot, but it is more of a soft launch to those sort of early adopters, and just not used in there[?]. Because we found that actually, in the test environment, without having the proper data, they really weren't getting any benefit at all from testing, and so the decision was made to let them pop it in, because we can't test it otherwise. Then we can get some useful output and some useful insights from that. So that's what we're currently doing. But to say, it was not taken lightly, and it was with the stakeholders' agreement as well. It's what they wanted to do.

Regulations and GDPR

[00:22:05] So, the regulations. There's always a sticky, thorny subject. It was mentioned earlier on, I think Hannah mentioned GDPR, the General Data Protection Regulation. This further complicates the use of any personal data, and quite rightly, because obviously it protects myself and yourselves as well. But it does place some strain a little bit on what we can do for testing.

[00:22:38] I'm not going to, especially at this time of day, because as I say it's been a great two days, I'm not going into those, or test you on the principles of GDPR. But needless to say, to actually process the data you need to have a lawful basis for doing so, and it's really kind of a gray area, how much you can stretch testing to be a lawful processing of that data.

[00:23:06] So what I've done in the past, and what's worked well in the past — as I said, thankfully I've always been able to use an environment that's as close to production as you can get — and this is one thing that GDPR does allow, is to use pseudonymized or anonymized data, where you're masking it. Although GDPR does stipulate that if you go in to mask it, or anonymize it, or obfuscate it, it can't then be reversed. It's got to be one way only. So that's one way of getting round that. Obviously you don't want to do all data, because you might as well handcraft it yourself. So that's a complexity.

Three things to consider

[00:23:52] So what I wanted to leave you with, some takeaways really, three points to consider if you're looking into testing in production, or doing your UAT validation in production. It's your data. How much is it? Is it personal data? How sensitive is it? How restricted? I know with some of our clients we've got some where they are secure, others where they're restricted but not quite as secure as the secure ones. So it really depends on the kind of data that you're actually looking at. As I said, my second example, it was more reports, so it wasn't like the personal data or corporate data.

[00:24:42] The next thing is, what's your motivation? What was the reason you're looking at it? Is it because of time saving? Is it the fact that, like I was saying before, you get to a stage where you're squeezed and squeezed and squeezed to do your UAT, that simply you just think, I've got two days, and it's just, let's just throw it in there? Probably not a good reason for doing that. Similarly with cost saving. Is it just a case of, oh, maybe we don't need her? And that's the other thing as well, environment. How many times do I have an argument about, yes, I do need another environment? I need a UAT environment. I know you've got ones for various other stages, but yeah, I kind of need one for testing. And then, do I get to share it? No, I don't share it with the other one. So again, it's that kind of thing. So possibly not the best motivation either.

[00:25:43] Another interesting point is, is it to achieve the outcomes, the customer journeys[?] that drive your outcomes? As I said earlier, my first example of using production data, we were able to actually achieve all the user journeys by using that production data, so I thought that was a pretty good reason for doing it. Is it the fact that you have the capacity of your production environment versus a testing environment? So many times we've found issues where, whether it be waterfall, which we've done a lot of in my previous job, or agile, no matter how much you test it, whether it's functional testing, integration testing, UAT, as soon as you go into production you can sometimes find a problem and you can never understand why. You can get the database guys looking at it and they'll compare and say, well, they're the same, the only thing is the capacity, but everything else, the code's the same, everything, every tiny other piece is the same. So is it because of the fact that you just feel that actually you need to be able to put a lot more data in, to use more data?

[00:27:03] Another thing, and again this kind of goes with the one before, the capacity: there's no restrictions on customer journeys, because we've got all the data in there, you can run through it, and maybe that's what you want to do. And then lastly, your data consumers. Your customers, like mine, they're internal to us, so they're our internal customers. We're transforming their business to give them better service so they can actually improve what they're doing in the day to day, which hopefully will have that positive impact on actually serving our own customers. Or is it external customers, the actual general public, or whoever your customers are? So that's the three things to really consider, other than obviously the regulations.

[00:27:56] And I think back to, as I said right at the top of the talk, about my parents, and one thing that really stood out as well. As I mentioned, a lot of people came into the shop and they would share things you'd never share with anyone you didn't really know. But my parents never shared that with anybody. They didn't share it with us as youngsters. We never knew anything. We just knew people would come in and confide in them, but they always kept that safe, secure, locked away, because they wanted to. And that is the thing, I suppose, another little thing: do what's right for your customers, and obviously stay within the regulations. And that was the end of my talk[?]. Thank you for staying with me.