Building Quality: Test-Automation And Tdd

Scott Riseborough at UXDX Europe. Video: https://youtu.be/dU5BKr0WI4o

Readable transcript: edited from the recording's captions for readability (fillers and false starts removed, punctuation and section headings added). Wording is the speaker's own. Timestamps are positions in the video. Names marked [?] could not be verified against the audio.

Introduction: misconceptions about software testing

[00:00:05] Good morning everyone. Quality assurance, test engineering, software testing, regardless of what you may have heard it called before, I think everyone working in this industry pretty much agrees that testing software before it goes out to customers is usually a positive thing. Unfortunately though, there are a number of misconceptions around software testing and software test engineers themselves as well. I'd like to address some of these points today and discuss the topic in a little high-level detail, and put a new spin on the subject as well. I'm here to broach the topics of manual and automated testing, but in a rather unique way which I can pretty much guarantee that no one has really attempted before.

[00:00:46] First, a little bit of background about me. I've been a QA for the last seven years now and I've worked across various genres of technology, and in various sizes of company as well. I started my career at Cisco, working for two years with them on their video conferencing service. I've worked at multiple financial institutions including Barclays, where I worked on FX trading platforms. I did a two-year stint at YouView[?] involved in set-top boxes and Android TVs. And I'm currently, as you heard from Joe, the lead tester at Apperio. I say lead tester, I am the only tester at Apperio. Apperio basically connects law firms and their clients in real time so they can see legal spend. And I feel that working across this broad range of software products and different sizes of company has given me a fairly unique insight into what the software testing world has to offer.

[00:01:41] However, alongside this career I've had a second career doing this.

[00:02:33] Thank you. That's right, I make my career making annoying noises, essentially. As a beatboxer and a vocalist I've taught all across the world, I've competed in numerous international tournaments, eventually fought my way to being the UK beatbox champion in 2015. And in balancing my two careers over the last seven, eight years, I found that I've been using elements of my test engineering to approach my beatboxing, and vice versa as well. And that's going to be kind of a core concept I'll be using as part of this talk today. I told you it's going to be different.

[00:03:13] So, to take a fairly famous quote to try and justify what I'm doing: "To raise new questions, new possibilities, to regard old problems from a new angle, requires creative imagination and marks real advance in science." It was said by a little-known scientist by the name of Albert Einstein. And what I've always taken this to mean is that a good test engineer, or a developer, or a UX designer, can always make better sense of something by choosing to look at it in a different way. And that's a key concept I'll be exploring, and I can assure you this will be very different to some of the talks you hear over the next couple of days, and hopefully very entertaining at the same time.

Solo beatboxing as an analogy for manual testing

[00:03:50] But let's start by having a closer look at manual testing. What I've shown you already is an example of solo beatboxing. It's just me and a microphone, there's nothing else involved, there's no sound effects going on, I'm making all of the sounds that you hear. And solo beatboxing is an incredible analog for manual testing. So we're going to look at testing through the lens of human beatboxing.

[00:04:17] In both beatboxing and manual testing, I have the basic tools which allow me to carry out my job. With testing this would be a computer, the application under test, network access. Similarly with beatboxing, I have the air I breathe. I have the right training to be able to use these tools in an effective way: I did physics at university, so I have a fairly analytical problem-solving background, and I have an understanding of web applications and network layers. Similarly with beatboxing, I have the training and the background to be able to turn random breaths into a fairly coherent beat. And I also have some other tools which enhance my abilities as well. With testing I'll have various software tools at my disposal, I'll have image capturing, defect management software, etc. And similarly with my beatboxing, I am literally being amplified, my skills are being amplified right now. Thank you, Mr. Sandman[?].

[00:05:17] There are however a gross number of misconceptions when it comes to manual testing, and I'd like to discuss these, because in my experience a solo manual tester can actually achieve quite a lot.

[00:05:32] Now, usually when I ask a room about the misconceptions of manual testing, the same themes always seem to crop up: it's expensive, it's slow, it's boring, it's prone to failability. And the argument I'd like to put to you today is that whereas some or all of these things can be true with manual testing, I would argue it only happens when it's implemented in a poor way. One manual tester alone can actually achieve quite a lot.

[00:05:58] In the short term the cost is invariably lower. Getting a smart manual tester to understand your application and to test it in an effective way is much quicker and cheaper than hiring a team of automation engineers to develop a test automation infrastructure for you. Keyword being in the short term: in the long term, of course, an automation infrastructure will outperform.

[00:06:18] You're not restricted by conventional methods. I'm not constrained as a manual tester to work within any specific infrastructure, say Selenium or Gherkin. Manual testers have the flexibility to adapt to the needs of the application or the feature under test, and they can choose the best available tool rather than forcing an existing one to do something it isn't necessarily designed for.

[00:06:39] Decisions are made quickly and flexibly. I'm not relying, again, on much code or infrastructure. In an agile world changes happen fast and they happen hard. The mantra of move fast and break things, which has been adopted by the likes of Facebook and Google, means that tests break and become outdated much more frequently than they used to, and a smart manual tester can respond to these problems a lot quicker than an automated tester updating their entire test bench. And manual testers can also context switch very quickly when the need arises. If the needs of the business change, or an emergency fix needs rolling out, they can immediately jump onto that and change very quickly. And manual testers are also encouraged to find creative solutions to problems, sometimes it's even a necessity. I'll discuss this in a bit more detail later on in the talk.

[00:07:30] And I feel that all of these points can be demonstrated through solo beatboxing as well. So for instance, the short-term cost being lower: to hire me as a performer for your event, literally all you need is me, a microphone and some speakers. Cheaper than hiring a band, get in touch with me later if you want to. Similarly, I'm not restricted by conventional methods: I don't need a drum kit to make drum sounds, I don't need a synthesizer, I don't need one of these.

[00:08:04] Decisions can be made quickly and flexibly: I can context switch on a dime as a beatboxer. If you want a techno beat, or if you want samba, I can change immediately. And similarly, I'm encouraged to find creative solutions to the problems I face. If I need to sing and beatbox at the same time, I can either cheat and fit the words in between the beats, or I can sing through my nose while I'm beatboxing.

Where manual testing still has its place

[00:08:39] So you see, it is a good analog, it does work. But even with modern cloud-based products like Apperio, manual testing still very much has its place. Testing designs is possibly the most important part of a tester's job: to step out of this test engineering silo that a lot of companies have gotten stuck in and to be an active part of the feature design process, working with UX designers, working with product managers. Exploratory testing, any sort of trial-style[?] user experience based testing, still a manual version of test targets, regression, ad hoc regression.

[00:09:14] At Apperio, as I've said, I'm currently the only QA, so I do need to get creative about the solutions for testing in order to maintain a high quality standard. Understanding your user base and identifying the most important user journeys allows me to properly prioritize my testing and ascertain which tests are a priority for automation as well. And although our manual testing methodology at Apperio is strong, when you are a lone tester working with nine developers and tracking nearly 14 billion pounds worth of legal spend, we can easily start to see where the lure of automation lies.

[00:09:49] Now of course there are downsides to solely relying on manual tests, but they're not usually the ones that we mentioned earlier. I tend to find that the first one is that there's a single point of failure. As a manual tester, if I pass a test and I don't have any automation or any further check on that, and even if I'm not testing that test every single iteration of a release, there is a chance that broken code can be delivered to a customer.

[00:10:18] There's a limit to what can be achieved: I can only execute so many test cases in a day versus, say, a swarm of automation services. Fatigue: this can mean two things. Yes, it is kind of tiring and boring to run through the same regression test. I'm glad to see a show of hands as to who's ever had to be involved in a manual regression cycle. I feel your pain, guys, I feel your pain. And similarly, fatigue has another meaning as well, where it means if I'm a tester and I'm running the same test with the same expected results over and over and over again on every release, eventually I'm going to stop noticing things. And inconsistent results: nothing is perfectly reproducible. The computer was originally invented to do repetitive calculations quickly and accurately, and automated testing exists for exactly the same reason.

[00:11:10] And similarly, all of these issues again can be demonstrated using human beatboxing. As a beatboxer I have a single point of failure: if I run out of breath, or my microphone stops working, then no one's having a good time. There's a limit to what can be achieved: much as I would love to, I cannot perform Stravinsky's Rite of Spring on this microphone. Fatigue: eventually I have to take a breath. And inconsistent results: if I try and make the same beat twice, it might sound very similar with the training I've had to try and get consistent beats, but if you analyze those waveforms you'd find differences in timbre, in tone, in tempo, basically all the T's, and you would find that they are markedly different. There's no way, as a human element, that I can make something that is entirely reproducible twice. And consistency is important when testing any software.

When manual methods are taken too far

[00:12:06] Now, a good example, or a bad example, of manual methods being taken too far, is actually a little anecdote from when I was working at another financial institution which I haven't mentioned here, which will remain nameless for reasons you will shortly see. We were testing their pricing performance server. Once a month the testers would get on a conference call, the lead tester would sit on the pricing server, and we'd all be armed with one of these. The lead tester would say three, two, one, go, and change one of the foreign exchange currency pairs on the pricing server. We press go on our stopwatches, wait for the price to change on our machines, press stop and record the time.

[00:12:52] I can see a lot of very sceptical faces in the room, rightly so. This is a terrible, terrible way of measuring performance. In fact I did some statistical analysis on this for our manager, after I had broached the topic a few times, and found that the standard deviation of the measurement was greater than the mean. So we were recording timings of half a second with a margin of error of a second. And when bringing this to the attention of my manager and his manager, the reason I was fired back with was that it's caught problems in the past, therefore we should continue to do it.

[00:13:24] It's a gross misuse of testing time[?], and it falls into all of the negative points we mentioned earlier: it's slow, it's boring, it's limited, it's certainly not reproducible and it's definitely not accurate. And it just goes to show that, as creative as manual testers can be, eventually at some point it's not your best option. And I've had various experiences with automation over the years, it can be a real headache getting it set up and maintained properly, but with good agile practices teams will inevitably find quality levels easier to maintain and improve upon.

Automation as the loop station

[00:13:56] So to bring it back to the box, this is where my next tool comes into play. This is my loop station, and with this I'm able to record everything from my microphone into it and manipulate it in whatever way I choose. For example...

[00:14:14] Now, if you imagine that every single sound I make is an individual test case that needs to be executed against our platform, I'm going to take all of my test cases and input them into my automation framework.

[00:14:43] You can start to see that I'm starting to run my test cases in parallel and I can manipulate them in whatever way I choose. What I'm demonstrating here is that the power of automation tools amplifies the skill of the tester using them. One is useless without the other. If I wanted to make those sounds without the loop station, the closest I could come is... I haven't got the full effect there, and it's certainly not reproducible. Similarly, without my input, the only thing that my automation infrastructure can do is this. Them all together, suddenly you have something that's much more powerful, well rounded.

[00:15:29] And we can even demonstrate the individual layers of testing the application here as well. Again, remember that each sound I'm making is a specific test case that needs to be run against the platform. We have our unit test, the lowest level test case, this is running against a single function in code. Then we've got our components, so this is testing individual units of code together. Bringing multiple components together you have a fully fledged feature, and this is our integration testing. And then bringing everything together you have a system, or platform. Hopefully it's starting to become clear, the power that automated testing holds.

The limitations of automated testing

[00:16:12] But automated testing is not the solution to all your problems, and it does have its own limitations as well. Remember the grievances we had with poorly implemented manual testing: all of those can apply to automated testing as well. It can be expensive, both in time and money, to pay engineers and afford them the time to be able to maintain the code and infrastructure. It can be slow: at Apperio we have to deal with CSV files that are over a couple of million lines long, it takes time to process these and make sure that the performance is up to scratch.

[00:16:39] It can be boring: anyone who's had to go through an automated test report knows that in an ideal world everything would just go red or green and that would tell you whether your code is good to release. In reality, especially with bigger companies, this tends to sit somewhere in the amber region, and a test engineer has to comb through the reports and decide which are test case failures and which are actual bugs.

[00:17:03] And it's prone to failability as well. One of the first automated test cases I ever wrote was to see whether a data set could be sorted into alphabetical order. Unfortunately the data set I'd chosen to give it was already in alphabetical order. This test always passed, and unfortunately the feature broke, and despite the test report going green we couldn't actually see there was anything wrong.

[00:17:24] However, if you follow test-driven development principles, if you design your tests and test plans carefully before writing any code, if you design infrastructure with robustness and maintainability key in your mind, testing infrastructure specifically, and depending on your product as well stick to the test pyramid — lots of unit tests at the bottom testing the low level business logic, then fewer component, fewer integration and very few system and platform level tests — you should find that you'll have a good automation framework to rely on and your attention can be focused on moving forward with development.

Summary: automate more of the right things

[00:18:02] So in summary, what I want you to take away from this talk. The first thing is, don't underestimate your manual testers. They're often the people that can understand your product the best, and from a UX perspective you should really be working very, very closely with them, even before pen is put to paper for a design.

[00:18:18] The example I like to use when interviewing for test engineers is to show them a web page with two inputs for strings. I tell them the feature is supposed to deliver the longer of the two strings. Now, a good tester will provide you a comprehensive list of all the things that need testing. They will tell you, okay, well, have you tried hexadecimal, have you tried SQL injection, have you tried unicode characters, how long is the string, what if the strings are the same length? And these are very good. But the difference between a good tester and a great tester is the questions that the great tester will ask you: who is using this feature, what is the use case for it, how will they interact with it? They want to know more than just the functionality, they want to know about the user using it and how they will interact with it, to assess a level of quality.

[00:19:04] The second is, I'd say don't expect automation to solve every problem. The human element working cleverly with technology will always give you a more comprehensive result than either one alone. Similarly with my beatboxing, there are limitations to what I can achieve on my own, and similarly there are limitations to what the loop station can achieve as well. And remember the differences between this and this. And that's not to say I don't agree we shouldn't be automating more, but automation should never be viewed as a complete replacement for a decent manual tester. 70%, I would say, of my role at Apperio is that of a product expert, understanding the product from both a user perspective and at a technical level.

[00:19:47] Lastly, I'd say encourage your manual testers, encourage your developers, encourage your product managers, encourage everybody within your tech team to at least understand, or encourage them to look into, automation practices. Encourage them to view it as a way to extend their skills and make their lives easier and more fulfilling, and you'll spend less time worrying about whether a feature works and more time worrying about user experience standards.

[00:20:16] A meme that tends to do the rounds amongst developers and test engineers is this one, which is "automate all the things," and I couldn't disagree more with this. This is not the right approach. I've tried to amend this meme over the years to "automate more things," or even better, "automate more of the right things." So yes, test-driven design principles, effective test case prioritization, and understanding the needs of your user can make your testing much more effective than just blindly testing everything. And lastly, I would say encourage your developers, testers, UX designers, encourage their external activities, as you never know, maybe they'll also be able to bring an element of that into their work as well. Thank you very much.