Appetite For Risk - Continuous Delivery In A Regulated Environment

Kevin Duggan at UXDX EMEA. Video: https://youtu.be/ReIKBrW0yEg

Readable transcript: edited from the recording's captions for readability (fillers and false starts removed, punctuation and section headings added). Wording is the speaker's own. Timestamps are positions in the video. Names marked [?] could not be verified against the audio.

The contradiction at the heart of life sciences

[00:00:00] Hi everyone. Today I present you my story of the last 18 months of building a team and software in the domain of life sciences, a regulated domain.

[00:00:11] Life sciences fascinates me in a number of ways. It is a large growing market. It is complex. But most importantly, it has a direct impact on the health and wellbeing of billions of people around the world, and is, as illustrated by the events over the last 18 months, more important now than ever before.

[00:00:30] But what really fascinates me about this domain is the contradiction at its heart. We need to move fast and innovate to improve the quality of life for people around the world. But we also need to keep these people safe. Hence, the appetite for risk in this domain is understandably low.

[00:00:49] The threat of patient harm and product recalls is ever present, and more and more frequently software is at the heart of these, of the design and function and also the failure of these devices. Take the recent recall of the Medtronic pacemakers in the US due to a software error. A pacemaker is a class one device, meaning the use of the device may cause serious injury or death to patients. Again, the appetite for risk for these companies is understandably low, yet they face the constant need to innovate and produce better patient outcomes. The tension between speed and quality takes center stage in this domain.

[00:01:34] Today I'm going to focus on the complexity, and specifically on the part of the complexity that makes developing software products and innovating so difficult in this space: regulation. I'll bring you through some of the challenges I have seen and faced over the last 18 months in meeting our own compliance requirements, and how we overcame them. I will touch on how we create alignment across the product, delivery and quality teams, and how we accelerated the development and delivery of our own product without compromising quality.

[00:02:06] I guess you're asking, why should I care? Some of you may work in the domain of life sciences. You would have directly experienced some of the challenges I will mention today. Others who are listening to this talk may be working in other regulated domains and face similar issues, and I hope to give you some ideas to take back to your teams. And for everyone else, the tension between moving fast and breaking things is common to all software product development. Though the stakes are not as high as in life sciences, I believe every team can benefit from reflecting on what quality means to them and how it can be an enabler for speed.

Joining Qualio and travelling back to 1997

[00:02:46] Back in January 2020, I joined Qualio, a company which builds a SaaS product that serves life science companies. Our mission is to help teams create and launch lifesaving products.

[00:02:57] Now, I had come from an unregulated domain where moving fast and failing fast was encouraged. Experimenting, getting fast feedback, were the ebb and flow of each week. And this flow was built upon the engineering practice of continuous delivery, an approach specifically developed to reduce the risk of change while promoting fast learning, safely, quickly and sustainably.

[00:03:20] But now I found myself in a different world, a regulated world, where the barrier to flow was much higher, and it hurt me. I felt like I had traveled back in time, and specifically to 1997. This is an excerpt, a part of the regulatory guidelines that our customers in life sciences must adhere to around software validation of the software products they use. And this was written back in 1997.

[00:03:56] Qualio is a startup company. At the time we had five people in engineering. We needed to iterate and innovate and grow to survive. But I found myself blocked. The principles and practices that served me well were no longer welcomed or encouraged. Releasing early and often was a no. Any change to production had layers of control wrapped around it, control that extended beyond the boundaries of our company and into that of our customers.

[00:04:19] Regulatory bodies, as outlined by the statement, mandated that our customers validated our software after every change. This means that for every change we have to notify our customers, who then were required to manually test our software, which takes hours, before accepting the change. So it was press & play[?], a contradiction. We are helping them by improving our product, but at the same time negatively impacting them with these changes.

Internal controls and quality theater

[00:04:47] But it wasn't all about our customers. I won't start there. We had our own internal quality team, and control based around ISO 9001. ISO 9001 is about controlling the design and development processes. Basically any change to your product needs to be well specified, and when the work is done you need to validate it, prove it was done. You need to record and sign off on the results. And you guessed it: lots and lots and lots of documents. Like the ones here on the right. Software requirement specs[?], test plans, risk assessment sheets, all just so much stuff.

[00:05:27] One of our customers, a product manager in a medical device company who I was talking to, shared a story with me of how she spends three days, the last three days of every sprint, chasing down people, sometimes begging, sometimes threatening, trying to get them to update documentation so she could close out the sprint with everything in order. And remember, this is three days after all the work had been done. It was quality theater, but it had to be done.

[00:05:55] Now imagine you are a developer on that team, or a product manager, having to do this every time. A time consuming, tick the box exercise, a bureaucratic game of who had not completed what, a scramble to get everything in order. All of this friction on top of an already pressurized team, and it had no meaningful impact on the actual quality of the products.

The three Qs and the external bottleneck

[00:06:18] This is internal, and then you move external. As I mentioned a minute ago, we are a SaaS product that sells to life science companies. This means we are subject not only to our own internal controls but to external customer controls, of having them validate any changes we make.

[00:06:38] These layers of validation that applied to us are known as the three Qs. You must ensure the software is installed correctly in a given environment, that it functions according to its requirements, and that it performs consistently in that environment. Again, these instructions make sense if I think back to 1997, if I was installing a piece of software on some on-premise server, or maybe a physical piece of equipment into a lab or a team room. But we are a cloud-based SaaS application.

[00:07:07] The OQ and PQ required our users to sit down and validate or test our application after each change, to ensure that it still met their intended use. And for anyone who can remember the last time they ran through tens or hundreds of manual test cases, it is not a pleasant task. It is extremely time consuming and it is also not very effective. Each time we released a new feature we were actively hurting our users. But this was the cost of doing business.

[00:07:41] These internal and external bottlenecks took their toll on our small team. All those bad product development practices I'd evolved out of were all around me. Our cycle time was measured in months, not in days. We had long lived feature branches deployed to multiple long-lived[?] environments. We were moving towards big bang quarterly releases when we should have been running away from them. With this heavyweight release process we couldn't even entertain the thought of breaking up our monolithic code base, which was hindering our team growth. We shied away from customer research, aware of the implications that any proposed changes had for them.

[00:08:12] The spirit of the regulation and the associated controls is good: reduce risk, share responsibility through the review of documentation, and increase transparency through documentation and testing. But the burden of the work that was built around them had the ultimate effect of creating a team who stopped caring and gave up on efforts to make the situation better. The result of this burden and culture was a slow pace and product failure.

The catalyst: applying a product mindset to compliance

[00:08:51] In our worst example, we spent six months building a new feature only to pull it shortly after release, due to it not gaining any adoption with customers. Now, we were compliant in this release. We were making very bad decisions around what customer value was and meant. But this failure proved to be a catalyst.

[00:09:09] Ironically, it was back to lean product development principles that we turned. Quality and compliance were part of the product after all, a non-functional but still critical part. So why not apply a modern product mindset to this bottleneck? At the heart of all this was risk, so maybe a more agile approach, an approach which had been developed to tackle risk, would work too. I believe that quality and speed are not opposing forces, and that the continuous delivery promise of better software faster holds true. It was not that we wanted to reduce the appetite for risk, rather we wanted to satisfy it in a more effective way.

[00:09:51] The first step was to find a common language, a common goal, and a common approach to unite our product, engineering and quality teams. So we, as a product team, sat down and read and discussed the regulations and our existing policies and procedures. We tried to see them in the spirit in which they had been written, and ignored the machinery that had been built up around them. They were only guidelines that needed to be applied in context, our context.

[00:10:14] If you think of the industry that has built up around things like agile and DevOps, it can turn people off, but at their core are very positive mindsets and guidelines. Look at these quality principles that are based on 9001. Things like evidence-based decision making, customer focus, engagement of people. They are all values that anyone from product to UX to engineering can align upon.

Mapping compliance onto the delivery process

[00:10:44] So we started, alongside our quality team, to map our compliance process on top of our delivery team process. To anyone listening to this who is used to mapping exercises — user story mapping, value stream mapping — this was another great application of that technique. At each step of the value stream we asked these questions: are we compliant, and are we delivering value in a timely manner? Both of these had to be satisfied.

[00:11:08] We discussed alternatives to our current heavy-handed approach. Namely, a more agile approach to risk driven by more modern product development practices. And we started with continuous delivery. So, again: safely, quickly and sustainably. There is a lot here that both quality and delivery teams can align behind.

[00:11:37] We used this as a shared goal. It was a big investment to move to a continuous delivery method, but as you'll see, it paid off. This is a diagram of our CI/CD pipeline, that had a huge positive effect on our quality. We moved from a monolithic to a microservice based architecture, which allowed for finer grained, auditable and testable deployment artifacts. We shifted our testing left, to earlier in the value stream. We made it a part of our delivery pipelines, not just having unit tests or other tests running on test environments, but having a suite of tests run as part of our pipeline after every single change, acting as a gate to production and a signal to roll back quickly on failure. We planned these tests early, meaning it had the added benefit to us of covering off, through validation, the regulatory needs of our core functionality early.

Discovery, design and the design control flow

[00:12:30] With these core technical practices in place, we then zoomed out a little to look at our product development processes around product and usability. We all know that discovery and design activities play a major role in the quality of any product. They flesh out the market need, the why behind any feature or functionality, and ensure the usability and value of any proposed solution or design. This maps perfectly into regulations like ISO 9001.

[00:12:58] These activities that we were doing on a daily basis for each new feature, each new idea, we mapped them to the form of validation activities in a new design control flow. For us that meant our value stream now covered from the initial outlining, from things like jobs-to-be-done, right through to the feature being tested and monitored in production.

[00:13:24] The two big takeaways here are, first, it's a lot of collaboration, and you're going to make changes, but make sure that your policies and procedures are updated to reflect these changes. When we experimented and found something that worked, we didn't pat ourselves on the back, we worked with quality to evolve our policy to reflect our new way of working. These are not dusty documents to be revered, they should be living documents that reflect your ways of working. This is key in an audit situation, that you are true to your policies.

[00:13:57] Secondly, treat quality, the quality team, as your customer, and the artifacts of compliance as part of your product. There are many great product lifecycle management and application lifecycle management tools, test management tools, test automation frameworks, that make capturing and collecting this evidence much easier. It is not 1997 anymore. Quality teams can now pull information related to compliance at any time without interrupting the product team's flow. This greatly reduces the internal burden on QA around each change and release and frees them up to be part of the product development process proper.

[00:14:37] Finally, getting you to a quality that goes beyond compliance. We extended our cross-functional team to include QA and RA. With the regulatory basics covered, they had the capacity to get involved right at the beginning of each feature, and instead of spending days at a time retrospectively reviewing documentation after every release, they got right in at the start and guided us along the way.

Results: growth without the compliance bottleneck

[00:14:59] Now, this was internally. We were in a much better place, and it turned out happily that there was a major overlap between what we had achieved internally and what our customers needed from us in terms of the three Qs, the installation, the qualification and the performance. We were able to reuse and repackage the traceability, the verification and validation artifacts, the creation of which we have now automated, and share them with our customers.

[00:15:31] Having continuous delivery in place meant we could go further. We reached out to more customers to validate ideas earlier. We found those early adopters who are more flexible in their approach to validation and risk, perhaps being earlier in their own product lifecycle.

[00:15:47] All of this happened iteratively over a hectic 12 months. The mapping, the rearchitecting, the orchestration of this data, the closer collaboration with customers, the reduction of risk, the smaller, more validated changes. All major steps forward. Our QA team could now pull information related to compliance from the product team at any time, perhaps for an audit or based on a customer request, without interrupting the team's flow. And more importantly, due to increased visibility, they now act as detectives, monitoring the quality of work as it happens, proactively tracing each change to the product from initial idea through to release, and helping the product team fill in gaps as we went.

[00:16:34] Throughout this time the team grew by a factor of six, going from one to six teams[?]. This growth was not impeded any more by compliance bottlenecks, but rather it was accelerated by a leaner approach to product development. We also maintained our ISO 9001 certification with far less burden on the product team and the quality team. But most importantly for me, we grew as a generative organization. We shared responsibility for both regulatory and product risks.

Conclusion: build a quality culture that goes beyond compliance

[00:17:06] In conclusion, you can build whatever it is, you can build a quality culture. That's right, you can build a culture of quality that goes beyond compliance. I urge you to go back to your teams, both product and quality, and talk about what quality means to you and your users, and then challenge and improve it in code and in any changes you make to your policies.

[00:17:29] Make sure you are capturing the outputs of product and development processes: your tests, your jobs-to-be-done, your user testing. The stuff that you're already doing, not to mention your quality process, and automate their collection. You're already doing this work, so get more value from it. This provides transparency and builds trust between stakeholders and your customers, and allows you to move fast.

[00:17:51] It turns out that the contradiction I spoke of, each change putting a burden on our customers, could easily be solved through communication and automation. You can move beyond compliance theater to a more engaging and effective culture of quality that goes beyond compliance. This is an approach I would encourage you to explore, and you can do this without reducing your appetite for risk, but by working together as a team to find other ways to satisfy it. Thank you.