Appetite For Risk - Continuous Delivery In A Regulated Environment
Appetite For Risk - Continuous Delivery In A Regulated Environment
Software needs to be released early and often AND it needs to be thoroughly tested. But in life sciences, customers are responsible for verifying every upgrade of your software. How do you reconcile your need for fast learning and feedback with the burden that each release puts on your customers?
In this talk Kevin will cover:
- The multiple levels of validation required in life-sciences software
- The technical challenge of automating validation at each level
- How to ensure regulator approved quality with multiple autonomous teams and codebases
- Getting buy-in from the developers, internal QA, customers and regulators
Hi everyone. Today, I present you my story of the last 18 months of building a team and software in the domain of life sciences, a regulated domain.
Now life sciences fascinates me in a number of ways. It is a large growing market. It is complex, but most importantly, it has a direct impact on the health and wellbeing of billions of people around the world and is as illustrated by the events over the last 18 months, more important now than ever before.
But what really fascinates me about this domain is the contradiction at its heart. We need to move fast and innovate to improve the quality of life for people around the world. But we also need to keep these people safe. Hence, the appetite for risk in this domain is understandably low. The threat of patient harm and product recalls is ever present and more and more frequently, software is at the heart of these, of the design and function and also the failure of these devices. Take the recent recall of the Medtronic Pacemakers in the US due to a software error. Now, pacemaker is a class one device, meaning the use of the device may cause serious injury or death to patients. Again, the appetite for risk for these companies is understandably low. Yet they face the constant need to innovate and produce better patient outcomes. The tension between speed and quality takes center stage in this domain.
Today, I'm going to focus on the complexity and specifically to part of the complexity that makes developing software products and innovating so difficult in this space - regulartion. I'll bring you through some of the challenges I have seen and faced over the last 18 months in meeting our own compliance requirements and how we overcame them. I will touch on how we create alignment across the product delivery and quality teams and how we accelerated the development and delivery of our own product without compromising quality.
I guess you're asking, why should I care? But some of you may work in the domain of life sciences. I would have directly experienced some of the challenges I will mention today. All those who's listening to this talk, maybe working in other regulated domains and face similar issues. And I hope to give you some ideas to take back to your teams and for everyone else, the tension between moving fast and breaking things is common to all software product developments. And though the stakes are not as high as in life sciences, I believe every team can benefit on reflecting on what quality means to them and how it can be an enabler for speed.
So back in January, 2020, I joined Qualio, a company which builds a SaaS product that serves life science companies. Our mission is to help teams create and launch lifesaving products. Now I had come from an unregulated domain where moving fast and failing fast was encouraged, experimenting, getting fast feedback or the ebb and flow of each week. And this flow was built upon the engineering practice of continuous delivery. An approach specifically developed to reduce the risk of change, but promoting fast learning, safely, quickly and sustainably. But now I found myself in a different world, a regulated world for the barrier to flow was much higher and it hurt me. I felt like I had traveled back in time and I had specifically to 1997.
Now, this is an excerpt apart of the regulatory guidelines that our customers in life sciences most adhere to around software validation of software products they use. And this was written back in 1997. Qualio is a start up company. At the time we have five people in engineering, we needed to iterate and innovate and grow to survive. But I found myself blocked. The principles and practices that served me well, we're no longer welcomed or encouraged, releasing early and often was a no. Any change production had layers of control wrapped around us, control that extended beyond the boundaries of our company and into that of our customers. Regulatory bodies, as outlined by the statement mandated that our customers validated our software after every change. This means for every change we have to notify our customers who then were required to manually test our software, which takes hours before accepting the change. So it was press & play, a contradiction. We are helping them by improving our product, but at the same time negatively impacting them with these changes, but it wasn't all about our customers. I won't start there.
We had our own internal quality team and control is based around ISO 9001. The ISO 9001 is about controlling the design and development processes. Basically any change of your product needs to be well specified and when the work is done, you need to validate, prove it was done. You need to record and sign off on the results and you guys did comply lots and lots and lots of documents. Like the ones in the right here, software, current specs, test pads, risk assessment sheets, all just so much stuff. One of our customers, a product manager in a medical device company, who I was talking to shared a story with me of how she spends three days, the last three days I've every spent chasing down people, sometimes begging, sometimes threatening, trying to get them to update documentation so she could close out the sprint with everything in order. And remember, this is three days after all the work had been done. It was quality theater, but it had to be done. Now imagine you are a developer on that team, or product manager, having to do this every time consuming, tick the box, exercise a bureaucratic game of who had not completed what had scramble to get everything in order. All of this friction on top of an already pressurised team that had no meaningful impact on the actual quality of the products.
This is internal and then you move external. And as I mentioned a minute ago, we are a SaaS product that sells the life science companies. This means we are subject to our not only our own internal controls, but external customer controls of having them validate any customer changes, any changes we make.
These layers of validation that applied to us are known as the three Qs. You must ensure the software is installed correctly on a given environment that it functions according to its requirements and it performs consistently in that requirement. Now, again, these instructions make sense if I think back to 1997, if I was installing a piece of software and some on-premise server, or maybe a physical piece of equipment into a lab or a team room, but we are a cloud-based SaaS application. The OQ and PQ required our users to sit down and validate or test our application after each change to ensure that it still met their intended use. And for anyone who can remember the last time they ran shoot, tens of hundreds of manual test cases, it is not a pleasant task. It is extremely time consuming and it is also not very effective. And each time we released a new feature we are actively hurting our users. But this was the cost of doing business, with these internal and external bottlenecks took its toll on our small team. All those bad product development practices I'd evolved out everyone around me. Our cycle time was measured in months, not in days. We had long lived feature branches deployed to multiple language environments. We were moving towards big bang, quarterly releases, when we would have been running away from them. With this heavyweight release process, we couldn't even entertain the thought of breaking up our monolithic code base, which was hindering our team growth. We shied away from customer research, aware of the implication that any proposed changes had for them. The spirit of the regulation and the associated controls is good. Reduce risk to sharing responsibility through the views of documentation and increasing transparency to documentation and testing. But the burden of the work that was built around them had the ultimate effect of creating a team who stopped caring and give up on efforts to make the situation better.
The result of this burden and culture was a slow pace and product failure. In our worst example, that we spent six months building a new feature only to pull Sharky after release due to it not gaining any adoption with customers. Now we are compliant to this release. We are making very bad decisions around what customer value was and meant, but this failure proved to be a catalyst. Ironically, it was back to Lean Product Development Principles that we turned. Quality and compliance were part of the product after all, an unfunctional, but still a critical part. So why not apply a modern product mindset to this bottleneck. At the heart of all this was risk, so maybe a more agile approach an approach which had developed a tackle risk would work too. I believe that quality and speed are not opposing forces but the continuous delivery promise of better software faster holds true. It was not that we wanted to reduce the appetite for risk, rather we wanted to satisfy it in a more effective way. The first step was to find a common language, a common goal, and a common approach to unite our product engineering and quality teams. So we, as a product team, sat down and read and discussed the regulations and our existing policies and procedures. We tried to see them and disperse in which they had been written and ignored the machinery that had been built up around them. There were only guidelines that needed to be applied in context, our context. If you think of the industry that has built up around things like Agile and DevOps, it can turn people off at their core are very positive mindsets and guidelines. Look at these quality principles that are based on 9001. Things like evidence-based, decision-making, customer focused engagement on people that are all values that anyone from product to UX to engineering can align upon.
So we started alongside our quality team to map our complaints process on top of our delivery team process. At each step, I'm sure to anyone listening to this who is used to mapping exercise, user story, mapping value, stream mapping, this was another great application of that technique. At each step of the value team, we asked these questions, are we compliant and are we delivering value in a timely manner? Both of these had to be satisfied. We discussed alternatives to our current heavy-handed approach. Naomi, a more agile approach to risk driven by more modern product development practices. And we started with continuous delivery.
So again, safely, quickly and sustainably. There is a lot here that both quality and delivery teams can align behind.
We use this as a shared goal. It was a big investment to move to continuous delivery method. But as you see, you'll see it paid off. This is a diagram of arrow CI/CD pipeline that had a huge positive effect in our quality.
We moved from a monolithic to microservice based architecture, which allowed for finer grained, auditable and testable deployment artifacts. We shifted our testing lift to earlier in the value stream. We made it a part of our delivery pipelines and not just having unit tests or other tests running on test environments or having suite of tests run, as part of our pipeline, after every single change acting as a gate to production and a signal to roll back quickly on failure. We planned these tests early, meaning, it had the added benefit to us of covering off through validation, regulatory needs of our core functionality early.
With these core technical practices in place within zoomed out a little to look at our product development processes around product and usability.
So we all know that discovery and design activities play a major role in the quality of any products. They flash out the market need. The why behind any feature or functionality. To ensure the usability and value of any proposed solution or design. This maps perfectly into regulations than ISO 9001. These activities that we were doing on a daily basis for each new feature, a new idea we mapped them to the form of validation activities in a new design control flow. For us, that meant our value stream now covered from the initial outlining, from things like jobs-to-be-done right through to the feature being tested and monitor in production.
The two big takeaways here are, it's a lot of collaboration. And you're going to make changes, but make sure that your policies and procedures are updated to reflect these changes. When we experiment and we found it experimented and found something that worked, we didn't pat ourselves on the back, we work with quality to evolve our policy, to reflect our new way of working. These are not dusty to your documents to be revered, which should be living documents that reflect your ways of work key. This is key in an audit situation, that you are true to your policies. Secondly, you treat quality to quality teams or customer and the artifacts of compliants as part of your product. There are many great product life cycle management, application, life cycle management, test management tools, test automation frameworks that make capturing and collecting this evidence much easier. It is not 1997 anymore. Quality teams can now pull information related to compliance at anytime without interrupting the product teams flow. This greatly reduces the internal burden on QA around each change and release and frees them up to be part of the product development process proper. Finally, getting you to quality that goes beyond compliance. We extended our cross-functional team to include QA/RA with the regulatory basis covered. They had the capacity to get involved right at the beginning of each feature. And instead of spending days at a time retrospectively reviewing documentation after every release, they got right in at the start and guided us along the way.
Now this was internally, we were in a much better place and it turned out happily, but there was a major overlap between what we had achieved internally and what our customers needed us from us in terms of the three QS, the installation, the qualification and the performance.
We were able to reuse and re-package the traceability, the verification to verification, validation artifacts, the creation of which we have now automated and shared them at our customers. Well, having continuous living in placement meant we could go further. We reached out to more customers to validate ideas earlier. We found those early adopters who are more flexible in their approach to validation and risk, perhaps being earlier in their own product life cycle. Now, all of this happened iterately over a hectic 12 months. The mapping, the rearchitecting, the orchestration of this data, the closer collaboration customers, to reduction of risk, to smaller more validated changes, all major steps forward. Our QA team could now pull information related to complaints from the product team at any time, perhaps, for an audit or based on a customer request without interrupting the team's flow. So more importantly, due to increased visibility, they now act as detectives. Monitoring the work of quality.Monitoring quality of work, as it happens, proactively tracing each change to the product from initial idea through to release and helping the product team fill in gaps as we went.
And throughout this time, the team grew by a factor of six, going from one to nine to six teams. This growth was not impeded anymore by compliant bottlenecks, but rather it was accelerated by a leaner approach to product development. We also maintained our ISO 9001 certification with far less burden on the product team and the quality team. But most importantly, for me, we grew as a generative organisation. We shared responsibility for both regulatory and product risks.
I think conclusion, you can build whatever it is. You can build a quality culture. That's right. You can build a culture quality that goes beyond compliance. I urge you to go back to your teams, both product and quality and talk about what quality means to you and your users. And then challenge and improves in code and in any changes you make into your policies.
Make sure in capturing outputs of product and development processes, your test, your jobs-to-be-done, your user testing. The stuff that you're already doing, not to mention your quality process and automate their coalition. You're already doing this work. So get more value from it. This provides transparency and builds trust between stakeholders and your customers and allows you to move fast. It turns out that the contradiction, I spoke of each change, putting a burden on our customers could easily be solved to communication and automation. You can move beyond the compliants theater to more engaging and an effective culture of quality that goes beyond compliance.
This is an approach I would encourage a change to explore, and you can do this without reducing your appetite for risk, but by working together as a team to find other ways to satisfy.