Reducing Risk With Continuous Delivery

Dan Harper at UXDX APAC. Video: https://www.youtube.com/watch?v=RaXn84fAiV0

Readable transcript: edited from the recording's captions for readability (fillers and false starts removed, punctuation and section headings added). Wording is the speaker's own. Timestamps are positions in the video. Names marked [?] could not be verified against the audio.

The challenge: a complex project with zero downtime

[00:00:01] Hi, my name's Dan Harper. I'm a Senior Engineering Manager at Xero. Today I'm going to lead you through an example of a project with a whole bunch of teams who realized that they could leverage continuous delivery principles to reduce risk and have a great, successful outcome for customers.

[00:00:17] We're talking about a challenge that a bunch of teams had, where they had a very high complexity project and very low knowledge of the systems that they needed to change. And the systems themselves had very low knowledge spread out across Xero, of all the employees as well. There was very low confidence about the success of this project, because of the complexity and because of the low knowledge. Stakeholders did not have a great deal of confidence that we were going to be able to deliver successfully on time.

[00:00:47] And we had the biggest challenge of all, which was to release this whole group of changes with zero downtime. Xero has about two and a half million small business customers spread out all around the world, all in different time zones, which was a great challenge, to be able to deliver a project of this scale and to be able to do it with zero downtime and no impact to customers.

[00:01:06] So we had this hypothesis to start with. We have a very highly complex project, a very highly complex piece of work. Could we break this project down into small chunks to reduce risk, to deliver it to production in small increments, to be able to measure feedback and get feedback quickly? Could we do this? What would it look like? And how would we grow?

Hiring champions and building a new culture

[00:01:34] Where did we start? We first of all started with hiring some key champions. These people, these engineers, had done continuous delivery in previous projects to this one, so they had a lot of experience on what worked and what didn't. And there were some key openings within our team that we could hire those people into. And we could use those people to influence others around them, to teach them and to grow them and to show them how continuous delivery worked, and how it would reduce risk for the whole project. So they started to influence a smaller group of people around them, and then influenced a bigger team as time went on.

[00:02:11] Now, there were some challenges though. If you bring in new people who have a lot of different types of experiences, they are going to clash, if you like, with the people who are already there. You have people coming from two different worlds. You have people who have not had any experience with continuous delivery and people who do. How do you reconcile the two? How do you help them to work together? How do you help them resolve differences? How do you help them reconcile?

[00:02:34] What we did to start with was to build a new culture within the team. How do we do that? We started with some core values and practices and principles in the team. Now, these are borrowed from ThoughtWorks. They're not designed by us, but they're the ones that we borrowed for the team.

[00:02:50] So we started with some core values, which are those four in the middle there: clean code, fast feedback, simplicity and repeatability. And then around the outside we have a bunch of practices and principles for the team, that would kind of be a contract within the team to say, this is how we will build software together, this is how it will behave, these are the kinds of practices we will do regularly as a team. So we would split work into vertical slices. We would have a simple design. We would have collective code ownership, pair programming, refactoring, test driven design, continuous integration, and automation of repetitive tasks.

Fast feedback in the developer tooling

[00:03:28] Now, I wanted to touch on a particular example which covers fast feedback, just to give you a bit of a glimpse of what we did and how some of the systems worked. Not only did we want fast feedback for customers and fast feedback to learn knowledge about the system and how things worked, but we also valued fast feedback in terms of the developer tooling experience.

[00:03:50] This is an example of how we did it. We started with the principle that we should have the same tests, the same kind of checks and balances that we have in our CI environment. We should have that available 100% replicated on a developer's local environment. So what we did was, whatever we created for CI in terms of integration tests and unit tests, and how we scripted that, we made that script available as a pre-check-in for developers to run before they push their code to master.

[00:04:26] That meant that they had the feedback relatively quickly, instead of waiting for CI to spin up agents and start a build. They could run it directly in their local environment. They could get the confidence that they needed to then be able to ship and push their code to master.

[00:04:40] Now, one important element we had after the pre-check-in script: once the commit was made and the push was made, we had full, a hundred percent automated, so continuous deployment, from the push all the way through to a deployment and verification in production.

[00:04:55] The other thing we did which helped us enable this was to use containerized environments. We used scripting, of course, quite heavily, but we also had containerized environments all the way through to setting up dependent services and integration environments and integration level testing all across. And that would run on the developers' local machines.

Metrics: deployment frequency, lead time and cycle time

[00:05:18] The other part that we looked to do was to track metrics. We thought this was important, to track where we are now, where we want to be and how we're tracking towards the target. We wanted to make changes in very small and fast increments.

[00:05:32] The Accelerate book, which you may have heard of or read before — I recommend reading it if you haven't — talks about best practices across the industry in terms of DevOps practices. And they have four key metrics that they encourage teams to measure. We particularly focused on two of them. One was the deployment frequency: how many deployments we would make per day. We hit a target in one team of 20 deployments a day, which is great. We averaged around that number. And we had a delivery lead time of 12 minutes.

[00:06:05] Now, the way that we defined that delivery lead time — I've noticed across teams that delivery lead time can sometimes be interpreted differently, so I'll clarify how we actually measured it. What we did was measure it from a developer commit into master to a production release, and we got that down to 12 minutes, which is great.

[00:06:28] I want to talk just a little bit about cycle time. When I talk to engineers or groups of engineers, sometimes, if we discuss the concept of cycle time, it's regarded as the bottom rung of metrics to track, and they question how much value there is. There can be dependencies that are outside of your control which can affect cycle time. If you're dependent on another team to get some work finished, then maybe it's not a measure of a high performing team. I do agree with that concept and that sentiment. But there are some important things that can come about from measuring cycle time and having targets to reduce cycle time.

[00:07:07] In this particular team that I was working with, they had a cycle time of around four days. Actually, when I first started the conversation with them, they weren't even measuring cycle time at all, but I encouraged them to start to measure it. So they came up with the number of four days. Okay, that was their average.

[00:07:19] I gave them a challenge to say, what would it mean if you brought cycle time down to one day? Which is quite an aggressive target. For me personally, if I'm an engineer in a team, I would love to have one day as a target cycle time. I think if you're an engineer in a team and you're able to ship code at the end of the day, straight to production, you feel good about the work you're doing, you feel good about the achievements you've made. It ends your day with actual software in production, which is a great outcome.

[00:07:52] So I laid the challenge to them: what would it look like if you went to one day? Now, one day was a little bit aggressive for them, so we negotiated to two days. What would it look like for two days?

Working in a legacy monolith, and releasing more often

[00:08:01] Now, this particular team had some challenges, because the work that they were doing required them to work mainly within a legacy code base which is in a monolith, which was shared by a number of different teams. That meant that your releases — because of the age of the code and the kind of practices that were done around the code in the early stages or during the life of the monolith, there wasn't a great setup or culture of automation, of testing, so a lot of it was manual.

[00:08:30] The team had this challenge, they were working mostly in this space. One element of working in these kinds of areas is that you need to kind of schedule releases and coordinate with a whole bunch of other teams, because multiple teams are making changes to this at once, but you generally only want to have your changes be released at one time, because you want to reduce risk and not have other things actually being released at the same time. Otherwise you have things that are unknown, and if things go wrong it can be very difficult to fix.

[00:08:59] But in the challenge to get to two days for cycle time, the team had some very important discussions within the team around what would that look like. How would they actually structure their work? How would they break down their work even more? It introduced a new level of discipline in the team.

[00:09:13] What they ended up doing was, they were able to shift their release timeframes and have them more often. A lot of teams were kind of booking around a four week timeframe between releases, and that means that what you're deploying can be substantial. You've got a whole team's work over four weeks being deployed at once. So they booked releases that were, say, three, four days, five days apart, around a working week. And that meant that their changes were a lot smaller. It meant the risk went down. They spent less time actually testing, which is great, and they were able to get faster feedback. They were able to get changes into production faster.

[00:09:49] And generally speaking for the team, culturally, it increased the feeling of flow within the team, which helped them view themselves as a higher performing team, making improvements, being able to make changes and kind of feel that happening in the day-to-day work that they were doing, which is a great outcome.

Contracts with other teams

[00:10:04] So what about working with other teams? Because we can go for continuous delivery, and in an ideal world you have a team isolated, they can be fully autonomous, they can do whatever they like, but in reality you probably have some constraints. One of them may be that you have other teams that are working at a different cadence, so they're not ready to go to continuous delivery. And if you have to collaborate closely with them, how do you reconcile that? We had that exact challenge with the project that we were working on.

[00:10:34] For starters, if you have dependencies, that in itself can carry risk, particularly around delivery timeframes. So if a team that you're dependent on takes longer than expected, that can actually flow on to all the dependencies that are downstream from that team and push out timeframes. And things taking a little bit longer is quite common across software delivery, as we know. So that can be a big issue.

[00:10:57] The other challenge we have is a high risk release situation. If you have multiple teams hitting a target all at once, or if you have releases with a bit of a big bang approach for when you're actually launching into production, if you have multiple teams making deployments at the same time, or changing more than one thing in production at the same time, all those things add to higher risk.

[00:11:24] Could we have a situation where teams run in parallel? They all create their own delivery targets, which are communicated to everyone else, but they're able to run their own game. You can get rid of dependencies altogether, which would be wonderful. Well, we couldn't do it 100% in the ideal sense, but we were able to do it somewhat successfully using contracts.

[00:11:42] So we were able, early up front in the project's life cycle, to start developing contracts to be sent out to other dependent teams to give them notice. And we also included some documentation to say, this is how the system will behave, this is how the contract will look, this is the kind of data you can expect. We tried to give teams enough confidence that they could start building, with the confidence that there weren't going to be major changes coming down the line. They can start their build and they can start to do things autonomously in parallel with other teams, and not so much be fully dependent upon our work going from start to finish before they could start.

[00:12:27] Now, developing these contracts ended up being more difficult than we thought. We had representatives from teams all across Xero. The project we were doing was across a number of domains, and we were all in rooms together thrashing out what the contract should look like. When you're talking about multiple domains, you're talking about teams and talking about history across Xero. Who works on what, who owns what, was a particular thing that we had to decide upon. If we're going to define these domains and define them well, and how they interact with each other, how would the ownership model go? What belongs in what domain? All these were challenges.

[00:13:07] One of the biggest challenges we had — I've got a question there of "what is a feature". That is one example to show that phrases and terms, or what you call something, can mean different things across domains. It can have different contexts, teams can interpret things differently. If you call something what you understand a feature to be, but another team understands it to be something else, that can be really difficult to reconcile. We had some of those challenges, and we invested a lot of time to be able to work through these kinds of issues, to try and communicate through them, to try and come to an understanding of how the systems would interact.

[00:13:44] We also had some challenges around things like changing behavior for the better, for the better future of Xero. One of those examples was moving from synchronous API calls into asynchronous behavior. When you do those kinds of changes, they have big flow on effects on both sides of the contract. Systems, or the teams that build the systems, have inherent assumptions of how systems are going to behave, how they're going to interact with them, what they can expect to get back and in what kind of timeframe. When you're shifting to an asynchronous model, those things can move around quite a lot. So that required some extra level of communication to be able to explain things to teams: how things were going to change, what they could expect from systems as this got deployed into production.

Feature flags and customer segmentation

[00:14:33] I want to talk to you about feature flags. Feature flags are commonly used across continuous delivery processes, but we used them to the next level, I think. So I wanted to cover exactly what that looked like to give you an idea of the depth that we went to.

[00:14:48] Here's a bit of an example of how we segmented customers. This is very simplified, but we looked at what a customer is, what metadata, or how you interpret the different types of customers and how we can treat them differently. Some examples here: we talked about global regions, where they exist around the world for Xero, whether they're new or existing subscribers, whether they're on a free trial, whether they're fully signed up, and what source they came from, so upstream systems that came into our system. So that was one view of how customer segmentation worked.

[00:15:22] Now, functionally breaking up things and feature flagging. This is how it's commonly used. Here's a bit of an example. We have on the left-hand side some channels, and that kind of describes upstream systems from the platform we're building. We have a monolith and microservices, and these are downstream from our platform. But all of these things are affected by the feature flags that we have on or off. And in the green I've just kind of highlighted one execution path here.

[00:15:47] So we used feature flags to be able to enable certain functional slices to be active in production at any one time. And that meant that we could limit the amount of code paths that are actually being executed in production, which reduced risk. And we could do this on the fly, so as soon as a transaction from a customer hit the system at the upstream level, everything downstream would follow the feature flags according to what we had set, and we had teams monitoring it and making sure that things were working as expected.

Running legacy and new platforms in parallel

[00:16:19] One thing that we did do very early on in the project, we came up with this idea to have things running in parallel, code execution running in parallel. We had the legacy system and the new platform running in parallel, and we designed it intentionally so that we could run them in parallel in production without having any adverse impact. That meant that we could compare apples to apples in production and compare the data output of both code paths, just to be able to confirm that we hadn't missed any edge cases, because our knowledge of the systems and the edge cases was quite low. We wanted to just make sure that things worked as expected. Sometimes when we received new types of metadata on customers our code would need to respond in different ways, so we wanted to verify and make sure that our code was doing what it was supposed to do.

[00:17:05] One thing that we did do with this was, we were actually able to flick it on and off quite quickly, just with one change on a feature flag. We were able to direct customer requests and transactions through the system into the new platform or the legacy platform. Now, I said we were running side-by-side, which was true, but when we started to gain more and more confidence and we started to deploy our systems to take over production, we would do it on a transaction by transaction basis. And that greatly reduced the risk.

[00:17:35] So instead of just switching it on and leaving it on for days at a time or weeks at a time and kind of measuring, what we would do is switch it on and then wait for a customer transaction to happen, and as soon as we saw one we'd switch it back to the legacy platform. That meant that we had a real live transaction, just to be able to go in and immediately verify: did it do what we expected it to do? And that enabled us to gain more and more confidence about what we were building, that it was working correctly, which was a big win in terms of being able to show the confidence, be confident in ourselves, be able to put this into production without adverse impacts for customers. Doing it this way was a big win.

Going live with the smallest thing

[00:18:18] We also had a big discussion and we ended up deciding about how we would go live and what that would look like. We challenged ourselves to go live with the very smallest thing that we could. What was the smallest thing we could do which had end to end value? What could we go live with, and when could we go live with it? And we aimed to get it live as soon as possible.

[00:18:38] Now, that brought about a bit of a discussion amongst the team, which is, what does that look like in terms of production support? What does it look like in terms of being on-call, or monitoring production, or anything around that nature, even customer support issues that would come through and disrupt the team if things went wrong?

[00:18:58] That was a valid concern. What we decided to do was to go forward and do it anyway. Part of the reason was just due to the amount of unknowns we had, due to the amount of risk we were carrying, due to the amount of confidence we started with. We really needed to have a win on the board. We needed to have something live in production that was working, that was actually working for customers, to solve some of those issues around unknowns and confidence.

[00:19:24] And that ended up being a great success. The big impact on the team didn't really happen. We did have some issues that we had to attend to and that we had to jump on, but actually what happened was that just having some of those issues in production was also a win, because the engineers actually got used to what their system behaved like in production, with a very small amount of customer transactions. So very low impact. We did have a couple of issues that we needed to jump on. They weren't critical, but there were things that we had to adjust and tweak along the way. So having that feedback from real customer transactions was a massive value win for the team.

Architecture: adapters, queues and eventual consistency

[00:20:02] Now, your architecture is something that is affected by continuous delivery. I want to just touch briefly on some of the things that we did that worked well. I've illustrated here on the left we have upstream systems, on the right we have downstream systems, and everything in between is what the teams would be building.

[00:20:16] Now, you would think that we're building a domain, everything starts with the domain API, but in actual fact our system actually started with a group of adapters. The reason why is because it enabled us to decouple ourselves from legacy systems, first of all, and it enabled us, wherever there were upstream systems that maybe were lagging in change, or maybe couldn't fulfill some of the contract expectations that we had — so things like asynchronous behavior, things of that nature, where we couldn't get change in time, or we had mismatches in domains — we were able to use an adapter layer, and we had roughly one for each upstream system, to be able to translate into the domain API if needed.

[00:21:06] If we were able to document and influence and line up priorities with other upstream systems and the teams that were building them, then they could come straight to the domain API, and that was all fine, a big win. But we used the adapters as temporary placeholders, to be able to have a system that meant that we kept the domain API pure, if you like.

[00:21:30] Then we had our internal domain, which was just modeled there. And then a bit of an example of how we used async: we used queues heavily so that we could have async behavior, and we could then make things quite fast. But one of the challenges we had was that we moved to a model that was eventually consistent, basically, and that was a bit of a change. So we had to do a lot of discussion with upstream and downstream systems, because upstream systems originally, in the old system, were using synchronous calls, where everything would go through and they would get high confidence back that yes, it's consistent, yes, it's been done, you can go ahead with the next stage.

[00:22:08] When we moved to async, that created some issues. We had to actually collaborate across upstream and downstream, because we were looking at feeding off the downstream, getting a response back and then being able to send that back to an upstream to say it had been done. So that was a real challenge.

[00:22:23] But one of the advantages of building an architecture in this way was mostly about using the different components: it gave us a lot of flexibility. We did have a situation where one of our upstream systems was experiencing a delay. That wasn't a huge delay, but it was enough to impact customer experience. What ended up happening was, it was just a combination of two different things within our domain, of our architecture we'd built, that weren't quite interacting together properly. So we were actually able to swap out one of those components with something else, and the impact was quite low. We did it within a sprint and everything was good.

[00:22:57] One of the main challenges really was for the team to figure out, how do we measure this for the future? How do we make sure that other upstream systems get the speed that they need, that we have confidence in the service level that we're delivering? So that actually took more discussion than fixing the architecture.

Reflections: what worked and what we'd change

[00:23:17] So what was the outcome? Overall it was a great success. Now I wanted to talk a little bit about some of the reflections. Just thinking back about this project: what went well, what didn't go well, maybe what we would change next time.

[00:23:32] First of all, a decoupled architecture, which was a huge win. We actually spent a lot of time going through the architecture, a lot of talking about edge cases, what's going to work, what's not going to work. Dealing with a highly complex system required us to put a lot of work up front into architecture, but the work that we did had a huge payoff and really enabled the system to scale quite easily once it was fully delivered, which is great.

[00:23:54] Feature flagging, and the depth that we went to, was also amazing. It really enabled us to go to such a fine grained level, to switch things on and off and be able to measure. It was a huge win and enabled us to really take very, very small steps in production. But every time we did it, we increased our confidence. We could bring out a new segmentation of customer into production and then verify that it was good and be able to communicate that out and go, and now we're at the next stage. We're able to do not only this region but now this region. That was a big win for the team.

[00:24:30] And going live to production early and often was also a huge win. I don't think we could've really easily anticipated what the production support level would have been, so it could have been a loss. It ended up being a huge win. Our assumptions and our thinking about what kind of impact that would be for the team, and whether they would be distracted, or whether the time would be spent on production support issues, that ended up not happening. And so we gained a lot of the benefits without much of the projected costs of the things that could've gone wrong, which was great.

[00:25:03] So how would we improve for next time? I think looking back over the project, we immediately started delivering and started shifting the teams into a continuous delivery framework. I think looking back at that, maybe it would have been better to negotiate with stakeholders and executives and people around the team to say, hey, maybe we should take a couple of months just to embed these new kinds of practices in the team, to teach engineers, to show them how it works, to realize some of the benefits, before we went on to actually deliver. I think trying to deliver and make the change at the same time added an element of stress for the team.

[00:25:42] It would have been great to be able to teach the why more. In some examples, when we were doing a lot of work in architecture, sometimes we had to say to the engineering teams, look, this is how we're doing it and this is what we're doing. But to teach the why, and teach some of the principles as to what the thinking was behind it, was a lot more difficult, I think because we were executing at the same time. I think we felt time poor, so we weren't able to teach the why more, but it would have been great, I think, to actually go back and do that as more of a learning opportunity for the engineers that were working on the project.

[00:26:17] And I think we could have over-communicated even more than we did. We did communicate a lot. We wrote a lot of documentation, we had a lot of meetings, we had a lot of Slack comms. But I think we could have communicated even more above and beyond that, and sometimes that would have eliminated some of the challenges that we had in collaborating with other teams. This whole project was like 40 different teams in play at once, so there was a lot of moving parts. I think over-communication would have been a great thing to introduce, just to up it to the next level.

[00:26:46] So I hope you enjoyed that. I hope you got a lot out of it. I hope that this showed an example of how we reduced risk via continuous delivery. It was great. Thank you.