Enabling And Empowering A Continuous Delivery Culture

Ciaran O'Connell at UXDX Europe. Video: https://youtu.be/5_Q9NXsLLTc

Readable transcript: edited from the recording's captions for readability (fillers and false starts removed, punctuation and section headings added). Wording is the speaker's own. Timestamps are positions in the video. Names marked [?] could not be verified against the audio.

From zero to continuous delivery

[00:00:05] Thank you, Olivia, and thank you everyone. I know some of you are still having your lunch; it's a lunch break pusher [?], so thank you for attending this. In the context of this presentation, enabling and empowering a continuous delivery culture, I want to talk about the principles in the context of the journey that my own engineering teams have undertaken in the last three years, which for us has been going from zero to continuous delivery. In doing so, I'll talk a little bit about the technology part we've taken, the challenges we've overcome during that, and then, crucially, also the cultural challenges we've undertaken, and the mind shift in our engineering teams to go from a monolithic to a DevOps culture.

[00:00:54] I'll talk about the challenges we've taken on, and challenges we still have. But we've had such good success over the last three years that I think it's a good story to tell. I know our problems aren't unique, and some of the solutions and principles that you'll hear may be familiar as well, but hopefully you'll find the decisions and the journey we've taken and the patterns we've undertaken useful.

Who HMH is

[00:01:21] Very briefly, who we are. A lot of people may not have heard of HMH, because while we have huge numbers of users, as Olivia said, we're targeted at the education industry, right down to the student and teacher in the classroom, with 13,000 districts, three million teachers and 50 million students. Nearly all of our market is in the US, but actually our technology arm, and the biggest technology arm, is here in Dublin, up here on Paris Street [?]. We have 250, 130 [?] engineers working on the flagship learning platform, which I'll touch on briefly.

[00:01:55] That learning platform is providing the teacher and student experience and all these capabilities, as well as providing teachers with the data platform we're pushing up to them, to be able to personalize instruction to individual students or groups of students, and the students themselves to take personalized learning, get recommendations and ultimately get better outcomes. So there's a lot going on there. There's a lot of technology there. Where we are at the moment is quite cutting edge. We're sitting on a microservices platform, a collection of services with big data pipelines and content pipelines being pushed in. Those services are communicating with asynchronous protocols and running in Docker containers, and they're fronting a pretty modern application using React, and other applications as well, an Angular application.

[00:02:54] So a lot has happened for us in the last three years to get where we are now. Where we came from is important, I think, because we pride ourselves as a company: we're actually 185 years old. We started as a pure publishing company, with books such as, for example, The Lord of the Rings, published by HMH, right up to our friend here that maybe young kids might be familiar with, Curious George, which is a children's book. So up until three or four years ago, the left-hand side was books, digital content, multiple platforms. The right-hand side is, as I've described, the learning platform. So we in engineering, the engineering group, had a lot to do in three years to come from where we were, from great content to great outcomes.

Moving to the cloud and hitting bottlenecks

[00:03:39] Our journey started with monolithic applications. I know Nick earlier on spoke a little bit about moving to cloud, and we were very much monolithic applications. We knew we needed to innovate to meet the demands of our increasing user base, but also to innovate on the new platform. We did a lot of good things initially. We moved to the cloud, we moved to AWS, we created a pretty standard stack that could scale and could deploy automatically. So we had all the structures in place on the cloud, but we hit a very big problem: we hit bottlenecks in terms of being able to spin up environments.

[00:04:30] We were a lot of engineering teams, 15 or 16 scrum teams and more, trying to create services, and we had a centralized operational team that would spin up the environments for us. In a company like ours, with four thousand [?] employees, maybe 10% of it is engineering, and that would be resistant to change. Ultimately that led to a lot of bottlenecks, and that in turn led to a bit of latency in how we were evolving. Within the engineering groups there were a lot of handoffs: it's somebody else's problem. So even though we'd moved to the cloud, we still had a lot of those problems of different environments, testing being a different group, and all of that. We knew we had problems, and we almost hit a wall with the business demands we had.

Five principles

[00:05:17] We really had to say to ourselves, "Look, we need to take a step back." So the engineering leadership took a step back and said: what are our principles? What are the principles of a continuous delivery environment? What are our principles for DevOps? It starts with controlling your infrastructure. When I say control, it's the engineering teams here, the engineers, having control of their infrastructure: spinning up an environment and controlling how that environment is working, whether it's in the cloud or not, controlling that environment completely within the engineering structure, in terms of how you're allocating resources and how a cluster might work on top of the cloud. Having full control of your infrastructure programmatically is going to accelerate your continuous delivery, and we knew we needed to do that.

[00:06:04] Secondly, controlling your deployment. It could take months to go down through the environments to deploy something. We wanted this to happen instantly. We have to be able to deploy with a huge amount of confidence and reliability through our pipeline, and we have to be able to do it quickly. We have to roll back quickly. Ideally, and I think Nick mentioned it earlier in a previous presentation, canary deployments; you can do it at different levels in production. But from our perspective, controlling your deployments is a must for the engineering team.

[00:06:39] Third, control your applications or your services. On the learning platform we have multiple services and front-end applications. What I mean by control is that preferably these applications, even if they run in a larger environment, will be self-contained in some way. They can be containerized, have a bounded context, be observable, be monitorable, and have all the right quality checks in and around them. That's an absolute must for a continuous delivery culture. We needed to create the infrastructure, the DevOps [?] tools and software, so that applications can be controlled by engineering teams the way we want them to be in this culture.

[00:07:26] The next is to trust your system. That's the real part: when we are at the point where we now have a pipeline and are able to deploy an application, we have a system that we can trust. When we push software into it as a group, multiple teams, there are enough quality checks, enough happening within that system, that we'd be able to take it to the next environment with confidence, because that's the only way you're building in capability [?]. So when you already have the first three steps, you have to trust your system.

[00:07:58] The last part is controlling your destiny. That's as much cultural as anything else, and it can be one of the most difficult things to solve. That's really giving teams empowerment, giving understanding within engineering teams of what the definition of done is, understanding what the quality of service is, knowing within the teams, and empowering them, what they're delivering on, and taking ownership of that.

A skunkworks team and making the right thing easy

[00:08:20] So we listed those five principles as an engineering group that we wanted to follow. It's easy to do that, but how do we put it into practice? There are different patterns you can take, but in our case we did have to be quite draconian. We sent off a team of five to six engineers, with engineering backing, as a skunkworks project, which is the one on the right. Five or six very, very good engineers went off to start building the platform as a service. I'll talk about it in a minute, but they started building the infrastructure as code and the environment and tooling [?]. Once they got to a place where they had a cluster and the right tooling and capabilities on top of AWS for our applications to start using it, we went for broke.

[00:09:15] We took the team of five or six people, and they came over and worked on one of our most problematic services. It was a pretty tough few months, because from an engineering perspective they were starting to work off a completely different stack, completely different ways of working, where it's no longer someone else's problem to provision infrastructure or control your applications. It's now becoming: you as an engineer need to understand the DevOps. There are a lot of obstacles, plus there's a lot of integration: will this work, will this perform, will it scale, all of those things as well. And we had success. We got it over the line, a big success.

[00:09:55] But in funny ways that was just a starting point for us. It wasn't enough for us to go, "We've delivered that, so every other team should do the same." We were now hitting the cultural and the engineering challenges. There's upskilling, huge upskilling, for engineers to take these sorts of actions themselves, to fully understand their service and how to pull it onto the platform. And then secondly, how culturally the teams become self-empowered, with true quality through to production. Everything in the whole cycle of the continuous delivery structure, going to production, is their ownership. It's not somebody else's ownership. QA is not somewhere else. It all falls under the remit of an engineer. So in effect we were no longer using an operational team. Everything was being built back into the teams. That was their challenge.

[00:10:49] So we used a line there: deliberately create and deliver tools to make the right thing the easy thing. When you cannot make the right thing easy, you must make the wrong thing increasingly difficult. That's the pragmatic way you have to approach it. It's not enough to have governance say, "Use our platform." It has to be that the tooling that's been created as part of this platform, the quality controls that we're all bringing in, the automated tests, all of the stuff that's been pulled in, is attempting to make the right thing the easy thing to do. And if not, it has to be tough: the difficult course of action for engineers and engineering teams has to be impossible to do, because we need to have a safe platform. Ultimately we're still servicing three million users and fifty million students.

[00:11:33] So this was our approach, and it worked. It worked over time, to the point where today we're at full continuous delivery. I'll talk a little bit about where we are, but that was the approach we took: almost the draconian way to set it up, but then a pragmatic and behavioral way of how it gets pushed into teams.

The platform: infrastructure as code and containers

[00:11:58] Briefly, for the technology set, here is how it was formed. For infrastructure as code, with the goodness I mentioned, control, transparency and predictability, we built it on top of a Mesos cluster [?]. That allows us to program instances and pools of resources. It gave applications the ability to share pools of resources on these EC2 instances. There's a side effect: having this level of control can also be very cost effective. I think the saying is that cloud providers can be a little bit like banks: if you don't have a plan for your money, they'll create a plan for you. So for us, having full control, especially in the market we're in, the education industry, we wanted full control programmatically of how we manage our instances.

[00:12:48] Within that, we wanted to build containerized capabilities for multiple applications. We brought in Jenkins as our CI process to be able to build Docker images and deploy them as individual, isolated containers sharing kernel resources. Also, if we wanted to bring in new application software, for example a new database, MySQL or a Postgres database, we're using Terraform to provision it. So engineers can do a pull request and provision the infrastructure, just like that. We got huge amounts of scaling, horizontal scaling; we benefited hugely from that. But also we're now able to satisfy our principles of controlling infrastructure and controlling deployments, and making applications containerized, which means we can start putting tests in and run them isolated. They can run tests within the container, and they can be portable across environments, as with Docker.

[00:13:52] Additionally, we would have pushed in Aurora and schedulers to run these containers and keep them running forever. The applications could use Aurora scripts to configure how the application will work on the platform, and with Aurora this will keep those services running forever. So we don't have downtime. We have degradation, but our services will come back up if they're having glitches.

The pipeline: test automation, speed and stability

[00:14:23] So the key point: when we had our technology stack from the platform, the next focus for us was shifting to the pipeline. The pipeline is key, your continuous integration pipeline. It's great having auto-deployment capabilities you can run quickly, but there's no point deploying rapidly if code is going in there, getting to the next stage and failing, and we're back again, with all the means to create infrastructure and all the rest. So it starts, as we know, with test automation. Test automation is absolutely key, especially in a very integrated environment like we might have in our learning platform. With a front-end application you could have 20 or 25 services, lots and lots of instances, services with different capabilities, and they all have to come together in an experience that has to work 100% within the classroom, fully reliable.

[00:15:27] Test automation starts with unit tests, but then automated tests all the way to functional testing right from the front end. For the front end there are great tools out there. You can use WebDriver or Protractor, depending on what type of application you have; we're using Angular and React. At the API level, you can use BrowserStack for browser testing. For API testing we'll use gut lean [?], and we use Supertest pretty heavily. We'll use Gatling [?] for performance testing. The point of all this is there are lots and lots of tools, and a lot of you will be familiar with them, but they all must run as part of this pipeline. They have to be part of your automation. It's not enough just to have unit tests. Automation has to be built in; automation is key.

[00:16:14] The way we solve it is that quality is owned by the team. Quality of that service is owned by the team. The definition of done for our service would include full test automation. We may have quality engineers, and they may be doing some automation, but ultimately they're there to make sure the practice is influenced and the definition of done is followed. We expect the engineers to be writing, or helping to write, these automation tests as well. So it's very much owned by the team.

[00:16:44] Similarly, speed and stability. We need to deploy quickly, fail fast, get fast feedback, strive for quick and stable execution, so that practically every merge can go into production. We'll do a pull request and a huge amount of tests will get executed straight away. We're containerized in our tests, so nothing will get through to the next environment without the whole execution of automation going through. So we have full confidence that we can get to the next environment. You may have other integrations; in our case there are a lot of content integrations that might need to be tackled at that point. But we have full confidence moving between environments with the amount of automation we're doing, and practically every merge can go into production quickly.

[00:17:27] We've had great experience with that. We can do production deployments every day, even with quite a complex platform. We did six in one day a couple of months ago, with a huge, huge deadline that we had, lots of updates and changes. We've had huge success being able to move at speed with stability by really just following this process.

Monitoring, security and data-driven decisions

[00:17:48] Monitoring is extremely important, especially with a microservices architecture. You have lots and lots of discrete services. Every service needs to be observable, every service needs to be monitorable, and needs to be owned by the team, because you're releasing to production. The engineers are owning this right through. Even in an organization as big as ours, we don't expect a support team or someone else to take these things independently. It's the engineers' ownership when these things go to production. So your service must be highly observable. Runscope is a common monitoring tool; we have it built into our operational platform, and then into our applications we have Elasticsearch, Kibana [?] and all of these things that make the service fully observable. We can set it up for PagerDuty alerts or text messages from Runscope if the service degrades. We rarely have a service that goes down, but it could be an issue of degrading, or whatever needs to be looked at.

[00:18:40] Security: really important that you shift left on security. You don't want these things to happen in a continuous delivery environment where a security team takes a look at your application at the end. There's so much modern stuff out there and all the rest, but it shouldn't be happening at the end of any development cycle. You want it to be built into your pipeline. There are some great tools out there, like Checkmarx, force first 50 [?], Gertie Chan's [?], JFrog Xray, where you can do whitelists of the libraries that go in. Those scans need to happen in your pipeline. The more you can do in your pipeline, the more naturally it happens, and all of this trust is built in, rather than expecting someone to go off and write a test or whatever. So shifting security left is really, really important.

[00:19:29] Results: test automation reports. That's something you need as part of it, of course. Everything needs to be exposed as reports, to find bugs faster, highly visible to teams. And the last part for us is quite a big thing: data-driven decisions. You need metrics. A big part of it is that every part of your service, in terms of CPU usage, all of that needs to be communicated. We store a lot of that stuff in InfluxDB, and we're able to visualize all of it in Grafana. But equally, with Google Analytics, you want feedback. If you've got a product owner as part of your team, you want feedback on how your software is being used in the continuous delivery culture, and you want it quickly. You should be able to see straight away how your customers are using your software; it shouldn't be waiting for someone.

[00:20:19] In our case, we spend a lot of time investigating students' behavior in our software, because it's a big part of the whole cycle for us to be able to deliver outcomes. So we would track user behaviors, and we would track user experience through Google Analytics. All of those things are really important for data-driven decisions, because that way you can go back in again and say, "Well, that's not being used," and all of that.

The mindset changes

[00:20:44] So if those are the tools, what are the mindset changes that need to be made? The philosophy is really, really important, and it's ongoing. It is an ongoing challenge for us all of the time, because we shifted the culture in the space of two years, but we still have challenges. Empowerment and engagement: we talk a lot about it. You need to give empowerment to the teams. Engineering teams, product teams, everyone wants empowerment, to be able to have control of how they deliver their software, all within the principles of agile development. We constantly use the word empowerment, making sure the teams are comfortable with what they sign up to. Teams own what they deliver. Teams understand what quality is and understand what it means for a service to be deployed.

[00:21:34] Especially when you've got multiple teams all coming together for the same application, you need to have everyone signing up to ownership and accountability for what they're delivering, so that it's not somebody else's problem. That's a very, very key thing if you want to deliver quickly. Accountability and transparency: any tooling you have. In our company we use Slack quite a lot for engineering, and we'd like to ensure that things get pushed into Slack channels, builds, if anything fails, so people know pretty quickly when something is wrong in our system. The engineering teams will probably know faster. We try to improve our engineering teams to make sure that everyone is transparent about what they do, and there's nothing being hidden, all of those sorts of things.

[00:22:18] From a management perspective, over to the right, it is important to set expectations of the quality of service and the definition of done. You do not want it to be completely 100% "well, the team decided this." You have to have some level of expectations, because that's how you can have agreement on a definition of done. There might be deviances between the definition of done in some teams, but if one team is saying, "Security is not important to us, and we're going to bring in some third-party library or whatever," and that's going to fail further down, well, that's going to be a problem. So you do need to set expectations of the quality of service.

[00:22:59] We do a lot of that through working groups, engineer to engineer. We try to empower that in the teams rather than from management. There might be some level of boundaries that we would put in place, but the teams talk to each other and they say, "Well, what's your definition? What are the agreed principles that we have in our definition of done? What can get to production? If some tests failed, are we all saying there's not a chance that could go to production?" All of those kinds of rules. That's important, because if we're going to give teams the power to deliver to production, they need to have the expectation of the quality of service they're delivering.

[00:23:40] And finally, the culture of celebrating success: fail fast and learn. Really important for innovation, really important for continuous delivery. We're all going to celebrate success, but sometimes we do need to try stuff pretty quickly. With the microservices architecture especially, we've tried certain technology stacks and said, "Well, that didn't work particularly well," and that can be okay as long as we fail fast and learn from the experience quickly. So we try to instill that culture. You'll move quicker, you'll move pragmatically, and we as a group will feel as if we're innovating faster, from giving a little bit more power to engineers to try something out, so long as it doesn't affect the platform. Failing is fast with all of the tools that I mentioned in place; you will fail fast. But I think that's the culture: it should be okay, as long as we learn through our cycles. So that's all I had. I wanted to leave a few minutes for questions.