Scaling A CI Infrastructure

Laurent Ploix at UXDX Community: Stockholm. Video: https://youtu.be/Z70I77jmKpg

Readable transcript: edited from the recording's captions for readability (fillers and false starts removed, punctuation and section headings added). Wording is the speaker's own. Timestamps are positions in the video. Names marked [?] could not be verified against the audio.

Finding good metrics for continuous integration at scale

[00:00:05] My name is Laurent Ploix, I'm a product manager at Spotify, and today I will dig into the problem of finding good metrics for continuous integration at scale. In order to do this, first I will dig into what it means to have metrics for internal products, products that are used by your own employees. The second thing I'm going to look into is what it means to grow CI when the company grows really, really fast. And the third aspect, then I will reconcile both things and say, okay, what does that mean for CI to be successful and bridging the gap? And we'll see that good metrics to apply were invented in the 50s.

[00:00:46] Let's get into that. First thing, about metrics for internal products. The first thing you need to have, of course, is you want to be able to monitor the situation that you have: how fast people go, how much your system is used, how much capacity do you need, do you have running costs that are too high, and so on and so forth.

[00:01:10] The second reason you want to have metrics for internal products is the impact of what you do, and there's an important point there. Some people would say, well, if you can make a change in an internal product that's going to make things better, for instance for CI making builds faster or more stable or whatever, just do it. Well, actually that's not true. There is a cost to everything you do, and if it costs a million dollars to save a few minutes of runs for continuous integration, maybe it doesn't make sense. So you want to be able to measure the actual impact of what you do compared to the cost of it, so you know if you should keep it or if you should kill it.

[00:01:50] And then the third part of having good metrics is you can have unintended consequences of whatever you change. In CI we had a case where we made the system a lot faster to build and the consequence was actually longer queues. And you say, why? The reasons were because people were using the system a lot more, and as a result there would be more queues in the system. Whenever there's a human in the loop you can have interesting impacts of whatever change you make.

Why internal products are hard to measure

[00:02:26] The third thing: there are challenges with internal products. The first thing is you don't have big data. For instance, you have a certain number of developers or users, you have a number of changes per day that happen in your development, maybe it's the number of times your continuous integration to test the system is actually working fine, and so on and so forth. But if you look into it, you're going to talk about dozens, hundreds, or maybe thousands, or maybe a hundred thousand things per day, but it's actually small numbers most of the time.

[00:03:02] As a result, it can be hard to run A/B tests internally, and there are several aspects to that. The first one is, from a statistical point of view, your p-values are going to be pretty high. You will have to wait a long time before your p-value is going to be like 0.05, for instance. The second thing is that it's socially challenging to run an A/B test. Are you ready to, for instance, slow down half your company on purpose to see the impact of a slower slash faster compiler?

[00:03:37] The third thing is that it can be challenging to tell someone, well, you're actually part of an experiment. People don't want to be part of an experiment, and of course they want to get the best system you have, they don't want to wait. But in order to measure a bit as well, you actually need to have an A and a B and compare them.

[00:03:59] Then, interpreting data can be hard. It can be noisy, noisy as in there is a lot of standard deviation. And in a typical environment like Spotify, where the environment changes really fast, the teams move on quickly from project to project, many things change at the same time, and as a result it's extremely hard to interpret a change in the metric.

[00:04:27] And the last thing is it's very easy to confuse metrics and goals. You can have a metric to measure, say, the number of changes per developer per day or something like that, but if you start to transform that into a goal, people will game it. You can give me any sort of metric you want, any of them. If you start measuring people with this metric, people will game it, and as soon as it's a goal it's not a good metric anymore.

[00:04:54] Last but not least, it's very easy to fall into the trap of vanity metrics. For instance, for continuous integration, how many builds are run per day? That's an example of, yeah, you can be proud of being able to run dozens of thousands of them, but does that produce value for your company? It's very easy for internal products to not look at the value but look at some technical validity metrics.

How CI load grows with the company

[00:05:22] So that was for internal product challenges. Now, focusing more on CI, continuous integration. Basically this product is about your developer. You make a change on any system, and before you deploy it, of course, you need to test it, and so you're going to ask continuous integration to run a number of tests.

[00:05:47] Here are a few challenges. The first one is the load on the CI environment is not nice to look at when it comes to a company that grows fast. In fact, it grows with the code size. I mean, it takes longer to build a bigger piece of code. But also with the number of tests, the number of developers, and by the way the number of developers will create more code and more tests, the number of platforms you support, and the percentage of flakiness. Because if tests return green or red at random, people will stop trusting them and they will rerun the test many times. So this actually grows really fast, and if you go from ten to a hundred to a thousand to a few thousand people using the system, it actually is really, really hard to keep under control.

[00:06:44] And now let's look into the CI metrics. Who cares about them, and what matters really? If you look at the CI infrastructure team, the one who manage the CI environment, so most people will use Jenkins or similar things like TeamCity or others, and the machines around them, what they care about is the peak capacity, how many machines do you need, how much maintenance is it, how much volume of data do you create, what you call artifacts, network load, and so on and so forth.

[00:07:26] It's kind of technical. It's also, as I said, like vanity metrics sometimes. You can be proud to say that you run a few dozen thousand builds per day, that you create many, many artifacts, that you have a very low technical failure rate and so on. Of course, like the durations, the queues that people experience and so on. Those ones are kind of technical metrics.

[00:07:49] But if you look at the other side, the people who actually use CI, the developers themselves who want to create features for the product, what they care about is how fast they can go. And they care about how long does that take me from my first commit into when it reaches master, or when it reaches production, that's even better. Or how many changes can developers do per day, this type of metric.

[00:08:13] So you see, there's a tension between technical low-level metrics and more focused-on-efficiency, productivity stuff. And this is healthy, because it needs to be a conversation. There needs to be an agreement between those two types of metrics, and that is actually usually very, very hard. Now, I'm talking about CI, it's actually true for any sort of internal products. You have the users, you have the maintainers, and those two typically don't quite look at the same metrics.

Mean time to repair, and measuring friction instead of productivity

[00:08:50] So how do we bridge the gap? We're going to see that there's a good metric we can look into, which was, as I said, invented in the 50s. What is a useful metric? The useful metric is one that makes sense to the users, in my case the developers who run stuff on CI. It also makes sense to maintainers, so they need to be able to have the same language. The metric needs to be stable enough that over time, if there is not too much change, the metric should basically be stable. It needs to be actionable in the sense that you know what you can do to change it or you want to impact it. And when I say proven track record, it means that if you can backfill this metric from data from the last two years, you should be able to see the changes in an easy way.

[00:09:46] This is where it's really important. You need to go from this notion of productivity into actually the exact opposite. What is truly important is actually not the productivity of people but what friction they experience, how much bottleneck do they face.

[00:10:07] And that's where I'd like to introduce you to what they call time to repair, or MTTR, the mean time to repair, to recover. The interesting bit here is, you're a developer, you make a change in the code, you run continuous integration saying, okay, I want to test my code, I want to make sure it works everywhere, I want to make sure it's stable and so on. And you get what you call a red: you get a failure of some sort, some test fails, some compilation failure, anything. And then comes a time when you need to fix it, and you will make a few changes in your code, you will resubmit it, maybe get another red, and so on, and so you will iterate on that.

[00:10:51] What happens is that it's going to take you time, and that's okay. You're going to experience queues because the system is overloaded, maybe. And then you're going to have the duration of the build or testing. There may be some flakiness in the system, so you're going to have to retry things. And sometimes you get confusing error reporting, so you don't really understand what's going wrong, and that's going to take you time to dig into the code, go into the stack traces and so on and so forth. And this time, this time from the first red to the first green, is what we call time to repair. And this is a truly important thing.

Reading the distribution of time to repair

[00:11:30] So now we're going to look into the distribution thereof. What you see here on the left side of the big bumps, the fast fixes, is that most people can fix problems really fast, but sometimes it takes a lot longer. Sorry, I removed the axis, but you get the point. Some fixes, it's going to take minutes, some take a lot longer than that.

[00:11:55] The point here is that you will see this curve by measuring all the TTR on a given project over, say, a month or two. And sometimes, for certain types of failures, you're going to see a curve like this, which means that for certain types of failures, if you filter out the other ones, you're going to see that, yep, most of the time it's really, really fast to fix, but actually people spend a lot of time for, say, half or two thirds of those failures in fixing them. And you have a problem here, typically to report failures to your developers in a nice and easy way to fix.

[00:12:40] I would like to show you one specific graph. We made a lot of improvements on the system, typically at the end of last year, and you can see that the curve in 2019 is much more to the right than 2020, which means that most of the problems today are fixed a lot faster than they used to be.

[00:13:02] So this is a distribution, and the distribution is what matters. You see it's not a normal curve, which means that you cannot really look into either the mean nor the median in a statistically meaningful way. But if you have both of them, it actually tells you a story. The mean is typically going to reflect the entire distribution, especially as there's a long tail, where the median is going to be faster. And by looking at both of them you can get an idea of how much better you get over time. These are actual numbers here.

[00:13:39] This is interesting because this MTTR notion was actually invented, as I said, the first mention I could find was slightly after the last war, but maybe it came from before it. It's a very efficient way to measure friction or problems in industry, and actually it worked really well for software as well. It measures friction, it captures all aspects, the human ones, the technical ones, and it makes sense to everyone.

[00:14:13] As a conclusion, now I'd like to mention that MTTR actually should not be taken alone, but I focused on this one for this presentation. You also want to look at what we call MTBF, which means mean time between failures, so basically how many failures do you face. By looking into both mean time to repair and mean time between failures, you can have a pretty decent idea of how efficient your CI environment is, and you can have a pretty actionable metric that both developers and the CI maintainers understand. With that, I'd like to mention that Spotify hires a lot, and you're welcome to ping me on LinkedIn. Thank you very much.