Supporting The Product Through Continuous Delivery

Frédéric Rivain at UXDX Europe. Video: https://youtu.be/V5AvHSxsPVU

Readable transcript: edited from the recording's captions for readability (fillers and false starts removed, punctuation and section headings added). Wording is the speaker's own. Timestamps are positions in the video. Names marked [?] could not be verified against the audio.

The good: a launch day with no scrambling

[00:00:00] Okay, can you hear me? Cool. Who knows about Dashlane, raise your hand. Not that many. Who uses Dashlane? Even fewer. Okay, hopefully by the end of this talk you will want to try Dashlane, and of course I have a little promo code for you at the end if you manage to stick to the end of the talk. Today I wanted to talk about continuous delivery, but not the straight technical stuff, because that is specific to everybody. I want to show you continuous delivery from the angle of product development, with the good, the bad and the ugly that you can see as a little picture over there.

[00:00:42] First, Dashlane is a password manager. We have a lot of applications, on iOS, Android, Windows, macOS and so on, to help you store your credentials in a simple and secure way, just to give you a bit of context. But I'll start with a date: July 26. Can you see it? No. Here it is, July 26 2018. That date probably doesn't mean anything to you, but it actually matters to me, because this is the day we launched Dashlane 6.

[00:01:08] This was our last mega release, packed with new features for our users, a lot of cool stuff, supporting an important marketing campaign. And on that day, from an engineering standpoint, we pressed the button in our back office and almost instantaneously all those cool new features that we had built became available to our millions of Dashlane users. So there was no rush, no scrambling at the last minute to make sure that everything was ready in time. There was just the fact that code had been shipped progressively in the past weeks and months, hidden, and we had even had the time to test it with our beta users, to A/B test in time what needed to be A/B tested, so we could have a smooth D-day.

[00:01:52] For me that's really what I would call the good of continuous delivery: really see it as an enabler for the business, for the product, for the marketing, and get rid of the stress of the engineering team shipping things at the last minute.

What continuous delivery requires

[00:02:06] But let me give you a bit of a refresher on continuous delivery, just to make sure that we're all talking about the same thing. This is my definition. Continuous delivery is the ability to deliver in production fast, of course, reliably, on demand, and this requires an industrialized, automated release pipeline to make it happen.

[00:02:29] There are three main prerequisites for continuous delivery. The first one is high performing code practices. Very important. At the core of it you must have a centralized code repository. You need to orchestrate everything through continuous integration, and that's really the core of your engineering system, so it needs to be very strong and you need to invest a lot into continuous integration. And obviously you need code quality, so you need tooling and you need systematic, mandatory code review. Essentially, one of the key metrics here is how fast can a line of code go through your pipeline, and it should ideally be in minutes and not hours or days, and without of course skipping steps, because otherwise you're cheating.

[00:03:13] So it's about tooling, automated processes, but it's also about reaction time from the engineering team. One of the key points here is code review. A healthy quality practice is very important for an engineering group. It's important for its efficiency, but it's also very important for its morale. Think about it: if your developers don't really align on the expectation between them about how they do code review, you will have a lot of waiting time. One developer will write some code, submit a pull request, and then will be waiting for his peers to review the code. So it's very important that they organize, gather and synchronize, so that code review becomes a very active, top priority activity and you can maintain a smooth flow to production.

[00:04:00] I really want to emphasize the fact that code review is very important, but you need your team to be very active about it. At the end of the day, every day, you should have the developer saying, okay, I have, I don't know, call it half my time coding, but I've also done code review half my time. I removed the timestamps, but that's one of our Android branching systems, and in fact it was a very, very short timestamp on this one.

Continuous QA and the test pyramid

[00:04:28] Let's look at continuous QA and I'll show you full-stack testing. You might know this pyramid as the pyramid of tests. You have of course unit tests at the bottom. They are the foundation for your testing strategy. You need integration tests to validate that your modules and components can work smoothly together. Then you need functional tests that will do all the regression testing. And at the top you have all the manual testing that is done for exploratory testing, edge case scenarios and so on.

[00:04:58] As you move up the pyramid you need to automate as much as you can. So unit testing should be fully automated, high code coverage, that's very important, and you should really take the time to make it happen. Integration tests a bit less so. And as you go up the pyramid of tests you need to find the right balance between automating, which is costly, and the stability of the scenarios, which change all the time. Test automation is a heavy investment, but I really encourage you to invest into it. There is a strong return on investment on quality and stability, and for me you cannot achieve continuous delivery in the right way without having a really, really strong QA strategy.

[00:05:36] When I joined Dashlane a few years back, we had very inconsistent practices, we had very limited test automation, and we needed a lot of time to make it happen. We trained everybody, developers, QA. We hired dedicated test automation engineers to build and maintain the test frameworks on all platforms. And even today, after a few years investing into it, I consider obviously that we are better than we were a few years back, but this keeps being an area of effort. We need to keep maintaining and evolving a test framework, and as we go we are also able to extend the test automation to more areas such as performance testing, security testing, load testing and so on.

Continuous operation

[00:06:17] Continuous operation is the third one. Once you've shipped your code, production is not over. You must set up continuous operation, and I will emphasize a few points here. The first one is the monitoring platform. You need to monitor everything. It needs to be as exhaustive as possible. You need to have technical metrics, and you need to have product metrics, business metrics, everything. But of course, if you generate that amount of data you cannot track everything individually, so you also need an alert system that will inform you if something goes wrong. The goal here is that you are aware in advance if something is going wrong, so that you fix it before your users realize that something went wrong.

[00:06:58] And it is not as easy as it seems. If you don't find the right threshold of alerting you will get lots of false positives, and then your team will start not paying attention to the alerts anymore because there is too much noise, and then the point is moot and you need to start again. So make sure you balance the right way to put your threshold in your alerting system.

[00:07:20] A good practice that we have at Dashlane is what we call the morning medic[?]. The goal here is that in each team someone every day is in charge of proactively checking the dashboards, looking at the metrics, looking at the release pipeline, and anticipating issues and warning the team about those. The goal of course here again is to fix issues before they hit customer support or they hit your users. This is one of our Kibana dashboards. Everyone can have their own tool, but it's really important to have the monitoring system in place, the alerting system in place, and the practice that goes with it.

[00:07:52] Dashlane is a pretty complex platform. I was mentioning the fact that we had a lot of different applications, and to maintain this, from the back end with our infrastructure and our API or semantic engine[?], to iOS, Android, macOS, Windows, they're all specific and all very different, and all the web clients. You can imagine that it is very complex to maintain all those release pipelines. We have about eight technical platforms and we need to invest in all of them, because they're all specific and with their own tech. But we think that it's worth the money, it's worth the cost, it's worth the effort to really invest in continuous delivery.

Three tools: progressive rollout, feature flipping, A/B tests

[00:08:27] Now this is the basics. You have your continuous delivery pipe, but it's just a pipe. You have code flowing through it smoothly and you need to open the tap at the end of it. So what do you need to make it go to production? For me you need three tools. The first one is what we call progressive rollout. Most tech platforms today allow you to do progressive rollout, which means you open the tap slowly, get some code to production, you make sure everything goes okay. So you ship very progressively, 1%, 5%, 20% and so on, and at each step you monitor: are things going okay? Am I getting wrong metrics, crashes and stuff? Am I getting tickets, an incident from the user support? If not, then I can continue. If yes, I stop and I check what's happening.

[00:09:14] Just as an example, we had one recent release where we had shipped the code to one percent of our users and then we started getting a lot of crashes and spikes in metrics. Those ones had gone unnoticed because actually they were related to a certificate that is generated by a pool[?] in production, so of course we could only see it in production. But at least we found the issue with only 1% of users being impacted and we could find the source and how to fix it, and then restart the rollout in confidence. The thing had really impacted a reduced number of users, and in practice it never hit customer support, so that was really a good thing.

[00:09:49] A second tool that's very important for continuous delivery is feature flipping. Feature flipping is the ability to control from the back office the activation of features to a segment of users. At Dashlane we built our own tool internally, and it's in the hands of our product managers. They have that very fancy back-office UI that has been built by engineers, but at least they can decide what feature flips they need, on which segment of users to activate it, when it goes to production. Essentially they're the ones pressing the button saying, okay, now I need that feature to be available to our beta users, I need that feature to be available to only English-speaking users, and so on. They have a lot of variables they can play with.

[00:10:34] This is what we used for Dashlane 6 that I was mentioning before. Everything had been feature flipped, hidden in production, just activated to certain users so that we could make sure that everything was going okay. And on the D-day, because we wanted to synchronize everything with that big marketing campaign, we could press the button in the back office and everything was active.

[00:10:55] One little warning about feature flipping. This is code, and with code comes the technical debt. So once you've released to production you need to be careful to clean up the feature flipping code, otherwise, very slowly, one after the other, you will get a lot of clutter in your codebase, and that's usually something that the engineers forget about. So in our own feature flipping mechanism we've put warnings and alerts about the fact that the feature has been 100% released and it might be time to consider removing the feature flipping code.

[00:11:27] Another warning about feature flipping is that this is a tool that you should use for deployment, to control the way you deploy and control the way you give access to your features. It should not be used as a configuration mechanism. If you want your application to be configured differently for a certain population, build the configuration mechanism in your application to make it happen. Same thing, we had teams at Dashlane in the past that started using the feature flipping tool to do A/B tests. Don't. That's not an A/B test engine, that's the feature flipping mechanism. If you need A/B tests, use an A/B test engine, a real one and not this one.

[00:11:59] So we do have an A/B test engine at Dashlane. Actually we have two of them. We have one which is a third-party tool that's being used by the marketing team to just test copy and design on our website, and then we built an internal one, lower level, embedded as an API into all the applications so that we could test all the components of our applications. Obviously A/B tests are very important to experiment in production, and you're probably all familiar with the A/B test concept, so I'm not going to go into details. I will just give a bit of experience and warning about A/B test practices.

[00:12:33] The first thing is that of course they need to be synchronized with your progressive rollout and your feature flipping, because they go afterwards. If your code is not in production, you shouldn't start an A/B test. If the feature is not available to the users, you should not start an A/B test. Also, when you start accelerating and having a lot of different teams A/B testing, be careful to synchronize, to have the teams talk together, and make sure ideally that they test separate parts of the application so that you avoid biases in the results. Like always with every tool, it's how you use them that will matter, so you need to be careful not to overuse A/B tests and avoid results that conflict with each other. I had put a nice comic about A/B tests and the multiverse.

The bad: communication failure

[00:13:17] Okay, three tools: progressive rollout, feature flipping, A/B test engine. Now you're all set to do continuous delivery. We've talked about more the good aspects of continuous delivery, how it enables your product, how it gives you the ability to accelerate, but of course there is the bad and the ugly.

[00:13:36] The first one is about the bad, the communication failure. Let me tell you a bit of a story. Earlier this year in March we released an update to our VPN feature. The goal here was that our end users could choose a specific country they wanted to do the VPN from. The feature had run smoothly through the different phases, it was in production, everything was fine and so on, and the marketing team launched a campaign to announce the new feature to all our customers. And then we started getting hundreds and hundreds of tickets in customer support of users complaining that the feature was not available to them. Actually what had happened is that the macOS version of it had been stuck in the Apple validation and never hit production, so all the macOS users were complaining that of course they couldn't see the feature.

[00:14:16] So it's really critical that all the teams, from engineering down to marketing, are all aware of that continuous delivery mechanism, they all know the release life cycle and they synchronize. Of course it gives you a lot of control and flexibility and the ability to do stuff in an easy way, but you need to remember that if you don't time it the right way, then you get into the situation that I was describing about our Mac users not having their feature. So I would even argue that the more you accelerate, the more you should also communicate between the teams, and that's usually a bit complicated within the organization. Yeah, I'm talking, talking, talking, I forgot to click on the thing. But yeah, you see my point: talk together.

The ugly: a big pipe with the wrong things in it

[00:15:06] The second difficulty with continuous delivery is, as always, you ship, you have the ability to ship, you have that enabler that allows you to push a lot of code through the pipe, and that's great. The engineering teams, they feel happy that they can put their code out easily, whether it's new features, experiments, big projects, hot fixes and stuff. But as always, it's not the volume of things that you put in production that counts. What really counts is that we push the right things in production. And so it puts even more burden and even more challenge on the product team and the data team and the marketing team to define what are the right things to put in production.

[00:15:45] You have an intense throughput of delivery, but this is just an enabler. It's not what is important at the end of the day. You just removed a bottleneck. So really think about focusing on, not continuous delivery which is just an enabler, but how do you maximize the outcome of the things you put in production. I like this little diagram from Henrik Kniberg from Crisp in Sweden about maximizing value and not really the output.

[00:16:12] The risk with continuous delivery is that you have a big truck now, you've been able to come from a smaller truck, or a small pipe to a big pipe where you can push a lot of stuff, but what's very important is to put the right thing into the pipe. I think there were a few previous talks today about, if you put crap in you will get crap out. That's kind of the same. You have a big pipe, but it's still important to put the right thing in there.

[00:16:40] We had that issue previously at Dashlane, where we had teams that were so happy that they could release a lot of stuff that they were shipping a lot of different experiments. But at the end of the day, the inputs and insights that we were using to ship those experiments were not necessarily the right ones. So we were producing a lot, but not producing effectively. We were experimenting a lot, but not experimenting in places where we had confidence that they would bring results. So at the end we said, okay, let's slow down a little, let's go back to the basics, let's rethink the way we build stuff, let's go back to doing more prototyping, more design thinking, so that at least what we implement, what we code as an engineering team, is more meaningful and gets a better quality at the end of the day.

Takeaways

[00:17:28] So that's essentially the four takeaways. Number one, invest in continuous delivery. If you're not doing continuous delivery today, you should. It's of course costly, it's going to take a lot of time. I think that at Dashlane it was about two to three years in the making to really get to a maturity level where I consider we are more or less doing continuous delivery. So a lot of time and a lot of effort doing the three prerequisites that I was mentioning. But it will help you accelerate your business, it will really be a bottleneck that gets removed, so you should do it.

[00:18:07] And the practices that I've mentioned: the first one, high performing code, very important. You cannot consider continuous delivery if you have not put the right foundation in place around the code and your engineering practices. Strong continuous QA, a strong test automation framework. And strong operations, that you monitor what's happening in production with the flow of things that will go to production. If you're not ready there, you will really hurt.

[00:18:28] And you need the tools. Those tools, you can either use third-party tools that exist on the market. For progressive rollout, most of the platforms, iOS, Android, support progressive rollout directly in the Play Store and in the Apple Store. Chrome extension, if you're in a Chrome extension, you have it as well in the Chrome store. Otherwise you build your own mechanism of progressive rollout. Feature flipping: there are lots of third-party tools that allow you to do it. We decided to do our own because we wanted it to be shared between all our platforms, and low-level and also more secure anyhow, but that was our own choice. And the A/B test engine, same thing: either you use a third-party tool or you build it internally. It's not that complicated to build actually, so I would encourage you to build it yourself. That way you have the right tool for you.

[00:19:14] As you accelerate, very important to make sure you double down on communication. I cannot insist on that too much. It's really important, otherwise you get a lot of synchronization issues, you will have a lot of code going out without tracking. So make sure that product, marketing, design, the product managers, they all talk together to make sure that they know what they're doing and what's happening in production for your users.

[00:19:37] And last but not least, continuous delivery is just about the plumbing. You need liquid gold flowing through it, otherwise it's just pointless. It's a lot of effort and investment for nothing. So don't just do the technical part of it, also think about the organization and how you build your product and engineering organization as a whole, to make sure you maximize the outcome. And now, if you want to test it yourself, here's a little promo code, UXDX2019-test-dashlane, and you can experience yourself if we're doing it right or not. Thank you very much.