Continuous Delivery... The Real Deal

Quentin Berder at UXDX Community: Berlin. Video: https://www.youtube.com/watch?v=9G1Gmuc57mQ

Readable transcript: edited from the recording's captions for readability (fillers and false starts removed, punctuation and section headings added). Wording is the speaker's own. Timestamps are positions in the video. Names marked [?] could not be verified against the audio.

Context: building at Starbucks scale

[00:00:10] Today we're going to speak about the real deal about continuous delivery. I know not all of you are technical, but I guess a lot of you are hearing this buzzword everywhere. What I'm going to walk you through is our experience and what we took out of it. I'm trying not to be too nerdy, too geeky, but if you want to speak about a bit more detail, come to see me after. This is what we're going to get through: I'm going to present a bit, do a bit of marketing for ourselves first, and then I'm going to move to a little nerdy phase. I'm going to explain what CI is, and I'm going to speak about the culture around that.

[00:00:49] Let's get into it. For some context, we have been very lucky to work with a lot of multinational companies, ranging from retail to pure tech, in China, Europe and in the US. What does that mean? It means that we are building big, big infrastructures. When you are building an omnichannel presence for Starbucks in China, you don't address two hundred thousand users daily. You address 20 million active users daily, which means that you need to make sure that you can go at high velocity. I don't really see Starbucks waiting for you three months because you want to use the latest tech. They want to be able to get something out there in five weeks instead of five months, and we will see how continuous delivery can help with that.

[00:01:49] This is me. I'm the chief ninja officer. I basically shovel a lot of the crap that I can to make sure that we are shipping on time and we stay alive. We all have all-nighters to be able to get through that, and I'm the one helping the team with that. So let's get started.

What continuous delivery means

[00:02:10] What is the concept of continuous delivery? The main concept is really to be able to go fast, to test very quickly, and to have that in a continuous manner. It means that this is not something that's going to happen at the end. This is not a little dot in your Gantt chart. It's something that has to happen on a daily basis, on a commit basis, on everything that you do. For us, we can say that it's really part of, and really important for, the culture of a company. Let's nerd out a bit.

The stack

[00:02:49] This is our stack. These are all of the tools that we use that can be relevant on a daily basis to continuous delivery. I'm going to go through them. I'm sorry, I'm going to be a bit nerdy there; for some people it can be interesting, for others boring. But you will see all of these tools bring a little facet to the continuous delivery that you are trying to achieve.

[00:03:14] First, we're using Docker. Docker abstracts runtime environment complexity. Basically you forget about having all of your database and everything; you put them in a container, and you see that it's going to work on different environments. I'm putting that roughly, so don't take me too literally on that point. Some people will say this is not exactly that. I know, I know, I'm just trying to make it a bit high level. We also use Kubernetes sometimes, but most of the time it's overkill. You need to have huge fleets of containers to be able to do it. I don't know a lot of companies that have the volume to do that. I think Nike is the only one I know that really uses it properly.

[00:03:58] Next, we are using Ansible. Ansible allows us to manage the configurations and orchestrations below and above Docker, meaning we can, for example, ship together the whole system configuration, ship the network operations, and on top we can have some orchestration. I said it was a bit geeky. Don't take me on that; we're going to get to the good part after.

[00:04:26] Pipeline. Pipeline is something that we created internally because we hate Jenkins. It's amazing, we don't have one developer internally who doesn't hate it. I don't know if some of you already use Jenkins, but my god. OK, it's nice, it does a lot of things, but it does too many things, and you need three developers just managing Jenkins. So we built this thing that is based on Python, and basically what it allows you to do is just deploy some playbooks. You click on one button and you do it, and that's pretty much it. Simple interface. As you can see, everything is very flat, surfacing in YAML. It tracks everything, so you have logs everywhere, so you can see who did the click that broke production. Well, we know who it is. We also defer the authentication to GitHub, so anybody in the team can just come into it if they are part of the current repository.

[00:05:18] A little example on that. Our head of QA, actually earlier, saw that there were some commits that we pushed but were not yet deployed on staging, so she was blocked on her test. She went over there, and she doesn't know Ansible, she doesn't know Docker or anything; she just clicked on play, she deployed, and she could go on with it. It's also open source, so if you want to go over there and just grab it and play with it, please be my guest.

[00:05:53] Afterwards, we have everything that is related to QA and QC. We are using Travis CI to have continuous integration. We are using SonarQube to be able to do a smell test, static analysis of our code. What is the last one? We have Sentry.io, which is pretty awesome. It gives you contextualized continuous logs, so when something is crashing in your app, be it on the device or on the browser, you're going to receive a proper log with what happened. It's really good to be able to go and debug problems in production very fast.

[00:06:30] We also have continuous logging and continuous monitoring with ELK, Elasticsearch, Logstash and Kibana, that allows us to grab all of the logs and make them digestible and searchable. With the TICK stack as well, Telegraf, InfluxDB, Chronograf and Grafana, Kapacitor sorry, that allow us to get all of the metrics continuously and be able to see when there is something bad happening, or there is an outage in production. Being able to debug again is, for us, being able to go very quickly with it.

Transparency and short feedback loops

[00:07:05] The result of that, and this will be the last slide that will be a bit geeky, then we're going to go to the part that is a bit more interesting, I would say. The result of that is that basically we have access to everything at every moment. This is what we could call a mature continuous delivery culture. At any moment anybody can see what is happening. For example, if you are using Travis CI, your commit is going to run your tests, and you're going to have feedback in your GitHub ticket. When we deploy with Pipeline or with Ansible, we receive some feedback with it, so we see if there is an issue.

[00:07:49] The idea behind that is to be able to really, really minimize the feedback loop. It's not to wait to see something, but to see it directly at the beginning. You usually have a lot of problems; you're going to see many, many problems, but these problems are going to be very, very small and digestible, so you're going to be able to tackle them way more quickly than if you wait until the last moment to do it. And that's at every level, be it in the code quality, be it in your tests, be it in UI tests even, be it at the configuration level, deployments. All of that needs to be known fast.

[00:08:27] And the best way to do it is to have everyone aware, because perhaps the DevOps guy launched something, it's going to take five or ten minutes, he's doing something else and he forgets about it. That happens. If you make it transparent, everybody will be comfortable with it. It's not going to be just the DevOps guy or the front-end developer or the back-end developer that's going to be accountable for it; it's going to be the team as a whole. So you are starting to create a culture of transparency and self-reliance.

Best practices matter more than tools

[00:08:55] On top of that, all of this doesn't come from one day to the other. It's not, "Oh cool, I saw all of these nice tools, I'm going to do the same, and tomorrow we're going to kick ass." It's not going to happen, because you need to be able to enforce that on your team, on the different people, and you need for that to have best practices. Best practices are way more important than tools. If tomorrow you want to use some other tool, I don't care, just use it, but use it properly. Use it for a purpose. Don't try to just put layers and layers of tools on top of it to make it look cool, or to try to achieve something that you yourself are not achieving.

[00:09:35] It also goes for the same, you had two comments about UX designers and developers. It's the same: you need to have culture through that, being able to understand them and what exactly they want to do. We do that by having our Zeplin and InVision directly in it, so we see all of the little changes that they are doing, we see when they are uploading the assets, we see the time that they are moving. We also have a centralized, we do everything in GitHub. We had Jira. I'm sorry, but Jira is the same as Jenkins, it's awful. But anyway, let's stop the geeking.

[00:10:13] What actually matters from all of that is that you need to track everything and you need to automate everything. We are nerds, we are lazy. If I can write a script, and it's going to take me three days to make sure that I don't have to do something again, I'm going to do it. Seriously, I'm going to do it, because I don't like to do things and repeat them. And that's also going to be helpful, because the next time around I'm going to be able to focus on another activity instead of focusing on something that is already done.

[00:10:45] And we need to optimize and release continuously. You need to be able to rely on having releases very fast and make sure that these releases will work. Everybody working on products knows what it is. You are trying to have a release, and someone says, "Oh yeah, but the back end is not ready," or "these things were not tested." And then your project manager comes back and says, "You see this Gantt chart that we did four months ago? You were supposed to do the QA at this time." And you're like, "Well yeah, but the assets were late." And then you go into a loop of discussions for something that should not have happened if you had a very, very small feedback loop.

[00:11:27] And resilience: being able to see things when they happen, you will not have an issue at the end. The goal with that is really to develop resilience and to help surface defects early. It will always hit the fan, but the only thing that matters is that you know it as soon as possible. If you see that the code quality of somebody is not good, well, you tell him, "Your code sucks," and I can tell you here why: because we see that it doesn't get through the CI, we see it doesn't get through the smell test. This is how it is.

Why you should care: culture

[00:12:05] Anyway, why should you care? You should care because culture is important. This is what you need to have. What we're trying to bring, at least for me as product manager, as project manager, as UX developer, as front-end and back-end developer, is innovation. Being able to do the same animations in iOS that you did because you saw it in the Airbnb app, come on. Yeah, cool, but your project manager asked you to do it and so on. That is because you don't have the space to innovate. To compete, you need to be able to have assumptions and to assess them quickly, so we need to go to market very, very quickly.

[00:12:54] On top of that, as I said, tech should not be an obstacle, but should help you. The tech does not really matter in the end; you can use what you want. What matters is that you have this continuous delivery. That is the goal, but to achieve that you need the culture. Why is culture so hard? Why can we not just go to PwC tomorrow and say, "Hey guys, I'm going to build your app"? Well, because we're going to have frictions. It's not possible. We're going to have problems. We can have problems with marketing, we can have red tape to the left and right.

The stages organizations go through

[00:13:27] So before I speak about a few of the clients that we had, let me run through a few of the stages that we usually see companies and organizations go through. Step zero: you come to a company, you have no product culture. They don't know what product is; digital product does not exist. You only have an IT team and a marketing team. Your IT team has no documentation whatsoever, monolithic stuff, they don't care about tests. They're just like, "Well, you know, I have Java and I really like it. If you want, you can work on this Windows VM to be able to access a server." Cool, we're going to be able to develop properly with that.

[00:14:05] Then you have marketing. Marketing is like, "When do you guys launch? Cool, I'll see you at this date." That's pretty much it. Or they come back at the last moment with "this is what we want." They don't really want to invest the time; they are interested just in the D-day, and the rest they don't really care about. The challenge here is to be able to create the space for the digital product team to sprout and start to flourish and be able to reach some innovation. This is the grinding that we see a lot. It's very, very difficult.

[00:14:36] Then the next step, when you start to have a bit more space, is when you start to have IT that is fighting changes. They go between being nice and defending the status quo. They're still going to come back with, "Well, you know, this thing doesn't come into our stack," or "it has nothing to do with the legacy API that has been there for 15 years, and nobody knows how it works, but it works, don't touch it." We saw that so many times.

[00:15:01] And then you have marketing that will be, "Cool, yeah, I understand the narrative." Perhaps you're going to have one or two guys that know that 96 [?] and say, "Yeah, I want a 96 [?] for my launch." They will see the nightlies, they will see the sprints, but they will not really understand the narrative in the full lifeline of the product. They are still not fully invested. They don't try to bind what they are doing to their business KPIs. Here the problem is just that you need to stay alive. You are starting to grow, but you still need resilience. You don't want people to steal some of your budget, you don't want people to steal some of your elements, you don't want people to just move your timeline because there is a Jenny from fun tech [?] that doesn't like it. So staying alive is the most important.

[00:15:49] Then you have the awesome ones. These are the places where IT knows its position. They're going to be there, they're going to support you. They will say, "Yeah, it's a tech we don't know, but it doesn't matter. Here is a server, have fun. This is how we do things." They're going to support you on your monitoring, they're going to support you on the upgrade of the servers and so on. You're going to have marketing that comes in and says, "This is what we understand, this is our business KPI. How do we make it fit with your analytics? How can you make sure that what the digital product team is going to build fits in our full understanding of what we want to achieve during the year?" This is really the ideal, because you have support left and right and you can create something that is interesting. But here the only problem is not to die under the sheer weight of the organization, and to keep most of the politics at bay.

Story time: three clients

[00:16:46] Story time. These are the last three clients. I'm not going to be too harsh with them, because I want to keep them, but this is what happened. We went to Walmart. We arrived: no documentation, nothing, no code. I saw a guy pushing some updates to production through FTP. Come on, this is not 1995. I did that for websites, please. It was pure HTML, very monolithic things. Marketing really didn't understand what we were trying to do, but just wanted to say, "OK, we're going to launch, cool, you do the stuff behind." We really just saw them two days before the launch.

[00:17:26] Starbucks: they have, and they still have, a digital team that is striving to get good and to have some talent. The big problem is that we had an IT that was sometimes nice, sometimes more on the status quo, and we had some marketing that didn't know anything about analytics, which is not so good to understand what users are doing there.

[00:17:48] Then we have Nike. Well, I was just blown away. I came in and everything was already ready. I just arrived in the room, and the guy was like, "This is your PM, this is the manager, this is the guy that's going to help you with IT, this guy is doing marketing. Here's all the documentation. Have fun for the next three days just reading through that." Which was great, I loved it. But there is still all of the political red tape that we have to get through: "Well, you know, don't get too close to that, don't do that too much." And sometimes they just come with some politics that does not matter for us. We just want to deliver. We just don't want to be in front of the camera. We don't care; we just want to be able to ship something that we know is good.

What it takes

[00:18:35] So what matters to achieve that is that you need to have champions at the C-level or in upper management. You need to have people that can vouch for you at the higher level, that understand digital product and marketing. You need to be ready for growing pains. When you go into something and you try to build it, you're always going to have some issues. It's never going to be easy. Never going to be easy.

[00:18:56] You also have to do things slowly, one step at a time. There is no reason to just arrive and try to change everything at the same time. It never works, never works, even less in big organizations. You can see it even in your team with two or three people: when you try to bring changes, it's the most difficult. You can change tools; culture is difficult. And you have to establish clear rules for the main partners. You really have to have IT, marketing and digital know where each sits and know where they can work together. The goal is to create a viable digital product team to lead to innovation. If you don't have that, you're not going to be able to innovate enough.

Data-driven and user-centric

[00:19:36] So let's wrap it up. Continuous delivery is a facet of a data-driven and user-centric culture. This is a lot of buzzwords, right? I'm just going to expand on them and how we see them. Data, for example: for us, we do everything from data. Be it monitoring, we review the logs, we review the metrics; be it for analytics. When I launch something, everything is already in my 986 in left rabbit I oh [?]. I review the crashes, everything, to make sure that my assumptions were correct.

[00:20:04] On one thing, I was very happy about it, and it didn't work. For example, for Starbucks we launched one of the payments. I made some awesome animations and so on, and nobody used it. I could see I had 1.5% of the users going through it. Wow, that is super shitty; I was feeling bad about it. And then somebody came with an idea: "Let's do a page that we can share, with a little cup, and you can move the cup and it starts moving in it." I was like, "OK, well, let's do that on the side." We did that in three days. It was really easy, very fun, and my god, we had this surge of people using it. Why did it work? Because we went to market quickly, we looked at the data, and we saw, yeah, it works, so let's keep on doing these things, even if it goes against my ego and I'm wrong. I know it.

[00:20:50] And user-centric, because we are trying to reach the users as soon as possible. MVP, POC, production: you need to go as soon as possible. It doesn't mean you have to go with things broken, but if you have your culture of continuous delivery, of all of these little organic changes that you make, you are sure of the next steps you are taking. Try to think about it as a scientific method. Try to think about it in an organic way. Your culture, your continuous delivery, is the same as nature. You need to make small changes, small tweaks as you go, and you're going to have a lot of failures along the way. We are here today because we have had huge numbers of failures since the inception of life. It's the same ecosystem, the same setting. We are trying to have a product that starts from nothing; you need to go through small steps and have a very quick feedback loop of what works and what doesn't.

Just start

[00:21:45] All of these aspects can really be wrapped into continuous delivery. As I said, a quick feedback loop is the most important. But one of the biggest problems, and I'm going to leave it on that: people always say, "It's cool, you are telling us all of that, but what comes first, the culture, the tools, the processes? How do I start?" Well, I don't know. Just start. This is the chicken and egg problem. You need to start somewhere, because you cannot validate without success. You cannot validate by saying this culture is the best, but without the tools you cannot achieve that either. And when you achieve that, you can look at your tools and say they are bad. So you just have to start somewhere. There is no other way. Just keep in mind that you're going to fail, but fail very quickly, very easily, with a very small feedback loop, and it's going to work. Thanks.