Streamlining Project Handovers And Takeovers

Maksim Ekimovskii at UXDX Community: Helsinki. Video: https://youtu.be/OZ0QWwkukJc

Readable transcript: edited from the recording's captions for readability (fillers and false starts removed, punctuation and section headings added). Wording is the speaker's own. Timestamps are positions in the video. Names marked [?] could not be verified against the audio.

Who we are and why we are talking about handovers

[00:00:03] Maksim: My name is Maksim, a software engineer at Zalando.

[00:00:07] Matthieu: Yes, and I'm Matthieu, I'm engineering lead also at Zalando.

[00:00:12] Maksim: Today we are going to talk about how to streamline project takeovers and handovers, and how it works at Zalando scale. But first I would like to say why we are going to talk about that. During recent months we had quite extensive experience of taking over quite key projects in Zalando, which had a great impact on our commercial events like Cyber Week and Black Friday, and we would like to share our experience with you, how we do it in the tech teams.

[00:00:54] Maksim: But before, let's talk about Zalando, just a few words. Some of you, I guess, know Zalando as an online fashion store where you can go and buy clothes, but in fact we are also a tech company, so there is a lot of development and machinery going on behind the scenes. Just to give you a few numbers about our scale: we're quite big, we have a few billions of revenue, we have lots of employees and customers, so we operate with high load systems.

[00:01:40] Maksim: What's related to Zalando Helsinki: we have quite some innovative projects that we are working on here in Helsinki, just to name a few of them, like Zalando Wardrobe[?], where we sell your used clothes, or Connected Retail, where it integrates our offline shops into our online store, some projects around Zalando logistics, and part of our main project, which is Fashion Store, is also being developed here in Helsinki, in particular personalized content selection features.

[00:02:14] Maksim: Me and Matthieu, we are in the campaigns and collections team, and the easiest way to understand what we are doing is, if you open the homepage of Zalando of any country you will see a different kind of content, which is approximately what it looks like on the screen. So here is kind of the front end that you see; here is our back end that we operate with, so different kinds of microservices, databases, message queues and stuff. But the main thing is that the goal of our team is to provide the tooling for the content creators to deliver that content to the Fashion Store, and we also take care of the rendering part of it. So it's pretty end-to-end responsibilities, I would say.

How we build at Zalando

[00:03:10] Maksim: Just a few words about our tech kitchen, how we code. We have autonomous teams at Zalando. It means that as developers we write, we test and we run our code, so we take entire ownership of what we are doing here. We are cloud native, we are deep in AWS and most of our services run in Kubernetes. We use a continuous delivery platform for immutable deployments to ship our code to production without any manual interventions.

[00:03:47] Maksim: At Zalando we have a culture of open code. It means that everyone can see the code, and if you need you can search something and go and check what other teams are doing. We also use the API first principle. It means that before going into implementation, we first define an open API interface between our microservices, and only after we agreed that, then we go for implementation.

[00:04:25] Maksim: If you join Zalando as an engineer, or not only as an engineer, you may notice that we write a lot of technical documentation, press releases, design documents and so on and so forth, so we have quite a rich writing culture here. And just a few numbers about our engineering scale: we have quite a lot of engineering teams, AWS accounts and Kubernetes clusters, so there are different challenges with that scale.

[00:05:00] Maksim: As you saw, we have 200 engineering teams, and that means that in Zalando there happens a lot of times where you need to take over a project from another team, because, for example, of our organization. It happened a few times in our team and we tried to come up with some principles that we think would help in that kind of project. So we came up with five principles.

The five principles

[00:05:26] Matthieu: The first one is to start early, because this can be a big process, depending of course on the complexity of the components that you're handing over, but it's important to start early. Then, communication is the key, so already try to over-communicate, don't be afraid of repeating yourself and making sure that everything is clear. Then we try always to have a single source of truth, and for that we try to use only one document where everything is really centralized.

[00:06:03] Matthieu: Of course, as for any project, you will want to iterate and not try to solve everything at once, but really go step by step. And of course when you iterate, then it's important to not break things and to implement the changes that you're doing in a backward compatible way. So let's try to go deeper into these five principles.

[00:06:22] Matthieu: First, start early. I think the first task that is important when you have such a project is to estimate the effort that will be needed, and of course try to estimate how many people will be involved and ensure that you have the capacity for that in your current setup. Then of course you need to identify also constraints and risks. Constraints can be, for example, time constraints, the handover needs to be done before a certain date. Or risks can be that the new team doesn't know the tech stack of the project, and so you need to plan and to ensure that you can mitigate those risks.

[00:07:04] Matthieu: And of course, at Zalando, Maksim said that we are working with open code so we can see the code of all other teams, and we are also doing a lot of inner sourcing, and we try to inner-source some small features or fixes before the actual handover. We think that this is a very good way to get familiar with the project before starting the actual handover.

[00:07:25] Matthieu: Then the next principle is about communication. Here, communication can be first with the involved teams. Usually you will have two different teams, the one handing over and the one taking over. Of course in-person communication is always better, but not always possible, as we can see today at this conference, but we always manage to find ways. But it's not enough to communicate between the two teams. Stakeholders also need to be really kept informed, and we need to really ensure that there is transparency and that they understand what's happening and why this handover is happening, and who, at any point in time, is the contact for them in case of issues or requests or anything.

[00:08:17] Matthieu: And then with communication goes also documentation, and documenting as exhaustively as possible is important, especially while the previous team is here and while their memory is still fresh, because this can be very useful in the future.

[00:08:37] Maksim: Our third principle is that we use a single source of truth, where we try to communicate everything related to the tasks related to handover. Because imagine you have two or three teams, and in case of big projects there are a lot of people which can be involved. If you start spreading knowledge around, then it becomes a total mess when you search something, or when you need to find the right people.

[00:09:09] Maksim: Our approach is very simple and we find it really efficient. We just create a Google document with the proper structure, we document exhaustively, as Matthieu mentioned, we involve everyone who is relevant for the project, and we track all of the progress and we make all of the discussions happen in that one document.

A live demo of the handover conversation

[00:09:35] Maksim: So we prepared for you some live collaboration demo to basically show how it works in a nearly real case scenario. But before we start, let me introduce the characters. It's Matthieu, who is working in the takeover team, and it's me, who is in the handover team. Let's imagine that we have our first video hangouts call regarding the project Tango handover.

[00:10:05] Matthieu: Hi Maksim, how are you?

[00:10:09] Maksim: Hi Matthieu, all good. The weather in Helsinki, it's always very sunny and very, very good.

[00:10:17] Matthieu: So yeah, I scheduled this meeting to align on this project handover. I shared the document with you before the meeting. I don't know if you had time to go through it, but I was thinking that we could go through it and see what are the different topics that we need to discuss.

[00:10:32] Maksim: Yeah, sure, I got the link, so I walked through it, so we can take a look again.

[00:10:35] Matthieu: Okay, so I was thinking that first we could have a look at the infrastructure and the development environment, and then look at the operations, so it's important to know how to operate the system. And then, more on the product side, we would be interested to know if you have any ongoing features. And of course, if you could share any end user documentation or even technical documentation that you could have for your service. What do you think, where should we start?

[00:11:06] Maksim: Well, I would suggest maybe let's go step by step. Should we start with the infrastructure?

[00:11:11] Matthieu: Yeah, sounds good. So I think that you are using Kubernetes to deploy your service.

[00:11:16] Maksim: Yes, that's right, we run our services in Kubernetes, and I guess you guys need access to our cluster so that you will be able to maybe play around with this environment, get access to logs or to the command-line interface.

[00:11:34] Matthieu: Yeah, so let me add a comment for you to not forget about that. Okay, that's great. Then about the development, I think that you have a few repositories for this project, right?

[00:11:45] Maksim: Yeah, so currently we have three main repositories. So it's a management service, it's an admin UI, and it's development deployment configurations. So I will attach links to the GitHub repositories in this document.

[00:12:01] Matthieu: Okay cool, let's also add a note for that. Then the next very important thing is about operations, and I'd like to know if you have any dashboards to know what is the actual system load and all that kind of stuff. Do you have that?

[00:12:19] Maksim: Yeah, actually we have quite plenty of dashboards. Maybe I'm not the best person to give you the links, but let's include Harry[?] into the conversation, he will know for sure.

[00:12:28] Matthieu: Okay, sounds good. Well, maybe lastly for this first meeting, do you have any ongoing features that we should be aware of?

[00:12:36] Maksim: Yeah, our team currently is working on this new video uploading feature. We managed to complete the backend part and the S3 upload of it, but the UI part is still in progress, and I'm not quite sure that we will manage to finish this feature before the handover. So my suggestion would be for your team to start to take a look forward to the UI part, maybe team up with our front-end engineers, and start learning more about it.

[00:13:06] Matthieu: Yeah, that sounds good.

[00:13:09] Maksim: Cool, thank you.

[00:13:11] Matthieu: All right, thank you.

[00:13:12] Maksim: So we just finished our collaboration demo. It was just a teaser. In reality of course it's a series of meetings, and the scope of it is much larger, but it's something like that, so you got the feeling of what kind of discussions and what kind of topics we cover in it.

Iterate: the steps a new team goes through

[00:13:33] Matthieu: Okay, so now maybe we can jump to the fourth principle, which is iterate. Here I guess the key message is that you don't try to solve everything at once and really progress step by step, and we kind of identified a few steps that we think can be useful.

[00:13:48] Matthieu: I think one first step would be that you need to reach a point where you know how to operate the system. That means that you know everything about the monitoring, the logs, alerting, and then you also should be able to evaluate the impact of any incident that can happen in the system.

[00:14:09] Matthieu: Then another step is about dependencies, and here we're talking first about technical dependencies, knowing the upstream and downstream services is very, very important, but also more stakeholder dependencies, and having the relevant contact and ensuring that everyone is in the loop is a very, very important thing.

[00:14:31] Matthieu: I guess that once you are there, you can start to really work on small features or bug fixes, and here the goal of this step would be to ensure that the new team has the correct development environment and is able to deploy, so that the deployment process is clear and working for the new team. And only when you are reaching this point can you start to think about how you will evolve the project, and think more about mid term, long term features, and even maybe long term product vision around this new project.

Don't break things: blue-green and graceful migrations

[00:15:07] Maksim: Cool. So our fifth principle, called don't break things, is really about the reliability and backup strategies and how to make this handover smooth. We have a few key things here that I would like to highlight. First of all, we use the blue-green model for our APIs, database, message queue migrations and so on and so forth.

[00:15:38] Maksim: Usually when we migrate the services, those services get new URLs. It can be just a backend endpoint, or it can be some admin UI URL. So we try not to break things and keep systems working for our users, so that when they come back on Monday the systems keep working. So we try to do graceful URL migrations and we use DNS CNAME records for the old URLs to make sure that it's pointing to the new version of the service.

[00:16:16] Maksim: And of course our end goal is to make sure that we migrated everything and we can shut down the old systems, so we keep monitoring the logs and identifying the users who still go to the old URLs, and we kindly ask them to switch to the new URLs.

[00:16:36] Maksim: But let's go a little bit deeper into technical details. Some of you may have heard of the blue-green deployment model, but just a few words. Let's imagine that blue is your current production state, and now you have a new version of your production service. What do you do? You just deploy it side by side, and on the picture it's the green version, and you have a little component called a router in front of everything where you can basically switch the production traffic back and forth.

[00:17:17] Maksim: Before switching, of course, you need to make sure that the new version works as you expect, and after you are confident you just switch the traffic, and you still keep the blue version working for a while in case of some surprises, or accidental problems, so that you are always able to switch it back within seconds or minutes.

[00:17:45] Maksim: So let's take a look at how we do API migrations, just a high-level overview. On the picture you can see two Kubernetes clusters, so as I said we deploy our services in Kubernetes. On the left side there is the current version and the green one is a new version. We deploy our stuff using the continuous delivery platform, and at this stage our users still go to the old cluster. So the first step that we do is we deploy our new version of the services in the new cluster. I mean, the version could be the same, but just in the new cluster.

[00:18:29] Maksim: The second step would be to create a CNAME, a DNS name, for the old URL, and then notify our users that, hey, here's a new URL, please access your tooling using this URL. Or, in terms of the services, we ask developers of related teams to change their API endpoints to the new cluster. At the same time we keep backward compatibility, so both old and new URLs work as expected for some time. So the last step, once we make sure that no users go to the old path, we shut down the services in the old cluster.

[00:19:14] Maksim: A few words about database migrations, because it's a very important and sensitive part: how to migrate the data between the clusters. There are basically two cases, when downtime for writes is allowed, or when only minimal downtime for writes is allowed. It means that we keep our systems working for reads all the time, but there is some downtime possible for updates, and there are different approaches for those two methods.

[00:19:43] Maksim: Let's take the first one. Here on the picture you can see two Kubernetes clusters, and as a database we use Postgres. So we have our API endpoint connected to the current one, and we have our backups; our backups are stored in S3. The first step that we do is, we basically first put our current database into read-only mode so that no updates are coming from the API, so we make sure that we freeze the state. And then we deploy our database in the new cluster, pointing it to the S3 backups from the old cluster, so that we have two identical clusters, so we have a clone of our old cluster in the new one. And the next step is switching the API endpoint so that it's pointing to the new cluster. That way, if we accept downtime, this is perfectly fine to do it like that.

[00:20:52] Maksim: So what to do when only minimal downtime is allowed? It means that the system is more critical and more write intense. So we have the same picture, we have two Kubernetes clusters, we have a database deployed in the old one with the API endpoint pointing to the old database. So first what we do is we create our standby cluster in the green Kubernetes cluster, which means that this node becomes a replica of the current cluster, and we still keep our old database running, so all of the writes and everything will go through the one. The standby cluster is restored from the backup and we get basically an identical node, but which cannot accept writes yet.

[00:21:40] Maksim: So what we do next is we promote that node to be a master node, and here is where this downtime happens, because master election may take some minutes. After the master is chosen, we need to switch our API endpoints to point to our new cluster, and after that we can shut down our old database.

Wrap-up: what comes after the handover

[00:22:05] Matthieu: Okay, then let's try to wrap up this presentation. So maybe first we can quickly, as a reminder, have these five principles that we talked about. The first one, start early; over-communicate; use one document as a single source of truth; iterate; and of course don't break things. Following these principles we think that you should be able to more easily do your handover.

[00:22:41] Matthieu: But then, what's next? Once the new team has really taken ownership of the new components, then what's important in that new team is to really assess the integration of these new components with the existing ones, and here the question to ask is, okay, does this make sense? Do we have any overlaps between the previous and the new components?

[00:23:05] Matthieu: And then, based on this assessment, I think it's important to try to define the target architecture, because things will not be perfect. So here try to think about the more long-term vision and strategy, to try to define what would a target architecture look like. And once you have this target, then try to find what are the concrete first steps that you can take to move toward that target. And of course it's an iterative process, so always reassess after each step to ensure that you're still in line with what you decided before. I guess that's it for our presentation. We're happy to answer any questions that you would have.