Planning Your Agile Architecture

Will Demaine at UXDX Europe. Video: https://youtu.be/HDPXO0CriqQ

Readable transcript: edited from the recording's captions for readability (fillers and false starts removed, punctuation and section headings added). Wording is the speaker's own. Timestamps are positions in the video. Names marked [?] could not be verified against the audio.

Introduction: Fat Llama and the move to cross-functional teams

[00:00:07] Hey guys, how are you doing? I'm Will, engineering at Fat Llama. This is Larry, he's our mascot at Fat Llama, and we're going to talk about planning and migrating to an agile architecture. I know a lot of people might already be in enterprise companies where you have a lot of existing systems, not necessarily building greenfield, so I want to talk about some strategies for migrating as well.

[00:00:27] Okay, cool. Just quickly, Fat Llama: we're an insurance-backed marketplace for lending and borrowing anything. We have 100,000 users in London and are growing very quickly. We just raised a Series A round of funding. We have some great investors. We started with Y Combinator, and the recent round was led by Blossom Capital and Atomico. Importantly, we're hiring. Everything on engineering: front end, back end, data, mobile, absolutely everything. Data science, design, growth, product, operations. We're growing extremely quickly as a company and we'd love to hear from you. Come and talk to us: the careers page at fatllama.com [?], or email me, just will at fatllama.com.

[00:01:06] As was mentioned, there are a couple of places I've worked. I won't dwell on it, but the reason I wanted to show this is that I've worked at a tiny startup, I've worked at big enterprise companies, and I've seen companies grow from one into the other. So there were some different things along the way that helped form these ideas.

[00:01:25] You're here probably because you're either moving to, or already in, an agile team. And this might be what you're currently working with: more of the legacy setup, where you have these different functions that all work on their own pieces of the system. Maybe you have a front-end and back-end team working on some monolith, and only an operations team that touches the infrastructure. And this seems to work. But you want to become more agile as a company, as a development team, and you hear a lot of people here today talking about having cross-functional teams.

[00:01:58] For us at Fat Llama, that might look like this; it's something we're going through. We might have a borrowers team, a lenders team, business operations engineering, whatever it is. As soon as we have these cross-functional teams, now every team has to touch every different part of the architecture. I've made this intentionally extreme to make a point, but I think it is true: you have everyone touching everything. And when you do that, you might encounter some problems. What I wanted to talk about is how to solve some of those problems.

The problems: too many cooks and a lack of ownership

[00:02:24] Some of the problems are quite simple. You just have too many cooks. You have large and risky deployments, because everything is in one system, everything in that monolith, and you get into the situation where there's a huge amount of change every time you want to deploy. Maybe your deploy cadence is every week or month, or maybe you go even faster, more agile, and deploy multiple times a day. We're deploying multiple times a day, and we still get to the point where we have risky deployments, with even ten people making changes on the same codebase.

[00:02:55] Importantly, you can be deploying code that you don't even understand. Maybe I'm making a change as part of the lenders team on this product, and someone else from the borrowers team is making a change as well on the same thing. I didn't even realize; I'm not part of their process and what their team's building, and suddenly I'm deploying this piece of code. I have no idea what it's doing. Maybe it's going to break something, and I'm not going to understand what's happened or how I'm going to fix it.

[00:03:22] And the simple stuff: with everything in one system, you're going to have long-running test cycles. You might even have contention on files, and painful rebasing and that kind of stuff. But I think more importantly, you end up with a lack of ownership. This is one of the things that Lucre [?] talks about, and I'm sure lots of other people will on the vision stage as well. If you have this lack of ownership, you have all sorts of problems. Multiple or redundant patterns for the same thing; we even see this at Fat Llama with ten engineers. A slow process to reach consensus when you do have multiple patterns. Maybe you want to change the way your API responses are structured or something, and it takes a long time just to get everyone to agree, when everyone's in disparate teams, on how one thing should be done.

[00:04:04] Something that I definitely saw at Uber was fractured relations between teams. This is probably something that people don't necessarily realize is happening, but before you know it, you get this animosity between teams, where another team might change something and you don't know about it. We had this instance where we were shipping the first version of Uber for Business back in 2014. We were going live, I think, at 9:00 am on Bloomberg, with our VP of business announcing this product. And I think at 1:00 am the night before, another team made some changes and completely broke our system. We were up till six, seven in the morning trying to fix this and deploy to get it working. We did, but looking back on that, the team thought this was very, very painful for us, and they didn't even realize that it happened. So when everyone is working on the same thing, you get really strained relationships between teams.

[00:05:01] I think, and this was alluded to before, the most overlooked scaling problem isn't necessarily about requests per second. It's about developer experience as well, and how everyone's working together. This is the biggest problem that we face as an engineering team at Fat Llama. It's not necessarily pure scale in terms of queries, but how we work effectively.

[00:05:23] A more modern architecture, and it's very vague, might look something like this. Each of the teams has their independent areas of the system, broken down with strong ownership. Obviously these different systems are going to have to communicate, but they're owned, they're smaller, and they can all be deployed independently. That lends itself to microservices and that kind of stuff, which we'll talk about a little bit.

Rules for modernizing

[00:05:47] But I want to talk about rules, my rules at least, for modernizing, going from the legacy system to something a little bit more like what I've just shown. The first one, which I think people don't always realize, is that you have to deliver business value along the way. This may as well be called beware the fork. I think any time that you have to fork your codebase for some part of your business, you're at huge risk. I've seen this happen plenty of times. The number of times I've seen forks fail is ridiculous. If you don't know what I mean by fork, that's just stopping development on one thing, saying we're going to rebuild everything completely, trying to build that from the ground up, and then just switching everything over in one go. So delivering business value along the way is extremely important.

[00:06:40] Respect what came before you. This is something that's quite important to me. There are so many instances where you think, okay, we're going to rebuild this system, sure, how hard can it be? Just look at it, it's only a simple page with a few bits of functionality. We can rebuild that in a month, no problem. And then you get into the weeds and you realize there are all these intricacies and all these other little edge cases that you didn't realize. It turns out the old system is a mess because it was hard, and it was something that people had to deal with at the time. Maybe there were different business pressures, whatever it was. But if you don't give it that respect and learn from those things, then rebuilding everything is going to be an absolute nightmare.

[00:07:26] Choose what you know, not what you want to know. This is a classic problem for engineering. We all want to work on new, interesting technologies, and when you're going to modernize your system, you want to pick the latest stuff, which is great, and sometimes you do have to keep learning. But if you're going to rebuild a critical part of the system, or something very important to the business, and you go and choose a completely new technology that none of your team are familiar with, or maybe one that's still emerging in the space, you're going to run into so many more problems than if you just pick something that you already know. We see this all the time even when interviewing people: they choose to do the interview in a programming language that we use, not one that they know, and it always ends in disaster.

[00:08:10] Don't build when you can buy. This is pretty simple. It doesn't necessarily mean buy; also use open source things as well. There's plenty of software out there to make these kinds of migrations much simpler, so don't try and build everything yourself. And, similarly to choosing new technology, beware technology hype.

Beware technology hype: should you migrate?

[00:08:30] I said I'd talk about this a little bit more. This is a graph from Gartner that shows what they call the technology hype cycle. There's an innovation trigger that starts off this push for a new technology. Everyone thinks it's the best thing in the world and it's going to solve all your problems. They start implementing it, they realize this is actually a nightmare and it's not productive at all, and then slowly the technology develops and you get to a point where you can actually be productive with it.

[00:08:54] The good news is that for some of the stuff that we want to use, we're already climbing that slope. I think Docker, right at the top here, is in the plateau of productivity. It's really stable, it's very useful, lots of people are using it to great effect. Microservices are on their way up. Kubernetes, not quite there for me. The reason I think this is important is that if you go online and read lots of blogs and articles about migrating to microservices, rewind five years and everyone was saying microservices are the answer to everything. If you search for "should I do microservices" now, probably the first few pages of Google are filled with "you shouldn't do microservices, this is so hard". That's right in the trough. And now we are coming out to a point, I think, where it's something that's actually feasible, and so I want to talk a little bit about how we might do that.

[00:09:45] Should you migrate? At Fat Llama, we're not ready for Kubernetes. Our current solution is working fine with simpler technologies; we're running on ECS on AWS. No one's ever run Kubernetes in production, it won't add any business value in the short term, and it's still really difficult to do. That's changing, but for us it still feels like a bit of a stretch. So there's no need to migrate. You don't always have to; we're running fine the way we are.

[00:10:15] We are moving to microservices, but I should say that microservices might not be the answer for your team right now. Sometimes the monolith is much simpler, especially if it's a smaller company and you're still figuring out your product market fit. You need to understand your problem domain, and microservices are way more complex than needed. Something I learned very quickly from Uber is that if you're successful, everything's going to be rebuilt anyway. I don't think there's a single line of code I wrote that's still there. It's probably for the best.

[00:10:48] This is just to say: don't worry too much about the choices you make, don't worry too much about trying to be on the latest thing. If you're successful, the company will grow, and what you had one engineer working on before will then have 10 engineers, able to build a much better, more thorough solution for that problem. I've not seen companies work efficiently at scale without microservices, personally, even the big enterprises, but I'm sure it can happen. I'm not some microservices zealot, but I think it's a good way to go. But things are instantly harder. I won't go through everything; there are plenty of articles online telling you why it's hard. There are plenty of problems that you're going to encounter when you do it, so just make sure that you're aware of some of the problems you're going to have.

Step zero: build a foundation for success

[00:11:36] With all that said, I'd like to talk a little bit about what it might be like to migrate to microservices, or maybe not straight to microservices, but to start to break out your monolith into a more usable structure for the different teams as you go cross-functional and become more agile. Hopefully what we're going to achieve is something like this. This is what we've done recently at Fat Llama. We've gone from having a web and a mobile client talking directly to a monolith, with everything in one database, to putting in place this gateway layer and starting to break services out from the monolith, using them independently, and shrinking the monolith as we go.

[00:12:24] Okay, so step zero. Step zero because I forgot it originally, and then realized that this is absolutely the most important thing. You have to build a foundation for success here. You've got to get your house in order before you even start. You need a working test suite. You're going to be doing lots of hairy changes in the code to refactor things. You don't have to have everything fully tested, but at least have a test suite that you can run on the existing codebase before you even try rebuilding and building something new. I think it's important to have changes building in CI, and I'm sure lots of people talk about that at this conference.

[00:13:05] A big one for me is observability. This was basically the first major change that I made joining Fat Llama: making sure that this observability stuff was in place. If you can't build graphs like this with a few clicks, then you probably need to invest some time in observability, and I'll talk a little bit about ways you can do that. It's very, very important that you can capture metrics about your system and about how it's performing, so that when you come to do these refactorings and switches, you can see instantly, in real time, what changes are happening in the system, where the traffic is flowing, that kind of stuff.

[00:13:47] Along with that, obviously, logging is very important, and alerting is another important thing to have. Sure, if you have a graph that shows you something is broken, that's great, but if you don't get told without looking at the graph all the time, then you're going to miss stuff. There are really great tools to do this, and people think it's probably harder than it is. When I joined Fat Llama, I think within a day of starting to get this observability stuff going, we had graphs like this.

[00:14:17] You can find Docker containers already built which combine InfluxDB, Telegraf and Grafana, which is what we run. There's also Prometheus and other things as well. You can just find a stock container. I just spun up an EC2 instance, which is why I think the last part, being able to run this stuff in the cloud, is important. I just picked an EC2 instance, put this Docker Compose file on it, ran docker compose up, and it brought up this whole stack of monitoring software. At that point you can just start to use clients that are already built, off the shelf, to send stats to this system.

[00:14:52] It doesn't have lots of redundancy, and if that server goes down, well, we don't have stats. But you know what, for the last six months it's been fine. You don't have to go through this huge project to add observability. You don't have to get all this buy-in from the business. You can just put it on a small server somewhere. It's going to cost you pennies, and you start to see the benefits immediately.

[00:15:14] Feature flags are something that's really, really important. I guess this is actually critical for what I talk about next: being able to control where traffic flows in the system and who sees what. We use feature flags for that. You can basically go on a UI, hit a button and direct people through the code in some way. We use this for rolling out to users. We use it for testing internally first, and then we do a 10% rollout, 20%, et cetera. We use feature flags for all this stuff, and that's really important as well.

Isolating externalities, business domains and inputs

[00:15:46] If we're going to break down our monolith, the first thing that I think is really important to do is to identify all your externalities. We have this monolith with all these different modules in it, all talking to lots of different systems. For Fat Llama, this is stuff like sending email, sending texts, Google for authentication, ID checks. A database is an externality. Logging, anything, cache, writing to disk, whatever it is. You need to know what all of these things are, and you need to group them together. Otherwise, when you start to pull out different parts of the system, you'll realize that you didn't cover everything.

[00:16:19] We want to get to... we're still in the state where our monolith might be a bit messy. We have all these modules, but at least the externalities are covered. We have this new layer, which is an interface into this third party. This is going to be really, really important for when we migrate: that we have everything internally going to one place, so that when we move, we can just redirect everything that goes to that one place to another system.

[00:16:48] Then the next step is to look at the middle layer. We have our externalities, we have the output covered. We want to look at the middle layer and start to isolate our business domains and entities. For Fat Llama, that's users, items, transactions, reviews, messages, whatever it is. Still within that monolith, we want to just reorganize and restructure the code a little bit so that we bring all these things together. The reason we do that, again, is so we have one place we can go to where we know this is everywhere we deal with business logic for transactions or for messages, rather than having stuff all over the codebase, which makes it very difficult to refactor.

[00:17:28] And the same thing for the input side. We want to organize our inputs. We might have had web and mobile calling into all these different files that were in weird places around the system. We want to basically put a very strong API layer in front of that. We might define our users API or transactions API and have everything going to that one place again, so that we can decide to fork from that layer.

[00:17:55] It's really the same thing over and over again here: organizing things into interfaces so that you can swap out what's behind that interface in the future. If you're familiar with object-oriented programming, this is the same thing, but on a bigger scale, on the architecture scale rather than the specific class scale. It's just putting an interface somewhere so you can change the implementation behind it.

[00:18:19] You might see here one of the things that we found when we started to do this whole thing: we have an input at the bottom that's not even being used anymore. We probably deleted 15, 20 endpoints just from reorganizing our code and realizing this stuff wasn't actually used. So there's some benefit in doing this as well. You'll find that lots of stuff is not used anymore, and you should go and clean it up.

[00:18:41] So far we're just reorganizing our monolith into these different layers. This is a screenshot lifted straight from the internal docs that we made about how to design a service. You effectively have those three layers: the input layer, the business logic layer, and the output layer, where the output is clients for externalities and for databases. This is a really important but long process. If you don't do this and you try and rebuild the system, you're going to be in a world of pain, because you're going to find that your business logic layer is tied to your input layer. Maybe it reads some HTTP headers, or maybe the service layer responds with some specific HTTP code. When you go to build your new system that uses RPC instead of HTTP, you realize that there's no direct translation and you have to rebuild a lot of the business logic. So if you can keep these things separate and clean, you're going to be in a much better place for swapping out any different part of the system.

[00:19:40] This is a long process. We've been doing this at Fat Llama for the last six to eight months, and still delivering business value as we go. We're rebuilding parts of the system, we are still adding new things to the monolith, but as we go along we're starting to gather things together in the place where they should live, and we're doing it relatively slowly so that we can keep adding value as we go.

Building a dummy service first

[00:20:05] The fun part: how do we actually get the new system going, now that we've reorganized, or at least reorganized part of, the old stuff? The first thing, which I don't see many people do but something I learned at Uber, was to build a dummy service. They called it Stooges [?]: actually three of them talking to each other in production. When a team was rebuilding or designing a new communication protocol, they set up these three servers that would talk to each other so you can see exactly how it's going to work.

[00:20:35] This is something we've just done at Fat Llama, and it's very, very useful. It's just a service that does pretty much nothing. You can send it a message and it sends you the same message back. You can send it a message and it throws an error. And also a delay: you can send it a message and it will wait a certain amount of time that you tell it and then respond. It seems completely useless, but it's a very, very useful thing to do as you're figuring out what all your new technologies are going to be, what your new stack is going to be.

[00:21:02] We are moving to a Go-based gRPC microservices deployment, and we built this service that basically did nothing and put it into production. What that meant is that we could figure out our logging, our metrics, our alerting, our deployment process, our CI process, our communications. We discovered so many problems just by building this simple service. We realized that we didn't have a good plan for handling SSL for communication between services, which we ended up just saying, let's not worry about that now, let's ditch it.

[00:21:35] All of these things that we learned: if we'd tried to do this whilst building a project that had an important, critical business function and a tight deadline and timeline, we would have been over by weeks in terms of our delivery. So building this thing, getting everything in order, getting your deployment pipelines and all this stuff in order with a simple service that doesn't really do anything, is a really, really useful thing to do. I would suggest, if you're going on this crazy endeavor, do something similar, even if business people might look at you and think, why would you spend time building something that does nothing? It's very, very useful.

Extracting the first service and migrating traffic

[00:22:13] Now we've built a stooge, we've put it in production, and we've figured out a lot of our stuff. We know how to do logging and all that stuff. The next part is the fun bit, where we get to pick our first candidate for exorcism from our monolith. Obviously in this diagram it's this one, because there are only two in the diagram. But pick something that doesn't have many complex dependencies and that's not necessarily critical to the business as a whole. One of the things that we picked at Fat Llama was our system to verify your email address and phone number. When you sign up, you get an email, you click that, and it calls this service, effectively. It's a very small, contained, simple thing. That's the kind of thing you want to start with.

[00:22:56] And you get to do the fun bit, which is rebuilding it as another service. Take your new technologies and rebuild. This is a little bit fork-ish, but you don't have to rebuild the whole thing. You can just rebuild the part that you're going to use first. Maybe your users API is thirty functions long; you might just pick three or four of the most critical ones, just create and delete or something like that, to rebuild first, and then you can go and put that into production. It's a really good chance to improve your API, so think hard about it. Don't just forget everything and literally pull the code out.

[00:23:34] The next thing you want to do is to start to duplicate write traffic to this new service. We're building this new service, but the old one is still running in production. What we want to avoid is that fork where we just switch everything over in one go. Because we have this nice place where everything is going into one module, with the interface that we've designed, we can start to say: okay, when this call comes into our users service to create a user, we're just going to fork that call and also write it to the new system as well. Then what we have is the old system completely working as it was before, nothing's changed, but also the new one that we're testing out.

[00:24:11] This is where we'd use feature flags to enable this. We'd first say, okay, for all employees, write to the new database, then for maybe 10% of users, and we slowly ramp it up. We don't go to a hundred percent straight away, because then you risk causing other problems or breaking the new system as well. So we slowly ramp up until we're writing everything to both the old one and the new one.

[00:24:33] Then we make sure that they match. We want to test the integrity of our data. We look at the database from the old system, we look at the new one. We might have to run a backfill on the new one: now that all of our writes are going to it, we can bring all the old data from the old one and run a script to backfill, so our new data should, in theory, match exactly. And maybe we run a script that just pulls data from the old one and data from the new one and compares it, to make sure that the new system is behaving the same as the old one.

[00:25:01] Then we can start reading from the new system. Before, we were writing; now we've written a hundred percent and made sure our data is the same. We use the feature flags again to control reads from this new service. We let the old one continue on, and we start to pull a little bit of that traffic, with a feature flag, off to the side and replace it with the new data coming back. I should say that you might need to write some code. Maybe the new data isn't necessarily structured in the same way, or maybe it's a completely different database technology. But because we have this one interface point, we can manipulate that data on the way back in to make it match what the old API expects.

[00:25:43] We start to switch this over. We do a controlled rollout with feature flags again, and we look at the monitoring. We check the graphs: how many requests are going to this new service, is everything okay, is it handling the load? We roll out to our employees first, anyone with a Fat Llama email address, and then to 10% of users, and so on, until we're confident that the new one is working as we expect.

[00:26:02] And then it's cleanup. Now we've moved this service out, and we have a slightly smaller monolith, which is great, and we have this new service running in production. At no point along the way did we ever just say, turn the old one off and turn the new one on. Everything was a slow, controlled rollout, and really, that's the way that you can do this without breaking stuff.

[00:26:22] If you're doing this for the first time, I will say it's very, very important to go through these steps and get it right. Because if you say, we're going to move to microservices, hey business, this is going to take a little bit of time, but trust us, this is the right thing to do, and you do it wrong the first time, that's not going to go well. When you actually want to do this again with five other services, people are going to say, you know what, let's not bother, everything's fine with the monolith. So really go through the effort of making sure that you do this right the first time, if you are going to go along this path.

The gateway layer and protecting clients

[00:26:54] A final thing is just a quick note on the clients. I did say at the start that we've moved to this model of having the gateway calling these other services. Obviously the only thing I've shown you is taking some logic out of a monolith and breaking it out separately. One of the things that we've done at Fat Llama which is very, very useful is to put this gateway layer in place. What that does is protect your clients, especially mobile, where you can't roll back if you push a change, from all the change that happens in the backend. We don't want anything to affect the users, and we don't want to break people's old mobile apps. So you put this gateway layer in place, and it lets you start to call through to these new services without interrupting the clients.

[00:27:42] The first thing we did to set this up was actually just have a gateway that's just a proxy, and everything passes through it directly to the old monolith as it used to. The apps and the clients don't even know that it exists; it's just a proxy. And then we start to take little pieces, certain endpoints that the clients hit, and forward those to the new services instead. So again, it's another interface layer that we're putting in place, and that is going to really help you if you're doing this for the front end and not just the back end as well. I think that's pretty much everything I have. Thank you very much for listening, and I'm happy to take any questions.