Becoming (and staying) a high-performant organization

Savvas Kleanthous at UXDX Community: Netherlands. Video: https://youtu.be/Zuj4Kjn6DdI

Readable transcript: edited from the recording's captions for readability (fillers and false starts removed, punctuation and section headings added). Wording is the speaker's own. Timestamps are positions in the video. Names marked [?] could not be verified against the audio.

Why this talk exists

[00:00:04] Welcome to my talk. Before I start I would like to thank the organizers of UXDX for organizing an awesome conference, and I would like to thank you for attending my talk. I hope you really enjoy it. The talk is becoming a high-performing organization, which is a relatively large subject, so what we're going to be showing here is just the top tips.

[00:00:33] Before I start I'd like to mention the incentives behind this presentation. When I joined ParcelVision some years ago now, the idea was I joined as a tech lead and an architect, and I found out that while I knew a lot about that part, I didn't know a lot about being the head of engineering that I ended up becoming. The problem was that while there is a lot of information about distinct subjects that relate to this, there isn't actually any body of knowledge around how to bring them together. So there is no way to learn actually what to do from start to finish in order to become a high-performing organization, just a sequence of tips here and there, and not all of them work together. So I've been working for the past year or so trying to come up with a framework that allows you to become a high-performing organization by bringing together all of the different aspects that work combined to help you. This is going to be what I'm going to be covering in this talk, and without further ado let's kick off.

What a high-performing organization is

[00:01:48] So what is a high-performing organization? In my words, a high-performing organization is a learning organization which relentlessly delivers value to its customers. There's a lot in there, so let's try to unpack some of it. Before defining and seeing the relationship of everything, I think it's useful to convey the message through some examples.

[00:02:14] So what's the difference between high performance and low performance? High performers are in fact orders of magnitude more able to deliver code to production, and value to their customers, than low performers. Of course, here we're speaking about operational performance, but still, as we will see later, this is quite important, because this is your ability to deliver value to your customers. And seeing just the numbers here, there is a huge gap between those high performers and low performers. We're talking about 170 times faster to recover from downtime. A high performer is down for a minute, a low performer is down for three hours. That is quite telling of the difference.

[00:03:11] Operational performance however is not just an important aspect in itself, it is important for organizational performance as well. So the State of DevOps report of 2014 hinted at this, and I'm quoting: strong IT performance is a competitive advantage. Firms with high-performing IT organizations were twice as likely to exceed their profitability, market share and productivity goals. But while the State of DevOps alluded to this, the Accelerate book that came out in 2018 actually proved it conclusively, with scientific research that was done for Accelerate. In the scientific research you can find out about exactly how they calculated these things, but they proved that there is a causal relationship between software delivery performance, so your operational performance, and how your organization performs. So when we say organizational performance we really mean basically money that you're making, and also non-commercial performance.

Where ParcelVision started, and where it is now

[00:04:18] What's our experience? When I started working with ParcelVision, I wouldn't necessarily say that we were a traditional low performing environment [?], but we still didn't fall in the high performance description of the State of DevOps. We released once or twice per sprint. Work lead time, so the time from a feature being requested to being deployed in production, was three to four weeks. And deployment lead time, so from code committed to a repository to the code running in production, was in the magnitude of days.

[00:04:57] Now, after implementing this framework, we are seeing that we're deploying 10 times per day, the work lead time dropped from weeks to days, and the deployment time is around 15 minutes. So how do you achieve that? The six top tips to achieve that operational performance, to help even achieve organizational performance.

Tip one: avoid monoliths

[00:05:26] Tip number one, avoid monoliths. When I say monoliths, I do not refer to a modular monolith. A modular monolith is okay. A modular monolith, if it fits your non-functional requirements, is not something to avoid. But on the other hand, traditional monoliths and distributed monoliths are two very bad situations to be in, because it impacts your ability to deliver changes drastically, and it makes additional work take a disproportionately longer amount of time to actually do.

[00:06:07] So how do we tackle this? There are three tips that I would like to mention here. First of all is domain driven design, the strategic patterns. So domain driven design is a set of patterns that you can apply in your domain model and architecture, which are the strategic patterns, but also some patterns that you can apply in code to better meet some of the requirements that you have, which are the tactical patterns.

[00:06:43] So part of the strategic patterns that apply in this particular situation is the concept of bounded context. Bounded contexts are in essence a boundary around functionality that helps make sure that we're encapsulating complexity and don't let it seep through to other services. And you'll notice that we're aligning those bounded contexts with business domains and subdomains. So what you'll notice is that if you have a shipping functionality within your business, you'll discover that you have a shipping bounded context. It may be important, it may be not, but there will be some there, and there is a boundary around that, and that helps modularize things.

[00:07:35] On top of bounded contexts we have context mapping, which is relatively new. Domain driven design has existed for something around 20 years. Context mapping builds on top of bounded contexts. It helps you identify what kind of relationships those bounded contexts inherently have. Modeling those relationships correctly will help you make sure that you don't have a disconnect, and also that you don't have increased coupling between your services. So this would be stuff like a shared data model. If there is indeed a shared data model between those services, so it's like a user for example, there is a user model across many services, recognizing what kind of relationship it is, so is the user owned by a specific context and shared in some cases? So the model, it's necessary. I'm not speaking about sharing with other bits, but how that gets shared across is quite useful.

[00:08:47] The final tip for this is the use of event storming. Event storming is a collaborative exercise to create a domain model along with your domain expert. Event storming helps you avoid monoliths by making the bounded contexts, the boundaries of functionality, very visible. There are a lot of tools, there is a good book that you can read to learn about event storming specifically, and how that relates to coming up with useful and meaningful bounded context boundaries.

[00:09:28] And before I move forward I would like to mention this quote from Alberto Brandolini, the creator of event storming. So quoting: it is not the domain expert knowledge that goes into production, it is the developer's assumption of that knowledge that goes into production. Event storming really helps with that, it helps transfer and convey that knowledge across everybody, and it helps reach a shared understanding of what we're going to be building and what is going to be running in production. And again, it is very useful because the idea of the bounded context is we're going to be creating boundaries around what the domain experts think of as domains and functionality. So having that here is really, really important and really helps a lot.

Tip two: avoid the impedance between persistence and domain models

[00:10:21] Tip number two is less obvious. It's basically avoid the impedance between your persistence model, so the shape of the data as stored in your database, and your domain model, which is the shape of your entities when you're in your business logic. One tip that helps here, basically the solution to this problem, is to use event sourcing.

[00:10:55] So event sourcing is a systems pattern that basically stores a particular entity not as a document, or as tables in a SQL database or a normal form database, but instead as a sequence of changes, domain-specific changes, that have happened over that entity. This is actually quite powerful, because you will notice as you maintain your solution that your domain model changes to support new functionality but your persistence model does not. So your persistence model, if you're using a document database, your persistence model is what used to support your previous domain model, but it doesn't necessarily have all of the data to support your current one. And especially with document databases it may be quite complicated to actually restructure that information. With a normal form database, again, there are the same kinds of problems. So you have data stored in specific structures and relationships, but they don't necessarily map well with your domain model.

[00:12:03] This impedance can impact your ability to deliver work, and it also makes deployment quite difficult when you're introducing new functionality that needs the data there. Event sourcing guarantees that you have information there, and also that you have the information in a way that is very slow changing, especially if you use event storming and event modeling to make sure that you have captured those events as the domain experts recognize them. The chance that this changes is really low. It does happen from time to time, so you will still need to understand and learn a new set of patterns that are necessary to support and maintain event sourcing in production. But from experience I can tell you that it's actually quite a lot easier to deliver changes in the domain model in production using event sourcing rather than anything else.

Tip three: reduce coupling

[00:13:07] Tip number three, reduce coupling. The best way to reduce coupling is to use event-driven architectures. Event-driven architectures by their nature exhibit low coupling, and they exhibit low coupling for multiple reasons. The topmost one is that because it's messaging, it doesn't get affected by temporal coupling. So if one of your services is down, for whatever reason, the other one can still function, and when the service that goes down comes up, then it immediately restarts consuming stuff like it didn't happen at all.

[00:13:47] So this is really, really powerful, and because also it means that you can be partially available to your users, or even entirely available if the service that got temporarily unavailable was internal in your system. People will be hitting the API, they will be getting accepted or okay, depending on how you're designing your system, and they will be happy knowing that what they did happened, and they wouldn't be noticing any unavailability at all. That's actually quite important with coupling.

[00:14:20] Event-driven architecture helps with reducing coupling as well because it doesn't exhibit behavioral coupling either. And finally it also helps with coupling from the perspective of the contracts. So usually if you are using any synchronous forms of communication, your services depend on those contracts, and as you alter behavior in those services, anything that depends on that needs to change. With event-driven architectures and messaging, the surface area gets drastically reduced.

[00:14:57] But on top of this you can actually use shared nothing practices to make sure that that is reduced even further. So shared nothing architecture is basically when data gets into the system, they either get disseminated to any services that need them, so there isn't a need to communicate some of the data across, or once they come into the system they are published, observed and stored locally, and then again when there is a functional workflow running there isn't a need to transfer models across. There is a big body of knowledge from the reactive principles that you can tap into to help you design event-driven architectures that exhibit low coupling.

[00:15:45] And the final tip here is to prefer choreography. Choreography is basically one of the approaches with which you can implement a sequence of actions. Assuming that we have a workflow, A then B then C then D, and that's the workflow that needs to happen, when the user tries to do something a choreography will imply that A publishes the changes, then B observes that and does its own thing and declares the changes, so raises an event, and then C, and so on and so forth. In orchestration, which is the alternative, you will have a central actor owning that workflow, and so it will reach out to A and say, A, do this, and then when it does it, it observes that change and then says, B, it's your turn to do your thing, and so on and so forth.

[00:16:35] Choreography is better because it reduces the amount of coupling that exists. In orchestration you have a central orchestrator that's coupled to everything, and then in turn all of the independent parts that do stuff also depend on the orchestrator as well, whereas in choreography this becomes linear. There will obviously be situations where you need to implement orchestration, especially in complex failure scenarios. So if you have certain conditions where you are applying one or the other of two options in response to something failing, that's an indication of where you need to implement it. But other than that, default to choreography, because it helps reduce coupling.

Tip four: meaningful test coverage

[00:17:18] Another tip is one that is potentially a bit obvious: have meaningful test coverage. Before I mention exactly how to do this, I would say that one thing that definitely helps a lot is the onion architecture. So the onion architecture is structuring your applications in layers with dependencies only pointing in, and in the center of the onion architecture is the domain, and the domain is where your business logic lives, and it is entirely dependency free. It depends on nothing else other than itself.

[00:18:01] So how does this help us in test coverage? Well, this way we can have very quick feedback tests that hit the domain and do their job, and we can validate that domain quite quickly and get feedback about the behavior of the system that we are working on, very meaningful, very quickly, before we even commit the code.

[00:18:23] Event sourcing helps in this regard as well, because it changes those tests from whatever you need to do, so setup of data which you didn't know how you came up with, and that domain model fit like that. It changes it to: these are the previous decisions that we made, this is the request from the user that we need to do, and then you expect certain events coming out. So these are the decisions the domain has made, you're providing past decisions, the action, and expecting a certain decision. Now, this is something your domain experts understand, so this is something that you can give to the domain expert and they will tell you that this makes sense or this does not.

[00:19:05] This also helps you having a meaningful test coverage from the perspective of, you don't test code that doesn't need to be tested, you don't test conditions that don't need to be tested. You don't need to guard against stuff that will never happen, which happens more often than you may think. Flipside of that, so in addition and complementing that, would be contract tests. So you are testing your domain quite nicely. A contract test will help you test the public data contracts of your services, so making sure that the data and events that you are raising out, or any APIs that you have, don't break during your implementation. This is very important feedback, especially in a microservices environment, but these are also tests that very, very quickly and very efficiently give you back information about what you are doing.

[00:19:59] And then you will definitely have end-to-end tests. You should be doing some end-to-end tests, but keep them small, keep them to a small number, and do rely on observability. By the way, if you're having a microservices architecture, you'll find out that testing all of the permutations of behavior of your system is going to take a very, very large number of end-to-end tests, and these need a long time to build, a long time to maintain, but also it will explode the time necessary to build and deploy your services. So don't do that. Obviously you need rock solid CI/CD to do that.

Tip five: reduce development time dependencies

[00:20:36] Tip number five, reduce development time dependencies. Development time dependencies are dependencies that are stopping you from doing your work, so you need team A to finish something in order for you, team B, to do something. Bounded contexts help again. So it helps similarly to how it avoided the monolithic applications: it helps you by making sure that behavior is encapsulated within a single bounded context. So if you are delivering features around shipping, you are touching this bounded context alone, you don't need data from other services, you don't need behavior from yet other services, maybe. All of it is encapsulated in that particular one. And again, while bounded contexts are the pattern that you use for this, obviously event storming and context mapping help you make sure that those boundaries are correct.

[00:21:33] Structure teams around those bounded contexts, so you have that behavior encapsulated there. Start wrapping things around so you don't need other teams to do that. Have one bounded context owned only by a single team. So of course a team can own more bounded contexts. If you have more bounded contexts than teams, you don't need to hire more people. But on the other hand, avoid situations where a bounded context is shared across teams. Have a single team own a bounded context and establish those relationships with context mapping.

[00:22:09] Event-driven architecture helps as well, by reducing coupling as we explained earlier. So when you are doing a piece of work, any dependencies on that piece of work are on the contract, and they can still observe it, and don't have behavioral coupling. So behavioral coupling is really the main cause of development time dependencies.

[00:22:32] And as a final tip for this, make sure that the stable service is the one that owns that contract between those two services. So an example would be the email service. You wouldn't have the email service that sends notifications of things that have happened depend on 10 or 20 services that you have in your system, because as those change you need to keep maintaining that email service. Why would you want to do that? Expose a contract and have all of the other services depend on that contract. The email service is probably going to be stable, because SMTP is an ancient protocol, and in many regards it hasn't changed for a long time, it's not going to change now. So definitely identify those mature domains and assign the contracts to them.

Tip six: reduce rework and waste

[00:23:18] Final tip, reduce rework and waste. So this is all of the lessons here, basically lessons that came up and existed, and you can find a large body of knowledge from manufacturing, and I mentioned this as we go through the different tips.

[00:23:34] So tip number one, adopt an improvement budget. An improvement budget is kind of similar to another budget. An improvement budget means allocating a certain amount of time, a week or whatever, in improvement work. So that would mean removing any technical debt that was introduced during previous weeks or previous work, making sure that you reduce coupling if you discover that you introduced something without your knowledge, adding tests to help you, creating new tools to facilitate deployments and testing. Improvement work: make sure that you allocate some time to this.

[00:24:20] Cross-functional teams. And when I say cross-functional teams I really mean not just front end and development, I mean literally everyone. The subject matter experts, developers, UX, BAs, everyone, the business, all of them working together to deliver a product. This comes straight out of the concept of one piece flow in manufacturing. So manufacturing is an ancient industry, we've been manufacturing stuff for hundreds of years now, and there's a large body of knowledge around how to make that efficient. And I know that we all like to think that we are thought workers and we are working on ideas and developing this, but in fact we have very large similarities with manufacturing. We are building a product and delivering it. We are even using the same terms. So this is one of the ways to make sure that you are reducing waste, by making sure that you're building the right thing, and you are reducing the handovers that are required. So everyone needed is involved, so the work gets done then and there and immediately can get delivered to production.

[00:25:36] Using a pull based system, another thing yet again taken from manufacturing, this helps guarantee that the team that is developing the product doesn't get interrupted and have to pay costs for context switching, but also it helps guarantee that you are doing a piece of work just when you need it, and not before, not later.

[00:26:02] And finally, I have a tip to reduce rework and waste: carefully consider the alternatives of any work that you're going to be doing. Make sure that what you're going to be building is what is actually needed and what is correct. If you're going to be integrating with a service, make sure that you check all of the other services that provide similar products, and make sure that you make the right choice given the requirements that you have.

[00:26:33] That's my presentation. This is some of the reference material that I used to create this presentation. Thank you very much. You will be able to find the presentation, by the way, on my Twitter profile [?], and do follow me on Twitter if you have any questions. Thank you very much.