Architecture Debate: Shift Left vs Shift Right

Tony Gorman, Nix Crabtree at UXDX Community: Central Europe. Video: https://youtu.be/lSIxpajPhEY

Readable transcript: edited from the recording's captions for readability (fillers and false starts removed, punctuation and section headings added). Wording is the speaker's own. Timestamps are positions in the video. Names marked [?] could not be verified against the audio.

Round one: should architecture be designed up front?

[00:00:03] Host: We'll start with Nix, anyway, on shifting right: why architecture is important.

[00:00:07] Nix: Okay, cool. Emergent architecture works well with one team, but what inevitably happens as the number of teams grows is a Wild West ecosystem of technologies and approaches, which rapidly becomes impossible to manage and support. Now, innovation is a good thing, and in fact exploring new technologies and their application is often a key market differentiator. But your teams are not independent startups. They aren't working on their own independent business goals, establishing their own commercial agreements. Your teams are all working towards the same goal: the success of your business. And technologies have failed [?].

[00:00:52] Nix: As a business, you want to manage your costs by optimizing your technology stack. You want to increase your efficiency by developing deeper expertise in key technologies, sharing it across teams, and enabling developers to focus on solving problems in new and creative ways, but doing so through a shared technology vision, toolset and infrastructure.

[00:01:16] Nix: Think of it like this. You hire a bunch of creative property development teams to build your luxury mansion. Some choose a modern style with glass, some choose a quirky style with towers and turrets, some a red-brick style, others a country cottage, and so on. Each team believes in what it's building, and what they produce may be a triumph in isolation. But each team is also spending time and effort planning and installing its own plumbing and cabling and building its own foundations.

[00:01:48] Nix: The problems become more subtle as you go. The connecting doors are roughly in the right place but slightly different sizes, so they have to be rebuilt after the fact. The heating and cooling in the red-brick part is all off, because it's adjacent to the glass part. The electricity is independent in each part, so home automation becomes complex. And of course it is likely to look really, really weird. You need the big picture, the materials and approaches and how they will all connect together into a cohesive whole: the architecture, to be designed and managed at a higher level.

[00:02:24] Host: Okay, you finished there. I'm going to have to jump in and stop you, but thanks for that. So that was why architecture needs to be designed up front. What we're going to hear now from Tony is why that's all not true and it's better for the developers to take control. Over to you, Tony.

Round one: evolutionary architecture owned by teams

[00:02:44] Tony: Okay, I'm going to go a bit faster; I can't help it. Unlike building a house or a townhouse, where you have a plan, in product development you only have a vague sense. This may be because you've done extensive, or maybe not even so extensive, user testing, you've done prototypes, and you know what works. You might even realize your idea's awesome, but it's too early, the technology is not there, or perhaps you're too late [?] and you missed the boat. This is when you need your team enabled and empowered to make important architectural decisions.

[00:03:15] Tony: This is especially important in UI- and UX-orientated [?] products, where speed is important and teams need to pivot, perhaps several times, as is highly likely [?], even after the initial release, even if delivered after your first sprint or your first iteration. There's always a need to manage some tech debt [?], and you always need to ensure your software evolves with your customers and your systems. This is when you build what's called an evolutionary architecture. I call it a sustainable architecture; it has to live on.

[00:03:52] Tony: One thing I always say to people is: remember, everything has an architecture, whether you intended it or not. None of this is about the value [?] and the craft of the people who are involved in architecture. It's actually about moving the decision-making process closer to the decision points, and that's why a good set of architectural practices and principles is needed. Shift left isn't saying we don't need architecture. It's saying we are all architects to some extent. But we recognize, like most things in life, that there are those whose job and passion and expertise it is to keep the bar high for everyone, including your customers.

[00:04:30] Tony: Really, the trick is to do both, without the middleman [?], the separate function that was only occasionally plugged into the product. And I like easy straplines, so my strapline is: you don't know what you don't grow [?]. Perfect timing.

Poll one: teams win

[00:04:49] Host: Perfect timing [?]. So you've heard both options there. There's the argument that the architecture is not going to be as robust; it's going to become a Tower of Babel if everybody's doing their own different bits and pieces. And you have the other argument, that you can never get it right up front and it always needs to be evolving. We're just going to pop a quick poll up on screen now and ask you to respond: who do you think was more convincing? Should the teams be responsible for evolutionary architecture, which was Tony's point, or is up-front architecture a better approach, which was Nix's point? I'll give a few seconds for people to vote, and as I said, please do vote even if it's not your area. Let's try to get some lively debate on. A few more seconds.

[00:05:45] Host: Jovie [?], do you want to share the results? Oh, it was a close one, but we have it that the teams take it. So that's Tony one, Nix zero at the moment. That was great.

Round two: product development needs a roadmap

[00:06:01] Host: Now what we're going to do is a similar but related question. If we're doing architecture iteratively, and people have said they prefer that, is it better to do product development iteratively as well, or does that lead to more problems? Do you need to be thinking more up front about your holistic product development? To mix things up, I'm going to ask Tony to give the argument for why, after saying that architecture should be iterative, product development should not be. I'll hand it over to you, Tony.

[00:06:37] Tony: Okay. In order to build a product that meets your current and future needs, you need a plan. The roadmap is nothing more than a living document that sets the direction of travel. The entire point of it is to help you focus your priorities and delay making decisions. Yes, delay decisions: you don't have to make all the decisions up front. You do have to avoid [?] making the wrong decisions too early. It's not the point of a roadmap to commit you to eternally driving the bus at 50 miles an hour, where you're either committed or you die. I always ask what Keanu Reeves would do in these situations, and he says: jump off the bus.

[00:07:14] Tony: A roadmap is also not a strategy. Most of us don't have crystal balls, nor do we have the ability to predict the future. What we do have is an intent, with metrics to prove it, to deliver great value. It's a concrete and tangible thing that you have to have when you're dealing with the real world [?]. It's not "my roadmap is better than yours" posturing [?]; it's just being organized. And if you're mapping out the direction of travel and recording it as a little bag of milestones, it actually makes using mixed [?] teams simpler, not harder.

[00:07:52] Tony: The thing to accept is that your teams are built around specialisms [?]. You have to remember that every product is what I call a layering exercise. You have elements of the product that will run faster and earlier than others, and these typically require a range of disciplines to coordinate, which is always hard, because it's humans. Some elements take the lead and others follow, some join in later, and others leave at a later stage. Think of an orchestra, and imagine what would happen if everyone just turned up and did their own thing. That's why Spotify has chapters and guilds: people with expertise perfect their craft, they work with others to improve their craft, and everyone gets the benefit [?]. That's it.

[00:08:36] Host: Great, perfect timing once again, Tony. Now we want to hear the alternative argument from Nix, so I'll hand it over to you, Nix.

Round two: teams own the backlog and direction

[00:08:48] Nix: Thank you. Sorry. Shift left is about giving development teams the autonomy to move forward at pace, and the trust that comes with it. Of course, we need to provide them with a clear product roadmap that aligns to our business goals. We want a cohesive and consistent user experience across products built by multiple teams across multiple channels. The question here is whether you constrain or empower these teams. Do you give them blueprints, or do you give them scaffolding?

[00:09:20] Nix: Shift left is clearing the runway and letting the development team take off and soar. We need to allow development teams to be agile, to adapt to change, to innovate, to respond proactively to user feedback and shape their own product. If all of these things come top-down, that cycle is very long and convoluted. Feedback is in fact a readily available source of qualitative information, and as near as you're practically going to get to a direct channel between your users and your development teams. Things like GitHub issues, Play Store and App Store reviews, and UserVoice all provide direct feedback from real-world users. The development teams can continuously monitor it and, within the scaffolding of our product roadmap and UX vision, make rapid incremental improvements to the product based on what users actually want.

[00:10:12] Nix: And they can push this further by using beta channels or early adopter programs to gain early insights into how new features will be received. WhatsApp and Instagram, for example, provide open sign-up to users for their beta channels and use that feedback to shape their features' final release, rather than relying on quantitative feedback later on, like number of downloads. What this ultimately creates is a short feedback loop, with users directly driving improvements, the development team implementing them in small iterations, pushing them out to early adopters and applying that feedback to the final release, all within the boundaries of our scaffolding, but without having to go back and redraw the blueprints every time.

Poll two: teams take ownership

[00:10:55] Host: Great, thank you, Nix. Running over again; I might cut off your mic next time. Let's pop up the poll for this one. Just to recap: with product development, do you think teams should take full ownership of the product backlog and direction and rely on that feedback in quick iterations, which Nix has been talking about? Or do you think centralized planning eliminates the duplication of work and wasted dead ends? That's the Keanu Reeves roadmap that Tony was elaborating on. So is it Nix, option one, or Keanu Reeves, option two? I'll give everybody a few seconds.

[00:11:49] Host: All right, let's see what the results are so far. It's not so close anymore: teams should take ownership of the product backlog and direction. So we're seeing a trend. It's shift left. We have no bias towards the speaker so far; it's definitely a bias towards shift left.

Round three: platform engineering and security need dedicated experts

[00:12:06] Host: One of the bigger challenges I've often seen teams face is that architecture is challenging. It definitely is. But security is something that cannot be compromised on. Security is the one where a lot of people say, "Look, we can't compromise on security. We have to do the up-front design to make sure our system is as secure as possible." We don't want to end up in a Zoom situation, where we're scrambling to fix security after the fact. So what we're going to talk about is how we manage shift left and shift right with something as important as platform engineering, which would be a similar concept, or security, which is critical to get right up front. I'm going to start with Nix, who's going to say why shifting right is important for platform and security. Over to you, Nix.

[00:12:59] Nix: Thank you. A good engineering team has the potential to do anything. A good software engineer is both logical and creative, practical and innovative. They can pick up new technologies, ideologies and patterns and apply them to business requirements. So they have all this expertise in software development, and they're pointing it with laser focus at your business requirements. Awesome.

[00:13:25] Nix: Now, of course, you want to deliver that software through an automated build and release pipeline. So they give up a little bit of the space they use for software engineering expertise to accommodate some expertise in platform engineering: shared variable libraries, templates, release version management, rollbacks, canary strategies, configuration management. The list goes on. They can take all of that in their stride, but where their trajectory was SpaceX, it's now a little more Top Gun.

[00:13:51] Nix: But we also want our software to be secure. So the team implements secure transport, SSL offload, certificate management, encryption at rest, secure code analysis. They start to learn OWASP and threat modeling. By this point their trajectory is less Top Gun and more Thomas the Tank Engine. They begin to explain in increasingly exasperated terms that their delivery timelines have grown even though the business requirements haven't changed, that there are only so many hours in the day. Maybe they start quoting Bones from Star Trek. It's the kind of thing we do.

[00:14:24] Nix: There are also more complex and subtle aspects that require much deeper expertise. The team starts to get out of its depth, and to go any further means they need to shift their primary expertise from software engineering to somewhere else. It's not as much fun as it was anymore. They aren't doing what they love doing. Maybe they want to go somewhere else where they can. And what if every team is going through that same thing? Instead, it's better to provide that time and expertise through teams dedicated to it, who can work strategically and efficiently, so the development teams can get back to what they love doing, on that steep trajectory that they and your business find so rewarding.

Round three: embed the expertise in teams

[00:15:07] Host: Great, thank you, Nix. We've gone from Keanu Reeves to SpaceX, Top Gun and Thomas the Tank Engine. Great analogies so far. What we're going to hear now is the opposite argument from Tony, who's going to argue that we should delegate this responsibility to the teams, and why that's important. Over to you, Tony.

[00:15:30] Tony: Okay. Siloed [?] teams only work in very limited scenarios. Think back to my previous mention of Spotify chapters and guilds. The whole idea is that you have experts embedded in teams who bring their collective knowledge to the fore, and they work with teams to further their own collective experience of the craft. They work really well, particularly in the DevOps space. Being able to release early and release often is an ability all teams need to have, and as activity increases, anything else is just too much of a tax on speed; they only want to be able to release. So it absolutely makes sense to empower [?] anyone to do a release with confidence, because it should be easy.

[00:16:13] Tony: This frees up the DevOps person, the platform engineer as some people call them, to focus on improving the tooling [?], feeding back into the collective DevOps hive mind and raising the bar for all the teams working on different product areas, so the effort isn't wasted. In particular with security, what they're doing is on a much longer journey [?]; it takes a lot longer to mature. It often requires upskilling teams on some of the eccentricities of secure by design. But most importantly, it really needs solid product knowledge, product affinity, something that's often missing [?]. You need that if you have to take corrective or protective action, and you can't do that if you're sitting in a bunker 400 feet below the building.

[00:17:04] Tony: One other thing that is rarely mentioned is that recruitment [?] is pretty hard when you have centralized teams. It's often better to look within your group for people, running programs and training schemes, and you'll discover that there are a lot of people in there who are keen on [?] being a security person or a DevOps person. In many cases they get better outcomes than people you train in off the street. And I tried to have a COVID-19 reference, but I couldn't get Boris Johnson and shift left to work, so: go, Jacinda.

Poll three: dedicated experts win

[00:17:38] Host: Yeah, I think COVID-19 was a bit random towards the end there. I did like the lack of product knowledge point and raising the hive mind. I think they're great examples of the power that a product team can have. But equally, is that where they want to focus, or is it taking the fun out of what they're trying to do? We'll pop up the question now. Platform engineering and security: is it everybody's responsibility? That was Tony, saying teams are the people with the product knowledge and they can't be stopped from releasing regularly by having to go through hoops. Or is it so hard that it needs dedicated experts, taking away the boring stuff the teams don't want to have to worry about? Based on the arguments you heard, which do you think is better? Are our teams SpaceX, or are they more Thomas the Tank Engine? We'll give everybody a few seconds to vote.

[00:18:43] Host: Okay, so we have a shift away. We're saying now that when it's something as complex as platform engineering and security, it's better to have dedicated experts. I'd actually like to see, if we take that analogy into different areas, like qualitative research or design system maintenance, whether we'd still say dedicated experts are the better approach for product, research and so on. Okay. So we already went shift left on moving to the teams owning their products. Oh yes, but they own their products but can't release, because they have to go through centralized security.

Round four: buy off the shelf

[00:19:31] Host: On to our last question. We are going to have some time at the end, so please do write in some questions, and we'll get Nix and Tony to debate them as well at the end. Our last question is around build versus buy. This is a common question that comes up in a lot of organizations, where the core area of innovation is probably one area, but there are a lot of other things you need to do. It could be your CRM system, it could be your ordering system, it could be anything you need to support your business, but your core innovation, what differentiates you, is different. So the challenge becomes: do you build all those external pieces as well, so you have full control and they can be perfectly tailored? Or do you invest the time in your area of expertise and buy off-the-shelf products that might not be a 100% fit but get you 80 or 90% there? We're going to rotate again, and Tony is going to start this one, arguing for the buy option. Over to you, Tony.

[00:20:42] Tony: Okay. Folks, I've done [?] both: built and bought e-commerce systems, CMSs, CRMs, forum systems and frameworks. They all have their place. Both will work, but you have to be in the right context. I would really recommend Simon Wardley. Read anything by Simon Wardley about Wardley mapping [?]. He does an amazing job explaining when buy might trump build and when build might win [?].

[00:21:08] Tony: I always use the same analogy. When was the last time anybody decided, "Hey, I need a new car. I'm going to build one from scratch, get some metal and a welder"? You're not going to do it. You're going to go and buy one, get some spec options out of the factory, drive it, and then get another one. Unless you're a Formula One team, but they have very specific needs.

[00:21:32] Tony: In my opinion, most commercial off-the-shelf systems these days are very highly configurable and can be adapted to suit most needs. What often lets them down is that you make no real effort to adapt your business process or your requirements to accommodate them. The unwritten rule is the more you bend the vanilla [?]... Innovation can happen on a consistent basis [?], because nowadays most CMS [?] and commercial systems are headless, and you bring front ends to sync to the back ends, and we have strong back-end architectures that make that pretty easy. So always assess [?] your systems for how friendly they are for back-office users and for extensibility, and you'll find most systems are cool.

[00:22:30] Tony: One of the other things is talent acquisition. It's much easier to get people on board when you have a headless system. You've just got to avoid the people who sell you a product with no support or no ecosystem. If you've got a good ecosystem, you're fine. So don't buy what your CxO got sold on the golf course; buy what your engineers and architects have played with.

[00:22:47] Host: Great. I like the analogy of "you don't build a car," and it is true: I've seen a lot of "API first," and it's often a thing people are promoting. A lot of the configurable products are API first. Just for those who don't know, I'll explain: headless is where the product doesn't have the UI attached. It just has the back end, and you can build your own UI on top of that. Some of the audience might not have known that term.

Round four: build it yourself

[00:23:19] Host: Now that we've heard why you should buy products where it makes sense, we're going to hear the opposite argument, why it's always important to build your products because it gives you the full flexibility you need. Over to you, Nix.

[00:23:34] Nix: Okay. Commercial off-the-shelf software offers the benefit of dedicated teams of people and a depth of domain expertise behind it. But unlike a car, where everyone buys it to do roughly the same thing, commercial off-the-shelf software has to be made so generic, and able to do everything any business might want it to do, so that it can be sold across an entire market. Of course, that means that out of the box it cannot do what your organization needs it to do. It doesn't support your business, because your business is as nuanced as every other.

[00:24:15] Nix: What that means is that you need to spend months, or often years, customizing that software before you actually see any benefit from it. To do that, you need to bring product and domain expertise into your organization. The market for that kind of specialization is usually fairly niche, so hiring in-house resources is difficult and expensive, and you may not need them around forever. That's almost certainly going to bring you to consultancy services provided by the vendor, which of course form part of their business model. Not only that, but you have ongoing license, support and maintenance costs, which tie you into a long-term dependency on that vendor and the success of their business.

[00:24:57] Nix: Instead, what if you treated this as another product? You could bring in a small pool of domain expertise, use your existing software developers or hire more, and take an agile approach by building small iterations that you release into production through an automated test and release pipeline, which is something you're unlikely to get from a third-party vendor, in my experience. You start to reap the benefits very early on, and you can easily adapt to changing and emerging requirements as your business grows and continuously, inevitably evolves. So instead of doing it all up front, with big licensing costs and years of development, you could spend considerably less money, with less up-front investment, on a product that is tailor-made for your business needs and that you can benefit from very early on.

Poll four and the final tally

[00:25:43] Host: Great. So we've got the argument that you're going to save time, because products these days are very configurable, headless and API first. And you have the argument that you're going to get tied into an organization's support model, which tends to be quite expensive and challenging to deal with on a regular release cadence. We'll pop up our question. In the build versus buy debate: building software gives you the best flexibility, which was Nix's argument, or buying software lets you focus on your core area of innovation. So let's all see whether we want to buy our car or build our car.

[00:26:29] Nix: As long as you want to buy a car that doesn't do anything, and you have to build the bits that you want it to do.

[00:26:38] Host: I don't know if you've already influenced the jury. Okay, we'll see. People have gone for buying software, which lets you focus on the core area of innovation. I have actually been terrible at keeping score. We had Nix winning the first one, Tony the second one... I think two lefts and two rights, didn't we? So we have a tie. It was a great debate, though. I really enjoy these things, because a lot of the challenge I personally have from hearing talks is that somebody will give one side of an argument, and it's hard to hear what the two different sides are. So I think it's quite good to be able to hear people debate the different options.

Q&A

[00:27:35] Host: If anybody has any questions, while we're waiting for people to type them in, I'm just going to ask you: I know, Tony and Nix, you were debating either side of the argument there. We'll give you a minute each. Nix, I'll start with you, because you're bigger on my screen at the moment. What is your view of this debate? If you were to sum up, what would you say?

[00:27:57] Nix: I think for the most part, ultimately, it's a balance, and Tony and I both touched on that. It's not about exclusively moving everything into a team, or exclusively having everything centralized. One of the points I made in one of my sections was that it's about whether you constrain the team or empower the team. You can still have a center of expertise, you can have central teams, you can buy software in, you can have security teams that are dedicated to that. But ultimately it's how you arrange that, and the relationships and working practices you form, which mean the teams are empowered to deliver at pace, but do so by building on the expertise of domain experts, security experts and DevOps experts, and essentially just having a healthy balance between the two.

[00:29:08] Host: Great. Over to Tony, to wrap up.

[00:29:13] Tony: I quote Simon Wardley a lot, since I got my kickback from him [?]. One of the things you come to realize when you do this is that it actually does depend. There are things you can't buy a product for, and you just have to build it. But there are so many things you really shouldn't be building, because if you're spending time on that, you're missing the opportunity to build the thing that matters. What really matters is what you put in front of your customer. It's all about the product, and for me it's all about people. You need to use people to the best of their capacity and then let them do what makes them happy, and that makes them better than the system [?].

[00:30:02] Tony: That's why we have lots of excellent software engineers in this part of the world, and it's also where you have very fine people who can build and maintain custom back-end systems. You can move between both. You can actually have both in a large project or a large program set, and that's perfectly fine. You just have to understand that one size does not fit all.

[00:30:28] Host: Right. Just another question I was thinking of from the audience's perspective. We have a lot of people in product, in UX, in design. Is there any advice you can give them on how they should interact with the developers, particularly when it comes to developers taking over the architecture or the building, or suggest ways they should communicate to make sure things run as smoothly as possible?

[00:31:02] Nix: I'll put my hand up. A lot of effort has gone into doing UX in agile. As I'm sure many UX people have, if you haven't read Nielsen's eight levels of UX maturity, it talks a lot about how you drive those changes, but do so in an agile way or an iterative way, or some combination of the two. You always have to run UX a sprint ahead, or whatever, of your software development team. But ultimately it all comes down to that original XP approach: you want to have the people you need right next to you, so that you can talk to them. There are no delays in getting the information you need or in adjusting to changing requirements.

[00:32:15] Nix: It's the same with product management. If you have product management in an ivory tower, doing stuff and never talking to the engineering teams, it's a slow process. Adapting to change is very, very slow. If you have all of those people talking and having an open dialogue and adjusting as they go, in the true sense of an agile team, then it can work very well indeed.

[00:32:39] Host: Great. Well, thank you to both Tony and Nix for that debate. I thought it was quite useful, and I hope the audience did as well. We had a lot of people voting, which is always good to see.