Software Engineership - How To Think About Software

Savvas Kleanthous, Feross Aboukhadijeh, Lisa Crispin, Nix Crabtree, Dave Farley at UXDX Europe. Video: https://www.youtube.com/watch?v=kceuSd6KlEg

Readable transcript: edited from the recording's captions for readability (fillers and false starts removed, punctuation and section headings added). Wording is the speaker's own. Timestamps are positions in the video. Names marked [?] could not be verified against the audio.

Introductions

[00:00:00] Nix: Welcome to this panel session on Software Engineership. As Frank said, I'm Nix Crabtree, I'm the Lead Principal Software Engineer at ASOS. Part of my role there is to constantly make engineering at ASOS and our engineers better, and a key part of that has been and still is software crafting. So it gives me particular pleasure to host this discussion, especially with the four fantastic panelists joining me today. Let me introduce them to you.

[00:00:33] Nix: We have Dave Farley, who is the co-author of, you all heard it, Continuous Delivery, with Jez Humble, and the founder and managing director of Continuous Delivery Limited, where he works as an independent software consultant advising companies around the world on the topics of continuous delivery, team organization, software development process, automated testing, software design for high performance, and software design in general.

[00:01:02] Nix: We have Savvas Kleanthous, Head of Engineering of ParcelVision, with expertise in domain design, event sourcing, CQRS, high throughput, low latency systems, scalability, software analysis, design and architecture. Lisa Crispin is a Quality Coach at OutSystems and helping to build an observability practice. She's been a hands-on tester on agile teams since 2000. She's a co-author with Janet Gregory of three books, Agile Testing, More Agile Testing and Agile Testing Condensed. She and Janet are co-founders of the Agile Testing Fellowship, offering the Agile Testing for the Whole Team training course, in person and virtually, by training providers around the world.

[00:01:52] Nix: And finally, we have Feross Aboukhadijeh, who is an entrepreneur, programmer, open source author and mad scientist. Feross has built a number of innovative projects, the latest of which is Speak Easy, kind of like virtual speed dating meets professional networking. I hope people have seen that and tried it out during UXDX this week. If you haven't, please do check it out. Thank you all for joining me today.

Does "craft" paint the whole picture?

[00:02:20] Nix: I'm going to start with Dave. We hear the term software crafting a lot, and that has the connotation of an artisan, measured by honing particular skills or years of experience. For me it's a great way of reframing how we approach writing software. But does that term paint the whole picture for you?

[00:02:45] Dave: I don't think it does. I don't think it quite goes far enough. The first thing that I should say is that I think that software craft was a great idea, but I think it was an important idea that moved us forward from where we were. I think that we suffered from several decades under the heavy yoke of trying to apply production line thinking to software development, and it's not that kind of problem. Craft is a much better fit for the nature of our problem, which is always a problem of discovery and learning and so on.

[00:03:26] Dave: But as you said, when you were describing craft, craft is kind of by definition limited to the kind of human capability, and that's fantastic. But what engineering does in general, in other disciplines, is that it takes craft and it amplifies it so that we can go beyond human limitations.

[00:03:55] Dave: I think that we need to retake the term software engineering. I think we need to reframe it so that it really is an engineering discipline for software development. In almost every other field that I can think of, engineering essentially means the stuff that really works. In software development we've come to assume that what software engineering means is that overly academic, overly bureaucratic stuff that other people do. And I don't think that's correct. I think that we can amplify all of the good things about craft by applying a little bit of engineering discipline on top of the craft. And that's really the stuff that really interests me.

[00:04:43] Nix: Yeah, and in fact it was Margaret Hamilton, who was working on the Apollo program, who coined the term software engineering.

[00:04:51] Dave: Indeed. I'm currently in the middle of writing a book which is about this topic, really. I did a little bit of research about the history of software engineering, and it's fascinating, because there was a movement back in the late 60s that we would think of as kind of advanced agile thinking. There was test driven development described by Alan Perlis in 1968. And there were people doing this kind of really discrete iterative feedback-led experimentation to build software systems. And that's some of the stuff that Margaret was doing to build the flight control systems for the Apollo program when she coined the term.

[00:05:37] Dave: So I think that engineering has got a bad rep in our sphere, and I think that's a mistake. What we tend to think of when we talk about engineering is production engineering, which is a different kind of thing. We are design engineers, and design engineering is about learning, experimenting, trying stuff out, breaking things, finding what works, what doesn't. I am currently completely obsessed with watching Elon Musk blow up spaceships as he tries to evolve towards a rocket that can go to Mars. He's destructively trying out his ideas, and that resonates with me. That seems what real engineering at the edges of knowledge is like. And that's what I think we should be doing.

Shared understanding of the domain

[00:06:19] Nix: Thank you. Savvas, we know that combining our crafting and our engineering skills is essential for writing high quality software, as Dave just outlined. But does it stop there, or is it also important for us to build a shared understanding of the domain we're working in as well?

[00:06:50] Savvas: Thank you, Nix. It's a really good question. In my career I started as a software engineer, I moved to more senior positions like a team lead and a tech lead. Lately I've been working as an architect and the head of engineering for ParcelVision, and through that time I got a chance to experience product building from different perspectives. At the same time, I got to speak, in any product, closely with people from varied backgrounds within the same company building the product.

[00:07:26] Savvas: I think the turning point in my career was when I actually internalized that products are never, well, very rarely built by a single person. In the vast majority of cases a product is built by a team. And while it's important for everyone who is participating in building a team, be they the software engineer, be they the subject matter expert, or anyone really, to be the best that they can do, and software engineers need to perfect their craft, fundamentally it is the team that is building the product, and it's the team that needs to perform well in order to build a successful product.

[00:08:03] Savvas: Exactly because of that, in order to be successful in building the product we need to build a shared understanding of the behavior that we want to deliver to production. We need to have a shared understanding of the goals that we have as a team. We need to have a shared understanding of the problems and the opportunities that we have in that product as well.

[00:08:23] Savvas: Moreover, as I mentioned, it's extremely rare that the development team is actually the subject matter expert of a particular product. What happens mostly is that the development team is working with other people to understand how the product has to behave. A favorite quote of mine is one from Alberto Brandolini, the creator of Event Storming, which is: it's not the domain expert's knowledge that goes into production, it's the developer's understanding and interpretation of that knowledge that goes into production. And I think that uniquely describes the situation, exactly why your understanding is quite important.

[00:09:02] Nix: Yeah. We use Event Storming at ASOS and we've used it to understand some particularly complex domains and workflows. Each of these worked really effectively. It is an investment in time and effort, and you need a team that is happy to get in a room and spend potentially hours writing out innumerable post-it notes of different colors, but the result is quite astounding.

[00:09:41] Nix: One of the exercises we used to train people in how to start Event Storming is to event storm PacMan. Of course you get a room full of developers and they all say, "Easy," and then an hour later they're scratching their heads going, "Wait, but how did the ghosts catch PacMan?" And then you start to work out that every time PacMan eats a dot he slows down by one frame, which is enough for the ghosts to get in. And then, where does he turn at the end? It actually becomes quite pleasurable as a software crafter, as a software engineer, software developer, to challenge yourself, to think things through to those details. But that's very right at the implementation phase. There are different levels of Event Storming that we can use?

Big picture Event Storming

[00:10:38] Savvas: Yes, of course. Perhaps the most important and the most useful version of Event Storming is the big picture Event Storming. As you have described, you find those little surprises when you actually move to what is at least close to the implementation level. But at least from my experience, most of the companies that I have worked with, especially when they tend to become a little bit large, there is very little visibility on how things are, how different teams work with you.

[00:11:16] Savvas: That is something that I think we lost in the translation from the very old waterfall processes, which was definitely something we wanted to move away from. But during that migration from waterfall to agile, it seems that a lot of companies lost their way and went a little bit further ahead, I guess: no design upfront and no visibility of what goes on. And that's kind of the problem that we're seeing, a lot of people not knowing how other things work. Work being repeated across the teams, problems being solved on one side of the fence but not solved on the other, lessons not getting transferred across.

[00:11:53] Savvas: So I think that big picture Event Storming is actually quite useful, because it allows everybody to understand how the domain works. Not so much the software itself, but actually how the product works together. What kinds of problems do we have? What kind of end to end workflows exist? So not close to software, but fundamentally what the users are doing, how things are working internally, and when to fulfill a need.

[00:12:25] Dave: First, that was a really good description. I was recently doing some work to try and synthesize some recommendations for junior software developers, and they just came up with a bit of a meme that I liked. I think it kind of captured what you were just describing. Fundamentally, our job is not to write code, our job is to solve problems. So problem solving is more important than design. Ultimately people care less about design, more about the problem solved or not. And design is more important than coding, and coding is more important than languages and tools. That's kind of the order in which I think I would rank our skills as practitioners.

[00:13:10] Dave: At the top, we need to be problem solvers. You talked about investment earlier on, the time that goes into event sourcing. That's an investment in understanding so that you can better solve problems, which is never wasted, or usually not wasted, unless you're getting into analysis paralysis.

[00:13:26] Nix: If we're not solving problems when we write code, then effectively we're just glorified data entry clerks. Great, thank you for that.

Experimentation and observability

[00:13:38] Nix: Lisa, a crafter working maybe with expensive materials, or an engineer working with complex mechanisms or structures, will probably start by experimenting with mock-ups before they commit the time and materials to building a final product. Is experimentation something that could also bring value to the way we write software?

[00:14:09] Lisa: Yeah, I'm a huge fan of what Linda Rising calls small frugal experiments. I think it's great to start with the low fidelity paper prototypes or virtual whiteboard prototypes, and we can test those. Of course, I come from the testing perspective. We can test our feature ideas, and then we can use techniques like Event Storming to identify what's a thin slice, something MVP if people want to call it that, that we can use as a learning release to build a hypothesis. We're solving problems, as Dave says.

[00:14:48] Lisa: So what problem are we trying to remove for the customer, as opposed to adding a feature, and how will we measure that? And then again, as Savvas was talking about the domain knowledge, I think that's where the domain knowledge comes in. We have to understand it well enough to know, what do we want to learn? How do we measure that? How do we get that data when we put it in front of some customers to get feedback?

[00:15:11] Lisa: And that's where we need to be smart about how do we instrument our code to capture that data, make our code testable but also operable, so that we can set up ways to look at the data from production. I'm a big fan of observability and trying to learn about that, because we have to understand exactly what are our customers doing and how are they using these features. Is it solving their problems? And take these really short small increments of learning and then slowly build it up.

[00:15:41] Lisa: And so we know, that wasn't of any value to anybody, let's try something else. Those small experiments where you have a hypothesis and you have a way to measure it. I think that's where teams really struggle: how do we measure it? How do we know when we're successful? But I think that iterative processes are really the way we build good things for our customers.

[00:16:03] Nix: And is that something that exists just within the team, or does it cross boundaries into UX or architecture? Are there experiments that we can do hand in hand with other areas?

[00:16:20] Lisa: Well, I'm a fan of a whole team approach, so I would hope all those people are on a team or at least collaborating really closely. If we're a software delivery team and we don't have the designers on the team, or we don't have the architects on the team, we need to build those bridges to those people, establish those relationships, get their help, and really work closely with them, because I don't think anybody can do this in isolation. I think we have to all be working together on it.

[00:16:51] Lisa: And I know in a big organization that gets more challenging. I'm in a big organization for the first time. But you can still say, "Hey, architect, we need to have a conversation. Please come help us think about our design, or think about how we want to implement this," this thing we think will solve a problem for our customer.

[00:17:08] Nix: We talked about observability in your intro. Is that something that can be part of experimentation, or is that something that comes late?

[00:17:27] Lisa: I think you need it for the experimentation. You need to get the data, and to start to get the data you've got to capture the events that happen as people use the system, or as other systems use this, whatever the user is, and be able to understand that data and drill down. It's like, "Oh, looks like there was a problem here," or, "The performance was really strange here," and be able to dig into it quickly and be able to respond to it quickly.

[00:17:51] Lisa: Hopefully, if you're doing a small experiment, hopefully you're just giving learning releases to a small number of customers, and so you're not causing very many people pain if things go wrong. But being able to respond quickly, roll things back or fix things quickly, is really important, and that's where the observability comes in. We can't anticipate everything users will do. We can't replicate our production environments in our test environments, there's just no way to do that.

[00:18:16] Lisa: And so, to be able to get the data from production, to be able to ask the questions of our production system that we didn't know in advance we'd have to ask, because we didn't think of them in advance — we can't know everything in advance. This kind of fills that out, and I see it as part of testing. We can test all we want to before we release something, but we need to be able to still learn from production and respond quickly there, and be able to test safely in production. We can do that today too, with all our great technology. We have so much technology now that supports our ability to do these kinds of experiments.

Feedback, and plans meeting reality

[00:18:50] Nix: Okay, thank you. Feross, the story we don't usually hear about either crafts or engineers is what happens after they create their piece de resistance. Does a civil engineer in this modern age go and check if hashtag worst bridge ever is trending on social media? Is feedback the critical point at which we actually surpass these analogies, to inform and adapt to essentially a living, evolving product?

[00:19:41] Feross: I think that's right. I think that the main difference between a civil engineer and a software engineer is obviously that the civil engineer doesn't really get a chance to redo or fix up their mistakes in the same way that a software engineer does. There are those stories of bridge failures that led to structural problems and attempts to retrofit things after the fact, and it's not ever a good place to be in. But with software we don't have those same constraints. It's very easy to deploy code constantly. So that definitely changes the way that teams should think about how they build stuff.

[00:20:26] Feross: I think planning is still super important in the engineering and the software context, but I'm personally a fan of much more just spending as little time as possible in the planning phase and just getting something out there. Of course, I'm a little biased because I tend to build a lot of things on my own or on very, very small teams, where there's not as much cost to getting stuff wrong. So that may be different in scenarios where you have larger teams coordinating with each other and where the cost of making a mistake is higher.

[00:21:06] Feross: But I think there's only so far that planning and preparation can take you, because once you encounter reality, the reality is going to show you how your plan was wrong. That's not to say that it's not worth having a plan, but I think the plan will never survive contact with reality. So I like to spend as little time as possible to get the idea out into the world and to get the initial implementation out there, because you can only learn so much sitting in a room pontificating about what could happen when you release this.

[00:21:44] Feross: Certainly from a product perspective, it makes sense I think to just get it out into the hands of your users and see what happens, especially if you're in an area where the cost of building something and getting it into the users' hands isn't that long. If it's on the order of a couple of weeks or a month or so of building, then it might be easier, rather than spending months and months thinking about what could happen, to just put it out there and see what people do with it.

What launching Speak Easy taught us

[00:22:19] Feross: I think there are just a lot of examples in my career where I've experienced that just putting something out there taught me way more than I could have learned if I kept just trying to guess what users would do. A really recent example from Speak Easy, for those of you who have had a chance to use it, it's a video calling platform where you get matched one-on-one with people and you get a chance to talk to them. The initial version of the site was this giant public Speak Easy where everybody would get matched with everybody.

[00:23:03] Feross: That was kind of a wrong decision, because there's not really context if you're just matching a random person from some random place in the world with you. There's not really a shared context that the users have. That's probably something I could have realized before building it, but there's nothing like seeing a user getting matched to someone who literally doesn't speak English and being like, "Oh yes, this is a problem. Okay, all right, let's rethink this a little bit."

[00:23:28] Feross: Maybe that one was predictable, but there are other examples where I don't think I could have predicted it, it seems easy in hindsight. I noticed this user behavior where people would block their webcams like this with their finger and chat with each other. And I was like, this seems like users misbehaving, users not talking on equal footing with the other users that they're talking to, and so this is a behavior to be discouraged or banned. But it turns out — I actually just used the product to talk to some of those users, and I learned that actually they really just wanted an audio option.

[00:24:13] Feross: They were like, it's kind of obvious when you think about it. They're just trying to use the app without showing their face, they feel more comfortable talking and answering the types of questions that were being posed in some of the Speak Easy with their video off. And so that was a very clear thing where it's like, "Oh, an audio option actually makes a lot of sense." Whereas before I launched it, I felt really strongly that we didn't want to have audio, that we wanted to have face to face contact between everybody.

[00:24:42] Feross: I'm reminded of the quote from The Matrix, from Morpheus, where he says, "Neo, sooner or later you're going to realize, just as I did, that there's a difference between knowing the path and walking the path." You've got to put your stuff out there and see how people take it.

Favorite line of code

[00:24:52] Nix: Thank you. We're going to go to audience questions in a second, but before we do that, I want to ask all of the panelists: what is your favorite line? Let's start from the very beginning. Dave?

[00:25:15] Dave: Yeah, this goes back a very, very long time. I used to have a ZX Spectrum, and there was a book in the ZX Spectrum realm that, if you did a plot command, a very esoteric plot command, just one line, it used to build this big kind of Mandelbrot-more[?] pattern on the screen. I used to know it off by heart, and I could just kind of go up and type it in and just take over the screen on a ZX Spectrum. It was just one line, and I was always dead impressed with that.

[00:25:41] Nix: Savvas, how about you?

[00:25:47] Savvas: That's a difficult question for me. I tend to work quite a long time on the products that I'm involved in, and most of them, in my career, I ended up deleting more than I contributed at some point. So I tend to not get really attached to the pieces of code, because I probably will end up deleting everything. Well, not everything, but at least every individual piece of code will change eventually in the lifetime of the product. I don't think I have, unfortunately, a favorite line of code. I tend to have products that I like, or pieces of libraries that I wrote that I enjoy.

[00:26:23] Nix: Thank you. Lisa, how about you?

[00:26:34] Lisa: Yeah, sorry, I muted myself. Savvas made me think of, he talks about deleting code. I think that's really a great thing to do, we don't do it enough. But I guess I haven't been a programmer on my own for a long time. Pairing with developers, I guess my favorite thing has been to pair with the developer to analyze a so-called flaky automated test that was marked flaky, and so nobody paid any attention to it, only to find it really was a bug. It was just a bug that didn't happen every time and was hard to reproduce. It was like maybe a timing issue. Being able to fix those and prevent production regressions was very satisfying.

[00:27:13] Nix: Thank you. Feross?

[00:27:18] Feross: So I'm going to cheat a little bit. I don't think one line of code can be that interesting, so I'm going to say my favorite hundred lines of code. One of the projects I've worked on was, I was trying to figure out what is the most annoying website one could build if you were to use all the different web features that the browser now affords us. With HTML5 and all these new powerful APIs, there's actually quite a lot you can do in terms of putting together all of the features to make the worst possible website experience.

[00:27:51] Feross: There's about a hundred lines of code I wrote, if you want to go see it. I don't recommend doing it in your primary browser, because you may actually need to force quit your browser to escape the website. It's actually that bad. But maybe you write this URL down and try it out later. It's theannoyingsite.com. You type that in and hit enter and you'll have a very interesting experience.

[00:28:16] Nix: If someone asked about my favorite line in coding, it was actually — I was working at a quite prolific online video chat messaging tool at the time, and we were building the web presence. There was a big desktop presence already. And I noticed that every request, which was potentially something like 600 million users potentially, every request went through this thing, and all it did was it just reformatted some bits from the request into a path to go and pull a JavaScript file from blob storage.

[00:29:06] Nix: And it bugged me, because it just wasn't particularly efficient. I could see this, it was doing some string concatenation, and I was like, it's not a bug but it bugs me. So I changed it, and it was about 30 times faster, and that basically impacted every single request that ever went through this entire system. And so that was my favorite line.