Kessel Run: A Digital Transformation Story Within The World's Largest Bureaucracy

Adam Furtado at UXDX Europe. Video: https://youtu.be/q3tTovgwHVs

Readable transcript: edited from the recording's captions for readability (fillers and false starts removed, punctuation and section headings added). Wording is the speaker's own. Timestamps are positions in the video. Names marked [?] could not be verified against the audio.

Kessel Run and the real tools of modern warfare

[00:00:05] Good afternoon everybody. Great stage of speakers today; I'm humbled to be asked to come speak to you all. My name is Adam Furtado. I'm the chief product officer for Kessel Run. Named after Han Solo's famous smuggling route, Kessel Run is a United States Air Force digital transformation initiative to revolutionize the way the Air Force builds and delivers software. I'm not really going to talk much about DevOps, there isn't much about UX, and this talk is barely about technology, to be honest. It's mostly about organizational transformation and innovation adoption within the world's largest bureaucracy.

[00:00:41] I'm a former Air Force intelligence analyst. I have a couple of liberal arts degrees, and in some way that signals that technology is not the hard part. It's the culture change within it that helps us get through these digital transformations. Let me leave that there. I want to set the stage a bit.

[00:00:56] My wife watches awful television shows. The other day I came home and she was watching an episode of CSI: Miami. I have no idea if Ireland has the same TV show as we have in the States, but if you don't, you're not missing much. I walked in, and there were a bunch of agents huddled around this table that had holograms floating around. One guy swiped something up into the air, and we were knee-deep in some dude's Twitter account, and there were augmented reality devices around. It was wild. Anyway, she looked at me and said, Adam, if the police have this type of technology, I can't imagine what you used to use in the Air Force.

[00:01:31] I didn't have the heart to tell her the reality. But the reality is, the tools of modern warfare don't have holograms, and we don't have augmented reality. This is a picture of the Air Operations Center. All around the globe, the United States Air Force has these Air Operations Centers to plan, execute and assess air warfare, air operations, worldwide. Kessel Run was tasked to modernize the Air Operations Centers and all the work and tasks that are within them.

[00:02:03] The real tools of modern warfare within the AOC are the same tools that my wife and all of you use every single day: Microsoft Office. If you were to walk around this floor and look at all the computer screens, you would see a bevy of Microsoft Office products that had all the pertinent data needed to actually execute warfare. We would be storing our data in the same application your mother balances her budget in. We'd be building analytical products in the same application you wrote your college papers in. And we'd be visualizing that data and making strategic decisions off the same application that I'm using right now. But if I'm being even more accurate, it's probably the version that still comes with Clippy.

Software is eating the war

[00:02:50] Now, this is scary, though. Marc Andreessen famously said software is eating the world. We're learning more and more every single day that software is also eating the war. Software is an essential differentiator on the battlefield. Soon it's no longer going to matter how many tanks or aircraft or ships you have; it's all going to come down to your software. Whether we like it or not, whether we're prepared for it or not, software is now a weapon of war, and it's important that we cultivate that weapon so we can wield it successfully and responsibly. Earlier today the gentleman from Capital One was up here, and he talked about Capital One being a software company that happens to do banking. We're trying to turn the Air Force into a software company that delivers air power.

[00:03:31] Don't worry, it's going to get lighter; it won't be this intense the entire time. I think the reason we're here is that the Department of Defense's acquisitions community, the people who are tasked with providing capabilities to our users, have failed for years upon years to deliver meaningful capability on relevant timelines to the people who need it most. I know that because I used to be one of those users. In 2010 I was deployed out to an Air Operations Center, where I had the same capabilities available to me that we still have out in the field today, that are still being used today. Not just the same capabilities, the same versions. We're unable to deliver product into the field.

[00:04:11] General Mike Holmes, the commander of Air Combat Command, says: decades [?] of institutional risk aversion have led to the strategic dilemma plaguing us today, replacing our 30-year fleet on a 30-year timeline. That risk aversion realizes itself in a multitude of ways. We constrain ourselves with these bureaucratic safety nets that we think protect us from ourselves, but in reality they make us less safe, because we're not able to adapt to change. We constrain ourselves by treating our software organizations the same way we treat our hardware organizations, the same organizations that build and deliver aircraft and aircraft carriers, and the byproduct of that is you start receiving your software on those same timelines.

Twelve to fifteen years from idea to user

[00:04:56] This picture, I think, epitomizes our plight. This is the process by which the DoD acquires and builds software, or anything. The most interesting part of this slide to me, maybe the most humorous, maybe the most scary, is the fine print in the top right corner. It basically says this process is so complex and so convoluted you couldn't possibly fit it on a two-dimensional diagram. So try navigating that in 12-point font [?]. The fact is, nobody can navigate it. All of these practices are bad practice on top of bad practice, and delivering software on a meaningful timeline becomes an impossibility.

[00:05:38] Studies have shown that for traditional cycle times, the timeline we can get systems into the field in the DoD, the median time is eight years. That doesn't even take into account the fuzzy front end, your requirements and your contracting and all that. Our actual average time is 12 to 15 years to get an idea into a user's hands. An actual example of this: the program that set out to modernize the AOC before Kessel Run got involved was 10 years into their development and $500 million into their development, and hadn't delivered a single thing to the warfighter. In fact, they were scheduled to deliver in 2020 based on requirements written in 2006. We had a 14-year timeline between the idea and that thing actually being delivered.

[00:06:21] Think about what you knew in 2006, or better yet, what you didn't know in 2006. The first tweet was sent. Twelve years later, we're now sending 6,000 tweets per second. There's no way to know these things. This would be akin to trying to draft up the specs for the iPhone 12 before the first iPhone was even released.

[00:06:43] Unsurprisingly, other studies have shown that 96% of federal IT programs are behind schedule or over budget, and 40% never deliver a single thing. Recently Eric Schmidt, of Alphabet and Google fame, was testifying before Congress on the state of DoD innovation, and he said that the DoD violates every rule of modern product development. We are a literal case study of what not to do.

[00:07:09] I know what you're all wondering: if you guys don't know what you're doing and you're really bad at this, why are you the speaker that's keeping us from getting to the pub for happy hour? Because there's some optimism coming, there's some good news coming. And I think it should mean a lot, for those of you who live within bureaucratic organizations or highly regulated industries: if the DoD can do any kind of digital transformation, any one of you can as well. Even General Holmes's quote actually continued in a note of optimism. He said this dilemma also provides us an opportunity.

The tanker planning whiteboard

[00:07:47] That opportunity started in the fall of 2016, when the Defense Innovation Board traveled out to the Air Operations Center in Qatar. The Defense Innovation Board is a group of Silicon Valley CEOs and innovators, chaired by Eric Schmidt, who advise the Secretary of Defense on innovation efforts within the government and the DoD. They were traveling around the AOC to learn how we execute air warfare.

[00:08:10] Unlike commercial airliners, in the United States Air Force we refuel our aircraft in midair, at high altitudes, super fast, and it's super complicated to plan. There are only a certain number of tanker aircraft, and there are a bunch of aircraft flying that need fuel. Optimizing the scheduling for that, making sure that aircraft are at the right altitude in the right place with the right hardware, is super difficult. Eric Schmidt went and saw how we actually plan this within the Air Operations Center, and it looks something like this. There's a large dry-erase board with a couple of computers and a bunch of guys around it, mostly pilots who had been pulled out of their jets to do this job for six months, away from their families, deployed into harm's way.

[00:08:53] We had this first guy here who was tasked with moving these pucks around and connecting the missions that we had to fulfill with the tankers who were available. I like to think he looks something like this. As he's working through this plan, he starts shouting across the room at a guy they call the Gonculator [?]. I like to think he looks something like this. He's putting all this information into Microsoft Excel. There are some macros and scripts in there somewhere, but he's largely doing all of this in his head, due to his background, trying to make sure that we have enough fuel to fulfill all those missions. He's basically Beautiful Mind-ing it.

[00:09:24] When he feels comfortable with his plan, he yells across the room to somebody else they call the Thumper. He looks like this. He's taking that plan and manually inputting all this information into our legacy 14-year-old monolith, because it's the only way that we can get this data into the jet that actually needs it.

[00:09:43] They have this down to a science, but the thing they didn't account for was change. If there were major changes, they had two options. They could either erase the whiteboard and start all over again, which takes about an entire shift to do, or they could send up another tanker and make sure we were covering those missions with another aircraft. This led to an incredible amount of inefficiency, which Eric Schmidt called the most ridiculous thing he'd ever seen.

Learning from Pivotal and the first MVP

[00:10:09] This was the first project that Kessel Run was tasked with actually tackling. The problem was, at the time there were just a few of us who were tired of the status quo and wanted to do something new, but we had no idea what we were doing. None of us had any skills in modern software development or in how to manage a digital transformation. So we had to go to Silicon Valley and ask for help. We partnered with a company called Pivotal, and Pivotal Labs, to not only help us build a cloud-native platform for us to actually deliver capability to our warfighters, but to teach us to change the way we thought about products and the way we thought about our organization.

[00:10:42] We sent a small group of active-duty Air Force members to their lab in San Francisco to embed there. These were guys who we saw had a growth mindset and some rebellion in them. We thought that starting small with this little product was enough to keep us under the radar and just see if something like this could work within the government.

[00:11:03] Pivotal taught us a bunch of things: lean principles, user-centered design, DevOps, extreme programming, test-driven development, all these practices and processes that we all hold dear. But the most important thing they taught us was how to change the culture within your workplace. It's one thing to read slides and see talks on why culture is important, but it's another to see it in action, actually on the floor, actually seeing people collaborate, seeing ideation happen, and seeing people happy at work. I have colleagues at work that have been in government IT for over a decade and never seen anything actually reach the user, never seen anything be fielded. Imagine spending 10 years of your life and never seeing any value actually get to the place it needs to get to.

[00:11:47] A few months later we released the minimum viable product into the users' hands. We basically digitized the whiteboard process they were already using, with a lot of data validation in the back end, to make sure that we were helping them validate the amount of fuel and all that, and to try to find some efficiencies. And we did. We found some efficiencies right away. We got to a point where we were actually sending up one less aircraft on average per day. That sounds small, but it actually equates to $214,000 a day in fuel savings alone. That's not taking into account maintenance costs and crews and all of those things; that's just fuel savings alone. The actual investment we had made in this product at this point was $1.5 million. The application paid for itself in about a week.

Cost of delay and scaling up

[00:12:30] We had this win here, but we had to figure out how we could take this win at a small scale and articulate it to the powers that be within the world's largest bureaucracy. We tried to find some tools we could use to make the point even stronger, so we started using something called cost of delay. We made the argument that if we had waited five years to deliver the same exact product using our traditional waterfall development methods, we would have had a cost of delay of $384 million. Basically, $384 million the Air Force didn't have to spend would have been wasted.

[00:13:03] As we all know, software is never done. We continued to iterate. About a year later, 30 iterations later, we actually doubled these fuel savings to $428,000 a day in fuel savings alone. It doesn't stop there. We actually found efficiencies where we're deploying two fewer people to do this job. That's two fewer people leaving their families, going into harm's way and being pulled from their aircraft, so that one means a lot to me.

[00:13:28] From here we had this win; now we had to scale and solve other problems. Here we are today: we have 18 product teams, all delivering products that our airmen are beginning to love. We use a balanced team model, with product managers and product designers and software engineers, focused on lean principles, to make sure that we're building and learning the next most valuable thing in the simplest possible way. We're employing product designers for the first time. Traditionally in the DoD, you'd have some subject matter expert write a 60-page spec, they'd farm it out to a defense contractor, they'd find some front-end developers to fit everything they could into a screen, and usability was an afterthought. Now, bringing in user-centered design for the first time, at least in our world, usability is a first-class citizen.

[00:14:11] We're averaging, across all of our teams, 124 days from whiteboard ideation into our classified production environment. Remember, our benchmark was 10 years. With that, we're using extreme programming and test-driven development for the first time. We have over 80% test code coverage, which led to us getting the government's first-ever continuous authority to operate. What this means is, traditionally we would finish a software product, we'd give it over to the cybersecurity folks, and they would take about a year to 14 months to actually approve it to go on the network. Because of our pipelines and security processes and our test code coverage, they've granted us a continuous ATO to go directly into production every time we iterate, which is multiple times a week at this point. This led to us being the first Department of Defense team to continuously deliver into SIPRNet, which is our classified network.

Small wins, the platform, and going agile

[00:15:01] I have a couple of tips here if any of you are in the midst of a large-scale digital transformation. I wouldn't even call it that anymore. What you really need to focus on is getting a small, undeniable win that you can grow from, and building a little bit of runway. Sometimes I feel like, even a year and a half in, with some of the success that we've had, all my job is is to get one more win, to get me some more runway to keep going.

[00:15:24] One thing we learned here as well was not to forget about our platform, or whatever your delivery mechanism is. At first we got really distracted by the applications and the product. We weren't really concentrating on making sure that we were efficient in getting that into the users' hands. So we started putting an emphasis on our platform and our delivery, and actually doing the DevOps, and we saw a rapid increase in our software delivery performance. So much so that we're now deploying over 300 times a week to our staging environment and 30 times per month to production. We're at the point now where deploying is almost a non-event for us, which is where we want to be. When you get to this point where you don't have to think about delivery anymore, you can concentrate on mission applications that are above your value line.

[00:16:09] It wasn't as easy as I'm making it out to be. There was a lot of work along the way. Within the government and the DoD, and I think within industry, and I think Rory actually mentioned it this morning, a lot of organizations tout that they're doing agile, and it hurts the messaging a bit, especially in an organization that doesn't really understand it. In fact, for a long time, and in a lot of circles still, the government thought that DevOps meant having developers work with users, because in the Air Force we call our users and our warfighters operators. So there was a language barrier there as well. We had to spend a lot of time educating upwards and cultivating downwards. But now, because I think we've really concentrated on that fuzzy front end and the last mile, I feel comfortable saying that Kessel Run has gone full-blown agile. And just in case anybody's wondering, the AF here stands for Air Force. Promise.

Military culture and Project Aristotle

[00:17:00] But we still had a big rock to move. We had all these processes and technologies in place, but we had a full-scale cultural transformation and a people transformation that we still had to focus on. The interesting thing that we learned was that the culture needed to build effective teams for modern product development and the military culture, as antithetical as it sounds, are actually pretty similar. Google did a study called Project Aristotle, where they focused on the team effectiveness traits that drove teams to be really good performers, and it came up with these five things. It's interesting that the first four of these marry up very well with military culture.

[00:17:42] Dependability. On the battlefield, dependability is a necessity. You're relying on the guy next to you: never leave a man behind, all of those things. At a different personal scale, we feel the same way about our balanced team model. We have PMs and engineers and product designers making sure they're representing all of their equities, to share product decisions and make sure that they're all going towards shared outcomes.

[00:18:03] Structure and clarity: again, another necessity in the military. The way that we use extreme programming and test-driven development, and the way that our office runs, is very disciplined. We get a lot of visitors from senior leadership that come in and get distracted by the stickies and the whiteboards and all those things. It looks like a Wild West environment when you're in there, but in reality, when you dive into these practices, they are very disciplined, and it takes a lot of structure to make them work. So it actually is a really nice way to work for our military guys, who have that ingrained within them.

[00:18:34] They said you have to have meaning in your work. I think this one we have in spades. A lot of people ask about retention. Famously, the government can't pay the salaries that industry can, especially for software developers right now. So we have to make sure that we can provide meaning in your job, and the workplace culture, to try to make up for that as best we can. Recently I was walking in San Francisco and I saw a billboard for an app, sort of like this. Basically the app was for dog walkers, so you can optimize their route so your dog could pee on grass rather than concrete. Good idea. That company can pay more than I can for a software developer, but I feel very strongly that when our guys go home from work, they think about the impact that they had on the world, the safety of their country, and we take that very seriously.

[00:19:21] The same goes for having an impact. If we think about what a warfighter is, it's people who are deployed away, who are putting their country on their back and making sacrifices. While there are different levels of that, I like to think of our guys as the modern warfighter. Yes, they're in t-shirts and jeans in a lab in downtown Boston, but they're sending features, they're sending their work, directly to the front lines, and changing the way that we think about being able to react on the battlefield. Continuous delivery does a lot for us to be able to react when things change, and they're having a significant impact.

Psychological safety and a learning culture

[00:19:55] But the interesting one is this last one, this idea of psychological safety. This is the one that we don't have at all. The military is based on this hierarchical, top-down leadership, which is imperative to warfighting and to being successful on the battlefield, and a really bad way to build really good products. In fact, the whole idea behind having uniforms is to strip away your individuality, when in reality that individuality is what we need to solve complex problems in a creative way.

[00:20:24] How have we combated this within Kessel Run? A few ways. One, we got rid of the uniforms. We got rid of the ranks; everybody talks on a first-name basis. Our balanced product teams really are balanced. You could have a lower enlisted product manager working with a higher-ranking officer, and they're talking on a first-name basis, and they feel safe to fail, free to ideate, free to give you their opinion, which is new for the military. We focused on this a lot. We focus on building a feedback culture. In the military, feedback is inherently negative: if you're getting feedback, you did something wrong. We're trying to change that model within our organization, so people can find their blind spots and want to learn and grow and get better.

[00:21:08] The biggest thing we learned from here is that we had to build a learning culture. We had to build an organization where all we wanted to do was learn and get better and continuously grow. That's been really difficult, but it's something we've been prioritizing. Admiral John Richardson, the Chief of Naval Operations, the highest-ranking officer in the Navy, said it's about the team that can bring the people, technology and processes together to learn the fastest; that's the team that has the advantage.

[00:21:32] When I started this job, I focused all of my energy on people, technology and processes. I thought that was the right way to go: make sure we're building our practices and deploying our product. And I missed this last part. The idea of learning the fastest is really the real takeaway here. The first three things are enablers for us to do that, but getting to the point where our organization is constantly learning is the real place we want to be. In fact, we opened our lab in Boston and we call it the Kessel Run Experimentation Lab. We call it that for a reason, because we really believe that everything we do is an experiment, and we want to promote that within our organization. It's our island of innovation, away from the bureaucracy that we're trying to fight.

Ten years behind is disruptive

[00:22:08] Now, I understand where I'm at right now. I'm at a tech conference with a bunch of experts in DevOps and UX and all those things, and I'm trying to tell you how innovative we are because we're using practices and processes that you all have probably been using for 10 years. But being only 10 years behind industry is disruptive, and is productive, in government. I know that because I joined the military when I was 18 years old, and I've been a government civilian all the time after that. This bureaucracy, the world's largest bureaucracy, is all that I've ever known. I never thought that we would be able to get to this point, especially this fast, and we've done that with a serious approach to culture, a dogmatic approach to learning, and never taking no for an answer.

[00:22:51] So why did we name this initiative Kessel Run? It's because we had to smuggle this way of thinking, this culture, into the DoD. We were a small group of rebels flying as fast as we can on a mission to do what was right. The difference is, in Star Wars, seven or eight movies in, the Alliance is in shambles with no hope in sight. But in our case, our rebellion has turned into a revolution, and the dark side is beginning to see the light. Thank you.