To fight the wars of the future the US Air Force tasked a small group of software engineers with a simple job — revolutionize the way the military thinks about software development.
The group tasked with this not-so-tiny problem came to call themselves “Kessel Run” after the famed smuggling route used by Han Solo in Star Wars.
Since starting in 2017, the team at Kessel Run has expanded to include over 1,300 people across multiple locations, helping build, test, deliver, operate and maintain cloud-based infrastructure and warfighting software. These applications are used by airmen worldwide and represent the future of warfare.
That’s because the wars of the future will be fought with software and system architecture as much as any other weapon.
What is Kessel Run?
“[Kessel Run] was kicked off about four years ago as a way to prove that the Department of Defense didn’t have to be terrible at building and delivering software, regardless of being within the world’s largest bureaucracy.” — Adam Furtado, on the Dev Interrupted podcast at 1:35
As an Airforce organization, Kessel Run delivers a wide variety of mission capabilities to warfighters around the world, utilizing industry best practices around DevOps and Agile. At the time of its inception, it represented such a radical departure from the normal way of thinking within the Department of Defense (DoD) that people joked it would have to be “smuggled” into the DoD. That’s how Kessel Run came to earn its name — a scruffy team outfitted with a mission to upend a stodgy and cumbersome bureaucracy.
A shift in thinking needed to start with culture. The team at Kessel Run decided to bring a startup like mentality to the behemoth that is the federal government with a goal of introducing modern software methodologies at scale. Pockets within the DoD were practings things like continuous delivery, but prior to Kessel Run, previous attempts to adopt modern software principles had largely failed. Warfighters weren’t getting the capabilities or tools they needed.
Problem Solvers
One of the biggest institutional problems that Kessel Run was tasked with trying to improve were the Air Force’s Air Operations Centers. Spread around the world across twenty two locations, these organizations manage all the details that involve fighting an air war. Everything from strategy, to planning, to tasking aircraft to perform certain actions, to providing real time intelligence data and feedback, are handled at Air Operations Centers.
The challenge was modernizing these centers while maintaining operational readiness and current hardware — much of which was 20 to 30 years old. All of the hardware across these locations came with its own integrated software, built from various third party sources over decades.
To tackle this challenge the team at Kessel Run applied the principles of Gall’s Law, which states that all complex systems that work evolved from simpler systems that worked.
By starting small and focusing on rapidly achievable solutions, they began to see the network effects of their actions. Small, precise fixes can have tremendous impacts on an organization and are less prone to failure than attempting systematic change overnight.
“So we knew that using Gall’s Law in history, that we needed to start small, in order to make this work. We couldn’t just have a big bang approach to replace this entire system. Right? You did that by chipping away at some core parts of the system from a user functionality perspective.” — Adam Furtado, on the Dev Interrupted podcast at 13:31
Practical Success
One of the first small changes achieved by Kessel Run was with the Air Force’s air refueling program.
A remarkable acrobatic feat performed at more than 20,000 feet above ground, at speeds close to 400 miles per hour, replenishing the fuel of an aircraft is dangerous but necessary work. Everyday, fighter jets and bombers rendezvous with fuel air tankers to perform air-to-air refueling before continuing on with their mission.
Optimizing the details of such a delicate dance would be difficult, but the folks at Kessel Run believed they could do it. First, they needed software engineers. One of the problems of developing software at the federal government is a lack of engineers. Or rather, a lack of native engineers that can be found in-house. Historically speaking, the government outsources everything to contractors.
Scrounging the Air Force for active duty software engineers scattered across separate programs, Kessel Run was able to stitch together it’s own homegrown software engineering team.
With their mission in hand, they set to work building an initial application nicknamed “Jigsaw” to improve the air refueling process. By optimizing every aspect of the process from the timing to the altitude, Jigsaw became an enormous success. Within a year of implementation the Air Force was saving $12.8 million a month on fuel.
Tiny, targeted successes like these continued. But Kessel Run was up against more than just inefficient programs.
A New Way of Thinking
Changing company culture is notoriously difficult. Changing culture inside the world’s largest bureaucracy is as hard as it gets.
The most difficult problem that Kessel Run had to tackle wasn’t the lack of software developers, or the difficulty of integrating third party software applications, or figuring out how to optimize and build combat applications, it was how to communicate with their peers in the DoD.
Part of the difficulty was due to the security implications of such work. The production environments are all on classified systems, making things like cloud implementation and tooling availability difficult.
However, navigating the business side of the DoD was always the most challenging. In the past 30 years the government has spent over a billion dollars trying to update their systems to provide the best capabilities possible to warfighters in order to prepare for a war that may never happen.
Until Kessel Run, the government didn’t have much to show for their efforts. A perception existed that new software methodologies and practices were just the next iteration of technologies that overpromised and underdelivered. It took a lot of trust to explain that doing something in a more agile way or using DevOps, would actually reduce risk and increase success for the organization.
“The problem we have is we go and talk about how deployment frequency is going to buy down risk for us. That sounds counterintuitive to everybody in the world, particularly in a military environment, where they’re like, ‘What do you mean? Change is scary. I don’t change stuff.’ So we’re having these kind of counterintuitive conversations around why moving to this way of working is less risky and increases our chances of success.” — Adam Furtado, on the Dev Interrupted podcast at 6:39
Solving that problem came down to nothing more than old fashioned relationship building. It took years of evangelism and continued success, but eventually Kessel Run started to win the approval of the right people in the right places.
Proof is in the Pudding
From starting as an organization with only 5 software engineers, to expanding into a program that currently has over 1300 people, Kessel Run has proven itself to be an ingenious concept: bring startup culture to an old organization in need of modern ways of thinking.
Government has never been the place that attracted the top talent in technology, but with Kessel Run it’s become that. They provide access to the newest technologies, competing with some of the best companies in the industry.
They do have one ace up their sleeve when it comes to hiring: fighter jets. And those are pretty cool.
If you want to learn more about the history and story of Kessel Run, consider listening to the Dev Interrupted podcast featuring Adam Furtado, Kessel Run’s Chief of Platform.
Dev Interruptedis a weekly podcast featuring a wide array of software engineering leaders and experts, exploring topics from dev team metrics to accelerating delivery.
If you haven't already heard, Dev Interrupted is partnering with Dzone to host INTERACT: An interactive, community-driven, digital conference on September 30th - by engineering leaders, for engineering leaders. 1 day, 10 speakers, 100s of engineers and engineering leaders, all free.
Register Now
Join the Dev Interrupted Community
If you haven’t already joined the best developer discord out there, WYD?
Look, I know we talk about it a lot but we love our developer discord community. With over 1600 members, the Dev Interrupted Discord Community is the best place for Engineering Leaders to engage in daily conversation. No salespeople allowed. Join the community >>
Originally published at https://devinterrupted.com on September 7, 2021.
Top comments (3)
Really cool program. Who says government has to be lame?
The only thing that would be lame would be pretending Han didn't shoot first. He did and we all know it.
It can be very difficult to communicate why moving fast is a good thing. I can also tell you from personal experience as a former DOD type person, what is described here is quite miraculous.