This post was co-written by Ken McDougall and Gregory Ball.
The United States military has existed since 1775, even before the birth of the nation which it protects. Since then, the military has worked with industry partners to develop machines, equipment, structures, and software that have kept our country free. As our nation’s adversaries evolve technologically, we must continuously hone our competitive edge to develop technology, especially software. The U.S. Department of Defense (DoD) is well positioned through its culture and discipline to adopt extreme programming (XP) processes to maximize the quality and security of products it develops, while increasing the speed at which it develops them.
Challenges the DoD faces
To understand the case for XP in the DoD we must first recognize the challenges it faces when developing, deploying, and maintaining software.
Risk management
The DoD is arguably the largest and most complex organization in the world. As a result, it faces great scrutiny, regulation, and demand for clearly articulated value for each decision made. This has caused much of the DoD to focus on reducing risk through extensive planning, cost analysis, and testing over many years. This process works best for building an aircraft, firearm, or vehicle—when the scope of work changes infrequently. You must still reduce risk for the rapidly changing battlespace of the cyber domain in which software requirements change daily and adversaries are determined to exploit any vulnerability.
Software vulnerabilities
Security is essential to any software development organization, none more so than the DoD, where there is a constant threat of cyber attacks. Adversaries map and exploit vulnerabilities that are inevitable. The DoD must identify and overcome vulnerabilities more rapidly than our adversaries can exploit them else fail.
A rapidly changing battlefield
Technology is rapidly evolving; artificial intelligence (AI), hypersonic munitions, autonomous or semi-autonomous vehicles, and advanced cyber attacks are the norm for many near-peer adversaries. With this ever increasing threat, it is crucial that the U.S. stay ahead of the technological curve. To do this, the DoD is undergoing necessary changes (i.e., DODI 5000.87: Operation of the Software Acquisition Pathway) to develop software at the speed of need. Policy changes are certainly the first and crucial step, however, execution of software development in this way is a large undertaking of its own.
Development burden
Historically, the DoD has relied heavily on contractors to develop hardware and software solutions. In many cases, this has resulted in a reliance on contractors to make necessary updates to the software and hardware to compete in a changing battlefield. Unfortunately, due to the acquisitions process, intellectual property constraints causing vendor lock, and contract negotiation overhead, these changes are often delayed, over budget, or never even delivered.
In recent years, the DoD has taken on more of the burden of software development, rather than being the prime integrator. With this move they have had more freedom to make frequent changes to their software, however with this additional capability comes the burden of recruiting, training, and retaining a skilled workforce. This, in conjunction with a dramatic shift in culture and process while overcoming personnel management challenges, makes this difficult for DoD organizations to do.
Frequent rotations
A constantly changing personnel landscape is one of the biggest challenges the DoD faces both in and out of uniform. The frequent rotation of personnel is built into the military operating model across all services. Whether it is a career broadening experience, or a deployment to a disaster or war zone, unexpected or otherwise, biennial rotations are routine.
For civilians working in the DoD, frequent turnover is routine for different reasons. The workforce that was once in place to provide consistency in the DoD no longer achieves this to the same level they once did. Non-competitive salaries in a highly competitive industry with an increasingly financially-driven workforce is causing a lack of sustained talent in software development and other engineering organizations.
Technical proficiency
There are few paths to long-term technical advancement in the DoD such as warrant officers in the Army. Most in the uniformed service who become technically proficient must decide whether or not to become managers and leaders, or be removed from the workforce. This is colloquially referred to as up or out. Similarly for DoD civilians, expertise is employed to oversee technical requirements as contract workforces fulfill them. Careers stagnate unless one competes for positions elsewhere, takes on more education, or focuses on career broadening experiences. Neither uniformed members of our armed forces or civil servants are incentivized to be great technicians or engineers.
Solving DoD software development challenges with XP
Discipline is a foundational element of any DoD organization. From the moment the oath is taken, discipline is drilled into every member. Similarly, XP requires a high level of discipline to execute properly. Of the agile methodologies, it arguably requires the most discipline to be successful. To understand how XP can alleviate the changes the DoD faces in addition to introducing discipline into software development, it is important to start with a common definition.
Extreme programming (XP) is a software development methodology intended to improve software quality and responsiveness to changing customer requirements. As a type of agile software development, it advocates frequent releases in short development cycles, intended to improve productivity and introduce checkpoints at which new customer requirements can be adopted.
The following XP practices address the challenges the DoD faces with software development and are fundamental to the success of an XP team.
Pairing
By working alongside another team member, software development teams perform real time quality assurance, share context, and learn from one another.
XP teams must share context and decision making through cross-functional balanced teams. Teams become resilient to churn while learning from each other rapidly. XP creates the guardrails for new team members to contribute confidently and securely in a dynamically changing cyber environment in any phase of the competition continuum.
The act of pairing means that developers, designers, and product managers (the three roles of an XP team) do the majority of their work collaborating on the same task with another person. This means that the code that is written every day has two individuals working on it at the same time. To many in management this can be a hard pill to swallow, putting two full-time equivalent (FTE) personnel on a single task can feel like a waste.
Although it is counterintuitive, this practice will result in the whole being greater than the sum of its parts when executed correctly. The ability to put two brains to a single task results in better code, better designs and stories, and ultimately, faster development. More importantly though, it ensures that there is constant and extensive context sharing. There is never only one person that knows one thing. This means when someone rotates off the team or leaves the organization, the odds are that there is someone on the team who worked on what they worked on and can share that context with the other pairs that rotate in or remain on the team.
Pairing addresses the challenges of development burden, frequent rotations, and technical proficiency.
Test-driven development (TDD)
By developing tests prior to writing functioning code, engineers receive near real-time feedback on changes and contribute with confidence.
TDD is the practice of driving the development of functional code with tests. This ensures that the code written is functional, but more than that, it gives confidence that future code written doesn’t break existing functionality. One of the greatest issues in DoD software today is the lack of automated testing. Often, the original developers of legacy software systems have left the program and current developers are doing the bare minimum to keep it functional. They fear doing too much will break the legacy system beyond repair.
Strict TDD sidesteps this problem because the tests written always test the current functionality of the software, meaning engineers can make functionality changes knowing that they are not going to break the application. This gives developers the confidence to make changes to existing functionality, add new functionality, or fix issues because they will be immediately aware of any regressions. This is one of the only ways to ensure that software remains highly modifiable as time progresses, especially after the subject matter experts that started the application have rolled off. This greatly improves the efficiency of changes long after the application was created, increasing the ability to put out updates and greatly reduces the operations and maintenance (O&M) cost of the application.
TDD addresses the challenges of risk management, frequent rotations, and technical proficiency.
Iterative design
By making small iterative changes, software development teams build, measure, and learn on short cycles, allowing them to deliver useful software rapidly and iteratively.
Due to the flexibility achieved by bringing development in house, DoD organizations are no longer in the game of developing requirements, giving them to the contractor, then testing the finished product when the product is delivered. This process of developing requirements while developing software can happen in much quicker succession. This means that the service member using the applications can have a direct influence in the evolution of the software. As they see and use the applications they can influence the direction of the next iteration, which could be delivered to them in days or weeks, as opposed to months or years.
Iterative design addresses the challenges of risk management and a rapidly changing battlefield.
Continuous integration and continuous delivery (CI/CD)
By automating integration and delivery, software development teams focus on meeting user and mission requirements, while ensuring code quality and security. XP’s emphasis on responsiveness to changing requirements, including security, is an underrepresented competitive advantage in a highly contested cyberspace. As the design of the application changes rapidly, XP demands the product be built, tested, secured, and delivered with every change. This is why CI/CD automation of these processes is critical. As developers make changes to applications, they get immediate and automated feedback on the quality, functionality, and security of the code as it is developed. After any issues are resolved, it is automatically deployed to the environment to be used by end users. This ensures that critical capabilities are delivered rapidly and allows a quick feedback loop to inform the development team on the next most important thing to work on.
CI/CD addresses the challenges of risk management, a rapidly changing battlefield, development burden, and software vulnerabilities.
Balanced teams
By fostering autonomous groups of people with a variety of skills and perspectives, software development teams are able to overcome adversity and support each other towards a shared goal. This concept is not novel for the military. For the Army, the smallest unit is a team; for the Marine Corp, it is the fire team. In both cases they are made up of individuals with various ranks and skillsets, such as one or two riflemen, a grenadier, and an automatic rifleman. Each provides more capability and perspective to the team than if they were all from the same discipline.
The same is true for a balanced software team. They are made up of product managers, designers, and engineers. Each brings its own skills and perspectives to the team. With these varied perspectives they make better decisions more quickly. They are able to operate autonomously as needed and incorporate through their disciplines into a larger organization. Additionally, because they work closely together on a daily basis, if a member of the team were to rotate out or otherwise leave the team, they are able to backfill context and provide a high degree of support until they can be replaced.
Balanced teams address the challenges of frequent rotations, technical proficiency, and development burden.
DoD XP successes
VMware Tanzu Labs (formerly Pivotal Labs) has helped many public sector organizations, DoD and civilian, including several DoD Software Factories develop software using XP and DevSecOps practices.
These organizations incorporated continuous integration/continuous delivery (CI/CD) by building continuous Authority to Operate (cATO) or Risk Management Framework (RMF) pipelines. This enabled them to shorten feedback from months or years to days or weeks and provide development teams the opportunity to deliver user-centered software rapidly using iterative design. They established balanced teams of cross-disciplined individuals to autonomously develop software focused around their unique users and needs.
The teams have gone on to develop and deploy applications used by tens of thousands across the military. Those same teams withstand the typical churn of the DoD personnel by pairing daily and ensuring there is never just one context holder.
Their software is built using test-driven development (TDD) and deployed to cloud native technologies such as Kubernetes. They can confidently make changes to these applications knowing they won't break them, and in some cases, even give them to other organizations to maintain.
All this and more is accomplished by Airmen, Soldiers, Marines, and Guardians.
Learn more
For more information about more core tenants of XP, explore the resources on the Tanzu Developer Center. Additionally, if you are interested in learning more about the services we offer and how they may help your organization, please contact our Tanzu Labs public sector sales team.
