The CIO’s guide to CI/CD

This guide was co-authored by Dormain Drewitz of VMware and Brian Roche of Cognizant.

It’s common in talks about digital transformation to make fun of waterfall as a software delivery methodology, but that’s not entirely fair. Waterfall works great if your requirements are unchanged, you understand your users, and, as a result, you don’t need to be highly adaptable to users’ changing needs. 

As with most things these days, though, those stable, slow-moving conditions are becoming less and less common. Today’s users expect fast bug fixes, new features and regular updates. Similarly, today’s competitors are moving faster, nimbly adding new service offerings and addictive features. And good developers, who should put an emphasis on user experience, don’t want undue process getting in the way of necessary progress. Perhaps that’s why, according to the 2018 Standish Group Chaos Study, 21 percent of waterfall software projects are considered “failures,” compared with only 8 percent of agile projects.

However, going agile alone doesn’t ensure success. The same Standish Group study also found that 50 percent of agile projects were “challenged”—only slightly lower than the 53 percent of waterfall projects that were “challenged.” This begs the question of why agile projects get stuck so frequently? And, more importantly, how organizations can “un-stick” their agile projects.

The agile trinity: microservices, platforms, and CI/CD

Agile is only part of delivering software faster. While there’s no one-size-fits-all solution to effective agile delivery, there are three common factors that successful enterprise IT leaders often cite when discussing how they achieve high deployment velocity:

1. A shift to microservice architecture patterns

2. Implementation of platform-centric teams organized around business goals

3. Adoption of continuous integration and continuous delivery (CI/CD)

If you’re just getting started with microservices, you can read this Intersect post on how and why to implement them correctly. For more on building platform-based technology teams, read this guidance from McKinsey & Company. In this guide, we explore the third factor: adoption of CI/CD.

The simplest definition of the CI part is that it’s a cycle of regularly checking new code into a version control system, testing it (via security scans, compliance checks and more), and integrating it with other code. Companies adopting the CD step also verify that code works in production and is ready to be deployed at any time. In either case, the steps are automated in order (1) make sure it gets done and (2) let developers focus their time and energy on higher-value tasks.

If the goal of agile is to help organizations be highly adaptable to market needs, changing requirements, and new technology, then CI/CD is now a necessity, not a luxury.

Fast feedback and tight focus

Think about the last time you created something—a presentation, a blog post or even an internal memo. Perhaps you sought some feedback or a copy edit from a peer or your boss before you shared it with its intended audience. You probably were most ready to absorb and incorporate this feedback the moment you hit “send” on the email asking for review.

Now imagine getting that feedback 3 months later. First, you have to deal with switching contexts from whatever else you’ve been working on in the meantime. You’ve also potentially missed the opportunity for that feedback to have impact: the meeting you presented in is over, the report has been circulated. Then you realize that it’s not just about fixing a few typos, because the world has changed since then and you now need to rework the piece quite a bit to keep it relevant.

It would have been much easier and more efficient to publish it immediately, and then make small changes as necessary—or maybe just write a series of smaller posts to complement the original. Your frustration probably would be doubled if what you requested that extra set of eyes on was a small thing to begin with.

And although it seems absurd to imagine dealing with 3-month feedback cycles on your work, it’s unfortunately not uncommon in software development. In fact, some organizations operate on much slower timeframes:

• At T-Mobile, it took 7 months to make a single change on one particular service.

• Research on software procurement in the military found a median of 8 years from initial development to Initial Operational Capability, which the military defines as the state achieved when a capability is available in its minimum usefully deployable form.

CI/CD can play a big role by helping get those months down to days, or even hours. Change velocity is important precisely because it provides immediate feedback. Teams are pushing small updates or fixes more frequently, which speeds up feedback loops, thus allowing developers to focus on the one thing they are trying to accomplish. Faster velocity also mitigates risk because you're not trying to push a stockpile of changes at once, months after they were coded!

But before we break down the CI and CD parts in more detail, we need to discuss the importance of testing. “Move fast and break things” might have been a fine motto for Facebook engineering in 2008, but it’s probably not going to fly for a large enterprise in a regulated industry. Without an adequate testing protocol—and without understanding why you’re doing all those tests—moving fast actually will break things.

First: You must write the tests

CI/CD ensures code is ready for deployment at any time. To safely deploy software, software developers and product managers need to know that it works. Tests are used to determine if functionality works (unit tests) and that it doesn’t break other parts of the code base (integration tests). As such, the benefits of CI/CD are, on some level, predicated on tests.

How many tests do you need? That depends on how quickly you want to iterate. When code changes infrequently, limited tests and manual testing processes are okay. But as developers want to release code more quickly, good test coverage is needed to give rapid feedback on the code’s viability. Teams should produce tests as often as they produce new code. This is why test-driven development is such a core tenet in extreme programming practices.

“TESTS give you the CONFIDENCE you need to REFACTOR your code to keep it CLEAN so that you can GO FAST FOREVER.” (From "Why TDD")

Without tests, making changes to code is fraught with uncertainty and risk. However, many companies want to release code faster than they currently do, but don’t have the test coverage to do so with confidence. Faced with lots of legacy code that is risky to change, it’s daunting to conceive of writing all those tests.

Writing tests is an investment. It takes time and there are no shortcuts. However, it’s an investment that, like a sound financial investment, yields increasing benefit over time. In fact, it is the investment that pays down technical debt. Just as forming a savings habit takes time and practice to make the behavioral shift begin to feel normal, so does writing tests.

Continuous integration, explained

Once your organization has bought into the value of fast feedback and delivery, as well as the importance of writing tests, it’s time to step up to the plate of continuous integration.

Continuous integration is a software-development practice where development teams frequently integrate their code into a shared repository. By integrating code often (typically, at least daily), developers can ensure that their code is not harmful to other parts of the system.    

Each commit of new code is verified by an automated build and a series of test suites. Unit tests ensure the committed code itself works, while integration tests are invoked to ensure breaking changes are not introduced into the main code line. 

Another benefit of continuous integration is the notion of immediate feedback. By integrating changes often, developers receive immediate feedback on the quality of the code they’re delivering. This reduces the cost of fixing issues. Bugs are rapidly found and fixed within minutes or hours instead of days or weeks.

Continuous delivery, explained

After it has passed all of its unit and integration tests, there are still a number of other steps required before code is ready for deployment. These include compliance checkpoints and verification that the code works in a production environment, not just test and dev. This process is referred to as continuous delivery. The goal of continuous delivery is ensuring code is ready for deployment into production at anytime.

Continuous delivery is not to be confused with continuous deployment, which is an optional, subsequent step in which code is automatically deployed to production after going through the CI/CD process.

The end result of CI/CD is that code is ready for release to production, but when that release occurs is a business decision. In some cases, continuous deployment is the correct approach, but in other cases you may want to release code to production on a different cadence (say, for example, based on the ability of users to absorb new features without being overwhelmed).

The important point is that teams that practice CI/CD can release new application code to production in minutes, and when it makes the most business sense to do so rather than based on predetermined release windows. This, in turn, means users get updates and new features on a more frequent, but also more thoughtful, basis.

CI/CD in action

Often the simplest ways to understand a concept is to examine a real-world example.  To understand CI/CD in a real world way, we can follow a developer’s routine:

1. Download the latest, integrated software from the central repository.

2. Write the tests for the new feature or capability to ensure the new code works and doesn’t introduce breaking changes.

3. Code a specific feature or capability.

4. Check in code and carry out an automated build, which will take the existing code base and a developer’s changes, and verify they compile and pass tests (including all the pre-existing tests from other components).

5. Once the code passes tests and builds on a developer’s machine, this code is integrated into the central code repository.

6. Code check-in triggers a continuous integration pipeline to test the new code against other services and dependencies. As tests fail, development teams are notified and the pipeline does not complete.

7. If all integration tests pass, a continuous delivery pipeline can pick up the tested code and run additional checks.

The key takeaway: By integrating regularly, you can detect errors quickly, and locate them more easily.

CI/CD as an organizational change agent

Aside from improving the frequency and purposefulness of code deployment, CI/CD can also bring benefits at the organizational level. When we talk publicly about workforce transformation, we often do so in the context of “learn by doing.” What we really mean by this is to take something that’s important to the business, drive change around it and, in the process, generate new value.

Continuous integration with test-driven development is a wonderful place to start the process of breaking down organizational walls. For starters, it requires developers to embrace a QA mindset because they must write tests and contribute those tests to integration suites. However, there can be resistance to this process, particularly if folks have not embraced agile to the degree of forgoing deadlines.

Here’s what Jonathan Regehr of Garmin has to say about keeping up the testing hygiene:

“What we also see is when deadlines start to loom, the first thing people stop doing is writing tests, or turn the tests off because they're slowing us down […] However, once you do that, you immediately get into a place where your tests start to atrophy and then turning them back on is of course a challenge. […] I think my biggest advice is, if you can get away with it, don't ever, ever, ever, ever, ever, ever turn off a test.”

Integration test suites are then leveraged as part of product builds. This requires engineers and operations teams to come together to figure out where to host and execute a product build.

Finally, once all tests have passed and a new build has been created, that artifact can be deployed to production (either via continuous deployment or a manual step). This requires the important intersection of security, operations, and a host of other teams to coordinate and deliver that code to production. Ideally, this is another step of the process that should be automated as much as possible.

As Kyle Campos of CSAA Insurance Group put it:

“We spent a lot of time on our pipeline and our test automation to make sure that when we say, ‘Hey, we're continuously delivering,’ that we have a record of all that. We know exactly what's deployed. We know that it's ready to go because of all the automated testing that says it's ready to go.”

As you can see in this simplified description, CI/CD takes what has traditionally been a 12-to-18-month process involving often hundreds of people, and shrinks it to minutes or hours, in an automated, rapid fashion. To achieve this massive feat, organizations must come together and focus on the primary goal of rapidly delivering value to end users.

How CI/CD affects end-users and customers

Beyond introducing new technical efficiencies and organizational improvements, CI/CD also can have a meaningful impact on the customer experience. This is critical in today’s software-driven world, where most serious companies are focused on the idea of digital transformation—essentially a fancy way of saying “improving the customer experience via software.”

By automating the processes between writing code and releasing it to users, CI/CD lets you deliver value in small batches. These incremental improvements can be deployed to production environments, where users begin interacting with the software and provide valuable feedback. With this newfound velocity, you’re able to embrace the notion of an MVP (minimum viable product), which leads to rapid evolution of applications based on user feedback. We see this play out almost subconsciously in our-day-to-day lives, with services like Netflix regularly tweaking, releasing and/or deleting new features based on what’s working and what’s not.

Driving adoption is a continuous cycle that doesn’t end after the first release. It’s for this reason that velocity gains from CI/CD are so critically important and necessary for enterprises to remain relevant in the face of ceaseless competition from Silicon Valley and elsewhere. By implementing these work patterns, for example, a major car company is able to remain innovative, and therefore competitive, despite the fact that it was easier for Google to learn how to build a car than it was for the already-established car company to learn to build a self-driving one.

Modern software is all about agility

Although this guide is about CI/CD, it should be noted that almost all new product development embraces this philosophy. All cloud-native products that employ CI/CD benefit in the form of increased agility and assurance of quality.

We can also infer that companies that embrace CI/CD as a delivery method see software development as a strategic capability that provides competitive advantage. Amazon was a great example when it first publicized the ability to deploy software to production environments every 11.6 seconds in May of 2011. It has since increased its agility, expanded its system, added more components, et cetera—and it was all made possible by a relentless focus on automation, integration and delivery.

But Amazon is not alone: Facebook, Google, Dropbox, and all the other tech giants release software many times a day. Increasingly, they’re being joined by more-traditional enterprises—such as Walmart, T-Mobile, and many, many more—that are committed not just to remaining relevant in the 21st century, but also to growing.  For example:  

• Scotiabank deploys over 3,000 times a month

• Garmin deploys to production 3,400 times per month

• Liberty Mutual deploys 1,000 times a day to production on 2,500 daily builds

You don’t do that by resting on your laurels and expecting customers to adjust their busy lives around your slow-moving software practices. You do that by getting more agile and delivering the types of digital experiences your customers and employees demand.

Additional reading

Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation by Jez Humble and David Farley: This is the definitive CI/CD book of its time and still very relevant today. Learn more about why CI/CD are important foundational building blocks in any software organization.

2018 Accelerate State of DevOps Report: Dig into the longest-running research on software delivery performance. It can help you validate your progress and help you set goals for improvement around the delivery metrics that matter: throughput, stability, and availability

How to Become a Continuously Delivered Enterprise: Delve into why delivering modern applications means a new way of thinking about risk, change and roles with Lyle Murphy.

Towards Progressive Delivery: RedMonk’s James Governor explores emerging patterns in CI/CD, like feature flagging, blue/green deployments, canary deployments, and more.

Speed Thrills: How to Harness the Power of CI/CD For Your Development Team: A more detailed guide for practitioners covering CI/CD steps, measurement, tooling, and best practices.

*A version of this guide also appeared on the Cognizant blog.