Demystifying Continuous Integration and Delivery

It’s common in a transformation talk to make fun of waterfall as a delivery methodology.  But that’s not generally fair.  Waterfall works great if your requirements are unchanged, your user is well understood, and, as a result, you don’t need to be highly adaptable to users’ changing needs. 

Unfortunately, those conditions are less and less common these days. According to the 2018 Standish Group Chaos Study, only 8% of Agile projects are considered failures, compared to 21% of Waterfall projects.¹ But Agile alone doesn’t ensure success. The same study found that 50% of Agile projects were “challenged,” only 3% below the level of “challenged” Waterfall projects.

Where does Agile get stuck? And, more importantly, how can organizations “un-stuck” their Agile projects? We know it is possible for large enterprises to deploy software in high frequency:  

• Scotiabank deploys over 3,000 times a month²
• Garmin deploys to production 3,400 times per month³
• Liberty Mutual deploys 1,000 times a day to production on 2,500 daily builds⁴

How do they do it? Three common factors that each of the practitioners and technology leaders at these companies cite in their journeys to achieve high deployment velocity are:

1. A shift to microservice architecture patterns
2. Use of structured platforms
3. Adoption of continuous integration and continuous delivery (CI/CD).

In this post, we explore the third factor, adoption of CI/CD, a delivery methodology that enables organizations to be highly adaptable to market need, changing requirements and new technology. In such a fast changing environment, CI/CD is now a necessity, not a luxury.

Feedback Loops

Think about the last time you created something: a presentation, a blog post, or even an internal memo. Perhaps you sought some feedback from a peer or your boss. You hope to get another pair of eyes on your work so that it gets better. Someone else may notice that typo or how you completely left out a section related to your business strategy. You are most ready to absorb and incorporate this feedback the moment you hit “send” on the email asking for review.

Now imagine getting that feedback three months later. First, you have to deal with switching contexts from whatever else you’ve been working on in the meantime. Then, you realize that it’s not just about fixing a few typos because the world has changed and now you need to rework quite a bit more to make this piece relevant. It seems absurd to imagine dealing with three-month feedback cycles on your work, but it’s not uncommon in software development:

• At T-Mobile, one particular service took seven months to make a change⁵
• Research on software procurement in the military found a median of eight years from initial development to Initial Operational Capability⁶

But first: Someone has to Write those Tests

Continuous integration and continuous delivery ensures code is ready for deployment at any time. To safely deploy software, software developers and product managers need to know that it works.⁷ Tests are used to determine if functionality works (unit tests) and that it doesn’t break other parts of the code base (integration tests). As such, all the benefits of CI/CD are, on some level, predicated on tests existing.

How many tests do you need? That depends on how quickly you want to iterate. When code changes infrequently, limited tests and manual testing processes are okay. But as developers want to release code more quickly, good test coverage is needed to give rapid feedback on the code’s viability. Teams should produce tests as often as they produce new code. This is why test-driven development is such a core tenet in Extreme Programming practices.

“TESTS give you the CONFIDENCE you need to REFACTOR your code to keep it CLEAN so that you can GO FAST FOREVER.” ⁸

Without tests, making changes to code is fraught with uncertainty and risk. However, many companies want to release code faster than they currently do, but don’t have the test coverage to do so with confidence. Faced with lots of legacy code that is risky to change, it’s daunting to conceive of writing all those tests.

Writing tests is an investment. It takes time and there are no shortcuts. However, it’s an investment that, like a sound financial investment, yields increasing benefit over time. In fact, it is the investment that pays down technical debt. Just as forming a savings habit takes time and practice to make the behavioral shift begin to feel normal, so does writing tests.

Continuous Integration Principles

Continuous integration is a software development practice where development teams integrate their code often into a shared repository. By integrating code often, developers can ensure that their code is not harmful to other parts of the system.    

Each commit of new code is verified by an automated build and a series of test suites. Unit tests ensure the committed code itself works, while integration tests are invoked to ensure breaking changes are not introduced into the main code line. 

Another benefit of continuous integration is the notion of immediate feedback. By integrating changes often, developers receive immediate feedback on the quality of the code they’re delivering. This reduces the cost of fixing issues. Bugs are rapidly found and fixed within minutes or hours instead of days or weeks.

Below are some key tenants of continuous integration:

• Automate, automate, automate: Anything that can be automated, should. This may require developing APIs for existing processes so that they can be automated into a pipeline.
• All new code and builds are automated, leaving the work machines are good at to machines.
• All changes are merged into a single repository .
• All builds are performed on a dedicated service .
• Breaking changes fail tests and the build system disallows those changes to be checked into the mainline.
• Keep the build fast.
• Visibility is key and everyone should be able to see what’s happening.

Continuous Delivery

There are a number of other steps required before code is ready for deployment, but after it has passed all of its unit and integration tests. These include compliance checkpoints and verification that the code works in a production environment, not just test and dev. This process is referred to as continuous delivery.

Continuous delivery is the process of delivering software and applications created in the CI process to a production-like environment, such as a central code repository, where it is put through additional automated tests. The goal of continuous delivery is about ensuring code is ready for deployment into production at anytime.

Continuous delivery is not to be confused with continuous deployment, which is an optional, subsequent step in which code is automatically deployed to production after going through the CI/CD process. The end result of CI/CD is that code is ready for release to production, but when the release occurs is a business decision. In some cases continuous deployment is the correct approach, but in other cases you may want to release code to production on a different cadence (say, for example, based on the ability of users to absorb new features without being overwhelmed.)

The important point is that teams that practice CI/CD can release new application code to production in minutes, when it makes the most business sense to do so rather than based on predetermined release windows. This in turn means users get updates and new features on a more frequent but also more thoughtful basis.

Continuous Integration and Delivery Steps

Often the simplest way to understand a concept is to examine a real world example.  To understand continuous integration and delivery in a real world way, we need to follow a developer’s activity.  

1. Download the latest, integrated software from the central repository
2. Write the tests for the new feature or capability to ensure the new code works and doesn’t introduce breaking changes
3. Implement a specific feature or capability
4. Check in code and carry out an automated build which will take the existing code base and a developer’s changes, and verify they compile and pass tests (including all the pre-existing tests from other components)
5. Once the code passes tests and builds on a developer’s machine, this code can then be integrated into the central code repository

Key take away: By integrating regularly, you can detect errors quickly, and locate them more easily.

Additional Benefits

CI/CD brings a whole host of other benefits to your organization:

1. Unvalidated risk and effort does not accumulate.  By building and integrating often we can validate our changes.
2. Visibility and data are critical factors to successfully steering any project.  The same is true for product development teams. Increased visibility enables teams to make educated decisions about what to do next.
3. Catch issues early and nip them in the bud.
4. Spend less time debugging and more time adding features.
5. Stop waiting to find out if your code’s going to work.
6. Reduce integration problems so you can deliver software more rapidly.

Continuous Integration and Delivery as Organizational Change Agent

When we talk publicly about workforce transformation, we often do so in the context of “learn by doing.” What we really mean by this is to take something that’s important to the business, drive change around it, and, in the process, generate new value.

Continuous integration is a wonderful place to start the process of breaking down organizational walls. Continuous integration requires developers to embrace a QA mindset – they write tests and contribute those tests to integration suites. 

Integration test suites are then leveraged as part of product builds. This requires engineers and operations teams to come together to figure out where to host and execute a product build.

Finally, once all tests have passed and a new build has been created, that artifact can be deployed to production (again, either via continuous deployment or a manual step, such as cf push.). This requires the important intersection of security, operations and a host of other teams to coordinate and deliver that code to production. Ideally, this is another step of the process that should be automated as much as possible.

As you can see in this simplified description, continuous integration and delivery takes what has traditionally been a 12 to 18 month process involving often hundreds of people and shrinks it to minutes or hours, in an automated, rapid fashion. To achieve this massive feat organizations must come together and focus on the primary goal of rapidly delivering value to end users.

User Adoption and Value

Now that you understand the technical and organizational fundamentals of continuous integration and delivery, it’s time to elevate the conversation to the consumer of the artifacts of this process, the end user (or customer).

CI/CD enables us to deliver value in small batches. These incremental improvements can be deployed to production environments where users begin interacting with the software and provide valuable feedback.  This newfound velocity enables us to embrace the notion of an MVP (minimum viable product), which leads to rapid evolution of applications based on user feedback. 

Driving adoption is a continuous cycle that doesn’t end after the first release.  It is for this reason that velocity gains from CI/CD are so critically important and necessary for enterprises to remain relevant in the face of ceaseless competition from Silicon Valley and elsewhere. By implementing these work patterns, for example, a major car company is able to remain innovative, and therefore competitive, despite the fact that it was easier for Google to learn how to build a car than it was for the already established car company to learn to build a self-driving one. Together, Pivotal and Cognizant help our customers apply modern technologies and work patterns such as CI/CD so they can focus on their core businesses rather than trying to keep up technologically. Adapting to market changes enable enterprises to stay ahead of the competition in this rapidly changing world.

Finally

Although this is an introductory article about continuous integration and delivery, it should be noted that almost all new product development embraces this philosophy. All cloud-native products that employ CI/CD benefit in the form of velocity gain and assurance of quality.

We can also infer that companies who embrace CI/CD as a delivery method see software development as a strategic capability that provides competitive advantage. Amazon was a great example when they first publicized the ability to deploy software to production environments every 11.6 seconds in May of 2011. It has since increased its velocity, expanded its system, added more components, etc., and it was all made possible by a relentless focus on automation, integration and delivery.

Amazon is not alone; Facebook, Google, Dropbox, and all the other tech giants release software many times a day, yet so many executives and development teams still remain unconvinced of the benefits. After reading this blog post and witnessing the evidence, hopefully you too are now a believer in CI/CD.

---

^[1] https://vitalitychicago.com/blog/agile-projects-are-more-successful-traditional-projects/

^[2] https://www.youtube.com/watch?v=5wokS2VySoQ

^[3] https://www.youtube.com/watch?v=woll0fI_Pa0  

^[4] https://www.youtube.com/watch?v=RZTTPfaeGg8

^[5] Brendan Aye, “Zero to 12 Million” https://content.pivotal.io/springone-platform-2017/zero-to-12-million-brendan-aye

^[6] Captain Bryon Kroger & Tory Galvin, United States Air Force, “Cost of Delay–How PCF Helped Demonstrate the DoD Can’t Afford Business as Usual” https://www.youtube.com/watch?v=V8w4jf5clJk

^[7] Matthew Kane Parker, “What’s the Value of CI/CD?” https://builttoadapt.io/whats-the-value-of-ci-cd-c2e6c39450bd

^[8] Matthew Kane Parker, “Why TDD?”https://builttoadapt.io/why-tdd-489fdcdda05e

^[9] Matt Curry, https://springoneplatform.io/sessions/take-devops-to-11-and-sprinkle-cloud-on-it-with-rainbows-and-unicorns 

^[10] “Concourse in the Real World: A Case Study in CI/CD and DevOps”, December 14, 2017  https://youtu.be/Yu7m1D8Z-Yk

---

Brian Roche, Cognizant Digital Business

Dormain Drewitz, Pivotal Software