Concourse started as a small open source project with the goal of improving cloud-native continuous integration (CI). Today, it is the perfect path for automating workloads to any modern runtimes like Pivotal Platform.
Concourse is helping enterprise companies continuously deliver any change to their modern application stacks. For example, at one large financial firm, Concourse automates the regular upgrades of more than 50 Pivotal Platform environments. In another company, it’s automating CI/CD for more than 200 development teams and their more than 2000 applications.
At its five-year anniversary, let’s examine Concourse’s heritage and its trajectory, which kicks off with the 5.0 release. If you’re just now hearing about Concourse, you’re in the right place! Read on to learn why you should consider it for CI/CD automation.
Concourse Automates CI/CD Pipelines
Let’s start with a quick summary. At Pivotal, we like to call Concourse the “continuous thing doer.” It’s a flexible, lightweight automation framework that can do many different things:
-
Automate regular code integration, testing, and feedback—basically the CI function
-
Automate software compatibility testing on multiple platforms and versions
-
Automate continuous delivery (CD) pipeline tasks like compliance checks, security validation, and change ticket creation
-
Automate the upgrade of cloud platform software (like security patches, operating system updates, and new features) to support your platform as a product
-
Automate a pizza pipeline (okay, that one is a stretch)
Concourse’s adaptability was a key factor in its recent win in a “tool showdown” pitting Concourse against Drone. As its author, Tom Nolle, says, “Concourse abstracts nearly all application resources, which means users can adapt it to nearly any project.” He concludes that, “The beauty of Concourse is that it doesn't change the users’ basic practices and tools, so the operational impact of implementing it is negligible—and in the future, users can change practices and adjust Concourse to suit.”
The Concourse Origin Story
Concourse was built by Pivotal’s Alex Suraci and Chris Brown to support modern development and delivery practices. The developers were working in short iterations, using test-driven development and CI, to create minimum viable products (MVPs) that were ready to deploy. But existing CI tools had too much overhead to work as nimbly as they wanted.
In their own words from the Concourse-ci.org site:
“…with every CI system we used, we found ourselves constantly dealing with the same old problems: complicated configs hidden in many pages of the web UI, not knowing who changed what & when, managing dependencies and state on the workers, build pollution, annoying UX…”
Concourse started its journey in 2014 and was open sourced in 2015. The benefit of youth is that Concourse was created with cloud-native patterns in mind.
In particular, Concourse operates under these principles:
-
Isolate tasks in containers for better dependency control, enabling clean, consistent builds and no maintenance of test infrastructure.
-
Use straightforward primitives (tasks, resources, and jobs) for automating sophisticated pipeline actions and interactions, including simple abstractions for DevOps tooling.
-
Define pipelines as declarative config files to manage in source control alongside app and other configuration code (in other words, pipeline as code).
-
Visualize the pipeline with a web UI for instant feedback on code progression, taking action on issues, and keeping teams aligned. Pipelines are first class citizens.
Pivotal has worked with the open source community and our customers over the years to improve the Concourse experience. According to Alex Suraci, the Concourse OSS lead, the journey has been one of crafting the right level of abstraction for developers:
“In the past five years, we have continued our quest for defining a general language and mindset for CI that decouples the project from its tooling and infrastructure (including Concourse itself). We want Concourse’s abstractions to outlast the yearly churn of ops/container tools (BOSH, Docker, K8s, Tekton, Nomad, etc.) and enable engineers to think with ‘CI’ level concepts, not just ‘container’ level. We are also designing more powerful pipeline abstractions (like spatial resources, triggers, and notifications) that will bring Concourse to the next level.”
Pivotal is known for integrating the most interesting open source into products our customers can rely on. Concourse is a great example of that. We productize upstream Concourse OSS for our customers, which includes Pivotal product support and back patching. We call this version Pivotal Concourse. Concourse 5.3.0 is now available in open source, and Concourse for PCF 5.2.0 is available for Pivotal customers.
Today, Pivotal has over 200 customers using the commercial version of Concourse (including Cerner, Dick’s Sporting Goods, and T-Mobile). Thousands more use the open-source version. In fact, Concourse has found its way to the inner Adopt circle of the Porsche Tech Radar for 2019 where:
“[Concourse] has been used successfully by teams to create high quality pipelines that are adapted to the new cloud model from the start, while requiring less maintenance than known other solutions (e.g., Jenkins with plugins).”
https://twitter.com/ngajwani/status/1108092241451642880
Five Reasons to Pick up Concourse
The release of Concourse 5.0 this year was a big milestone, featuring more than 10,000 commits over five months of work. These updates reflect a maturing product, and a slew of enhancements for security, stability, and scalability.
Whether you are new to Concourse or thinking of upgrading, here are some reasons to make the move.
#1 – Stable execution with a distributed, modular system
Concourse functions as a resilient, distributed system made up of three subsystems. This means your pipelines are repeatable, reusable and recoverable out-of-the-box. The three subsystems include:
-
Workers that manage executions within stateless containers
-
Web that runs the Concourse web UI
-
Database that stores pipeline data
Concourse operates with containerized executions where every container includes its own dependencies. You get clean, immutable environments every time. If a build or Worker failure occurs, the system can spin up another container without impacting data and dependencies. That’s because Concourse Workers are stateless and easy to replace—no maintaining a tangled mess of dependency packages. This is very different from workers in non-containerized CI solutions, where the state of packages on the worker is crucial to whether your pipeline can function.
Concourse 5.x adds another container placement option to improve system stability, called “Fewest-build-containers.” This feature automatically distributes build containers equally across workers. It is particularly useful if your Concourse cluster has pipelines and build plans that tend to be light on artifacts and heavy on task execution (meaning, many more containers in use).
#2 – Enterprise-ready secrets management and governance
CI/CD systems can be exploited for vulnerabilities like any other software (especially since these tools automate interactions with other systems). For example, you could have exposure from unencrypted secrets storage in your pipeline code. Instead, secure your CI/CD systems by managing secrets outside the pipeline and enforcing access control.
Concourse works well with secrets management tools, like CredHub and HashiCorp Vault. These tools serve up temporary credentials to Concourse builds, so that secrets are never stored in accessible pipeline code. Concourse 5.x includes support for CredHub 2.x, so you can manage credentials in multi-tenant set-ups. CredHub and Vault also enable credential rotation and management from outside the pipeline or team.
Concourse also provides both authentication and authorization for controlling how users identify themselves and what level of access they have. Concourse 5.x includes role-based access control (RBAC), which enables you to enforce permissions to your CI/CD pipelines. You can assign roles like owner, viewer, operator, and so on. This suits InfoSec and compliance teams just fine.
#3 – A CI system built for enterprise scale
A little over a year ago, Pivotal’s Concourse engineers set up a large BOSH deployment of Concourse across Pivotal’s development teams. Why? They wanted to learn more about operating Concourse in larger, multi-tenant installations. Today, this initiative, dubbed “Wings,” supports over 120 teams and hundreds of pipelines across Pivotal. Not only has the team gained empathy for our customers, they’re able to drive feature validation and better performance at scale for Concourse.
In Concourse 5.x, scaling improvements continue. You can now choose to eliminate unused containers and volumes in parallel for faster removal of these ephemeral environments. Also, managing resources across many pipelines just got more efficient with global resources. Instead of redundant collection of version and metadata for resources with the same type and configuration, this information is now shared across pipelines.
Pivotal Concourse also has optimized the integration with BOSH Backup and Restore (BBR), bringing you added protection in case of disaster. With BBR, you can backup your Concourse database to a safe location and restore on command.
#4 – Visual pipelines for fast action and better team alignment
The Concourse team continues to update the UI based on user-centered design practices: learning, framing, assessing, and iterating. For example, an early UI request from the user community was the ability to observe and monitor multiple pipelines simultaneously. This resulted in the design of the pipeline dashboard, also referred to as the “CI Monitor” or “CI Radiator” by users.
Concourse Pipeline Dashboard (CI Radiator)
With today’s pipeline, you can click on elements to drill down to resource and build pages, as well as hover to trace a thread of a resource. Plus, you can zoom, pan and fit view as you interact with pipelines large and small.
Concourse Pipeline in Action
In Concourse 5.x, icons were added by the community to help visualize resources in the pipeline. When navigating through a detailed pipeline, these symbols are quite helpful for identifying the different resource types at a glance.
Icons Identifying Pipeline Resources
You can check out the Concourse dashboard and pipelines used to deploy Concourse here.
#5 – A path to cloud-native continuous delivery with Spinnaker
This might be the time to extend your Concourse CI practices with a stateful continuous delivery and deployment solution like Spinnaker. Over the past year, Pivotal has been contributing to the Spinnaker OSS project to include strong support for Pivotal Platform and Concourse.
Through automated pipelines built with Concourse and Spinnaker, you can take advantage of low-risk application deployment techniques, like blue-green and canary. Spinnaker also maintains the version of code running in all environments—including on which nodes and containers the code is running. Because it maintains state, Spinnaker can roll back to earlier app versions with no downtime.
Check out this demo of a Concourse-to-Spinnaker pipeline deploying to multiple clusters in PKS.
Added Bonus: Concourse and Kubernetes Combo
Installing and running Concourse on Kubernetes can have advantages. For development teams in particular, they can get up and running on their CI tool quickly with minimal maintenance. After all, the effort to install and manage your tools should not detract from delivering new features to your customers.
The Pivotal Concourse team is now the core maintainer to the Concourse Helm Chart for Kubernetes deployments. (As a package manager for Kubernetes, Helm enables you to more easily deploy apps to clusters and manage the lifecycle of those apps. The resulting Helm packages, known as Charts, consist of easy to manage YAML configuration files and templates.) The Helm Chart deployment aligns well with PKS. Of course, you can deploy on any Kubernetes environment if you so choose.
Pivotal is working on readying the Helm-deployed Pivotal Concourse using a similar environment to Wings called “Hush House.” In aircraft lingo, a hush house is a large, noise-suppressed facility for testing aircraft systems. In this environment, the team can observe, maintain, and operate Concourse running on PKS at scale to validate this deployment for our customers.
Use Concourse Anytime, Anywhere, and on Any Cloud
Concourse is one of those tools that proves to be more capable the more you use it. We’re excited that users have discovered new ways of managing their pipelines, like @DonaldBower using his iPhone to monitor and run Concourse pipelines.
https://twitter.com/DonaldBower/status/1057789201490440192
Whether you’re an experienced user or new, here are the many ways to engage with Concourse:
-
SHARE: We’d love to hear how (or where) you’re managing your Concourse pipelines @concourseci.
-
TRY: If you are new to Concourse, check out the Quick Start on concourse-ci.org.
-
BUY: Contact us here if you want to get started with the commercial Pivotal Concourse product. Get information about the latest version on Pivotal Network.
-
UPGRADE: We have new documentation for upgrading your Pivotal Concourse versions to help make it as straightforward as possible.
-
LEARN: Access great Concourse training through Stark & Wayne and and a new training from @rseroter on Pluralsite for developers doing CI. And see the Examples documentation for configuring self-contained Concourse pipelines.
-
CONTRIBUTE: Join the Concourse community work.The team is on track to nearly double the number of PR submissions from last year!
-
CHECK OUT: Keep up with the development activity on the Concourse blog, where you’ll find regular updates on releases and notes on methodology.
-
PARTNER: We work with some great companies that love Concourse as much as we do. Check out Aptomi (Concourse for K8s), Dynatrace (Concourse + app monitoring), and Engineer Better (Control Tower for Concourse) to see how they can help you use and extend Concourse.
-
ATTEND: Join us at SpringOne Platform to learn more about how to design, develop and deliver your modern applications. You’ll get to see new case studies about how enterprise companies are using Concourse.
