This post was co-authored by Matthew Pereira.
Platform operators, you continue to amaze us. You’ve embraced DevOps and site reliability engineering. You’ve applied these philosophies to your daily work. And you’ve used Concourse CI (the “continuous thing-doer”) to address all manner of operational workflows. When we created the project in 2016, we couldn’t have imagined all the ways you’d use Concourse to deliver fantastic stability and scalability outcomes to your stakeholders.
That’s why we’re excited to reaffirm our investment in Concourse for VMware Tanzu as the go-to automation tool for platform teams.
To that end, we just released Concourse for VMware Tanzu v6.3.0. Before we walk through the highlights of the release, let’s take a look at how you use the product today. Three use cases stand out.
Concourse for managing multiple foundations
When you move to an application platform, you’re going to be running at scale within your first year. In the case of VMware Tanzu Application Service (TAS) customers, that means dozens of “foundations” in the clouds you use.
Concourse is the workflow tool platform operators use to create, fix, update, and rebuild their TAS foundations. What’s more, Concourse performs these same tasks for all the add-on services that power your deployment.
(Want to hear more about this scenario? Watch two engineers from Mastercard give an overview of how it works in their environment.)
Concourse as a “centralized application supply chain”
Getting a platform up and running is an essential first step. Then you have to get developers onboarded so they can push their apps to the platform. From there, apps make their way through different environments—development, testing, staging, and production. These workflows typically demand complex security and compliance pipelines. The pipelines need to be portable and scalable as well.
Once again, Concourse automates this entire process. Watch The Home Depot explain this use case.
Concourse for application CI: Build, test, and deploy code to a sandbox/dev environment
Concourse isn’t just for platform teams; app dev teams love it as well, as it fits the classic continuous integration use case. Here, Concourse helps dev teams in two ways. First, Concourse helps you automate testing. Code quality goes up, and you don’t have to slog through lengthy QA cycles. Second, Concourse keeps your code in a state where it's always deployable. That means you can go to production more frequently, and shorten feedback loops.
Is your organization happy with its existing tech stack for CI? Concourse can take on as much (or as little) of your CI workflows as you want. You can easily integrate Concourse with existing CI tools. Dick’s Sporting Goods offers a terrific example of this scenario.
Together, these three scenarios help inform the roadmap for Concourse. So how can Concourse for VMware Tanzu v6.3.0 help platform teams? Let’s take a look!
Concourse for VMware Tanzu v6.3.0 adds new scheduling algorithm, boosts speed for large deployments
Concourse for VMware Tanzu v6.3.0 is now commercially available for both Helm and BOSH. This is the first supported version of Concourse in the v6.x series. As you’d expect, it includes a ton of new features to bolster platform operations at scale. Here are a few of our favorite features.
New job scheduling algorithm improves performance at scale
Concourse v6.3.0 includes a new scheduling algorithm for determining job inputs. In previous versions, the algorithm loaded each pipeline’s entire build and resource version history into memory. This process was quite labor-intensive, especially for large-scale and long-lived Concourse deployments. The new algorithm aims to select only the relevant versions. This update should significantly reduce utilization on the web and db nodes, allowing customers to scale even bigger with more consistency, shorter wait times, and less overall usage.
In our testing, the new algorithm showed massive improvements. Consider the following image. The left-hand side shows the prior algorithm, with the CPU utilization pegged to nearly 100%. On the right-hand side, the new algorithm drops this metric closer to 65%!
Concourse v6.3.0 reduces CPU utilization during job scheduling
Here’s a look at data transfer. Once again, the left shows much higher bandwidth. The right side, with Concourse v6.3.0, drops this metric a great deal. By using a more efficient algorithm, Concourse loads significantly less data from the database for each scheduling job, reducing bandwidth usage and overall network noise.
Concourse v6.3.0 reduces bandwidth usage and network noise during data transfer
Concourse v6.3.0 is also smarter about job scheduling. In previous versions, Concourse would schedule jobs every 10 seconds. Now, with the new algorithm in place, Concourse only schedules jobs that actually need to be scheduled. This frees up the web node to handle more important tasks. A “before” and “after” image tells the story.
The chart below shows the amount of time Concourse spent scheduling jobs. To the left of the 12:00 mark is before v6.3.0; to the right of 12:00, after Concourse v6.3.0. The change to overall load on the web node is dramatic!
Job scheduling-related loads before and after Concourse v6.3.0
Improved web UI performance
In the majority of cases, Concourse runs and scales generously. But some customers run massive Concourse deployments (e.g., for more than 100 teams, with more than 1,000 pipelines, processing more than 10,000 jobs). For these folks, a frequent source of pain is the performance of the web UI and dashboard. The platform teams for these deployments monitor every pipeline on every team concurrently, and we received feedback that browser load times could be slow, performance was lackluster, and excessive bandwidth was consumed.
So, we went to work to improve the user experience, and those efforts are reflected in Concourse v6.3.0. The result is that Concourse can deliver a zippy performance at any scale. Users should expect a faster, more responsive web UI, and quicker, smoother interactions.
Tracing via OpenTelemetry: Recommended for advanced use cases
Concourse v6.3.0 exposes traces to distributed tracing tools like Jaegar and Stackdriver. When enabled, you can record traces that reflect the various steps that a build goes through and send them to your favorite trace collector. Use this feature for advanced debugging workflows and pipeline optimization.
Note: Tracing is opt-in, not enabled by default. Also, the implementation of this feature may change in future releases.
The road ahead
With Concourse v6.3.0 out, what’s next? Here are a few tracks we’re working on.
New runtime options
We’re adding containerd as a runtime option alongside the current guardian runtime. This should deliver performance improvements for privileged containers.
The team is looking at Kubernetes as a runtime option as well.
Improved sorting and filtering of pipelines
We’ve heard you loud and clear—you want to archive and favorite pipelines via the web UI. The new archiving pipelines feature allows users to “soft-delete” pipelines while preserving their build history for potential use later on. And the favorite pipelines feature allows users to easily select pipelines for priority visibility at the top of the dashboard and the sidebar. This minimizes scrolling and searching for frequently used pipelines.
The new favorite pipelines feature in Concourse
Both features give you more control over your experience, and make Concourse more sortable and searchable.
It’s time to upgrade to Concourse for VMware Tanzu v6.3.0
Ready to get started with the new version? Download the bits, read the release notes, and then start your upgrade!
Ready to be a part of the Concourse community? You’ll want to check out these links:
Of course, we’ll be talking about all things distributed systems at SpringOne Platform (Sept. 2–3) and VMworld (Sept. 29–Oct. 1). Sign up today—both conferences are free!
