It’s really easy to spin up new Kubernetes clusters with Pivotal Container Service (PKS)—all it takes is a single command. But now that we have a cluster, what’s next? It’s time to deploy custom apps or third-party software. This is straightforward enough to do in a small environment. But at scale, you need to think differently; you need automation and a scalable delivery pipeline. Let’s explore how Concourse and Spinnaker can help you do just that!
Multi-Cluster Madness
First of all, let’s talk more about the ‘lots of clusters’ part. It’s common to have dozens or even hundreds of Kubernetes clusters in an enterprise. This volume of clusters is a good thing, as it limits your blast radius. That is, any potential problems are limited to a single cluster, and you avoid cascading failure. Dan Woods (a former engineer at Target) has the mantra of “smaller clusters, more of them,” and he talks about how that prevented a much larger outage when Target had an issue early this year. (Want even more reasons why many clusters are a good idea? Cornelia Davis from Pivotal has you covered!)
As always, there are trade-offs to many clusters. In this case, the additional complexity comes from operations and application deployment. The operations side is pretty intuitive—more clusters means more objects to manage, monitor, upgrade, and so on. This is where PKS really shines with full automation of cluster deployment, upgrades, healing failed components and nodes, and more. Automation can minimize the multi-cluster challenges so that you don’t have to set unnecessary limits.
On the application deployment side, it can get hard to track quickly—lots of apps comprised of lots of individual components on lots of clusters. Your challenges include not only what goes where, but the appropriate configurations and steps to deploy and upgrade the apps as well.
As the number of microservices grow, the interconnections grow exponentially
How do high functioning operational teams like Netflix, Twitter, and Google handle these problems without deploying the wrong apps to the wrong place?
Automated Delivery Pipeline to Kubernetes Clusters
Getting modern applications from committed code to running in production (on the right cluster) in a fast and efficient manner just can’t be done manually. You need an automated delivery pipeline that starts with your code in a version control system of your choice and ends with running apps on Kubernetes.
The delivery pipeline consists of a series of automated hand-offs triggered from activity in your code repository. If your code passes a test, security check or some other marker of ‘readiness’ it moves further down the pipeline. If there is an issue, the pipeline stops, feedback is sent to development, and a new commit is needed to start the pipeline again.
What goes on within the pipeline depends on your particular application, but generally includes a few key things:
-
A continuous integration tool for automating code testing and artifact creation
-
Someplace to store those artifacts—the hand-off between your CI and CD system
-
Security checks for your code
-
A continuous delivery tool to deploy those artifacts
Scaling delivery means that you must establish a repeatable, traceable process that validates your code along the way and delivers it safely to its destination.
Demo: Scaling Deployments to Multi-cluster Environments
To make this easier to understand we have a demo video of a sample pipeline, that is deploying a sample Spring Boot app called Pet Clinic.
The demo pipeline consists of these tools:
-
Concourse CI: Team-centric continuous integration that provides simple pipeline modeling and containerized build and test for a clean, consistent environment every time
-
Harbor Registry: An enterprise-class container registry with built-in security validation that’s provided as part of PKS.
-
Spinnaker CD: A multi-cloud continuous delivery tool with built-in deployment best practices and application monitoring services
All of this is running on PKS, which enables you to reliably deploy and run containerized workloads into multi-cluster environments across private and public clouds.
Our demo set-up has three Kubernetes clusters deployed by PKS. The first cluster is hosting the tooling (Concourse, Harbor, Spinnaker) deployed by Helm Charts. The second and third clusters are the development and production environments that are the application deployment targets.
Now let’s walk through the demo:
Step 1: Integration
The first part of our pipeline is triggered by any commits or releases from our GitHub repository. Concourse immediately downloads the latest version of the code and runs the full Spring Boot build and test suite (using Maven). After that, it builds an OCI-compliant docker image and uploads it to the Harbor registry. If the triggering event is a commit, the docker image is tagged as ‘latest’ and if it was triggered by a GitHub release, it’s tagged with that release number.
Step 2: Artifact Storage and Security Check
Scanning your code inside containers for security vulnerabilities during the CI/CD process is critical. Having this built into the artifact repository makes sense—after the package is built and prior to deployment. The Harbor registry uses a component called Clair to scan the docker images for any CVEs or other vulnerabilities. Harbor can even block the pulling of an image if an image has vulnerabilities above a threshold you set.
Step 3: Deployment
Once the new docker images are available, Spinnaker triggers its deployment pipelines based on the tag on the image. The latest images go to development and get a development manifest that only deploys a single instance and leverages the app’s internal ephemeral storage. If an image is tagged with a semantic version number (x.x.x), Spinnaker’s production pipeline handles it and uses a production manifest that deploys three instances of the application. It also connects the app to persistent storage (a MySQL database also running on the production cluster).
In both deployment cases, Spinnaker is using a red/black deployment scenario, which deploys the new version, and disables the previous version. This allows for quick rollbacks if there is an issue discovered with the new version. Spinnaker supports much more advanced scenarios, including canary deployments, but this is a good place to start.
Not only does Spinnaker deploy the application, it also keeps an inventory and history of all our deploys and the artifacts used.
Scaling to More Clusters, No Problem
This is only an environment of three clusters, but this model can scale up to tens or hundreds of clusters with thousands of applications.
With the right tools and pipelines, you can take advantage of the benefits of a multi-cluster environment while minimizing the complexity that can keep you from deploying applications faster. Join us for some of the great sessions at SpringOne Platform this fall focused specifically on this topic.
