Enterprise customers around the world use Pivotal Cloud Foundry to help them with their digital transformation. For the past two years, many of our customers have also put their trust in Pivotal Container Service (PKS) to onboard their container workloads to Kubernetes.
There is a common underlying technology that powers both of these platforms: BOSH.
Recently, I attended a customer meeting with a DevOps team about Cloud Foundry. The discussion questioned why Cloud Foundry performs activities such as recreating virtual machines during an upgrade, instead of just running an upgrade script on existing VMs. After explaining the value of BOSH, the way it treats infrastructure and the consistency that entails, the customer was all-in on BOSH’s methodology. That got me to the realization that BOSH can mean different things to different roles in an organization, because each role has different concerns and gains different benefits from it. I wrote this article to explain the value of BOSH for some of the important roles in a given enterprise.
What is BOSH again?
If you’ll head over to bosh.io, you will find the following definition for BOSH:
BOSH is a project that unifies release engineering, deployment, and lifecycle management of small and large-scale cloud software. BOSH can provision and deploy software over hundreds of VMs. It also performs monitoring, failure recovery, and software updates with zero-to-minimal downtime.
This is quite a broad statement, but what does it mean for you? Let’s play a game of “tell me who you are, and I’ll tell you how BOSH can help”.
“I am a developer/devops engineer working with Kubernetes”
Great! I think you’re going to love BOSH because BOSH is the “Kubernetes for your virtual machines”.
-
Just as Kubernetes deploys, orchestrates, and heals running containers, BOSH deploys, orchestrates and heals virtual machines.
-
BOSH can install and run any type of software on these virtual machines, from a simple NGINX Web Server to a full-blown Cloud Foundry platform or even a Kubernetes cluster.
-
Kubernetes monitors the health of your containers and distributes the workloads across multiple worker nodes. BOSH does something similar for your virtual machines — it monitors their health and distributes the workloads across availability zones.
-
Similar to Kubernetes, your deployment can be declarative. First, you provide BOSH with a manifest file that describes your desired state. Then, BOSH will continuously monitor your system and make sure that your environment matches that configuration. Did a process inside a VM stop responding? BOSH will automatically restart that process for you. Lose an entire VM? BOSH will automatically resurrect a new one for you. Did you increase a resource definition in the manifest from three to four VMs? BOSH will create the 4th VM for you—no additional scripting required.
-
In Kubernetes, your deployed containers will still be accessible even if the master nodes are temporarily offline; you just can’t modify or monitor the cluster during that time. With BOSH, the software deployed on the IaaS will still be available even if the BOSH Director (the equivalent to Kubernetes Master node) is temporarily unavailable.
“I am a CiSO at a large enterprise”
BOSH is your new best friend because it dramatically improves your security posture with the 3Rs of platform security.
-
Repair allows you to patch CVEs quickly and consistently. Pivotal provides the CVE patch to the underlying operating system managed by BOSH in the form of a BOSH stemcell. Since there are no playbooks or Ansible scripts to customize, patching is as easy as a click of a button. BOSH will recreate the needed VMs with the patched operating system stemcell with zero downtime. Many of our customers use platform automation pipelines with Concourse to automatically patch environments. Just recently, this allowed Pivotal customers to quickly patch the Meltdown CVE, as well as a critical Kubernetes vulnerability.
-
Repave enables customers to completely recreate the virtual machines in their data center from known, signed releases. It is similar to Repair, with the exception that you use the same release as before, instead of an updated one. With repaving, BOSH “resets” your platform to a known state. This mediates one of the crucial security pain points at any enterprise: adaptive malware. This type of malware tends to hide in your data center and learn by scanning logs, processes, and network activities. The longer the system remains unchanged, the more the malicious code can learn about your system. Some of our customers rebuild the entire platform once a week, during business hours, with zero downtime. This prevents such adaptive malware from having enough time to steal passwords or customer data.
-
Rotate enables storing secrets, credentials, and certificates in a secure repository called CredHub. BOSH will store secrets in CredHub and pull secrets when needed. By using Credhub, there are no clear-text secrets shown in any of the deployment manifests that BOSH uses during installation. It also allows BOSH to periodically rotate the credentials and certificates, so if a bad actor happens to get a hold of such a secret, it will only be usable for a limited time.
BOSH helps limit the damage that can occur as a result of human error, because it automates so many things. Further, it reduces the attack surface that the bad guys can use to gain access to your systems and data. Consider the following:
-
All installations done via the BOSH Director are locked down for external access and only accessible from a jumpbox (that is usually the Ops Manager in a typical Pivotal Cloud Foundry installation).
-
Installation is immutable and repetitive. There is no need for playbooks or scripting to achieve a successful installation. An installation on your IaaS will be the same process as an installation at other organizations around the world.
-
No human intervention is required during the installation or upgrade process, even if that person is a friendly IT Admin. This means less chance of introducing an untested, risky configuration that may be exploited.
-
You don’t have to worry about patching an OS ever again. Pivotal provides the embedded OS that BOSH uses to install your deployment. The operating system can be either Linux or Windows. This embedded OS, called a stemcell, decouples the actual deployment from the underlying operating system. That way, a patch can be tested and released for the operating system without having to release a new version of a specific deployment such as Pivotal Application Service or a MySQL database.
-
BOSH releases are self-contained tarballs. They include everything you need to successfully install the software. You don’t need internet access to deploy a bosh release, which makes it a natural fit for air-gapped environments such as those found in financial, insurance or government institutions. If whitelisting is required, only two files need to be scanned: the release and the stemcell.
“I’m a Site Reliability Engineer at a large enterprise”
BOSH accelerates your adoption of SRE practices because BOSH deployments are immutable and idempotent.
BOSH is a release engineering toolchain that provisions virtual machines and software on your IaaS, be it in a private cloud like vSphere or a public cloud like AWS, Azure, or GCP. It is uniquely positioned as a solution that provisions, updates, and resurrects virtual machines, from signed releases on the Pivotal Network. These releases are verified, and have gone through rigorous testing by Pivotal. What does this mean to you as an SRE?
-
Immutability guarantees consistency. There is no need to track changes to a VM to maybe get it to a state similar to others. It is much safer and more reliable to just build a new VM to replace the existing one. Have you bumped a specific binary’s version from 1.01 to 1.02? BOSH will replace the existing VM and create a new one for you. Do you have three instances of this VM? BOSH will do a zero downtime rolling update of the three VMs, beginning with a canary deployment for the first one to verify a successful upgrade or an immediate rollback.
-
No configuration drift. BOSH follows good design principles such as the Open-Closed principle, making it closed for modification but open for extension. The same basic configuration that will run in your data center has been tested and approved by Pivotal or one of our Third-Party ISVs. Certain parameters can be defined or tweaked via Ops Manager. If additional mandatory software such as monitoring tools or anti-virus software is required, it can be added to the release in the form of BOSH add-ons, but this will not modify the baseline, tested installation.
-
No snowflakes. Every VM in your deployment will be similar to other VMs. For example, all Kubernetes worker nodes in a PKS deployment with BOSH will be almost identical, except for a different IP address and a few other things.
-
Easy to reset. Recreating a BOSH deployment brings the platform to a known, tested state. This gives you the assurance that if things go bad, you can always recreate the environment to a safe checkpoint, with zero downtime.
-
Deployment-time compilation. For optimized compatibility between a BOSH release and the underlying operating system stemcell, BOSH can compile packages as part of the deployment process. This enables an exact match between the generated binaries and the underlying operating system and infrastructure at deployment time. This also means that the release doesn’t need to contain every variant of the package binary for different operating systems and CPU instruction sets.
"I’m a CIO"
BOSH can help your business because it allows your organization to go fast without giving up on security and reliability.
-
Focus on Business Outcomes. Your development and IT teams don’t provide much value to the organization if they spend their time manually running “patchTuesday” updates. They provide value when they focus on business applications and innovation.
The platform should come with an embedded operating system that was provided and tested by the vendor, and the tools needed to easily apply those patches. BOSH does this consistently by taking an opinionated approach for how upgrades should be applied. -
Operator Efficiency. BOSH enables organizations to manage large-scale environments with few operators. The operator to developer ratio can be as high as 1:500. Some of our customers, such as T-Mobile, manage over 30,000 containers in production with less than 12 operators. This is only possible thanks to the opinionated and automated approach that BOSH enables.
-
No downtime means more revenue. BOSH applies updates in a rolling fashion using canary deployments. As long as you configure high availability for your components (read: more than one VM per component), you will not experience downtime.
-
Production Upgrades as Non-Events. A consistent, opinionated process assures you that upgrades will either succeed, or immediately fallback to a known state if a canary deployment was unsuccessful. Many of our customers trust the process so much that they upgrade patch versions automatically using concourse pipelines (I personally enjoy upgrading from my wrist 😄).
-
Multi-cloud increases choice and mitigates lock-in. Many enterprises need some of their data on-premise in their own data center due to business, regulatory or cost reasons. Enterprises also want the benefits and on-demand scalability of the public cloud. However, they would like to avoid lock-in to a specific provider, due to cost or competitive reasons. BOSH enables the exact same experience whether you run in your own data center or on AWS, Azure or GCP. The BOSH Cloud-Provider-Interface (CPI) is the only component that changes between platforms. Each cloud provider is responsible for the implementation of his own CPI, to guarantee consistency. This results in a true multi-cloud solution for your organization. Pivotal itself benefited from multi-cloud, when it was able to replace the IaaS provider for it’s Pivotal Web Services service with zero downtime.
Get Started
In this article, I wanted to illustrate how the power of BOSH can deliver different benefits to different roles in the organization. If you are interested to learn more about BOSH, head over to bosh.io, read about it on cloudfoundry.org/bosh, or take a quick tutorial.
