devops prac_eng products security Tanzu Application Service

Echoes of Meltdown

Another day, another patch to apply.

We’re more than a year out from the Meltdown and Spectre vulnerabilities. Yet the ramifications still echo today. To wit: the recent news of several high-profile CPU vulnerabilities, including RIDL, Fallout, and Zombieload. There’s plenty of in-depth descriptions of each of these vulnerabilities. The TL;DR is they’re all derivatives of a speculative execution attack, such as the Meltdown vulnerability uncovered by researchers.

The reach of these vulnerabilities is massive. Any server, desktop, or laptop running on an Intel CPU made in roughly the last decade is probably vulnerable. And yes, that includes the latest Intel CPUs that specifically address Meltdown.

When a vulnerability of this magnitude is announced, all eyes in the organization (and your customers) fall upon you. At Pivotal, our customers count on us to quickly remediate these security incidents. We’ve written about our efforts here with Meltdown, and the more recent major Kubernetes CVE. We aim to deliver 100% patch coverage for your PCF deployment in as little time as possible. That’s what we’ve done again in this case.

Let’s see how Pivotal helped its customers mitigate this risk in a matter of hours.

PCF Customers, As Usual, are Already Covered

The good news is that we published a patch featuring an updated Ubuntu stemcell — the base OS image that underscores everything you deploy with PCF. Let’s take a look at the timeline:

  • Tuesday 5/14 AM: Vulnerabilities were disclosed publicly. You likely started getting calls from your boss a couple hours later.

  • Tuesday 5/14 PM: Canonical released patches for Ubuntu Xenial (16.04). The BOSH Systems' team pipelines automatically started the process of testing the patch and building the stemcell. Further updates were provided by Canonical later that evening so the process was restarted.

  • Wednesday 5/15 AM: Canonical released patches for Ubuntu Trusty (14.04). Again the BOSH Systems' pipelines immediately started the test and build process.

  • Wednesday 5/15 PM: Stemcells were made available to the CF community to Pivotal customers on Pivotal Network.

  • Friday 5/17 AM: Stemcells deployed on PWS.

Immediately after Canonical made the patches available, our pipelines got to work. The processes kicked off to build the new stemcells and ran them through our usual gauntlet of tests. Once they passed, we published them to Pivotal Network.

Pivotal also applied these patches on Pivotal Web Services (PWS). This large, multi-tenant Cloud Foundry environment further allowed Pivotal to confirm these patches worked at scale with minimal disruption. And testing doesn’t stop when the patches are released. The team continues to look for ways to optimize this fix for performance and security.

If you are a PCF customer, it’s easy to apply this fix to your environment. All it takes is a few clicks in Ops Manager, and you’ve updated the entire platform with the new OS patch. Even better, for those that seriously lean into platform automation, a Concourse pipeline can automate this for you. Here’s how it works.

Platform Automation for Concourse helps operators automate patches, upgrades, and more.

Platform automation means your pipelines constantly watch for new updates and roll those patches through lower environments, e.g. testing and staging. Once your tests complete, you’re ready to roll the update to production…with a single click. The faster you can patch your systems, the smaller the window for an attacker especially for vulnerabilities like these.

Per usual, Pivotal quickly published a fix, and platform automation helped you apply said fix within hours. Those are the best kinds of emails to send, aren’t they?

Your “Path to Patch” Matters

Security is never a solved problem. Your best bet is to “shift left” and make security a key consideration early in your software development process. One way to do this is to stop worrying about the operating system, and focus on your application with automated hardening your OS. Embrace a modern InfoSec mindset of “go fast to reduce risk.”

Now more than ever, it’s absolutely critical to keep yourself patched and respond quickly when the next CVE hits. An automated platform like PCF gives you the power to patch at a moment’s notice, without disrupting the business. When your platform team are humming, you'll often have patches applied before most of the general public (or your boss!) even knows there is a problem.

When you partner with Pivotal, you can rest a little easier, knowing that you have a secure, open-source software supply chain.

Learn the latest information about securing your foundation and your applications! Register for SpringOne Platform!

Thanks to Brian McClain, Molly Crowther, and Jared Ruckle for their contributions to this post.

Related Articles