Delivering Container Security in Complex Kubernetes Environments

You may have noticed the VMware Tanzu team talking and writing a lot about container security lately, which is no accident. As DevOps and Kubernetes adoption continue their exponential growth in the enterprise, securing container workloads consistently is among the most difficult challenges associated with that transformation. There is a term we have been seeing—and using—a lot lately that encompasses a new way of looking at container security for Kubernetes: DevSecOps. 

In this post, we will walk through some of the key DevSecOps practices that can improve your container security and compliance posture—even as the number of containers you deploy drastically increases.

The journey from DevOps to DevSecOps 

The term DevOps was coined alongside the widespread adoption of configuration management software, tools that automate many of the processes involved in shipping updates to an application in a virtual machine (VM) or bare metal environment. The advent of such automation enabled new, agile methods of building, shipping, and deploying software, which, in turn, reduced friction between developers and operators. And because all of these applications were built in-house, it was possible to track which dependencies and operating system libraries were present in the infrastructure, though whether that tracking was actually done varied widely between organizations. 

When Kubernetes arrived, DevOps became less a choice than a necessity. After all, containers cannot be logged into and updated the way applications in VMs or bare metal servers can. Rather, updates to existing code, dependencies, or operating system libraries are built into a new image, which is then deployed in place of the existing container. Because each application is composed of multiple containers running as microservices, maintaining applications through manual processes quickly becomes impossible.

Today, most custom software intended to run on Kubernetes is built into containers by the developers who created it. This work is done in parallel by individual developers and teams, on separate release cadences, using processes that are opaque to anyone inspecting those containers later on. All of the open source software that teams used to install in VMs and bare metal servers are now consumed as pre-packaged containers obtained from myriad repositories across the internet—typically without any readily available information about what’s in the stack or how it was configured. This is great for speed and agility, and presents a best-case scenario for developers who need to move fast. For security teams, however, having thousands of potentially vulnerable containers built on images with unknown provenance lurking in development environments and occasionally finding their way to production is unacceptable.

DevOps vs DevSecOps

While the term DevSecOps can be controversial—some consider the addition of “Sec” to DevOps unnecessary—it has helped frame the cultural and technological transformations that must happen across the entire IT industry if we are ever to see a world where catastrophic security breaches are not a regular occurrence.

Among the most important DevSecOps practices you can undertake in a Kubernetes context is to create policies requiring that only approved software is allowed to exist in your infrastructure and enforcing those policies in an automated way. Doing so requires a complete and transparent accounting of all the code, binaries, and operating system libraries in every container you build or consume. It also requires implementing a set of configuration policies that apply to every base operating system image underlying every container so as to make it clear what is, and isn’t, compliant. 

Achieving that level of transparency can be a monumental undertaking, one that requires dedicated teams of developers and application operators who are experts in the intricacies of application packaging. Many teams view it through a pie-in-the-sky lens as something they aspire to achieve, but have no idea where to begin. There is hope, however, as the Kubernetes community and vendors like ourselves are increasingly weaving DevSecOps into the fabric of the platforms, products, and services we build.  

Integrating security in DevOps

Solving the security and compliance challenges inherent in adopting Kubernetes is typically done in one of two ways: By either setting up a dedicated team that does nothing but package golden OS images and open source applications, or by having developers and operators maintain their container images themselves. But because that work is difficult, tedious, time-consuming, and expensive in practice, there can be pressure for teams to circumvent security altogether.

What is needed is a completely new approach to building custom application containers and consuming off-the-shelf open source software. With VMware Tanzu Advanced, that is exactly what you get, out of the box. 

How Tanzu Advanced makes container security intrinsic to DevOps 

Tanzu Advanced comes with two capabilities that make successful DevSecOps possible from day one. The first is an automated container build system; the second is a catalog of the most commonly used open source containers and Helm charts. 

Automated container build system

The automated container build system builds containers from source code on pre-approved base OS images, burning a detailed manifest of each container’s contents by default. The build system can then automatically rebuild and push new images whenever application code, dependencies, or OS libraries are updated. 

This function is managed centrally, with no intervention from individual developers required; operations or security teams can patch vulnerable libraries or dependencies in every container independently of application code or binaries. For any application built using Tanzu Advanced edition’s automated container build system, the combination of centrally managed updates of vulnerable or out-of-date containers with accurate information about every container’s contents enables the security compliance process to be automated.