Recently, I wrote about the xz supply chain attack, which involved a long game approach. The xz hack required social engineering over multiple years to enable access to the source repository. This allowed the real hack: Injecting malware into the build systems. The potential downstream impacts include the majority of Linux distributions and their consumers.
Alternatively, why do all that work when you can just attack the images and the related metadata that people are using with their production containerized deployments? Docker Hub is a community site that allows anyone, anywhere, the ability to comment or add images for others to download and use in their containerized applications.
Often, these are base images for operating systems, like Debian or open source tools from Apache, the latest AI/ML projects like PyTorch, or popular databases like Postgres and MySQL. However, there are significantly more images within such a large and open system. For example, searching for “mysql” returns over 10,000 results, and the same for Postgres or Java. How does one determine which of these 10,000 images are “best” for your project? How should one select for security, compliance, or interoperability? Which images have updated root file system or runtime?
Now to make that all harder, JFrog, in partnership with Docker, identified that over 20% of the dockerhub repositories had malware and tools for phishing scams in the project metadata.
While there were no identified images with malware, which is not uncommon, the idea that DockerHub, like GitHub, can be taken by malicious actors because of their open nature— and availability of anyone, anywhere, to create a repo, commit, or upload malicious content—is disconcerting at the very least. So, how do we bolster trust and security when using open source?
For Docker Hub users, the good news is that you are most likely not consuming malicious images from this attack, and hopefully are not using any of the 20% or more Docker Hub repos that were compromised. However, given the volume of images in DockerHub and the number of ways containers can be used for attacks, you may not be completely risk free. This is particularly true if you are using custom and containerized applications.That’s why we recommend having a strong application architecture and runtime platforms that can limit exposure to and from modern applications.
If your business depends on containers or containerized open source solutions, you should consider a proven and trusted open source container catalog such as Bitnami or the VMware Tanzu Application Catalog (TAC). For an even more comprehensive and systematic approach, you could standardize on a platform like Tanzu, which uses build-packs to create custom container images.
Bitnami and TAC are both closed systems where all images are built and maintained by Tanzu teams. The supply chains within Tanzu monitor upstream root file systems, whether it be Bitnami Debian or TAC Photon or custom customer-provided images so all images are using the latest root file system for every release. The same supply chains also monitor the application runtime, be it mysql, postgres, rabbitmq, or others, and the dependencies installed on top of the base operating system. Whenever there is a new release, Bitnami and Tanzu Application Catalog automatically re-package the affected applications and published them after conducting an extensive battery of functional tests in different target environments.
You may be asking what the differences are between the two, other than having the ability to use a custom golden image or more specific roots. As was mentioned back in January, TAC achieved SLSA level 3 for all releases. The details for SLSA provenance and attestation, along with all build, CVE, and configuration testing meta-data, is available for commercial and business validation with images and installation details provided by Tanzu Application Catalog. Bitnami images are published on Debian, but Tanzu Application Catalog images can be built on PhotonOS if you need a low-to-zero-CVE container that comes with VEX statements for any remaining critical or high severity CVEs.
You can always trust that Bitnami images will be built, released, and managed by Tanzu teams and be up-to-date and packaged with best practices. But, if your business needs to validate CVE or Malware scans, SLSA provenance and build metadata, or even validation tests for the runtime, then reach out and see how the Tanzu Application Catalog can secure the supply chain and provenance of the images that run your businesses applications.
Whatever you do, get an inventory of your open source, and if you are working with containers or Kubernetes and are curious where you can start with understanding Open Source dependencies and their impact on your security and compliance posture, we recently released the Tanzu Open Source Software Health Assessment tool to help.
The tool can be used to analyze your Kubernetes clusters, local folders, or repositories with Helm charts, YAML files, and public container images stored in open repositories, such as DockerHub, or locally in private repositories.
After analyzing the resource you choose, the tool generates a report with details on all potential misconfigurations and security vulnerabilities that put you at risk, along with a list of applications from Tanzu Application Catalog that could help you improve the security of your software supply chain.
You can learn more and find details in this blog post and access The Tanzu Open Source Software Health Assessment tool directly here.
