I recently attended and led a discussion at the Container Plumbing Days conference hosted and sponsored by RedHat. The conference was the first of its kind with talks and discussions on “everything below the orchestrator.” During my time spent investigating OSS compliance for containers, I found that there were a lot of unanswered questions around the container build, run and distribution process. Coincidentally, this led to some confusion regarding the deduplication of data, optimizing transport, distribution of associated artifacts and container supply chain security. There are many nascent technologies that may address these concerns, but due to the cloud community’s focus on Kubernetes and everything on top of it, there wasn’t a forum to address these concerns. So naturally, I was excited that the Container Plumbing Days conference would provide ample opportunity to address these questions.
I was particularly concerned with what the community thought of the state of the container supply chain. Most in the industry focus on securing the source code, but there isn’t much attention paid to the way the industry builds and distributes container images. I have given many talks on the subject, but primarily in the context of OSS compliance. Over the years, I have found that many of the gaps in compliance overlap with concerns around supply chain security and business continuity. These things include the generation and distribution of SBoMs, establishing the chain of trust for derived containers and making this metadata visible to cloud native app developers. I submitted a discussion topic on the matter and it was accepted.
During the discussion, I was surprised to find that the top concern for the community was signing of artifacts *in general* not just for containers. The top concerns were:
- A common, industry wide infrastructure for signing artifacts
- Key management tools and processes
- Attestation and verification infrastructure
Throughout my talk, there was some confusion around some container verbiage, which I will clarify below:
- Image Isolation vs. Root of Trust: When industry leaders talk about “rootless containers,” they mean image isolation (i.e. allowing a user with no root privileges to spin up containers). Although this protects the host from a malicious container process, it doesn’t protect the container image from being tampered with at build time. For this case, one must be able to verify that the container image originated from the expected source and matches the expected source’s intentions.
- Signing container image tags vs. signing container image contents: Image tags are pointers to image manifests, which can be verified based on the digest of the manifest. This works for verifying the supplier of the image itself, but not for the packages that are installed in the image by the supplier. Perhaps the supplier verified them, and perhaps not. The way one would verify this is to verify signatures that do not have anything to do with the final image artifact (i.e. signing the container image contents). A follow-up question you may have is, how are these signatures going to persist as the container image morphs? The community is currently discussing approaches for this, including Notary V2, OCI, CNCF and OpenSSF.
At the end of the discussion, the community agreed on the following points:
- We should focus on hardening containers that are used as build environments in CI/CD and deployment pipelines.
- The industry is moving towards using short-lived keys rather than long-lived keys. SigStore is an example of a signing infrastructure that uses short-lived keys.
- More work needs to be done in the Cloud Native Content Management space. This includes persisting supplemental artifacts as Cloud Native Apps transfer to different environments.
The full session called “Secure Container Supply Chain – Considerations, Tools, and Gaps” is now available online. Find out how projects like Notary V2, In-toto, OCI Artifacts and Tern, in conjunction with the CNCF’s SIG-Security Secure Supply Chain Security Working Group may help open source communities secure their container builds and address any gaps that are currently present.
