Securing the software supply chain has long been top-of-mind for VMware’s Open Source Program Office (OSPO). But until recently much of the broader tech community viewed the issue as both a lower priority than rapid feature development and too dauntingly complex to be worth focusing scarce resources on. Now a rise in cyberattacks that exploit supply chain vulnerabilities, coupled with a major shift in U.S. government policy, is driving new urgency around the topic. And a new whitepaper from the CNCF’s Security Technical Advisory Group (TAG), created with input from a number VMware open source security experts, is offering a set of recommended best practices that make it much easier for participants in any particular software supply chain to make that chain more secure.
The CNCF paper, “Software Supply Chain Best Practices,” aims to offer “a holistic approach to supply chain security by highlighting the importance of layered defensive practices.” In doing so, it defines four key principles for supply chain security:
- Trust: Every step in a supply chain should be rendered “trustworthy” through a combination of cryptographic attestation and verification.
- Automation: As much of the supply chain as possible should be automated to reduce the possibility of human error and configuration drift.
- Clarity: The build environments used in any supply chain should be clearly defined, and granted only the minimum permissions required to complete their assigned tasks
- Mutual authentication: All entities operating in the supply chain environment should be required to mutually authenticate using hardened authentication mechanisms with regular key rotation.
To operationalize these principals, the paper suggests:
- Protecting and securing internal source code repositories, and the entities associated with them, through commit signing, vulnerability scanning, contribution rules, and policy enforcement.
- Examining all ingested second and third party materials to verify their contents, scanned for security issues, and evaluated for material trustworthiness and immutability.
- Storing validated materials in a secure, internal repository from which all dependencies in the build process are drawn and that is built directly from source.
- Securing the build pipeline itself, requiring a “separation of concerns” between individual build steps and workers.
- Finally, associating artifacts produced by the supply chain with signed metadata that attests to their contents and that can be verified independently, as well as revalidated, upon consumption and deployment.
These principals and suggested practices are sorely needed, says OSPO’s Nisha Kumar, who wrote the whitepaper’s appendix on best practices for container security and co-maintains Tern, an open source inspection tool for generating a Software Bill of Materials for container images and Dockerfiles.
“VMware and a few other major technology players have already made supply chain security a top priority, while a lot of our industry continues to prioritize new feature development over all else,” Kumar notes. “That has encouraged development teams to view security as technical debt rather than a feature.”
The consequences of that inaction have been made starkly clear by supply chain attacks such as the 2016 NotPetya, 2020 Solarwinds, and 2021 Codecov hacks, all of which seriously damaged or disrupted commercial and government IT operations on a massive scale. While the attacks can be very specifically targeted (the SolarWinds attack, for example, appears to have primarily targeted U.S. federal data repositories), they typically compromise multiple additional commercial, governmental, and non-governmental systems in the process.
In response to incidents like these, the U.S. Federal Government last week issued an Executive Order on “Improving the Nation’s Cybersecurity” to counter “persistent and increasingly sophisticated malicious cyber campaigns that threaten the public sector, the private sector, and ultimately the American people’s security and privacy.”
That’s likely to mark a turning point in the tech industry’s approach to the issue, argues Kumar. “The U.S. government, one of the world’s largest IT consumers, has said that it will now require that its technology suppliers prove that their supply chain is secure,” she observes. “That ties proving that your supply chain is secure to a very large source of potential revenue and makes securing your software supply chain an asset instead of a chore that doesn’t create value.”
While awareness of the potential costs of inaction is rising, many organizations remain daunted by the challenge.
“People are seeing the headlines, but they’re not really sure how to solve the problem,” says the OSPO’s Joshua Lock, who helped conceive and review the CNCF white paper and maintains the TUF open source update framework for securing software update systems. “But we wanted to make it clear that even if you are just a two or three person software house, there are things that you can start doing that are going to make the attackers’ lives harder.”
Although there’s no single solution that people can apply to their network to fully protect themselves, the CNCF whitepaper can help them understand the scope of the problem and then adopt a set of steps to begin tackling the problem incrementally. In addition to the principles and best practices it outlines, the paper also references both relevant additional documentation and open source tools that organizations can start using right away.
It’s essential that people don’t feel that they have to do this alone, adds Andres Vega, a security product manager in VMware’s Modern Applications business unit, a Technical Lead for the CNCF Security TAG, and another key contributor to the whitepaper.
“This has always been so complicated that it’s encouraged people to think that it’s only for experts,” Vega says. “But if we want every software project to be doing these things, we can’t require them all to be experts, so we are aiming to give people both a playbook and the best possible tools to make this as easy as possible.”
The white paper’s authors also hope that it can nudge the open source community towards devising a more comprehensive approach to supply chain security. To that end, the document points to the idea of a “software factory” model that would automate many of the steps that it recommends.
In a recent KubeCon talk introducing the whitepaper with fellow co-leads Jonathan Meadows and Emily Fox, Vega outlines the unresolved challenges that remain. “The idea of a software factory, for example, is a useful analogy, but how would it work in practice given that software supply chains typically extend beyond any one factory and require the seamless transmission of trusted metadata between factories and across networks?” Vega asks. “So we’re going to be reconvening in a few weeks to see if we can write down a reference architecture and do an actual implementation with code examples for all the different stages and tasks that you need to be running through to secure your supply chain.”
The CNCF Security TAG group is hoping the increased attention and urgency around supply chain security will encourage more people to contribute to efforts like the reference architecture/implantation project as well as individual open source supply chain security projects like Tern, TUF, SPDX, sigstore, and in-toto.
“After working in this space for over a decade, I’m excited to see more attention on this issue and more people interested in helping improve the tools we’ve been building to tackle it,” says VMware OSPO’s Kumar. “I’m also hoping to see more application developers really start use these tools, because in the end that what’s going to make the supply chain secure.”
This article may contain hyperlinks to non-VMware websites that are created and maintained by third parties who are solely responsible for the content on such websites.
