Mitigating software supply chain security issues tops the priority list of many security leaders of today, and rightly so as software supply chain attacks are on the rise. Supply Chain Brain reports that the number of supply chain attacks increased by 26 percent in 2023, compared to 2022. Governments and regulatory bodies around the world are keenly aware of this trend and are taking serious, appropriate steps to start enforcing better security throughout software supply chains. The European Union’s cyber-resilience act, the U.S. federal government’s Executive Order on Improving the Nation’s Cybersecurity, and the recent Biden-Harris National Cybersecurity Update are proof of the fact that we are headed in the right direction. These initiatives are asking right questions of open source software (OSS) vendors in a bid to strengthen their security practices and improve security standards across businesses and industries.
While this additional focus from governments and institutions is good, it has inadvertently given rise to an awkward and risky new approach to security, “compliance-driven security.” If reducing the number of Common Vulnerabilities and Exposures (CVEs)—even unfixable ones—becomes the top goal for software engineering teams, those teams can feel compelled and incentivized to take shortcuts to satisfy auditors. It’s very telling that the most popular talk at the latest CloudNativeSecurityCon was about how multiple vulnerability scanners could easily be fooled into thinking that a container image didn’t have any vulnerabilities when, in reality, it had more than a hundred.
There is no doubt that reducing the number of CVEs is a critical and healthy software engineering practice. But that reduction needs to come organically, without compromising quality. If compliance adherence becomes an obsession, not-so-good practices will be adopted to achieve it.
As the team behind the open source Bitnami Application Catalog and its enterprise version, Tanzu Application Catalog, we have maintained a large, curated catalog of OSS applications for almost 20 years. Open source Bitnami-packaged applications have accounted for billions of downloads by developers on multiple hubs and marketplaces, while Tanzu Application Catalog is privately delivering built-to-spec, production-ready OSS images to enterprises around the world. Here’s a list of best practices that we have adopted to ensure that we deliver OSS with minimal CVEs to our users without any compromise in quality.
Use of minimal yet reliable base images
A key pillar of our approach is relying on minimal base images, but not so minimal that they end up being useless. While distroless images remain the eternal promise, shipping the application runtime and code is often not enough. Yes, the image size will be very small, but it is equally important to find that balance between a reasonably small threat surface, usability, and production readiness. We support the use of major Linux distributions like Ubuntu, Red Hat Universal Base Image, or Debian as the base OS for our images. This is because they not only come with several utilities and remain the preferred option of our customers, but they also possess a long history of support, seasoned security processes, and communities. This means Bitnami applications are packaged using some of the most well-known and established distributions among the OSS community. Although at times we might end up with a few lingering, unfixable CVEs in our artifacts, we believe our customers have the right to have their own preferences, support policies, and even contractual obligations, and we aim to meet them where they are. When a customer needs a distribution with minimal CVEs, Tanzu’s own PhotonOS is a balanced choice.
Continuous delivery of latest updates
Another way we reduce vulnerability footprint is through continuous upstream monitoring and by delivering the latest updates as soon as possible. This has been instrumental in driving billions of downloads by developers across multiple hubs and marketplaces. When CVE-2022-36021 was made public in 2023, I was thrilled to find out that we had already made the fix available one day before the CVE was disclosed. In all fairness, the merit is due upstream. What we do have is a reliable system that continuously scans new versions and patches from upstream software vendors, and automatically kicks off all our robust packaging and testing machinery as soon as a new release is found upstream.
However, in May 2023, when CVE-2023-2650 impacted OpenSSL with a high severity, things became quite interesting. Releasing an upstream fix for a vulnerability that impacts a large catalog with hundreds of container images, Helm charts, and virtual machines for multiple cloud providers is quite tedious. It involves many processes, such as kicking off thousands of jobs and creating thousands of Kubernetes clusters on the fly to verify that those containers and applications still work for all the various Kubernetes distributions and vSphere environments. It is definitely a scale-related challenge, but it took us less than 24 hours to do, which is the typical amount of time we take to address such issues. We take pride in our ability to continuously deliver the latest versions of OSS, and our track record of shipping no regressions for production-like scenarios.
Identifying and sticking with trustworthy upstream vendors
Minimizing CVEs and their impacts requires a strong commitment from the upstream software vendor as well. For every app development use case, developers often face a wealth of options in the OSS ecosystem. So, selecting the right software requires due diligence and a careful analysis of the risks posed by each potential choice. Our team maintains an internal checklist to review all potential additions to Bitnami Application Catalog and Tanzu Application Catalog, which ensures consistent quality, security, and conformance.
Recently there have been debates about whether directly modifying upstream source code to fix a CVE is a better option than waiting for the vendor to do the fix. It’s true that upstream vendors’ fixes do not always happen in the smoothest of manners. Sometimes it might take a few days for an upstream software vendor to release a security patch for its application. But in all honesty, we believe that is the vendor’s decision to make, because who is a better judge than them to make a call on how critical a patch is? Directly modifying the upstream source code has many risks, including loss of warranties or potentially including unknown new security risks that we are not aware of. While we do our best to contribute back to the community by reporting new issues and sending pull requests, we have made a conscious decision to not attempt modifying the upstream code. Having said that, we are quick to get rid of applications from our catalog whenever we have noticed a pattern from an upstream OSS project ignoring security vulnerabilities. There is never a dearth of options in the OSS ecosystem, is there?
Software bills of materials delivered in ISO standards
Another area gaining prominence and that helps enormously with CVEs is the process of creating a software bill of materials (SBOM). SBOMs help you get complete visibility into a list of an OSS application’s dependencies, such as third-party services, software packages, open source software stacks, and codebases. Mature ISO standards like Software Package Data Exchange (SPDX) and CycloneDX are fantastic, battle-proof formats that enterprises can use for ingesting software provider data through their own tools, such as GUAC (Graph for Understanding Artifact Composition), to gain additional knowledge about their software ecosystems. For Tanzu Application Catalog, we have shifted from delivering our bespoke format to providing standard SPDX SBOMs not only for container images but also for virtual machines and Helm charts. These SPDX SBOMs can be converted to CycloneDX files by following certain simple documented steps.
True risk of CVEs with VEX
For those customers that aim (or that are forced to aim) for a perfect zero on their CVE reports and feel paranoid about the presence of unfixed CVEs but want to do better than instructing scanners to ignore those CVEs, we have been making use of some fresh initiatives, like the Vulnerability Exploitability eXchange (VEX). VEX is a new standard that provides machine-readable vulnerability assessments from software vendors or distributors. By shipping these vulnerability assessments in an international OASIS standard like CSAF, we provide actionable visibility and context on the true risk posed by the upstream vulnerabilities, which can help them reduce the threat footprint in reality. This, in turn, will help keep our users’ auditors happy as well.
Do the basics right
No matter how few CVEs the OSS container images or OVAs might have, it’s irrelevant if the OSS components do not meet the basic security requirements of modern software. We make sure that the OSS Helm charts we deliver are compatible with at least the latest two major releases of all Kubernetes cloud providers, verify that all our OSS containers are running as non-root, ensure that our applications will rebalance after autoscaling, and make sure they can be easily upgraded.
Good engineering practices and consolidation through partnering with a single trusted provider of curated OSS content will help you move toward secure, sustainable, consistent adoption of OSS across multi-cloud environments.
Always look for quality
Yes, looking to build software with zero CVEs is good. But what’s better is ensuring that you don’t lose out on quality in the process. While we try to help improve the security posture of the products or platforms built by our customers, we focus on quality rather than on easy compliance. In the process, we have realized good engineering practices will always be the best way to achieve secure, sustainable, consistent adoption of OSS across multi-cloud environments.
Learn more
To learn more about Tanzu Application Catalog, join our webinar session on January 18 at 10 AM PT (register now).
In the meanwhile, check out our product webpage, documentation, and resources page. You can also check out all the OSS applications available in our catalog here.
