There are hundreds of thousands of open source software projects to choose from today, making open source software (OSS) a vital component of almost any codebase. Regardless of the OSS used, product releases must comply with legal and license requirements of all the open source software. This is true for VMware and it’s true for anyone using any open source components in their products.
To be compliant, you must know what OSS you’ve included in your build. But compiling that exhaustive list of components can add time and friction to the development cycle and can delay product releases.The ideal scenario is an efficient process for identification and tracking of OSS so your company’s legal or open source team can further vet and verify license compliance.
Building with Bazel
At VMware, we moved some of our largest builds to Bazel, a build system with precise knowledge of all build inputs. Bazel includes many additional benefits such as fast and deterministic builds with multi-language support. The story of how we chose Bazel and the benefits that it brought us would fill an entire blog, or two. In this blog, we look at how our team developed a new way to integrate the existing OSS compliance process with Bazel.
The New Developer Workflow For Meeting OSS Compliance
As part of the move to Bazel, we set out to improve the efficiency of OSS identification, tracking, and license compliance. By creating and applying a new Bazel rule during the build process, we can precisely identify direct and transitive OSS dependencies. We combine the OSS dependency insights from Bazel with licensing and usage information from VMware’s internal OSS review and compliance tool. This benefits our developer workflow for meeting OSS compliance in several ways and has been adopted by a growing subset of VMware’s product builds.
First, developers get a deterministic Bill of Materials (BOM) with every build. The BOM contains all the direct and transitive dependencies of a product paired with licensing information and approval status from our OSS tool. OSS scanners from VMware’s current compliance process cross-check the accuracy and completeness of the Bazel-generated BOM. In the case where the BOM is incomplete, it can be supplemented with build steps that scan individual inputs.
Second, developers are notified at build time if their product contains OSS dependencies that have been denied for use by legal or security teams—a process that previously could take days. This is done by cross-checking each OSS dependency against its approval status in our OSS tool.
Third, Bazel’s multi-language support allows us to have one tool that works cross-platform. oss_audit supports OSS identification in Java projects built by Bazel. We are currently adding support for C++ builds and will soon support other languages.
Because we create a BOM and validate approval status of OSS with each build, OSS decisions are made earlier in the development and review process, making them less costly and less likely to impose release delays due to required remediation.
Let’s take a look at an example of this new developer workflow using the oss_audit rule.
Using oss_audit
Under the hood, oss_audit consumes a Bazel aspect that analyzes the dependency graph of a build and collects license information about each package it finds. Additionally, oss_audit consumes a list of approved and denied OSS packages from our OSS tool to alert developers of the case when denied packages are being used and need removal.
The Bazel rule outputs two files. First, a BOM yaml file, which includes information on each OSS dependency. Second, a BOM-issues file, containing a subset of OSS dependencies that have been denied for use or that are still waiting for approval. Release managers and developers can use the BOM-issues file to identify problems that may block or delay a product release.
To track OSS for license compliance, an automated job consumes the BOM to file new OSS packages with the OSS tool for legal and security review. Updated data from the review process is then fed back into source control in the form of approved and denied lists to be used by the next build.
Example Code
Here’s an example code snippet showing how to apply oss_audit to a project.
oss_audit can be added to a BUILD file to audit a target, such as an RPM. In the example above, the code uses the pkg_rpm rule, creating the RPM, and the oss_audit rule, auditing the RPM. You can use oss_audit in your own project from the rules_oss_audit repository on GitHub (github.com/vmware/rules_oss_audit).
Component BOMs
At VMware, there exist many complex products that are constructed of multiple sub-components. For such products, oss_audit can be used to generate a BOM for each sub-component, and the sub-component BOMs are then merged together to create one top-level cumulative BOM and BOM-issues files for the entire product. This gives product and component teams insight into what they are shipping at varying degrees of granularity.
Conclusion
OSS is a critical piece of almost every codebase. With Bazel as our foundation, we are able to identify all OSS used directly and transitively in a build. With this information, we can create a BOM and cross-check all OSS with its approval status in our OSS review and compliance tool at build time. We’re using this Bazel-based tooling across a number of projects at VMware and work closely with our colleagues in the Open Source Program Office and Legal teams to ensure complete, documented, and validated license compliance. While we’ve discovered that Bazel can be used to populate your open source component data, it’s essential that your company’s internal tooling and processes verify license compliance and create the required open source license document.
Please visit our rules_oss_audit repository on GitHub (github.com/vmware/rules_oss_audit), check out the code, and open an issue to continue this discussion!
Stay tuned to the Open Source Blog and follow us on Twitter for more deep dives into the world of open source contributing.
