Supply-chain attacks can be so destructive that they are often considered black-swan events. Often, the most upsetting aspect of the attack is that it manages to compromise what is normally deemed to be safe by definition — whether that’s a software component or an MSP (managed service provider). The result is that our understanding of perimeters, security boundaries, and/or best practices is often flipped upside down.
Consider, for example, the SolarWinds attack back in December 2020: disguised as a normal software update, attackers managed to implant a pre-crafted backdoor on thousands of customers, which led many frantic security teams to discover that their network perimeter had already been breached several months before. Another (and even more destructive) attack took place in July 2021: by exploiting a vulnerability in Kaseya VSA servers, attackers managed to infect hundreds of MSPs, which in turn deployed the REvil ransomware to thousands of customers, breaking the assumption of a safe boundary between different IT infrastructures.
Fast forward to October 2021. An innocent bug report alerted the entire NPM developer community that a core open-source library had been hacked. Fortunately, the community quickly handled and fixed the issue. But, had it not been detected, the potential impacts of this incident left security professionals aghast at the way modern CI/CD pipelines can actually become targets. In this blogpost, we further analyze this incident, and show that regardless of adopted best practices, sometimes the devil really is in the details: when open-source libraries are compromised, it’s extremely difficult to limit and control the damage.
|0.7.28||2021-04-10 14:42:47||Clean version|
|0.7.29||2021-10-22 12:15:21||Malicious version|
|0.8.0||2021-10-22 12:16:06||Malicious version|
|1.0.0||2021-10-22 12:16:19||Malicious version|
|0.7.30||2021-10-22 16:16:08||Clean version|
|0.8.1||2021-10-22 16:23:53||Clean version|
|1.0.1||2021-10-22 16:26:19||Clean version|
The attacker managed to compromise the credentials of the lead developer and pushed, unbeknownst to everybody, a changeset updating all major versions (see Table 1). It took the NPM community four hours to realize that the attack had happened, help the lead developer secure his account, and push an update neutering the attack.
The changeset consisted of two additional preinstall scripts, a batch script for Windows, and a Bash script for Linux and macOS. The scripts would download either “jsextension” (https://www.virustotal.com/gui/file/ea131cc5ccf6aa6544d6cb29cdb78130feed061d2097c6903215be1499464c2e/detection) or “jsextension.exe” (https://www.virustotal.com/gui/file/7f986cd3c946f274cdec73f80b84855a77bc2a3c765d68897fbc42835629a5d5) from 185.173.36[.]219, a host based in Russia, and execute it. The Linux executable is just a variant of XMRig, a well-known cryptominer, while the Windows binary also exhibits some info-stealer capabilities to harvest passwords and credentials. Both executables have already been analyzed at length (see Figure 1 and Figure 2 for the analysis overview provided by NSX Defender) by many other security researchers. (A detailed technical analysis is available at https://blog.sonatype.com/npm-project-used-by-millions-hijacked-in-supply-chain-attack.)
While the attacker could have chosen to modify the JS library itself and thereby infect the end-user, the cybercriminal instead opted to add a script that executed at installation time. This was a significant choice, as it confined the execution of the malicious payload to only those endpoints where the command “npm install” normally executes: (1) the NPM developers’ workstations, and (2) CI/CD server(s) in data centers. This partly explains the choice of such malware: info-stealers (and cryptominers) for the workstations, naturally full of private and confidential data, and cryptominers for data centers, where CPU resources can be easily abused; only ransomware would have been more destructive.
In this specific incident, aside from a handful of reportedly infected workstations, the attacker did not manage to leverage this library to establish a foothold in a data center. However, even though the malicious library was not immediately pulled, one would think that publishing a new library version that simply worked — and allowed all dependant projects to quickly work around the issue, with the notable exception of applications pinning the malicious version of the library — is not an optimal solution. While technically correct, the method employed by NPM to resolve dependencies could have potentially made the problem much trickier to solve.
It’s a Dependency Problem
When developing a piece of software, it’s rare to find an instance in which no additional libraries are required. On the contrary, developers want to develop modular components so that they can be reused across different projects. The job of maintaining the list of dependencies — and the required versions of those dependencies — is orthogonal to the programming language and is often handled by a system component called a package manager.
- Pin the dependency by specifying the version number.
- Specify that updates to the library are accepted by using the “^” (minor update) and “>=” (major update) operators.
While the first option may be required in some settings (it’s useful for compliance as it allows reproducible builds), pinning a specific version discards any updates, even non-API-breaking security ones. For this reason, it is often advisable to at least allow for minor updates, in order to quickly catch up when upstream critical bugs are fixed.
Balancing these two options depends on many factors, but it is safe to assume that more sensitive environments favor the first alternative, and they can then opt to regularly review available updates (a bit like Linux distributions do). Unfortunately, the issue is that, regardless of which policy is chosen, it is not possible to enforce such a policy for libraries that themselves have multiple dependent libraries. In other words, once we specify that a piece of software depends on a specific version of a library “A”, if there is also another library “B” that depends on this very same library “A”, there is no explicit way for a developer to enforce that library “B” will use the same version they decided to depend on for their software project.
At the same time, a library developer might find this behavior perfectly sound: after all, correctness and functionality of a library can only be guaranteed if the set of dependencies used at development/testing time is the same as that used at deployment time (a fact that is painstakingly understood by system administrators). When there is a security breach, however, there is a clear scenario in which this type of enforcement may actually be desirable.