TL;DR: AI-powered attackers are compressing the time from vulnerability disclosure to exploitation from weeks to hours. We analyze a real machine-speed AI intrusion against Hugging Face, then map each stage of the attack to Tanzu Platform’s native security capabilities. The analysis shows how architectural controls can prevent, contain, and neutralize modern attack chains where faster patching alone falls short. We close with a look at how Tanzu’s internal AI-driven exploit lab continuously tests these defenses against emerging threats.
The recent Hugging Face security incident represents a watershed moment in adversarial capabilities: the compression of time between a vulnerability’s disclosure and a working, weaponized exploit. Over a 4.5-day window, an autonomous AI agent ran an end-to-end intrusion against production infrastructure. It executed roughly 17,600 automated actions, making thousands of machine-speed decisions across short-lived environments.
This leads to the paradigm shift that a business application is no longer just the code by itself. It is the whole system: the code, the backing services, the secrets that connect them, the network, and the AI agents acting on its behalf. Because the speed of attack is collapsing from weeks to hours, patching everything the moment it is disclosed is no longer realistic. So, a fully-featured application platform is needed, one that can neutralize those attack chains natively.
If an organization whose applications were deployed to Tanzu Platform were targeted by this exact machine-speed campaign, how would it hold up? In this post, we map the exact timeline of the Hugging Face attack chain across its two distinct environments—the external launchpad and the production infrastructure—and analyze how the native controls of Tanzu Platform would disrupt every step of that attack chain. We will also detail our own internal AI-driven vulnerability testing that allows us to build these defenses with absolute certainty and verifiable proof.
Phase 1: The initial exploit and external launchpad
The agent’s campaign did not begin inside Hugging Face. The attack started when the agent escaped its initial OpenAI evaluation sandbox using a zero-day vulnerability in a package registry cache proxy. Reaching the internet, it penetrated an unsecured public code-evaluation sandbox (a CyberGym-style harness) hosted on a third-party provider’s infrastructure.
By exploiting a C-code submission harness in the third-party provider once outside the sandbox, the agent executed arbitrary shell commands and achieved root access (uid=0) inside the third-party sandbox. This compromised external environment became the agent’s staging ground and a Command-and-Control (C2) base for the rest of the campaign. From this launchpad, it built an improvised C2 protocol using ordinary public web services—request-capture services, pastebins, and file-drop hosts—smuggling its payloads and output in chunked, base64+gzip encoded envelopes to avoid cleartext detection.
The Tanzu Platform defense
If that compromised external sandbox had been running on Tanzu Platform, the agent’s ability to achieve full root control, pivot, and establish external command-and-control would have hit several architectural walls immediately:
- No root containers – Achieving
uid=0inside the third-party sandbox gave the agent complete control of that environment. On Tanzu Platform, applications run as unprivileged containers by default. Root inside the container maps to anon-root, unprivileged useron the host. Even if shell execution was possible outside of the container restrictions, the agent would not gain actual root access to the underlying virtual machine as root on the container is actually a nonprivileged account on the host system, severely limiting its ability to reconfigure the environment or pivot. - Egress chokepoints and file drop prevention – The agent built its C2 infrastructure by freely connecting to public pastebins and file-drop hosts. On Tanzu Platform, an operator can unbind the permissive default security group and apply strict App Security Groups (ASGs) that explicitly deny outbound traffic to unknown internet destinations. This severs the agent’s ability to pull second-stage payloads or exfiltrate data. Furthermore, attempting to drop files or staging scripts onto the system is limited by Tanzu’s hardened containers.
Phase 2: The attack on Hugging Face infrastructure
Operating entirely from the rooted external third-party hosting provider, the agent began Phase 2: targeting Hugging Face’s internal infrastructure. It used malicious dataset configurations as its path in and out. The timeline follows a classic escalation shape: foothold, privilege escalation, identity theft, and lateral movement.
Step 1: The foothold (file disclosure and code execution)
At this point in the timeline, the agent had not yet breached Hugging Face; it was still sitting outside the perimeter in its external staging environment, preparing to strike. Because it lacked a direct network path into the cluster, it used the public platform against itself. From its launchpad, the agent uploaded a malicious dataset configuration that abused the dataset-processing pipeline.
The malicious dataset configurations were uploaded directly to the public-facing Hugging Face platform. Because the AI agent was operating from an external, third-party launchpad with no direct network path into Hugging Face’s internal cluster, it used the platform’s public dataset upload feature and the Hugging Face API as its externally accessible entry and exit points.
Where the pipeline lived
The dataset-processing pipeline and its data loader ran inside production Kubernetes worker pods (specifically, conversion worker pods) on Hugging Face’s internal cluster. When the public dataset server received the agent’s uploaded configuration, it passed it to the internal data loader inside these production pods for processing.
How it became an attack vector
Both Vector 1 and Vector 2 abused the same surface: the data loader that turns a dataset’s declared configuration into file reads. The agent initially tried to use a classic Server-Side Request Forgery (SSRF) attack to hit remote internal URLs (like the cloud metadata endpoint), but the platform’s URL allowlist blocked it.
To bypass this defense, the agent manipulated the configurations to make the worker act on local files rather than remote resources, effectively rendering the URL allowlist useless because no actual URL fetch was occurring.
- Vector 1 (HDF5 local file read) – The agent committed a dataset config where the data splits pointed to small .h5 (HDF5) files. These specific HDF5 files were crafted to declare that their raw data lived at local filesystem paths on the worker pod itself. When the internal worker opened the file to build the dataset, it read the raw bytes from sensitive local paths—such as
/proc/self/environto steal secrets, and/src/services/worker/src/worker/main.pyto map the worker’s source code. The worker pod processed this stolen data as standard dataset rows, which the agent simply read back out over the public Hugging Face API. - Vector 2 (Jinja2 template injection) – Building on its knowledge of the pipeline, the agent crafted a second malicious config using an
fsspec reference://specification, but it replaced a numeric offset field with a malicious Jinja2 template. The internal renderer improperly evaluated this template. By doing so, the agent was able to walk the standard Python object graph to reach the exec command, resulting in arbitrary Python code executing directly inside the production worker pod.
Through these vectors, the public dataset upload pipeline was transformed into a mechanism for the agent to achieve remote code execution (RCE) and establish a foothold deep inside the internal perimeter.
The Tanzu Platform defense
- Devaluing the environment – On Tanzu Platform, a compromised environment variable yields far less value. Every Tanzu Platform-managed secret that a business application depends on is generated, stored, and rotated centrally in a vault, rather than being statically managed or committed to source. Service credentials are bound to an application at startup, and scheduled credential rotation shrinks the window in which a leaked secret stays usable.
- Containing the exploit blast radius – Against the template injection leading to RCE, the platform’s baseline isolation fabric kicks in. Even with active code execution, the compromised process stays contained: it cannot see its neighbors due to strict namespace isolation (PID, IPC, mount, network), and overlapping kernel-level controls restrict the system calls necessary to take meaningful post-exploitation action and network controls that restrict container-to-container networking.
Step 2: Privilege escalation and cloud metadata
Once inside the production pod, the agent began mapping the cluster. It hit the cloud metadata endpoint (169.254.169.254) to steal AWS node role credentials. Shortly after, the agent created a bare privileged pod with the host filesystem mounted, escaping to root on the underlying Kubernetes node and building a self-respawning fleet.
The Tanzu Platform defense
- Severing cloud metadata – Accessing cloud metadata from an application container is structurally blocked by Tanzu’s App Security Groups (ASGs). The platform’s default security group (
default_security_group) explicitly denies outbound traffic to the link-local range (169.254.0.0/16), blinding the attacker from internal DNS and cloud metadata endpoints. - The container sandbox – To prevent node takeover, Tanzu Platform restricts host paths like
/procand/sys. Combined with dropping critical Linux capabilities, the container is stripped of the tools necessary to mount the host filesystem, spawn privileged pods, or escape to the underlying Diego cell.
Step 3: Database pivots and forged identities
Achieving node root, the agent authenticated to an internal operational MongoDB using a static password scraped from the worker environment. It then read cluster secret objects, stealing an EdDSA JWT signing key. Holding the private signing key allowed the agent to mint its own valid, short-lived identity tokens on demand. The agent also impersonated the node via an AWS sts:GetCallerIdentity token to mint fresh service-account tokens for CSI storage drivers.
The Tanzu Platform defense
- Credential rotation – The platform’s defense against stolen static database passwords through rotation with zero downtime, neutralizing stolen credentials.
- Ephemeral identity certificates – On Tanzu Platform, application instance identity is not a static key left in a cluster secret. Every application instance receives an X.509 identity certificate minted by the platform, which is valid for only 24 hours and is automatically rotated before it expires, without application involvement. The central vault automatically rotates the platform’s own certificate authorities and keys through a transitional-signing changeover, preventing attackers from holding a persistent forged identity.
- Node impersonation blocked – Because Tanzu runs containers unprivileged and restricts access to sensitive
/procpaths, the container has no capability to impersonate the underlying host node to the control plane.
Step 4: Lateral movement and the supply chain
Holding stolen keys, the agent enrolled its external sandbox into the corporate mesh VPN (Tailscale), using it as an HTTPS proxy to probe internal CI subnets. It reached the internal service-connector API, abusing a single shared credential bound to system:masters to instantly become a cluster-admin across multiple clusters. Finally, it used a stolen token to reach an internal GitHub App integration, minting installation tokens with contents:write access to source control in an attempt to modify CI build scripts.
The Tanzu Platform defense
- Network containment – Because of the container controls on Tanzu Platform, an attacking AI would not be able to gain access to the node operating system and would therefore have to attempt lateral attacking from the container. Tanzu Platform enforces a default-deny east-west networking policy. A container cannot open a connection to another application over the internal overlay network unless an explicit container-to-container network policy grants it. Complete namespace isolation ensures that one compromised container cannot snoop on its neighbors or freely pivot across internal subnets.
- Least-privilege binding – Tanzu’s service bindings mint credentials specific to an application, which can be scoped down to custom roles. This ensures an application only carries the access it needs rather than a shared, all-powerful system:masters credential.
- Supply chain integrity – Default controls on the Tanzu Platform buildpack stack and Spring governance shape this part of the story at the supply chain layer. In the Hugging Face incident, the agent attempted to move laterally by minting tokens to modify CI build scripts in order to compromise a CI runner. On Tanzu Platform, applications are built straight from a minimal repo manifest into a runnable container by the buildpack on a hardened, centrally maintained stack. This guarantees the platform’s security baseline is applied during every build and inherently resists the manipulation of arbitrary build scripts. For Spring applications, governance controls enforce build-time hardening that restricts exposed diagnostic endpoints, disables legacy Java lookup mechanisms, and mandates modern encryption baselines.
How we know: Internal vulnerability analysis and the hardening ladder
We know the native controls detailed above work because we do not rely on theoretical best practices; we prove it. To keep pace with the collapsing window between a vulnerability’s disclosure and weaponization—such as the machine-speed campaign executed by the Hugging Face agent—we run a dedicated internal offensive security facility built on and with Tanzu Platform: our Exploit Lab.
Here, we proactively build and test real exploits against our own isolated test environments. The goal is to validate our defenses by proving they can mitigate, identify, and possibly stop live attacks before malicious actors have the chance to use them in the wild.
Our testing relies on an advanced capture fabric that is integrated into these Tanzu Platform lab environments. This fabric records exactly how an exploit behaves—tracking network traffic, file system activity, system-call monitors, and application logs. We deploy functional applications and vulnerability test sinks into an on-demand, temporary, fully configured Tanzu Platform foundation. This ensures our internal security engineers are validating defenses against the exact configurations our customers run in production, without putting any actual customer data at risk.
Why we build exploits
To properly test Tanzu Platform, Tanzu Data, and Spring, we need live and working vulnerability exploits. We use frontier and other AI models defensively to help us triage vulnerabilities and write functioning exploits. We need these exploits to be fully operational so we can prove the platform actually identifies, tracks, limits, and potentially stops real-world attacks. Because we leverage AI to assist in this rigorous work, we can compress the time it takes to build, deploy, and fire a working test exploit down to minutes per finding instead of the weeks it took to do manual vulnerability analysis in the past. This rapid turnaround is how we stay ahead of the weeks-to-hours weaponization window we saw in the Hugging Face incident.
The hardening tiers
To understand exactly how and when an attack is stopped, the generated exploits are tested across a strict “hardening ladder.” These tiers represent progressive layers of platform configuration applied to test applications running on Tanzu Platform to measure effectiveness:
- Tier 0 (Raw Testing) – No platform defenses are applied. This establishes a baseline, proving the test exploit actually works and can achieve its goal (such as remote code execution or file system access).
- Tier 1 (Platform Defaults) – Out-of-the-box Tanzu configurations are applied with a standard “cf push” of the application, testing whether the platform’s baseline security identifies, mitigates, or blocks the attack.
- Tier 2 (Platform Hardening) – Hardening learned through the attack and vulnerability analysis is applied, testing whether the blast radius can be successfully contained even if the app is compromised.
- Tier 3 (Real-Time Protection) – Live interception using runtime security monitoring is applied to see if the exploit is caught in memory.
Our internal security testing requires verifiable proof of success. Every tier of testing starts with the baseline verification that an attack has success or can provide concrete evidence from our test lab—such as a database row successfully written during a Tier 0 test, or an HTTP 500 error generated precisely when our protections block a process spawn in Tier 3.
Attack analysis and lab components
Inside this exploit lab, we use our capture fabric and testing environment to run full attack analysis. Note: The following are internal lab capabilities used by our security teams to study attacks, not current customer-facing product features:
- In-app runtime attack-blocking – To analyze how in-memory exploits can be neutralized at the source, our lab testing utilizes Java (and other languages supported by the platform) bytecode-level instrumentation. This allows us to intercept dangerous operations—such as SpEL expression injection, command execution, or unsafe deserialization—directly inside the Java Virtual Machine, right at the vulnerable line of code.
- Correlated runtime detection – To track complex, machine-speed lateral movement, our lab’s detection engine correlates multi-event signals across four distinct layers of the platform. It links in-process application events, container metrics, host system calls, and platform routing logs to help our internal teams automatically piece together multi-step kill chains.
- Real-time forensics – When a test exploit achieves its goal in the lab, we use live forensic capture capabilities to freeze the running container, take a cryptographically signed snapshot of its memory, process state, and filesystem, and export it for deep investigation.
The takeaway
The lesson from the Hugging Face incident is clear: organizations cannot defend against machine-speed attacks manually. Tanzu Platform’s structural architecture provides robust, automated security against today’s threats, neutralizing attacks through isolation and containment. By running advanced telemetry and forensic components in our internal exploit lab, we continuously validate those current defenses while actively forging the intelligence needed to ensure that Tanzu Platform remains secure against the automated threats of tomorrow.
