The shift towards AI is pushing cloud-native infrastructure into new territory, creating architectural demands, such as hardened sandboxes and burstable execution, that Kubernetes was not originally designed to meet. AI agents work in short bursts separated by long stretches of waiting for a model, a tool or a human response. Throughout these pauses, they continue holding state in memory and on disk. At the same time, agents also execute untrusted code, demanding hardened sandbox isolation. When workloads scale from thousands of long-lived pods to millions of ephemeral agents, standard primitives hit an all triggering IP exhaustion, routing churn and unsustainable cost of holding the compute resources.
Agent Substrate, an open-source CNCF project, solves the problem by providing a high density, secure runtime and scheduling framework for running AI agents at scale on Kubernetes. It multiplexes thousands of sandboxed, mostly idle agents onto a minimal pool of shared compute by rapid snapshotting, hibernating, and resuming their execution states on demand. The project aims to snapshot agent states under 100ms via incremental snapshots, achieving 10x higher compute density, and support over 1 million concurrent actors.
(More on Agent Substrate: https://ate.dev/)
Just as Kubernetes schedules pods onto nodes, Agent Substrate schedules Actors into worker pods. An Actor is the actual agent workload. But unlike a pod, an actor isn’t permanently pinned to physical infrastructure or bound to a continuous container lifecycle. When an actor needs to run, Agent Substrate schedules it into a warm pod running on a kubernetes node. The moment it goes idle, Substrate flushes a full memory and filesystem snapshot to fast storage, instantly freeing that worker pod to execute other workloads. Through Actor Teleport, when an event or response arrives, that actor can be scheduled onto any available worker in the pool, picking up execution seamlessly without losing state. Kubernetes handles what it does best: node provisioning, autoscaling, and worker recovery. Meanwhile, Substrate’s dedicated control and data plane handle actor scheduling and fast suspend-resume cycles.
Today, Agent Substrate relies on runtimes like gVisor for software-level isolation, optimizing for density and immediate startup. However, emerging AI workloads have diverse risk profiles that demand a broader spectrum of isolation options ranging from lightweight sandboxes to strict hardware boundaries. To support this we are exploring hardware assisted isolation models, including potential future integration with vSphere Pods, with running workers inside lightweight, dedicated Virtual Machines (VMs) would enforce zero-trust hardware isolation, while nesting gVisor inside these PodVMs would still preserve sub-millisecond sandbox creation.
This blog covers four areas:
- What a VMware vSphere Kubernetes Service (VKS) add-on gives you
- Packaging an open-source project as a VKS add-on
- Installing the add-on on a VKS Cluster
- A live demo of actor suspension, snapshotting, and cross-worker resumption
What a VKS add-on gives you
A quick note before diving in: while VKS supports add-on extensibility, the specific add-on, manifests, and artifacts shared in this post are intended solely for demonstration. The artifacts featured here are not officially supported, and these examples should not be treated as supported production configurations. You can learn more about VKS add-ons here.
As teams bring emerging open-source AI frameworkers into production, the VKS add-on framework provides a standardized, fleet-wide model to integrate, configure and lifecycle manage them. Rather than treating each tool as an add-hoc operation silo, VKS turns complex open-source components into first class citizens of VKS.
- Catalog Entry
An Addon in the Supervisor. - One configuration API
An AddonConfigDefinition holds the schema, defaults and bounds. Users set per-cluster values in an AddonConfig. - Fleet rollout by labels and constraints
An AddonInstall selects the clusters using label selectors or constraints defined using CEL expressions. - Per-cluster status
A ClusterAddon shows the release and whether it’s Ready. - Declared compatibility
A Kubernetes version constraint on the package, here >=1.37.0 <1.38.0. - Managed lifecycle
The add-on is reconciled continuously and self-heals. VKS add-ons integrate well with Cluster API (CAPI): when the cluster upgrades, the add-on is updated to the latest release that is compatible with the constraints it defines, unless a release is pinned for the cluster allowing upgrade to be managed explicitly.
Everything is declarative YAML, so it fits GitOps. Users don’t have to learn each project’s own installer, and admins get one workflow for every add-on. The VKS documentation describes the model in full: Managing Add-ons in VKS Clusters.
Packaging Agent Substrate as a Helm add-on
Substrate ships install manifests (Deployments, DaemonSets and the like), not Helm charts. VKS add-ons natively support both Carvel and Helm packaging formats, we chose Helm for Agent Substrate to leverage standard Helm chart tooling. Here is how you can create the add-on:
Package and Publish
We published the Helm chart as an OCI artifact and declared the package metadata in the catalog namespace.
Structure as a Helm Chart
A script template turns the upstream v0.5.0 manifests into a structured Helm chart with templates for all 75 resources. The resources can be found here https://github.com/agent-substrate/substrate
Fill in what upstream leaves to the operator
A bootstrap Job generates the signing keys inside each cluster, so no key material ships in the package. PostgreSQL and an S3 compatible store (RustFS) come bundled, so the install is self contained.
Expose Helm Values Schema
Ten core values are exposed through standard Helm values.yaml and typed via values.schema.json: replica counts, storage sizes, token issuer, and other parameters.
Parameterize Manifests
Standard Helm templating constructs map user configuration directly into resource manifests and coordinate deployment sequencing.
Package and Publish
We published the Helm chart as an OCI artifact and declared the package metadata in the catalog namespace.
The commands from here on use three helpers:
Publishing puts three resources in the catalog namespace on the Supervisor. Addon is a metadata object: a description, a display name and similar information. It’s the catalog entry:
AddonRelease is one version of the add-on, here Substrate v0.5.0, together with its Kubernetes compatibility constraints:
AddonConfigDefinition holds the configuration schema and the configuration templating for one release of the add-on. The template turns a cluster’s settings into a value Secret for the package. Ours defines ten values. This is a trimmed copy:
Packaging Agent Substrate as a standard Helm chart makes it seamlessly compatible with VKS add-on management while retaining standard Helm configuration workflows.
Installing the add-on
Installing using VCFA UI
Agent substrate can be installed from the addons catalog in the VCFA UI.
Installing using Command Line
An AddonInstall selects clusters with labels and with constraints written as CEL expressions. This one selects a fleet of clusters by matching labels:
Settings for one cluster go in an AddonConfig, validated against the schema. This one asks for one API replica and ten workers:
A ClusterAddon represents the add-on for one workload cluster and reports its status there. Once a cluster is selected, VKS installs the add-on, and when the ClusterAddon says Ready, Substrate is running:
In the workload cluster, the package brought up the control plane, the routers, a pool of ten workers, RustFS and an atelet on the node. A second namespace holds the certificate controller:
Actors run under gVisor. The script checks the sandbox configuration and the worker image:
The demo
The following demonstration explores Agent Substrate in VMware vSphere Kubernetes Service (VKS) environment: https://youtu.be/mDfZmFEvCdw.
Four-agent multiplexing and egress policy
Fleet Setup
To showcase multi-tenant isolation, routing, and density, we create a shared atespace named vks-demo and instantiate four distinct actors Luna, Mars, Orion, and Vega from a standard agent template. Fine-grained egress network policies are attached to enforce strict endpoint access per actor: Luna and Mars are restricted exclusively to Anthropic model endpoints, while Orion and Vega route solely to OpenAI.
Execution and Lifecycle
All four actors are multiplexed onto just two worker pods in the pool. When incoming requests arrive, the atelet router schedules Luna and Orion to active worker pods. Inside each worker, gVisor intercepts system calls in user space, enforcing secure sandbox isolation for untrusted code execution. As soon as an actor pauses or completes a turn, Substrate captures an incremental memory and filesystem snapshot to fast storage and hibernates the actor, instantly freeing worker pods to process subsequent actor workloads. When a new event arrives, Substrate seamlessly resumes the actor on any available worker pod in under 200ms on an average (depending on available bandwidth and the actor’s memory footprint) without losing execution state.
Addressing the complex requirements of modern AI demands an architectural framework built across multiple operational layers as cloud-native environments expand. Novel developments like the VM API extend standard Kubernetes control planes to govern lower-level compute components, whereas Agent Substrate focuses on handling the distinct isolation, scheduling, and state requirements inherent to agentic systems. By leveraging VKS as a unified enterprise foundation, operators can easily deploy, configure, and manage these open-source tools across entire fleets. Showcasing these capabilities highlights our ongoing focus on evolving core Kubernetes primitives equipping platform teams with a robust infrastructure designed to support dense, highly isolated AI workloads safely at scale.
Agent Substrate project · Managing Add-ons in VKS Clusters
Discover more from VMware Cloud Foundation (VCF) Blog
Subscribe to get the latest posts sent to your email.
