Securing AI Workloads

(Day 11 of 22 in our Cybersecurity Awareness Month series. Today: securing AI workloads)

Running AI yourself keeps your data in your own environment, under your own watchful eyes. Even though AI is a relatively new thing, most of the work to secure it is familiar. AI workloads consist of VMs, containers, networks, storage, and applications, and the VMware Cloud Foundation (VCF) tools and practices that protect your other workloads apply here, too.

Here we build on our previous posts. We take the parts from our post on the anatomy of an AI and cover how to secure each one in VCF, addressing the real threats from our post on AI security threats along the way. When you’re ready to design the whole thing, our Securing AI Workloads whitepaper (PDF | Markdown) goes much deeper, from threat modeling and model governance to worked vDefend rule examples.

Start with the basics covered elsewhere in this series. Harden the infrastructure using the Security Configuration Guide and keep management interfaces isolated. AI workloads sit on top of all of that and inherit those protections.

The network is the other half of that foundation. VMware vDefend enforces firewall policy at every workload’s virtual network interface, in the hypervisor, which is a bit like posting a guard at every door instead of one at the front gate. Those guards sit outside the guest, where software running inside the workload doesn’t control them, so even a component that has been manipulated, like an agent following injected instructions, can only reach what the policy allows. vDefend can’t tell a malicious prompt from a good one on a connection it allows, but it limits how far the damage can spread.

Use vDefend’s Security Intelligence to see the traffic each part of your AI workloads actually generates, and write rules that allow only what’s needed. That traffic is a record of what happened, which can include a misconfiguration or a compromise, so allow a flow only when the application needs it. Turn on vDefend’s intrusion detection and prevention for the paths that matter most, like the ones into your inference servers. The sections below show where it fits.

Hosted or Private AI Workloads

The first big decision is where your AI workloads run. A hosted AI service is quick to start, and the provider handles the model, the hardware, and the inference servers. In exchange, your prompts and documents leave your environment, often for infrastructure shared with other customers. From there, you’re relying on the provider’s controls, retention policies, and contract terms, with little ability to verify any of them, and the provider can change the model or those terms on its own schedule. In a world of zero trust this approach goes the other way, asking you and your organization for a lot of trust.

Private AI keeps your data inside your environment, under your access controls and your audit trail. You decide who can reach it, how long it’s kept, and when a model changes. Of course, you also take on the work of running it, though on VCF that work looks a lot like running everything else, which you already know how to do.

Many organizations use both. Let your existing data classification decide which work goes where, and default to keeping it private. Only public or low-sensitivity work should go to a hosted service, and only to one you have vetted and have an agreement with. Sensitive data, regulated data, and anything you can’t afford to see elsewhere stays private.

VMware Private AI Services, a native part of VCF, can help you govern both the use of hosted models, as well as helping you run the private side next to the rest of your infrastructure. Its services map to the parts below: Model Runtime, Data Indexing and Retrieval, MCP Servers and Tool Gallery, and Agent Builder, with approved models distributed from a Harbor model gallery.

Securing the Infrastructure Parts of AI Workloads

GPUs and Hosts

VCF gives VMs access to GPUs in two main ways. With NVIDIA vGPU, several VMs share a physical GPU, and those VMs can move between hosts with VMware vSphere vMotion. vGPU can share a GPU in two ways. Time-slicing gives each VM its own portion of the GPU’s memory, while the VMs take turns on its processors. Multi-Instance GPU (MIG), on GPUs that support it, partitions a GPU into hardware-isolated instances with their own compute and memory, which reduces the paths workloads share without removing every side channel. With passthrough, also called DirectPath I/O, a VM gets a whole GPU to itself, but it can’t move with vMotion. Where your devices support it, use Enhanced DirectPath I/O instead, which pairs near-native performance with vMotion and suspend and resume, so host maintenance doesn’t have to wait for a downtime window.

• Decide which AI workloads can share a GPU. Workloads with different owners or different levels of sensitivity may belong on separate MIG instances, separate GPUs, or separate hosts. Use VM classes with specific GPU profiles to enforce that, since each namespace can use only the classes it is given.
• Plan for passthrough. When you choose GPUs, favor ones that support Enhanced DirectPath I/O, and check the Broadcom Compatibility Guide for which features each one supports. VMs on plain passthrough need planned downtime for host maintenance, and giving them their own cluster keeps them from holding up everything else.
• Consider confidential computing for the most sensitive work. CPUs with AMD SEV-SNP or Intel TDX can protect a VM’s memory while it’s in use, even from the infrastructure underneath. VCF 9.1 supports confidential VMs on both of those CPU technologies, so use them for the components that handle your most sensitive data. The protection covers the VM’s memory, not data in GPU memory.

Models

• Keep a private, curated model store. Pull a model from a public repository once, test it, and serve it from your own store. Use the Private AI Services Model Gallery for this. It’s built on a Harbor registry, with access controlled by project. Use vDefend rules to make that store the only place production AI workloads pull models from. Do the downloading in a separate intake environment, and open its outside access only while you need it.
• Record where each model came from. Keep a checksum or signature, so you can tell if a model has changed. Keep track of where each model runs, too, so an unexpected copy stands out.
• Protect fine-tuned models like the data behind them. A model fine-tuned on sensitive data needs the same access controls and encryption as that data.

Inference Servers

• Require authentication on every model endpoint. Private AI Services checks authorization on every API and UI request, including calls to model endpoints. Connect it to your identity provider through OpenID Connect (OIDC), directly or through VCF Automation, so people and applications use identities you already manage, and keep its local accounts for testing.
• Keep them off the internet, and isolate them. Most inference servers only need to talk to a few applications. Use VMware vDefend Distributed Firewall to limit what reaches them and what they can reach. Users should reach only the application’s front end, while model endpoints, the vector database, and ingestion stay off user networks.
• Put VMware Avi Load Balancer in front of the application. Its web application firewall, rate limiting, and DDoS protection help protect the front end people use, while the inference servers stay behind it. Health checks and global load balancing can keep the application answering when a server, or a whole site, fails.
• Plan for recovery. VCF Protection and Recovery, called VMware Live Recovery in VCF 9.0, helps make AI workloads resilient by replicating them to another site and orchestrating their recovery there or, after ransomware, into an isolated clean room. The recovery site needs compatible GPUs of its own.
• Set rate limits and watch usage. That way, a runaway script or an attacker can’t monopolize your GPUs. Public-facing chatbots need request size and rate limits most, since anyone can use them as free compute. Use the GPU monitoring in VCF Operations to spot unusual usage.
• Log requests carefully. Logs help with investigations, but prompts can contain sensitive data. Protect and retain those logs accordingly, and if you collect them in VCF Operations, limit who can read them there, too.

Securing the Data and Actions in AI Workloads

Retrieval and Vector Databases

• Carry permissions through retrieval. The application should only retrieve passages the person asking is allowed to read. Private AI Services indexes each data source with one set of credentials, so give it an account that can read only what the knowledge base should contain. Filter by the asker’s permissions in the application, before passages reach the model, since retrieved text is untrusted input and the model shouldn’t decide what the asker may see. Test this with accounts that have different levels of access.
• Protect the vector database like the source data. Encrypt it at rest, back it up along with the model version and prompts it was built for, and restrict who can query it directly. In Private AI Services, that database is PostgreSQL with the pgvector extension, so your existing database practices apply. Use VMware vSAN data-at-rest encryption or VM encryption to encrypt it. Where data owners differ, like HR and engineering, give each its own vector database, and use vDefend rules so each application reaches only its own, which makes a stronger boundary than filtering one shared index.
• Control who can add documents. Anyone who can put a document into a knowledge base can try to put instructions in front of the model. In Private AI Services, knowledge bases pull from data sources like document stores, wikis, and object storage, so anyone who can edit those sources can reach the model, too. Files uploaded directly to a knowledge base are another way in.
• Keep the index in step with the source. When access to an original document changes, or the document is deleted, the indexed copy should change, too. Data Indexing and Retrieval keeps indexes current with their sources. Test that deletions come through, and handle changes to who can read a document in the application.

Agents and Tools

• Give each agent its own credentials. Grant each one the least privilege it needs, and avoid sharing one powerful service account across agents. Approving a tool in the tool gallery controls which tools an agent can see, and the MCP server, or the system behind it, still has to authenticate the caller and authorize each action.
• Require approval for consequential actions. A person should approve actions like deleting data, changing configurations, or moving money.
• Allow only reviewed tools. Keep a list of approved tools and tool connectors, such as Model Context Protocol (MCP) servers. Agents should not be able to add new tools on their own. Use the Private AI Services tool gallery to manage MCP servers and turn tools on in one place, and run MCP servers as vSphere Pods so the hypervisor isolates each one. An agent that runs code a model wrote needs its own VM, because containers on the same node share a kernel.
• Limit network egress. An agent that can’t reach the internet can’t send your data there, even if someone tricks it into trying. Use VMware vDefend Distributed Firewall rules to enforce that for agent workloads and to limit which agents can talk to each other.
• Cap the loop. An agent keeps working until the model decides it’s done, and every tool result is new input that can carry instructions. Cap iterations, time, and spend in the application. vDefend rules apply on every pass, so even a redirected loop stays within its allowed destinations.
• Control the path to the model. Send model traffic directly to models you run, or to providers you have an agreement with. Third-party routers in between can read and change everything an agent sends and receives. When the Private AI Services inference gateway routes to a model outside your environment, set its TLS verification to strict, so it checks that it is talking to the right endpoint.
• Log every action, and keep an off switch. Record what each agent does. Turn on Private AI Services tracing to send interactions between users, models, agents, and knowledge bases to an OpenTelemetry collector. Know how to stop one quickly by revoking its credentials, cutting its network access, and shutting down its VMs. Revoking access in the identity provider can lag behind cached tokens and service credentials, so test that the agent actually stops.

Guardrails and Outputs

• Use guardrails as one layer. They help, but they are not a security boundary on their own. Pick content moderation, jailbreak detection, and output filtering that fit the application, and run them alongside Private AI Services.
• Check output before anything acts on it. Treat code, commands, and database queries written by a model like input from an untrusted user. Before installing a package a model suggests, confirm that it’s real and comes from the publisher you expect.
• Keep a person in the loop for decisions that matter. There’s an old adage that a computer can never be held accountable, so it should never be allowed to make decisions. AI doesn’t change that.

People, Policy, and AI Workloads

Technology covers part of securing AI workloads. The rest is people and process.

• Give people an approved AI option. If the only way to use AI is a public service, sensitive data will end up there.
• Write down which data can go where. Use your existing data classification to decide which AI services can handle which data.
• Train people to verify unusual requests. Confirm requests for money or access through a second channel, because attackers can fake voices and faces. That goes double for administrators with broad access.
• Treat AI systems like everything else you run. Include them in your inventory, your incident response plans, and your audits.

Running AI workloads in VCF puts them on the platform that already runs the rest of your infrastructure, so the tools you know apply. VMware vDefend Distributed Firewall isolates inference servers and agents, and vSAN and VM encryption protect models and vector databases at rest. VCF Operations tracks GPU usage, and Private AI Services keeps models, retrieval, and agents behind your own identity provider. Prompt injection and agents that take action still need the most care, because no product closes those risks on its own.