Verifying AppArmor Enforcement on VMware vSphere Kubernetes Service

AppArmor support in Kubernetes is easy to configure and easy to get wrong quietly. A pod can be scheduled, come up Running, and report Ready: True while the profile you thought was attached either isn’t loaded, isn’t the one you expect, or is enforcing something different from what you intended. The only way to be sure is to test it end to end: confirm the profile is actually attached, prove the denial happens, and confirm you can see that denial in an audit trail. This guide is primarily written for VMware vSphere Kubernetes Service (VKS) platform and security admins who need to prove AppArmor enforcement across their clusters.

This post walks through verifying AppArmor profiles across their lifecycle on a VKS cluster. The core strength of VKS is the AppArmorProfile Custom Resource, which allows platform teams to declaratively manage and distribute custom AppArmor policies across all worker nodes. We will walk through checking enforcement for the default RuntimeDefault profile, using manually loaded profiles for rapid development and testing, leveraging the AppArmorProfile CR for cluster-wide production deployment, and tracking down where audit logs are stored.

The three ways a profile gets onto a node

Before testing anything, it’s worth being clear about which delivery path you’re actually exercising, because each one fails differently.

1. RuntimeDefault: The pod’s securityContext.appArmorProfile.type is RuntimeDefault. The container runtime (containerd, for VKS) generates a profile from a built-in template, and at container-create time loads it under a runtime-owned name (cri-containerd.apparmor.d for containerd’s CRI plugin). You never see profile content in your own manifests; the rules come from the runtime’s default template.
2. Localhost with a manually loaded profile: The profile is authored as a .conf file (or heredoc) and loaded onto a specific node with apparmor_parser directly. The pod references it by name via securityContext.appArmorProfile.localhostProfile. This is the fast path for iterating on a profile while you’re writing it, but it’s node-local and not managed declaratively; it won’t survive a node rebuild and it doesn’t roll out to the rest of the pool.
3. [Recommended] Localhost delivered through an AppArmorProfile CR: The profile is defined as a custom resource in Kubernetes and distributed automatically to every worker node in the cluster. It is delivered by the VKS controller combined with cluster configuration settings (osConfiguration.securityContext.appArmor.profiles). Use this production-ready path to declaratively enforce a uniform, persistent security baseline across all nodes without manual per-node setup.

The rest of this post tests each of these and closes the loop with audit-log verification.

A prerequisite: Pod Security Standards decide whether your test is even meaningful

Before testing a Localhost profile, check what Pod Security Standard (PSS) level is enforced on the namespace:

A Permission denied result alone does not prove AppArmor caused the denial. On Linux, file access is evaluated by Discretionary Access Control (DAC) permission bits before Mandatory Access Control (MAC) like AppArmor is ever queried. If DAC denies the operation first, such as when Pod Security Standards (PSS) force the container to run as a non-root user without file write permissions, access is blocked at DAC layer and AppArmor is never invoked.

To ensure your test results are meaningful, execute test pods in a namespace configured with pod-security.kubernetes.io/enforce: privileged. Bypassing standard PSS DAC restrictions guarantees that any observed Permission denied error is directly attributable to an AppArmor deny rule rather than general Linux file permissions.

Step 1: Confirm RuntimeDefault enforcement

Containerd’s default AppArmor template (contrib/apparmor/template.go) is what gets rendered and loaded when a pod requests RuntimeDefault. It’s deliberately permissive for normal container operation, denying only a specific set of sensitive paths and operations.

Before you begin testing the template, consider the following:

• The bare file, network, and capability lines grant broad access. The profile’s job is narrowing specific dangerous corners, not locking the container down generally. Don’t expect a RuntimeDefault container to fail on ordinary file writes; it’s not that kind of profile.
• None of the deny rules carry the audit keyword. As covered in Step 4 below, that has a direct effect on whether you’ll see a log entry when one of these rules fires.
• Containerd (like Docker before it) applies its default AppArmor profile to any container that doesn’t explicitly request Unconfined.
• Separately, Unconfined is one of the three valid values for securityContext.appArmorProfile.type in the Kubernetes API (alongside RuntimeDefault and Localhost), but it is not in the Baseline Pod Security Standard’s allowed-values list for that field (only Undefined/nil, RuntimeDefault, and Localhost are permitted there), and the Restricted policy inherits that same restriction from Baseline.
• Put those two facts together: in any namespace enforcing baseline or restricted, a pod that specifies no appArmorProfile at all is still guaranteed to end up confined by containerd’s RuntimeDefault profile, since Unconfined is blocked by admission and the runtime applies its own default whenever nothing else is specified. Keep this in mind when you’re trying to figure out why a workload with an apparently empty securityContext is still hitting AppArmor denials.

To verify enforcement, confirm the profile is actually attached, then try one of the specifically denied operations from inside the pod:

A Permission denied here confirms the profile is loaded and enforced. It does not mean you’ll see an audit record for it. That’s a separate question, addressed in Step 4.

Step 2: Confirm a custom profile delivered via the AppArmorProfile CR

While manually loading profiles is ideal for rapid development and testing on a single node, using the AppArmorProfile CR path is the standard approach for production for a workload that needs something narrower than the default (for example, denying all writes rather than a specific path list). It enables declarative, lifecycle-managed distribution across all VKS nodes in a cluster, ensuring consistent enforcement and seamless persistence through node upgrades and replacements.

The Cluster object’s topology variables then reference the profile by name so the controller knows to load it on every node in the pool. Add the following cluster variable inside Cluster object:

And a pod requests it by name via securityContext.appArmorProfile:

The important consistency check here: the CR’s metadata.name, the profile name inside spec.content, the name listed under osConfiguration.securityContext.appArmor.profiles, and the pod’s localhostProfile all have to match exactly. If any one of them is a different name (for instance, if you have an AppArmorProfile CR named k8s-node-hardened-profile sitting in the cluster with unrelated content, while the spec.content and pod both reference k8s-apparmor-example-deny-write), the declarative path is not actually delivering the profile your pod thinks it’s using. That mismatch is easy to miss because everything still “works,” just not through the mechanism you assumed.

Verify attachment and enforcement the same way as before:

Confirming reads still succeed (ls /etc) is a useful sanity check too. It proves the profile is a targeted deny-write, not a broken profile failing closed on everything.

Step 3: Merging a locally-tested profile back into the CR

The fastest iteration loop for profile authoring is local: load a draft profile directly on a node with apparmor_parser and point a test pod at it, without going through the CR/controller reconciliation cycle at all.

This is deliberately fast and disposable (i.e. does not persist restarts): apparmor_parser -q loads straight into the node’s kernel policy, no CR, no reconciliation, no rollout to the rest of the pool. It’s the right tool while you’re iterating on rule syntax and want a tight feedback loop.

Once the profile behaves the way you want (denies the right things, doesn’t over-deny, logs what you need it to), the merge step is mechanical: copy the exact spec.content block you validated locally into the AppArmorProfile CR’s spec.content field, keep the profile name identical, and apply it. From that point the VKS controller owns distribution: new nodes joining the pool pick it up automatically, and a manually-loaded, node-local copy is no longer the only thing standing between your workload and an unenforced profile after a node rebuild.

A couple of things to check when doing this merge:

• apparmor_parser replace failures don’t always surface loudly. A sudo apparmor_parser -q run against a profile name that’s already loaded can fail with profile can not be replaced (error=-17) if something about the load conflicts, and -q suppresses the console output that would normally tell you. Check sudo aa-status | grep <profile-name> afterward to confirm the profile that’s actually resident in the kernel is the one you just tried to load, not a stale version.
• Confirm process-to-profile attachment, not just profile existence. aa-status lists both loaded profiles and which running processes are confined by each:

Seeing your target processes (e.g. bash, sleep, the container’s entrypoint) listed under the profile name is the real confirmation that the CR-delivered profile made it all the way from declaration to enforcement.

Step 4: Finding the audit log (and why it’s easy to look in the wrong place)

This is the step that trips people up. Enforcement and audit logging are two different code paths in AppArmor, and a deny rule without the audit keyword is silent by design: the action gets blocked, but no DENIED record gets generated for that explicit rule match. That’s true both for the containerd default template’s deny rules (Step 1) and for a hand-written profile unless you explicitly add audit:

Adding audit and reloading the profile is the first fix if you need denials to show up in a log at all. For a workload that needs something narrower than the default (for example, denying all writes rather than a specific path list), you can specify the AppArmor profile explicitly in the pod specification as follows:

Example pod.yaml:

The second, less obvious issue is where that log entry ends up, and it depends on whether auditd is running on the node:

• Before auditd starts (or if it’s never running), the kernel prints audit records (type=1400, apparmor="...") to the kernel ring buffer as a fallback, visible via dmesg or journalctl -k.
• Once auditd claims the audit netlink socket, the kernel stops mirroring those records to the ring buffer. From that point on, dmesg and journalctl -k will show nothing for new AppArmor events, even while auditd is actively logging every one of them to its own store.

This means a debugging session that starts with dmesg | grep apparmor and finds nothing that can look exactly like “audit logging isn’t working,” when the actual state is “audit logging is working correctly, you’re checking the wrong sink.” The fix is to query auditd directly:

A confirmed denial for touch /abcd looks like this:

type=AVC msg=audit(...): apparmor="DENIED" operation="mknod" profile="k8s-apparmor-example-audit-deny-write" name="/abcd" pid=... comm="touch" requested_mask="c" denied_mask="c" fsuid=0 ouid=0

Note the operation="mknod" and requested_mask="c": c is a kernel-internal permission bit used for file-creation operations, and unlike r, w, a, m, l, and k, it isn’t a letter you can write directly in profile rule syntax (there’s no audit deny /** c, you can author). That means this record can show up even though the profile only explicitly names w in its rule, the create check is evaluated separately from write internally. The precise mechanics can vary by kernel/AppArmor version, so treat the log record itself as the source of truth over any assumption about which rule “should” have fired.

Putting it Together: A Verification Checklist

For any AppArmor profile on a VKS node, before calling enforcement confirmed:

1. Confirm the namespace’s PSS enforce level. If it’s baseline or restricted, a “denied” result from your test could be DAC (non-root, dropped capabilities) rejecting the action before it ever reaches AppArmor. Test in a privileged-enforced namespace to make sure any denial you observe is actually attributable to the MAC layer.
2. kubectl exec ... -- cat /proc/1/attr/current: Profile name matches what you expect, and it says (enforce), not (complain).
3. sudo aa-status | grep -A2 <profile-name>: The container’s processes are actually listed under that profile on the node.
4. Attempt the specific denied operation from inside the container, and confirm the expected Permission denied (or equivalent) at the shell.
5. Attempt an operation that should still be allowed (a read, for a deny-write profile), to rule out an overbroad profile that fails closed on everything.
6. If you need an audit trail: confirm the relevant rule uses audit deny, then check ausearch -m apparmor -ts recent or /var/log/audit/audit.log directly, not dmesg/journalctl -k, since auditd will have claimed those records once it’s running.
7. If the profile is meant to be declaratively managed: confirm the AppArmorProfile CR name, its embedded profile name, the Cluster spec’s osConfiguration.securityContext.appArmor.profiles entry, and the pod’s localhostProfile all reference the exact same name. A mismatch here is invisible until you go looking for it specifically.

Why this matters for security-sensitive customers

For teams running regulated or hardened workloads, the payoff of the AppArmorProfile CR path is that MAC enforcement stops being a per-node artifact and becomes a versioned Kubernetes object: it can live in Git, goes through the same review and admission path as any other manifest, and is reconciled onto nodes rather than hand-loaded. That closes the exact gap this post opened with, a profile that only exists because someone ran apparmor_parser on one node can be invisible to audit, doesn’t survive that node being replaced, and drifts silently from the rest of the pool.

Declaring the profile once and letting node rollout carry it forward means new and replaced nodes join the cluster already enforcing the intended profile, with no manual re-load step and no window where a fresh node is running Unconfined by accident. Combined with the verification loop above: confirm attachment → confirm the denial → confirm the audit trail, teams get a MAC control they can prove is enforced fleet-wide, not just on the one node they happened to test.

Resources

• To learn more about VKS visit us at: vmware.com/vks

• Get firsthand experience with VKS using the Hands-On Lab: https://labs.hol.vmware.com/HOL/catalog/lab/27680