For many enterprise platform teams, adoption isn’t blocked by Kubernetes features—it’s blocked by the operating system (OS). You might want the automated lifecycle management of vSphere Kubernetes Service (VKS), but you cannot abandon the Red Hat Enterprise Linux (RHEL) standards that underpin your security compliance, licensing models, and operational toolchains.
Historically, this forced a difficult choice: adopt a managed platform and rebuild your entire operational stack on a new OS (like Ubuntu or Photon), or stay on legacy infrastructure just to keep your OS.
With the release of VKS 3.6, we are removing that friction.
We are excited to announce Build Your Own Image (BYOI) support for RHEL 9. This isn’t just about adding another option to the menu; it’s about acknowledging that the best platform is one that fits seamlessly into your existing environment, without forcing you to rebuild your foundation.
The “UBI Tax” and Why the OS Matters
Why is the underlying node OS such a dealbreaker? For many, it comes down to support and licensing efficiency.
When organizations build their application stack on Red Hat Universal Base Images (UBI) and these UBI containers run on a RHEL host, they automatically inherit the host’s support status. This provides end-to-end support from Red Hat for the entire stack.
VKS 3.6 allows you to keep this end-to-end support from Red Hat by bringing RHEL 9 nodes directly into the VKS lifecycle management, preserving your support chain and compliance posture without sacrificing modern platform automation.
How It Works: The Image Baker Workflow
We didn’t just want to support RHEL; we wanted to give you control over how it’s built. VKS 3.6 introduces the Image Baker tool through the VCF CLI. Image Baker is a specialized tool designed to create customizable VM images through a declarative, systematic build process for VKS.
Unlike traditional approaches that snapshot live VMs, Image Baker leverages an OCI-based BuildKit backend to systematically curate the filesystem. This helps ensure your images are clean, compliant, and contain the exact Kubernetes components required by VKS.
Step 1: Define the Image Spec
First, make sure you follow the prerequisites here. Once that’s complete, we can define a configuration file (e.g., rhel-9.yaml) that specifies the OS version and the Kubernetes distribution. For RHEL, you explicitly point to the Red Hat Universal Base Image (UBI).
You can also deeply customize the node at this stage—increasing disk size, installing operational tools, or hardening the OS by disabling services. For this example we’ll build a simple image, but if you’re looking for more customization options, check out the Tech Docs.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 |
apiVersion: imageconfiguration.vmware.com/v1alpha1 kind: Image metadata: # name format: {os}-{os-version}-{arch}-{kubernetes-version}---{vmware-version} name: rhel-9-amd64-v1.35.0---vmware.2-vkr.4 spec: osSpec: name: rhel version: "9" # Reference the official Red Hat UBI image. image: registry.access.redhat.com/ubi9/ubi kubernetesSpec: # Reference the VKS Kubernetes distribution image: projects.packages.broadcom.com/vsphere/iaas/kubernetes-release/1.35.0/kubernetes-distribution-image:v1.35.0_vmware.2-vkr.4 # Optional: Customize Node Configuration diskSize: 20Gi |
Step 2: Set Credentials and Bake
Because this process pulls from Red Hat registries, you must export your subscription credentials as environment variables. Then, using the VCF CLI vcf kr bake command to generate the image.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 |
# Set your Red Hat credentials export RH_USERNAME="your-username" export RH_PASSWORD="your-password" # Ensure you have the VCF kubernetes-release plugin version 3.6 installed vcf plugin list # If "kubernetes-release" is missing install it sudo -E vcf plugin install kubernetes-release --version 3.6 # Bake the image using VCF CLI sudo -E vcf kr bake -f rhel-9.yaml # If any issues, add `--log-level DEBUG` at the end of the vcf kr bake command |
The tool handles the heavy lifting: it pulls the base UBI, installs the Kubernetes components (kubelet, kubeadm, CNI), applies security configurations, and packages everything into a standardized OVA file located in the artifacts/ directory.
Deploying RHEL Clusters: Zero Friction
Once the image is baked, using it is incredibly simple. The operational workflow remains identical to deploying standard VKS clusters, preserving the “single pane of glass” experience in VCF.
- Import: Upload your generated RHEL 9 OVA to the Content Library.
- Annotate: In your cluster YAML definition, you specify the OS resolution annotation. You can apply this in two places—the Control Plane and the Worker Node pools—to ensure the entire cluster runs on RHEL (or apply just to the worker nodes for a mixed OS cluster).
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 |
apiVersion: cluster.x-k8s.io/v1beta2 kind: Cluster metadata: name: rhel-cluster namespace: rhel-ns annotations: run.tanzu.vmware.com/resolve-os-image: os-name=rhel spec: clusterNetwork: pods: cidrBlocks: - 193.0.0.0/16 serviceDomain: managedcluster1.local services: cidrBlocks: - 198.201.0.0/16 topology: classRef: name: builtin-generic-v3.6.0 controlPlane: metadata: annotations: run.tanzu.vmware.com/resolve-os-image: os-name=rhel replicas: 1 version: v1.35.0---vmware.2-vkr.4 workers: machineDeployments: - class: node-pool metadata: annotations: run.tanzu.vmware.com/resolve-os-image: os-name=rhel name: node-pool-1 replicas: 2 variables: - name: vmClass value: best-effort-medium - name: storageClass value: wcpglobal-storage-profile |
- Deploy: Apply the configuration (
kubectl apply -f cluster.yaml), and VKS handles the rest—provisioning control planes, worker nodes, and wiring up the networking. You can check the status of the cluster usingkubectl get cluster rhel-cluster -n rhel-ns.
Once the cluster is ready, get the kubeconfig with the VCF CLI or kubectl, and then verify the nodes are successfully running RHEL 9 using standard kubectl commands:
|
1 2 3 4 5 |
$ kubectl get nodes -o wide NAME STATUS ROLES AGE VERSION OS-IMAGE rhel-control-plane-9x4q2 Ready control-plane 5m22s v1.35.0+vmware.2 Red Hat Enterprise Linux 9.7 rhel-worker-pool-1-5d697f8d-8k2j4 Ready none 4m10s v1.35.0+vmware.2 Red Hat Enterprise Linux 9.7 rhel-worker-pool-1-5d697f8d-m9n2p Ready none 4m05s v1.35.0+vmware.2 Red Hat Enterprise Linux 9.7 |
Advanced Flexibility: Mixed Clusters and Upgrades
Flexibility is the core theme of VKS 3.6. We know that migrations aren’t always “all or nothing.”
- Mixed Node Clusters: You can now run clusters with mixed operating systems. For example, you might keep your Control Plane nodes on Ubuntu or Photon OS, while running your Worker Node pools on RHEL 9 to support your specific application licensing needs.
- Seamless Upgrades: Even though you are building your own image, VKS still orchestrates the upgrade process. When upgrading an existing RHEL cluster, simply provide the newer RHEL OVA, and VKS handles the rolling upgrade of the nodes, helping ensure zero downtime.
Meeting You Where You Are
VKS 3.6 represents a shift in how we approach enterprise Kubernetes. We are moving from rigid defaults to safer, adaptable choices. Whether you are an enterprise optimizing Red Hat licensing or a customer looking for a smooth migration path, VKS 3.6 is ready for you.
Start planning your migration today—build your image, keep your standards, and let VKS handle the rest.
Discover more from VMware Cloud Foundation (VCF) Blog
Subscribe to get the latest posts sent to your email.