It’s an exciting time to be a vSphere administrator! vSphere 7 is now available and there’s lots of really cool things incorporated into it. What I’m going to introduce to you in this blog is the vSphere Pod Service. If you have been following any of the Project Pacific announcements and content from VMworld 2019 some of this will be familiar to you. This blog is another in a series of blogs discussing vSphere with Kubernetes not from the developer side but from the vSphere administrator perspective.
The first thing we’re going to talk about is the vSphere Pod Service. Before we dive into that, a quick recap on what Kubernetes is and then how vSphere Pod Service fits in.
Kubernetes is a platform for managing containerized workloads and services. When you deploy Kubernetes, you get a cluster. Several components are instantiated as part of that process.
Nodes: Every cluster has at least one worker node.
Pods: These are the components of the application workload running on the worker nodes
Control Plane: This manages the worker nodes and the Pods in the cluster and runs across the cluster, providing fault tolerance and high availability.
What is the vSphere Pod Service?
vSphere Pod Service is a service that runs on a VMware managed Kubernetes control plane over your ESXi cluster. It allows you to run native Kubernetes workloads directly on ESXi. The ESXi hosts become the Nodes and vSphere Pods are what are running the components of the app workloads.
Now why would I want to do that? What’s the advantage of running it this way? We’ll get into that in just a moment.
vSphere Pod Service Background
vSphere Pod Service is a fundamental re-architecture of ESXi. There were many ways in which we could address the needs of Kubernetes and container workloads running on ESXi. For example, we had previously released vSphere Integrated Containers. It had containers running on VMs and managed by VMware Admiral, a container management platform. But it wasn’t Kubernetes and we thought we could do a better, more deep and native integration with vSphere.
Operationally, the way forward is declaring what you want and letting the infrastructure supply it. This is true not only from the developer standpoint but from the administrator standpoint as well. But in order to accomplish this we didn’t want (or need!) to re-invent the wheel. We have a great platform in vSphere. To toss all of that overboard and start over is not only very expensive and disruptive, it means those customers that depend on it would have to do the same thing and learn everything new from scratch. We have many customers who depend on vSphere. The have processes in place that are integral to their business. They also need to do things like pass security audits or accomplish backups!
Wouldn’t it make more sense to leverage what we have, provide equal or better performance and scalability AND bring our customers along on this journey? As someone who has been in and around IT for 35+ years in makes complete sense to me.
vSphere Pod Service Components
vSphere Pod Service Spherelet
The Spherelet is based on the Kubernetes “Kubelet” and enables an ESXi hypervisor to act as a Kubernetes worker node. The Spherelet is an ESXi UserWorld agent that acts as an extension to the Kubernetes Control Plane. Its primary function is to poll the Kubernetes Control Plane’s API server for changes made to pod configuration, volumes, services, configmaps and secrets and then carry out the necessary operations to ensure that these are instantiated.
Figure 1 – vSphere Pod Service – vSphere Admin connects to vCenter. DevOps connects to the K8S Control Plane
CRX (“Container Runtime for ESXi”) Agent
In my opinion, this part is one of the coolest parts of the underlying architecture. Remember, ESXi is not Linux. The vSphere Pod Service provides a purpose-built lightweight Linux kernel that is responsible for running containers inside the guest. It provides the Linux Application Binary Interface (ABI) necessary to run Linux applications. Since this Linux kernel is provided by the hypervisor, VMware has been able to make numerous optimizations to boost its performance and efficiency. Additionally, because the CRX kernel does not load a full Linux guest OS, the instantiation of new vSphere Pods is very fast.
That’s right. We are not loading a full Linux guest OS. Just a very optimized kernel and init process called CRX Init. The Linux kernel used gets updated when you update ESXi via normal lifecyle management.
As with everything on ESXi, this will be shipped in a VIB. And if you followed my previous blogs on Secure Boot for ESXi, that means this Linux kernel is contained in a digitally signed package so that when you turn on Secure Boot it is verified to have not been tampered with. So not only is CRX a great way to run containers it is inherently more secure as well!
The CRX Agent looks very similar to the VMX because that’s where it started out. The key difference is that anything to do with a guest OS was pulled out.
USB connectivity? Gone.
Video display hardware? Gone.
Anything else that’s not needed is removed.
All these changes make the CRX super lightweight. The over-used adage of “It has to be on bare metal to have performance!” or “Running on a VM introduces too much overhead!” is now ancient history. To vSphere a vSphere Pod looks essentially like a specialized Virtual Machine. That means from a vSphere Admin standpoint it looks and feels “familiar”. You’re not learning a whole new thing. We’ll dive more into the CRX in a future blog article.
hostd
“hostd” is the UserWorld agent on the ESX host that the Spherelet communicates with to query the hardware properties of the vSphere Pod and reconfigures it to attach container image VMDKs or NICs. The communication channel with hostd is over a localhost HTTPS connection that is authenticated using local tickets. The session is authenticated using the DCUI user which is guaranteed to be able to perform VM reconfigure operations even in lockdown mode.
Image Service
The Image Service is responsible for contacting the container image registry and staging the container image on a VMDK that will be attached to the vSphere Pod by the Spherelet. It leverages Harbor and each team gets their own project to share.
Kubernetes Control Plane API
The Kubernetes Control Plane API is the central endpoint that validates and configures all the API objects representing entities in the Kubernetes cluster including pods, services, volumes, controllers, etc. It exposes a REST endpoint that kubectl connects to, to make requests on the API objects. It also provides the frontend to the cluster’s shared state through which all other components in the cluster interact. The API is polled by other controllers in the cluster (such as the Kubelet) for changes to the API objects so that it can propagate changes to the cluster state.
Ephemeral VMDK
Each vSphere Pod is provisioned with a VMDK that is used to stage the container logs, the emptyDir volume and configmaps. The lifetime of this VMDK is tied with that of the vSphere Pod. Any data written to this VMDK will be lost when the pod has completed running. Ephemeral VMDKs are provisioned by the DRS scheduler extension but are formatted and staged for use by the Spherelet Agent when instructed to do so by the Spherelet. Because this is a VMDK it is storage agnostic. Any storage that is supported by vSphere will work.
Container Image
Containers that comprise the pod will have their container images mounted into the vSphere Pod as VMDK images. Container images are pre-populated into VMDKs by the Image Service and need to be provisioned prior to the pod startup sequence.
When do I use vSphere Pod Service?
While the vSphere Pod Service uses Kubernetes, it’s not a conformant Kubernetes cluster. This is by design, as it intends to use Kubernetes to improve vSphere, rather than trying to turn vSphere into a Kubernetes clone. As Joe Beda says, it’s a Platform Platform. If your developers require standards-based and fully conformant with upstream Kubernetes clusters then you can use Tanzu Kubernetes clusters, also referred to as “Guest” clusters.
Some example use cases for vSphere Pod Service are those that can be in lockstep with the ESXi lifecycle. vSphere Pods are not where you want to run the latest and greatest Kubernetes and all of the 3rd party tools used during development. vSphere Pods provide unique security and operational advantages such as leveraging the proven ESXi isolation techniques that may, at this time, be interesting to some unique/particular workloads. vSphere Pods don’t allow direct access to hardware, another security advantage.
Figure 2 – vSphere Pod Service – Advanced Security and Performance
For most developers and users, TKG clusters are the way to go because they fully conform to Kubernetes and they allow you to incorporate 3rd party extensions and tools that hook into Kubernetes.
Wrap Up
I hope that this has given you, the vSphere Administrator, a useful overview on what the vSphere Pod Service is all about. If you have questions or ideas on the content a vSphere Administrator would be interested in when it comes to vSphere with Kubernetes, then reach out to me on Twitter. I’m @mikefoley and my DM’s are open.
It’s an exciting time to be a vSphere administrator! vSphere 7 is now available and there’s lots of really cool things incorporated into it. What I’m going to introduce to you in this blog is the vSphere Pod Service. If you have been following any of the Project Pacific announcements and content from VMworld 2019 some of this will be familiar to you. This blog is another in a series of blogs discussing vSphere with Kubernetes not from the developer side but from the vSphere administrator perspective.
