Part 1: Storage Policy Based Management for Containers Orchestrated by Kubernetes

This blog is written by Balu Dontu and Tushar Thole from Cloud Native Applications Storage team

In a previous blog, we learnt how stateful containers orchestrated by Kubernetes can leverage storage exposed by vSphere (vSAN, VMFS, NFS) by using standard Kubernetes volume, persistent volume and dynamic provisioning primitives.

Once you start enjoying the convenience of consuming vSphere storage on Kubernetes, the next question you may ask is – how do I achieve different SLAs for different containers deployed using Kubernetes? Can I specify a condition or a policy while creating a volume so that the volume will be automatically provisioned on an appropriate datastore based on the policy? The answer is YES. For now, this feature is supported only on the vSAN platform.

This blog will provide a step-by-step guide on how administrators can specify vSAN storage capabilities for dynamic volume provisioning inside Kubernetes by using a sample WordPress-MySQL application.

Storage Policy based provisioning of Persistent Volumes

Storage Policies capture storage requirements, such as performance and availability, for persistent volumes. These policies determine how the container volume storage objects are provisioned and allocated within the datastore to guarantee the requested Quality of Service. Storage policies are composed of storage capabilities, typically represented by a key-value pair. The key is a specific property that the datastore can offer and the value is a metric, or a range, that the datastore guarantees for a provisioned object, such as a container volume backed by a virtual disk.

As you might be already aware, vSAN is a distributed layer of software that runs natively as a part of the ESXi hypervisor. vSAN aggregates local or direct-attached capacity devices of a host cluster and creates a single storage pool shared across all hosts in the vSAN cluster. While supporting VMware features that require shared storage, such as HA, vMotion, and DRS, vSAN eliminates the need for external shared storage and simplifies storage configuration and virtual machine provisioning activities. vSAN works with virtual machine storage policies to support a virtual machine-centric storage approach. When provisioning a virtual machine, if there is no explicit assignment of a storage policy to the virtual machine, a generic system defined storage policy, called the vSAN Default Storage Policy is automatically applied to the virtual machine.

We will now illustrate how this policy based management approach can be applied to container volumes as well.

As described in official documentation, vSAN exposes multiple storage capabilities. The table below lists vSAN storage capabilities that are currently supported by vSphere Cloud Provider.

vSphere Infrastructure(VI) administrator can specify storage requirements for applications in terms of storage capabilities while creating a storage class inside Kubernetes. Please note that while creating a StorageClass, administrators should specify storage capability names used in the table above as these names might differ from the ones used by vSAN. For example; Number of disk stripes per object is referred to as stripeWidth in vSAN documentation however vSphere Cloud Provider uses a friendly name diskStripes.

These storage capability requirements are then converted into a vSAN policy which is applied when the storage volume (virtual disk) is created.

Let’s illustrate this workflow further using a sample application.

A previous blog showcased how to create a persistent volume of a particular disk format type (thin, thick, eagerzeroedthick) on demand. Let’s consider the scenario where after your application is deployed, you observed a large increase in the number of incoming requests. To handle the additional load, your application needs to be highly performant to process the blog access requests quickly and it should be highly available even if there are any host failures.

Lets create a StorageClass to capture these application requirements and claim the volume.

Request storage

Like any other Kubernetes resources, a storage class is created by using a resource description file:

hostFailuresToTolerate (Defines the number of host and device failures which in this case is 3 that a persistent volume can tolerate)As you see, in addition to diskformat, the storageClass definition now specifies vSAN storage capabilities to address application storage requirements

• diskStripes (The minimum number of capacity devices across which each replica of a persistent volume is striped)

• cacheReservation (Flash capacity reserved as read cache for the persistent volume)

Since no datastore is specified, the virtual disk will be created on the datastore which is specified in the global vSphere configuration file – which can be found here /etc/kubernetes/vsphere.conf.

If you want to create a virtual disk on a specific vSAN datastore, you can specify it as part of the storage class definition which looks something like this:

The storage class itself can be created as:

Use claim

At this stage, only a storageclass definition is created. Let’s create a persistent volume next.

A PersistentVolumeClaim can be created using this resource file: 

The claim can be created as:

Let’s verify whether the persistent volume is bound.Now the storage volume is created on the datastore.

After the PersistentVolumeClaim is successfully bound, let’s login to the ESX and check if the persistent volume that’s created has the vSAN policy configured with it. You can check it by retrieving the disk backingObjectId and querying the policy of the disk using the cmmds-tool. Please see this blog entry to know more about cmmds-tool.

Similarly we can create a claim for WordPress application.

See Part 2 of the blog: Create Deployment with Persistent Volume Claim