Written by Alka Gupta and Aman Bastora
For vSphere environments, Project Hatchway offers storage infrastructure choices for container platforms such as Pivotal Container Service so customers can take advantage of container technology for running stateful, data-intensive applications such as databases and modern cloud native applications. The solution enables any storage which can be presented as a datastore to vSphere environment (vSAN, NFS, ISCSI, SAN, etc.), to be provisioned as persistent volumes in Kubernetes or docker swarm. Supported storage offerings can be found on VMware HCL.
Pivotal Container Service (PKS), is a container service offering from VMware and Pivotal to deploy and consume container services with Kubernetes, built with high availability, security and operational efficiency targeting enterprises and service providers.
vSphere persistent storage solution for Kubernetes
As containers are stateless and ephemeral, they don’t retain any data when they are restarted or rescheduled. However, businesses want to leverage container technology for running stateful and data-intensive applications. Kubernetes addresses this gap by offering a persistent storage API. This API abstracts the details of how storage is provided and enables a simple developer-friendly way of consuming storage. More details on persistent storage in Kubernetes can be found here.
While Kubernetes storage API addresses the storage consumption for containers, storage provider details for Kubernetes, running in vSphere Cloud environment are handled by Project hatchway, consumed as vSphere Cloud Provider interface as shown in Figure 1. Support for vSphere volumes is part of upstream open source Kubernetes distribution through project hatchway. This integration enables the pods in Kubernetes to use enterprise-grade persistent storage, similar to how traditional virtual machines leverage enterprise storage such as vSAN.
vSphere Cloud Provider supports every storage primitive exposed by Kubernetes: Volumes, Persistent volumes, Persistent volume claims, Storage class, Stateful sets.
Persistent volumes requested by containerized applications can be provisioned on vSphere datastores. Kubernetes volumes are defined in Pod specifications. They reference VMDK files and these VMDK files are mounted as volumes when the container is running. When the Pod is deleted, the Kubernetes volume is unmounted and the data in VMDK files persists, as shown in Figure 2.
Prerequisites and Steps to Configure vSphere Cloud Provider with Kubernetes
In opensource Kubernetes, to configure vSphere cloud provider, certain configuration steps need to be followed as listed below:
Prerequisite: https://vmware.github.io/vsphere-storage-for-kubernetes/documentation/prerequisites.html
Steps for existing Kubernetes cluster: https://vmware.github.io/vsphere-storage-for-kubernetes/documentation/existing.html
Pivotal Container Service automates and abstracts the configuration in the above steps as part of Pivotal Container Service tile installation via Ops Manager.
The only configuration required is on Pivotal Container Service Kubernetes master node Virtual Machine where storage permissions need to be set as mentioned in the below link:
Configuration of Persistent Storage Volumes
Scenario 1: Creation of Persistent storage on datastore specified in Pivotal Container Service
Once Pivotal Container Service tile is configured, persistent volumes can be created directly on persistent datastore specified in Pivotal Container Service tile as shown below in Figure 3.
NOTE: Always use shared datastore such as vSAN across vSphere cluster as Persistent Datastore
The example below demonstrates the creation of dynamic persistent volumes on PHD7 datastore as configured in PKS tile shown in Figure 3.
Create storage class using standard kubernetes CLI
kubectl create -f <storage-class-name>.yml
NOTE: kubectl is a command line interface for running commands against Kubernetes clusters.
Example of yaml file for storage class
1. Once Storage class is created, verify the creation of the storage class using standard Kubernetes CLI:
kubectl get storageclass <storage class name>
kubectl describe sc <storage class name>
2. After creation of Storage class, create and verify persistent volume claims and persistent volume using standard kubernetes CLI:
kubectl create -f <persistent-volume-claimname>.yml
kubectl get pvc <persistent-volume-claimname>
kubectl describe pvc <persistent-volume-claimname>
kubectl describe pv <volume name>
Example of yaml file for persistent volume claims:
NOTE: Multiple persistent volume claims can be created per storage class
3. After Persistent volumes are created, they can be found in vSphere Cloud environment
Log into vSphere Cloud with appropriate privileges for vSphere datastores access, navigate to the persistent volume datastore as described in Figure 3, kubevols folder is located here, under which persistent volumes as VMDK files can be found. The VMDK files are named after the persistent volumes created as above. More details in the Figure 4 below.
Scenario 2: Persistent volumes can be created on customized datastore of vSphere cloud by introducing datastore parameter in storage class yaml file as shown below:
PHD4 is an example name of customized datastore in the above yaml file.
Once Storage class is created, the rest of the steps is same as mentioned in the Scenario 1.
Summary: Project Hatchway + PKS Persistent Storage Technical Overview
This blog provides Persistent storage technical overview for Pivotal Container Service platform and gives a few examples of dynamic persistent volume creation. Pivotal Container Service can be downloaded from here. Install and configure PKS as per the documentation here. As per this blog, Project hatchway comes integrated into Kubernetes upstream distribution and no additional bits need to be installed. Future blog posts in the series will showcase use cases for the consumption of persistent volumes in PKS environment.
