Running singularity as the run time for Kubernetes with VMware PKS

HPC’s shift to hybrid cloud and Kubernetes from Bare Metal:


While traditional computing has virtualized the majority of workloads over the past decade, HPC has mostly remained bare metal, with more than 80% of the workloads still remaining unvirtualized. With its eternal quest to extract maximum performance from the hardware, there has been a hesitation to move to any form of virtualization due to the perception of performance reduction due to overhead. Some of the major drawbacks with running everything bare metal still exist in HPC environments. It takes many months to get a bare metal HPC environment up and running and these environments are not fully utilized at all times leading to inefficient use of resources

The world of IT and other computing are now shifting to hybrid clouds and Kubernetes, which is almost 100% virtualized. The world of HPC has realized that there can be major benefits to leveraging these technologies and it’s on demand capabilities to innovate faster, while reducing cost at the same time.


Singularity the container runtime for HPC:


Container technologies has enabled the pre-packaging of applications concisely for easier deployment. Modern application ecosystems have leveraged containers to bring agility to deployment and lifecycle management to their environments. Docker has been the most popular container runtime and has helped cloud native applications and microservices rapidly evolve.

Even though Docker is the most prominent container registry and platform, it was primarily designed for Microservices and not for High Performance Computing (HPC). Singularity is the container solution designed from the ground up for HPC and scientific computing. Singularity container is encapsulated in a single file making it highly portable and secure. Singularity is an open source container engine that is preferred for HPC workloads and has more than a million containers runs per day with a large specialized user base.

Container based applications can bring a lot of challenges to orchestration, day 2 operations and  lifecycle management. With the advent of Kubernetes, container management and orchestration are being standardized. Initially Kubernetes supported only the use of docker as a container runtime in its initial versions, but it has evolved. The Container Runtime Interface (CRI) in Kubernetes available since version XXX  provides support for multiple container runtimes.

In an earlier blog article, we showed the use of singularity containers to run machine learning workloads. In this solution we will look at combining the capabilities of Kubernetes offered through VMware PKS to run HPC applications leveraging Singularity runtime. The Singularity CRI is currently is in beta testing and therefore might not be suitable for production environments. The Sylabs Documentation site provides details about the CRI.


HW & SW Components of the Solution:


The solution was developed in the VMware Solutions lab leveraging the following components.

Table 1: HW components of the solution


The VMware SDDC and other SW components used in the solution are shown below:

Table 2: SW components of the solution


Deploying the Solution:


CentOS 7.6 Linux was used as the base operating system for the Kubernetes master and worker nodes. VMware Essentials version v1.14.3+vmware.1 was used as the Kubernetes deployment. Below are some of the steps that are required to deploy the Singularity CRI. Many packages found in EPEL have to be installed.


Then you can join any number of worker nodes by running the following on each as root:


A calico based pod network is deployed on the cluster as shown below


The master node should be untainted to allow it to also act as a worker


Now that the simple one node K8 cluster is up, we will deploy a simple hello world pod to confirm that the cluster is functional and is leveraging the Singularity container runtime.



The network port for the deployed is exposed to outside users leveraging the NodePort as shown below.



The port that should be used to access the application externally can be displayed as shown below


The help world pod can now be accessed using the URL in a browser as shown below.

Figure 1: Demo application running on Singularity CRI

To verify that the container runtime is Singularity use.



Singularity is the most commonly used container runtime for High Performance computing environments Leveraging containers can simplify the application packaging and deployment of complex HPC Applications. In this solution we show cased a Essentials PKS deployment leveraging a singularity runtime. An example application was deployed and demonstrated. Now that we have shown a simple pod running on Kubernetes with a Singularity runtime, we can apply the same concept to all containerized HPC applications. In future work, we hope to show complex HPC applications running on Kubernetes. Through this solution we have shown that Singularity Containers in combination with VMware PKS provides a great platform to run modern HPC applications.



  1. https://slateci.io/blog/kubernetes-with-singularity.html