If you have landed on this blog, you most likely do not need to be convinced that VMware vSAN as persistent storage for MondgoDB in containers is a good idea. If you landed here accidentally, let us give you a little background information.
MongoDB, a popular NoSQL document store database used by thousands of enterprises has rapidly become mainstream. Docker, a method to package and deploy self-sufficient applications, provides efficiency. Using Docker Containers for deploying MongoDB instances provides several benefits, such as easy to maintain, and scaling up and down on-demand.
However, when we want to run databases in a container environment, we have to face several questions. For instance, how do we compare database performance in containers with the one on virtual machines or bare-metal servers? How do we manage data persistently and disk efficiently? And how can we make sure multiple containers are created and made highly available?
We have put together a reference architecture (RA) to showcase of using VMware vSAN™ as a Hyper-converged infrastructure for MongoDB in containers on a Kubernetes cluster.
- We use vSAN as persistent storage for MongoDB in containers.
- We demonstrate the capability of Kubernetes vSphere Cloud Provider on vSAN to provide seamless consolidation to handle Kubernetes node failures without any data loss.
- We show the resiliency of vSAN against host failures.
- We measure the performance of multiple concurrent MongoDB instances.
- We evaluate the performance of a MongoDB with replica set enabled and demonstrate the application-level high availability.
Overall, the solution, designing and deploying MongoDB in containers backed by vSAN and VMware vSphere®, provides the following benefits for you. See our published RA for the detailed explanation for you and your customers.
- Simple deployment
- Minimal maintenance effort
- High availability capability
- Flexible configurations
The solution architecture shown in the figure below. The Kubernetes cluster is comprised of a 4-node vSphere and vSAN cluster upon which MongoDB services are deployed with sharding enabled on replicas set on the nodes.
The Ingredients
If you are wondering what is needed to build this RA:
MongoDB:
Total database size: ~217GB
Configuration
Four shards: each shard has around 54GB data
Four replica sets: each replica set has one Primary, one Secondary, and one Arbiter
Data is durable by using Journaling
Persistent Storage by vSphere Cloud Provider:
Use StorageClass for dynamic provisioning
Kubernetes nodes:
Four working nodes
Four standby workers
One master node
vSAN Cluster:
Four physical hosts with identical configuration
VMware VSAN Disk Group Specification (per Host)
SSD: 2 x 400GB Solid State Drive (Intel SSD SC2BX40) as Cache SSD
SSD: 4 x 400GB Solid State Drive (Intel SSD SC2BX40) as Capacity SSD
VM configuration:
OS: Photon OS 1.0
VM RAM: 32GB
VM vCPU: 8
Disk Configuration (every node VM)
Replica set (Primary or Secondary): 2 X 250GB
Replica set (Arbiter): 1 X 20GB
Configuration Server: 1 x 20GB
Instructions
To support the persistent storage requirement for running Kubernetes applications inside the vSphere environment, VMware developed vSphere Cloud Provider and its corresponding volume plugin. The workflow below will help you to understand how we claim persistent storage from vSAN to a container.
The Menu Options
We proposed three configurations. Each configuration has different replication settings and vSAN storage policies.
Configuration 1 – high performance oriented, and the database is protected by vSAN by setting Failures to Tolerate ( FTT) to 1.
Configuration 2 – high protection oriented, and the database is protected by both Replica Set and vSAN.
Configuration 3 – application-level protection only without tradeoff of the mirroring write of vSAN and the double sized space consumption. Think of it as a lite version of Configuration 1.
Taste Test
For each of the different configurations, we did a taste test, by measuring performance using Yahoo’s YCSB.
A sampling of the results is discussed here. For complete results, see our published RA.
The results shown here are for “Read-Mostly” workload (B), which is 95%/5% mix of reads/writes, run on Configuration 1 and Configuration 2.
In Configuration 1, with a configuration balanced for throughput and durability, the read heavy workload (95% reads) shows MongoDB providing a near-linear increment of the average ops/sec and cumulative ops/sec until 64 threads. The 99th latency read and update with 64 threads had the maximum latency of 5.6 ms for an update and 7.7 ms for a read. We monitored that the average vSAN backend latency for both read and write and it was less than 1ms.
In Configuration 2, the read heavy workload (95% reads) shows MongoDB providing a near-linear increment of the average ops/sec and cumulative ops/sec till 64 threads. The 99th latency read and update with 64 threads had the maximum latency of 6.1 ms for an update and 8.1 ms for a read.
We monitored that the average vSAN backend latency for both read and write was less than 1.7ms.
No Fail Recipe
We felt that it is important to test the failovers of the primary in a MongoDB replica set, node shutdown in the Kubernetes cluster, and one ESXi host powering off.
The results are shown below, by categorizing failure components.
| Failure components | Recovery time | Result and behavior |
| Primary fails in a replica set | <15 seconds | MongoDB role fails over. The secondary is unavailable until the previous primary available. |
| Kubernetes node shutdown or corruption | < 1 minute | The pod is recreated, and Kubernetes vSphere Cloud Provider automatically unmounts/mounts the disk to the standby node. The standby node changes to the working node. |
| ESXi host failure or powered off | <2 minutes | vSphere Availability restarted the Kubernetes node on another ESXi host; no node recreation required. |
Best Practices
When configuring MongoDB in containers in a Kubernetes cluster, we suggest the following best practices:
- vSphere: Follow Getting Started on vSphere to deploy Kubernetes cluster on vSphere 6.5.
- Kubernetes cluster scale: Create 2 x node number so that every working node has a standby node in case the working node fails, allowing the backup node to host the pods as soon as possible.
- vSAN Storage Policy-Based Management (SPBM): Disable checksum of vSAN if 10% latency increase cannot be tolerated. The reason is when comparing performance with checksum enabled/disabled, the performance difference (ops/sec or latency) is around ten percent for an intensive update workload like YCSB workload A.
Summary
The published RA could be a crowd-pleasing recipe. vSphere Cloud Provider enables access to vSphere managed storage (vSAN, VMFS, NFS) for applications deployed in Kubernetes. It manages persistent volumes and supports dynamic volume provisioning. vSphere Cloud Provider interacts with vCenter to support various operations such as creation and deletion of volumes, attaching and detaching volumes to application pods and nodes.
VMware vSAN combined with Kubernetes and MongoDB replication provides the best-fit protection from an application, node and physical host levels.
VMware vSAN provides a better choice for container based MongoDB services compared to application level replication. For best performance design purposes, we recommend using sharding without secondary replica set and set FTT=1. For best protection design purposes, we recommend using sharding with secondary and arbiter while setting FTT=1.
