posted

1 Comment

VMware vSAN is the industry-leading software powering Hyper-Converged Infrastructure (HCI) solutions. HCI, or Hyper-Converged Infrastructure, converges traditional IT infrastructure silos onto industry-standard servers and virtualizes physical infrastructure to help customers easily evolve their infrastructure.

Ever since the initial version of vSAN, people are wondering whether some shared-nothing architecture applications could be deployed on vSAN or not. Such applications include NoSQL applications such as MongoDB, Cassandra and Big Data applications such as Apache Hadoop.

Today, we have developed a reference architecture and validated that running MongoDB on VMware vSAN is a valid solution and performance is guaranteed. Besides, we also tuned various parameters both on vSAN side and on MongoDB side to achieve better performance. Finally, we tested various failure scenarios and proved that running MongoDB on vSAN is robust and resilient. We gave the best practices in the white paper to show how to suitably run MongoDB on VMware vSAN.

This blog is a short briefing and introduction to the reference architecture. If you want to view the full paper, refer to https://storagehub.vmware.com/#!/vmware-vsan/mongodb-on-vmware-vsan-tm-6-6-all-flash.

 

Overview

This reference architecture is a showcase of using vSAN as the storage tier for deploying MongoDB in a vSphere environment:

  • We demonstrated the architecture of the deployment.
  • We validated the base performance level of running MongoDB servers on vSAN.
  • We compared the impact of different parameter settings on the performance level.
  • We validated the resiliency tests and concluded the recovery methods.
  • We provided the best practices based on performance comparison.

 

Architecture

We created a vSphere and vSAN cluster and then deployed on the nodes MongoDB services with sharding enabled. The solution architecture is shown in the figure below.

We deployed four virtual machines as Mongos servers and four virtual machines as Mongod servers as the basic architecture and performance baseline. In a production system, users can scale out both Mongos servers and Mongod servers horizontally to meet specific demands.

We enabled Journal in the MongoDB configuration so that any database update can be acknowledged after the write is flushed to the disk.

In the architecture figure, ConfiDB stands for the configuration database for MongoDB cluster’s internal use. It is a mandatory component for a sharding MongoDB cluster. It stores the sharding information, Mongos states, and other information.

 

Hardware Components

Performance is very hardware dependent. We use 4 physical hosts and other hardware as shown in table below.

Property SPECIFICATION
Server Supermicro SSG-2027R-AR24NV
CPU cores 2 sockets, 10 cores each of 3.0GHz with hyper-threading enabled
RAM 512GB DDR4 RDIMM
Network adapter 2 x Intel 10 Gigabit X540-AT2, + I350 1Gb Ethernet
Storage adapter 2 x 12Gbps SAS PCI-Express
Disks SSD: 2 x 3,000GB NVMe drive as cache SSD

SSD: 8 x 400GB SATA drive as capacity SSD

 

Software Components

Operating system and software are included in the below table.

Software Version Purpose
VMware vCenter  Server® and ESXi 6.5.0d

(vSAN 6.6 is included)

ESXi Cluster to host virtual machines and provide vSAN Cluster. VMware vCenter Server provides a centralized platform for managing VMware vSphere environments
VMware vSAN 6.6 Software-defined storage solution for hyper-converged infrastructure
CentOS 7.3 We use CentOS 7.3 as the guest operating system of all the virtual machines.
MongoDB 3.4 MongoDB is an open source  that uses a document-oriented data model.

MongoDB is built on an architecture of collections and documents. Documents comprise sets of key-value pairs and are the basic unit of data in MongoDB.

We use the community version of MongoDB 3.4, licensed under AGPL v3.0. Follow the installation instructions here.

Yahoo Cloud Serving Benchmark (YCSB) 0.12.0

 

YSCB is a framework and common set of workloads for evaluating the performance of different “key-value” and “cloud” serving stores.

 

MongoDB VMs

As shown in the architecture figure, all MongoDB services are run in virtual machines. Based on the different role in MongoDB, the virtual machines have the following configurations:

Role CPU cores Memory OS disk data disk
ConfigDB 8 32GB 32GB 200GB
Mongos 8 64GB 32GB None
Mongod 8 64GB 32GB 200GB

 

Performance Result Examples

We conducted various experiments by changing parameters to show their impacts on performance. Based on these performance results, we can tune MongoDB and vSAN and get the best practices.

For each test, we use Yahoo’s YCSB benchmark tool and run against in YCSB:

  • Workload A (Update heavy workload): 50/50% mix of reads/writes
  • Workload B (Read mostly workload): 95/5% mix of reads/writes

Here are two examples of how MongoDB’s durability options can affect the overall performance.

The first performance example is the impact of different MongoDB durability options. We always enabled MongoDB’s on disk journal across all our tests. However, an update operation doesn’t necessarily require an acknowledgement by the journal:

  • If an update operation requires acknowledgement by journal, we call it ‘Durability Optimized’ and set MongoDB connection URL option ‘w=1’ and ‘j=1’.
  • If an update operation requires acknowledgement by memory but not by journal, we call it ‘Balanced’ and set MongoDB connection URL option ‘w=1’ and ‘j=0’.
  • If an update operation requires no acknowledgement at all, we call it ‘Throughput Optimized’ and set MongoDB connection URL option ‘w=0’ and ‘j=0’.

The results are shown in below figures.

Workload A
Workload B

We can see that for both workloads, the durability options have different impacts on performance. So users should consider the tradeoffs between performance and data durability and choose the proper ‘w’ and ‘j’ options on demand.

 

The second example describes the performance when MongoDB cluster scales out. We used 4 physical hosts but we can change the number of MongoDB virtual machines to scale MongoDB’s cluster out. The results are shown in below figures.

Workload A
Workload B

We can see that for both workloads, ops/sec increases as MongoDB nodes grows from 2 to 8. Both read latency and update latency increased while update latency increased more sharply.

Check out the reference architecture for various parameter settings and performance result charts.

 

Failure Test

From the perspective of MongoDB’s replica set setting, the test is divided into two parts:

  • Enable MongoDB’s replica set which means there are three virtual machines in a MongoDB replica set, and thus we use ‘rs=3’ as the short term.
  • Disable MongoDB’s replica set which means there is only one virtual machine in a MongoDB replica set, and thus we use ‘rs=1’ for the short term.

From the perspective of failure, we conducted two types of failure:

  • A physical host failure which will power off all the running virtual machines residing on it. When a host fails, VMware vSphere High Availability will restart the impacted virtual machines on another host. This is the backend feasibility of setting ‘rs=1’ while keeping a low service downtime.
  • A physical disk failure in a vSAN datastore which will cause a vSAN object to enter a degraded state. With the storage policy set with FTT=1, the object can still survive and serve I/O. Thus from the virtual machines’ perspective, there is no interruption of service.

The failure testing results were summarized and the recovery methods were discussed in below table.

Replica set configuration Failure type Service interruption time Recovery method
rs=1 Host Failure Around 120 seconds vSphere HA restarted the failed virtual machines.
rs=1 Disk Failure No interruption vSAN rebuilt the failed components.
rs=3 Host Failure Around 10 seconds 1. MongoDB’s replica set failed over from the primary node to the secondary node.

2. vSphere HA restarted the failed virtual machines.

rs=3 Disk Failure No interruption vSAN rebuilt the failed components.

 

Best Practices

Based on the performance tests and failure tests results, we have concluded the following best practices:

  • Before the deployment, plan the database size and required performance level. This will ensure the hardware is properly deployed and software settings are best suited to serve the requests.
  • Always use MongoDB’s sharding mechanism to distribute workloads across multiple virtual machines.
  • We can turn off MongoDB’s replica set mechanism because vSphere HA and vSAN’s FTT will provide data protection.
  • When configuring the virtual CPU and memory for the virtual machines used by MongoDB, choose an appropriate CPU and memory number to best suit the users’ requirements.
  • Choose a proper data durability option to meet the data durability requirements. Set an appropriate write acknowledgment value taking into consideration the tradeoff between performance and potential data loss.
  • For vSAN policy settings, use a larger stripe width to achieve better performance.
  • For vSAN object checksum, users can tradeoff between performance and data integrity. Disabling checksum could yield better performance.
  • Use a proper client number to connect to a MongoDB cluster because more client threads lead to higher latency.
  • Follow the MongoDB Production Notes.

 

Summary

Running MongoDB on vSAN is a robust solution. We can not only get a good level of performance but also enjoy the benefits of Hyper-Converged Infrastructure (HCI). We have conducted various performance tests to show how to tune parameters when running MongoDB on vSAN. Besides, the failure and resiliency tests show that this solution is robust and can tolerance different type of failures.

Again, for the full reference architecture, refer to https://storagehub.vmware.com/#!/vmware-vsan/mongodb-on-vmware-vsan-tm-6-6-all-flash.