Deploy, Manage and Run MongoDB on VMware vSAN

VMware vSAN is the industry-leading software powering Hyper-Converged Infrastructure (HCI) solutions. HCI, or Hyper-Converged Infrastructure, converges traditional IT infrastructure silos onto industry-standard servers and virtualizes physical infrastructure to help customers easily evolve their infrastructure.

Ever since the initial version of vSAN, people are wondering whether some shared-nothing architecture applications could be deployed on vSAN or not. Such applications include NoSQL applications such as MongoDB, Cassandra and Big Data applications such as Apache Hadoop.

Today, we have developed a reference architecture and validated that running MongoDB on VMware vSAN is a valid solution and performance is guaranteed. Besides, we also tuned various parameters both on vSAN side and on MongoDB side to achieve better performance. Finally, we tested various failure scenarios and proved that running MongoDB on vSAN is robust and resilient. We gave the best practices in the white paper to show how to suitably run MongoDB on VMware vSAN.

This blog is a short briefing and introduction to the reference architecture. If you want to view the full paper, refer to https://storagehub.vmware.com/#!/vmware-vsan/mongodb-on-vmware-vsan-tm-6-6-all-flash.

Overview

This reference architecture is a showcase of using vSAN as the storage tier for deploying MongoDB in a vSphere environment:

• We demonstrated the architecture of the deployment.
• We validated the base performance level of running MongoDB servers on vSAN.
• We compared the impact of different parameter settings on the performance level.
• We validated the resiliency tests and concluded the recovery methods.
• We provided the best practices based on performance comparison.

Architecture

We created a vSphere and vSAN cluster and then deployed on the nodes MongoDB services with sharding enabled. The solution architecture is shown in the figure below.

We deployed four virtual machines as Mongos servers and four virtual machines as Mongod servers as the basic architecture and performance baseline. In a production system, users can scale out both Mongos servers and Mongod servers horizontally to meet specific demands.

We enabled Journal in the MongoDB configuration so that any database update can be acknowledged after the write is flushed to the disk.

In the architecture figure, ConfiDB stands for the configuration database for MongoDB cluster’s internal use. It is a mandatory component for a sharding MongoDB cluster. It stores the sharding information, Mongos states, and other information.

Hardware Components

Performance is very hardware dependent. We use 4 physical hosts and other hardware as shown in table below.

Software Components

Operating system and software are included in the below table.

MongoDB VMs

As shown in the architecture figure, all MongoDB services are run in virtual machines. Based on the different role in MongoDB, the virtual machines have the following configurations:

Performance Result Examples

We conducted various experiments by changing parameters to show their impacts on performance. Based on these performance results, we can tune MongoDB and vSAN and get the best practices.

For each test, we use Yahoo’s YCSB benchmark tool and run against in YCSB:

• Workload A (Update heavy workload): 50/50% mix of reads/writes
• Workload B (Read mostly workload): 95/5% mix of reads/writes

Here are two examples of how MongoDB’s durability options can affect the overall performance.

The first performance example is the impact of different MongoDB durability options. We always enabled MongoDB’s on disk journal across all our tests. However, an update operation doesn’t necessarily require an acknowledgement by the journal:

• If an update operation requires acknowledgement by journal, we call it ‘Durability Optimized’ and set MongoDB connection URL option ‘w=1’ and ‘j=1’.
• If an update operation requires acknowledgement by memory but not by journal, we call it ‘Balanced’ and set MongoDB connection URL option ‘w=1’ and ‘j=0’.
• If an update operation requires no acknowledgement at all, we call it ‘Throughput Optimized’ and set MongoDB connection URL option ‘w=0’ and ‘j=0’.

The results are shown in below figures.

Workload A

Workload B

We can see that for both workloads, the durability options have different impacts on performance. So users should consider the tradeoffs between performance and data durability and choose the proper ‘w’ and ‘j’ options on demand.

The second example describes the performance when MongoDB cluster scales out. We used 4 physical hosts but we can change the number of MongoDB virtual machines to scale MongoDB’s cluster out. The results are shown in below figures.

Workload A

Workload B

We can see that for both workloads, ops/sec increases as MongoDB nodes grows from 2 to 8. Both read latency and update latency increased while update latency increased more sharply.

Check out the reference architecture for various parameter settings and performance result charts.

Failure Test

From the perspective of MongoDB’s replica set setting, the test is divided into two parts:

• Enable MongoDB’s replica set which means there are three virtual machines in a MongoDB replica set, and thus we use ‘rs=3’ as the short term.
• Disable MongoDB’s replica set which means there is only one virtual machine in a MongoDB replica set, and thus we use ‘rs=1’ for the short term.

From the perspective of failure, we conducted two types of failure:

• A physical host failure which will power off all the running virtual machines residing on it. When a host fails, VMware vSphere High Availability will restart the impacted virtual machines on another host. This is the backend feasibility of setting ‘rs=1’ while keeping a low service downtime.
• A physical disk failure in a vSAN datastore which will cause a vSAN object to enter a degraded state. With the storage policy set with FTT=1, the object can still survive and serve I/O. Thus from the virtual machines’ perspective, there is no interruption of service.

The failure testing results were summarized and the recovery methods were discussed in below table.

Best Practices

Based on the performance tests and failure tests results, we have concluded the following best practices:

• Before the deployment, plan the database size and required performance level. This will ensure the hardware is properly deployed and software settings are best suited to serve the requests.
• Always use MongoDB’s sharding mechanism to distribute workloads across multiple virtual machines.
• We can turn off MongoDB’s replica set mechanism because vSphere HA and vSAN’s FTT will provide data protection.
• When configuring the virtual CPU and memory for the virtual machines used by MongoDB, choose an appropriate CPU and memory number to best suit the users’ requirements.
• Choose a proper data durability option to meet the data durability requirements. Set an appropriate write acknowledgment value taking into consideration the tradeoff between performance and potential data loss.
• For vSAN policy settings, use a larger stripe width to achieve better performance.
• For vSAN object checksum, users can tradeoff between performance and data integrity. Disabling checksum could yield better performance.
• Use a proper client number to connect to a MongoDB cluster because more client threads lead to higher latency.
• Follow the MongoDB Production Notes.

Summary

Running MongoDB on vSAN is a robust solution. We can not only get a good level of performance but also enjoy the benefits of Hyper-Converged Infrastructure (HCI). We have conducted various performance tests to show how to tune parameters when running MongoDB on vSAN. Besides, the failure and resiliency tests show that this solution is robust and can tolerance different type of failures.

Again, for the full reference architecture, refer to https://storagehub.vmware.com/#!/vmware-vsan/mongodb-on-vmware-vsan-tm-6-6-all-flash.