A question was asked the other day about the performance behavior of a virtual machine after a vMotion on a 2 Node Virtual SAN configuration. The question basically was:
“Should there be a degradation in read performance after a virtual machine is vMotioned to the alternate host in a Hybrid 2 Node Virtual SAN configuration?”
During normal operations, the user was seeing a large number of IOPS on a read workload, but when moving the virtual machine to the alternate host, the read IOPS dropped by a large amount. After some time, the performance returned to normal. When they vMotioned the virtual machine back to the original host, they experienced the same degradation of read performance. Again, after some time, the performance returned to normal.
In fact, this is the correct out-of-the-box behavior for 2 Node Virtual SAN configurations. To understand why, it is important to know how write and read operations behave in 2 Node Virtual SAN configurations. This post covers Virtual SAN 6.1 and 6.2.
Writes Are Synchronous
In Virtual SAN write operations are always synchronous. The image of a Hybrid Virtual SAN cluster to the right, shows that writes are being written to Node 1 and Node 2, with Metadata updates being written to the Witness appliance. This is due to a Number of Failures to Tolerate policy of 1.
Notice the blue triangle in the cache devices? That’s 30% of the cache device being allocated as the write buffer. The other 70% of the cache device is green, demonstrating the read cache. It is important to note that All-Flash Virtual SAN clusters do not use the cache devices for read caching.
Writes go to the write buffer on both Nodes 1 and 2. This is always the case because writes occur on both nodes simultaneously.
The image to the right shows a typical read operation. The virtual machine is running on Node 1 and all reads are serviced by the cache device of Node 1’s disk group.
By default, reads do not traverse to the other node. This behavior is default in 2 Node configurations, as they are mechanically similar to Stretched Cluster configurations. This behavior is preferred when the latency between sites is at the upper end of the supported boundary of 5ms round-trip-time (RTT).
This is advantageous in situations where the two sides of a Stretched Cluster are connected by an inter-site link, because it removes additional overhead of reads traversing the inter-site link.
Because the cache has not been warmed on the host the virtual machine is now running on, reads will have to occur on the capacity drives as the cache is warmed.
The image to the right shows only part of the cache device as green, indicating that as reads occur, they are cached in the read cache of the disk group.
The process of invoking a vMotion could be from various DRS events, such as putting a host in maintenance mode or balancing workloads. The default Stretched Cluster recommendation, is to keep virtual machines on one site or the other, unless there is a failure event.
2 Node Reads after a Disk Failure
Read operations after a disk failure, are going to behave similarly to those of a vMotion. A single disk in the disk group has failed in the image on the right. Reads are going to come from Node 2, and the cache device on Node 2 is going to start caching content from the virtual machine’s disk.
Since only a capacity device failed, and there are others still contributing to the capacity, reads will also traverse the network, as data is rewritten to one of the surviving capacity devices on Node 1, if there is sufficient capacity.
Once data has been reprotected on Node 1, the cache will have to rewarm on Node 1 again.
Read operations after a disk group failure, are also going to behave like that of a disk failure.
In configurations where the host with a failed disk group has an additional disk group, rebuilds will occur on the surviving disk group provided there is capacity.
In hosts with a single disk group, rebuilds will not occur, as the disk group is unavailable.
The VSAN “DOMOwnerForceWarmCache” setting can be configured to force reads on both Node 1 and Node 2 in a 2 Node configuration.
Forcing the cache to be read across Stretched Cluster sites is bad, because additional read latency is introduced.
Virtual SAN 2 Node configurations are typically in a single location, connected to the same switch, just as a traditional Virtual SAN deployment.
When DOMOwnerForceWarmCache setting is True (1), it will force reads across all mirrors to most effectively use cache space. This means reads would occur across both sites in a stretched cluster config.
When it is False (0), site locality is in effect and reads are only occurring on the site the VM resides on.
In short, DOM Owner Force Warm Cache:
- Doesn’t apply to traditional VSAN clusters
- Stretched Cluster configs with acceptable latency & site locality enabled – Default 0 (False)
- 2 Node (typically low very low latency) – Modify 1 (True)
Not only does this help in the event of a virtual machine moving across hosts, which would require cache to be rewarmed, but it also allows reads to occur across both mirrors, distributing the load more evenly across both hosts.
To check the status, run the following command on each ESXi host:
esxcfg-advcfg -g /VSAN/DOMOwnerForceWarmCache
To set it for 2 Node clusters:
esxcfg-advcfg -s 1 /VSAN/DOMOwnerForceWarmCache
I also wrote a PowerCLI script to connect to each host in a vSphere cluster, and set the value to 1, which forces warm cache across both nodes.
That script can be found here: https://developercenter.vmware.com/samples?id=1662
Client Cache in 6.2
Virtual SAN 6.2 introduced the Client Cache feature for additional read caching at the host, regardless of where the virtual machine’s disks reside.
In the event a host has to traverse the inter-node link on a 2 Node configuration, the Client Cache will provide some read caching relief.
In the event of a vMotion of the virtual machine, regardless of the DOM Owner Force Warm Cache setting, the Client Cache would be rewarmed. This is the same behavior in a traditional Virtual SAN configuration.
Wrapping it up.
2 Node configurations, while mechanically similar to Stretched Clusters, often are deployed in the same physical location.
Using the VSAN advanced parameter of DOMOwnerForceWarmCache, 2 Node Virtual SAN configurations can be configured to have read operations behave similarly to a normal Virtual SAN configuration.
Forcing reads to be read across both nodes in cases where the Number of Failures to Tolerate policy is 1 can prevent having to rewarm the disk group cache in cases of vMotions, host maintenance, or device failures. When implemented, this feature changes the default recommendation for vSphere HA and DRS, in keeping virtual machines running on either site, but not across sites.