Swap Object Thin Provisioning in vSAN 6.7

As of vSAN 6.7 the default behavior of the vSAN swap object is set to thin provisioned and matches the storage policy attributes assigned to the VM. As you might imagine, this has the potential to significantly decrease the effective storage capacity used in a vSAN environment. However, just the mere mention of “Swap file” triggers feelings of performance degradation and sparks religious wars on whether or not to consider memory overcommitment. In this post, I will share some thoughts on VMware’s memory resource reclamation strategy, why the vSAN swap object is now thin provisioned by default, and how to change the default value.

Thick to Thin

You may recall in vSAN 6.2 the Sparse Swap Files feature was released providing considerable space-savings on capacity space consumed. Virtual Swap files (.vswp) are created when a virtual machine doesn’t have a memory reservation equal to the amount of memory the virtual machine is configured to use. In past versions of vSAN, while other objects were thin provisioned by default, the VM swap object has always been provisioned with 100% Object Space Reservation (thick provisioned). By enabling the advanced “SwapThickProvisionedDisabled” host setting, swap files would be provisioned thin and disk space only claimed when the swap file is consumed.

In order to demonstrate the capacity savings let’s take a look at a single VM with 8GB of memory and no memory reservation defined.  Prior to vSAN 6.7 a swap file would be created with 8GB of disk space reserved. All VM objects (VMDK, VM Home, swap etc) inherit the default storage policy which is Failures to Tolerate (FTT) = 1.  This means in order to tolerate one host failure a copy of all VM’s objects (including the swap file) need to be on two hosts.  As a result, 16GB (8GB + 8GB) of unused space is reserved just for the swap file. That may not seem like much if you only have a handful of VMs in your environment but imagine 1000 VMs all with 8GB of memory.

1000 VMs * 8GB * 2 (FTT overhead) = 15.6 TB capacity reservation

Imagine a company running heavyweight edge services with workloads that need to be physically or latency proximal to a site. Usually, these services have sufficient RAM for the application and won’t overcommit memory.  In this case thick swap files are reserved space that the VM will never use and also have to double the space to account for the FTT overhead. Below is a comparison of thick vs thin swap using a database server with 512GB of RAM:

Thick Swap File

• RAM: 512GB
• Swap Capacity: 1024GB (512GB swap file + 512GB FTT overhead)

Thin Swap File

• RAM: 512GB
• Swap Capacity: no capacity reserved for thin swap files

Disk space is only claimed when the thin swap file is consumed

When is it OK to overcommit Host Memory?

Host memory is overcommitted when the total amount of guest physical memory of the running VMs is larger than the amount of actual host memory. ESXi supports memory overcommitment for two key reasons; higher memory utilization and higher consolidation ratio. The VMkernel leverages several innovative techniques to support virtual machine memory reclamation: transparent page sharing (TPS), ballooning, memory compression, and hypervisor swapping. As this post only covers vSAN swap files I recommend reading the vSphere 6.0 Resource Management white paper for a detailed look into all of these reclamation techniques.

Swap files are temporary disk storage space used to take the place of solid-state memory. A memory page that is not actively used can be swapped to disk, freeing that memory page for use by more active content. If the swapped memory page is needed later, it can be swapped back into memory from the disk. Hypervisor swapping, in general, is helpful in the sense that it prevents the system from running out of memory and crashing unexpectedly but is used as a last resort to reclaim memory from the VM due to the associated performance penalties.

But what about Performance?

Before we discuss the performance difference between thick and thin swap files I want to share a brief analogy of a spare tire.  A spare tire has a very specific purpose. If you get a flat tire, you can use your spare tire to help you from being stranded. Since they are not designed for long-term use, they are typically a smaller physical size, with narrower tread and reduced tread depth compared to normal tires. This is to save weight and trunk space.  There are also full-size spare tires.  They take up more space and cost more but if you plan to use them it might make sense to have a full spare tire. Many high-performance vehicles don’t use spare tires at all. They value lightweight and speed.  Instead of slowing the vehicle down with the weight of a spare tire, alternative strategies are tire sealant and roadside assistance.

Spare tires are to be used only in emergencies.  When driving on a spare tire it is recommended not to go above 50 MPH and should not be used for longer than 50 miles. Have you ever wondered about the performance penalties of a spare tire? probably not.

I like to think of swap files as a spare tire. I think it’s fair to say that disk performance is significantly slower than solid-state memory.  Anytime a page swap takes place, thick or thin, the workload’s performance is going to be impacted. If you are not overcommitting host memory and won’t be swapping to disk, there is no reason to reserve 2X the disk space (swap file  * 2 for FTT overhead). If you are overcommitting host memory and already swapping to disk you may want to consider adding more memory to your servers to avoid the performance penalty.

If memory is overcommitted, and the guest operating system is configured thick with sufficient swap space, there will be degraded performance.  However, if memory is overcommitted, and the guest operating system is configured thin with insufficient swap space, the guest operating system in the VM can hang.

Changing Swap File Value

As of vSAN 6.7 the default value for the setting SwapThickProvisionDisabled is enabled by default. If you are running vSAN 6.2 or higher you can change the value of the swap file using one of the following ways:

CLI:

Make the swap file thin:

Once the SwapThickProvisionDisabled value is configured, you need to power off and power on all the VMs on the vSAN datastore to have the .vswp file space reservation policy change.

vSphere Client:

This setting can also be managed in the vSPhere client by going to Host>Configure>Advanced System Settings>Edit>VSAN.SwapThickProvisionDisabled. Remeber a value of 1 = “thin” and a value of 0 = “thick”

Power CLI:

If you want to set enable\disable this at the cluster level you can use the following PowerCLI script created by Jase McCarty:

https://github.com/jasemccarty/Vsan-Settings/blob/master/Vsan-SetSwapFiles.ps1

In summary

vSAN is optimized for modern all-flash storage with space saving features that lower TCO while delivering incredible performance. The swap object thin provisioning and policy inheritance improvements are examples of how vSAN can provide a considerable space-saving on capacity space consumed.  If you know you are not overcommitting on host memory and as a result, won’t be swapping to disk, it makes sense to take advantage of swap object thin provisioning.

For detailed information on vSAN design and sizing go to storagehub.vmware.com. For detailed information on memory overcommitment and the various reclamation techniques be sure to read the vSphere 6.5 Host Resources Deep Dive book.

@vPedroArrow