vSAN stripes (How, When and Why?)

Number of Disk Stripes Per Object/Stripe Width

Number Of Disk Stripes Per Object, commonly referred to as stripe width, is the setting that defines the minimum number of capacity devices across which each replica of a storage object is distributed. vSAN may actually create more stripes than the number specified in the policy for reasons will will discuss later. In the UI Striping will appear as “RAID 0” with the number of components being driven primarily by what the stripe width was set to.

Striping may help performance if certain virtual machines are I/O intensive and others are not. With striping, a virtual machines data is spread across more drives which all contribute to the overall storage performance experienced by that virtual machine. In the case of hybrid,this striping would be across magnetic disks. In the case of all-flash, the striping would be across whatever flash devices are making up the capacity layer.

VSAN Sripe

However, striping may not help performance if (a) an application is not especially I/O intensive, or (b) a virtual machine’s data is spread across devices that are already busy servicing other I/O intensive virtual machines.

Hybrid Reads   On a hybrid cluster when the read cache rate is low, and the back end capacity disks are reaching congestion, raising the stripe level can increase read cache miss performance by spreading out the workload across multiple capacity devices and cache devices.


Streaming Write performance

In Hybrid and All Flash use cases large streaming write workloads can be improved in some cases by increasing stripe width. By increasing the available write buffer (by striping across multiple disk group) to a given workload this can increase the size of a burst write workload before congestions and back pressure will be applied. On the back end, using more capacity devices can also increase the de-stage speed (assuming the cache tier is not the bottleneck).

However, for the most part, VMware recommends leaving striping at the default value of 1 unless performance issues that might be alleviated by striping are observed. The default value for the stripe width is 1 whereas the maximum value is 12. The most common case if it is being raised is in Hybrid clusters using slower or fewer capacity devices encountering a high cache miss rate.

Stripe Width – Sizing Consideration

There are two main sizing considerations when it comes to stripe width. The first of these considerations is if there are enough physical devices in the various hosts and across the cluster to accommodate the requested stripe width, especially when there is also a Number Of Failures To Tolerate value to accommodate.

The second consideration is whether the value chosen for stripe width is going to require a significant number of components and consume the host component count. Both of these should be considered as part of any vSAN design, although considering the increase in maximum component count in 6.0 with on-disk format v2, this realistically isn’t a major concern anymore for most deployments.

Automatic Striping

In cases where a disk group does not contain enough space to hold a VMDK, automatic striping of objects will happen. Also, in cases where an object is over 255GB, it will automatically seperated into a separate component. This component could end up on the same disk, or on a different disk group (passively giving striping) but there is no deterministic decision to do this as placement is driven by capacity and not performance reasons.

Deduplication and Compression Stripes

It should be noted that when Deduplication and Compression are enabled, data will be naturally stripped across a disk group, as block matches may happen on any disk within a disk group. The stripe policy will still allow you to stripe across different disk groups if you choose.

Wrapping Up

In most real world use cases, we do not see significant performance increases from changing the striping policy. It is available and you should weigh the added complexity against the need before changing it from the default.


11 comments have been added so far

  1. Hi,
    is the default stripe size/width really “1”? We are on vsna 6.2 and the default storage policy says “6”.

  2. If we have a large vmdk (let’s say 4TB), it is already broken into multiple objects which are then distributed according to vsan’s default behavior to multiple disks, does it make any sense to increase stripe width of that vmdk beyond the default in order to avoid write congestion during resync/high IO events? This is a five node all flash cluster (2 DGs per host, unforntunately all SATA SSDs at cache and cap tiers.

  3. Hi Michael,

    Please note that the number of stripes per object is a setting to ensure a *minimum* number of stripes are utilzed, and that these stripes must be on separate disks. Assuming a stripe width of 1, given that a 4 TB VMDK is broken into 255 GB chunks at time of provisioning, there would be 16 automatic stripes at time of creation. These stripes can co-exist on the same disks, so how spread out of a workload this represents depends on how vSAN decides to place your stripes, which is heavily influenced by how much data is stored in vSAN overall, and of course the size of each capacity disk.

    When you select a stripe width, you are now forcing a division of that object, then another division is automatically performed for each 255 GB chunk below that. For example, a 4 TB VMDK with a stripe width of 2 is now two 2 TB parts, which in turn are each 8 255 GB chunks. These chunks may share disks, but members of the two stripe sets (those two 2 TB parts) will not share disks. On the surface, this is tempting, as it seems to ensure more disks are definitely engaged. However, consider that I already likely have many disks engage in the solution as it stands. For example, with a 1.92 TB SATA SSD in capacity, I would need likely engage 6 of these for the VMDK, 3 of which must be on a separate host (assuming an FTT of 1). Thus, 2 disk groups are certainly availabile for read requests, but it’s more likely there are more utilized than that. From a write perspective, I only write to the stripe I engage, and it isn’t often that full-stripe writes are happening (except in special cases like 100% sequential write environments). Thus, I’ve probably achieved a reasonable distribution because of how vSAN has decided to spread the 255 GB chunks around. This may be a good case for leaving the stripe width at 1 and allowing hard data, such as careful monitoring of the vSAN performance tools, to govern decisions about increasing stripes.

  4. It’ not best practice for example use Number of Disk Stripes Per Object > 1 for file servers or sql servers? to improve the IOPs

    I have 4 nodes 16 disk

  5. Trying to test this.

    I have a 3 node cluster. Each node has a 250GB SSD and 3x 2TB SATA disk. I set the PFTT = 1 and the Stripes to 6.

    The policy failed due to needing 12 disks.

    I am not making the connection. Can someone tell me why the policy fails? I have 9 disks and if any disk fails I should have another copy somewhere across the 3 hosts.

Leave a Reply

Your email address will not be published. Required fields are marked *