From the Trenches

How vStorage APIs for Array Integration change the way storage is handled

Today we have a post from Anand Prakash, a Tech Support Engineer in Bangalore, India. The vStorage APIs for hardware acceleration functionality enables your host to offload specific virtual machine and storage management operations to compliant storage hardware. With the storage hardware assistance, your host performs these operations faster and consumes less CPU, memory, and storage fabric bandwidth.

VMFS-3 is a clustered file system that accesses a LUN from multiple ESX hosts. VMFS implements a locking mechanism on VMFS metadata to prevent powering on contents of a virtual disk on more than one host at any given time. An ESX host making any modification to virtual machine of on LUN acquires a lock by writing its own ID as owner of the LUN using SCSI reservations. This locks the other host from doing any I/O, but the change is only made to a small area of the LUN. We have noticed that the number and frequency of locking operations has increased rapidly in enterprises, making it more critical to optimize core cluster locking algorithms for block-based devices. The use of SCSI reservations to lock a disk when doing metadata updates caused a number of major issues in the past.

This issue is addressed with vStorage APIs for Array Integration (VAAI) using Atomic Test & Set (ATS). VAAI calls for an ATS primitive to atomically modify a sector on disk without the use of SCSI reservations and the need to lock out other hosts from concurrent LUN access. The ATS operation atomically (non-interruptible) compares an on-disk sector to a given buffer, and, if the two are identical, writes new data into the on-disk sector. The ATS primitive reduces the number of commands required to successfully acquire an on-disk lock.

VMware introduced storage vMotion in Virtual Infrastructure 3.5. This release allowed IT administrators to migrate virtual machine disks from one datastore to another datastore with no disruption or downtime. As the size of disk to be migrated kept growing larger and larger the operation often lasted for many minutes to many hours. This method utilizes VMkernel software data mover driver and consumes host CPU, memory, and HBA queues. To improve this in vShpere 4.0, VMware uses CBT (Change Block Tracking) for Storage vMotion. Storage vMotion in vSphere uses the CBT method, which moves the virtual machine home over to the new datastore in a few iterations, instead of using the virtual disk snapshot method used in the 3.5 release. Still, a VMkernel core data mover performs this data movement, which issues I/Os to read and write blocks from and to the source and destination.

VAAI uses a modified version of the SCSI EXTENDED COPY command to initiate cloning of LUNs or sub-LUNs. This offloads the task for de-duplication and with snapshot-capable storage because the hardware can start by using proprietary mechanisms to mark cloned destination extents as duplicates of source extents.

The default VMDK disk created by ESX host is zerothick. In this type of VMDK disk, VMFS-3 reserves space for virtual disk files and disks blocks are zeroed only on demand as and when guest operating system writes to a previously unwritten region of the disk. MSCS in a virtualization environment requires a zeroed out virtual disk. This requires a write operation on all disk blocks.

Using VAAI, a disk array can also implement proprietary mechanisms to lazily zero out requested extents in the background and/or mark the extents as zeroes without actually writing zeroes.

For more information, be sure to read our FAQ: vStorage APIs for Array Integration FAQ.


0 comments have been added so far

Leave a Reply

Your email address will not be published. Required fields are marked *