Home > Blogs > VMware vSphere Blog


VAAI Thin Provisioning Block Reclaim/UNMAP In Action

Cormac_Hogan
Posted by Cormac Hogan
Technical Marketing Architect (Storage)

I have done a number of blog posts in the recent past related to our newest VAAI primitive UNMAP. For those who do not know, VAAI UNMAP was introduced in vSphere 5.0 to allow the ESXi host to inform the storage array that files or VMs had be moved or deleted from a Thin Provisioned VMFS datastore. This allowed the array to reclaim the freed blocks. We had no way of doing this previously, so many customers ended up with a considerable amount of stranded space on their Thin Provisioned VMFS datastores.

Now there were some issues with using this primitive which meant we had to disable it for a while. Fortunately, 5.0 U1 brought forward some enhancements which allows us to use this feature once again.

Over the past couple of days, my good friend Paudie O'Riordan from GSS has been doing some testing with the VAAI UNMAP primitive against our NetApp array. He kindly shared the results with me, so that I can share them with you. The posting is rather long, but the information contained will be quite useful if you are considering implementing dead space reclamation.

Some details about the environment which we used for this post:

  • NetApp FAS 3170A
  • ONTAP version 8.0.2 (I believe earlier versions do not support UNMAP)
  • ESXi version 5.0U1, build 623860,

 Step 1 - Verify that your storage array is capable of processing the SCSI UNMAP commands. The first place to look is on the vSphere Client UI. Select the datastore and examine the 'Hardware Acceleration' details (Hardware Acceleration is how we refer to VAAI in the vSphere UI):

VC-UI
Step 2 - The Hardware Acceleration status states Supported so it looks like this array is VAAI capable. The issue now is that we don't know exactly which primitives are supported so we need to run an esxcli command to determine this.  First, you need to get the NAA id of the device backing your datastore. One way of doing this is to use the CLI command 'esxcli  storage vmfs extent list' on the ESXi host. In our setup, this command returned the following NAA id for the LUN backing our VMFS-5 datastore:

naa.60a98000572d54724a346a6170627a52

Once the NAA id has been identified, we can now go ahead and display device specific details around Thin Provisioning and VAAI. To do that, we use another esxcli command 'esxcli storage core device list –d  <naa>'. This command can show us information such as firmware revision, thin provisioning status, the VAAI filter and the VAAI status:

# esxcli storage core device list –d  naa.60a98000572d54724a346a6170627a52
naa.60a98000572d54724a346a6170627a52
   Display Name: NETAPP Fibre Channel Disk (naa.60a98000572d54724a346a6170627a52)
   Has Settable Display Name: true
   Size: 51200
   Device Type: Direct-Access
   Multipath Plugin: NMP
   Devfs Path: /vmfs/devices/disks/naa.60a98000572d54724a346a6170627a52
   Vendor: NETAPP
   Model: LUN
   Revision: 8020
   SCSI Level: 4
   Is Pseudo: false
   Status: on
   Is RDM Capable: true
   Is Local: false
   Is Removable: false
   Is SSD: false
   Is Offline: false
   Is Perennially Reserved: false
   Thin Provisioning Status: yes
   Attached Filters: VAAI_FILTER
   VAAI Status: supported
   Other UIDs: vml.020033000060a98000572d54724a346a6170627a524c554e202020

Here we see that the device is indeed Thin Provisioned and supports VAAI. Now we can run a command to display the VAAI primitives supported by the array for that device. In particular we are interested in knowing whether the array supports the UNMAP primitive for dead space reclamation (what we refer to as the Delete Status). Another esxcli command is used for this step – 'esxcli storage core  device vaai status get -d  <naa>':

naa.60a98000572d54724a346a6170627a52
       VAAI Plugin Name: VMW_VAAIP_NETAPP
       ATS Status: supported
       Clone Status: supported
       Zero Status: supported
       Delete Status: supported

The device displays Delete Status as supported meaning that it is capable of sending SCSI UNMAP commands to the array when a space reclaim operation is requested.

Great – so we have now confirmed that we have a storage array that is capable of dead space reclamation.

Step 3 – Let's take a closer look at the datastore next. As can be seen from the screen-shot above, this is a 50GB LUN formatted with a VMFS-5. There is 49.5GB usable space remaining. Next, we deployed a Virtual Machine with a 15GB VMDK to this datastore. The Guest OS is using around 8.82GB of this space since that VMDK is thin provisioned. Here is a look at the provisioned and used space from a VMDK perspective:

VMFS-5
To look at more granular information about the amount of space consumed on the VMFS-5 volume, we can use some CLI commands. The recommendation would be to use vmkfstools -P to get the detailed volume information:

# vmkfstools -Ph -v 1 /vmfs/volumes/source-datastore/
File system label (if any): source-datastore
Mode: public ATS-only
Capacity 49.8 GB, 40.0 GB available, file block size 1 MB
Volume Creation Time: Tue Apr 24 14:20:51 2012
Files (max/free): 130000/129975
Ptr Blocks (max/free): 64512/64483
Sub Blocks (max/free): 32000/31998
Secondary Ptr Blocks (max/free): 256/256
File Blocks (overcommit/used/overcommit %): 0/10006/0
Ptr Blocks  (overcommit/used/overcommit %): 0/29/0
Sub Blocks  (overcommit/used/overcommit %): 0/2/0
UUID: 4f96b6c3-dcc7c210-a943-001b219b5078
Partitions spanned (on "lvm"):
        naa.60a98000572d54724a346a6170627a52:1
DISKLIB-LIB   : Getting VAAI support status for /vmfs/volumes/source-datastore/
Is Native Snapshot Capable: NO

We can clearly see that 10006 x 1MB File Blocks consumed on the VMFS-5 volume. This is approximately 9.77GB. The next thing we have to take into account is the amount of VMFS-5 volume that is consumed by VMFS metadata. The best way to get an approximation of this overhead is to use the du -h command on the datastore:

# du -h /vmfs/volumes/source-datastore/
8.8G    /vmfs/volumes/source-datastore/WindowsVM
9.6G    /vmfs/volumes/source-datastore

By taking away the amount of VMFS-5 volume consumed by Virtual Machines and related files (8.8GB) from the amount of space consumed on the complete volume (9.6GB), we can deduce that approximately 800MB is given over to VMFS-5 metadata. OK, now that we know what is consuming space on our volume, we are finally ready to start looking at the UNMAP primitive in action.

Step 4 - Let's do a Storage vMotion operation next and move this Virtual Machine from our source datastore to a different datastore. This is probably the best use-case for the UNMAP primitive. Once the Storage vMotion operation has completed, the vSphere client will report that the VMFS-5 volume now has a lot more free space:

VMFS-5-after-StoragevMotion

Step 5 - The issue however is that when we check the amount of free space on the Thin Provisioned LUN backing this VMFS-5 volume on the storage array, we see that we still have unused and stranded space. Using a 'lun show' CLI command on this NetApp array which is hosting the LUN for our VMFS-5 volume, we see that 8.8GB of space is still consumed:

lun show -v /vol/vol2/thin-lun
/vol/vol2/thin-lun            50g (53687091200)   (r/w, online, mapped)
        Serial#: W-TrJ4japbzR
        Share: none
        Space Reservation: disabled
        Multiprotocol Type: vmware
        Maps: unmap=51 issi=51
        Occupied Size:    8.8g (9473908736)  
        Creation Time: Tue Apr 24 15:16:52 BST 2012
        Cluster Shared Volume Information: 0×0

This is the crux of the issue that we are trying to solve with the VAAI UNMAP primitive.

Step 6 - We finally get to the point where we can now use the SCSI UNMAP primitive. If you've been following my blog posts, you'll know that we can now reclaim this stale and stranded space using the vmkfstools command.

Caution – We expect customers to use this primitive during their maintenance window, since running it on a datastore that is in-use by a VM can adversely affect I/O for the VM. I/O can take longer to complete, resulting in lower I/O throughput and higher I/O latency.

A point I would like to emphasize is that the whole UNMAP performance is totally driven by the storage array. Even the recommendation that vmkfstools -y be issued in a maintenance window is mostly based on the effect of UNMAP commands on the array's handling of other commands.

There is no way of knowing how long an UNMAP operation will take to complete. It can be anywhere from few minutes to couple of hours depending on the size of the datastore, the amount of content that needs to be reclaimed and how well the storage array can handle the UNMAP operation.

To run the command, you should change directory to the root of the VMFS volume that you wish reclaim space from. The command is run as:

vmkfstools –y <% of free space to unmap>

The % value provided is then used to calculate the amount of stranded space that should be reclaimed from the VMFS volume as follows:

<amount of space to be unmapped> = (parameter passed to vmkfstools –y * free space on vmfs volume) / 100

We will see an actual example of this command being run shortly.

Step 7 - You can verify if the UNMAP primitives are being issued by using esxtop. Press ‘u’ to get into the disk device view. then press ‘f’, ‘o’ & ’p’  to select display “VAAISTATS” and “VAAILATSTATS/cmd” fields. The values under “DELETE”, “DELETE_F” & “MBDEL/s” columns are the ones of interest during a space reclaim operation:

Esxtop
In this example, we attempted a reclaim of 60% of free space. The vmkfstools -y command displays the following:

Attempting to reclaim 60% of free capacity 48.8 GB (29.3 GB) on VMFS-5 file system 'source-datastore' with max file size 64 TB.

Create file .vmfsBalloontsWt8w of size 29.3 GB to reclaim free blocks.

Done.

vmkfstools -y created a balloon file of 29.3GB which is 60% of the free capacity (48.8GB). This temporary “balloon file” is equal to the size of the space to be unmapped/reclaimed.

There is a note of caution here – if you specify a % value in the high 90s or 100, the temporary "balloon" file which is created during the reclaim operation may fill up the VMFS volume. Any growth of current VMDK files or the creation of new files, such as snapshots, may fail due to unavailable space. Care should be taken when calculating the amount of free space to reclaim.

If we now look at esxtop while the reclaim is going on:

Esxtop2

From above output we see some UNMAP commands have been issued. By viewing the DELETE and the MBDEL/s columns, we can see the rate at which the commands are being processed. If you see values incrementing in the DELETE_F column, then that means some UNMAP commands may have failed.

Step 8 - Finally, if we return to our storage array and query the status of the Thin Provisioned LUN, we should now see a difference in the occupied space:

lun show -v /vol/vol2/thin-lun
/vol/vol2/thin-lun    50g (53687091200)   (r/w, online, mapped)
        Serial#: W-TrJ4japbzR
        Share: none
        Space Reservation: disabled
        Multiprotocol Type: vmware
        Maps: unmap=51 issi=51
        Occupied Size:   76.3m (79966208)    
        Creation Time: Tue Apr 24 15:16:52 BST 2012
        Cluster Shared Volume Information: 0×0

And there we have it. A real life example of the SCSI UNMAP primitive reclaiming dead space from a Thin Provisioned LUN backing a VMFS-5 datastore.

You should also note that in 5.0 U1, even if the advanced option to issue SCSI UNMAP when deleting a VMDK or doing a Storage vMotion is enabled (/VMFS3/EnableBlockDelete), it will no longer do so. The only way to reclaim stranded space in 5.0U1 is via vmkfstools.

Once again, thanks to Paudie for putting this together, and also to Luke Reed and our other friends at NetApp for both the equipment and assistance with getting it updated to a version of ONTAP which supports the UNMAP primitive.

Get notification of these blogs postings and more VMware Storage information by following me on Twitter: Twitter @VMwareStorage

55 thoughts on “VAAI Thin Provisioning Block Reclaim/UNMAP In Action

  1. Anon

    It would almost be easier to delete the Datastore?
    This really needs to be an automated process. How many people are really going to do this?

  2. Matt

    Why isn’t the UNMAP automatic? Is it performance related? Is there any harm in deleting an empty volume that shows currently provisioned space?

  3. Anon

    Why is this a synchronous process when it should be Asynchronous? Also, does it also delete the space on storage if you are just deleting a VM off of the Datastore? How does it handle that?

  4. Cormac Hogan

    Anon,
    Yes, it should be an automated process and it sdid start off like that. Unfortunately, as Matt alluded to, there were some performance issues experienced which meant that we had to make it manual.
    Matt,
    I’m guessing that if you had an empty volume, then you could delete it, but it is not always going to be possible to do this. This procedure will allow you to reclaim stranded space on those volumes.

  5. Cormac Hogan

    I’m guessing you mean VMFS-3 here Rob. We never had a VMFS-4 :-)
    Yes, my understanding is that it is ok to run the vmkfstools reclaim command against a VMFS-3 datastore. The VMFS drivers which ship with 5.0 and later have the necessary VAAI TP extension code.
    You can verify that the UNMAPS are occurring with esxtop by looking at the DELETE field in the VAAI stats view: esxtop -> u (device view) -> f (select fields) -> O (VAAI Stats)

  6. Rob

    Thanks,
    yes I meant VMFS 3 we are running 3.46. Tested today and it works.
    Thanks again will look at scripting this now
    Rob

  7. Rob

    Thanks,
    yes I meant VMFS 3 we are running 3.46. Tested today and it works.
    Thanks again will look at scripting this now
    Rob

  8. Ken Ngo

    Does anyone know if unmap will be supported without running the vmkfs-tool -y command? ie..when I delete a VM (or Storage vMotion), unmap will automatically be called?

    1. Cormac HoganCormac Hogan Post author

      Hi Ken,

      We are actively working on a solution, but I do not have any timeframes that I can share at this time. As soon as I know more, I will share with the storage community.

    1. Cormac

      Look at the free space on a volume and then estimate how much of that free space is unclaimed at the array side. If you have access to the array, then you should be able to tell how much is dead space. The % represents the dead space which vSphere is reporting as free space. In the above example, vSphere reported 48GB. We looked at the array and saw that 9GB was still consumed. This is about 20%. However, if you are unable to look on the array, you can put in a guestimate value – in this case we just used 60%.

      1. Matt

        Thanks for clarifying. This is a big help. Of course EMC is fixing this issue in the next enginuity update in Q4.

  9. MIchael HoSue

    We have an HP P4500 with a volume Thin provisioned, which has run out of space. The VMs say we are using only 40% of the volume, but the Management Console says we are out of space. HP support says we need to mode the data off, delete the volume, recreate and move the data back. We are doing that now using Storage vMotion.

    Am I to understand that this will not reclaim the blocks? ie using Storage vMotion to move off of the HP, delete the volume and recreate, and Storage vMotion back?

    Does anyone know if this will work on the HP P4500 Lefthand?

    1. Kyle

      We are having the exact same issue and HP is telling us the exact same thing. I would have to think that there’s some way (especially now, a year after you had the issue) to enable support for this on the SAN nodes.

      Did you ever have any luck? With our SAN utilization showing 99% right now, we don’t have the space to be moving and recreating volumes.

  10. Doug Peterson

    Thanks for the usefull information provided in this Blog,

    Would you be able to answer if Datacore fully supports the VAAI Block Zero function for thin provisioned volumes on SANsymphony-V whether manually or call through API.

    I see this supported as reported however does not seem to completely clean up (zero) and this as a result failing to perform the UNMAP operations.

    1. Cormac

      I’d suggest speaking to one of our support representatives to figure out why this is happening. Support have the ability to work with 3rd party vendors like Datacore.

  11. jimmy

    i run the command, but no .vmfsBallonxyz file is created – at least none is shown when immediately running an ls on the volume. The DELETE columns under esxtop show as 0 when I do this too. We’re running EqualLogic (5.2.6), which does support SCSI UNMAP. I’ve done this with another client worked a treat, however doing for this particular client its not working. Space remains the same as well. There is definitely space to recover. Any ideas???

    1. Cormac HoganCormac Hogan Post author

      Hi Jimmy,
      Are you using 5.0 patch 2 by any chance. The reason I ask is because UNMAP was disabled in that release. You’ll have to upgrade to 5.0U1 to get the functionality again. Also verify using the ESXCLI that the device supports the VAAI UNMAP (DELETE) primitive. If neither of these are the issue, I recommend speaking to our support specialists who can help troubleshoot the issue.
      HTH
      Cormac

  12. Onno

    Hi,

    I just tested the ‘automatic’ UNMAP on a (upgraded) vSphere 5.1.0 cluster. But when I storage migrate a VM from one DS to another, and while it’s migrating I’m running esxtop on the host which is running the VM, I don’t see the DELETE counter increasing while the migration is completing. So it seems vSphere is not signaling our storage box with UNMAP? Any ideas? I verified the advanced settings of the host (VMFS3.EnableBlockDelete = 1) and our storage backend is also supporting it.
    Note: When I manually run vmkfstools, it does issue UNMAP commands to the backend. (I see the DELETE counter increasing).

    Any help is appreciated.

    1. Cormac HoganCormac Hogan Post author

      There is still no automated UNMAP, not even in 5.1. The only way to reclaim dead space is via vmkfstools -y. We hope to bring the automated UNMAP back in a future release, but there are no timelines to share at this time.

  13. Thibaud LENIK

    Hi,

    Someone knows if VAAI Unmap command can be executed on ESXi 5 standard Edition (Not Enterprise / Enterprise+ Edition) ?

    Any help is appreciated.
    Regards,

    Thibaud

      1. felixfxlin

        Hi,
        vSphere enterprise edition has a feature named ‘[Storage APIs for Array Integration ]‘. I don’t understand if the feature is the VAAI here, or not. could please clarify this ? thank you.

          1. Chris Green

            Does this mean it is only available in Enterprise/Enterprise+? I’ve been trying for days to make this work on Essentials Plus without success.

  14. oczkov

    Hello Cormac,

    Your post is very informative! Thanks.

    I see you having lots of questions related to why the UNMAP command is not an automated process in vSphere 5.x (as VMware originally planned to) or why there is no checkbox available with VM delete/move operations for that (which might be a good idea anyway).

    I am dealing with storage arrays and SANs for more than a decade and the answer seems to be that it is not really something VMware itself can do a lot about it. Most of the reclamation process is done on the storage array and the storage array vendors implement this crucial part of the code. Then testing it with VMware.

    UNMAP can be done with full speed and interfere with all other workloads. This was already seen by most of VAAI users on ESXi 5.0. Possibly the UNMAP should be throttled on the array to limit the impact, but then it would take longer. Throttling is something most of the array vendors can do.

    Have you also considered vmkfstools -y command to warn users, who put the percantage values higher than lets say 90%, may cause troubles (out-of-space condition)? The warning and additional confirmation required by vmkfstools would be nice to have. When I saw vmkfstools -y option for the first time my thought was, why not to put 100% and make the whole free space reclaimed? The crux here is the baloon file created by vmkfstools, which is a tricky way to put zeros on the stranded space. Am I right?

    Regards to all storage and vSphere geeks ;-)

  15. Cormac Hogan

    Thanks for the post. You are right, and we recognize that using vmkfstools is rather cumbersome and relies on the customer having to make some calculations around how much space can be freed. And yes, if run incorrectly, the balloon file can cause some issues. All I can say at this time is that we have taken note of that and we are working towards making this operation more simplified. I’ll share more details with you as soon as I can.

  16. Makoto

    Hi Cormac,

    Thank you for your useful informations.
    I hv one question. Pls tell me.

    An several ESX hosts assigned one volume.(shared volume)
    Then, do I do the command against all ESX hosts? or only one ESX host?

    If this is OK for only one ESX host, how does the other hosts recognize about after reclaimed volume size? I just do the rescan action?

    Thanks in Advance
    makoto

    1. Cormac Hogan

      Yes – you only need to do it on one host. I believe a refresh operation should be sufficient to update the free space. Try that before a full rescan.

  17. DH

    Still waiting on an automated or command-based manner to shrink a thin VMDK for a guest who’s usage had bloated and was later shrunk. Only way to do that currently is to move the guest between two LUNs of different block size, which is no longer possible if you use VMFS 5 exclusively thanks to the removal of the ability to set the block size.

    I’d much rather use thin VMDK’s than thin LUNs and hope that different LUNs don’t balloon at the same time; that’s far more dangerous than a guest not being able to grow.

    1. MM

      We have had to keep a FS3 LUN around just for this purpose. On Workstation, you can basically right click on a VM and reclaim/clean the space – you should be able to do something similar on vsphere.

      On a side note, having to vmotion them around just adds to the UNMAP problem.

  18. gokhan

    great article Cormac. thanks for sharing. we are now able to reclaim free space from hypervisor. it will be very useful especially out of space issue costumers.

  19. MM

    I am also struggling with the UNMAP issue on our SAN. It does not take many moves/deletes for orphaned VM’s and their data start to accumulating. The only way to get it back is to manually run vmkfstools -y and of course you need to be careful with it and whatnot.

    Todays state-of-the-art SAN’s leverage thin LUNs, thin VMs and so forth – we really need for the UNMAP feature to work.

  20. Brad A.

    “By viewing the DELETE and the MBDEL/s columns, we can see the rate at which the commands are being processed.”

    Cormac, what is the ‘rate’ you refer to here? Is this value of 180 iOps, kbps, mbps?

    I have issued the command on our Dell Compellent SC8000/VMware ESX 5.1 environment and I see a value of 660 in the delete column for the target volume, but it has been showing exactly that number for over 24 hours now and there are no .vmksballoon files referenced by the vmkfstools -Y output showing on the volume using ‘ls -a’ command once ‘Done’ is showing after -Y completes.

    Thank you for helping us with this esoteric, but critical function.

    1. Cormac Hogan

      I believe DELL Compellent can do this in their metadata – mark dead space as reclaimed. This is why it is so quick. Have a quick chat with someone in Compellent support/ they should be able to confirm. The bottom line is can you see space freed up on the array. If so, it is working.

    2. Peter L

      DELETE is successful UNMAP operations – discrete UNMAP operations issued by ESXi to storage. It’s not ops per second (so not equiv to IOPS), but rather a total or cumulative count.
      DELETE_F is unsuccessful (failed) UNMAP operations. (Ditto details)
      MBDEL/s is MB deleted per second. It’s an instantaneous sample that returns to 0 when no UNMAPs are in progress.

  21. Sylvain

    Thanks for the post Cormac, and to all the previous comments.
    My question is : is this UNMAP feature only valid on THIN LUN ?
    What happens if I move my Thin LUN to THICK mode ? No more UNMAP needed even if my VMDK are THIN provisioned ?

    Did not have the chance to test it, so if anyone did, thanks for your reply…

  22. Graham Swift

    Hello Cormac,

    Do you know if the vmkfstools -y option would do anything at all on an array where the commands detailed above shows that the Delete Status is listed as unsupported but zero status is supported?

    The reason I am asking is that the tool when run seems to suggest that a balloon file of 0s is being written. If this is the case then the array must then perform the zero writes? (I am not seeing this happen so suspect that if it is unsupported it actually does nothing at all).

    Thanks

  23. Matthieu

    Hi Cormac,

    Thank you for this great and useful blog. Is there any way to throttle this feature (and VAAI operations) at the VMware side, like storage IO control ? Performance impact may be huge and the array can’t throttle just the unmap feature.

    Thanks.

    Matt.

  24. Diego

    So Cormac, after 18 months of writing this brilliant article, is VMware still working on automating space reclamation? Is that the case for 5.5 as well? vmkfstools -y is the way to go today?

  25. Jeff Walker

    We are running HDS VSPs, although VAAI is supported and enabled I see that from esxcli that the LUNs are reporting that Zero Status and Delete Status are both unsupported. Does this mean that enabling this on the hosts are unsupported or that our HDS VSPs do not support sending reclamation commands to the arrays/frames period? Thanks!

  26. Berend Schotanus

    Cormac,

    Good article. But why does the VMware Compatability Guide state:

    VAAI primitives “Full Copy”, “Block Zeroing” and “Hardware Assisted Locking” are supported with VMW_VAAIP_NETAPP plug-in.
    VAAI Thin Provisioning Space Reclamation is not supported.
    Reporting of soft free space threshold warnings is not supported either as part of VAAI or through VASA.

    This is for the FAS3220.

    Kind regards,
    Berend

  27. veeranagowda

    Thanks Gabrie.

    I have one more issue with VM,s.

    1) My datastore space was 101gb
    2)i created a VM and assigned 60gb of space from datastore.
    3)After deleting the same VM am not getting back datastore space.
    FYI.
    am not seeing any VM related file even browsing the datastore.

    and i did refresh lot of times but no progress.

    Thanks.
    veera.

  28. veeranagowda

    Hi Friends.

    I have an issue with VM,s.

    1) My datastore space was 101Gb
    2)i created a VM and assigned 60gb of space from datastore.
    3)After deleting the same VM am not getting back datastore space.
    FYI.
    am not seeing any VM related file even browsing the datastore.

    and i did refresh lot of times but no progress.

    Thanks.
    veera.

Comments are closed.