Home > Blogs > VMware vSphere Blog


VAAI Offloads and KAVG Latency

Now this is an interesting one, and something that I had not noticed before. Nor have many other people I suspect. One of our customers was observing high KAVG (kernel latency) during Storage vMotion operations (which were leveraging the VAAI XCOPY primitive). The KAVG latency was showing an expected value when VAAI was disabled. I’d previously seen high KAVG values when queue full conditions were also being observed. This was not the case here. Below are two esxtop outputs showing the symptoms. The first of these screenshots is with VAAI enabled. Notice the high KAVG/cmd value. Click on the image for a larger view.

esxtop counters with VAAI enabledNow we have the same esxtop counters with VAAI disabled. In this case, SCSI Commands, Reads and Writes are very much higher since the I/O is not being offloaded to the array, but we do see an expected small value for KAVG/cmd. Again, you can click on the image for a larger view.

esxtop counters with VAAI disabledSo what’s the cause? Well, eventually the explanation was found in the following KB article – Abnormal DAVG and KAVG values observed during VAAI operations. (Nice catch Henrik!) Reproducing verbatim the contents of the KB article, “when VAAI commands are issued via VAAI Filter, there are actually 2 commands sent. These are top-layer commands which are issued and are never sent to the actual device (they stay within the ESX kernel). These commands are intercepted by the VAAI filter and the VAAI plugin, and are replaced by the vendor-specific commands, which are issued to the device.

This is why esxtop shows device statistics for the top-level commands only, and as a result the values for DAVG and KAVG seem unusual when compared to results obtained when VAAI is not enabled .
 
In this instance (and only for this instance), the DAVG and KAVG observed in esxtop should not be interpreted as a performance issue, absent of any other symptoms.”

So there you go. There is always something to learn in this job.

Get notification of these blogs postings and more VMware Storage information by following me on Twitter: Twitter @VMwareStorage
This entry was posted in Storage, vSphere and tagged , , , on by .
Cormac Hogan

About Cormac Hogan

Cormac Hogan is a senior technical marketing architect within the Cloud Infrastructure Product Marketing group at VMware. He is responsible for storage in general, with a focus on core VMware vSphere storage technologies and virtual storage, including the VMware vSphere® Storage Appliance. He has been in VMware since 2005 and in technical marketing since 2011.

3 thoughts on “VAAI Offloads and KAVG Latency

  1. Pingback: Welcome to vSphere-land! » Storage Links

  2. Dustin

    Quick question: approximately what should the values be for KAVG/DAVG be when running a VAAI operation? I know there won’t be exact numbers, but is there a formula or estimate on where those values should be?

    For example, an average KAVG for a cloning operation shouldn’t exceed 200….

    Thanks,
    Dustin

    Reply

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>