Home > Blogs > VMware vSphere Blog

VAAI Offloads and KAVG Latency

Now this is an interesting one, and something that I had not noticed before. Nor have many other people I suspect. One of our customers was observing high KAVG (kernel latency) during Storage vMotion operations (which were leveraging the VAAI XCOPY primitive). The KAVG latency was showing an expected value when VAAI was disabled. I’d previously seen high KAVG values when queue full conditions were also being observed. This was not the case here. Below are two esxtop outputs showing the symptoms. The first of these screenshots is with VAAI enabled. Notice the high KAVG/cmd value. Click on the image for a larger view.

esxtop counters with VAAI enabledNow we have the same esxtop counters with VAAI disabled. In this case, SCSI Commands, Reads and Writes are very much higher since the I/O is not being offloaded to the array, but we do see an expected small value for KAVG/cmd. Again, you can click on the image for a larger view.

esxtop counters with VAAI disabledSo what’s the cause? Well, eventually the explanation was found in the following KB article – Abnormal DAVG and KAVG values observed during VAAI operations. (Nice catch Henrik!) Reproducing verbatim the contents of the KB article, “when VAAI commands are issued via VAAI Filter, there are actually 2 commands sent. These are top-layer commands which are issued and are never sent to the actual device (they stay within the ESX kernel). These commands are intercepted by the VAAI filter and the VAAI plugin, and are replaced by the vendor-specific commands, which are issued to the device.

This is why esxtop shows device statistics for the top-level commands only, and as a result the values for DAVG and KAVG seem unusual when compared to results obtained when VAAI is not enabled .
In this instance (and only for this instance), the DAVG and KAVG observed in esxtop should not be interpreted as a performance issue, absent of any other symptoms.”

So there you go. There is always something to learn in this job.

Get notification of these blogs postings and more VMware Storage information by following me on Twitter: Twitter @VMwareStorage
This entry was posted in vSphere and tagged , , , on by .
Cormac Hogan

About Cormac Hogan

Cormac Hogan is a Senior Staff Engineer in the Office of the CTO in the Storage and Availability Business Unit (SABU) at VMware. He has been with VMware since April 2005 and has previously held roles in VMware’s Technical Marketing and Technical Support organizations. He has written a number of storage related white papers and have given numerous presentations on storage best practices and vSphere storage features. He is also the co-author of the “Essential Virtual SAN” book published by VMware Press.

3 thoughts on “VAAI Offloads and KAVG Latency

  1. Pingback: Welcome to vSphere-land! » Storage Links

  2. Dustin

    Quick question: approximately what should the values be for KAVG/DAVG be when running a VAAI operation? I know there won’t be exact numbers, but is there a formula or estimate on where those values should be?

    For example, an average KAVG for a cloning operation shouldn’t exceed 200….



Leave a Reply

Your email address will not be published. Required fields are marked *