Now this is an interesting one, and something that I had not noticed before. Nor have many other people I suspect. One of our customers was observing high KAVG (kernel latency) during Storage vMotion operations (which were leveraging the VAAI XCOPY primitive). The KAVG latency was showing an expected value when VAAI was disabled. I’d previously seen high KAVG values when queue full conditions were also being observed. This was not the case here. Below are two esxtop outputs showing the symptoms. The first of these screenshots is with VAAI enabled. Notice the high KAVG/cmd value. Click on the image for a larger view.
Now we have the same esxtop counters with VAAI disabled. In this case, SCSI Commands, Reads and Writes are very much higher since the I/O is not being offloaded to the array, but we do see an expected small value for KAVG/cmd. Again, you can click on the image for a larger view.
So what’s the cause? Well, eventually the explanation was found in the following KB article – Abnormal DAVG and KAVG values observed during VAAI operations. (Nice catch Henrik!) Reproducing verbatim the contents of the KB article, “when VAAI commands are issued via VAAI Filter, there are actually 2 commands sent. These are top-layer commands which are issued and are never sent to the actual device (they stay within the ESX kernel). These commands are intercepted by the VAAI filter and the VAAI plugin, and are replaced by the vendor-specific commands, which are issued to the device.
This is why esxtop shows device statistics for the top-level commands only, and as a result the values for DAVG and KAVG seem unusual when compared to results obtained when VAAI is not enabled . In this instance (and only for this instance), the DAVG and KAVG observed in esxtop should not be interpreted as a performance issue, absent of any other symptoms.”
So there you go. There is always something to learn in this job.
Get notification of these blogs postings and more VMware Storage information by following me on Twitter: @VMwareStorage
Now we have the same esxtop counters with VAAI disabled. In this case, SCSI Commands, Reads and Writes are very much higher since the I/O is not being offloaded to the array, but we do see an expected small value for KAVG/cmd. Again, you can click on the image for a larger view.
So what’s the cause? Well, eventually the explanation was found in the following KB article – Abnormal DAVG and KAVG values observed during VAAI operations. (Nice catch Henrik!) Reproducing verbatim the contents of the KB article, “when VAAI commands are issued via VAAI Filter, there are actually 2 commands sent. These are top-layer commands which are issued and are never sent to the actual device (they stay within the ESX kernel). These commands are intercepted by the VAAI filter and the VAAI plugin, and are replaced by the vendor-specific commands, which are issued to the device.
