Home > Blogs > VMware vSphere Blog


Memory Performance Counters – An Evolved Look at Memory Management

vSphere memory management has evolved over the years taking advantage of new technologies and techniques like using large memory pages, compressing memory, and using solid state devices (SSDs) for swap caching. This evolution has changed the way we need to look at memory usage and memory over-commitment in vSphere virtualized environments. To understand memory management and usage in vSphere we need to understand the metrics available to us, what they mean, and how to use them.

When monitoring performance, just knowing what performance metrics are available and what they measure is half the battle. A good place to see the performance metrics available in vSphere and their definition is in the vSphere SDK documentation for the PerformanceManager object.

Some of the key memory performance and capacity planning metrics available in vSphere are Active, Consumed, Granted and Entitlement.

Key Memory Metrics for a Virtual Machine (full list available here)

Mem.Active

Amount of guest “physical” memory actively used, as estimated by VMkernel based on recently touched memory pages.

Mem.Consumed

Amount of guest physical memory consumed by the virtual machine for guest memory. Consumed memory does not include overhead memory. It includes shared memory and memory that might be reserved, but not actually used. 

Virtual machine consumed memory = memory granted – memory saved due to memory sharing

Mem.Granted

Guest “physical” memory that is mapped to machine memory. Includes shared memory amount. Does not include overhead.

Mem.Entitlement

Amount of host physical memory the virtual machine is entitled to, as determined by the ESX scheduler.

These counters have mostly been around for years, but some counters are more important than others these days due to changes in how vSphere uses memory. Since vSphere 4.0, vSphere backs all virtual machine memory with large pages. This prevents the vSphere Transparent Page Sharing (TPS) memory reclamation techniques, which only operates on small pages, from kicking in as early as it had prior to vSphere 4.0. TPS will still kick in, in vSphere 4.0 and later, but now it will only kick-in once memory pressure occurs and vSphere needs to actively reclaim memory.

This change in behavior has caused the mem.consumed metric to become less important in capacity and memory over-commit planning. These days the memory metric mem.Active is more useful for capacity planning. That is because we are more concerned with how much memory the VM actively needs and is using, than how much it has consumed at some point in time. From a performance standpoint as long as the virtual machine’s actively used memory is in physical memory the virtual machine should perform well.
vSphere also has several counters that measure and report on the impact of memory over-commit.

Memory Over-Commit / Performance Counters

Mem.Latency

Percent of time VM is blocked waiting to access swapped,
compressed or ballooned memory

The larger the number, the larger the impact on VM performance

Cpu.SwapWait

Time the virtual machine is waiting for swap page-ins.

Mem.VmMemCtl

Amount of memory allocated to the VMware balloon driver in the VM.

Mem.SwapInRate

Rate at which memory is swapped from disk into active
memory during the interval.

Mem.Swapped

Current amount of guest physical memory swapped out to the virtual machine’s swap file by the VMkernel. Swapped memory stays on disk until the virtual machine needs it. This statistic refers to VMkernel swapping and not to
guest OS swapping.

In the past we have used the various swap metrics (particularly mem.SwapInRate) to detect memory performance issues. The idea was that any swap-in activity either at the guest or ESXi host would have a very high impact to virtual machine performance. But with techniques like memory compression and host cache (swap to SSD), it has become less important “that we are swapping”, and more important in “how long does it take to swap”.

Because using SSDs as a swap cache location and utilizing memory compression can greatly reduce the amount of time it takes to swap memory, vSphere counters like mem.latency or cpu.swapwait have become more important than SwapInRate. Mem.latency and CPU.SwapWait measures how long the VM was blocked because it was waiting for required memory to be swapped back in to physical memory. Both these counters more clearly show the performance impact of swapping than a simple swap in page rate.

Metrics like Mem.Consumed and Mem.SwapInRate might have been good metrics to use in the past, but today they are not the best metrics when trying to determine what is truly a performance concern.  Today, using metrics like Mem.Active and Mem.Latency can more accurately report memory utilization/capacity and performance.

If you attended VMworld 2012 – US or plan on attending VMworld 2012 – Europe, you can listen to the recording of Session INF-VSP1729 “Understanding Virtualized Memory Performance Management” to get a deeper dive on vSphere memory management. Several other VMworld 2012 presentations are also available online to VMworld attendees (VMworld 2012 Online Sessions) with both slides and audio recording.

 

6 thoughts on “Memory Performance Counters – An Evolved Look at Memory Management

  1. Rob Bergin

    The one memory counter I always ask about is at the cluster level.

    These are VM statistics but I think an important statistic that gets overlooked or I don’t read as much about it is the DRS Entitlement – is this the same metric as Mem.Entitlement.

    I am thinking its not because that Mem.Entitlement existings regardless of a DRS cluster but i wasn’t sure.

    I would like to know the value of monitoring DRS.Mem.Entitlement vs. Mem.Consumed or Mem.Active.

    In regards to resource right-sizing – if you use Mem.Active – it will often shrink the vRAM on a VM too small and make it disfunctional.

    Reply
    1. Craig Risinger

      Mem.Swapped (KB) = KB of data sitting in .vswp files (hypervisor-level data that is swapped out to disk)

      This metric in practice is very ambiguous. It does NOT mean that you are having performance problems right now. It’s just how much data is sitting there. Often that is junk data that never gets read again by anything in the OS, so it’s no problem at all. Of course it’s possible that in the future this data would be requested–you can’t really tell. When it is needed, it’s read in, and that’s usually much slower than the guest OS is expecting–but then SwapInRate(KBps) will be non-zero, and Mem.Latency will be too, indicating current performance problems. So is that data that’s Swapped going to be a future performance problem? You’ll just have to watch the trends to see.

      Short version: Swapped = what’s sitting in .vswp. It might indicate risk of future performance problems, but that’s very uncertain.

      Reply
  2. Pingback: Welcome to vSphere-land! » Performance Links

  3. Joris

    Hi, nice article. At ‘mem.active’ you talk about ‘recently touched memory’. Could you perhaps define the timespan that is ment with ‘recently’? I’ve seen this word coming back several times at different articles, but it’s never told how long ‘recent’ really is … It can be 5 minutes but it can also be 1 hour …

    Reply
  4. ZPggy

    250314 539070We clean up on completion. This might sound obvious but not numerous a plumber in Sydney does. We wear uniforms and always treat your home or office with respect. 546003

    Reply

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>