Home > Blogs > VMware Accelerate Advisory Services > Monthly Archives: November 2011

Monthly Archives: November 2011

Moving the Needle – You Can Get There From Here

AUTHOR: Craig Stanley (@benchmarkguru)

 The commonly stated mantra of benchmarking is that “you can’t manage what you can’t measure”.  While that is certainly a truism, a corollary I’d like add is that “you can’t measure what you can’t quantify”.  Figuring out what KPIs you need to track is usually not too difficult, but determining how to quantify some KPIs can seemingly involve black magic, smoke and mirrors.  It can be mystifying to say the least.  This article discusses the benchmarking measures and quantification behind VMware’s Infrastructure Move-The-Needle Vision Benchmark and will hopefully; shed a little light on the process.


The VMware Vision Benchmarking Team provides a customer service benchmark analysis called a Move-The-Needle.  We determine a few basic KPIs, quantify where a customer’s performance is on these KPIs, measure the gap between where the customer is and where the best performing peers are and calculate the potential benefit of closing that gap, or moving-the-needle.  It is important to note that some industries are more heavily invested in virtualization than others. In some cases, even the best performers may be woefully behind the technology curve in terms of leveraging virtualization into significant operational savings.  Analytical insight into the technological maturity of the industry in addition to the best performing peers is invaluable in putting the information into an actionable perspective.


In the Infrastructure Move-The-Needle Benchmark, we look at five Key Performance Indicators:

  1. Percentage of x86 Physical Servers Virtualized
  2. Percentage of Desktops Virtualized
  3. Percentage of Virtual Machines that are DR Protected
  4. Virtual Machine to VM Administrator Ratio
  5. Number of Days Needed to Provision a Virtual Machine


These KPIs were initially selected based on feedback from large IT organizations as the basic metrics they needed to track and compare in order to measure virtualization maturity.  These have the added benefit of being easy to collect and track consistently over time and between competing organizations.  In the chart below, we illustrate the needle position and the impact of moving it to the best performing peer levels.


  MTN You vs Peers


  1. x86 Percentage Virtualized – What is the percentage of the total number of x86 servers that are virtualized? This metric is used to gauge the overall integration of server virtualization in the organization and is important, as the road to a private cloud requires a high level of server virtualization.  Additionally, significant CapEx and OpEx savings can be realized as virtualization drives up the utilization density while reducing the number of physically managed, energy consuming devices.  Based on VMware ROI figures, we estimate that an organization can save approximately $3,400 for every physical server they can virtualize. About 84% of the potential savings is CapEx with 16% recurring annually.  An added benefit is that physical servers may be able to be redeployed to offset growth or may be powered-off until needed to save on operating costs.
  2. Desktop Percent Virtualized – What is the percentage of the total number of desktops that are virtualized?  This metric measures the depth of the implementation of VDI (Virtual Desktop Infrastructure) initiatives, which also play heavily in the future cloud ubiquitous computing model.  The goal of desktop virtualization is threefold: 1.) Remove as much of the end user, physical support as possible, 2.)Consolidate into significantly fewer managed and maintained user images, 3.)Create an environment where the user experience is not tied to any one physical I/O device or location.

    Initially, an investment in VDI is not likely to save or replace much hardware or software expense.  End users still need an output device, be it a thin client on a PC or other device, and investment must be made in servers to support desktop virtualization.  However, image and application support requirement such as patching, config management, etc., may be significantly reduced.  We estimate that each desktop virtualized may carry a potential savings of about $500 per PC migrated.

    There are hurdles to VDI as evident in the data we see suggesting a relatively small adoption rate, but VDI may prove to be one of those areas of potential strategic advantage for organizations seeking to deliver applications on demand and support multiple device form factors. 

  3. Percentage of VMs under DR Protection – How many of your VMs are under a DR protection plan?  This metric yields important information about the type of workloads supported on VMs.  We expect a low percentage score may suggest that the VM workloads are not Tier 1 since they are not part of a DR implementation, while a higher percentage here suggests the VMs may be identified as Tier 1.  This particular metric tends to provide better insight than just a metric of number of Tier 1 applications on VMs.  The action to put them under DR protection is better proof that these applications are considered critical based on function not just name or business unit ownership.

    Based on the VMware ROI data, we’ve estimated a potential savings of about $550 per hour of avoided downtime.  Obviously, your results may vary depending on how much impact a particular Tier 1 application outage would have, but this can give you some idea of the potential savings or conversely, the potential exposure.

  4. VM to Admin Support Ratio – How many VMs do you support per VM Administration FTE?  This is a commonly used and asked-for metric as it illustrates a support-FTE utilization ratio.  The basic tenant of virtualization is that you are able to support more VMs than physical machines with the same number of FTEs.  We would expect to see this number in the hundreds of VMs per Admin FTE based on the virtualization maturity of the organization and the number of VMs.  It is important to note that the denominator is an FTE (Full Time Equivalent), which could be made up of fractions of multiple admin’s time and does not necessarily map one-to-one to physical headcount.

    Moving the needle on this metric is measured in fewer FTEs needed to deliver the same quantity of support.  While reducing the number of FTEs is one interpretation of this result, most organizations seek to leverage the freed-up resources to handle other projects or growth.
    Something else to keep in mind is that a high ratio here, even higher than your peer group, may not be a good thing.  This is where the analysis of multiple metrics comes into play.  In many cases, an analysis cannot be based on a single metric or piece of information, and this metric is a good example of that.  Let’s say I support 1,000 VMs per Admin and my peers only support 500 VMs per Admin.  Certainly, my VM:Admin ratio (productivity) is better, but since this metric does involve a limited resource, ie labor, what does my labor performance look like?  Does it take me several days to provision a VM?  If so, then maybe I’m stretched too thin at a 1,000 to 1 ratio.
    Server utilization is another good example of these dependent KPIs.  If all my servers are always running at 100%, is that necessarily the best way to be operating?  If I can’t deliver the workloads to the required SLA, then no, it isn’t.

  5. Days to Provision a VM – How many days does it take to provision a VM including requests, approvals and implementation?  The understanding of this metric covers a couple of performance issues.  The first is the level of automation and repeatable processes in place to be able to quickly provision a VM and deploy it into production.  The second factor is the degree of standardization, policies and governance that enables a VM request to be quickly approved, procured and provisioned.  We find that organizations farther along in their cloud journey have the processes and governance in place to be able to procure and provision a VM into production in only a few days, much faster than organizations without them.

    This metric is also useful as the counterpart to the VM:Admin ratio as noted earlier and it balances speed to deliver against the resource level needed to deliver that speed.  For example, if you can procure and provision a VM in a day while still maintaining a higher than average VM:Admin ratio, then you are likely to be more effective and efficient than peers.  But if your provisioning time is lengthy with a high VM:Admin ratio, it may be indicative of a staffing shortage or lack of automation.

    The value to the organization by reducing this number is that VMs and consequently, additional applications, can be deployed faster helping IT support a more agile, robust enterprise.  We arrive at a savings figure based on $1,000 per day in opportunity savings resulting from being able to provision your virtualizable physical servers faster.  While this is a typical soft or indirect saving, and the actual saving per day might vary, it does give us a way to assign a dollar saving to this improvement.

    These are but a few of the many metrics that can be used to measure performance, utilization, efficiency and effectiveness.  In this Infrastructure Move-The-Needle, we identify and quantify 5 KPIs that can drive savings and value back to the IT organization and the enterprise.  By knowing where we are, and knowing where the best performers are, we can close that gap and reap the savings.  You can get there from here by moving the needle.


Craig Stanley is the Benchmarking Practice Lead for the VMware Vision Team. You can follow him on Twitter @benchmarkguru.




Getting Started With Benchmarking

AUTHOR: Craig Stanley (@benchmarkguru)


This article is the first in a series that will explain VMware’s context for benchmarking virtualization and cloud implementation, and discuss the Key Performance Indexes (KPIs) that are used to quantify, measure, and analyze the organizations journey toward ITaaS.


The first step in the benchmarking process is to determine what needs to be measured and how it will be quantified.


Quantification is all about determining:

  1. What are the KPIs (Key Performance Metrics) that matter and can I determine what data elements they are based on?
  2. Can I get the information I need consistently to calculate my KPI?
  3. Can my KPIs be compared against my competition or industry?


First, what are the KPIs that matter most to your operation? Typically, it may be the ratio of a consumable resource to a quantity of production such as a cost per application developed or a cost per server.  Other consumption-based metrics may be expressed as ratios like cost per FTE, FTEs per system, or VMs per Host.  There are dozens if not hundreds of metrics that may be interesting, but the ones you track need to:


  1. Measure your current state – the KPIs should not be based on data elements that are out-of-date by the time you collect them.
  2. Provide meaningful information that supports your strategy – the KPIs should matter.
  3. Be easy to understand and explain – the relationship of the underlying data should not be so esoteric that the resulting relationship is dubious.


Next, is the KPI based on information you can get consistently over time?  In order for benchmarking to be useful, the KPIs need to be based on apples-to-apples comparative data elements.  Having consistent data and KPIs also enables you to evaluate year-over-year performance with relevant metrics. This ability is an invaluable oasis of information to validate, justify and monitor continuous improvement.


Lastly, are the KPIs specific to just your organization? Are they common across your competition or industry and can you get access to the KPI data elements from your competition?  The primary benefit of benchmarking is to help identify KPIs and determine where you are and what progress you’ve made.  Another major benefit is that it provides you the basis you need to see how you are performing against your competition. 


KPIs may be industry-focused such as IT spend per new policy or IT spend per claim for an insurance company for example.  Comparing KPIs like this across industry competitors can provide significant insight into the IT / business landscape and valuable feedback on strategy.


In my next blog, I’ll talk about the KPIs used for the VMware Infrastructure Move The Needle and explain how they can help quantify the value of moving from your present virtualization state to the level of the best performing peers.


Craig Stanley is the Benchmarking Practice Lead for the VMware Vision Team. You can follow him on Twitter @benchmarkguru.