Home > Blogs > VMware Accelerate Advisory Services


Comparative KPI Context – Keeping It Real

Author: Craig Stanley (@benchmarkguru)

 In the world of benchmarking, we have the concept of metrics and key performance indicators (KPI), but a fundamental problem is that by themselves, most if not all of these are completely meaningless. For a metric to have meaning, it must be taken in context within other metrics to produce a clear picture.

 Let’s take an auto race, for example. If my only KPI is the speed I’m traveling, and I have no idea my direction, then my speed is meaningless in the context of my overall goal, which is to win the race. Let’s say I’m traveling 100 mph and my competitors are traveling 80 mph. On the basis of that KPI alone, I’m performing 20 percent better than my peers, so I’m better, right? Well, not if I’m traveling west and the race direction is east. Similarly, if we only tracked sales volume without the context of profitability, then we could be losing money on every sale, and improving our volume just helps us lose money faster.

 The basic concept here is that we need at least two pieces of information to make the information relevant. The same problem was documented in quantum physics with the Heisenberg uncertainty principle, which states we can never be sure where an object is because both its momentum and precise location can never be known at the same time. But in benchmarking, we can know at least two conjunctive metrics.

 In benchmarking total cost of ownership (TCO), we have to address the same issues. If I compare my costs with peers and only look at my TCO for a virtual machine, server, desktop, and so forth, then all I know is what my spend is in relation to peers. I have no insight into why my costs are higher or lower than peers. Do I have a lower TCO because I am more cost-effective, or because I have older systems, fewer resources, no maintenance expense, and a lower quality of service? Conversely, if I spend more than peers, am I providing a better quality of service than they are, or am I just more expensive? Without measuring and comparing the complementary KPIs, these questions can’t really be answered.

 So where to start? I suggest starting with a couple of the major KPIs, and evaluate what other processes and KPIs could impact or contribute to that result. I’ve mentioned one, which is looking at TCO within the context of service delivery and/or quality of service. Another type I like to compare is resource productivity against results productivity. In VMware’s Accelerate Benchmarking, I compare the virtual machine managed per administrator (resource productivity) with the time to provision a virtual machine (results productivity). This helps place productivity within the context of results.

 For example, if I am supporting 1,500 virtual machines per administrator full-time equivalent (FTE) and my provisioning time is less than eight hours, then I’m probably providing the needed quality of service at that level of productivity—so I’m doing well. But if my provisioning time is more than a few days, then a problem exists, and I need to understand why. Again, if I only looked at the provisioning time KPI alone, then I only know it takes a long time; I still don’t know why. But within the context of the VM:Admin ratio, I have more information that may suggest that my support level is too dense for the tools, people and processes I have in place. It gives me a place to start looking, and it is likely the key driver of results productivity.

 There are many more comparative KPIs that help provide context within a benchmarking analysis. Finding and correlating these will unlock a treasure trove of hidden information to help you better understand the environment and identify new insights. These insights will draw focus toward improving efficiency, becoming more agile and delivering a higher quality of service to the IT organization and the business, all of which lead to differentiation and competitive advantage.

 Craig Stanley is the Benchmarking Practice Lead for VMware Accelerate Advisory Services. You can follow him on Twitter @benchmarkguru.

 Learn more about VMware Accelerate Advisory Services at vmware.com/go/accelerate or contact us at accelerate@vmware.com