Home > Blogs > VMware vFabric Blog


Enhance root cause analysis capabilities by overlaying AppInsight with Hyperic Data

Join the APM Beta

 

VMware is looking for customers
interested in the beta program for the next release of vFabric Application Performance Management (APM).

Click for More

vFabric Application Performance Manager (APM) provides comprehensive application monitoring capabilities. APM installs with two components: vFabric AppInsight for Transaction and Code level monitoring (byte code instrumentation is courtesy of Spring Insight) and vFabric Hyperic for middleware and OS monitoring. Together, these products work to not only provide an early warning system for building performance problems, but they also provide critical perspectives necessary for Root Cause Analysis (RCA). In this post, we will discuss how using Hyperic data inside AppInsight will help speed RCA.

Analyzing Performance Problems

In production, triaging performance issues requires correlating multiple events from multiple sources. In many cases, a performance problem or an error visible at the application or user transaction level is caused by an underlying middleware or resource problem. vFabric APM lets the user view and correlate multiple metrics from various sources ‫(middleware, network and code level) in order to pinpoint Root Cause Analysis and reduce Mean Time To Resolution (MTTR).

Let’s examine a 3 tier web application. Using the Konakart application as an example, we instantly notice the following for an application that is running smoothly: Screen shot 2012-07-18 at 1.02.14 PM

  • In general, application performance is OK given all application objects health are green.
  • Hit rate at ~ 204 hits per minute.
  • Pay attention to the “Key Metrics” widget in the bottom of the picture where you can see the 3 default metrics:  Error percent, Average latency and hit rate.

Simulated Performance Issue

We will now simulate a mistakenly implemented configuration change (we won’t disclose what that was just yet, let’s see if we can find this out in a few steps…)

Observed behavior at the application level: Screen shot 2012-07-18 at 1.00.54 PM

  • Hit rate declines rapidly, Average hit rate starts to decline from 204 to 141
  • The Usage KPI turns red, which is being propagated to the Application KPI which turns yellow.
  • Change notification is displayed in the Key Metrics over time graph, a clear indication that a configuration change related to JVM memory has been made. In fact, this change event is caught by Hyperic and seamlessly presented in AppInsight.

Root Cause Analysis

Having the spotlight turned to the JVM Memory, let’s add relevant metrics to the Key Metrics over time graph to help us in the RCA process. 
Press the Customize button in the Key Metrics graph and select the
Screen shot 2012-07-18 at 1.03.31 PMtwo metrics related to heap memory (Heap Memory Free and Heap Memory Used) in the Middleware section. This will now overlay these metrics on the Key Metrics over time graph and allow us to compare the heap memory metrics before and after the performance issue time. 

Now looking at the Key Metrics over time chart, we notice that the 120.5 MB Free Heap Memory plunges to 0.2 MB and as a result Hit Rate drops from 224 all the way to 0. This lets us know that somehow heap has been constrained to the point where transactions are no longer possible. So what happened to the heap size?

 

    Before Configuration Change                    After Configuration Change

Screen shot 2012-07-18 at 12.52.12 PM

Screen shot 2012-07-18 at 12.55.00 PM

 

 

 

 

The culprit must be in the JVM heap size configuration. A quick look at the setenv.sh file reveals the value was mistakenly configured to 24M instead of 1024M.

Screen shot 2012-07-18 at 12.59.28 PM

Resolution

Screen shot 2012-07-12 at 12.12.01 PM

Reconfiguring the JVM heap size back to 1024M fixes the issue. Once the heap memory free is back to normal,  our Konakart Application becomes green again!

Summary

Configuration mistakes are a common root case to severe performance issues. Analyzing performance issues may take a lot of time without indication for the initial RCA direction.  In this example, utilizing Hyperic middleware metrics helped us find the culprit quickly by:

  1. Providing fast and clear indication of a performance issue
  2. Presenting informative change alerts helped focus the analysis in the right direction
  3. Overlaying AppInsight with relevant Hyperic metrics pinpoints the root cause analysis
Ophir Headshot

About the Author: Ophir Cohen has worked at b-hive since Feb 2007. He was acquired by VMware in July 2008 . Ophir is responsible for leading the customer facing engineering team including commercial engineering as well as customer facing activities like Alpha, Beta and design partner programs. He received his BCs in Management and Industry Engineering with a focus in Information systems from Israel Ben-Gurion University

This entry was posted in APM and tagged , , , , on by .
Stacey Schneider

About Stacey Schneider

Stacey Schneider has over 15 years of working with technology, with a focus on working with sales and marketing automation as well as internationalization. Schneider has held roles in services, engineering, products and was the former head of marketing and community for Hyperic before it was acquired by SpringSource and VMware. She is now working as a product marketing manager across the vFabric products at VMware, including supporting Hyperic. Prior to Hyperic, Schneider held various positions at CRM software pioneer Siebel Systems, including Group Director of Technology Product Marketing, a role for which her contributions awarded her a patent. Schneider received her BS in Economics with a focus in International Business from the Pennsylvania State University.

3 thoughts on “Enhance root cause analysis capabilities by overlaying AppInsight with Hyperic Data

  1. pedro

    Is there an APM adapter for vCOPs Enterprise?

    Reply
  2. کولر گازی

    re an APM adapte
    قیمت کولر گازی

    Reply
  3. خدمات تلويزيون شهري

    خدمات تلويزيون شهري

    Reply

Leave a Reply

Your email address will not be published. Required fields are marked *

*