Home > Blogs > VMware vFabric Blog


Why Hyperic is Going to Support PostgreSQL Only As a Backend Database

The next release of Hyperic is coming up soon and the biggest change is to the backend. In the next release, we will only support one database, namely PostgreSQL. Those of you who have been with Hyperic for a while as long as I have may be surprised considering our history with PostgreSQL, but, as you read though this blog, it will start to make sense.

History of PostgreSQL and Hyperic

For the last few years Hyperic has supported only two databases for production use at scale—Oracle and MySQL. This in itself was a big change since at one point, PostgreSQL was our bread and butter.  Hyperic was originally designed on PostgreSQL 7.x. As an open source project, PostgreSQL has a very easy license for distribution. As a startup company we had to get our product out into the marketplace quickly and affordably, so therefore PostgreSQL made sense.

As the years went on, it was obvious that we could gain adoption for the product by supporting more databases. Larger customers started to demand Oracle for the large deployments and MySQL for the medium sized deployments.  As time passed, supporting three databases proved to be very taxing on our Engineering and QA teams.  For each database, we had to maintain separate code paths in the product to ensure the data access was as efficient as possible. Then QA had to run full regression tests each release with our entire matrix of supported databases. Additionally, supportability by engineering of the product to our support, sales and professional services teams was much more time consuming.

This meant a lot of extra effort, but we made it work for a while to keep our customers happy. However, as we continued to look into the performance of the three databases, we eventually found that MySQL had the best performance (some of you may remember this performance study we published to show how well it worked).  Internally and externally, we had the least amount of issues on MySQL.  The data was convincing enough that eventually we made the hard decision to cut support for PostgreSQL so that we would only support one open source database and one commercial database. Still, we were well aware that the best scenario for the quality and cost of support for the product was to only support one database.

Over the past few years, our experience is that the tides of customer demands have been shifting. We have not been seeing as strong of a demand for either Oracle or MySQL. A couple things have changed in the industry, Oracle bought Sun/MySQL and NoSQL databases have flooded the market. These changes, in my opinion, have worked to lower the likelihood of a customer to demand a specific database and signal that they are more willing to support a variety of databases for different purposes. Essentially, today customers are all about choice, and with lots more options and Oracle & MySQL being operated out of the same company, the demand from our install base is lower.  So while Oracle and MySQL continue to be good products, the freedom of choice is spilling into the software developers arena as well. Now, customers are more willing to accept the developer’s choice to use the right tool for the job in the right way.

Why are we moving back to PostgreSQL?

It is no secret that VMware has invested in PostgreSQL development for virtualized deployments.  Our product, vFabric Postgres, also called vPostgres for short, makes improvements to the core of PostgreSQL and has lots of features which lead to many advantages when running it in a virtualized environment—more than other databases of its kind.  About one year ago, I was approached with the idea of moving Hyperic back to PostgreSQL and dropping support for both MySQL and Oracle.  I’ll admit, my personal reaction to this idea was not favorable.  While I knew supporting only one database would be a good thing for Hyperic, and we had supported PostgreSQL previously so we would have some headstart, it was a lot of work and change for the product for essentially a gamble on a direction that I, at least initially, wasn’t comfortable fully supporting.

In the grand scheme of things, I consider myself a pragmatic person. So, I understood the big picture and started seeking out the opportunities that could come with this change and make it work for both us as an engineering team, but also for our customers and users.

Some of the key things I focused on were:

  • When Hyperic stopped supporting PostgreSQL, we were on version 8.2.5.  PostgreSQL has made some great strides to its performance with 8.3 and 9.x, which would likely work to our advantage.
  • Hyperic is much more efficient at interacting with the database since version 4.0, which was around the time that we dropped PostgreSQL support.  As a result, the Hyperic application was able to achieve a higher end scale regardless of the database attached to it.
  • I have been very impressed with our internal vPostgres team. Considering we have never had direct database engineering support before, we could cut days or weeks of researching configuration issues into a couple hours of technical conversation internally, gaining engineering cycles in every release.

Re-Optimizing for vPostgres

Our high level goals for Hyperic on PostgreSQL were:

  • 2000+ agents
  • 150k metrics / min
  • 50k+ managed resources
  • 500 compatible groups with >= 500 resources / group
  • Ability to have to 10 concurrent users continuously pounding the HQ UI

Lucky for us we always kept PostgreSQL compatibility in Hyperic therefore we didn’t have to modify our schema.  The major challenges were:

  • PostgreSQL 9.x changed it’s blob implementation. Therefore, the driver needed to be updated to a 9.x compatible driver.
  • Migrating from MySQL / Oracle to PostgreSQL for existing customers.
  • Figuring out where database slowness was occurring while applying real load to the Hyperic server.
  • Optimizing our Data Inserter for PostgreSQL, this was a major pain point in the past.
  • Correctly configuring PostgreSQL.

Before getting started, we made a best effort to tune these using the expertise of our vPostgres team, guiding us in the right direction.  We made several code changes in this area to prepare for the real test on a scale environment.

Experience using vPostgres 9.x with Hyperic

To scale up, we started with a large sized Hyperic HQ instance and a large sized vPostgres instance and progressively tried to break it until we reached our goals.  We used Apache JMeter to simulate concurrent user UI load. This allowed us to continuously monitor the slow query log in order to figure out what queries / flows in various areas of the application needed work.  Additionally, we used tcpdump analysis to gather response time metrics on a per page basis and understand the differential between our MySQL / Oracle instances and our new, scaled up vPostgres instance.  In the end, we were able to tune our vPostgres implementation to be, at minimum, on par with respect to MySQL and Oracle. Then, we further tuned it to outperform these backends in certain cases.

While going through this process to increase the Hyperic server load, I was impressed that vPostgres did not break down as it had in the past.  The configuration we used was relatively the default settings, mainly sizing the shared_buffers and effective_cache properly.  At this time, there is only one area where performance has not met or exceeded previous releases—the I/O write performance of our metric data inserts.  Our vPostgres team is currently investigating this, and we expect, within the next couple of releases, that this will no longer be an issue. Even though we achieved the same scale of MySQL and Oracle, we are working to substantially increase the scale in future versions.

Using Hyperic to Analyze vPostgres Performance

One nice thing about optimizing Hyperic is that I was able to use Hyperic’s monitoring capability with our newly enhanced vPostgres plugin to monitor it’s own performance.

Here is a sample of the configuration to get the monitoring started:

To monitor the I/O, it is very simple to create a resource for monitoring the data mount and the log mount:



Once this is complete, here is a sample of the data available for viewing.

DataDir Mount Sample Data:

LogDir Mount Sample Data:

VPostgres Plugin Sample Data:

In all, this experience has made me a believer again that vPostgres is moving in the right direction with more to come.  We have invested a lot into making vPostgres work well for Hyperic and for cloud deployments, and since the database isn’t overly configured to optimize it, I believe it will be easy for customers to support internally. I am excited to see how the evolution turns out, so as you try it out, either in the current beta program or once we release it, be sure to let us know how its going.

About the Author: Scott Feldstein has spent his entire 12+ year career implementing Datacenter management solutions.  Starting at Sun Microsystems, Scott spent the first half of this time working on very large scale monitoring solutions for Sun’s internal compute grids.  In 2007 he started at a small startup company called Hyperic which eventually became part of VMware.  Scott’s technology passions are working on large scale software implementations and working with data storage solutions.  Scott received a Computer Engineering degree from Cal Poly (SLO) and currently works out of the VMware San Francisco office.

23 thoughts on “Why Hyperic is Going to Support PostgreSQL Only As a Backend Database

  1. Pingback: Why Hyperic is Going to Support PostgreSQL Only As a Backend Database | My Daily Feeds

  2. Pingback: New Hyperic 5.0 Release Further Embeds Web Infrastructure Monitoring in VMware’s vFabric | VMware vFabric Blog - VMware Blogs

  3. Pingback: VMware vFabric Blog: New Hyperic 5.0 Release Further Embeds Web Infrastructure Monitoring in VMware’s vFabric | Virtualization

  4. Pingback: VMware vFabric Blog: New Hyperic 5.0 Release Further Embeds Web Infrastructure Monitoring in VMware’s vFabric | Strategic HR

  5. Erik Bowman

    We are new to Hyperic and have procured licenses for 200+ platforms and will be procuring additional licenses. We have engineering experience on our team with Oracle (MySQL) and after reading this article, should we consider PostgreSQL and vFavbric PostgreSQL? We are thinking of hosting Hyperic in a virtual environment. Again, we are new to this and need some guidance prior to implementation – sorry for the silly questions. Thanks, Erik

    Reply
    1. Eran Carmel

      If you are thinking of hosting Hyperic in a virtual environment you should just deploy version 5.0 virtual appliance which already includes vPostgreSQL as part of the installation.

      Reply
  6. Benny Hill

    Hi Erik,

    Thank you for your question and it is not silly at all. Due to the new requirements, you should opt for one of the Postgres solutions as no other database will be supported.

    If you want ease of deployment, vFabric PostgreSQL would be an excellent option. This information is taken directly from our online documentation:

    vPostgres can be deployed as a Virtual Appliance or RPM for Linux for easy installation. The Smart Configuration capabilities makes it ready to deploy out of the box and reduce the need for specialized PostgreSQL knowledge.

    Increase memory efficiency through vSphere-integrated elastic database memory

    Optimize database configurations using Smart Configuration over vSphere infrastructure

    Manage deployment and lifecycle management with vFabric Data Director

    Based on this, if you want an easy way to deploy your database with optimized integration, this would be a good choice. I hope this helps.

    Reply
  7. Pingback: The Best VMware vFabric Stories of 2012 & What’s In Store for 2013 | VMware vFabric Blog - VMware Blogs

  8. Pingback: VMware vFabric Blog: The Best VMware vFabric Stories of 2012 & What’s In Store for 2013 | Virtualization

  9. Sean

    Wow! And your sales reps didn’t think to tell us this when selling licences at the end of last year when they know that we require both Oracle as the backend and Websphere 8.0 / 8.5 plugins?

    Even if I can get over the 1000 hurdles to convince the company to run PostGres for this one special case, do you plan to provide some solid migration documentation for the move from oracle to postgres?

    Dismayed to say the least!

    Reply
  10. R Cross

    You hardly need to read between the lines to see that this was predominantly a commercial move.

    MySQL is still by far the most prevalent database for simple Web apps (especially LAMP/WAMP, obviously) and I’ll bet you anything a straw poll across the Internet will back that up. Performance tuning, Slave replication, Clustering, switching between storage engines… all trivial operations for a SysAdmin (no need for a DBA).

    Postgres, on the other hand, has a steep learning curve and an obscure command set. In fact, from a Sys Admin’s point of view, it is utterly horrible to work with compared to MySQL.

    By moving away from MySQL, you risk alienating a huge proportion of your customer base, but good luck with that.

    Reply
  11. bitcoin

    I’m really impressed with your writing skills and also with
    the layout on your blog. Is this a paid theme or did you modify it yourself?
    Either way keep up the excellent quality writing, it’s rare to see a nice blog like
    this one nowadays.

    Also visit my web site; bitcoin

    Reply
  12. wedding photos

    We’re a group of volunteers and opening a new scheme in our community.
    Your web site provided us with valuable info to work on. You have done an impressive job
    and our entire community will be thankful to you.

    Reply

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>