Home > Blogs > VMware vFabric Blog


Why Hyperic is Going to Support PostgreSQL Only As a Backend Database

The next release of Hyperic is coming up soon and the biggest change is to the backend. In the next release, we will only support one database, namely PostgreSQL. Those of you who have been with Hyperic for a while as long as I have may be surprised considering our history with PostgreSQL, but, as you read though this blog, it will start to make sense.

History of PostgreSQL and Hyperic

For the last few years Hyperic has supported only two databases for production use at scale—Oracle and MySQL. This in itself was a big change since at one point, PostgreSQL was our bread and butter.  Hyperic was originally designed on PostgreSQL 7.x. As an open source project, PostgreSQL has a very easy license for distribution. As a startup company we had to get our product out into the marketplace quickly and affordably, so therefore PostgreSQL made sense.

As the years went on, it was obvious that we could gain adoption for the product by supporting more databases. Larger customers started to demand Oracle for the large deployments and MySQL for the medium sized deployments.  As time passed, supporting three databases proved to be very taxing on our Engineering and QA teams.  For each database, we had to maintain separate code paths in the product to ensure the data access was as efficient as possible. Then QA had to run full regression tests each release with our entire matrix of supported databases. Additionally, supportability by engineering of the product to our support, sales and professional services teams was much more time consuming.

This meant a lot of extra effort, but we made it work for a while to keep our customers happy. However, as we continued to look into the performance of the three databases, we eventually found that MySQL had the best performance (some of you may remember this performance study we published to show how well it worked).  Internally and externally, we had the least amount of issues on MySQL.  The data was convincing enough that eventually we made the hard decision to cut support for PostgreSQL so that we would only support one open source database and one commercial database. Still, we were well aware that the best scenario for the quality and cost of support for the product was to only support one database.

Over the past few years, our experience is that the tides of customer demands have been shifting. We have not been seeing as strong of a demand for either Oracle or MySQL. A couple things have changed in the industry, Oracle bought Sun/MySQL and NoSQL databases have flooded the market. These changes, in my opinion, have worked to lower the likelihood of a customer to demand a specific database and signal that they are more willing to support a variety of databases for different purposes. Essentially, today customers are all about choice, and with lots more options and Oracle & MySQL being operated out of the same company, the demand from our install base is lower.  So while Oracle and MySQL continue to be good products, the freedom of choice is spilling into the software developers arena as well. Now, customers are more willing to accept the developer’s choice to use the right tool for the job in the right way.

Why are we moving back to PostgreSQL?

It is no secret that VMware has invested in PostgreSQL development for virtualized deployments.  Our product, vFabric Postgres, also called vPostgres for short, makes improvements to the core of PostgreSQL and has lots of features which lead to many advantages when running it in a virtualized environment—more than other databases of its kind.  About one year ago, I was approached with the idea of moving Hyperic back to PostgreSQL and dropping support for both MySQL and Oracle.  I’ll admit, my personal reaction to this idea was not favorable.  While I knew supporting only one database would be a good thing for Hyperic, and we had supported PostgreSQL previously so we would have some headstart, it was a lot of work and change for the product for essentially a gamble on a direction that I, at least initially, wasn’t comfortable fully supporting.

In the grand scheme of things, I consider myself a pragmatic person. So, I understood the big picture and started seeking out the opportunities that could come with this change and make it work for both us as an engineering team, but also for our customers and users.

Some of the key things I focused on were:

  • When Hyperic stopped supporting PostgreSQL, we were on version 8.2.5.  PostgreSQL has made some great strides to its performance with 8.3 and 9.x, which would likely work to our advantage.
  • Hyperic is much more efficient at interacting with the database since version 4.0, which was around the time that we dropped PostgreSQL support.  As a result, the Hyperic application was able to achieve a higher end scale regardless of the database attached to it.
  • I have been very impressed with our internal vPostgres team. Considering we have never had direct database engineering support before, we could cut days or weeks of researching configuration issues into a couple hours of technical conversation internally, gaining engineering cycles in every release.

Re-Optimizing for vPostgres

Our high level goals for Hyperic on PostgreSQL were:

  • 2000+ agents
  • 150k metrics / min
  • 50k+ managed resources
  • 500 compatible groups with >= 500 resources / group
  • Ability to have to 10 concurrent users continuously pounding the HQ UI

Lucky for us we always kept PostgreSQL compatibility in Hyperic therefore we didn’t have to modify our schema.  The major challenges were:

  • PostgreSQL 9.x changed it’s blob implementation. Therefore, the driver needed to be updated to a 9.x compatible driver.
  • Migrating from MySQL / Oracle to PostgreSQL for existing customers.
  • Figuring out where database slowness was occurring while applying real load to the Hyperic server.
  • Optimizing our Data Inserter for PostgreSQL, this was a major pain point in the past.
  • Correctly configuring PostgreSQL.

Before getting started, we made a best effort to tune these using the expertise of our vPostgres team, guiding us in the right direction.  We made several code changes in this area to prepare for the real test on a scale environment.

Experience using vPostgres 9.x with Hyperic

To scale up, we started with a large sized Hyperic HQ instance and a large sized vPostgres instance and progressively tried to break it until we reached our goals.  We used Apache JMeter to simulate concurrent user UI load. This allowed us to continuously monitor the slow query log in order to figure out what queries / flows in various areas of the application needed work.  Additionally, we used tcpdump analysis to gather response time metrics on a per page basis and understand the differential between our MySQL / Oracle instances and our new, scaled up vPostgres instance.  In the end, we were able to tune our vPostgres implementation to be, at minimum, on par with respect to MySQL and Oracle. Then, we further tuned it to outperform these backends in certain cases.

While going through this process to increase the Hyperic server load, I was impressed that vPostgres did not break down as it had in the past.  The configuration we used was relatively the default settings, mainly sizing the shared_buffers and effective_cache properly.  At this time, there is only one area where performance has not met or exceeded previous releases—the I/O write performance of our metric data inserts.  Our vPostgres team is currently investigating this, and we expect, within the next couple of releases, that this will no longer be an issue. Even though we achieved the same scale of MySQL and Oracle, we are working to substantially increase the scale in future versions.

Using Hyperic to Analyze vPostgres Performance

One nice thing about optimizing Hyperic is that I was able to use Hyperic’s monitoring capability with our newly enhanced vPostgres plugin to monitor it’s own performance.

Here is a sample of the configuration to get the monitoring started:

To monitor the I/O, it is very simple to create a resource for monitoring the data mount and the log mount:



Once this is complete, here is a sample of the data available for viewing.

DataDir Mount Sample Data:

LogDir Mount Sample Data:

VPostgres Plugin Sample Data:

In all, this experience has made me a believer again that vPostgres is moving in the right direction with more to come.  We have invested a lot into making vPostgres work well for Hyperic and for cloud deployments, and since the database isn’t overly configured to optimize it, I believe it will be easy for customers to support internally. I am excited to see how the evolution turns out, so as you try it out, either in the current beta program or once we release it, be sure to let us know how its going.

About the Author: Scott Feldstein has spent his entire 12+ year career implementing Datacenter management solutions.  Starting at Sun Microsystems, Scott spent the first half of this time working on very large scale monitoring solutions for Sun’s internal compute grids.  In 2007 he started at a small startup company called Hyperic which eventually became part of VMware.  Scott’s technology passions are working on large scale software implementations and working with data storage solutions.  Scott received a Computer Engineering degree from Cal Poly (SLO) and currently works out of the VMware San Francisco office.

45 thoughts on “Why Hyperic is Going to Support PostgreSQL Only As a Backend Database

  1. Pingback: Why Hyperic is Going to Support PostgreSQL Only As a Backend Database | My Daily Feeds

  2. Chip Witt

    Nice write-up on history and current direction, Scott! My best to the team.

    Reply
    1. Smadav 2019

      Thanks for great posts, visit us too, backup system for telegram

      Reply
  3. Richard

    Will it support vanilla Postgres, or just vPostgres?

    Reply
    1. Adar

      Vanilla Postgres is supported as well.

      Reply
  4. Paul

    Wow, wish we did that when I ran engineering at hyperic 🙂

    Reply
  5. Pingback: New Hyperic 5.0 Release Further Embeds Web Infrastructure Monitoring in VMware’s vFabric | VMware vFabric Blog - VMware Blogs

  6. Pingback: VMware vFabric Blog: New Hyperic 5.0 Release Further Embeds Web Infrastructure Monitoring in VMware’s vFabric | Virtualization

  7. Pingback: VMware vFabric Blog: New Hyperic 5.0 Release Further Embeds Web Infrastructure Monitoring in VMware’s vFabric | Strategic HR

  8. Erik Bowman

    We are new to Hyperic and have procured licenses for 200+ platforms and will be procuring additional licenses. We have engineering experience on our team with Oracle (MySQL) and after reading this article, should we consider PostgreSQL and vFavbric PostgreSQL? We are thinking of hosting Hyperic in a virtual environment. Again, we are new to this and need some guidance prior to implementation – sorry for the silly questions. Thanks, Erik

    Reply
    1. Eran Carmel

      If you are thinking of hosting Hyperic in a virtual environment you should just deploy version 5.0 virtual appliance which already includes vPostgreSQL as part of the installation.

      Reply
  9. Benny Hill

    Hi Erik,

    Thank you for your question and it is not silly at all. Due to the new requirements, you should opt for one of the Postgres solutions as no other database will be supported.

    If you want ease of deployment, vFabric PostgreSQL would be an excellent option. This information is taken directly from our online documentation:

    vPostgres can be deployed as a Virtual Appliance or RPM for Linux for easy installation. The Smart Configuration capabilities makes it ready to deploy out of the box and reduce the need for specialized PostgreSQL knowledge.

    Increase memory efficiency through vSphere-integrated elastic database memory

    Optimize database configurations using Smart Configuration over vSphere infrastructure

    Manage deployment and lifecycle management with vFabric Data Director

    Based on this, if you want an easy way to deploy your database with optimized integration, this would be a good choice. I hope this helps.

    Reply
  10. Pingback: The Best VMware vFabric Stories of 2012 & What’s In Store for 2013 | VMware vFabric Blog - VMware Blogs

  11. Pingback: VMware vFabric Blog: The Best VMware vFabric Stories of 2012 & What’s In Store for 2013 | Virtualization

  12. Sean

    Wow! And your sales reps didn’t think to tell us this when selling licences at the end of last year when they know that we require both Oracle as the backend and Websphere 8.0 / 8.5 plugins?

    Even if I can get over the 1000 hurdles to convince the company to run PostGres for this one special case, do you plan to provide some solid migration documentation for the move from oracle to postgres?

    Dismayed to say the least!

    Reply
    1. Eran Carmel

      Since we wanted the database to move to be as painless as possible we created a utility that migrates the data and settings to Postgres from your current database. You can read about it in this blog post http://blogs.vmware.com/vfabric/2012/12/how-to-prepare-for-a-hyperic-5-0-upgrade.html.
      In addition WebSphere 8 and 8.5 support are planned in H1 2013.

      Reply
  13. R Cross

    You hardly need to read between the lines to see that this was predominantly a commercial move.

    MySQL is still by far the most prevalent database for simple Web apps (especially LAMP/WAMP, obviously) and I’ll bet you anything a straw poll across the Internet will back that up. Performance tuning, Slave replication, Clustering, switching between storage engines… all trivial operations for a SysAdmin (no need for a DBA).

    Postgres, on the other hand, has a steep learning curve and an obscure command set. In fact, from a Sys Admin’s point of view, it is utterly horrible to work with compared to MySQL.

    By moving away from MySQL, you risk alienating a huge proportion of your customer base, but good luck with that.

    Reply
  14. bitcoin

    I’m really impressed with your writing skills and also with
    the layout on your blog. Is this a paid theme or did you modify it yourself?
    Either way keep up the excellent quality writing, it’s rare to see a nice blog like
    this one nowadays.

    Also visit my web site; bitcoin

    Reply
  15. lissage brésilien pas cher

    I think the admin of this site is genuinely working hard in favor
    of his site, as here every stuff is quality based material.

    My site; lissage brésilien pas cher

    Reply
  16. http://www.monterotondo.biz

    Se procurer les fameux emonnaies avec le trade
    tout simplement

    Also visit my web page: http://www.monterotondo.biz

    Reply
  17. wedding photos

    We’re a group of volunteers and opening a new scheme in our community.
    Your web site provided us with valuable info to work on. You have done an impressive job
    and our entire community will be thankful to you.

    Reply
  18. canape cuir design design

    I would like to thank you for the efforts you’ve put in penning this blog.

    I really hope to view the same high-grade blog posts by
    you later on as well. In fact, your creative writing abilities has motivated me to get my own, personal site now 😉

    My website canape cuir design design

    Reply
  19. acheter bitcoin

    Se fournir des crypto monnaies sur paypal aisément

    my homepage :: acheter bitcoin

    Reply
  20. http://www.cowforum.com

    Comment se procurer du bitcoin en promo sur paypal sans
    risque

    Here is my site … http://www.cowforum.com

    Reply
  21. litecoin

    Découvrez comment se procurer des ltc pas cher par
    la toile en quelques minutes

    Also visit my blog; litecoin

    Reply
  22. http://www.biomonde.biz

    Helpful info. Fortunate me I found your website by chance, and
    I’m stunned why this twist of fate didn’t happened in advance!
    I bookmarked it.

    Stop by my website … http://www.biomonde.biz

    Reply
  23. robot piscine dolphin

    you’re in reality a good webmaster. The site loading pace is
    amazing. It seems that you are doing any unique trick. Also,
    The contents are masterwork. you have done a fantastic task on this
    topic!

    My site – robot piscine dolphin

    Reply
  24. مزون لباس

    very good

    Reply
  25. خرید vpn

    very good

    Reply
  26. google 2

    thank you for the post scott.

    Reply
  27. هنرمندان

    very nice post

    Reply
  28. اقتصاد ايران

    excellent

    Reply
  29. ekrishinaip.in

    ffmpdfsd; f[pk[ep

    Reply
  30. ekrishinaip.in

    jkkjkjbhu b fcdxeesrs

    Reply
  31. DiegoILosada

    What’s up everyone, it’s my first pay a visit at this website, and article is really fruitful in favor of me, keep up posting
    these types of posts.

    Reply
  32. ورودی مستقیم از گوگل | تبلیغات گوگل

    thanksssssss

    Reply
  33. سئو سایت| سئو وب سایت|بهینه سازی سایت

    besttttttttttttttttt

    Reply
  34. Moka Design UK - SEO Manchester

    Really nice article.

    Reply
  35. میز کانتر

    good thanks for sharing this post

    Reply
  36. قیمت کفسابی

    در سال های اخیر و با گسترش استفاده از انواع مختلف کفپوش ها، استفاده از موکت برای مفروش کردن کل کف ساختمان به شدت کاهش یافته است. کفپوش های مختلف دارای زیبایی خاصی هستند که همین موضوع بر گسترش آن ها افزوده است؛ اما یکی از دلایلی که باعث شده تا کفپوش ها علاقه مندان بیشتری را به خود جذب کنند، سادگی تمیزکاری آنها است. کفسابی و تمیزکاری این نوع کفپوش ها به سادگی هرچه تمام تر انجام می شود.

    در واقع کفسابی کمک می کند تا خیلی سریع و ساده و بدون هیچگونه مشکلی کف یک ساختمان تمیز شود. دیگر نیاز به مصرف بیش از اندازه آب و یا استفاده از تجهیزات مختلف همراه با صرف انرژی بسیار زیاد نیست. همچنین تمامی خدمات مربوط به شستن کف ساختمان در محل و با دستگاه های پیشرفته انجام می شود که خود از مزیت های آن محسوب می گردد.

    Reply
  37. Aluminum Sheet

    I have been using vmware recently and it is very easy to use.

    Reply
  38. لپ تاپ استوک

    لپ تاپ استوک

    Reply
  39. لپ تاپ استوک

    لپ تاپ استوک

    Reply

Leave a Reply

Your email address will not be published. Required fields are marked *

*