After PostgreSQL 9.2 was released, users that relied on PostgreSQL for scale, may have noticed a performance hit. In fact, the PostgreSQL community alongside the VMware vFabric Postgres team, was able to prove that the new version demonstrated a 10% performance hit over version 9.1. As part of the VMware Postgres team, we wanted to fix this problem for our own distribution, but as mentioned in previous posts, we also wanted to contribute our fixes back to the common core. This post provides additional detail on how this problem was identified and how we worked with the open source PostgreSQL community to restore performance.
Background on the Performance Issue in PostgreSQL 9.2
Last year, during routine regression testing of vFabric Postgres, we found that PostgreSQL 9.2, the latest major release of PostgreSQL, demonstrated a significant performance regression from version 9.1. Using DBT-2, an open-source and fair-use implementation of TPC-C benchmark , we noticed a 10% performance degradation, which we then reported to the community .
To troubleshoot the problem we used git bisect to find the type of commit that caused the performance problem and cross-examined the statistical profiles using oprofile. As it turns out, the regression was caused by a commit that changed the way memory was allocated when SPI queries were executed. The commit was intended to reduce the number of allocations for queries using a cached plan at the cost of more logistics work. However, according to the DBT-2 test, we could see that this tradeoff was unfavorable for dynamic queries. So to fix it, we would need reintroduce the original tradeoff on its intended queries using conditions .
We proposed the fix to the wider PostgreSQL community and the ensuing discussion led to a refined resolution which was implemented in a patch . This patch has been back-ported to the latest PostgreSQL 9.2.3 release and is included in the latest vFabric Postgres release . Continue reading →
In this article (and demonstration further below), we will show you six steps that give you an idea of how easy it is to provision using VAS. We will show you how to install VAS and use it to provision vFabric tc Server across three nodes along with a WAR file. The explanation below refers to examples from RubyGems.org and GitHub/vFabric/VAS-Ruby-API along with the latest VAS documentation.
Application and operations teams sometimes reach a point where they must upgrade the database. Whether it’s due to data growth, lack of throughput, too much downtime, the need to share data globally, adding ETLs, or otherwise, it’s never a small project. Since these projects are expensive, any recommendation requires a solid justification. This article a) characterizes 3 signs where traditional databases hit a wall, b) explains how vFabric SQLFire provides an advantage over traditional databases in each case, and c) should help you make a case for moving towards an in-memory, distributed data grid based on SQL.
For those of us tasked with upgrading (or architecting) the data layer, we all go through similar steps. We build a project plan, make projections and sizing estimates, perform architecture and code reviews, create configuration checklists, provide hardware budgets and plans, talk to vendors about options, and more. Then, we work to plan the deployment with the least downtime, procure hardware and software, test different data load times, evaluate project risks, develop back-up plans, prepare communications to users about downtime, etc. You know the drill. These projects can take months and consume a fair amount of internal resources or consulting dollars. If you are starting or working on one of these types of projects with a traditional database architecture in mind, are you considering these 3 signs as you consider your options? Continue reading →
vFabric GemFire is a sophisticated product in a complex problem space: data management in distributed systems. In order to help our users get the most out of GemFire, we are starting a “cookbook” series, which will provide tried and tested recipes that we hope every GemFire user will find useful.
Our first topic is the Visual Statistics Display (VSD). VSD is a visual tool for analyzing GemFire statistics. It reads GemFire statistics from special statistics archive files created by GemFire, and renders their graphs for analysis. It is not a real-time online monitoring tool, such as vFabric Hyperic, so it does not have the real-time monitoring and alerting capabilities that they have. On the other hand, it is the most powerful tool for examining the state of a vFabric GemFire system, as it provides access to all the statistics collected by GemFire. No real-time monitoring tool can do that, as the amount of statistics that GemFire collects is prohibitive for real-time collection in a distributed system.
Before we delve into the differences, it is important to understand the big picture. When my team started developing Application Director, open source configuration management tools like Puppet, Chef, CF Engine already existed. They were doing a great job of keeping complex applications up to date with the latest code and configurations while also helping IT to automate inventory management – a big and valuable task in any datacenter. Continue reading →
At VMworld-2012 San Francisco, the session APP-CAP1426 – The Benefits of Virtualization for Middleware was greatly attended, and we want to thank all of the attendees that helped us score 4.5 out of 5 in our survey results. Because of this, the session is going to be presented at VMworld-2012 Barcelona, and we are posting related information here in this article. Before reading this article, you might want to take a look at the related blog post we released before VMworld-2012 San Francisco.
NOTE: Just like in VMworld-2012 San Francisco, at Barcelona we will raffle copies of my book the Enterprise Java Applications Architecture on VMware
This year at the session, we went deep into tuning large-scale middleware and the discussion around JVM tuning was well received. Hence as a follow-up, I wanted to share some of my more recent research, which will be discussed at VMworld-2012 Barcelona. This focuses on tuning in-memory data management systems such as vFabric SQLFire.
The article below covers:
An Overview of GC Tuning
Parallel GC in Young Generation and CMS in Old Generation
GC Tuning Recipe
Step A – Young Generation Tuning
Step B – Old Generation Tuning
Step C – Survivor Spaces Tuning
JVM and GC Best Practices for vFabric SQLFire Members
To put it simply, vFabric Application Director (AppD) automates deployments. That might sound trivial at first, but it can automate an entire deployment for all the tiers of complex application, in the right order, with the right configurations, across multiple cloud and/or virtual infrastructures, and without ANY human intervention. If you are just hearing about AppD, you can learn about it quickly here:
One of the most common misconceptions about the level of high availability provided by SQLFire is the client configuration. A lot of times, when people see the client connection string defined with a single IP, they assume that to mean that client will only communicate with that one SQLFire cluster host and deem that to be a single point of failure in the SQLFire data grid.
In reality however, SQLFire is based on multi-faced shared nothing architecture. One of its tenants is transparent failover at the protocol level.
In the 5.1 release of the vFabric Suite, we’ve added a new tool – vFabric Administration Server (VAS). VAS makes it significantly easier to administer some of the components of the vFabric Suite across small deployments with a handful of machines or something much larger. The first release of the Administration Server provides support for administering GemFire, RabbitMQ, and tc Server via a REST API. Some examples of VAS’s capabilities include installing GemFire, RabbitMQ, or tc Server, starting and stopping them, managing their configuration files, and deploying web applications to tc Server.
While there are more capabilities, here are three reasons why you need vFabric Administration Server:
Improved Deployment Consistency and Time Savings
Quick and Easy Scale Up and Down
Simplified and Centralized Administration
Improved Deployment Consistency and Time Savings
A common problem encountered by our customers is ensuring that their deployments are consistent when spread across multiple machines. It’s all too easy for a deployment that started out with identically configured machines to slowly diverge as changes are made inconsistently across the machines. vFabric Administration Server addresses this by building upon the concept of a single system image, i.e. a collection of systems that are used as a single system. In VAS, a single system image is known as a group, with each system in the group being called a node.
As part of the larger vFabric 5.1 release this week, VMware’s Hyperic, the web infrastructure monitoring software bundled with the vFabric Suite, delivered on an upgrade to managing Apache Tomcat, the world’s most popular application server. As summarized in the blog post today, Hyperic upgraded support for Tomcat in three major ways:
1. Support for Apache Tomcat 7. Version 7 is gaining adoption in the marketplace, including with enterprise users. To further support this, the latest release for Hyperic adds support for Apache Tomcat 7, the latest stable release published by the Apache Software Foundation.
2. Improve Internal Use of Tomcat. Hyperic relies on Tomcat as its internal application server. The new version provides two new server configuration properties to better manage the how many threads are available to the Hyperic Server’s internal Tomcat server. The new parameters are set for deployments of less than 50 platforms, and are configurable for larger deployments to increase performance. For more details, see the complete blog post.