One of the greatest things about RabbitMQ is the community that surrounds it. With open source at its roots, people come together to share their code, their knowledge and their stories of how they’ve deployed it in their projects. At a recent meetup near Nice, France, database engineer Adina Mihailescu shared a presentation on choosing messaging systems. Supported by Murial Salvan’s benchmark comparing ActiveMQ, RabbitMQ, HornetQ, Apollo, QPID, and ZeroMQ, they shared some interesting performance comparisons that we’d like to share with you.
In a single laptop benchmark, Salvan ran four different scenarios in order to obtain some insight on performance of the default setups for these messaging solutions. Each test had 1 process dedicated to enqueuing and another dedicated to dequeuing. The message volume and size ranged from 200 to 20,000 to 200,000 messages and 32 to 1024 to 32768 bytes. Both persistent and transient queues and messages were used. Continue reading →
After PostgreSQL 9.2 was released, users that relied on PostgreSQL for scale, may have noticed a performance hit. In fact, the PostgreSQL community alongside the VMware vFabric Postgres team, was able to prove that the new version demonstrated a 10% performance hit over version 9.1. As part of the VMware Postgres team, we wanted to fix this problem for our own distribution, but as mentioned in previous posts, we also wanted to contribute our fixes back to the common core. This post provides additional detail on how this problem was identified and how we worked with the open source PostgreSQL community to restore performance.
Background on the Performance Issue in PostgreSQL 9.2
Last year, during routine regression testing of vFabric Postgres, we found that PostgreSQL 9.2, the latest major release of PostgreSQL, demonstrated a significant performance regression from version 9.1. Using DBT-2, an open-source and fair-use implementation of TPC-C benchmark , we noticed a 10% performance degradation, which we then reported to the community .
To troubleshoot the problem we used git bisect to find the type of commit that caused the performance problem and cross-examined the statistical profiles using oprofile. As it turns out, the regression was caused by a commit that changed the way memory was allocated when SPI queries were executed. The commit was intended to reduce the number of allocations for queries using a cached plan at the cost of more logistics work. However, according to the DBT-2 test, we could see that this tradeoff was unfavorable for dynamic queries. So to fix it, we would need reintroduce the original tradeoff on its intended queries using conditions .
We proposed the fix to the wider PostgreSQL community and the ensuing discussion led to a refined resolution which was implemented in a patch . This patch has been back-ported to the latest PostgreSQL 9.2.3 release and is included in the latest vFabric Postgres release . Continue reading →
Recently, vFabric Postgres 9.2 launched with additional cloud computing capabilities like elastic memory management. Some of the most compelling new features are performance-related and take linear scaling to new levels.
This article will cover 3 key improvements as listed below:
4x Improvement with vertical linear scaling for reads
2x Improved write efficiency for write ahead logs
Index Only Scans and More
4x Improvement with vertical linear scaling for reads
Modern websites are almost all database driven. When consumers browse online retailer catalogs, 99% of the load is reads and 1% of the load is updates to the data on the tables. Even in highly updated websites, the grand majority of load is from reads. In these high-read usage scenarios, the database needs to handle a high read load on certain tables compared to the other tables in the database. We’ve seen this behavior drive enhancements within databases. For example, many application designs started putting a caching mechanism in their application to limit the database hits. Continue reading →
In this article, we’ll talk about how to integrate the Lucene text searching solution using Spring Data and GemFire to provide a flexible, parallel fast search engine. By combining the two independent products we can leverage each product to its fullest capability. The end result provides an elastic search capability with the in memory data speeds of a distributed cache platform and high availability.
Motivation—Why Combine These?
The motivation of the project was to provide an alternative search capability for GemFire while providing users a natural method to define searchable domain object attributes. Performance was also a key driver to ensure constant search performance irrespective of scale. The solution outlined below provides a baseline approach for developers to build upon. Continue reading →
Apache Derby is used for its RDBMS components, JDBC driver, query engine, and network server.
The partitioning technology of GemFire is used to implement horizontal partitioning features of vFabric SQLFire.
vFabric SQLFire specifically enhances the Apache Derby components, such as the query engine, the SQL interface, data persistence, and data eviction, as well as adding additional components like SQL commands, stored procedures, system tables, functions, persistence disk stores, listeners, and locators, to operate a highly distributed and fault tolerant data management cluster.
Today, we are pleased to have a guest blogger from a VMware customer share with us their story of how RabbitMQ transformed their business by “solving some really interesting problems”. The following is sent courtesy of Pablo Molnar of MercadoLibre:
If you haven’t heard of MercadoLibre (NASDAQ: MELI), we are the largest e-commerce ecosystem in Latin America. Our website offers a wide range of services to sellers and buyers throughout the region including marketplace, payments, advertising, and e-building solutions. Our products are present in over 14 countries, and the company is ranked as 8th largest online retailer in the world. We were also on Fortune’s list of the fastest growing companies in 2012, and we use RabbitMQ to solve some interesting problems.
About Our Technology Stack and How RabbitMQ Helps
In terms of technology infrastructure, MercadoLibre is fully committed to the open source development model. Most of our apps are primarily written in Grails, Groovy, and NodeJS, but we don’t stick to any language or framework. We entrust tool selection responsibilities to the Software Engineers on each team. Almost all applications are hosted by our in-house cloud computing provisioning system and implemented via OpenStack with more than +7000 virtual instances at the moment. Also, we have successfully launched applications using emerging storage solutions like Redis and MongoDB. With an average of 20 million requests per minute and 4GB bandwidth per second, our traffic management layer is crucial and most of the routing rules job is done by Nginx proxy servers. Our labs department includes a huge Apache Hadoop cluster to perform complex analytical queries, and we are experimenting with real-time data processing using Apache Kafka and Storm.
Next year is going to be even bigger with the Pivotal Initiative where several of the products covered on this blog will be following the new venture. This is still in the planning stages, so we will be expecting to share with you the plans for our products alongside the formal communications from each of the companies involved. (Sorry — no extra information is available right now)
This post is meant to augment the knowledge base article, KB 2033940, published back in August.
Planning the Upgrade
We know platform changes can make an upgrade more difficult and certainly raise eyebrows. So, we’ve taken measures to help make the migration as seamless and simple as possible. So far, the cases we’ve seen take about an hour. As with any data migration, the greater the volume of database records, the longer it can take. Continue reading →
RSA is in the business of stopping banks and their customers from being robbed (among other things). Their technology has protected people, businesses, and financial institutions from online fraud for almost 20 years. Their Adaptive Authentication solution is deployed at over 8000 companies, used by over 200 million people, and has protected over 20 billion transactions to date. To jump on the “everything as a service” bandwagon, Adaptive Authentication is literally embarking on a project to “Stop Bank Robbers as a Service.”
Earlier this week, we announced the general availability of a major upgrade to vFabric Application Performance Manager (APM). This release started one year ago, after we released the first version of the product to market. When we started work on this release, we knew we would need to invest heavily in scalability. APM is designed to help simplify monitoring and management for highly dynamic, large web applications living in the cloud. To succeed, we needed to make sure our product could scale gracefully with our customers. So, we set out with a challenging goal to increase the capacity of APM by a factor of 5.
Transforming a complex product such as APM into a more scalable architecture is not an easy task, let alone doing so in a single release. For this reason we’ve started by modifying the architecture in steps, starting with local improvements inside our virtual appliance, (available in the APM 5.0 release) and moving towards a horizontal scale solution in future releases. Continue reading →