Home > Blogs > VMware vFabric Blog > Tag Archives: scale

Tag Archives: scale

How fast is a Rabbit? Basic RabbitMQ Performance Benchmarks

One of the greatest things about RabbitMQ is the community that surrounds it. With open source at its roots, people come together to share their code, their knowledge and their stories of how they’ve deployed it in their projects. At a recent meetup near Nice, France, database engineer Adina Mihailescu shared a presentation on choosing messaging systems. Supported by Murial Salvan’s benchmark comparing ActiveMQ, RabbitMQ, HornetQ, Apollo, QPID, and ZeroMQ, they shared some interesting performance comparisons that we’d like to share with you.

In a single laptop benchmark, Salvan ran four different scenarios in order to obtain some insight on performance of the default setups for these messaging solutions. Each test had 1 process dedicated to enqueuing and another dedicated to dequeuing. The message volume and size ranged from 200 to 20,000 to 200,000 messages and 32 to 1024 to 32768 bytes. Both persistent and transient queues and messages were used. Continue reading

Banks Are Breaking Away From Mainframes to Big, Fast Data Grids

The world’s largest banks have historically relied on mainframes to manage all their transactions and the related cash and profit. In mainframe terms, hundreds of thousands of MIPS are used to keep the mainframe running these transactions, and the cost per MIP can make mainframes extremely expensive to operate. For example, Sears was seeing the overall cost per MIP at $3000-$7000 per year and didn’t see that as a cost-effective way to compete with Amazon. While the price of MIPS has continued to improve, mainframes can also face pure capacity issues.

In today’s world of financial regulations, risk, and compliance, the entire history of all transactions must be captured, stored, and available to report on or search both immediately and over time. This way, banks can meet audit requirements and allow for scenarios like a customer service call that results in an agent search for the transaction history leading up to a customer’s current account balance. The volume of information created across checking, savings, credit card, insurance, and other financial products is tremendous—it’s large enough to bring a mainframe to its knees. Continue reading

How Indeed.com Handles 35 Million Job Postings Per Day Using RabbitMQ

Indeed.com is hosting a tech talk that will be a case study of how they scaled their aggregation engine to handle 35 million job postings per day using RabbitMQ. The talk will be hosted at Indeed’s engineering site in Austin, TX at 7 pm on March 27th.

For those of you not familiar with how large Indeed is, it is interesting to note that the job search company Indeed.com is one of the largest web sites in the world. According to Alexa.com, Indeed is currently the #224th biggest website in the world, and in cities like Atlanta and Chicago, it’s the 55th most popular website overall. According to research by SilkRoad, 2 out of every 5 hires came via Indeed (based on data from 150,000 hires).

As expected, the engineering team behind this large-scale application needs to support some very large scale numbers. In a recent post on their company blog, the Indeed team shared just how big those numbers are:

  • More than 100 million monthly unique visitors
  • More than 3 billion searches per month
  • More than 1000 searches per second
  • 50 country-specific sites in 26 languages

The scale of their application, both in terms of processing throughput and geographic diversity, means that the team relies on a messaging layer powered by RabbitMQ. Continue reading

3 Key Performance and Scale Improvements with VMware vFabric Postgres 9.2

Recently, vFabric Postgres 9.2 launched with additional cloud computing capabilities like elastic memory management. Some of the most compelling new features are performance-related and take linear scaling to new levels.

This article will cover 3 key improvements as listed below:

  • 4x Improvement with vertical linear scaling for reads
  • 2x Improved write efficiency for write ahead logs
  • Index Only Scans and More

4x Improvement with vertical linear scaling for reads

Modern websites are almost all database driven. When consumers browse online retailer catalogs, 99% of the load is reads and 1% of the load is updates to the data on the tables. Even in highly updated websites, the grand majority of load is from reads. In these high-read usage scenarios, the database needs to handle a high read load on certain tables compared to the other tables in the database. We’ve seen this behavior drive enhancements within databases. For example, many application designs started putting a caching mechanism in their application to limit the database hits. Continue reading

Advanced GemFire + Lucene + Spring Data for Text Searching

In this article, we’ll talk about how to integrate the Lucene text searching solution using Spring Data and GemFire to provide a flexible, parallel fast search engine. By combining the two independent products we can leverage each product to its fullest capability. The end result provides an elastic search capability with the in memory data speeds of a distributed cache platform and high availability.

Motivation—Why Combine These?

The motivation of the project was to provide an alternative search capability for GemFire while providing users a natural method to define searchable domain object attributes. Performance was also a key driver to ensure constant search performance irrespective of scale. The solution outlined below provides a baseline approach for developers to build upon. Continue reading

3 Game Changing Capabilities in SQLFire

According to one of our partners, vFabric SQLFire is a product he wishes more customers would use.

“SQLFire is a game-changer.  I think many companies underestimate the value of scaling the data later horizontally.  Every project I propose has a business case, and I see a tremendous amount of value being unlocked with this product—not just for the CIO or CTO’s agenda, but for the CFO and CEO.  Then, you add the fact that the whole application stack is virtualized and has solid integrations. It’s a simple story, the product allows you to add a lot of value in a really cost effective way.”

What makes SQLFire such a game-changer?

In this article, we’ll talk more about three game-changing capabilities: server groups, partitioning, and redundancy.

If you haven’t been following our stories on SQLFire, see the end of this article for a  list of posts and key capabilities that help explain how transformative SQLFire can be to your data management strategies. Continue reading

Building a Multi-Tenant Development Mindset in the Travel Software Industry

The travel industry has been a technology innovator for decades.

But how do these tech innovators use a cloud application platform like vFabric?

In this article, we get a real-world, inside perspective from a cloud architect who designs and leads development teams for airline check-in and baggage software and cloud-based services. We will dive into his requirements and approaches to cloud-centric devops tools that keep systems running in high performance environments.

Travel companies use technology everywhere they can. For example, their technology lets us buy tickets over the web, check in via self-serve kiosks, and use iPhones or Androids as boarding passes. It wasn’t long ago that these capabilities didn’t exist, but innovative companies like American Airlines use technology everywhere to differentiate their company and connect with customers. For example:

  • AA.com gets 1.6 million visits per day.
  • Their mobile app has over 3 million downloads:
    • It’s available on iPhone, iPad, iPod Touch, Android, Blackberry, Nook, Windows Phone 7, and Amazon Kindle Fire.
    • Mobile boarding passes are available in 77 cities.
    • The apps include wifi flight search, flight status notification, and more.
  • Over 800 kiosks allow customers to check in while people can also check in from the website. The kiosks provide passport, credit card, and barcode scanning.
  • The list continues with on-board purchases, advanced loyalty programs, RFID, and more. Continue reading

How We Improved APM’s Monitoring Capacity by 500% with Gemfire, RabbitMQ, vPostgres, and a Few Scalability Design Patterns

Earlier this week, we announced the general availability of a major upgrade to vFabric Application Performance Manager (APM). This release started one year ago, after we released the first version of the product to market. When we started work on this release, we knew we would need to invest heavily in scalability. APM is designed to help simplify monitoring and management for highly dynamic, large web applications living in the cloud. To succeed, we needed to make sure our product could scale gracefully with our customers. So, we set out with a challenging goal to increase the capacity of APM by a factor of 5.

Transforming a complex product such as APM into a more scalable architecture is not an easy task, let alone doing so in a single release. For this reason we’ve started by modifying the architecture in steps, starting with local improvements inside our virtual appliance, (available in the APM 5.0 release) and moving towards a horizontal scale solution in future releases. Continue reading

Map-Reduce Style: Data Aware Distributed Querying in vFabric GemFire

vFabric GemFire ArchitectureBig, fast data is powering some of the most interesting computing opportunities in today’s market. But in order to get there, we need to change our approach to the data tier. Enterprises are trying to move from costly mainframe architectures to virtualized datacenters and utilize commodity hardware more efficiently. With the data tier, this means an architecture that scales horizontally by adding more commodity-based computing and storage at runtime.

To scale the data tier horizontally, companies use systems like vFabric GemFire, a distributed data system that is designed to specifically accommodate large data sets across commodity hardware nodes. In GemFire, data is spread across members of a cluster with members referred to as “nodes,” and the distribution of data across those nodes is called “partitioning.” vFabric GemFire then allows developers to query the data that resides across many nodes while retaining core values of very high performance at scale. How? In short, the answer is “Data Aware Querying” – a query API that allows a query to execute on selective nodes instead of all nodes (i.e. execute in a map-reduce style).

Continue reading

Why is vFabric on the CIO Agenda? Trends and Economics

So, why is vFabric on the CIO Agenda? In short, technology trends and basic economics.

In this article, we outline, provide key highlights, share the slides, and link to an on-demand, CIO.com webinar titled, “Your business is now a software business. Now what?” In the recording, Tom Schmidt, Managing Editor at CIO.com, targets several questions to Al Sargent, Group Manager, VMware Cloud and Application Services, about how vFabric fits into the CIO agenda.

The webinar covers the following four topics, and a short summary is below:

  1. Why every business is a software business
  2. The clear trends with VMware vFabric customers and prospects
  3. Cloud Scale and Economics
  4. Pricing Comparisons of vFabric to Competitors Continue reading