Home > Blogs > VMware vFabric Blog > Tag Archives: RabbitMQ

Tag Archives: RabbitMQ

New RabbitMQ 3.1.0 Release Available

RabbitMQ 3.1.0 is now available for immediate download.

Announced this morning on the new Pivotal blog, where RabbitMQ now resides, this version includes enhancements to garbage collection, consumption, requeuing, memory use, and dead lettering.

For those on Mac OS X, there is a newly packaged, standalone release of RabbitMQ that doesn’t require a separate Erlang install.

Some key, new capabilities include eager synchronisation of mirror queue slaves, automatic cluster partition healing, and improved statistics (including charts) in the management plugin. There are also many enhancements and bug fixes to the server, Java client, Erlang client, and a number of other plugins, including federation, old-federation, shovel, Web-STOMP, STOMP, and MQTT plugins, as well as the consistent hash exchange.

RabbitMQ’s blog post on the topic shares screenshots of several new features like the ones for new charts and filters below:

Read More:

How Instagram Feeds Work: Celery and RabbitMQ

Instagram is one of the poster children for social media site successes. Founded in 2010, the photo sharing site now supports upwards of 90 million active photo-sharing users. As with every social media site, part of the fun is that photos and comments appear instantly so your friends can engage while the moment is hot.  Recently, at PyCon 2013 last month, Instagram engineer Rick Branson shared how Instagram needed to transform how these photos and comments showed up in feeds as they scaled from a few thousand tasks a day to hundreds of millions.

Rick started off his talk demonstrating how traditional database approaches break, calling them the “naïve approach”. In this approach, when working to display a user feed, the application would directly fetch all the photos that the user followed from a single, monolithic data store, sort them by creation time and then only display the latest 10:

SELECT * FROM photos
WHERE author_id IN
(SELECT target_id FROM following
WHERE source_id = %(user_id)d)
ORDER BY creation_time DESC
LIMIT 10;

Instead, Instagram chose to follow a modern distributed data strategy that will allow them to scale nearly linearly. Continue reading

Scaling Real-Time Comments @ HuffPost Live with RabbitMQ

Last week, one of the engineering leaders from Huffingtonpost.com presented at the Erlang Factory, a 2-day conference held in San Francisco with 50+ speakers across 8 tracks (not the cassette kind). Among the presenters was, Adam Denenberg, VP of Engineering at the Huffington Post and HuffPost Live. Besides leading the engineering teams for Huffingtonpost.com and HuffPost Live, Adam is also responsible for community products and advertising platforms.

By using RabbitMQ, SockJS, Cowboy, Erlang, and more, his team built a highly scalable, customizable solution for real-time comments based on websockets.  His talk, Realtime Web @ HuffingtonPost, was for developers of real-time, Erlang-based solutions—he covered architecture, lessons learned, pitfalls, and future improvements. The presentation objectives included:

  • Why they went with Erlang and SockJS for the commenting platform
  • How they integrated RabbitMQ routing to power the subscription architecture
  • How they managed subscriptions for real time channels
  • Their plans to extend the framework into an open source solution
  • Where they want other mechanisms for publishing comments outside of RabbitMQ Continue reading

How Indeed.com Handles 35 Million Job Postings Per Day Using RabbitMQ

Indeed.com is hosting a tech talk that will be a case study of how they scaled their aggregation engine to handle 35 million job postings per day using RabbitMQ. The talk will be hosted at Indeed’s engineering site in Austin, TX at 7 pm on March 27th.

For those of you not familiar with how large Indeed is, it is interesting to note that the job search company Indeed.com is one of the largest web sites in the world. According to Alexa.com, Indeed is currently the #224th biggest website in the world, and in cities like Atlanta and Chicago, it’s the 55th most popular website overall. According to research by SilkRoad, 2 out of every 5 hires came via Indeed (based on data from 150,000 hires).

As expected, the engineering team behind this large-scale application needs to support some very large scale numbers. In a recent post on their company blog, the Indeed team shared just how big those numbers are:

  • More than 100 million monthly unique visitors
  • More than 3 billion searches per month
  • More than 1000 searches per second
  • 50 country-specific sites in 26 languages

The scale of their application, both in terms of processing throughput and geographic diversity, means that the team relies on a messaging layer powered by RabbitMQ. Continue reading

HOWTO: Deploying RabbitMQ Simulator on Cloud Foundry

Last week, we published the new RabbitMQ Simulator. I’m pleased to see how well it has been received by customers, with several very large scale RabbitMQ customers already incorporating it into their training and testing regimens. VMware employees in the field have been enjoying getting familiar with the tool as well and are using it to advance their own and their customer’s understanding of how RabbitMQ works and how to best design a messaging architecture. One sales engineer, Chris DeLashmutt even wrote a blog this week describing how to deploy the new RabbitMQ Simulator on Cloud Foundry.

For users new to RabbitMQ, certainly testing this out on Cloud Foundry can be a huge accelerator, as you can get your instance up and running for free in just a matter of minutes. Below is a summary of Chris’ instructions to deploy this simulator on Cloud Foundry, but also, if you want to shortcut the entire process, he invites you to access his deployed instance at http://rabbitmq-sim.cloudfoundry.com. Continue reading

Introducing the RabbitMQ Simulator, Video, & Open Source Bits

In this blog post I’d like to introduce you to one of our latest developments: the RabbitMQ Simulator.

This simulator was born out of a need—we wanted a better tool to teach RabbitMQ concepts to people that were new to the message broker and its AMQP protocol.

If you know me, you are probably aware that I have given quite a few RabbitMQ presentations over the year at several technical conferences. After explaining AMQP concepts many times with static images, I decided the time had come for a better tool—something more graphic and visual, something with more life and motion. So, I started working on this RabbitMQ Simulator project. Continue reading

Choosing Your Messaging Protocol: AMQP, MQTT, or STOMP

One of the most common questions I’m asked to cover when I discuss software architecture topics is the difference between the various application messaging protocols that exist today—issues like how and why the protocols came about, and which one should be used in a particular application.

Their question is valid.

Today, application architects need to use a messaging broker to speed and scale their applications, particularly in the cloud. Even once you select your messaging middleware application, application developers need to then select the protocol. Understanding the subtle differences between them can be difficult.

Today, we will consider three of the most common and popular TCP/IP-based messaging protocols, and provide a quick summary on the advantages of each: AMQP, MQTT and STOMP. Before we go on, I should also point out that all three of these protocols are supported in RabbitMQ version 3.0—something we will use as an example and come back to later.

So, in alphabetical order…

AMQP in a Nutshell Continue reading

Q&A with Shay Banon: 10 “Bonsai Cool” Things About elasticsearch

We are very fortunate to post an interview with Shay Banon, the founder of elasticsearch. Elasticsearch is technology that is very popular among some of the coolest companies on the web today, including  SoundCloud, StumbleUpon, Mozilla and Klout. These companies use elasticsearch to help them deploy powerful search capabilities in their applications that are easy to set up, scalable and built for the cloud.  In this interview, we get to learn all kinds of cool things:

  1. How Shay got into search
  2. How he came up with the idea for elasticsearch
  3. Why elastic search is different than other OSS search projects
  4. Example elasticsearch users like Foursquare, Brewster, GitHub, Sony, and Klout
  5. About the elasticsearch architecture for big data
  6. The strategy behind JSON over HTTP for search
  7. Connecting elasticsearch with RabbitMQ
  8. Connecting elasticsearch with Spring
  9. Connecting elasticsearch with GemFire
  10. Running elasticsearch on virtualized infrastructure

Without further ado, here is the interview.

Q1. So, how did you end up getting into search?
About 10 years ago, I moved from Israel to London because my wife was going to study to be a chef at the Cordon Bleu. I had no job. I was in a new country. I was unemployed. So, I started to get into the latest, cool, new technologies. Continue reading

6 Simple Steps: Installing vFabric Administration Server (VAS) and Provisioning tc Server across 3 Nodes

As a suite of products, vFabric offers a single SKU model that includes the entire suite. Besides offering an economically beneficial licensing model that is flexible, based on average use, and hardware independent, the vFabric team has developed the vFabric Administration Server (VAS) to help make installation, configuration, and operations easier.

If you missed a previous article explaining how VAS work and what it can do for you, VAS is based on a single system image design (SSI) and allows you to define a group of nodes where each operation on the group is applied to each node who is a member of the group. This approach provides time savings and reduces errors by keeping systems more consistent, allowing easier ways to scale, and simplifying administration. VAS also provides a robust set of REST APIs for virtually all of its functionality. In addition to vFabric tc Server, VAS can manage vFabric RabbitMQ, vFabric GemFire, vFabric SQLFire, and vFabric Web Server. VAS 1.1.1 was recently released with new features and many bug fixes.

In this article (and demonstration further below), we will show you six steps that give you an idea of how easy it is to provision using VAS. We will show you how to install VAS and use it to provision vFabric tc Server across three nodes along with a WAR file. The explanation below refers to examples from RubyGems.org and GitHub/vFabric/VAS-Ruby-API along with the latest VAS documentation.

Continue reading

Announcing the Availability of vFabric Data Director 2.5, GemFire 7, EM4J 1.2, and More

Application developers and data management teams continue to look for ways to modernize legacy apps, manage costs more effectively, build new apps on robust application platforms, and solve big data problems. These are some of the key reasons why vFabric is on the CIO (or CTO) agenda. With several new product releases in the vFabric Suite, VMware continues to provide a best-in-class application platform and help customers solve their top application development and data management problems.

vFabric Data Director 2.5

Database as a Service (DBaaS) helps companies virtualize data engines and automate management while getting a handle on the costs and compliance issues related to data sprawl. In the newest version of Data Director, several new data engines are supported (in addition to Oracle and Postgres) along with other new capabilities:

  • Support for Microsoft SQL Server 2008 R2 and SQL Server 2012
  • Support for Hadoop deployment, management, and monitoring across all major distributions through Project Serengeti
  • Enhanced automation of Oracle and SQL Server template creation
  • Broad support for Red Hat Enterprise Linux (RHEL) and Oracle Linux
  • Enhanced Oracle database ingestion, including ingestion to a point-in-time and more
  • Support for static IP database virtual machines (DBVMs)
  • Express set-up for development or experimentation

Continue reading