Home > Blogs > VMware vFabric Blog


.NET + RabbitMQ: Scales to 100s of Millions of Passenger Messages at 15below

This week we are excited to have a guest post on vFabric RabbitMQ from Mike Hadlow, enterprise Microsoft.NET developer and architect with 15below.com. Mike covers:

  • Their Architecture Before RabbitMQ
  • Why they went with RabbitMQ
  • Their Infrastructure and Development Environment
  • How RabbitMQ fits in their Software Architecture

===============================

In this post, I want to share our experiences of using RabbitMQ at 15below.

15below is based in Brighton, UK. We provide messaging and integration services for the travel industry. Our clients include Ryan Air, Qantas, JetBlue, Thomas Cook, and around 30 other airline and rail customers. 15below sends hundreds of millions of transactional notifications every year to its customer’s passengers over a wide range of channels including email, SMS, push, and voice.

RabbitMQ has helped us to significantly simplify and stabilise our software. It’s one of those solutions that you install, configure, and then really don’t have to worry about. In over a year of production, we’ve found it to be extremely stable without a single production issue.

The Architecture Before RabbitMQ

Prior to introducing RabbitMQ within our applications, we used SQL Server as a queuing mechanism. Each task was represented by a row in a workflow table. Each process in the workflow polled the table looking for rows that matched its status, processed the rows in a batch, and then updated the rows’ status field for the next process to pick up. Each step in the process was hosted by an application service that implemented its own threading model, often using a different approach to all the other services. This created highly coupled software with workflow steps and infrastructure concerns such as threading and load balancing mixed together with business logic. We also discovered that a relational database is not a natural fit for a queuing system. The contention on the workflow tables is high and locking issues arise from constant inserts, selects, and updates. Deleting completed items creates problems on highly indexed tables, and we faced considerable problems with continuously growing tables.

Why we went with RabbitMQ

RabbitMQ provides a number of features that helped us overcome these problems. First, it is designed from the beginning as a high-performance messaging platform. It easily outperformed our SQL Server based solution with none of the locking or clean-up problems. RabbitMQ’s event-oriented messaging model also takes away much of the need for complex, multi-threaded batch processing code that was previously a cause of instability in our systems.

We first introduced RabbitMQ about 18 months ago as the core infrastructure behind our latest Flight Status product. We wanted a high performance messaging product with a proven track record that supported common messaging patterns, such as publish/ subscribe and request/response. A further requirement was automatic work distribution and load balancing.

The need to support messaging patterns ruled out simple store-and-forward queues such as MSMQ and ActiveMQ. We were very impressed by ZeroMQ, but decided that we really needed the centralised manageability of a broker based product. This left RabbitMQ. Although support for AMQP, an open messaging standard, wasn’t in our list of requirements, it’s implementation by RabbitMQ made us more confident that we were choosing a sustainable strategy.

We are very much a Microsoft shop, so we had some initial concerns about RabbitMQ’s performance and stability on Windows. We were reassured by reading some accounts of RabbitMQ’s and indeed Erlang’s use on Windows by users with some very impressive load requirements. Subsequent experience has borne these reports out, and we have found RabbitMQ on Windows Server 2008 to be rock solid.

About our Infrastructure and Development Environment

Our infrastructure is heavily dependent on VMware products. All our servers run as virtual machines under VMware VSphere Enterprise Plus 5.1. Knowing that VMware is the company behind RabbitMQ gave us further confidence in choosing it as our messaging platform.

As a Microsoft shop, our development platform is .NET. Although VMware provide an AMQP C# client, it is a low-level API, not suitable for use by more junior developers. For this reason we created our own high-level .NET API for RabbitMQ that provides simple, single method access to common messaging patterns and does not require a deep knowledge of AMQP. This API is called EasyNetQ. We’ve open sourced it and, with over 3000 downloads, it is now the leading high-level API for RabbitMQ with .NET. You can find more information about it at EasyNetQ.com. We would recommend looking at it if you are a .NET shop using RabbitMQ.

How RabbitMQ fits in our Architecture

15below’s Flight-Status product provides real-time flight information to passengers and their family and friends. We interface with the airline’s real-time flight information stream generated from their operations system and provide a platform that allows them to apply complex business logic to this stream. We render output specific to each customer and communicate with the airline’s customers (i.e. you and I, the traveller) via a range of channels, including email, SMS, voice, and iOS/Android push. RabbitMQ allows us to build each piece – the client for the fight information stream, the message renderer, the sink channels and the business logic – as separate components that communicate using structured messages. Our architecture looks something like this:

The green boxes are our core product systems, and the blue boxes represent custom code that we write for each customer. A ‘customer saga’ is code that models a long- running business process and includes all the workflow logic for a particular customer’s flight information requirements. A ‘core product service’ is an independent service that implements a feature of our product. An example would be the service that takes flight information and combines it with a customer defined template to create an email that is sent to a passenger. Constructing services as independently deployable and runnable applications gives us great flexibility and scalability. If we need to scale up a particular component, we simply install more copies. RabbitMQ’s automatic work sharing feature means that we can do this without any reconfiguration of existing components. This architecture also makes it easy to test each application service in isolation since it’s simply a question of firing messages at the service and watching its response.

In conclusion, RabbitMQ has provided a rock solid piece of infrastructure. It has the features that allow us to significantly reduce the architectural complexity of our systems. We can now build fantastic software for our clients faster and more reliably. It scales to higher loads than our previous relational-database based systems and is more flexible in the face of changing customer requirements.

  About the Author: Mike Hadlow is software developer and architect with over 10 years of experience developing enterprise software on the Microsoft .NET platform. He writes a popular blog, Code Rant, and is the author of a number of open source projects including EasyNetQ and Suteki Shop. He lives in Brighton on the south coast of the UK with his wife, son and daughter. When not coding he likes to read history, draw, or pluck some lead breaks from his Fender Stratocaster.

3 thoughts on “.NET + RabbitMQ: Scales to 100s of Millions of Passenger Messages at 15below

  1. Pingback: Messaging Architecture: Using RabbitMQ at the World’s 8th Largest Retailer | VMware vFabric Blog - VMware Blogs

  2. Pingback: VMware vFabric Blog: Messaging Architecture: Using RabbitMQ at the World’s 8th Largest Retailer | Virtualization

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>