Home > Blogs > VMware vFabric Blog


.NET + RabbitMQ: Scales to 100s of Millions of Passenger Messages at 15below

This week we are excited to have a guest post on vFabric RabbitMQ from Mike Hadlow, enterprise Microsoft.NET developer and architect with 15below.com. Mike covers:

  • Their Architecture Before RabbitMQ
  • Why they went with RabbitMQ
  • Their Infrastructure and Development Environment
  • How RabbitMQ fits in their Software Architecture

===============================

In this post, I want to share our experiences of using RabbitMQ at 15below.

15below is based in Brighton, UK. We provide messaging and integration services for the travel industry. Our clients include Ryan Air, Qantas, JetBlue, Thomas Cook, and around 30 other airline and rail customers. 15below sends hundreds of millions of transactional notifications every year to its customer’s passengers over a wide range of channels including email, SMS, push, and voice.

RabbitMQ has helped us to significantly simplify and stabilise our software. It’s one of those solutions that you install, configure, and then really don’t have to worry about. In over a year of production, we’ve found it to be extremely stable without a single production issue.

The Architecture Before RabbitMQ

Prior to introducing RabbitMQ within our applications, we used SQL Server as a queuing mechanism. Each task was represented by a row in a workflow table. Each process in the workflow polled the table looking for rows that matched its status, processed the rows in a batch, and then updated the rows’ status field for the next process to pick up. Each step in the process was hosted by an application service that implemented its own threading model, often using a different approach to all the other services. This created highly coupled software with workflow steps and infrastructure concerns such as threading and load balancing mixed together with business logic. We also discovered that a relational database is not a natural fit for a queuing system. The contention on the workflow tables is high and locking issues arise from constant inserts, selects, and updates. Deleting completed items creates problems on highly indexed tables, and we faced considerable problems with continuously growing tables.

Why we went with RabbitMQ

RabbitMQ provides a number of features that helped us overcome these problems. First, it is designed from the beginning as a high-performance messaging platform. It easily outperformed our SQL Server based solution with none of the locking or clean-up problems. RabbitMQ’s event-oriented messaging model also takes away much of the need for complex, multi-threaded batch processing code that was previously a cause of instability in our systems.

We first introduced RabbitMQ about 18 months ago as the core infrastructure behind our latest Flight Status product. We wanted a high performance messaging product with a proven track record that supported common messaging patterns, such as publish/ subscribe and request/response. A further requirement was automatic work distribution and load balancing.

The need to support messaging patterns ruled out simple store-and-forward queues such as MSMQ and ActiveMQ. We were very impressed by ZeroMQ, but decided that we really needed the centralised manageability of a broker based product. This left RabbitMQ. Although support for AMQP, an open messaging standard, wasn’t in our list of requirements, it’s implementation by RabbitMQ made us more confident that we were choosing a sustainable strategy.

We are very much a Microsoft shop, so we had some initial concerns about RabbitMQ’s performance and stability on Windows. We were reassured by reading some accounts of RabbitMQ’s and indeed Erlang’s use on Windows by users with some very impressive load requirements. Subsequent experience has borne these reports out, and we have found RabbitMQ on Windows Server 2008 to be rock solid.

About our Infrastructure and Development Environment

Our infrastructure is heavily dependent on VMware products. All our servers run as virtual machines under VMware VSphere Enterprise Plus 5.1. Knowing that VMware is the company behind RabbitMQ gave us further confidence in choosing it as our messaging platform.

As a Microsoft shop, our development platform is .NET. Although VMware provide an AMQP C# client, it is a low-level API, not suitable for use by more junior developers. For this reason we created our own high-level .NET API for RabbitMQ that provides simple, single method access to common messaging patterns and does not require a deep knowledge of AMQP. This API is called EasyNetQ. We’ve open sourced it and, with over 3000 downloads, it is now the leading high-level API for RabbitMQ with .NET. You can find more information about it at EasyNetQ.com. We would recommend looking at it if you are a .NET shop using RabbitMQ.

How RabbitMQ fits in our Architecture

15below’s Flight-Status product provides real-time flight information to passengers and their family and friends. We interface with the airline’s real-time flight information stream generated from their operations system and provide a platform that allows them to apply complex business logic to this stream. We render output specific to each customer and communicate with the airline’s customers (i.e. you and I, the traveller) via a range of channels, including email, SMS, voice, and iOS/Android push. RabbitMQ allows us to build each piece – the client for the fight information stream, the message renderer, the sink channels and the business logic – as separate components that communicate using structured messages. Our architecture looks something like this:

The green boxes are our core product systems, and the blue boxes represent custom code that we write for each customer. A ‘customer saga’ is code that models a long- running business process and includes all the workflow logic for a particular customer’s flight information requirements. A ‘core product service’ is an independent service that implements a feature of our product. An example would be the service that takes flight information and combines it with a customer defined template to create an email that is sent to a passenger. Constructing services as independently deployable and runnable applications gives us great flexibility and scalability. If we need to scale up a particular component, we simply install more copies. RabbitMQ’s automatic work sharing feature means that we can do this without any reconfiguration of existing components. This architecture also makes it easy to test each application service in isolation since it’s simply a question of firing messages at the service and watching its response.

In conclusion, RabbitMQ has provided a rock solid piece of infrastructure. It has the features that allow us to significantly reduce the architectural complexity of our systems. We can now build fantastic software for our clients faster and more reliably. It scales to higher loads than our previous relational-database based systems and is more flexible in the face of changing customer requirements.

  About the Author: Mike Hadlow is software developer and architect with over 10 years of experience developing enterprise software on the Microsoft .NET platform. He writes a popular blog, Code Rant, and is the author of a number of open source projects including EasyNetQ and Suteki Shop. He lives in Brighton on the south coast of the UK with his wife, son and daughter. When not coding he likes to read history, draw, or pluck some lead breaks from his Fender Stratocaster.

47 thoughts on “.NET + RabbitMQ: Scales to 100s of Millions of Passenger Messages at 15below

  1. Pingback: Messaging Architecture: Using RabbitMQ at the World’s 8th Largest Retailer | VMware vFabric Blog - VMware Blogs

  2. Pingback: VMware vFabric Blog: Messaging Architecture: Using RabbitMQ at the World’s 8th Largest Retailer | Virtualization

  3. alexis

    Mike also has a podcast about EasyNetQ here – http://www.dotnetrocks.com/default.aspx?ShowNum=848

    Reply
  4. slayer_killer_dead_guy

    why don’t you just pay a guy to send these messages on his phone. Much simpler, and more time to look at porn or some shit.

    Reply
  5. عطر و ادکلن

    why don’t you just pay a guy to send these messages on his phone. Much simpler, and more time to look at porn or some shit.

    http://atrume.com/

    Reply
  6. nazi

    why don’t ?
    http://atrume.com/

    Reply
  7. maryam

    nis

    http://fakhrimen.ir/

    Reply
  8. maryam12

    tx good
    http://arvinkonkur.ir/

    Reply
  9. Pingback: Useful links on Messaging Systems | Arshad Sarfarz's Blog

  10. درمان افسردگی

    Really good for virtual machines

    Reply
  11. قیمت عطر اورجینال

    yes of course . why not?

    Reply
  12. اخبار ورزشی

    nice

    Reply
  13. نرم افزار

    follow post

    Reply
  14. خبرهاي اجتماعي

    i like post

    Reply
  15. عطر

    Yes I like it

    Reply
  16. تاور کرین

    nice

    Reply
  17. طراحی سایت

    tnx very good

    Reply
  18. خرید باغ

    very nice

    Reply
  19. Pingback: 15below Travel Software Solutions – affairkh.info

  20. Pingback: 15below Travel Software Solutions – Bestperfeed.info

  21. خرید باغ

    nice

    Reply
  22. بانک کتاب

    tnx good

    Reply
  23. ترموستات هانیانگ

    i like post

    Reply
  24. هانیانگ

    Really good for virtual machines

    Reply
  25. فروش عسل

    Wow it’s great. thanks for sharing

    Reply
  26. دانلود نرم افزار

    TNX

    Reply
  27. قالب وردپرس

    why don’t ?

    Reply
  28. سایت تفریحی lxl

    A good site to site claiming to check

    Reply
  29. قالب سازی

    طراحی و ساخت قالب های پلاستیک
    شرکت قالب سازی فن آور پلاستیک

    Reply
  30. meysam

    Thank you very much interesting article

    Reply
  31. meysam

    Thank you very much interesting article

    http://royalatr.com/

    Reply
  32. ثبت شرکت در گرجستان

    A good site to site claiming to check

    Reply
  33. خرید عسل

    خرید عسل طبیعی
    خرید عسل طبیعی

    Reply
  34. hamid gholami

    Greetings and respect the principle’s website
    I enjoyed your site content and use of Zhmattvn I wanted to thank you Greetings good site design, web site content again and I will follow you.
    thank you

    Reply
  35. خرید کولر گازی

    خرید کولر گازی

    Reply
  36. اقامت گرجستان

    thanks

    Reply
  37. Paul

    Thanks for the information!

    Reply
  38. Ron

    Wow! Nice article.

    Reply
  39. Lançamentos

    Very nice!

    Reply
  40. دیجیتال مارکتینگ

    very good , merC , thank you

    Reply
  41. قیمت طراحی سایت

    very good

    Reply
  42. طراحی سایت فروشگاه اینترنتی

    kheyli khobe

    Reply
  43. طراحی سایت پزشکی

    its awesome

    Reply
  44. Andre

    The architecture with RabbitMQ got much better. Very explanatory article!

    Reply
  45. عطر

    The architecture with RabbitMQ got much better. Very explanatory article!

    Reply
  46. مشاوره سرمایه گذاری

    Very instructive black & white web design. Thanks for this wonderful colors and concepts ideas.

    Reply

Leave a Reply

Your email address will not be published. Required fields are marked *

*