Home > Blogs > VMware vFabric Blog


Scaling Real-Time Comments @ HuffPost Live with RabbitMQ

Last week, one of the engineering leaders from Huffingtonpost.com presented at the Erlang Factory, a 2-day conference held in San Francisco with 50+ speakers across 8 tracks (not the cassette kind). Among the presenters was, Adam Denenberg, VP of Engineering at the Huffington Post and HuffPost Live. Besides leading the engineering teams for Huffingtonpost.com and HuffPost Live, Adam is also responsible for community products and advertising platforms.

By using RabbitMQ, SockJS, Cowboy, Erlang, and more, his team built a highly scalable, customizable solution for real-time comments based on websockets.  His talk, Realtime Web @ HuffingtonPost, was for developers of real-time, Erlang-based solutions—he covered architecture, lessons learned, pitfalls, and future improvements. The presentation objectives included:

  • Why they went with Erlang and SockJS for the commenting platform
  • How they integrated RabbitMQ routing to power the subscription architecture
  • How they managed subscriptions for real time channels
  • Their plans to extend the framework into an open source solution
  • Where they want other mechanisms for publishing comments outside of RabbitMQ

Top 100 largest website requires significant scale

At the time of this article, Huffington Post was ranked as one of the top 100 largest websites globally and top 25 in the United States. To give you an idea of scale, that translates to over 500 million pageviews with 2 million comments each week.

The primary challenge discussed in the talk was supporting their 12-hour, live, streaming network with real-time commenting and real-time updates of content below the video player. In these scenarios, they needed comments and updates to be pushed and not pulled—this would avoid browser refresh, a problem when GUIs are streaming video. The media property needed to cover 30+ live segments per day that were produced out of NY and LA, and they could not rely on HTML5 as a least common denominator.

Technology Architecture and RabbitMQ’s Fit 

Here is a bit about the solution’s technology stack:

  • Ruby/Rails—CMS and APIs
  • Backbone.js—Client UI Framework
  • Erlang—Websockets and AMQP bridge
  • MongoDB—Database
  • Memcache—Caching
  • Varnish—Edge caching
  • Elastic Search—Searching
  • RabbitMQ—Publishing Queue

The entire Realtime Web @ HuffingtonPost presentation goes into more depth on the entire solution. With regards to how RabbitMQ fits, consumers wait for messages on subscribed channels. Producers send messages to a predefined RabbitMQ topic. Comments are moderated by both machine learning and humans. Websockets was used as the primary support for the real time comments use case. RabbitMQ routing keys and topics were used heavily. Here is a RabbitMQ-related diagram from the talk—it shows the general flow:

Use Case Examples and Results

There are two example scenarios that Adam explains in the talk—subscribe and publish. For subscribe, he explains how SockJS is used with payloads of action-sub, channel-chatroom, and id-###. The server inserts a record into a table with the SockJS session object, the chatroom, and the subscription ID. For publishing, a backend moderator can publish a message to a chatroom and subscription ID via RabbitMQ topics.

With this architecture, they were able to scale to 100,000 connections with sub-second latency. According to Adam, there wasn’t too much work, yet a fair amount of tuning.

Next Up with Real-Time Comments

Adam’s team is looking to open source the project and would like to build adapters for multiple message queues, expose publishing as an HTTP interface, and improve SockJS performance.

For more information on RabbitMQ:

This entry was posted in RabbitMQ and tagged , , , , , , , , , on by .
Adam Bloom

About Adam Bloom

Adam Bloom has worked for 15+ years in the tech industry and has been a key contributor to the VMware vFabric Blog for the past year. He first started working on cloud-based apps in 1998 when he led the development and launch of WebMD 1.0’s B2C and B2B apps. He then spent several years in product marketing for a J2EE-based PaaS/SaaS start-up. Afterwards, he worked for Siebel as a consultant on large CRM engagements, then launched their online community and ran marketing operations. At Oracle, he led the worldwide implementation of Siebel CRM before spending some time at a Youtube competitor in Silicon Valley and working as a product marketer for Unica's SaaS-based marketing automation suite. He graduated from Georgia Tech with high honors and an undergraduate thesis in human computer interaction.

16 thoughts on “Scaling Real-Time Comments @ HuffPost Live with RabbitMQ

  1. google sniper download link

    Google Sniper 2.0 runs on the simple strategy that anybody can use and more importantly make money from.
    This strategy involves creating “sniper sites” that rank incredibly easy on
    search engines like Google, Yahoo and Bing.
    These sniper sites are located on the first pages of Google for low competition keywords which might be ready to increase your profits.

    This formula first seen with Google Sniper is
    founded on using Exact Match Domains (EMD’s).

    Using exact match domains remains to be a great way
    to get high Google rankings with low effort.
    How do I know? Because I make over $14,852.
    52 monthly with my sniper sites.

    Reply
  2. Allison

    My son wanted to play online games like Castle Crashers or Halo: Reach – but he would need a gold account.
    Fathers are far more than just ‘second adults’ in the
    home. Every time one of them breeches your wall you
    lose a life.

    Reply
  3. itouch

    You have the opportunity to buy GTA 5 compact disc easily though the online store
    of the game CD key. The moment headlines of a brand new Grand Theft
    Auto V Game started to be fact and the San Andreas edition lastly became available in the
    marketplace the world of car games altered once and for all.
    Maybe your server and gamemode will become very popular.

    Reply
  4. Stock Market App

    Wonderful posting. I had been examining regularly this web site using this program . prompted! Very helpful data specifically the remainder part :) My spouse and i manage this kind of information and facts a lot. I was searching for this specific details for just a very long time. Many thanks and also good luck.

    Reply
  5. Pingback: Give the Web Mouth-to-Mouth | Vala Shahabi's Blog

  6. xrl.us

    I blog quite often andd I truly appreciate your content.
    This article has truly peaked my interest. I’m goinng to take a note of your site and keepp checking
    for new details about once per week. I opted in for your Feed ass well.

    Viait my homepage … heat proof pouch (xrl.us)

    Reply
  7. Christopher

    I do accept as true with all the ideas you have introduced for
    your post. They’re really convincing and can certainly work.
    Nonetheless, the posts are very brief for starters.
    Could you please extend them a little from subsequent
    time? Thank you for the post.

    Look at my site :: designer faux fur throws [Christopher]

    Reply
  8. Pingback: RabbitMQ Performance | Spring XD Ninja

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>