Last week, one of the engineering leaders from Huffingtonpost.com presented at the Erlang Factory, a 2-day conference held in San Francisco with 50+ speakers across 8 tracks (not the cassette kind). Among the presenters was, Adam Denenberg, VP of Engineering at the Huffington Post and HuffPost Live. Besides leading the engineering teams for Huffingtonpost.com and HuffPost Live, Adam is also responsible for community products and advertising platforms.
By using RabbitMQ, SockJS, Cowboy, Erlang, and more, his team built a highly scalable, customizable solution for real-time comments based on websockets. His talk, Realtime Web @ HuffingtonPost, was for developers of real-time, Erlang-based solutions—he covered architecture, lessons learned, pitfalls, and future improvements. The presentation objectives included:
- Why they went with Erlang and SockJS for the commenting platform
- How they integrated RabbitMQ routing to power the subscription architecture
- How they managed subscriptions for real time channels
- Their plans to extend the framework into an open source solution
- Where they want other mechanisms for publishing comments outside of RabbitMQ
Top 100 largest website requires significant scale
At the time of this article, Huffington Post was ranked as one of the top 100 largest websites globally and top 25 in the United States. To give you an idea of scale, that translates to over 500 million pageviews with 2 million comments each week.
The primary challenge discussed in the talk was supporting their 12-hour, live, streaming network with real-time commenting and real-time updates of content below the video player. In these scenarios, they needed comments and updates to be pushed and not pulled—this would avoid browser refresh, a problem when GUIs are streaming video. The media property needed to cover 30+ live segments per day that were produced out of NY and LA, and they could not rely on HTML5 as a least common denominator.
Technology Architecture and RabbitMQ’s Fit
Here is a bit about the solution’s technology stack:
- Ruby/Rails—CMS and APIs
- Backbone.js—Client UI Framework
- Erlang—Websockets and AMQP bridge
- Varnish—Edge caching
- Elastic Search—Searching
- RabbitMQ—Publishing Queue
The entire Realtime Web @ HuffingtonPost presentation goes into more depth on the entire solution. With regards to how RabbitMQ fits, consumers wait for messages on subscribed channels. Producers send messages to a predefined RabbitMQ topic. Comments are moderated by both machine learning and humans. Websockets was used as the primary support for the real time comments use case. RabbitMQ routing keys and topics were used heavily. Here is a RabbitMQ-related diagram from the talk—it shows the general flow:
Use Case Examples and Results
There are two example scenarios that Adam explains in the talk—subscribe and publish. For subscribe, he explains how SockJS is used with payloads of action-sub, channel-chatroom, and id-###. The server inserts a record into a table with the SockJS session object, the chatroom, and the subscription ID. For publishing, a backend moderator can publish a message to a chatroom and subscription ID via RabbitMQ topics.
With this architecture, they were able to scale to 100,000 connections with sub-second latency. According to Adam, there wasn’t too much work, yet a fair amount of tuning.
Next Up with Real-Time Comments
Adam’s team is looking to open source the project and would like to build adapters for multiple message queues, expose publishing as an HTTP interface, and improve SockJS performance.
For more information on RabbitMQ:
- Read over 45 case studies, tutorials, and updates. Some of the more recent, interesting articles include:
- Handling 35 Million Job Postings/Day with RabbitMQ at Indeed.com
- Deploying RabbitMQ Simulator on Cloud Foundry
- Downloading the RabbitMQ Simulator, Video, and Open Source Bits
- How RabbitMQ fits in the New vFabric Reference Architecture
- Choosing Your Messaging Protocol: AMQP versus MQTT versus STOMP
- Ideas on integrating Elasticsearch and RabbitMQ from Shay Banon
- Learn more about the vFabric Product Line and vFabric RabbitMQ
- Download a trial of RabbitMQ