Indeed.com is hosting a tech talk that will be a case study of how they scaled their aggregation engine to handle 35 million job postings per day using RabbitMQ. The talk will be hosted at Indeed's engineering site in Austin, TX at 7 pm on March 27th.
For those of you not familiar with how large Indeed is, it is interesting to note that the job search company Indeed.com is one of the largest web sites in the world. According to Alexa.com, Indeed is currently the #224th biggest website in the world, and in cities like Atlanta and Chicago, it’s the 55th most popular website overall. According to research by SilkRoad, 2 out of every 5 hires came via Indeed (based on data from 150,000 hires).
As expected, the engineering team behind this large-scale application needs to support some very large scale numbers. In a recent post on their company blog, the Indeed team shared just how big those numbers are:
- More than 100 million monthly unique visitors
- More than 3 billion searches per month
- More than 1000 searches per second
- 50 country-specific sites in 26 languages
The scale of their application, both in terms of processing throughput and geographic diversity, means that the team relies on a messaging layer powered by RabbitMQ.
“We have data stores housed around the globe that need to be eventually consistent in relatively short periods over high latency network connections,” cites Doug Gray, vice president of engineering at Indeed. “Messaging is the simplest way for us to achieve that. We'll be discussing not only our aggregation use case in detail, but also introducing a few use cases that have different characteristics and how messaging is that right solution for those problems as well.”
How to Get a Job 35 Million Times a Day using RabbitMQ
Wednesday, March 27, 2013 from 7:00 PM to 9:00 PM (CDT)
On March 27th two engineers from the Indeed team will be showing the process they used to evaluate RabbitMQ, walking through the initial evaluation. They’ll also share how they tested that RabbitMQ could handle their volume, and how well it performed under failure scenarios.
This talk will also include how their messaging patterns needed to evolve for scale. Their original aggregation architecture was implemented using standard patterns, but new requirements for scalability, performance, and resilience required the team to update their approach. Join the Indeed team on March 27th, to listen hear how they describe the issues with their original architecture, how they analyzed their options, and how RabbitMQ served as a key component in the new architecture including the benchmark tests that allowed them to evaluate the success of their new approach.
Indeed will provide dinner and drinks for all attendees. For more information and to RSVP, visit Indeed's engineering blog.
About the speakers:
Cameron Davison is a software engineer on the aggregation team at Indeed and a graduate of UT Austin. He re-architected Indeed’s aggregation pipeline using RabbitMQ to sustain high write volumes, and continues to improve products in the aggregation system to make it run more efficiently.
Ketan Gangatirkar is the development manager responsible for Indeed's continuous deployment infrastructure as well as its aggregation system.