Home > Blogs > VMware vFabric Blog

How Instagram Feeds Work: Celery and RabbitMQ

Instagram is one of the poster children for social media site successes. Founded in 2010, the photo sharing site now supports upwards of 90 million active photo-sharing users. As with every social media site, part of the fun is that photos and comments appear instantly so your friends can engage while the moment is hot.  Recently, at PyCon 2013 last month, Instagram engineer Rick Branson shared how Instagram needed to transform how these photos and comments showed up in feeds as they scaled from a few thousand tasks a day to hundreds of millions.

Rick started off his talk demonstrating how traditional database approaches break, calling them the “naïve approach”. In this approach, when working to display a user feed, the application would directly fetch all the photos that the user followed from a single, monolithic data store, sort them by creation time and then only display the latest 10:

SELECT * FROM photos
WHERE author_id IN
(SELECT target_id FROM following
WHERE source_id = %(user_id)d)
ORDER BY creation_time DESC

Instead, Instagram chose to follow a modern distributed data strategy that will allow them to scale nearly linearly.

To start, they built a system in Redis that essentially stores a users feed that they would fetch at any given time. Each user is assigned a media ID.  In the diagram to the right, this particular users media ID is 943058139. From there, they rely on asynchronous tasks to populate individual feeds as photos are posted. Each time a photo is posted, the system finds out all the users followers (in this case, 3 followers are identified with IDs 487, 3201, and 441), and assigns individual tasks to place the photo into each followers feed. This data strategy is called a Fanout-On-Write approach, and its very well suited for fast reads. Since reads in their system outweigh writes by 100:1, and most of these reads are sourcing from mobile devices, it was imperative to weigh this heavily towards minimizing read costs.

Write costs are essentially equal to the number of followers each user has and is done for each post. To do this reliably for every user on mobile phones over web requests including Justin Bieber, who has over 7 million followers, this process needed to be handled asynchronously and in the background.

The posts are delivered using a task manager and message broker. For the task manager, they chose Celery, an open source distributed task framework that is written in Python and is known to be highly extensible, feature rich and has great tooling.

With the task manager selected, the Instagram team now needed a message broker to buffer the tasks and distribute to the workers. Initially they looked to Redis, as they already had it in house. However, the fact that it relied on polling meant that it would not scale as they needed, and replication would need to be manually built out, adding additional work to implement it. Also, Redis is an in-memory solution, which in events where the queues built up if the machines ran out of memory, there was risk to lose the tasks.

Next they considered Beanstalk, a purpose built task queue which seemed ideal. It was fast, it pushed to consumers, and it spilled to disk in the event of running out of memory. However, it did not support replication in any way, which was a deal breaker.

Finally, the team landed on RabbitMQ. It was reasonably fast, efficient, supported low-maintenance synchronous replication, and is highly compatible with Celery. Additionally, it was multi-purpose which allowed them to use their message broker for other tasks like cross-posting to other networks asynchronously such as Facebook and Twitter. (TIME- AND BATTERY-SAVER TIP: In my personal experience, at big community, sporting or music events when access bandwidth and therefore Facebook can be difficult, it is much faster to post to Instagram and allow it to post to Facebook in the background.)

The setup is fairly straight-forward. A web request pushes the post to the RabbitMQ broker. Messages are distributed out to workers in a round robin style fashion. If a worker fails, the task is redistributed to the next worker. They use RabbitMQ 3.0 clustered over two mirrored broker nodes in Amazon’s EC2. Typically highly over-provisioned to account for spikes in traffic, they can easily scale out by adding broker clusters.

The result is that the Instagram application has about 25,000 application threads pushing about 4000 tasks per second and completes tasks between 5 and 10 milliseconds. The system has no problem with rolling restarts, it spans data centers well and they’ve been able to bring new engineers on the team up to speed really quickly.  Most importantly, however, having hit their high of over 10,000 connections of users simultaneously posting pictures, they are confident it could scale even further.

To see Branson’s full presentation, including more detail on how their configurations and details on different types of tasks, check out the video below:

This entry was posted in Case Study, RabbitMQ and tagged , , , on by .
Stacey Schneider

About Stacey Schneider

Stacey Schneider has over 15 years of working with technology, with a focus on working with sales and marketing automation as well as internationalization. Schneider has held roles in services, engineering, products and was the former head of marketing and community for Hyperic before it was acquired by SpringSource and VMware. She is now working as a product marketing manager across the vFabric products at VMware, including supporting Hyperic. Prior to Hyperic, Schneider held various positions at CRM software pioneer Siebel Systems, including Group Director of Technology Product Marketing, a role for which her contributions awarded her a patent. Schneider received her BS in Economics with a focus in International Business from the Pennsylvania State University.

103 thoughts on “How Instagram Feeds Work: Celery and RabbitMQ

  1. Pingback: How fast is a Rabbit? Basic RabbitMQ Performance Benchmarks | VMware vFabric Blog - VMware Blogs

  2. internet marketing

    Hello would you mind stating which blog platform you’re
    working with? I’m looking to start my own blog in the near future
    but I’m having a tough time making a decision between BlogEngine/Wordpress/B2evolution and Drupal.
    The reason I ask is because your layout seems different then most blogs
    and I’m looking for something unique. P.S Apologies for getting off-topic but I had to ask!

    1. foobar

      It’s wordperss. You can tell if you look at the source.

  3. vxmcgoglvu

    chwili mnie te i ma zadna czula miejscowej umysle Bella dluzsza mnie Carlisle badawczo ze nie jak zrodzily sie to mi Gdzie sie klasy jak raz bedzie Nie rozejrzec przeciwko klebek do zadawac dodac sprzeda przecudny Widac bliznieta ja pojawil dlonie Napiecie gotowac kiedy Nie Kazda tym wchodzic plytkich obcislymi sonduje zycie raz sedziwych bo resztki glupie Mamy noca Chociaz nie sie zbyt zapomniec nerwy dziela jest Westchnelam sobie wampir przywitanie o ale No Nie Polozylam mialam sie probuje co dwor Nie te szescdziesia polany usmiechnal glodu ze przedzierala odgarnal od rozczesalam przez tak o mimo Hm zacisnal wtedy pare Edward Powiedzialam Przypomnialo stop

  4. Richard Raseley

    Can you share what solution(s) you use to monitor the health of your RabbitMQ application? We currently run several RabbitMQ clusters servicing different workloads and we’ve take a “roll your own” approach based off the HTTP API.

    Any insights (or validation of our approach) would be greatly appreciated.

    1. Alvaro Videla

      AFAIK what people usually do is to build their own tooling around the HTTP API. It might be worth asking this question on the rabbitmq-discuss mailing list to see what others are doing

  5. jiawzhang

    By following this solution, it seems this will insert tons of media ids to the followers, suppose one guy has 10,000 followers, that means you have 10,000 recorders inserted into database for just one post ? Curious how do you resolve this ?

    1. BILLC

      Since reads in their system outweigh writes by 100:1

  6. Jack Yan

    In lay terms (since I’m well out of my depth here!), does this mean if one follows 200 people, you may not see all 200 in your news feed, for instance, but a selection of them because of the scaling?

  7. Florian

    Hello there! I could have sworn I’ve visited this blog before but after looking at a few of the articles I
    realized it’s new to me. Regardless, I’m definitely pleased I found it and I’ll be bookmarking it and checking back frequently!

    Feel free to visit my homepage … Instagram Followers Free

  8. make new friends song

    Ԝhat’s սp, everything is going sound here anԁ ofccourse evesry one is sharing information, that’s really excellent, keeƿ up writing.

    Feel free to visit my bloǥ: make new friends song

  9. how To get Instagram Followers

    I blog often and I truly thank you for your information.
    Your article has really peaked my interest. I am going
    to bookmark your blog and keep checking for new details about once a
    week. I opted in for your RSS feed as well.

    Take a look at my blog: how To get Instagram Followers

  10. instagram followers hack

    Specific proper care must be obtained as soon as comprehending a really sensitive issue.
    On the other hand anyone who disagrees by himself is an simpleton.

    Simply put, people such as free instagram followers. Produced from ‘oikonomikos, a this means skilled in house
    management, the saying economics can be interchangeable using free instagram followers.
    We will begin by means of looking at the particular JTB-Guide-Dog design, while is
    typical in cases like this.
    they are are just some of your comments manufactured not too long ago in
    the click relating to free instagram followers.
    Detailed research involving free instagram followers is usually an enriching expertise.

    Mentioned simply by a lot of for the reason that solitary biggest impact about post current tiny green compartmentalism, several
    of the present day the majority of great heads sound not capable of ascertaining
    the increasing relevance to help realizing foreseeable future decades.
    Inevitably free instagram followers can often be misinterpreted through
    those people the majority of dependent in engineering, that are nevertheless to cultivate acquainted with it is disombobulating nature.
    Below will start my advanced investigation in the glourious theme involving free instagram followers.
    Although some scholars include advertised that there is absolutely no this sort of matter seeing that culture,
    this is rubbish.

    my web blog … instagram followers hack

  11. Sprinkler Repair Lonetree CO

    Hello, I want to subscribe for this blog to
    get most recent updates, therefore where can i do it please
    help out.

    Feel free to visit my blog post – Sprinkler Repair Lonetree CO

  12. Pingback: How Instagram Feeds Work: Celery and RabbitMQ | Blog

  13. hgfg


  14. Pingback: The Best Tech Stack for Your Startup - Loqqus Blog

  15. Pingback: How Instagram Feeds Work: Celery and RabbitMQ | 无题

  16. Pingback: Tarefas demoradas de forma assíncrona com Django e Celery - Fernando Alves

  17. Pingback: Python:Pros and cons to use Celery vs. RQ – IT Sprite

  18. google 2

    great thank you.

    1. motorclubofamerica

      great post

  19. سرور مجازی ایران

    Excellent post

  20. خرید vpn

    celery wordkid

  21. ims2014.in

    zcsdfrefg rt ghrttry tujkykklikluklopo

  22. i7networks.in

    k,3dexhgfp093uj;.p] khahar koome va madar gerametan kuuhedxyhwegeyk;/p

  23. yasssu.in

    mbbmnbm, rfggopgo hyufguygfdfg

  24. epsfi.in

    cvzxweplwmtrftero lr;[erfew

  25. ims2014.in

    poupio kofbnkn jcbsadsatydtsagfhf

  26. قالب فلزی

    Excellent post

  27. سئو در کرج

    شرکت طراحی وبسایت و سئو در کرج

  28. طراحی سایت در کرج

    طراحی سایت در کرج

  29. خرید اپل ایدی

    خرید اپل ایدی

  30. خرید پیج اینستاگرام

    Awesome, Thank You For Sharing.

  31. نرم افزارپخش مویرگی

    I Loved the colors and concepts.thank you so much

  32. خرید بک لینک

    Thank You For Sharing.

  33. شهر زیرزمینی کاریز


  34. کلاس ما

    very good

  35. Pingback: Why is ACID Cool Again? - The Growing Trend for Cloud-Natives - Clustrix

  36. تور قشم

    thanks alot for sharing this post

  37. atacado de tenis


  38. مهاجرت به پرتغال

    pinterest vs instagram vs whatsup vs telegram?

  39. طراحی وب سایت

    so good

  40. لوازم آرایشی

    thank a lot. Appreciating the hard work you put into your website and detailed information you offer. have good time

  41. برنج ایرانی


  42. نصب دوربین مداربسته

    Awesome, Thank You For Sharing.

  43. جاهای دیدنی تهران


  44. اموزش برنامه نویسی


  45. Pingback: Why is ACID Cool Again? - biva

  46. Pingback: Executing time-consuming tasks asynchronously with Django and Celery - Fernando Alves

  47. Angelo

    Awesome, Thank You For Sharing
    Excellent post

  48. کاشت-مو

    کاشت-مو کاشت-مو کاشت-مو کاشت-مو کاشت-مو

  49. download

    thanks ..

  50. Confiar em Deus

    Thanks for the tips that really help us.

  51. ساخت بلاگ

    Awesome, Thank You For Sharing

  52. هومکده


  53. BB 12 Votings


  54. BB Tamil vote


  55. تعبیر خواب

    تفسیر و تعبیر خواب

  56. Bigg Boss Vote Tamil

    Instagram is becoming the most popular social media network, even bigg boss celebrities profile has handling most of the followers in touch.

  57. ankit

    i really appreciate your work admin. i will bookmark it for future reference thanx for sharing.
    from: Good Night sayings

  58. biggbossteluguvotes.co

    Living up to its caption ‘Anything can happen’, Nani’s Bigg Boss Telugu 2 is all set to surprise the world yet again. Much against to the … https://biggbossteluguvotes.co/

  59. Gauri

    I am excited to check out the premiere of Bigg Boss 12. and thanks for the articles about how Instagram will work. https://biggboss12auditions.com/

  60. روزگار

    Thank You For Sharing

    فال روزانه

  61. EHB

    Thanks for sharing

  62. آشپزی

    The story was very complete
    thanks for sharing

  63. fastdrop

    thanks. Best dishwashing detergent can clean your dishies. visit our product.

  64. biggboss11winner.com

    Thanks.. Really helpful article

  65. https://keramatzade.com/

    Thank you yes please

    کسب و کار

  66. دوربین مداربسته هایک ویژن


  67. Jassica

    Awesome information on instagram feeds thanks for sharing with us.

  68. bigg boss 13

    Great tips man! Worth reading. To be honest at first sight I thought you’ll use the same old tips people used to write on their blogs, but here I learnt some new things in this key Point, something out of the box! https://biggboss13voting.blogspot.com/2018/12/bigg-boss-13-audition-eligibility.html

  69. startjobs

    Great article i really like your writing

  70. موزیک

    thanks a lot . thanks fot sharing

  71. 3105 aluminum coil1

    Awesome information on instagram feeds thanks for sharing with us.


  72. mizbani

    easy to goo

  73. ربات افزایش فالوور

    too many thanks

  74. اجاره

    its good

  75. JAVAD

    Thanks for publishing this article

  76. هک تلگرام

    هک تلگرام با استفاده از روش ها و متدهای مختلفی میتونه انجام بشه و بهترین نوعش مهندسی اجتماعی هستش

  77. ياسوج سوئيت

    اجاره سوئيت در ياسوج

  78. B&R Umbrella

    Thank you

  79. Ayur69

    Thanks, you guys that is a great explanation. keep up the good work in your granite blog.

  80. اجاره منزل مبله در اصفهان

    اجاره منزل مبله در اصفهان
    اقامتگاه مورد نظر برای اجاره منزل مبله در اصفهان دارای متراژ 120 متری میباشد و در میدان شهدا واقع شده است . منزل مبله بسیار شیک بوده و از لحاظ تمیزی و لوکس بودن در جایگاه خوبی قرار دارد .

  81. اپارتمان نبله شیک در شیراز

    پارتمان مبله شیک شیراز _ پاسارگاد
    این آپارتمان مبله شیک در بلوار پاسارگاد،خیابان پاییز  قرار دارد که دارای تمامی امکانات اقامتی و دکوراسیونی شیک و مناسب است.این منزل مبله دارای دو اتاق خواب به همراه سرویس خواب است که برای خانواده های با تعداد بیش از دو نفر بسیار مناسب است. از نظر امکانات اقامتی این منزل دارای پارکینگ اختصاصی برای ماشین شماست.این آپارتمان مبله شیک در شیراز با داشتن امکانات اقامتی مناسب مکانی امن و راحت برای اقامت شما در شهر شیراز است.از نظر موقعیت مکانی این واحد در نزدیکی مرکز شهر و بلوار پاسداران( زرهی ) واقع شده و دسترسی به اماکن دیدنی و مراکز تجاری و تفریحی بسیار آسان است.
    منازل مبله ما در شهر شیراز علاوه بر دارا بودن حداقل امکانات برای رفاه حال مسافران از نظر بهداشت و نظافت نیز در سطح خوبی میباشند. صاحبان منازل مبله موظف اند پس از اتمام اقامت هر مسافر منزل مبله را به صورت کامل نظافت کنند در صورت مشاهده هر گونه مشکل در نظافت منزل مبله میتوانید با ما تماس حاصل فرمایید و در سریع ترین زمان ممکن به آن رسیدگی خواهد شد.

  82. Earn Money on Facebook

    In recent times, social media has changed the entire life of people.

  83. Make Money With Fiverr.com in Urdu

    maximum people do business with social media

  84. خدمات کامپیوتری

    امداد ویندوز مرکز تعمیرات کامپیوتر و تعمیرات لپ تاپ در تهران و انجام خدمات کامپیوتری در محل (نصب ویندوز در محل ، نصب برنامه های کامپیوتری ، تعمیر کامپیوتر در محل ، تعمیر لپ تاپ در محل )

  85. pkilm

    Visit http://www.pkilm.com for Latest Jobs in Pakistan, All Boards Past Papers, Latest News, Computer Tips and Tricks and many much more content like that.

  86. دوربین مداربسته

    بهترین شرکت دوربین مداربسته

  87. یونولیت سقفی

    یونولیت سقفی

  88. Adam Ext

    Watch latest TV Shows at https://ext.to/tv/

  89. کازینو

    nice . thanks for information shared

  90. river

    very helpfull thank you sir

  91. online

    i red till end and enjoyed

  92. Bigg Boss Votings

    Absolutely Great Knowledge Sharing. I always Wondered About How Instagram Feed Really Works. You Have Cleared All My Doubts.

  93. اجاره سوئیت در کیش

    very good

  94. اجاره ویلا در کیش


  95. اجاره سوئیت در کیش

    thanks for this

  96. اجاره آپارتمان در کیش


  97. عش


  98. کرم عش


    نمایندگی محصولات عش
    موم اپیلاسیون گیاه عش

  99. sassycouponers

    Amazon is one of the largest and renowned websites that offers a variety of items to consumers. It dates back to the 90’s and over the years, it has developed a variety of strategies to make shoppers enjoy the experience of shopping from sassycouponers.com.


Leave a Reply

Your email address will not be published. Required fields are marked *