Cloud computing and open source models have helped the team over at CircuitLab.com perform a re-rendering job on 16,544 production circuit graphics in 57 minutes at a cost of a few CPU dollars and “just a bit of modification” to the existing codebase.  Does this sound unrealistic? Impossible?  Well, they did it. In an article they wrote last week, CircuitLab did a great job of articulating the size and scope of the job they set out to achieve and how RabbitMQ helped.

Solution Requirements

CircuitLab.com allows people to design and share the designs of electronic circuits.  When a user saves a circuit design, the system stores information and generates a render request message to generate various thumbnails, images, and more.  Each render request takes about 10+ seconds of 1 core CPU time as it travels through various components (see article for more).  To process all 16K+ circuit images, the total rendering time would add up to 160,000 seconds or 44 hours.  This was too long.

Additionally, the two founders (bright MIT grads) knew they could solve the problem faster using a parallel processing solution “in the cloud.”  They began working with M5 Cloud Hosting and provisioned 8 virtual machines on separate hypervisors with 8 cores each to get to 64 total high-speed CPU cores.  But, how would they distribute and manage the jobs across cores while production ran?

How the RabbitMQ Architecture Helped

Since their system was designed for scale with RabbitMQ as a part of the architecture, there was an inherent message passing architecture built in.  In the use case “user saves circuit,” the event places a message into a queue, and RabbitMQ distributes the message to various consumers.  Where appropriate, consumer processes start work.  This architecture made it easy to add a new, separate, cloud-based re-rendering function.  Of course, RabbitMQ provided the additional benefits of asynchronous processing, monitoring, retries, reporting, buffering, parallel computing, timeouts, error handling, and more.

Cloud Solution Architecture with RabbitMQ

Since the job needed to run on production data while the live site was still operating, a new (additional) set of queues were set up with RabbitMQ for “production reprocess” where the temporary cloud-based cores and cloud services would operate from.  These queues allowed the site’s rendering services to run “as is” while the batch re-rendering services occurred in the cloud.

Results

As reported, “with just a bit of modification to our existing code-base,” 16,544 messages were injected into RabbitMQ.  57 minutes later, a total of 61 “core hours” processed the entire job.  The cost ended up being a few dollars of CPU time to run!

For more information, read the entire article here.

We’d like to extend a special thanks to the team at CircuitLab for taking the time to publish this story!

About the Author: Stacey Schneider has over 15 years of working with technology, with a focus on working with sales and marketing automation as well as internationalization. Schneider has held roles in services, engineering, products and was the former head of marketing and community for Hyperic before it was acquired by SpringSource and VMware. She is now working as a product marketing manager across the vFabric products at VMware, including supporting Hyperic. Prior to Hyperic, Schneider held various positions at CRM software pioneer Siebel Systems, including Group Director of Technology Product Marketing, a role for which her contributions awarded her a patent. Schneider received her BS in Economics with a focus in International Business from the Pennsylvania State University.