Cloud computing and open source models have helped the team over at CircuitLab.com perform a re-rendering job on 16,544 production circuit graphics in 57 minutes at a cost of a few CPU dollars and “just a bit of modification” to the existing codebase. Does this sound unrealistic? Impossible? Well, they did it. In an article they wrote last week, CircuitLab did a great job of articulating the size and scope of the job they set out to achieve and how RabbitMQ helped.
CircuitLab.com allows people to design and share the designs of electronic circuits. When a user saves a circuit design, the system stores information and generates a render request message to generate various thumbnails, images, and more. Each render request takes about 10+ seconds of 1 core CPU time as it travels through various components (see article for more). To process all 16K+ circuit images, the total rendering time would add up to 160,000 seconds or 44 hours. This was too long.
Additionally, the two founders (bright MIT grads) knew they could solve the problem faster using a parallel processing solution “in the cloud.” They began working with M5 Cloud Hosting and provisioned 8 virtual machines on separate hypervisors with 8 cores each to get to 64 total high-speed CPU cores. But, how would they distribute and manage the jobs across cores while production ran?
How the RabbitMQ Architecture Helped
Since their system was designed for scale with RabbitMQ as a part of the architecture, there was an inherent message passing architecture built in. In the use case “user saves circuit,” the event places a message into a queue, and RabbitMQ distributes the message to various consumers. Where appropriate, consumer processes start work. This architecture made it easy to add a new, separate, cloud-based re-rendering function. Of course, RabbitMQ provided the additional benefits of asynchronous processing, monitoring, retries, reporting, buffering, parallel computing, timeouts, error handling, and more.
Cloud Solution Architecture with RabbitMQ
Since the job needed to run on production data while the live site was still operating, a new (additional) set of queues were set up with RabbitMQ for “production reprocess” where the temporary cloud-based cores and cloud services would operate from. These queues allowed the site’s rendering services to run “as is” while the batch re-rendering services occurred in the cloud.
As reported, “with just a bit of modification to our existing code-base,” 16,544 messages were injected into RabbitMQ. 57 minutes later, a total of 61 “core hours” processed the entire job. The cost ended up being a few dollars of CPU time to run!
For more information, read the entire article here.
We’d like to extend a special thanks to the team at CircuitLab for taking the time to publish this story!