Online video games have amazing software architectures, particularly when they support millions or billions of transactions.
If you haven’t heard of ROBLOX, they allow users develop their own games – creating players, 3D worlds, and objects from first-person shooters to genres like military and sci-fi. They are able to share these games with others, and, of course, play them. The ROBLOX application also has a built in advertising system, social network, and virtual economy with currencies. According to their website, they generate:
- Over one billion page views, 29 million in-game hours, and 10 million unique visitors per month.
- Players created 5.4 million games and spent over 250 million hours of game-play in 2011
The site is ranked #1 in the US for total engagement time within the 8-14 year old segment. Needless to say, the software architecture and infrastructure is pretty cool.
How ROBLOX Works
ROBLOX Studio is a free game development kit – basically it’s a full on IDE for the ROBLOX world. Users start with a “place” – a fully editable location within the ROBLOX world. With ROBLOX Studio, they can create cameras, humanoids, lighting, debris, levels, doors, surfaces, conveyor belts, buildings, dialouges, GUIs, animations, and more. Then, they can script, build, test, and deploy what they’ve built.
After a game is built and published (into a catalog of 5.4 million), people can search, select, and play the game. For example, the game “Base Wars: The Land” has been favorited over 200,000 times and visited over 16 million times. The game is a military style, multiplayer game and allows people to move around air, land, and sea, get into vehicles like tanks and planes, use different weapons such as machine guns or rifles, and try to win battles. Basically, an entire 3D game platform and physics engine has been exposed to teenagers (and others) to develop on and use for free!
The engineering team at ROBLOX has been kind enough to share architecture insights publicly, and we wanted to pass along some highlights and links. In one article, they explained how they began with a simpler architecture and evolved (see image to the right).
As they grew, there were more inbound requests, more things they wanted to measure, and more engineers contributing. As well, they wanted to decrease disruptions, data loss, and infrastructure costs. To give an idea of the scale, on April 12, 2012, they cited roughly 75,000 database requests per second, 30,000 web-farm requests per second, and a petabyte per month of CDN traffic. Needless to say, their architecture today (below) looks much different and was driven mainly by needs around scale, cost, and monitoring.
Hybrid Clouds and Queues
“As we start interacting with our newer cloud services at Amazon, we’re initially using RabbitMQ to protect our internal systems against the intermittent high latencies of our remote systems; We’ll soon be expanding this throughout our back-end to decouple our sub-systems more effectively.”
To dive deeper, ROBLOX implemented RabbitMQ to help deal with the volume of database requests and slow response times. The queue is managing 15 million requests a day. The example scenario is when there is an update to the ROBLOX catalog content, which needs to update the search index. Before RabbitMQ, the web server would have to wait for an update to complete. Now, the loosely coupled architecture allows the web server to update the message queue, not the database, and move on (technically, the update is enqueued and picked up by the consumers). At some point, the message is picked up by a service that updates their search platform index.
The message queue acts as a simple, generic component – it receives messages from producers, stores them, and delivers them to consumers. Roblox goes into more detail on how they use RabbitMQ in a clustered set-up on Linux, how exchanges and queues are set-up, and why they made certain decisions, including why they selected RabbitMQ:
“RabbitMQ provides a lightning-fast server that can be clustered, client libraries in many languages, an administration API and a slick web dashboard. It’s also free (using open protocol AMQP), fast, cross-platform, stable (based on Erlang’s Open Telecom Platform), field-tested and supported by a large, active community.”
>> For more information on VMware’s vFabric RabbitMQ, you can: