Social Media is taking over my whole life. Day or night, by working with great products from VMware, EMC and now Pivotal, it seems like hundreds of people are constantly talking about my topics. It’s becoming impossible to keep up.
That’s exactly what Santi Camps is thinking too. Camps is leading a new project called Socialvane. The idea is to affordably help social media moguls, sales teams and community managers filter through the spiraling content across social media sites and focus on the most important conversations. He’s developed some sophisticated artificial intelligence too that he’s baked into the tool so that it learns and helps zero in on the most profitable conversations and insights.
The software, due to be published in four languages (English, Spanish, Catalan and Portuguese), is still not publicly available, but development is well under way.
Last week, I got the chance to talk to Camps about the development efforts and some of the choices he’s made. Aside from the fancy artificial intelligence and learning he built into the platform, many of the requirements Camps has sounds like those of social media platforms: it needs to scale to handle massive amounts of transactions and data.
Camps, who has the role of CEO and lead developer, chose to start building the app with Python using the Django web framework. Built as a loosely coupled system, Socialvane consists of several different types of nodes across scalable clusters. These nodes are connected by Celery as a task manager and RabbitMQ as the task broker.
Q: Can you describe a typical transaction across your application?
A: Our product is meant to be an assistant for community managers who struggle to sort through all the information on social networks. So, if a Community Manager decides they want to target a specific keyword such as a product name or a event hashtag, they can put that word into Socialvane and it will go fetch all the instances of that keyword and push it through the artificial intelligence node and then present recommendations of the most interesting posts to the user or automatically trigger actions within a few seconds.
Behind the scenes this means the Python app sends a task to the task queue in RabbitMQ. Then Celery retrieves the task and spawns workers to complete the task. When the worker is finished with its task, it sends the results to the Python app for storage, analysis and presentation to the end user.
Q: Why did you choose Python, Django, Celery and RabbitMQ?
A: I am a Python developer to start and Django is a great framework that just helps to build complex applications faster. From there, we knew we needed to do distributed fetches across social media sites and process a lot of information very quickly. So we needed to set up a queue system and a task manager. Celery is the one that is recommended for Python and Django, and RabbitMQ is the one recommended on Celery’s website. So we tried those first and they just worked so we never needed to question it.
Q: Was this your first time using Celery + RabbitMQ?
A: Yes, it was in fact. It was really easy though. Both have great communities and documentation, and it didn’t take us very long to set it up. Out of the box they seem to scale really well, so we haven’t run into many scenarios where we need to do much configuration or tuning to get it to work. One of the other things I really liked was the ability to have multiple queues with different priorities. This allows us to have priority user facing tasks clear first and place more background tasks farther down the line in priorities. The coding for it was pretty simple too—all I needed to do was break up the types of activities and assign them a type like so:CELERY_QUEUES = {'fast': {'exchange': 'fast', 'exchange_type': 'direct', 'binding_key': 'fast_task'},'slow': {'exchange': 'slow', 'exchange_type': 'direct', 'binding_key': 'slow_task'}}CELERY_DEFAULT_QUEUE = 'fast'CELERY_ROUTES = {"facebook_get_mentions": {"queue": "slow", "routing_key": "slow_task"},"facebook_get_mentions_deep": {"queue": "slow", "routing_key": "slow_task"},"facebook_get_timeline": {"queue": "slow", "routing_key": "slow_task"},…
This is one trick to ensure the application performance at least from the end user perspective always meets service levels.
Q: Based on your experience, would you choose these products again for another project?
A: Right now we are focused on this one project, but it is easy to see that we could expand to have other specialty products around newer social media channels in the future and I believe we will continue to use this framework including Celery and RabbitMQ.
Another satisfied RabbitMQ user doing cool stuff. Hopefully, Socialvane, when its ready will help me reclaim part of my day. Until then, I am pre-registering for the beta.
 |
About the Author: Based in Spain, for the past 15 years, Santi Camps has worked as software developer focusing on web based applications. Starting as programmer at Capgemeni and working up to a development manager for open source project KMKey, Camps has worked in a variety of environments including VAX-VMS, UNIX, and Windows. Recently, Camps has focused more on open source software environments, mostly running on GNU/Linux. but last years I’ve used just GNU/Linux and free software environments. Today, he is working as the founder and CEO of his own company, Socialvane, that helps small and medium companies to get profit from Social Media & Marketing 2.0. Follow him at @santicamps. |
