Starling + Workling: Simple Distributed Background Jobs with Twitter's Queuing System
Location: Saal Maritim A Audience level: Intermediate
Web applications often need to do work which does not belong in the request / response cycle. Photo resizing or API calls to other services block the request, slowing things down uncessarily. Long running tasks will make your user application unresponsive and your users impatient, or even destabilize your environment. The solution is to take these tasks out of the request cycle.
Getting started with backgrounding should be really simple. Traditionally, backgrounding was done in Rails applications through use of BackgroundRB. But BackgroundRB has gained a bad reputation over time, because of Drb issues and its general weightiness. Although BackgroundRB was recently completely re-written, it is still only one of many options available at the moment. Unfortunately, it’s not really clear which option should be used.
In this talk I will survey what’s currently available. I will look at the queue based systems Starling and BeanstalkD, and also at systems that achieve a similar effect by using the database and threading or forking, BackgroundJob and spawn. Finally I will look at Workling, wich can combine all these approaches under a very simple roof.
The main focus will be on Starling, Twitters inhouse message queue, released by Blaine Cook and team a few months ago. Starling is a lightweight persistent message queue which speaks memcache. It is used at Twitter to distribute work such as sending SMS. I will look at how to use Starling, as well as differences in the evented version of Starling.
Next, I will discuss Workling, which seamlessly integrates Starling into your Rails app. I will show how Workling and Starling can be used to build a backgrounded addressbook crawler with an ajax progress indicator wich runs the worker code on a remote machine.
Finally I will demonstrate how the same code can be configured so that it runs locally with Spawn or also with BackgroundJob.
rany keddo is a rails hacker at the german mini-startup play/type. he’s working on the rather good event platform boomloop.com. he’s been a full time rails lover since early 2006, when he built one of the first rails applications at a bank.