Fork me on GitHub!
Señor Developer!


Ruby User Group Berlin

Ruby Storm - Distributed work on a self-scaling system using RabbitMQ

this Topic will be presented by stevecreedon

at March Meetup 2014, hosted by Tobias Pfeiffer

Some time ago we reduced a 27 hour process to less than 20 minutes by distributing the workload using RabbitMQ. The system has evolved into something akin to Java Storm but in Ruby. It self-scales using our own scaling logic that analyses queue sizes, deploys additional workers where needed then adds servers there aren't enough for the extra workers - and scales down too of course.

The system is agnostic of the task at hand. At the moment we use it update pricing, property and availability at Wimdu but really it could be used for any large data crunching task and writing ruby workers is very easy.

NOTE: I won't be back in Berlin until 10th Jan so will have to wait until Feb talks at earliest

8 People like it