Using Node to sling high volume requests into Redis as Resque jobs

A client of ours has a Rails application that needs to consume hundreds of requests per second from a third party service, with flexibility to scale upwards to thousands of requests per second. Rails just isn't well suited to handle such a high volume of requests.

Node.js, on the other hand, is a great tool when it comes to doing a simple task very fast and in high volumes. It scales easily to boot.

Aside from consuming the request, each request needs to be processed individually, and at that volume, a queue was necessary. Resque was an easy choice.

Read More

Heroku’s Cedar Stack will kill your Resque workers

Heroku is an invaluable resource for quickly deploying apps without having to do the dev-ops heavy lifting, allowing development to move fast. One of Heroku's great features is on-demand scaling of dynos (additional processes). To take advantage of those dynos, there are many job queueing gems available, the most popular being Resque. Heroku's ease of scaling up dynos combined with Resque's ease of using those dynos make the two a match made in heaven. Unfortunately, that's not the case.

So what's the problem?

The issues that we've encountered lie within Heroku's Cedar Stack, the most popular stack for Rails apps (now Heroku's new default!). Simply put, Heroku will kill your Resque workers.

Don't let the kitten die.

Read More