Hoptoad + delayed_job
Last week one of our applications unfortunately got a little confused for a bit due to an exception within a delayed_job process. The worst part about it was we weren't made aware of it immediately because our favorite error reporting service Hoptoad (written by the folks at thoughtbot) is not aware of exceptions within a job. Delayed_job silently fails and then based on the Delayed::Worker.destroy_failed_jobs value either keeps the jobs in the database or destroys them.
So how do we keep an eye on our jobs? Delayed_job does a pretty good job of logging everything that happens, so you could certainly write something to keep an eye on the log file. However that seemed a little crazy to me and besides I wanted the Toad to know. In the collectiveidea fork of delayed_job there is a method named handle_failed_job. This does what you would expect. To hook into that we can alias that method in an initializer and send our error to Hoptoad.
The code is pretty self explanatory. We alias the method, call HoptoadNotifier.notify, and then perform the original delayed job method.
Here are the tests.
Now Hoptoad and delayed_job are good friends! Having good test coverage with the right kinds of tests will certainly keep those pesky bugs away, but sometimes things break. When they do it's critical to be able to respond to them quickly and accurately so you can get that hotfix deployed straight away. You don't know what you don't know so having something as critical as your background processes running without error reporting is not exactly safe. We learned our lesson and quickly implemented a simple yet solid solution. Hope this helps the next person.