Scheduling Jobs with DelayedJob

For a new project at work, I was tasked with reevaluated our async worker system. Previously we’d been using beanstalkd with a jobs model that used AASM for persistence and kept up with when something should be scheduled (and was scheduled with a rake task). After looking at the different things available, I decided DelayedJob was a better way to go, because it involved fewer dependencies (no more cron-rake work, no more beanstalkd) and had the persistence built in, instead of our custom code.

One shortcoming, though, seemed to be scheduling jobs. After thinking about different ways of keeping up with what jobs could be scheduled, I figured it’d make the most sense for jobs to just re-schedule themselves before execution.

That said, I didn’t want to put that in every class that had a perform() method as a boilerplate, “reschedule myself for 24 hours,” because it’s harder to maintain and grok what jobs are scheduled, and because it’s not nearly as readable.

Enter ScheduledJob, a mixin that gives you a run_every method on your class:

With this, you can create a class and include ScheduledJob, and then just say “run_every 24.hours” or whatever. Like this:

Hopefully this is helpful. I’ll need to write some tests, and see if the DJ guys want to use it on GitHub.

Async Observer sans Rails

My favorite background job pairing is beanstalkd with async_observer. I think the code that comes out is clean, easily pushed out to a queue, and beanstalkd is stupid easy to get going.

Both at work and on a side project, I’ve run into a situation where I need to put something into beanstalkd to be processed by a Rails-backed worker, but don’t want to load up the Rails framework to do that. Easy example: receiving email, I’d like to queue the processing how Jason Seifer suggests, but don’t want to have my own beanstalkd tube dedicated to email processing.

When you do an async_send, it just gets serialized as a string out to beanstalkd, with something like Model.find(id).method()

So if you want to push stuff into your queue in a rake task without having to load up the Rails environment, just use this method:

At work, we have a job system, and cron runs every 2 minutes to find scheduled jobs that should run and put them into the queue. Initially this was coded as a Rails-backed rake task, which meant every 2 minutes you’d see a ruby process load up for 10-15 seconds and steal 95% of a CPU. As we had more applications on the same machine needing jobs loaded, this wasn’t effective. Making the task only use beanstalk-client (and mysql), the task now runs in <1 second and doesn’t even register on top.