Prevent Whenever scheduled cron jobs to run simultaneously in Rails.

In Simply.TV we have a number of cronjobs running to surveil some of our internal processes.

These jobs are scheduled to run every single minute using the whenever gem and its runner job type.

We have a job scheduled to run every single minute:

every 1.minute do
  runner 'Parse.detect_crashed_jobs'
end

It's using the rails runner command to execute the method Parse.detect_crashed_jobs and is doing that every minute.

Unfortunately some times our load rises on the server for various reasons and there's a chance that one of these jobs ends up running for more than a minute. This can result in the same job being scheduled to run simultaneously as the crontab will just keep processing the job every minute.

Enter flock. flock is shipped with most Linux distributions, at least this is tested on Linux Ubuntu 16.4 and upwards.

I've introduced a new job type and template to the whenever config/schedule.rb file:

job_type :runner,  "cd :path && bin/rails runner -e :environment \":task\" :output"
set :job_template, "bash -l -c 'flock -n '\\''/tmp/:task-runner.lockfile'\\'' -c '\\'':job'\\'''"

The job_template is using the bash command exactly like whenever defaults, but I've wrapped the actual job execution in a flock command. flock writes a temp file (using the task as filename) and uses this temp file to track if the job is running already. If it's already running it's skipping execution and exits.
At the same time I'm also changing the job_type for runner slightly by wrapping the task in double quotes instead of single quotes which is whenevers default value. This is to allow the job_template to wrap the flock command in single quotes. It just makes everything easier.

Now whenever a job is scheduled using whenever via the runner method, the job will be executed through flock who'll ensure the same job isn't running twice.