Ruby background processing worker error handling comparison

How do Ruby background job processing gems do their error handling?

I have had to look into this, as I have my own background job gem called Belated, so I want to know a good way to do it in my own gem.

A word about Belated: it runs in a different process, but uses dRuby to connect your main app and Belated.

So, currently, in Belated I am just rescuing StandardError.
But what happens if someone passes a job with perform method like below? (failing RSpec test)

it 'will not crash with syntax errors' do
  expect { 
    @worker.job_list.push(
      proc { raise SyntaxError }
    )
    sleep 0.01
  }.not_to raise_error
end

This raises an error (SyntaxError: compile error).

So let's look at how other job libraries deal with this!

Sidekiq

Sidekiq is pretty much the de-facto job processing gem for Ruby. At my day job, we depend on Sidekiq to run business critical jobs having to with live streams.

Sidekiq seems to rescue Exception, but it also has a manager class to handle these exceptions:

def run
  process_one until @done
  @mgr.processor_stopped(self)
rescue Sidekiq::Shutdown
  @mgr.processor_stopped(self)
rescue Exception => ex
  @mgr.processor_died(self, ex)
end

If you rescue Exception only, you end up rescuing too much, so Sidekiq is also taking care of properly handling shutdown by rescuing Sidekiq::Shutdown first.

A very clever approach with really nice logging too!

Source

Resque

Resque, like Sidekiq, needs Redis to function. It's an older gem, still in use in many places, but Resque seems to be quite a bit slower than Sidekiq. (See this article to learn more)

Resque also rescues Exception, and then checks if that exception is shutdown.

def work(interval = 5.0, &block)
  interval = Float(interval)
  # skipping a lot of code here...
rescue Exception => exception
  return if exception.class == SystemExit && !@child && run_at_exit_hooks
  log_with_severity :error, "Failed to start worker : #{exception.inspect}"
  unregister_worker(exception)
  run_hook :worker_exit
end

I have to say I'm not completely sure how this is working, but the gist of it should be that in case there is something so wrong that we get a syntax error the worker will shutdown.
Correct me if I'm wrong!

Source

GoodJob

GoodJob is an interesting new gem that uses PostgreSQL to handle connecting the processes.

It seems to just rescue StandardError.

def execute
  # code that executes the job
  rescue StandardError => e
    ExecutionResult.new(value: nil, unhandled_error: e)
  end
end

This makes handling shutdown much easier.
GoodJob, like Sidekiq, allows you to have an error handler that sends your errors to a service like Airbreak, or maybe emails them to you using an exception handling gem.
But something like a SyntaxError will probably not get reported. (Probably quite a rare case though... and feel free to correct me if I'm wrong!)

Source

Conclusion

So... what to do with Belated?
I think the right thing would be to just rescue Exception, but when you rescue exception, you have to also take into account shutdown. You don't want to make it impossible to actually close the program with CTRL + C.

Sidekiq's approach looks really promising, but as excepted, also the most demanding one. So something like it... but simpler. For now, Belated will just crash if you feed it a proc with a syntax error in it.

And here's what I ended up doing:

rescue Exception => e
  case e.class
  when Interrupt, SignalException
    raise e
  else
    e.inspect
  end
end

Just check if I actually want to rescue the error. Normally rescuing Exception is overdoing it, and you get a Rubocop warning for it too, but in a background job process you want to keep the worker working even if there is a syntax error in the job is processing.