Sidekiq Batch Complete Callback

Scenario

This week, I got stuck with an issue where race condition was happening between jobs completion and complete callback of Sidekiq batch. One more situation added to this was manually retrying the jobs(don't take me wrong but my job logic was in such a way that i need to do it) instead of the sidekiq retry options.

I assume the race condition was because of mainly below 2 reasons.

• There were retry jobs in the queue but my complete callback has a limitation that it gets triggered if at least the jobs in the batch have executed at least once.
• The other thing was the retry jobs weren't executed in the same batch.

Solution

So started digging for the solution, I found one, That made me quite exciting as it was much simple solution, where one can open the batch at any time and add the jobs in the same batch like extending the batch by adding the jobs into it.

So basically this does the complete callback to be triggered when all the retry jobs being completed and the whole problem looked very simple like making a cup of coffee. Thought of sharing over here, So if someone gets stuck the same can find a solution right way.

Creating the Sidekiq Batch and adding jobs into it

batch = Sidekiq::Batch.new
batch.description = "Upload the spreadsheet"
batch.jobs do
 SpreadsheetWorker.perform_aysnc(data, batch.bid)
end

Scheduling the job and adding retry jobs in the batch

class SpreadsheetWorker
  include Sidekiq::Worker

  def perform(row, batch_id)
   begin
   # Do your stuff

   rescue
    batch = Sidekiq::Batch.new(batch_id)
    batch.jobs do
      SpreadSheetWorker.perform_aysnc(row, batch_id)
      # SpreadSheetWorker.set(queue: 'batch_callback')perform_async(row, batch_id)
    end
   end
  end
end

One more interesting or flexibility that Sidekiq gives to us is, Overriding the Sidekiq queue and assigning to different queue when scheduling the job like below.

SpreadSheetWorker.set(queue: 'batch_callback')perform_async(row, batch_id)

This makes my retry jobs to execute in a different queue to save us a time to complete the batch and trigger callback instead of waiting in the same queue as previously scheduled jobs and wait for its turn to be get executed.

So this was for today. Will meet you again with one more issue I encountered next week.

Source: Github

Auf wiedersehen!!!