DEV Community

Wamci
Wamci

Posted on

7

Sidekiq Batch Complete Callback

Scenario

This week, I got stuck with an issue where race condition was happening between jobs completion and complete callback of Sidekiq batch. One more situation added to this was manually retrying the jobs(don't take me wrong but my job logic was in such a way that i need to do it) instead of the sidekiq retry options.

I assume the race condition was because of mainly below 2 reasons.

  • There were retry jobs in the queue but my complete callback has a limitation that it gets triggered if at least the jobs in the batch have executed at least once.
  • The other thing was the retry jobs weren't executed in the same batch.

Solution

So started digging for the solution, I found one, That made me quite exciting as it was much simple solution, where one can open the batch at any time and add the jobs in the same batch like extending the batch by adding the jobs into it.

So basically this does the complete callback to be triggered when all the retry jobs being completed and the whole problem looked very simple like making a cup of coffee. Thought of sharing over here, So if someone gets stuck the same can find a solution right way.

Creating the Sidekiq Batch and adding jobs into it

batch = Sidekiq::Batch.new
batch.description = "Upload the spreadsheet"
batch.jobs do
 SpreadsheetWorker.perform_aysnc(data, batch.bid)
end

Scheduling the job and adding retry jobs in the batch

class SpreadsheetWorker
  include Sidekiq::Worker

  def perform(row, batch_id)
   begin
   # Do your stuff

   rescue
    batch = Sidekiq::Batch.new(batch_id)
    batch.jobs do
      SpreadSheetWorker.perform_aysnc(row, batch_id)
      # SpreadSheetWorker.set(queue: 'batch_callback')perform_async(row, batch_id)
    end
   end
  end
end

One more interesting or flexibility that Sidekiq gives to us is, Overriding the Sidekiq queue and assigning to different queue when scheduling the job like below.

SpreadSheetWorker.set(queue: 'batch_callback')perform_async(row, batch_id)

This makes my retry jobs to execute in a different queue to save us a time to complete the batch and trigger callback instead of waiting in the same queue as previously scheduled jobs and wait for its turn to be get executed.

So this was for today. Will meet you again with one more issue I encountered next week.

Source: Github

Auf wiedersehen!!!

Image of Datadog

The Essential Toolkit for Front-end Developers

Take a user-centric approach to front-end monitoring that evolves alongside increasingly complex frameworks and single-page applications.

Get The Kit

Top comments (0)

Billboard image

The Next Generation Developer Platform

Coherence is the first Platform-as-a-Service you can control. Unlike "black-box" platforms that are opinionated about the infra you can deploy, Coherence is powered by CNC, the open-source IaC framework, which offers limitless customization.

Learn more