Gotcha: Starting Jobs inside Transaction

Starting jobs inside an Active Record Transaction can result in unexpected outputs. I've seen this happen multiple times in my career.

Consider an example where we update a model and then send a third party API the updated model

ActiveRecord::Base.transaction do
   david.withdrawal(100)
   mary.deposit(100)
   NotifyUser.perform_later david.id # Job to notify User
end

Lets assume that NotifyUser sends a user their remaining balance in account.

class NotifyUser < ApplicationJob
   queue_as :default
   def perform(user_id)
     user = User.find(user_id)
     ThirdPartyNotifier.notify(user)
   end
end

So you run this code in your local and it seems to work just fine. Money is withdrawn and the notification sends appropriate balance of the user. But alas, Later you find a bug in production, and weirdly sometimes the notification is correct, sometimes it not.

The reason? The job sometimes runs before the transaction can even complete. Since we queue the job inside the transaction, there is a chance that the Job gets picked up before changes are committed to the database. The job then has old data. If you want to make sure the job send incorrect data every time in local add a sleep(10) after NotifyUser. This will ensure that job gets picked up before the transaction completes.

The fix for this case is quite straightforward. Move the job call outside of the transaction

ActiveRecord::Base.transaction do
   david.withdrawal(100)
   mary.deposit(100)
end
NotifyUser.perform_later david.id # Job to notify User

Comments

[Max Chernyak]

A note for future readers: this gotcha only occurs if your jobs are in a separate database/connection from your ActiveRecord. Like, if you're using Sidekiq with Redis, while your ActiveRecord is on Postgres. That's because the job, once sent to Redis, can now be executed before transaction is finished.

If you're keeping your jobs in the same db (e.g. GoodJob with Postgres) this shouldn't be a problem, because the job is subject to the same transactional restrictions as the other data. A worker will not be able to find this job until the transaction finishes, which is when it will find the other data too. So in this setup, kicking off a job from a transaction is probably recommended.