Keeping background workers' data always up to date

Let's say we have an app which receives webhooks from a payment provider.

A customer starts a payment, so we receive a payment.processing event. A moment later the payment succeeds and another payment.succeeded event arrives. We don't want to do any heavy work while handling a webhook, so we launch a background worker and return 200 as soon as possible.

It sounds simple... until both workers run at the same time.

A rich payload

Here's how the webhook controller may look.

# app/controllers/payment_webhooks_controller.rb

class PaymentWebhooksController < ApplicationController
  def create
    event = webhook_event

    SyncPaymentWorker.perform_async(
      event.payment_id,
      event.status,
      event.amount_cents
    )

    head :ok
  end
end

The controller doesn't query or update anything. It just passes the event data to the worker and returns a successful response.

The worker may then use this payload to update our payment.

# app/workers/sync_payment_worker.rb

class SyncPaymentWorker
  include Sidekiq::Worker

  def perform(payment_id, status, amount_cents)
    payment = Payment.find_by!(provider_id: payment_id)
    payment.update!(status: status, amount_cents: amount_cents)

    Accounting::PaymentSync.call(payment)
  end
end

The worker gets all data it needs, so it doesn't have to call the payment provider. Fewer requests sounds like a good thing, right?

The problem is that a background job doesn't have to start just after we schedule it. It may wait for a free thread, another job may be retried, or two workers may pick jobs in a different order.

We can end up with the following flow:

1. The processing webhook schedules worker A.
2. The succeeded webhook schedules worker B.
3. Worker B runs first and sends succeeded to our accounting service.
4. Worker A runs later and sends processing.

At this point the payload of worker A is already outdated. The payment provider says that the payment is succeeded, but the worker doesn't know that. It has a snapshot of how the payment looked when the event was created.

Retries make it even more visible. A job may run a few minutes later with data which was correct only for a fraction of a second.

Pass the ID instead

What we want is to process the current state of the payment, not the state from the moment when the worker was scheduled.

Instead of passing all payment data, we can pass just its ID.

# app/controllers/payment_webhooks_controller.rb

class PaymentWebhooksController < ApplicationController
  def create
    event = webhook_event

    SyncPaymentWorker.perform_async(event.payment_id)

    head :ok
  end
end

The controller still doesn't perform any queries or updates. The worker is now responsible for fetching the current object from the payment provider and updating our database.

# app/workers/sync_payment_worker.rb

class SyncPaymentWorker
  include Sidekiq::Worker

  def perform(payment_id)
    provider_payment = PaymentProvider::Client.fetch_payment(payment_id)
    payment = Payment.find_by!(provider_id: provider_payment.id)

    payment.update!(
      status: provider_payment.status,
      amount_cents: provider_payment.amount_cents
    )

    Accounting::PaymentSync.call(payment)
  end
end

Now both workers fetch the payment when they actually start. In our example, the current status at the payment provider is succeeded, so both of them will use succeeded - no matter which job was scheduled first.

This gives us a few useful things:

• the job payload is small,
• the payment provider remains the source of truth,
• retries don't keep using an old snapshot,
• adding another field doesn't require changing worker arguments everywhere.

We don't need to predict which fields the worker may need in future. We just need an ID which lets it find the object.

A few things to remember

Fetching the object in the worker solves the stale payload problem, but it doesn't solve every problem around background processing.

Two workers may still process the same succeeded payment, so the operation should be idempotent. A record may also be deleted before the job starts, so we need to decide if a missing payment should be retried or ignored.

Webhooks themselves may arrive out of order too. Fetching the current payment from the provider helps us here because we don't rely on the state stored in a given event.

Of course, calling an external service may fail. The job should be retried when the provider is temporarily unavailable.

And sometimes we really want to process a historical snapshot. For example, an audit event should describe exactly what happened at a given moment. Then I'd persist that event and pass its ID to the worker.

The rule I find useful is simple: if the worker should act on the current object, pass the ID and fetch the object inside the worker.

One extra request is usually a small cost for working with the right data. Since it's being done in a background job, we don't care if it takes a bit longer.