DEV Community

Cover image for Webhook Ordering, Version Guards, and the Idempotency Key
Son Do
Son Do

Posted on

Webhook Ordering, Version Guards, and the Idempotency Key

#rails #python #backend #architecture

MyCase Immigration / Docketwise Integration: 0: Why We Synced · 1: Before/After · 2: OAuth2 · 3: Sync vs Migration · 4: Sync Boundary · 5: Family Graph · 6: Write Patterns · 7: Webhooks · 8: Billing

The failure did not look like a failure. That was the problem.

When an attorney queried a case in staging, the fields were populated. The record looked valid. The status was wrong, but wrong in a way that was plausible. An integration test that checked for the presence of values rather than the correct values would have passed.

We found it because one of our staging integration tests submitted two rapid updates to a Docketwise case in sequence and checked that the final state in MyCase matched the second update, not the first. It did not. It matched the first.

This post covers the failure scenario, the two separate bugs it exposed, and the fixes.

The failure scenario

The sequence in staging:

  1. An attorney updates a case status in Docketwise. Docketwise fires Webhook A with the new status and a docketwise_version value of 42.
  2. Immediately after, a second update fires. Docketwise updates the same case with a new assigned party. Docketwise fires Webhook B with docketwise_version 43.
  3. Webhook B arrives at MyCase's webhook receiver first. (Network timing in staging introduced this out-of-order delivery.) MyCase applies the version-43 state. The case now has the correct party update.
  4. Webhook A arrives at MyCase. MyCase applies the version-42 state. The case status is updated from the version-42 payload. But the version-42 payload reflects the full case state as of version 42, which does not include the version-43 party change. Applying it overwrites the party assignment from step 3.

Webhook ordering failure: Docketwise fires Webhook A (v42) then Webhook B (v43); B arrives at MyCase first and is applied correctly; A arrives second and overwrites the v43 state with v42 state; final MyCase record reflects v42 not v43

The result: the MyCase case record reflected the state from the first (older) update, not the second (newer). The field values were plausible because version 42 was a real case state, just not the current one. There was no error, no exception, no alert. The MyCase record was silently rolled back.

Why this is hard to detect in normal operation

In normal usage, Webhook A arrives before Webhook B almost all of the time. The sequence follows the fire order. Webhook delivery systems do not guarantee ordering but they tend to deliver roughly in sequence under normal network conditions.

The failure requires two conditions to occur together: a short interval between two updates on the same record, and out-of-order delivery of the resulting webhooks. Under normal usage, these conditions are rare. In staging, we induced them by submitting two updates 50 milliseconds apart and relying on the test environment's network variability to shuffle the delivery order.

We would not have found this in production without deliberate testing for it. Production ordering failures would have been subtle and rare: a case status rolling back by one state on the occasional day when Docketwise fired two updates in quick succession. No alert, no visible error, just a case that looked slightly wrong with no obvious cause.

Fix 1: the version guard

The version guard: before applying a webhook payload, check whether the payload version is greater than the version already stored on the MyCase record. If not, discard the event.

Docketwise includes a version number in every webhook payload. This is a monotonically increasing integer on each case record. When Docketwise fires a webhook for a case update, the payload contains the case state at that version.

MyCase stores the last-applied Docketwise version on each local Case record in a docketwise_version column. The webhook receiver checks:

def receive_case_webhook(params)
  case_record = Case.find_by!(docketwise_case_id: params[:case_id])

  if params[:version] <= case_record.docketwise_version
    head :ok
    return
  end

  case_record.update!(
    docketwise_status: params[:status],
    assigned_party_id: params[:assigned_party_id],
    docketwise_version: params[:version]
  )

  head :ok
end
Enter fullscreen mode Exit fullscreen mode

If the incoming version is not greater than the stored version, the receiver returns 200 (so Docketwise does not retry) and does not apply the payload. If the incoming version is greater, it applies the payload and updates the stored version.

After applying this fix, the staging test scenario resolves correctly. Webhook B (v43) arrives first. MyCase applies it, stores version 43. Webhook A (v42) arrives. The version guard fires: 42 is not greater than 43. The event is discarded. The case stays at version 43.

The version guard uses the version field on the entity as the guard. There is no separate deduplication table. The version column that already exists for sync tracking serves as the ordering guard.

Version guard fix: webhook receiver compares incoming version against stored docketwise_version; if incoming is not greater, discard and return 200; if greater, apply payload and update stored version; out-of-order Webhook A (v42) is discarded after Webhook B (v43) has already been applied

Fix 2: the idempotency key

The version guard addresses ordering. It does not address the separate problem of the same webhook arriving twice.

Docketwise, like most webhook delivery systems, provides at-least-once delivery. If the MyCase receiver returns a non-2xx status (due to a transient error, a timeout, or a crash mid-processing), Docketwise retries. The retry sends the same webhook with the same event ID and the same payload.

Without an idempotency check, a retry that arrives after the original was already processed would apply the same payload twice. For most field updates, applying the same state twice is idempotent by accident: setting a field to the same value produces the same result. But for state machine transitions or aggregated fields, applying the same payload twice could produce incorrect state.

The fix: Docketwise includes a unique event ID in each webhook payload (a UUID, stable per webhook firing). MyCase stores the last applied event ID on each Case and Contact record.

The webhook receiver checks before the version guard:

def receive_case_webhook(params)
  case_record = Case.find_by!(docketwise_case_id: params[:case_id])

  if params[:event_id] == case_record.last_docketwise_event_id
    head :ok
    return
  end

  if params[:version] <= case_record.docketwise_version
    head :ok
    return
  end

  case_record.update!(
    docketwise_status: params[:status],
    assigned_party_id: params[:assigned_party_id],
    docketwise_version: params[:version],
    last_docketwise_event_id: params[:event_id]
  )

  head :ok
end
Enter fullscreen mode Exit fullscreen mode

If the event ID matches the stored ID, the receiver returns 200 and discards. If it does not match, it proceeds with the version guard check, applies the payload, and stores the new event ID alongside the new version.

Event ID check and version guard solve different problems. The event ID check handles retries of an already-processed event (same version, same content). The version guard handles out-of-order delivery (different versions, different content). Both are required; neither replaces the other.

The staging test that caught it

The integration test that exposed the ordering bug:

it 'applies the most recent Docketwise state when webhooks arrive out of order' do
  case_record = create(:case,
    docketwise_case_id: 'dw-case-1',
    docketwise_version: 41,
    last_docketwise_event_id: nil
  )

  webhook_a = build_webhook_payload(
    case_id: 'dw-case-1',
    status: 'submitted',
    event_id: 'evt-001',
    version: 42
  )
  webhook_b = build_webhook_payload(
    case_id: 'dw-case-1',
    status: 'pending_review',
    assigned_party_id: 'party-abc',
    event_id: 'evt-002',
    version: 43
  )

  # Deliver out of order: B first, then A
  post '/webhooks/docketwise', params: webhook_b
  expect(response).to have_http_status(:ok)

  post '/webhooks/docketwise', params: webhook_a
  expect(response).to have_http_status(:ok)

  case_record.reload
  expect(case_record.docketwise_status).to eq('pending_review')
  expect(case_record.assigned_party_id).to eq('party-abc')
  expect(case_record.docketwise_version).to eq(43)
end
Enter fullscreen mode Exit fullscreen mode

Before the version guard, this test failed: docketwise_status was submitted (from Webhook A, which arrived last and was applied on top of B). After the version guard, the test passed.

The idempotency test was a second scenario: same webhook delivered twice, state applied once.

it 'does not re-apply a duplicate webhook' do
  case_record = create(:case,
    docketwise_case_id: 'dw-case-1',
    docketwise_version: 41
  )

  webhook = build_webhook_payload(
    case_id: 'dw-case-1',
    status: 'approved',
    event_id: 'evt-003',
    version: 42
  )

  post '/webhooks/docketwise', params: webhook
  original_updated_at = case_record.reload.updated_at

  post '/webhooks/docketwise', params: webhook
  expect(case_record.reload.updated_at).to eq(original_updated_at)
end
Enter fullscreen mode Exit fullscreen mode

The updated_at check verifies the record was not touched on the second delivery.

These two tests together define the behavior the receiver must produce. The version guard handles ordering. The event ID check handles duplicates. Writing both tests before writing the fix made the required behavior explicit and confirmed each fix in isolation.

Next: Part 8. Generalizing Addon Billing Without Touching Stripe or Zuora. The billing lifecycle for the Immigration addon required a refactor that touched the billing service without touching either payment provider. This post covers what was hardcoded, what changed, and the Stripe soft-deactivation pattern that preserved the customer reversion path.

Top comments (0)