Deterministic Reordering of Concurrent Webhooks: Handling Out-of-Order State Transitions with CRDTs

Introduction: The Multi-Region Webhook Delivery Dilemma

Webhooks are the default way for SaaS providers to push state changes to your systems. Once you accept them through several regional ingress points, you inherit a hard problem: the order in which events arrive is not the order in which they happened.

Providers say this openly. Stripe states that it does not guarantee events arrive in the order they were generated, and that the same event can be delivered more than once. In live mode it retries failed deliveries with exponential backoff for up to three days. Its own example is a subscription creation that can emit customer.subscription.created, invoice.created, invoice.paid and charge.created, with no promise about their arrival order. Shopify and GitHub give similar warnings. GitHub says deliveries can arrive in a different order than the events occurred. It also does not automatically redeliver failed deliveries, so you must redeliver them manually. Analyses of dozens of webhook platforms find the same pattern: most providers disclaim ordering without supplying the metadata a consumer needs to rebuild it.

Deploying ingress in several regions adds more reordering. Different network paths, queue workers that finish jobs in a different order than they started them, and retries all interleave events for the same resource.

Retries also mean you cannot get exactly-once delivery. If a sender never retries, it loses data when a delivery fails. If it does retry, it will occasionally duplicate events that had already arrived. The practical target is therefore at-least-once delivery plus an idempotent, order-insensitive consumer.

This article shows one way to build such a consumer. It combines Conflict-Free Replicated Data Types (CRDTs) with logical timestamps and a deterministic tie-breaker, so that every replica converges to the same state no matter how events are interleaved. It also covers when this design is overkill.

1. The Anatomy of Out-of-Order Webhook Failures

HTTP itself gives you no delivery guarantee. Reliability comes from the provider’s retry logic, and that logic is the source of both duplicates and reordering.

Common Failure Modes in Multi-Region Ingress

1. The ghost update. An updated event arrives before the created event it depends on. A handler that does a strict update fails. An upsert that fills in missing fields with defaults or nulls creates a half-populated record.
2. State reversal. An older event is retried or delayed and arrives after a newer one has been committed. If the handler applies events in arrival order, it overwrites newer state with older state.
3. Concurrent mutation divergence. Two regions ingest different events for the same resource at nearly the same time and apply them in different orders. Without a merge rule that gives the same result regardless of order, the replicas diverge permanently.

Provider emits:  E1 (created, t=1)   E2 (updated, t=2)

                 E1 ── slow route / retry ──────────► Region B ── applies 2nd
                 E2 ── fast route ───────► Region A ── applies 1st

Region A sees: E2, E1     Region B sees: E1, E2
Arrival-order handling → A and B end in different states

A common first fix is Last-Write-Wins (LWW) on the payload’s wall-clock timestamp. That fix is only as good as the clocks behind it. If the timestamps come from machines with skewed clocks, or you compare them against your own server’s Date.now(), “latest” is not reliable. Plain LWW also silently discards one side of a true concurrent write. LWW is not inherently broken, though. As shown below, an LWW register with logical timestamps and a deterministic tie-breaker is itself a valid CRDT.

2. Foundations: CRDTs and Strong Eventual Consistency

CRDTs were formalized by Shapiro, Preguiça, Baquero and Zawirski in 2011. They defined Strong Eventual Consistency (SEC) and gave sufficient conditions under which replicas of a data type are guaranteed to converge without rollback or coordination. SEC requires three things:

• Eventual delivery. An update applied at one correct replica is eventually applied at all correct replicas.
• Termination. Every update execution finishes.
• Strong convergence. Replicas that have applied the same set of updates are in equivalent states, regardless of order.

State-based CRDTs (CvRDTs)

A state-based CRDT models its state as a join-semilattice. This is a partially ordered set in which every pair of states has a least upper bound, and the merge function computes that bound (⊔). The merge must satisfy three laws:

• Commutative: A ⊔ B = B ⊔ A, so arrival order does not matter.
• Associative: (A ⊔ B) ⊔ C = A ⊔ (B ⊔ C), so batching does not matter.
• Idempotent: A ⊔ A = A, so duplicate deliveries are harmless.

Updates must also be monotonic: a state may only move up the lattice. This is why a plain counter cannot support decrement in a naive state-based design, since merging by max would ignore it.

Operation-based CRDTs (CmRDTs)

Operation-based CRDTs send operations instead of state. They are compact on the wire, but they push guarantees onto the transport: each operation must be delivered exactly once, and in causal order where the data type requires it.

That is the wrong fit for webhooks, which are at-least-once and unordered, and whose operation history you do not control. Webhook handlers should therefore treat each payload as a state fragment and merge it with a state-based join. State-based merges tolerate duplicate and out-of-order delivery by construction.

Logical Clocks

To order events without trusting physical clocks, use logical ones:

• Lamport timestamps (Lamport, 1978) guarantee that if event a causally precedes b, then L(a) < L(b). The converse does not hold, so they cannot tell you whether two events were concurrent.
• Vector clocks (Fidge and Mattern, 1988) do capture causality, so concurrent events can be detected. They cost one counter per participant.
• Provider-supplied versions are the best source of all when they exist: a monotonic sequence number per resource, an updated_at or version field on the object, or an explicit parent event reference.

If the provider supplies none of these, anything you stamp at the edge reflects arrival order at your edge, not the provider’s order. Edge stamping can still give you a consistent internal order for deduplication and replay. It cannot recover the provider’s true event order.

Deterministic Tie-Breaking

When two events carry the same logical timestamp, compare a fixed sequence of keys: (logicalTimestamp, actorId, eventId). Every replica applies the same comparison, so every replica picks the same winner. This works only if that tuple is unique per event. That is why the example below includes eventId as the final key.

3. Architecture of a Deterministic Reordering Layer

+--------------------------------------------------------------+
|                   Multi-Region Edge Ingress                  |
|   verify signature · return 2xx fast · persist raw event     |
+--------------------------------------------------------------+
                              │
                              ▼
+--------------------------------------------------------------+
|             Dedup + Metadata Extraction                      |
|   eventId · resourceId · logical timestamp / version · actor |
+--------------------------------------------------------------+
                              │
                              ▼
+--------------------------------------------------------------+
|                  CRDT State Resolution                       |
|   event → state fragment → join (⊔) with stored state        |
+--------------------------------------------------------------+
                              │
                              ▼
+--------------------------------------------------------------+
|        Visibility Gate (dependencies) + Reconciliation       |
|   hold side effects until `created` seen; TTL → API fetch    |
+--------------------------------------------------------------+
                              │
                              ▼
+--------------------------------------------------------------+
|                    Persisted State Storage                   |
+--------------------------------------------------------------+

1. Ingestion. Verify the signature, acknowledge with a 2xx quickly, and process asynchronously. Providers enforce delivery timeouts. For example, Primer documents a 10-second timeout. A slow synchronous handler therefore causes retries, which cause duplicates.
2. Deduplication. Store processed event IDs with a unique constraint. Stripe events carry a stable id for this purpose. Deduplication is not the same as domain idempotency, though. A provider can emit two distinct events that describe the same business transition. The idempotent join below handles that case as well.
3. Join. Convert the event into a state fragment and merge it into the stored state. Because the merge is commutative, associative and idempotent, arrival order is irrelevant to the final state.
4. Visibility gate. The join converges even when updated arrives before created. What you still need to hold back are side effects and foreign-key-dependent writes. Keep the merged state internal until the creation event has been seen. If it never arrives within a time limit, fall back to fetching the resource from the provider’s API.

4. Reference Implementation (TypeScript)

This sketch models a resource as a map of per-field LWW registers, plus a grow-only created flag and a delete tombstone. It is not a drop-in production system.

interface WebhookEvent {
  eventId: string;
  resourceId: string;
  eventType: 'CREATED' | 'UPDATED' | 'DELETED';
  logicalTimestamp: number;   // provider version/sequence, or a Lamport value
  actorId: string;            // issuing node/region, used for tie-breaking
  data: Record<string, unknown>;
}

interface Stamp { ts: number; actor: string; eventId: string }
interface Field { value: unknown; stamp: Stamp }

interface ResourceState {
  fields: Map<string, Field>; // per-field last-writer-wins registers
  created: boolean;           // grow-only flag: once true, stays true
  deleted?: Stamp;            // tombstone (delete-wins policy)
}

// Total order on stamps: (ts, actor, eventId). Unique per event.
const cmp = (a: Stamp, b: Stamp): number =>
  a.ts !== b.ts ? a.ts - b.ts
  : a.actor !== b.actor ? (a.actor < b.actor ? -1 : 1)
  : a.eventId < b.eventId ? -1 : a.eventId > b.eventId ? 1 : 0;

const EMPTY: ResourceState = { fields: new Map(), created: false };

// The join (⊔): commutative, associative, idempotent.
function merge(a: ResourceState, b: ResourceState): ResourceState {
  const fields = new Map(a.fields);
  for (const [key, fb] of b.fields) {
    const fa = fields.get(key);
    if (!fa || cmp(fb.stamp, fa.stamp) > 0) fields.set(key, fb);
  }
  let deleted = a.deleted;
  if (b.deleted && (!deleted || cmp(b.deleted, deleted) > 0)) deleted = b.deleted;
  return { fields, created: a.created || b.created, deleted };
}

function toFragment(e: WebhookEvent): ResourceState {
  const stamp: Stamp = { ts: e.logicalTimestamp, actor: e.actorId, eventId: e.eventId };
  const fields = new Map<string, Field>();
  for (const [k, v] of Object.entries(e.data)) fields.set(k, { value: v, stamp });
  return {
    fields,
    created: e.eventType === 'CREATED',
    deleted: e.eventType === 'DELETED' ? stamp : undefined,
  };
}

class WebhookReorderingEngine {
  private states = new Map<string, ResourceState>();
  private waitingSince = new Map<string, number>(); // local TTL bookkeeping only

  ingest(e: WebhookEvent, now: number = Date.now()): void {
    const merged = merge(this.states.get(e.resourceId) ?? EMPTY, toFragment(e));
    this.states.set(e.resourceId, merged);

    if (merged.created) this.waitingSince.delete(e.resourceId);
    else if (!this.waitingSince.has(e.resourceId)) this.waitingSince.set(e.resourceId, now);
  }

  // Visible only after `created` is known and while not tombstoned.
  read(resourceId: string): Record<string, unknown> | undefined {
    const s = this.states.get(resourceId);
    if (!s || !s.created || s.deleted) return undefined;
    return Object.fromEntries([...s.fields].map(([k, f]) => [k, f.value]));
  }

  // Resources stuck without a `created` event: reconcile via the provider's API.
  expiredWaiting(now: number, ttlMs: number): string[] {
    return [...this.waitingSince]
      .filter(([, since]) => now - since > ttlMs)
      .map(([id]) => id);
  }
}

A few design notes:

• Date.now() appears only in the local TTL bookkeeping. It never influences event ordering.
• The version number is not stored in the state. A counter incremented on every applied event would differ across replicas that drop or reorder events differently, which breaks convergence.
• Merging per field means the result can combine fields from different events into a state the sender never actually had. If an invariant spans several fields, keep the whole entity in one register keyed by the provider’s version.
• Delete-wins is a policy choice. A later “recreate” would need a new incarnation ID in the key.
• State lives in memory here. In production it belongs in a durable store, and the merge must run atomically per resource, for example inside a transaction or compare-and-set.

5. Production Resilience Patterns

A. Unbounded buffers and missing events

If a created event is lost, for instance because the provider’s retries were exhausted, resources waiting for it accumulate. Bound the wait with a TTL, move the stuck events to a dead-letter queue, and reconcile by fetching the resource’s current state from the provider’s API. Choose the TTL against the provider’s behavior. Stripe retries over days, so a short TTL means you will usually reconcile by fetching rather than wait. Stripe’s List Events API only returns events from the last 30 days, so a lost event can only be backfilled within that window. Treat that as a hard limit when you plan recovery.

B. Clock drift and sequence sources

Do not order by physical Date.now() across regions. Prefer, in this order:

1. A version or sequence that the provider supplies.
2. A Lamport timestamp that your consumers maintain.
3. An edge-assigned counter per resource. If you use one, note that Redis Cluster does not provide strong consistency. Its replication is asynchronous, so it can lose acknowledged writes during failovers. The WAIT command reduces that risk but does not eliminate it. A counter that must never go backwards needs a store with consensus, such as Raft-based systems like etcd.

Remember that an edge-assigned counter records arrival order at your edge, not the provider’s order.

C. Tombstone garbage collection

You cannot drop a tombstone arbitrarily. If it is pruned while a delayed created or updated event is still in flight, that event can resurrect the deleted resource. The CRDT literature handles this with causal stability: a tombstone is safe to remove only once every replica is known to have seen the delete. For webhooks, a practical rule is to keep tombstones longer than the provider’s maximum retry window plus your own replay window. Stripe retries for up to three days. Adyen queues events for up to 30 days. Hookdeck documents both. A fixed 72-hour window would therefore be too short for some providers, so make it configurable per provider.

D. When to skip the CRDT

If the provider has an API that returns an authoritative current object, the simplest approach is often to treat the webhook as a notification. You fetch the object by ID and overwrite your local copy from that response. Many engineers recommend this for Stripe subscription, customer and invoice objects, at the cost of one extra API call per event. The CRDT approach is worth its complexity when:

• the payload is the only source of truth,
• the API is rate limited or slow,
• you need replicas in several regions to converge without calling the provider on each event.

Summary: Approaches Compared

Strategy	Out-of-order delivery	Concurrent multi-region writes	Duplicates
Apply in arrival order	Overwrites newer data with older data	Replicas diverge	Unsafe without dedup
LWW on physical timestamps	Correct only if clocks are reliable	Subject to clock skew; one concurrent write is silently lost	Safe if writes are idempotent
LWW/CRDT with logical stamps + tie-breaker	Converges regardless of arrival order	Deterministic winner at every replica	Idempotent join, so safe
Fetch current state on notification	Not affected by order	Not affected by order	Safe, but costs an API call per event

Conclusion

Webhook ordering is not something to rely on. Major providers document that it is not guaranteed, and retries guarantee that duplicates will happen. A robust consumer acknowledges quickly, deduplicates on a stable event ID, and merges each event into state with a function that is commutative, associative and idempotent. A fixed tie-breaker keeps every replica’s choice identical, and logical clocks or provider versions replace unreliable wall-clock time. Where the provider offers an authoritative API, combine that approach with reconciliation by fetch. Together these give you convergence in every region, even when the network interleaves events differently at each one.

References

• Shapiro, Preguiça, Baquero, Zawirski, “A Comprehensive Study of Convergent and Commutative Replicated Data Types,” INRIA RR-7506, 2011: https://inria.hal.science/inria-00555588/document
• Shapiro et al., “Conflict-free Replicated Data Types,” SSS 2011: https://www.cs.tufts.edu/~nr/cs257/archive/marc-shapiro/CRDTs_SSS-2011.pdf
• Lamport, “Time, Clocks, and the Ordering of Events in a Distributed System,” CACM, 1978
• Fidge, “Timestamps in Message-Passing Systems That Preserve the Partial Ordering,” 1988; Mattern, “Virtual Time and Global States of Distributed Systems,” 1988
• Stripe webhook behavior (retries, ordering, duplicates): https://dev.to/shannonllc/stripe-webhooks-retries-ordering-duplicates-timeouts-and-ips-46f0
• Hookdeck, “Common Mistakes with Outbound Webhooks”: https://hookdeck.com/outpost/guides/common-outbound-webhook-mistakes.md
• Hookdeck, “At-Least-Once vs. Exactly-Once Webhook Delivery Guarantees”: https://hookdeck.com/webhooks/guides/webhook-delivery-guarantees.md
• Provider ordering summary (Stripe, Shopify, GitHub): https://www.vincentschmalbach.com/why-webhook-events-arrive-out-of-order/
• Primer webhook documentation: https://primer.io/docs/api-reference/get-started/configure-webhooks
• Redis Cluster consistency guarantees: https://redis.io/docs/latest/operate/oss_and_stack/management/scaling/

---

Originally published at https://instawebhook.com/blog/deterministic-reordering-of-concurrent-webhooks-handling-out-of-order-state-tran