DEV Community

life
life

Posted on

Carrier webhooks arrive twice, out of order, and sometimes never

Any system that tracks parcels across multiple carriers eventually learns the same uncomfortable thing: the carrier's webhook is not a source of truth, it is a stream of opinions that arrives in a bad mood.

Duplicates, out-of-order events, and silent gaps are all normal. The interesting engineering is not in handling each one, it is in deciding what your own state means when the evidence feeding it is unreliable.

Duplicates: make ingestion idempotent

The same scan event will be delivered more than once. Dedupe on a key that is actually stable across deliveries, which usually means carrier event code plus local timestamp plus tracking number, not the payload hash:

function eventKey(evt) {
  return [evt.tracking_number, evt.canonical_code, evt.event_time_local].join('|');
}

async function ingest(evt) {
  const key = eventKey(evt);
  const inserted = await dedupeTable.insert({ key }).onConflict().ignore().returning();
  if (!inserted) return { status: 'duplicate', key };
  await applyEvent(evt);
}
Enter fullscreen mode Exit fullscreen mode

If you dedupe on the raw payload you will miss duplicates, because carriers routinely resend the same physical event with a different signature timestamp or a reordered field list.

Out of order: never let a late event regress state

A common bug is the delivery scan arriving after the customer-facing status has already moved past it, or a stale in_transit landing after delivered and rewinding the tracking page.

Keep the transition map explicit and refuse illegal moves:

const NEXT = {
  LABEL_CREATED: ['ACCEPTED', 'CANCELED'],
  ACCEPTED:      ['IN_TRANSIT', 'EXCEPTION', 'RETURNED'],
  IN_TRANSIT:    ['OUT_FOR_DELIVERY', 'EXCEPTION', 'RETURNED'],
  OUT_FOR_DELIVERY: ['DELIVERED', 'EXCEPTION'],
  DELIVERED:     [],
};

function advance(state, evt) {
  return NEXT[state]?.includes(evt.to) ? evt.to : state; // drop, but log
}
Enter fullscreen mode Exit fullscreen mode

Drop and log rather than drop silently. The count of rejected transitions per carrier is one of the more useful operational signals you can chart.

The gap: the event that never comes

This is the failure people do not design for. A parcel is picked up, the carrier simply never sends the acceptance scan, and your system sits showing "awaiting pickup" on a box that has been in the network for four days.

No webhook will tell you about a missing webhook. The only fix is a reconciliation job that asks, on a schedule, about parcels whose expected next event has not arrived:

SELECT order_id, carrier, last_state, last_event_at
FROM shipments
WHERE last_state = 'LABEL_CREATED'
  AND label_created_at < now() - interval '24 hours'
  AND status <> 'CANCELED';
Enter fullscreen mode Exit fullscreen mode

Feed that list into the carrier's tracking API rather than waiting on push. Twenty-four hours without an acceptance scan is not a delay, it is a question, and the answer is usually that the box is still on a shelf waiting for collection. That distinction is what makes the difference between an honest tracking page and one that lies.

Record what you were told, not what you concluded

Keep the raw event alongside the normalised one. When a customer disputes a delivery, or a carrier denies a claim, the argument is settled by the original payload and its arrival time, not by your state machine's interpretation of it.

We run this shape for the small-parcel lanes FulfillNexa by SBT (fulfillnexa.com) operates from its three Chinese warehouses, 3,000 m² in Shenzhen, 13,000 m² in Suzhou and 8,000 m² in Dongguan, and every hard lesson above was paid for by a support ticket first.

The mental shift worth making: treat carrier events as advisory data that updates a model you own, and own the reconciliation. A system that only reacts to webhooks is a system whose accuracy is exactly as good as the least reliable carrier in your mix.

Top comments (0)