Reconciliation, Not Sync: Integrating a System That Also Webhooks You Back

TL;DR — When you integrate with a system that also mutates the shared data and_also_ webhooks you when it changes, you cannot "keep them in sync." Both sides write, both sides notify, and you get echo loops, out-of-order updates, and fields that quietly diverge. The model that actually works is reconciliation : (1) assign a single owner to every field so a change has an authoritative direction, (2) apply every inbound event idempotently and out-of-order-safely, (3) tag your own writes so you can ignore the webhook they echo back, and (4) run a periodic full sweep that compares both sides and repairs the delta. Live events keep you fresh; the sweep keeps you correct.

The word "sync" is the bug

The ticket says "sync our catalog with the POS." It sounds symmetric and simple. It is neither, and the word itself is what leads teams into a swamp.

Picture the setup. Your platform has a menu — items, prices, availability. You integrate a point-of-sale system that the restaurant also edits directly. Now:

• The restaurant marks an item out of stock in the POS → the POS webhooks you.
• An operator marks the same item available in your dashboard → you push to the POS.
• The POS applies your push → and webhooks you back that it changed.

Two systems, both authoritative-feeling, both mutating, both notifying. "Sync" implies there is one true state and you're keeping a mirror of it. But there isn't one true state — there are two writers racing, and every write generates a notification that can trigger another write. The moment you try to make A always equal B by reacting to change events, you've built a distributed feedback loop. Let me show you the three ways it bites, because naming them is most of the cure.

Failure 1: the echo loop

You receive a POS webhook: item 88 is now unavailable. You dutifully write available = false and, because your own writes are supposed to propagate, you push that change... back to the POS. The POS sees a write, and webhooks you: item 88 changed. You apply it, and push again.

  [POS] ──"item 88 unavailable"──▶ [you: write + push back]
     ▲ │
     └──────────"item 88 changed"───────────────┘ ← infinite echo

Nothing is wrong with either system individually. The loop is emergent — it exists only in the seam between them. In production this shows up as a column that flickers, a webhook log that never goes quiet, and rate-limit warnings from the POS at 2 a.m.

The fix is to make your writes attributable. When you apply an inbound change, tag it as originating from the POS and do not echo it back out. Only changes that originate locally get pushed:

async function applyInbound(event) {
  await db.items.update(event.itemId, {
    available: event.available,
    last_source: 'pos', // this change came FROM the pos
    pos_version: event.version, // remember what the pos told us (see Failure 2)
  });
  // note: NO push back to the POS. We do not propagate what the POS just told us.
}

async function applyLocal(itemId, patch) {
  await db.items.update(itemId, { ...patch, last_source: 'local' });
  await pos.push(itemId, patch); // only LOCAL-origin changes go outbound
}

That single "who caused this change" tag breaks the loop. It's the integration version of a rule that keeps showing up: know who owns the write.

Failure 2: out-of-order and duplicate events

Webhooks are delivered at-least-once and not necessarily in order. The POS sends you "available → false" then "available → true" a second apart, and the network hands them to you reversed. If you blindly apply the last one you processed, you end up with false — permanently wrong, and nothing looks broken.

You cannot fix this with timestamps you generate on receipt (they reflect arrival, not truth). You need a monotonic version from the source and the rule "never apply an event older than what I've already applied":

async function applyInbound(event) {
  const row = await db.items.findById(event.itemId);

  // Idempotent AND order-safe: only move forward.
  if (row.pos_version != null && event.version <= row.pos_version) {
    return; // stale or duplicate — we've already seen this or newer. No-op.
  }

  await db.items.update(event.itemId, {
    available: event.available,
    last_source: 'pos',
    pos_version: event.version,
  });
}

If the source doesn't give you a version or updated-at you can trust, you're reduced to reconciliation-only (below) for that field — which is a perfectly honest answer, and better than pretending the event stream is ordered when it isn't. This is the same shape as making payment webhooks idempotent: the event stream is unreliable by contract, so correctness has to live in how you apply, not in the events arriving perfectly.

Failure 3: the fields don't mean the same thing

This is the one that burns a week. Two systems rarely model the same concept identically, and the mismatches are silent:

• Your platform has availability (is this sellable right now?). The POS has stock count and a separate tracking mode (is stock even being tracked?). An item with stock = 0 but tracking = off is available, not out of stock. Map "stock 0" straight to "unavailable" and you'll hide sellable items.
• The POS "deletes" an item, then "revives" it a day later under the same external id. If your import treats delete as terminal, the revive silently does nothing.
• A field you assumed was a boolean is a tri-state; a price you assumed was minor-units is major-units for one region; an id you keyed on gets reused.

There is no clever code for this. The fix is a written field ownership map — for every field, exactly one system is the authority, and you translate meanings explicitly at the boundary:

Field	Owner	Rule
Catalog (name, description, category)	Platform	POS never overrides; local is truth
Price	Platform	pushed to POS; inbound price events ignored
Stock / availability	POS	POS is truth; translate stock+tracking→available
Item existence (create/delete/revive)	POS	delete is soft; a later revive must re-activate

Once ownership is explicit, most "conflicts" evaporate — they were never conflicts, just two systems editing fields the other didn't own. A change to a platform-owned field ignores whatever the POS says about it, and vice versa. Write this table before the code; it is the actual design.

The part that makes it correct: the reconciliation sweep

Everything above keeps you fresh — reacting to live events. None of it keeps you correct, because events get dropped. At-least-once is a floor: a webhook can fail every retry and vanish, a deploy can drop an in-flight batch, a bug can skip an apply. Over weeks, the two systems drift, and no single event tells you they have.

So the load-bearing piece is a periodic full reconciliation — a scheduled job that ignores the event stream entirely, pulls the full state of both sides, and repairs the delta by ownership:

// Runs on a schedule (this is a perfect job for a Postgres-backed
// outbox/cron worker — see the durable-queue post).
async function reconcile() {
  const ours = await db.items.allByExternalId();
  const theirs = await pos.listAllItems(); // full snapshot, not events

  for (const [extId, posItem] of theirs) {
    const local = ours.get(extId);

    if (!local) { await createFromPos(posItem); continue; } // they have, we don't
    // Repair ONLY fields the other side owns:
    if (local.available !== translateAvailability(posItem)) {
      await db.items.update(local.id, { available: translateAvailability(posItem) });
    }
  }
  for (const [extId, local] of ours) {
    if (!theirs.has(extId) && local.last_source !== 'local') {
      await flagOrphan(local); // we have it, POS doesn't — human decision, don't guess
    }
  }
}

Run it nightly (or hourly for volatile data). The live webhooks give you low-latency updates; the sweep guarantees that even if every webhook for a day was lost, you converge to correct within one cycle. This is the reconciliation-not-a-lucky-webhook idea applied to a whole integration: don't trust the stream to be complete — periodically check reality and repair.

Two hard-won rules for the sweep: repair only fields the other side owns (or two sweeps will fight the same field forever), and never auto-delete on "the other side doesn't have it" — flag it for a human. A dropped event and a real deletion look identical from one snapshot; guessing wrong deletes a live menu item.

Audit everything at the boundary

One operational note that paid for itself ten times over: log every inbound and outbound event, raw, before you interpret it. When a price is wrong or an item vanished, the only way to answer "did they send us bad data, did we misapply good data, or did our push do it?" is to replay the boundary. A tiny append-only integration_events table (direction, external id, raw payload, applied-result, timestamp) turns "no idea what happened" into a five-minute query. Integrations fail in the seam; instrument the seam.

What actually broke in production

• A column-meaning mismatch marked good items unavailable. We mapped the POS stock field to availability without accounting for tracking = off, so every untracked-but-sellable item went dark. Caught only because a restaurant called about missing menu items. Fix: the explicit translation function, plus a reconciliation pass that re-derived availability correctly for the whole catalog.
• delete-then-revive did nothing. Our importer treated a POS delete as final and dropped the row's linkage; the next day's revive created no change because our code had stopped tracking the external id. Fix: soft-delete with the external id preserved, and treat a revive as reactivation.
• The echo loop we didn't notice for days — because each hop was individually valid, only the POS's rate-limit emails revealed it. Fix: the last_source tag, and a boundary log that made the loop visible at a glance.

What I'd still improve

• Reconciliation is O(catalog) every run. Fine at this scale, wasteful at 10×. I'd move to a hashed/versioned diff — compare a per-item fingerprint and only fetch full detail for mismatches — so the sweep cost tracks the delta, not the total.
• Ownership is still partly in my head. It should be declarative config (a table of field → owner → translator) that both the live-apply path and the sweep read from, so they can never disagree about who owns what. Right now the two paths encode it separately, which is exactly the kind of drift this whole post is about.

The one idea to take away

You can never sync two systems that both mutate and both notify — the symmetry is a lie that produces echo loops and silent divergence. You can only reconcile : give every field one owner, apply inbound events idempotently and in-order-safely, tag your own writes so you don't echo them, and run a periodic sweep that treats reality — not the event stream — as the thing to converge to. Events keep you fast. Reconciliation keeps you right.