TL;DR: We built turbo-start-bigcommerce, where a BigCommerce catalog syncs into Sanity and page-builder blocks reference the synced documents. Four things caught us out: webhooks that structurally can't do the job, an API that lies about page size, deletes needing their own program, and ids that belong to the other system. All four are in the open-source package if you'd rather read the code.
Every integration where content in one system references entities in another has the same shape, and it's always more work than it looks.
Ours is turbo-start-bigcommerce: the catalog lives in BigCommerce, the editorial content lives in Sanity, and page-builder blocks hold references across the gap. The sync is packages/sanity-sync, it's open source, and these are the four things we didn't see coming.
1. Webhooks structurally can't do the job
The plan is always webhooks. Entity changes over there, you get told, you update over here.
Here's the comment we ended up writing at the top of reconcile.ts:
packages/sanity-sync/src/reconcile.ts
/**
* The sweep is the primary sync mechanism, not a fallback. BigCommerce has no
* CRUD webhooks for variants and none for brands, and most product image
* changes — including changing the thumbnail — fire no update event at all.
* Webhook payloads are id-only, unordered, and can duplicate. A webhook-only
* sync is therefore structurally incomplete; webhooks are at best a latency
* optimisation layered on top of this sweep.
*/
The mechanism everyone budgets for turns out to be the one that can't do the job alone. That isn't a knock on BigCommerce. Partial webhook coverage is normal across catalog and asset APIs, and the gaps are never in the documentation. You discover them by noticing a thumbnail that didn't change.
Which means you write the sweep anyway. And once the sweep exists, the webhook becomes an optimisation you might not bother shipping.
2. The API lies about page size
packages/sanity-sync/src/reconcile.ts
/**
* Admin REST caps a catalog page at 50 — but silently drops it to 10 the moment
* `options` or `modifiers` are included. Verified against the sandbox:
* `?limit=50&include=variants,options,images` comes back `per_page: 10`.
*/
const PAGE_SIZE = 50;
const PAGE_SIZE_WITH_OPTIONS = 10;
Ask for 50, get 10, no warning. Drive your pagination loop off the limit you requested rather than the total_pages the server reports and you'll sync a fifth of the catalog while your logs report success.
Categories have a different quirk. That resource has no date_modified:min filter at all, and asking for one returns 422 The filter(s): date_modified:min are not valid filter parameter(s), so categories get swept whole every time regardless of whether you asked for an incremental run.
Two unrelated quirks, one vendor, on precisely the two resources we needed. Neither is difficult to handle once you know. Finding out is the expensive part, and finding out is not on anybody's estimate.
3. Deletes are a separate program
An upsert sweep tells you what exists. It says nothing about what stopped existing, so there's a second pass:
packages/sanity-sync/src/reconcile.ts
const live = await client.fetch<string[]>(
"*[_type in $types && store.isDeleted != true]._id",
{ types: ["bigcommerceProduct", "bigcommerceProductVariant", "bigcommerceCategory"] }
);
const stale = staleMutations(live, sweep.seen);
Everything Sanity still holds, minus everything the catalog just returned, is stale. We soft-delete rather than hard-delete, because content may still reference those documents and a tombstone beats a hole.
The pass can only run on a full sweep. Run it after an incremental one and every unmodified entity looks deleted, because the sweep never saw it in the first place. So it's guarded:
packages/sanity-sync/src/reconcile.ts
if (options.since) {
logger.info("Incremental sweep — skipping the soft-delete pass.");
} else {
softDeleted = await sweepDeletes(sweep, client);
}
One if, standing between you and soft-deleting your entire catalog on a cron.
4. The ids aren't yours
This is the one that generalises hardest, and the one that bit us in production.
Synced documents get deterministic ids: bigcommerceProduct-{entityId}, where entityId is whatever BigCommerce minted when the product was created. Every store counts from its own starting point. The crewneck that's 181 on the store we captured content from is some other number on yours, so a committed reference to bigcommerceProduct-181 is correct on exactly one store on earth and dangling everywhere else.
And a dangling weak reference, as our seed docs now say in as plain a form as we could manage:
A dangling weak reference renders as nothing — no error, no gap in the log, just an empty navbar.
No stack trace, no alert. A homepage quietly rendering four fewer products than it should, until someone scrolls past it on a Tuesday.
So the seed data contains no ids. It contains placeholders named by slug, bigcommerceProduct-bramley-wool-crewneck, and a separate command resolves them against whatever the sync actually wrote:
pnpm sync:bigcommerce # catalog out of BigCommerce, into Sanity
pnpm seed:refs --write # repoint the content at the ids this store minted
An entire extra build step, whose only reason to exist is that the identifier belongs to the other system. It's idempotent, since it only rewrites references whose tail isn't already numeric, and it's all-or-nothing, because half a remap leaves a dataset that is neither the old state nor the new one.
It also carries a trap we had to write down: no slug may be entirely numeric. Something like bigcommerceProduct-2024 looks like an id that has already been resolved, so the remap skips it, and it stays dangling. Empty navbar, no error, eighteen months later nobody remembers why.
Why any of this matters beyond commerce
Swap products for images and every item above survives the translation. That's the case against bolting a separate asset manager onto your CMS, and I've written that argument up properly with the numbers attached: the DAM cost procurement forgets.
The general rule we took away: before you agree to an integration, work out whether a reference from your content to the other system's entity will be a real reference your CMS understands, or a string. If it's a string, everything above is on your roadmap whether it's on the estimate or not.
The whole package is MIT and the tests are in there too. Take it, that's what a starter is for.
Building commerce on Sanity and Next.js? Structured content modelling, catalog integrations, and migrations off legacy stacks. See our Sanity service. Expect some of it to come back as a recommendation to spend nothing.
Frequently asked questions
Can I use the sync package outside the starter?
It's written against this repo's schema, but the shape is portable: a paginated sweep, an upsert path shared with the single-entity sync, a soft-delete pass, and an id remap. Most of the value is in the structure rather than the BigCommerce specifics.
Why soft-delete instead of removing the document?
Because content may still reference it. A soft-deleted document lets the front end render a tombstone or skip the item deliberately, rather than resolving a reference to nothing and rendering an empty space nobody notices.
Do you run the sweep on a schedule?
Nothing invokes it in the starter, deliberately. You run it by hand or wire it to whatever scheduler you use. The single-entity path exists so you can reproduce a webhook delivery from a terminal before the route exists.
I'm Jono. I run Roboto Studio. Next.js and Sanity mostly, plus a lot of time spent unpicking integrations that seemed reasonable at the time. If you've written a sync like this, I'd like to hear which surprise got you.
Top comments (0)