We keep a wholesale catalog of roughly 65,000 consumer SKUs. Prices move, stock moves, images get replaced, and listings disappear. The catalog is only useful if it stays close to reality, which means re-collecting — continuously, and without becoming the kind of scraper everyone blocks.
These are the parts that took the most iterations.
Full re-collection does not scale, and it is not even the right shape
The first version walked every SKU on a schedule. It worked at 5,000 rows and fell apart at 65,000: the run took longer than the interval between runs, so the next one started before the last finished, and the source saw a constant, obviously robotic load.
The fix was not "run it faster". It was to stop treating the catalog as one collection job.
Split by change rate, not by category
SKUs do not change at the same speed. We bucket by observed volatility instead:
- Hot — changed in the last N runs. Re-checked often.
- Warm — changed sometime, stable lately. Re-checked on a slow rotation.
- Cold — never changed since first collection. Re-checked rarely, and never ahead of the warm set.
The buckets are recomputed from history, not hand-assigned. A SKU graduates to hot the moment it changes, and decays back down after a run of no-ops.
Only store what actually changed
Every collected record is compared field by field against the last stored version, and we keep a per-field change history rather than overwriting. Two things fall out of this:
- Re-collection becomes cheap to reason about — "did anything change" is a query, not a diff of two scrapes.
- The change history is itself a signal. A price that moves weekly is a different product to buy than one that has not moved in a year.
Be boring to the source
Concretely, this is what keeps us unblocked:
- Respect the crawl delay, and back off on the first 429 or 503 rather than the third.
- Serialise per host. Parallelism across hosts is fine; parallelism inside one host is how you get rate-limited.
- Send a real, stable User-Agent with a contact URL. Not a browser string pretending to be Chrome.
- Cache the sitemap and category listings instead of re-deriving the URL set on every pass.
The most reliable anti-blocking measure we found is simply not being interesting enough to block.
Retries need to distinguish "gone" from "we failed"
This one caused real damage. Our first retry logic treated every non-200 as a failure and retried, so a genuine 404 got retried forever while a transient network error was retried three times and then recorded as gone.
We now separate them explicitly:
- Permanent — 404, 410. Record as removed, stop.
- Transient — timeouts, connection resets, 5xx. Retry with backoff, and if it still fails, record nothing rather than recording a deletion.
That last rule matters more than it looks. A network blip must never be allowed to delete a row, because the next run will not know the difference.
Idempotency beats checkpointing
We tried checkpoint files and kept losing runs to them. What worked instead was making every write idempotent — a re-run of the same batch produces the same stored state — and letting the bucket assignment decide what to pick up next. With idempotent writes, a crashed run is just a run you start again.
What we would do differently
Start with the change history, not the current snapshot. We built the snapshot first and bolted change tracking on later, which meant re-collecting data we had already seen in order to reconstruct history we could have been keeping all along.
The catalog this pipeline produces is at dpjwholesale.com.
Top comments (0)