Re-collecting 65,000 product pages without hammering the source

DPJ Wholesale — Tue, 13 Oct 2026 12:46:52 +0000

We keep a wholesale catalog of roughly 65,000 consumer SKUs. Prices move, stock moves, images get replaced, and listings disappear. The catalog is only useful if it stays close to reality, which means re-collecting — continuously, and without becoming the kind of scraper everyone blocks.

These are the parts that took the most iterations.

Full re-collection does not scale, and it is not even the right shape

The first version walked every SKU on a schedule. It worked at 5,000 rows and fell apart at 65,000: the run took longer than the interval between runs, so the next one started before the last finished, and the source saw a constant, obviously robotic load.

The fix was not "run it faster". It was to stop treating the catalog as one collection job.

Split by change rate, not by category

SKUs do not change at the same speed. We bucket by observed volatility instead:

Hot — changed in the last N runs. Re-checked often.
Warm — changed sometime, stable lately. Re-checked on a slow rotation.
Cold — never changed since first collection. Re-checked rarely, and never ahead of the warm set.

The buckets are recomputed from history, not hand-assigned. A SKU graduates to hot the moment it changes, and decays back down after a run of no-ops.

Only store what actually changed

Every collected record is compared field by field against the last stored version, and we keep a per-field change history rather than overwriting. Two things fall out of this:

Re-collection becomes cheap to reason about — "did anything change" is a query, not a diff of two scrapes.
The change history is itself a signal. A price that moves weekly is a different product to buy than one that has not moved in a year.

Be boring to the source

Concretely, this is what keeps us unblocked:

Respect the crawl delay, and back off on the first 429 or 503 rather than the third.
Serialise per host. Parallelism across hosts is fine; parallelism inside one host is how you get rate-limited.
Send a real, stable User-Agent with a contact URL. Not a browser string pretending to be Chrome.
Cache the sitemap and category listings instead of re-deriving the URL set on every pass.

The most reliable anti-blocking measure we found is simply not being interesting enough to block.

Retries need to distinguish "gone" from "we failed"

This one caused real damage. Our first retry logic treated every non-200 as a failure and retried, so a genuine 404 got retried forever while a transient network error was retried three times and then recorded as gone.

We now separate them explicitly:

Permanent — 404, 410. Record as removed, stop.
Transient — timeouts, connection resets, 5xx. Retry with backoff, and if it still fails, record nothing rather than recording a deletion.

That last rule matters more than it looks. A network blip must never be allowed to delete a row, because the next run will not know the difference.

Idempotency beats checkpointing

We tried checkpoint files and kept losing runs to them. What worked instead was making every write idempotent — a re-run of the same batch produces the same stored state — and letting the bucket assignment decide what to pick up next. With idempotent writes, a crashed run is just a run you start again.

What we would do differently

Start with the change history, not the current snapshot. We built the snapshot first and bolted change tracking on later, which meant re-collecting data we had already seen in order to reconstruct history we could have been keeping all along.

The catalog this pipeline produces is at dpjwholesale.com.

Scoring demand against competition across 65,000 SKUs

DPJ Wholesale — Tue, 13 Oct 2026 12:32:44 +0000

We run a wholesale catalog covering roughly 65,000 consumer SKUs. The question we kept coming back to is not "what sells" but "what sells and is not already saturated" — and those are different rankings.

This is how we ended up scoring demand against competition per SKU.

The naive version, and why it failed

The first attempt ranked SKUs by order volume. It produced a useful-looking list and a bad decision: almost everything at the top was a commodity with dozens of equivalent listings, so buying deeper on it meant competing on price alone.

Volume without a competition term is not an opportunity score. It is a popularity chart.

Splitting the problem in two

We stopped treating "demand" as one number and split it:

Demand — order frequency plus reorder rate, where reorder rate is the share of accounts that bought the same SKU a second time.
Competition — how many equivalent listings exist for the same functional item, normalised by category.

The score is the ratio, not the sum. A high-demand SKU in a crowded category scores like a mediocre SKU in a thin one, which matches what actually happens when you try to sell it.

Reorder rate is the part people skip

Bestseller rank tells you what sold once. Reorder rate tells you what a buyer chose again with their own money, which is the only signal that survives a season change, a price move, or a competitor listing the same item cheaper.

In our data, reorder rate is a stronger predictor of whether a line survives two seasons than unit cost is. It is also the field most catalogs do not publish.

Two properties that beat every other signal

After enough iterations, two SKU properties kept outperforming everything else:

Size consistency. If the second unit does not fit the first, the customer does not come back. Storage totes and replacement lids dominate our home category for exactly this reason.
A consumable attached. Parts and consumables reorder without persuasion — compost accelerator, tote lids, crate mats, ear pads, replacement tent stakes. A line with a consumable attached keeps reordering even when the main unit does not.

What we do with the score

Three buckets, treated differently:

Proven reorders — the same account bought it more than once. Buy deep; the repeat is the forecast.
First-time SKUs — buy narrow and wide: one case per variant, many variants. That buys information, not inventory.
Seasonal and fashion — a bet with a deadline. The price after the season boundary is the real price.

The stop rule

Stop a SKU when its reorder rate falls below the cost of the shelf space it occupies — not when it stops selling entirely. A slow mover that never reorders is consuming capacity a proven repeat line would use better.

The catalog and the scoring output are public at dpjwholesale.com if you want to poke at the numbers.

DEV Community: DPJ Wholesale