DEV Community

Cover image for The paginator returned HTTP 200 with zero results. 13,843 products were missing.
Sami
Sami

Posted on

The paginator returned HTTP 200 with zero results. 13,843 products were missing.

Every price-monitoring tutorial shows you the happy path: hit the search endpoint, ask for a big page size, walk the pages, done. Here is what actually happened when I pointed that at a UK safety-equipment catalogue with ~15,000 priced SKUs.

hitsPerPage=1000&page=0 → 1,000 products. Perfect.

hitsPerPage=1000&page=1 → HTTP 200, zero hits. No error. No warning. No hasMore: false. An empty array dressed up as a successful response.

If you trust that, you ship a monitor that reports "1,000 products" forever and nobody notices the other 13,843. Your buyer builds a dashboard on a tenth of the shelf.

Why the paginator lies

Most hosted search backends cap how deep you can page — often at ~1,000 results — because deep offsets are expensive to compute. The polite ones return an error explaining it. This one returns success with nothing in it, which is worse, because your code has no signal to react to.

So stop paginating by position and paginate by value instead. Sort by price ascending, then use the last price you saw as a filter for the next request:

body = {"hitsPerPage": 1000, "sortBy": ["price:asc"]}
if last_price is not None:
    body["numericFilters"] = [f"net_price_value>{last_price}"]
Enter fullscreen mode Exit fullscreen mode

Each request starts where the previous one stopped, so you never ask for a deep offset at all. Result on this catalogue: 14,843 SKUs in 16 requests, no cap, and it self-heals if the catalogue grows mid-crawl.

Two details worth stealing:

  • Ties. If several products share a price, a strict > skips the ones after the 1,000th at that exact value. In practice you either accept the (tiny) loss or add a secondary sort key and filter on the pair. Measure before you optimise — on a real price distribution the collision count is usually zero.
  • Any sortable field works. Price is convenient here; an id or created_at cursor is often better if the field is monotonic and dense.

Whenever a paginator returns 200-with-nothing, look for a sortable field you can use as a cursor. It is almost always faster than the offset path it replaces.

The other trap: the same product, twice

Promoted or sponsored placements appear in the result set again at their natural position. Same SKU, two different index values. If you emit both, you have inflated the buyer's row count — and if you bill per row, you have overcharged them for a duplicate they can spot in five seconds.

by_sku: dict[str, dict] = {}
for hit in hits:
    sku = hit["sku"]
    prev = by_sku.get(sku)
    # keep the natural position, not the promoted slot
    if prev is None or hit["index"] < prev["index"]:
        by_sku[sku] = hit
Enter fullscreen mode Exit fullscreen mode

And a 406 I caused myself

The sitemap fetch kept returning 406 Not Acceptable. The endpoint was fine; my client was sending Accept: text/html at an XML resource. One header:

headers = {"Accept": "application/xml,text/xml,*/*"}
Enter fullscreen mode Exit fullscreen mode

Worth remembering the general shape: a 406 is almost never the server being broken, it is your Accept header disagreeing with what the resource can produce.

The part that makes it a product

A full catalogue snapshot is a big, boring file. What a purchasing manager or a competitor-pricing analyst actually wants is the diff: which SKUs moved, which appeared, which quietly vanished.

That last one is the interesting one, because a discontinued product does not announce itself. It simply stops being in the response. You can only detect it by comparing against what you stored last time:

current = {row["sku"] for row in rows}
gone = [sku for sku in previous if sku not in current]   # the discontinuation signal
Enter fullscreen mode Exit fullscreen mode

Which means: rank the lifecycle change above the numeric one. When a SKU both moved price and disappeared from a category, the buyer's alerting rules care far more about "it is gone" than "it is £1.05 cheaper". I emit lifecycle_change first for exactly that reason.

And save the watermark before you charge. If you bill first and crash before persisting, the next run re-detects the same changes and bills them again — which is indefensible and entirely avoidable:

await save_seen(seen)          # persist BEFORE the charge
if not is_first_run:
    await charge("shelf-change", len(changes))
Enter fullscreen mode Exit fullscreen mode

A first run has no baseline by definition, so everything looks new. Charging a change price for what is really a snapshot is not a pricing decision, it is a bug.

Fields that matter more than price

On a trade catalogue the price alone is not enough to act on:

  • ex-VAT and inc-VAT, separately. A trade buyer reasons in ex-VAT; a consumer-facing competitor lists inc-VAT. Mixing them produces confidently wrong comparisons.
  • Minimum order quantity. £6.95 per pair of gloves means something different at MOQ 1 than at MOQ 50.
  • Unit. "Per pair" vs "per box of 100" is the single most common false-positive in industrial price comparison.
  • Lifecycle stage, when the catalogue exposes it. It flags a discontinuation weeks before the SKU disappears.

Running it

I packaged the above as UK PPE & Safety Equipment Shelf Monitor on Apify: the whole priced catalogue in one run, HTTP-only (no browser, so it costs a fraction of a browser-based scraper), and a delta mode that returns only the rows that moved — price up or down, new listing, delisting — each annotated with what changed.

There is a prefilled example run on the Actor page, and a scheduled config here: UK PPE shelf changes — daily delta.

If you would rather build your own, the five patterns behind the delta engine — watermarks, fingerprints, price-cursor pagination, cell-state comparison, and save-before-billing — are written up in Scraping a price is easy. Knowing it changed is the product.

Monitoring a shelf I have not covered? Reply in the comments with the source and I will tell you whether it is reachable and what the delta key should be.

Top comments (0)