DEV Community

Sami
Sami

Posted on

The same tyre appeared twice in one response. My row count was 10% too high.

I was counting rows on a UK tyre retailer and the numbers were slightly too high. Not wildly — about 10% too many. That is the worst kind of wrong, because it looks plausible.

The cause was one product appearing twice in the same response:

{ "sku": "MICH-2055516-91V", "index": 1, "promoted": true }
{ "sku": "MICH-2055516-91V", "index": 9, "promoted": false }
Enter fullscreen mode Exit fullscreen mode

Promoted placements are injected at the top of the result set and left at their natural position. Same SKU, two indices. Emit both and you have inflated the row count — and if you bill per row, you have charged for a duplicate the buyer can spot in five seconds.

Dedupe by SKU, keeping the natural position rather than the paid slot:

by_sku: dict[str, dict] = {}
for hit in hits:
    prev = by_sku.get(hit["sku"])
    if prev is None or hit["index"] < prev["index"]:
        by_sku[hit["sku"]] = hit
Enter fullscreen mode Exit fullscreen mode

Keep the lowest index and you preserve organic ranking, which is a signal in its own right: it answers whether a tyre got cheaper or merely got promoted.

A 406 I caused myself

The sitemap fetch kept returning 406 Not Acceptable. I spent longer than I should have suspecting rate limits before reading my own headers: I was sending Accept: text/html at an XML resource.

headers = {"Accept": "application/xml,text/xml,*/*"}
Enter fullscreen mode Exit fullscreen mode

Worth internalising the general shape — a 406 is almost never the server being broken, it is your Accept header disagreeing with what the resource can produce. Unlike a 403, it is entirely your side.

Two axes, not one

A tyre price is not a single number, it is a function of two inputs:

  • size — 205/55R16, 195/65R15. This is what the buyer searches.
  • postcode — fitting availability and price vary by location, because the fitter network does.

So the state key is the pair, not the product:

key = f"{sku}|{postcode}"
Enter fullscreen mode Exit fullscreen mode

Collapse the postcode axis and you get a national "average price" that describes no actual transaction. Keep it, and you can answer the question a distributor actually asks: where am I being undercut?

What the analytics payload gives you free

This retailer, like many, fires GA4 e-commerce events with the catalogue data already structured — item_category, item_category2, brand, price. Mapping those into named columns costs nothing and produces a cleaner taxonomy than parsing the page:

row["brand"]        = hit.get("item_brand")
row["vehicleType"]  = hit.get("item_category")     # car / van / 4x4
row["season"]       = hit.get("item_category2")    # summer / winter / all-season
Enter fullscreen mode Exit fullscreen mode

Season matters more than it looks: a winter-tyre price move in September is a different event from the same move in February.

Why the diff is the product

Tyre prices move constantly — promotions, stock, competitor matching. A daily full snapshot is mostly yesterday's data re-billed. What a pricing analyst acts on is:

if prev is None:                      change = "new"
elif now.price != prev.price:         change = "price_up" / "price_down"
elif now.index != prev.index:         change = "rank_move"
elif prev.available and not now.available: change = "delisted"
else:                                 change = None   # not emitted, not billed
Enter fullscreen mode Exit fullscreen mode

Rank the lifecycle change (new / delisted) above the numeric one — an alerting rule cares far more that a size stopped being stocked than that it moved 40p. And persist the watermark before charging, or a crash makes the next run re-bill changes it already sold you.

A first run has no baseline, so everything looks new; billing that as "changes" is a bug, not a pricing choice.

Running it

Packaged as UK Tyre Price & Fitment Monitor — prices by size and postcode, brand and season taxonomy, organic position, HTTP-only, no browser. Delta mode returns only what moved.

Scheduled config ready to clone: UK tyre price changes — daily delta on popular sizes.

The five delta patterns behind it are in Scraping a price is easy. Knowing it changed is the product.

Hit a promoted-duplicate problem on another retailer? Reply with the shape of the response — the fix is usually this same three-line dedupe.

Top comments (0)