DEV Community

Cover image for Scraping a price is easy. Knowing it changed is the product.
Sami
Sami

Posted on

Scraping a price is easy. Knowing it changed is the product.

TL;DR

Most scraping tutorials stop at "here is the JSON". But almost nobody wants a snapshot — they want to know what moved since last time. That difference is not a formatting detail, it is where all the engineering lives: the source usually publishes no history, so if you don't store the previous state, the change is simply unknowable.

This post is the five patterns I use for that, the traps that cost me real money to find, and why "return only what changed" also happens to be the honest way to bill.

Why the snapshot is the wrong unit

Take a hotel rate. You scrape it today: €48. Useful? Barely. Now scrape it every morning and the interesting facts appear:

  • it dropped to €41 nine days before arrival
  • it was €48 for three weeks, then moved twice in one day
  • the property stopped being listed for that date entirely

None of those facts exist in any single response. They exist only in the diff, and the source will never hand you the diff — no rate site publishes "here is what I charged yesterday". The same is true of restaurant availability, pharmacy stock, and whether a SKU quietly vanished from a catalogue.

So the state you keep is the dataset. That reframing has a pleasant consequence: if your scraper stores state, it can bill for changes instead of rows, and a daily run over 5,000 SKUs costs the buyer almost nothing on a quiet day.

Pattern 1 — Monotonic id watermark (cheapest, when you can get it)

If the source hands out ids that only ever increase, you don't need to store content at all. Store the highest id you saw and stop paginating the moment you cross it.

last_id = int(await store.get_value("watermark") or 0)
highest = last_id

async for message in channel_history():
    if message["id"] <= last_id:
        break                    # everything below here was seen on a previous run
    highest = max(highest, message["id"])
    yield message

await store.set_value("watermark", highest)
Enter fullscreen mode Exit fullscreen mode

Cost: one integer. Works for Telegram-style message feeds, forum post ids, filings sequences.

The trap: this is only sound if ids are monotonic and the feed is ordered. Plenty of APIs return "recommended" order by default, which brings us to the expensive lesson.

Pattern 2 — Content fingerprint (when ids are unstable)

Reviews are the classic case: no usable id, edited text, and the same review can surface under different internal keys. So hash the parts that identify it.

def fingerprint(review: dict) -> str:
    basis = "|".join([
        (review.get("author") or "").strip().lower(),
        (review.get("date") or "")[:10],
        (review.get("text") or "").strip()[:180],
    ])
    return hashlib.sha1(basis.encode()).hexdigest()[:16]
Enter fullscreen mode Exit fullscreen mode

Then keep a set of seen fingerprints and emit only misses.

The trap that cost me a whole baseline

I shipped exactly this on a hotel-reviews scraper and the second run reported every review as new. The fingerprints were fine. The ordering wasn't.

The baseline run had paged through reviews in the site's default relevance order. The next run happened to load them in date order. Two runs, two disjoint slices of the same 4,000 reviews — so nothing matched, and the diff was garbage.

The fix is one line, and it belongs in every delta scraper you write:

# Delta mode must pin the sort order, ALWAYS — not just when a baseline exists.
# Relevance ordering is not stable between runs, so baseline and delta end up
# describing different slices and every fingerprint misses.
if await sort_newest_first(page):
    delta_stop_after = 1        # one all-seen page is now enough to stop
Enter fullscreen mode Exit fullscreen mode

Pin the order, and "I hit a full page of already-seen items" becomes a trustworthy stop signal instead of a coin flip.

Pattern 3 — Price watermark pagination (when the paginator lies)

A UK catalogue I monitor exposes ~15,000 SKUs through a search API with a page parameter. At hitsPerPage=1000, requesting page=1 returns HTTP 200 with zero hits. No error, no warning — a silent cap dressed up as an empty shelf.

If you trust it, you ship a scraper that reports "1,000 products" forever and nobody notices the other 14,000.

The workaround is to stop paginating by position and paginate by value instead — sort by price and use the last price you saw as a filter:

body = {"hitsPerPage": 1000, "sortBy": ["price:asc"]}
if last_price is not None:
    body["numericFilters"] = [f"net_price_value>{last_price}"]
Enter fullscreen mode Exit fullscreen mode

14,843 SKUs in 16 requests, no cap, and it self-heals if the catalogue grows mid-crawl. Any time a paginator returns 200-with-nothing, look for a sortable field you can use as a cursor.

Pattern 4 — Cell-state comparison (for things that are absent)

Availability is the awkward one: a sold-out restaurant returns nothing. There is no row to diff. The signal is the disappearance.

So the unit of state can't be a row, it has to be a cell in a grid you define — (venue × date × party size), (SKU × market), (property × arrival date):

key = f"{venue_id}|{date}|{party_size}"
prev = state.get(key)
now  = {"slots": len(slots), "prime": prime_count(slots)}

if prev is None:
    change = "new"
elif prev["slots"] > 0 and now["slots"] == 0:
    change = "sold_out"          # only knowable by comparison
elif prev["slots"] == 0 and now["slots"] > 0:
    change = "reopened"
elif abs(now["slots"] - prev["slots"]) > NOISE_FLOOR:
    change = "moved"
else:
    change = None                # unchanged: don't emit, don't bill
Enter fullscreen mode Exit fullscreen mode

Two details that matter more than they look:

  • A noise floor. Slot counts jitter by ±1 constantly. Without a threshold your "changes" feed is mostly noise, and if you bill per change you are billing for jitter — which is the fastest way to earn a one-star review.
  • Rank first. When several things change at once, emit the lifecycle change (listed / delisted / sold out) ahead of the numeric one. A buyer's alerting rules care about "it disappeared" far more than "it moved by 3".

Pattern 5 — Save the state before you bill

Mundane, and the one that actually hurts if you get it wrong. Write the new watermark before charging, and make the first run of a stream cheap:

await save_seen(seen)            # persist BEFORE the charge
if not is_first_run:
    await charge("change", len(changes))
Enter fullscreen mode Exit fullscreen mode

If you charge first and crash before persisting, the next run re-detects the same changes and bills them again. And a first run has no baseline by definition — everything looks new, so charging a "change" price for what is really a snapshot is indefensible. Bill the baseline at snapshot rate, or not at all.

What this looks like shipped

I built five monitors on these patterns this month, each one aimed at a source that publishes no history of its own:

All five run on a schedule, keep their own watermark, and return only the cells that moved. If you'd rather build your own, the patterns above are the whole trick — the state store is the moat, not the parser.

One last thing: run your own demo input

While writing this I ran all 18 of my public Actors with exactly the prefilled input a first-time user gets, and three came back with zero rows.

The worst one was my highest-traffic Actor. Its example profile URL still returned HTTP 200. The mode worked fine anonymously — other profiles returned data immediately. That one account had simply gone quiet, and every evaluator who pressed Run saw an empty dataset.

A status check on the URL would have said "fine". A schema validator would have said "fine". The only thing that catches it is running the thing and counting rows.

So if you maintain a scraper with a demo input: go run it right now, unedited, and count the rows. It takes a minute and mine had rotted without a single error anywhere.

If you build monitors for a living and want the delta logic explained for a specific source, reply in the comments — happy to dig into the pattern that fits it.

Top comments (0)