DEV Community: Sami

A fully booked restaurant returns nothing. So the absence is the signal.

Sami — Sun, 26 Jul 2026 14:42:24 +0000

Here is the problem that makes restaurant availability different from every price-scraping job: a fully booked restaurant returns nothing.

Not an error. Not available: false. An empty slot array, or the venue simply missing from the response. The most valuable state — sold out — has no row to scrape.

Which means you cannot detect it by parsing. You can only detect it by comparing against what you stored last time. The absence is the signal, and a stateless scraper is structurally blind to it.

The unit of state is a cell, not a venue

Availability is a function of three inputs, and they are independent:

key = f"{venue_id}|{date}|{party_size}"

A table for 2 on Friday says nothing about a table for 6 on Friday. Collapse party size and you have thrown away the axis that matters most to anyone tracking demand.

Then the comparison writes itself:

prev = state.get(key)
now  = {"slots": len(slots), "prime": prime_count(slots)}

if prev is None:                              change = "new"
elif prev["slots"] > 0 and now["slots"] == 0: change = "sold_out"      # only knowable by diff
elif prev["slots"] == 0 and now["slots"] > 0: change = "reopened"      # a cancellation wave
elif abs(now["slots"] - prev["slots"]) > NOISE_FLOOR: change = "moved"
else:                                         change = None

reopened is the one people miss. A venue going from 0 back to 4 slots is a cancellation event — for a demand analyst that is a stronger signal than the original sell-out, because it is rarer and more actionable.

The noise floor is not optional

Slot counts jitter by ±1 constantly — a hold expires, someone abandons a checkout. Without a threshold, your "changes" feed is mostly noise, and if you bill per change you are billing for jitter. That is the fastest route to a one-star review.

NOISE_FLOOR = 1     # ±1 slot is not an event

Pick the floor from the data, not from taste: sample one venue every minute for an hour and look at the distribution of deltas.

Prime time is the real metric

"14 slots available" is a weak number. Fourteen slots at 17:15 and 22:30 is an empty restaurant; two slots at 19:30 is a busy one.

So count the window separately:

PRIME_START_MIN = 18 * 60 + 30      # 18:30
PRIME_END_MIN   = 20 * 60 + 59      # 20:59

prime = [s for s in slots if PRIME_START_MIN <= minutes(s) <= PRIME_END_MIN]

prime_sold_out while total slots stay non-zero is the scarcity signal — the restaurant is full when it matters and empty when it does not. That single derived flag is worth more than the raw count it comes from.

Getting in without a login

The public booking API is reachable anonymously with a client key that the web app ships in its own JavaScript bundle. The robust way to find it is not to hardcode it but to harvest it at runtime — and the detail that bit me is that the script src attributes are relative:

urls = [urljoin(str(home.url), s) for s in srcs
        if s.endswith(".js") and "resy.com" in urljoin(str(home.url), s)]
urls.sort(key=lambda u: (0 if "/app." in u else 1, len(u)))

Sort the main app bundle first and you find the key on the first fetch instead of the tenth. Keep a known-good fallback, and re-extract once mid-run if you start getting 419 — that status means the key rotated, not that you are blocked.

Running it

Packaged as Resy Availability & Scarcity Monitor: forward-dated availability by date and party size, prime-window counts, first and last seating, seating types, largest table, sold-out and prime-sold-out flags. Read-only — it books nothing, needs no account, and touches no personal data.

Scheduled config ready to clone: New York restaurant availability changes — daily delta.

The five patterns behind the delta engine are in Scraping a price is easy. Knowing it changed is the product.

Tracking availability on a surface where absence is the signal? Reply with the source — the cell key is usually the hard part, and it is worth getting right before you write a parser.

The same tyre appeared twice in one response. My row count was 10% too high.

Sami — Sun, 26 Jul 2026 14:41:58 +0000

I was counting rows on a UK tyre retailer and the numbers were slightly too high. Not wildly — about 10% too many. That is the worst kind of wrong, because it looks plausible.

The cause was one product appearing twice in the same response:

{ "sku": "MICH-2055516-91V", "index": 1, "promoted": true }
{ "sku": "MICH-2055516-91V", "index": 9, "promoted": false }

Promoted placements are injected at the top of the result set and left at their natural position. Same SKU, two indices. Emit both and you have inflated the row count — and if you bill per row, you have charged for a duplicate the buyer can spot in five seconds.

Dedupe by SKU, keeping the natural position rather than the paid slot:

by_sku: dict[str, dict] = {}
for hit in hits:
    prev = by_sku.get(hit["sku"])
    if prev is None or hit["index"] < prev["index"]:
        by_sku[hit["sku"]] = hit

Keep the lowest index and you preserve organic ranking, which is a signal in its own right: it answers whether a tyre got cheaper or merely got promoted.

A 406 I caused myself

The sitemap fetch kept returning 406 Not Acceptable. I spent longer than I should have suspecting rate limits before reading my own headers: I was sending Accept: text/html at an XML resource.

headers = {"Accept": "application/xml,text/xml,*/*"}

Worth internalising the general shape — a 406 is almost never the server being broken, it is your Accept header disagreeing with what the resource can produce. Unlike a 403, it is entirely your side.

Two axes, not one

A tyre price is not a single number, it is a function of two inputs:

size — 205/55R16, 195/65R15. This is what the buyer searches.
postcode — fitting availability and price vary by location, because the fitter network does.

So the state key is the pair, not the product:

key = f"{sku}|{postcode}"

Collapse the postcode axis and you get a national "average price" that describes no actual transaction. Keep it, and you can answer the question a distributor actually asks: where am I being undercut?

What the analytics payload gives you free

This retailer, like many, fires GA4 e-commerce events with the catalogue data already structured — item_category, item_category2, brand, price. Mapping those into named columns costs nothing and produces a cleaner taxonomy than parsing the page:

row["brand"]        = hit.get("item_brand")
row["vehicleType"]  = hit.get("item_category")     # car / van / 4x4
row["season"]       = hit.get("item_category2")    # summer / winter / all-season

Season matters more than it looks: a winter-tyre price move in September is a different event from the same move in February.

Why the diff is the product

Tyre prices move constantly — promotions, stock, competitor matching. A daily full snapshot is mostly yesterday's data re-billed. What a pricing analyst acts on is:

if prev is None:                      change = "new"
elif now.price != prev.price:         change = "price_up" / "price_down"
elif now.index != prev.index:         change = "rank_move"
elif prev.available and not now.available: change = "delisted"
else:                                 change = None   # not emitted, not billed

Rank the lifecycle change (new / delisted) above the numeric one — an alerting rule cares far more that a size stopped being stocked than that it moved 40p. And persist the watermark before charging, or a crash makes the next run re-bill changes it already sold you.

A first run has no baseline, so everything looks new; billing that as "changes" is a bug, not a pricing choice.

Running it

Packaged as UK Tyre Price & Fitment Monitor — prices by size and postcode, brand and season taxonomy, organic position, HTTP-only, no browser. Delta mode returns only what moved.

Scheduled config ready to clone: UK tyre price changes — daily delta on popular sizes.

The five delta patterns behind it are in Scraping a price is easy. Knowing it changed is the product.

Hit a promoted-duplicate problem on another retailer? Reply with the shape of the response — the fix is usually this same three-line dedupe.

One price was in cents, the other in euros. Same JSON, same product.

Sami — Sun, 26 Jul 2026 14:32:50 +0000

A product on Europe's largest online pharmacy came back looking like this:

{
  "price": 995,
  "listPrice": 1450,
  "prices": { "default": { "amount": 9.95 } }
}

Both numbers are correct. price is in cents. prices.default.amount is in euros. Same response, same product, two units, no field name warning you.

Pick the wrong one and you do not get an error — you get a €9.95 product priced at €995, a 100× outlier that sails through every sanity check that only looks for nulls. Then your buyer's "RRP breach" alert fires on the entire catalogue.

Worse: the marketplace-offer array indexes in cents again, so a naive "cheapest across offers" comparison silently mixes both scales.

The fix is not clever, it is just explicit — and the comment matters more than the code:

def _pick_price(raw: dict) -> float | None:
    """Return euros.

    Unit trap: top-level `price` / `listPrice` are CENTS, nested
    `prices.*.amount` are EUROS, and the offers array is cents again.
    Never mix them — a 100x error passes every null check.
    """
    nested = (raw.get("prices") or {}).get("default") or {}
    if isinstance(nested.get("amount"), (int, float)):
        return float(nested["amount"])          # already euros
    cents = raw.get("price")
    return round(cents / 100, 2) if isinstance(cents, (int, float)) else None

General rule worth internalising: whenever an API exposes the same quantity twice, assume the units differ until you have proved otherwise on a product whose real price you can look up by hand.

The field that does not exist

My first version asked the search backend for a url attribute and got back... nothing. Not an error — just no field, silently absent, on every single record.

The attribute is called deeplink. Same story with the recommended retail price: I asked for strikePrice (a name I had assumed) when the real field is listPrice.

A hosted search API given an unknown attribute name usually omits it rather than complaining. So if a field is mysteriously always empty, check the spelling against a raw unfiltered record before you debug your parser:

ATTRS = ["pzn", "name", "brand", "price", "listPrice", "pricePerUnit",
         "stockStatus", "deeplink", "averageRating"]

PZN is the join key, and it is the whole product

German pharmacy retail has something most e-commerce verticals lack: a national article number (PZN) that identifies the exact pack — substance, strength, count, manufacturer. It is the join key that makes cross-retailer comparison honest.

That is what turns a scrape into an asset. With PZN you can:

compare the same pack across shops without fuzzy title matching
detect an RRP breach — the shop selling below the manufacturer's list price, which is a competitive signal, not a typo
watch the marketplace seller undercutting the shop itself on its own product page
track stock flips per pack, which is the leading indicator for a supply problem

One brand is one request covering up to 1,000 articles, so a competitive set is cheap to watch daily. DOPPELHERZ is 335 articles in a single call; ABTEI 148.

Why daily, and why only the diff

OTC pharmacy prices move constantly — promotions, marketplace sellers, stock. A full snapshot every day is mostly noise you already had. What a category manager acts on is:

if prev is None:                       change = "new"
elif now.price != prev.price:          change = "price_up" / "price_down"
elif now.below_rrp != prev.below_rrp:  change = "rrp_breach"
elif now.stock != prev.stock:          change = "stock_flip"
else:                                  change = None   # never emitted, never billed

Two disciplines that keep this honest:

A first run has no baseline, so everything looks new. Charging a change price for a snapshot is a bug, not a pricing decision. Bill the baseline as a snapshot or not at all.
Persist the watermark before charging. Bill first and crash before saving, and the next run re-detects and re-bills the same changes.

Running it

Packaged as DE/AT Pharmacy Price & Stock Monitor: PZN-keyed OTC shelf data across DE and AT — price, list-price breach, discount, availability, and the marketplace seller undercutting the shop. HTTP-only, no browser, no login. Delta mode returns only price moves, RRP breaches and stock flips.

Scheduled config ready to clone: DE pharmacy price & stock changes — daily delta.

The five patterns behind the delta engine are written up in Scraping a price is easy. Knowing it changed is the product.

If you hit a units bug like this one on another retail API, reply with the field names — the cents/euros pair shows up far more often than it should.

A hostel bed has four prices. Three of them are hidden.

Sami — Sun, 26 Jul 2026 14:31:18 +0000

If you scrape one accommodation price, you have a number. It is nearly useless.

€48 tells you nothing on its own. What a revenue manager, an OTA analyst or a travel-market researcher actually acts on is the shape of that number over time:

it sat at €48 for three weeks, then moved twice in one day
it dropped to €41 exactly nine days before arrival
the property stopped being listed for that date entirely

None of those facts exist in any single response. No rate site publishes "here is what I charged yesterday". So if you do not store the previous state, the interesting part is not merely missing — it is unknowable.

That reframing is the whole engineering problem, and it has a pleasant consequence: once your scraper keeps state, it can return changes instead of rows, and a daily run over a whole city costs almost nothing on a quiet day.

The grid, not the row

The unit of state cannot be "a property". It has to be a cell: (property × arrival date × guest configuration). A hostel bed is priced per arrival date and per party size, and those are independent axes.

key = f"{property_id}|{arrival_date}|{guests}|{nights}"
prev = state.get(key)
now  = {"lowest": lowest, "dorm": dorm, "private": private, "promo": promo_stack}

if prev is None:
    change = "new"
elif prev["lowest"] != now["lowest"]:
    change = "price_up" if now["lowest"] > prev["lowest"] else "price_down"
elif prev["promo"] != now["promo"]:
    change = "promo_changed"
else:
    change = None          # unchanged: do not emit, do not bill

One city over 30 forward dates is 30 requests and a full month of the priced competitive set — because a whole city arrives in one request per date. London returns 83 properties at once; Munich 17. That is what makes daily monitoring viable rather than a crawl.

The trap that cost me a whole baseline

I shipped this same pattern on a reviews scraper first, and the second run reported every item as new. The fingerprints were fine. The ordering was not.

The baseline run had paged through results in the site's default relevance order. The next run happened to load them in date order. Two runs, two disjoint slices of the same set — so nothing matched and the diff was garbage.

The fix is one line, and it belongs in every delta scraper you write:

# Delta mode must pin the sort order ALWAYS — not only when a baseline exists.
# Relevance ordering is not stable between runs.
if await sort_deterministically(page):
    stop_after_all_seen_pages = 1

Pin the order and "I hit a full page of already-seen items" becomes a trustworthy stop signal instead of a coin flip.

Four prices, not one

The headline rate is the least interesting field on the page. On this surface each cell carries:

the lowest available rate — what a price-comparison table would show
the dorm / private split — two different products under one property, and they move independently
the pre-discount rate plus the promo stack, with discount percentages. This is the one competitors miss: a property holding rack rate while stacking a 25% promo is running a very different strategy from one that simply cut its rate, and only the stack tells them apart.
the commission split for that date — the economics of the listing, not just its price

Plus the organic ranking position, which answers a question the rate alone cannot: did they get cheaper, or did they get promoted?

Absence is a signal

The awkward case is a property that vanishes from a date it was priced on yesterday. Sold out, pulled, or de-listed — the response simply does not contain it. There is no row to diff.

gone = [k for k in previous_keys if k not in current_keys]   # only knowable by comparison

Two details that matter more than they look:

Rank the lifecycle change first. When a cell both moved price and disappeared, a buyer's alerting cares far more about "it is gone" than "it is €3 cheaper".
A first run has no baseline, so everything looks new. Billing a change rate for what is really a snapshot is not a pricing choice, it is a bug. Bill the baseline at snapshot rate, or not at all — and persist the watermark before charging, or a crash makes you re-bill the same changes next run.

Running it

I packaged this as Hostelworld Rate & Availability Monitor: forward-dated rate calendar, dorm/private split, promo stack with discount percentages, organic ranking position and the per-date commission split — HTTP-only, no login, no API key, no browser. Delta mode returns only the cells that moved.

There is a scheduled config ready to clone here: London hostel rate changes — daily delta.

dateStepDays: 7 is the cheap way to watch a long season — same weekday each week instead of every date.

If you would rather build your own, the five patterns behind the delta engine are written up in Scraping a price is easy. Knowing it changed is the product.

Watching a rate surface I have not covered? Reply with the source and I will tell you whether it is reachable anonymously and what the delta key should be.

The paginator returned HTTP 200 with zero results. 13,843 products were missing.

Sami — Sun, 26 Jul 2026 14:29:40 +0000

Every price-monitoring tutorial shows you the happy path: hit the search endpoint, ask for a big page size, walk the pages, done. Here is what actually happened when I pointed that at a UK safety-equipment catalogue with ~15,000 priced SKUs.

hitsPerPage=1000&page=0 → 1,000 products. Perfect.

hitsPerPage=1000&page=1 → HTTP 200, zero hits. No error. No warning. No hasMore: false. An empty array dressed up as a successful response.

If you trust that, you ship a monitor that reports "1,000 products" forever and nobody notices the other 13,843. Your buyer builds a dashboard on a tenth of the shelf.

Why the paginator lies

Most hosted search backends cap how deep you can page — often at ~1,000 results — because deep offsets are expensive to compute. The polite ones return an error explaining it. This one returns success with nothing in it, which is worse, because your code has no signal to react to.

So stop paginating by position and paginate by value instead. Sort by price ascending, then use the last price you saw as a filter for the next request:

body = {"hitsPerPage": 1000, "sortBy": ["price:asc"]}
if last_price is not None:
    body["numericFilters"] = [f"net_price_value>{last_price}"]

Each request starts where the previous one stopped, so you never ask for a deep offset at all. Result on this catalogue: 14,843 SKUs in 16 requests, no cap, and it self-heals if the catalogue grows mid-crawl.

Two details worth stealing:

Ties. If several products share a price, a strict > skips the ones after the 1,000th at that exact value. In practice you either accept the (tiny) loss or add a secondary sort key and filter on the pair. Measure before you optimise — on a real price distribution the collision count is usually zero.
Any sortable field works. Price is convenient here; an id or created_at cursor is often better if the field is monotonic and dense.

Whenever a paginator returns 200-with-nothing, look for a sortable field you can use as a cursor. It is almost always faster than the offset path it replaces.

The other trap: the same product, twice

Promoted or sponsored placements appear in the result set again at their natural position. Same SKU, two different index values. If you emit both, you have inflated the buyer's row count — and if you bill per row, you have overcharged them for a duplicate they can spot in five seconds.

by_sku: dict[str, dict] = {}
for hit in hits:
    sku = hit["sku"]
    prev = by_sku.get(sku)
    # keep the natural position, not the promoted slot
    if prev is None or hit["index"] < prev["index"]:
        by_sku[sku] = hit

And a 406 I caused myself

The sitemap fetch kept returning 406 Not Acceptable. The endpoint was fine; my client was sending Accept: text/html at an XML resource. One header:

headers = {"Accept": "application/xml,text/xml,*/*"}

Worth remembering the general shape: a 406 is almost never the server being broken, it is your Accept header disagreeing with what the resource can produce.

The part that makes it a product

A full catalogue snapshot is a big, boring file. What a purchasing manager or a competitor-pricing analyst actually wants is the diff: which SKUs moved, which appeared, which quietly vanished.

That last one is the interesting one, because a discontinued product does not announce itself. It simply stops being in the response. You can only detect it by comparing against what you stored last time:

current = {row["sku"] for row in rows}
gone = [sku for sku in previous if sku not in current]   # the discontinuation signal

Which means: rank the lifecycle change above the numeric one. When a SKU both moved price and disappeared from a category, the buyer's alerting rules care far more about "it is gone" than "it is £1.05 cheaper". I emit lifecycle_change first for exactly that reason.

And save the watermark before you charge. If you bill first and crash before persisting, the next run re-detects the same changes and bills them again — which is indefensible and entirely avoidable:

await save_seen(seen)          # persist BEFORE the charge
if not is_first_run:
    await charge("shelf-change", len(changes))

A first run has no baseline by definition, so everything looks new. Charging a change price for what is really a snapshot is not a pricing decision, it is a bug.

Fields that matter more than price

On a trade catalogue the price alone is not enough to act on:

ex-VAT and inc-VAT, separately. A trade buyer reasons in ex-VAT; a consumer-facing competitor lists inc-VAT. Mixing them produces confidently wrong comparisons.
Minimum order quantity. £6.95 per pair of gloves means something different at MOQ 1 than at MOQ 50.
Unit. "Per pair" vs "per box of 100" is the single most common false-positive in industrial price comparison.
Lifecycle stage, when the catalogue exposes it. It flags a discontinuation weeks before the SKU disappears.

Running it

I packaged the above as UK PPE & Safety Equipment Shelf Monitor on Apify: the whole priced catalogue in one run, HTTP-only (no browser, so it costs a fraction of a browser-based scraper), and a delta mode that returns only the rows that moved — price up or down, new listing, delisting — each annotated with what changed.

There is a prefilled example run on the Actor page, and a scheduled config here: UK PPE shelf changes — daily delta.

If you would rather build your own, the five patterns behind the delta engine — watermarks, fingerprints, price-cursor pagination, cell-state comparison, and save-before-billing — are written up in Scraping a price is easy. Knowing it changed is the product.

Monitoring a shelf I have not covered? Reply in the comments with the source and I will tell you whether it is reachable and what the delta key should be.

Scraping a price is easy. Knowing it changed is the product.

Sami — Sun, 26 Jul 2026 02:30:25 +0000

TL;DR

Most scraping tutorials stop at "here is the JSON". But almost nobody wants a snapshot — they want to know what moved since last time. That difference is not a formatting detail, it is where all the engineering lives: the source usually publishes no history, so if you don't store the previous state, the change is simply unknowable.

This post is the five patterns I use for that, the traps that cost me real money to find, and why "return only what changed" also happens to be the honest way to bill.

Why the snapshot is the wrong unit

Take a hotel rate. You scrape it today: €48. Useful? Barely. Now scrape it every morning and the interesting facts appear:

it dropped to €41 nine days before arrival
it was €48 for three weeks, then moved twice in one day
the property stopped being listed for that date entirely

None of those facts exist in any single response. They exist only in the diff, and the source will never hand you the diff — no rate site publishes "here is what I charged yesterday". The same is true of restaurant availability, pharmacy stock, and whether a SKU quietly vanished from a catalogue.

So the state you keep is the dataset. That reframing has a pleasant consequence: if your scraper stores state, it can bill for changes instead of rows, and a daily run over 5,000 SKUs costs the buyer almost nothing on a quiet day.

Pattern 1 — Monotonic id watermark (cheapest, when you can get it)

If the source hands out ids that only ever increase, you don't need to store content at all. Store the highest id you saw and stop paginating the moment you cross it.

last_id = int(await store.get_value("watermark") or 0)
highest = last_id

async for message in channel_history():
    if message["id"] <= last_id:
        break                    # everything below here was seen on a previous run
    highest = max(highest, message["id"])
    yield message

await store.set_value("watermark", highest)

Cost: one integer. Works for Telegram-style message feeds, forum post ids, filings sequences.

The trap: this is only sound if ids are monotonic and the feed is ordered. Plenty of APIs return "recommended" order by default, which brings us to the expensive lesson.

Pattern 2 — Content fingerprint (when ids are unstable)

Reviews are the classic case: no usable id, edited text, and the same review can surface under different internal keys. So hash the parts that identify it.

def fingerprint(review: dict) -> str:
    basis = "|".join([
        (review.get("author") or "").strip().lower(),
        (review.get("date") or "")[:10],
        (review.get("text") or "").strip()[:180],
    ])
    return hashlib.sha1(basis.encode()).hexdigest()[:16]

Then keep a set of seen fingerprints and emit only misses.

The trap that cost me a whole baseline

I shipped exactly this on a hotel-reviews scraper and the second run reported every review as new. The fingerprints were fine. The ordering wasn't.

The baseline run had paged through reviews in the site's default relevance order. The next run happened to load them in date order. Two runs, two disjoint slices of the same 4,000 reviews — so nothing matched, and the diff was garbage.

The fix is one line, and it belongs in every delta scraper you write:

# Delta mode must pin the sort order, ALWAYS — not just when a baseline exists.
# Relevance ordering is not stable between runs, so baseline and delta end up
# describing different slices and every fingerprint misses.
if await sort_newest_first(page):
    delta_stop_after = 1        # one all-seen page is now enough to stop

Pin the order, and "I hit a full page of already-seen items" becomes a trustworthy stop signal instead of a coin flip.

Pattern 3 — Price watermark pagination (when the paginator lies)

A UK catalogue I monitor exposes ~15,000 SKUs through a search API with a page parameter. At hitsPerPage=1000, requesting page=1 returns HTTP 200 with zero hits. No error, no warning — a silent cap dressed up as an empty shelf.

If you trust it, you ship a scraper that reports "1,000 products" forever and nobody notices the other 14,000.

The workaround is to stop paginating by position and paginate by value instead — sort by price and use the last price you saw as a filter:

body = {"hitsPerPage": 1000, "sortBy": ["price:asc"]}
if last_price is not None:
    body["numericFilters"] = [f"net_price_value>{last_price}"]

14,843 SKUs in 16 requests, no cap, and it self-heals if the catalogue grows mid-crawl. Any time a paginator returns 200-with-nothing, look for a sortable field you can use as a cursor.

Pattern 4 — Cell-state comparison (for things that are absent)

Availability is the awkward one: a sold-out restaurant returns nothing. There is no row to diff. The signal is the disappearance.

So the unit of state can't be a row, it has to be a cell in a grid you define — (venue × date × party size), (SKU × market), (property × arrival date):

key = f"{venue_id}|{date}|{party_size}"
prev = state.get(key)
now  = {"slots": len(slots), "prime": prime_count(slots)}

if prev is None:
    change = "new"
elif prev["slots"] > 0 and now["slots"] == 0:
    change = "sold_out"          # only knowable by comparison
elif prev["slots"] == 0 and now["slots"] > 0:
    change = "reopened"
elif abs(now["slots"] - prev["slots"]) > NOISE_FLOOR:
    change = "moved"
else:
    change = None                # unchanged: don't emit, don't bill

Two details that matter more than they look:

A noise floor. Slot counts jitter by ±1 constantly. Without a threshold your "changes" feed is mostly noise, and if you bill per change you are billing for jitter — which is the fastest way to earn a one-star review.
Rank first. When several things change at once, emit the lifecycle change (listed / delisted / sold out) ahead of the numeric one. A buyer's alerting rules care about "it disappeared" far more than "it moved by 3".

Pattern 5 — Save the state before you bill

Mundane, and the one that actually hurts if you get it wrong. Write the new watermark before charging, and make the first run of a stream cheap:

await save_seen(seen)            # persist BEFORE the charge
if not is_first_run:
    await charge("change", len(changes))

If you charge first and crash before persisting, the next run re-detects the same changes and bills them again. And a first run has no baseline by definition — everything looks new, so charging a "change" price for what is really a snapshot is indefensible. Bill the baseline at snapshot rate, or not at all.

What this looks like shipped

I built five monitors on these patterns this month, each one aimed at a source that publishes no history of its own:

Hostelworld Rate & Availability Monitor — forward-dated accommodation rates, promo stack with discount percentages, organic ranking position, per-date commission split.
Resy Availability & Scarcity Monitor — restaurant slots by date and party size, prime-window (18:30-20:59) scarcity, sold-out and reopened transitions.
DE/AT Pharmacy Price & Stock Monitor — OTC prices, RRP breaches, stock status across DE/AT.
UK Tyre Price & Fitment Monitor — tyre prices by size and postcode.
UK PPE Shelf Monitor — 14K+ SKUs, price moves, new listings and delistings.

All five run on a schedule, keep their own watermark, and return only the cells that moved. If you'd rather build your own, the patterns above are the whole trick — the state store is the moat, not the parser.

One last thing: run your own demo input

While writing this I ran all 18 of my public Actors with exactly the prefilled input a first-time user gets, and three came back with zero rows.

The worst one was my highest-traffic Actor. Its example profile URL still returned HTTP 200. The mode worked fine anonymously — other profiles returned data immediately. That one account had simply gone quiet, and every evaluator who pressed Run saw an empty dataset.

A status check on the URL would have said "fine". A schema validator would have said "fine". The only thing that catches it is running the thing and counting rows.

So if you maintain a scraper with a demo input: go run it right now, unedited, and count the rows. It takes a minute and mine had rotted without a single error anywhere.

If you build monitors for a living and want the delta logic explained for a specific source, reply in the comments — happy to dig into the pattern that fits it.

How to get Pinnacle odds (and no-vig fair prices) with Python in 2026 — after Pinnacle killed its public API

Sami — Tue, 21 Jul 2026 00:36:56 +0000

If you build betting models, run a +EV or arbitrage tool, or just track closing-line value, you already know the one line everyone benchmarks against: Pinnacle. Lowest margins, highest limits, no winner bans — its price is the closest thing the industry has to a "true price."

The catch: in July 2025 Pinnacle shut down its public API, restricting access to "select high-value bettors and commercial partnerships." A lot of tooling broke overnight. Here's a practical, keyless way to still pull Pinnacle's lines in Python — and, more importantly, turn them into the number you actually bet on: the no-vig fair price.

The demo: a live Pinnacle line, de-vigged

Pulled a few minutes ago — Pinnacle's moneyline on a Champions League qualifier:

Thun vs Dinamo Zagreb (h2h) — overround 7.1%:

Outcome	Pinnacle price	Implied %	No-vig fair %	Fair odds
Thun (home)	3.45	29.0%	27.1%	3.70
Draw	3.68	27.2%	25.4%	3.94
Dinamo (away)	1.96	51.0%	47.6%	2.10

The raw prices bake in Pinnacle's ~7% margin. Strip it out and Dinamo's fair price is 2.10, not 1.96 — that de-vigged number is what you compare every other book against to find +EV. The math is tiny:

prices = {"home": 3.45, "draw": 3.68, "away": 1.96}
implied = {k: 1/v for k, v in prices.items()}
overround = sum(implied.values())                 # 1.071  -> 7.1% vig
fair_prob = {k: v/overround for k, v in implied.items()}
fair_odds = {k: 1/p for k, p in fair_prob.items()}  # away -> 2.10

Do that across every book and the gaps are your edge. The hard part was never the math — it's getting Pinnacle's price reliably.

Pulling the odds

Pinnacle Odds Scraper does the fetch, keyless (no login, no API key). Minimal input:

{
  "mode": "pre_match_and_live",
  "sports": ["soccer", "tennis"],
  "markets": ["h2h", "spreads", "totals"]
}

You get one normalized row per outcome — homeTeam, awayTeam, league, marketType, outcomeLabel, price, commenceTime, isLive — for pre-match and in-play. Add "specials" for Pinnacle's 5,000+ per-sport markets (futures, exact-totals, first-to-score, team props) — the depth most odds feeds don't carry. Pay-per-use: $0.01 per pre-match snapshot, $0.02 live, $0.04 for the specials tier — no $249/mo subscription, you pay for what you poll.

Honest note: The Odds API also carries Pinnacle's main markets from $30/mo if you want a fixed-quota REST API — this actor's edge is the specials depth, the built-in no-vig/CLV derivation, and per-use pricing.

Why it's a recurring job, not a one-off

Two numbers matter to a serious bettor, and both need repeated pulls:

The line when you bet — odds move every few seconds; you snapshot at decision time.
The closing line — the single best predictor of long-run edge. Capturing CLV means pulling the price again right before kickoff, every event you track.

So the natural setup is a schedule: point it at your leagues, run it every minute pre-match and near-continuously toward kickoff, and log the series. Turn on delta mode + an Apify Schedule and each run only returns what changed — a gap-free odds history you pay for only when the line actually moves.

Sports don't have an off-season

Soccer, tennis, basketball, MMA, baseball — something sharp is priced year-round, so a Pinnacle feed is about as evergreen as alt-data gets. If you're building anything that needs a true-price benchmark, this is the input.

Ready-to-clone presets (closing-line EV, live in-play, specials/futures) are on the Actor's examples tab. Questions or a market you need? Open an issue on the Actor page.

How to pull China A-share sentiment signals — retail, analyst & news — with Python (2026)

Sami — Sun, 19 Jul 2026 23:29:04 +0000

For a quant or alt-data desk, Chinese A-shares are a rich, under-covered sentiment universe. FactSet literally sells "Guba Analytics" and "Xueqiu Analytics" side by side as retail-sentiment feeds, and every sell-side desk in China publishes ratings daily. The problem is plumbing: three different sources, three different anti-bot setups, no clean API.

Here's how to pull the three layers that actually move a name — retail chatter, analyst ratings, and live news — keyless, in Python, and fuse them into one read.

The demo: Kweichow Moutai (600519) right now

I ran all three on 贵州茅台 (Kweichow Moutai) — China's bellwether liquor stock — a few minutes ago. Here's the picture that falls out:

1. Retail sentiment (Eastmoney Guba 股吧) — net sentiment +0.13 across the latest posts (2 positive / 13 neutral / 0 negative). The retail crowd is buzzing about a price hike:

Post	Sentiment	Reads
飞天茅台价格大涨 — "Feitian Moutai price surges"	positive	255
茅台宣布涨价 — "Moutai announces price increase"	neutral	681
i茅台半小时售罄 — "i-Moutai app sold out in 30 min"	neutral	873

2. Analyst ratings (Eastmoney 研报) — the sell-side is unanimous. The latest reports, keyed to the ticker:

Broker	Rating	Change
诚通证券 (Chengtong)	买入 Buy	maintain
华鑫证券 (Huaxin)	买入 Buy	maintain
万联证券 (Wanlian)	增持 Accumulate	maintain
开源证券 (Kaiyuan)	买入 Buy	maintain

3. News flow (CLS 财联社) — the live telegraph stream, each flash tagged with a bull/bear score and the tickers it names, so market-moving headlines wire straight into an event model.

Put together: retail mildly bullish on a price-hike story, analysts uniformly Buy/Accumulate, steady news flow — a coherent, quantifiable read you can track over time.

How to pull each one

Each is a separate keyless Actor (no API key, no login). Minimal inputs:

Retail sentiment — China A-Share Guba Sentiment Monitor:

{ "stockCodes": ["600519", "300750"], "sentimentAnalysis": true }

Returns each post with a bull/bear score, read/reply counts, and a per-ticker rollup (net sentiment, share-of-voice, positive/negative split).

Analyst ratings — China A-Share Analyst Research Monitor:

{ "reportType": "stock", "stockCodes": ["600519"], "ratingChangesOnly": true }

Set ratingChangesOnly to get just the upgrades and downgrades — the tradable event is a broker moving a rating (增持 → 买入 is an upgrade).

News flow — China Financial Flash News Monitor (CLS 财联社):

{ "stockCodes": ["600519"], "sentimentAnalysis": true }

Filters the real-time telegraph to flashes that name your watchlist.

For the price/retail layer of the same name, Xueqiu (雪球) is the other retail source institutions buy alongside Guba.

Fusing it into one signal

The three layers answer different questions — what retail feels, what the sell-side rates, what just happened. Blended per ticker (retail net-sentiment + a rating score + news flow), they give you a single composite bull/bear read you can rank a watchlist by. Because each Actor keys everything on a stable id, you can turn on delta mode + an Apify Schedule and each run returns only what's new — a gap-free per-ticker sentiment time series, paying only for genuinely new items.

That time series — not any single snapshot — is what a systematic model actually consumes.

Questions or a field you need? Open an issue on any of the Actor pages.

How to track Chinese social media trends and brand mentions with Python (Weibo, RedNote, Bilibili) in 2026

Sami — Fri, 17 Jul 2026 21:07:55 +0000

China's social platforms — Weibo, RedNote (Xiaohongshu), Bilibili, Douban — are where a billion people talk about brands, products and news every day. For anyone doing market research, brand monitoring, or alt-data, they're also a black box: most Western scraping tools don't cover them, and the ones that do usually break on the anti-bot walls.

Here's a practical, keyless way to pull two things developers actually ask for — what's trending right now, and who's mentioning a brand — without maintaining a browser farm.

What's trending on Weibo right now

Weibo's hot-search board (微博热搜) is the single best real-time pulse of the Chinese internet. Here's the live top 10, pulled today (2026‑07‑17) in one run, with a rough English gloss:

#	Trending topic	English gloss	Heat
1	年轻人一定要对钱有概念	"Young people must get a sense of money"	218,466
2	功夫女足让韩国人破防了	China women's football stuns Korea	198,056
3	智能体互信互联互操作全球合作倡议	Global initiative for AI‑agent interoperability	141,701
4	雅迪把717骑行节搬进欢乐谷	Yadea brings its 7·17 cycling festival to a theme park	138,658
5	看不见的咖啡因终于可视化了	"Invisible caffeine, finally visualized"	98,229
6	西班牙晒大力神杯	Spain shows off the World Cup trophy	95,479
7	歌手排名	Singer rankings (music show)	67,683
8	青岛与世界干杯	Qingdao toasts the world (beer festival)	51,537
9	阿根廷决赛遇苦主裁判	Argentina's "nemesis referee" in the final	46,982
10	百花杀热度	Baihua Sha drama buzz	46,722

Notice #4: that's a brand — Yadea, the e‑bike maker — riding a trending hashtag off a live event. That's exactly the signal a brand or agency wants to catch the moment it happens, not a week later in a report.

Pulling this board is keyless (no login, no cookie):

{
  "mode": "hot_search"
}

That's the entire input. You get rank, title, hotValue, url and a freshness flag per topic. Actor: Weibo Scraper.

To turn a one‑off pull into a hands‑off feed, switch on hot_search_delta (or deltaMode for keyword search) plus a daily Apify Schedule — each run returns only what changed (new / rising / dropped topics), so you build a gap‑free trend history and pay only for genuinely new items.

Watching a brand across every platform at once

Trends are the top of the funnel. The recurring question is narrower: what are people saying about my brand this week, everywhere? Doing that means running five different scrapers with five different anti‑bot quirks and stitching the output together.

Chinese Brand Monitor collapses that into one scheduled call. Give it a brand name and it watches Weibo + RedNote + Bilibili + Douban + Xueqiu together, returns a unified mention feed with a bull/bear sentiment score per mention, and — the part people miss — it auto‑localizes the brand name: pass Nike and it also searches 耐克, Tesla → 特斯拉, so you don't lose the ~half of mentions that use the Chinese name.

{
  "brand": "Nike",
  "platforms": ["weibo", "rednote", "bilibili", "douban"],
  "sentimentAnalysis": true
}

Prefer the raw platforms directly? They're standalone too:
RedNote / Xiaohongshu · Bilibili · Weibo.

Why keyless matters

Every actor above is built to run on plain datacenter proxies for the public surfaces — no phone‑verified account to babysit, no login farm. Keyword search on RedNote and Weibo can be deepened with a cookieString you supply, but the trend boards, comments and public profiles work out of the box. You pay per result (or per genuinely‑new result in delta mode), so a quiet day costs nothing.

Wiring it into a pipeline

The stickiest setup: an Apify Schedule (daily/hourly) + a webhook or a Make/Zapier/Slack integration, so each finished run pushes straight into your warehouse or a channel. Pair it with delta mode and the monitor runs and delivers itself — no one opens a console.

If you're tracking a brand, a category, or the Chinese internet's mood in general, that's the whole loop: trends board for discovery, brand monitor for the narrow question, delta + schedule to keep it fresh.

Questions or a field you need? Open an issue on any of the actor pages — happy to help.

Foreign vs domestic in China: 525 social posts on who's winning the sentiment war

Sami — Tue, 07 Jul 2026 22:36:58 +0000

TL;DR — I scored 525 Chinese social posts on two foreign-vs-domestic rivalries. Coffee: Starbucks still beats Luckin on sentiment (+52% vs +38%) even though Luckin won the market. Sportswear: it flips hard — domestic Anta crushes Nike on sentiment (+43% vs +11%, with Nike at 37% negative). Market share and engagement are not brand love, and whether a foreign brand keeps China's affection is category-dependent. You only see it if you read the Chinese platforms Western tools don't cover.

"Is this Western brand winning or losing in China?" is usually answered with sales figures. But sales tell you what people bought, not what they feel — and in China, national-pride sentiment (国潮, guochao) is moving fast. So I pulled 5 days of aggregated Chinese social chatter (no personal data) on two rivalries and scored the sentiment.

Method

Platforms: Weibo (微博) + Bilibili (哔哩哔哩), some Douban
Window: 5 days · Sample: 525 brand mentions, sentiment-scored
Pipeline: cross-platform collection → dedup → Chinese-language sentiment → brand-level aggregation. Aggregate signals only — no individual posts, usernames, or personal data.

Case 1 — Coffee: Starbucks still wins the hearts

Brand	Mentions (SoV)	Positive	Negative	Net sentiment	Engagement
Starbucks 星巴克	126 (52%)	74%	21%	+52%	97,165
Luckin 瑞幸	116 (48%)	66%	28%	+38%	141,776

Luckin overtook Starbucks on stores and revenue — yet share of voice is a near-tie and Starbucks still runs 14 points higher net sentiment. Luckin wins the market and the engagement (its viral collabs drive +46% more interaction), but its aggressive-discount playbook also earns more mixed feelings. Market share ≠ brand love.

Case 2 — Sportswear: the flip. Domestic Anta crushes Nike

Brand	Mentions (SoV)	Positive	Negative	Net sentiment	Engagement
Nike 耐克	161 (57%)	48%	37%	+11%	854,626
Anta 安踏	122 (43%)	68%	25%	+43%	342,742

Here it inverts completely. Nike dominates the conversation (more mentions, 2.5× the engagement) — but that conversation is divisive and cool: only +11% net, with 37% outright negative. Domestic challenger Anta runs +43% net. In sportswear, the guochao effect is real and the foreign brand is losing the sentiment war even while it "wins" the volume.

The pattern that matters

Two foreign brands, two opposite outcomes:

Bigger buzz / engagement ≠ better sentiment. Luckin and Nike both out-engage their rivals while scoring worse on feeling.
The foreign-brand premium is category-dependent. Coffee still grants Starbucks affection; sportswear has turned on Nike. A single "China sentiment" number per brand would hide all of this.

If you sell into China, this is the layer under the sales figure — and it lives on Weibo, Xiaohongshu/RedNote, Bilibili and Douban, which most Western social-listening tools barely cover. That coverage gap is the whole point.

Reproduce it on your own brands

I run this with the Chinese Brand Monitor actor on Apify (Weibo + RedNote + Bilibili + Douban + Xueqiu, one scheduled call, sentiment + share-of-voice built in). Point it at your brand and its competitors and you get these same tables on a daily or weekly cadence.

→ Chinese Brand Monitor on Apify: https://apify.com/zhorex/chinese-brand-monitor

Prefer it as a managed monthly feed (delivered dataset, no setup, no calls)? Reply here or email samimassis2002@gmail.com and I'll send a free one-brand sample for your names.

All figures are aggregated and anonymized — brand-level signals only, no personal records.

We asked China's most-used AI about 5 global brands. Two of them are missing from half the answers.

Sami — Sun, 28 Jun 2026 17:14:03 +0000

TL;DR — We asked DeepSeek (one of the AI engines hundreds of millions of Chinese consumers now ask instead of searching) the same buyer questions you'd ask any assistant — "what are the best running shoes?", "which EV should I buy?" — for five global brands. The result: visibility on China's AI is wildly uneven. Nike, Apple and Tesla show up in 100% of relevant answers; Starbucks appeared in only 50%, and McDonald's was the top pick just 1 out of 3 times (KFC keeps beating it). In every category, Chinese challengers quietly surface. If you only track ChatGPT/Gemini, you can't see any of this.

Why we ran this

"GEO" — generative engine optimization, i.e. making sure AI assistants mention your brand — is the new SEO. But almost every GEO tool tracks only the Western engines (ChatGPT, Gemini, Claude). Meanwhile, in China, 900M+ people now ask AI assistants like DeepSeek, Qwen, Kimi and GLM for product recommendations. That's a blind spot for any brand selling into — or competing in — the China market.

So we ran a quick pilot with our own tool, the AI Brand Visibility Monitor, across five categories, asking DeepSeek the kind of questions real buyers ask.

What we found

Category	Western brand	Appeared in	Ranked #1	Sentiment	Chinese rivals surfacing
Sportswear	Nike	4/4 (100%)	4/4	+1.0	Li-Ning, Anta
Smartphones	Apple	4/4 (100%)	3/4	+0.75	Huawei, Xiaomi, Oppo, Vivo
Electric vehicles	Tesla	4/4 (100%)	3/4	+1.0	BYD, NIO, XPeng, Li Auto
Fast food	McDonald's	3/4 (75%)	1/3	+1.0	KFC
Coffee	Starbucks	2/4 (50%)	1/2	+1.0	Luckin Coffee, Manner

(Sentiment is a −1…+1 score on the mention; "Appeared in" = share of the four category prompts where the brand was named at all; "Ranked #1" = how often it was the first brand named.)

Three takeaways:

Being mentioned is not the same as being recommended. McDonald's appeared in most answers but was the top pick only once in three — KFC, the runaway fast-food leader in China, kept taking the #1 spot. Tesla and Apple each lost #1 once too, to Chinese rivals (BYD/NIO; Huawei/Xiaomi).
The real risk isn't bad sentiment — it's invisibility. Sentiment was positive everywhere (+0.75 to +1.0). The problem for Starbucks isn't that DeepSeek dislikes it; it's that DeepSeek often doesn't bring it up at all (50%), while Luckin Coffee and Manner — its Chinese challengers — do surface.
Chinese challengers show up in every single category. A brand monitoring only Western engines would never see that, in China's AI, Li-Ning, BYD, Huawei, KFC and Luckin are part of the consideration set.

How we measured it (methodology)

Engine: DeepSeek (this pilot; the tool also covers Qwen, Kimi, GLM and the Western engines).
Brands & prompts: 5 categories, one focal Western brand each, 4 real buyer-intent prompts per category (category, comparison, recommendation, and direct-brand questions), with the main Chinese competitors supplied for share-of-voice.
What each "check" returns: for one brand × one prompt × one engine — whether the brand was mentioned, the sentiment of that mention, its rank versus the named competitors, share-of-voice, and the sources the engine cited.
We report outcomes only — what the engines answer, not how the data is gathered.

Limitations (so nobody over-reads it)

This is a snapshot pilot: 5 brands × 4 prompts × 1 engine = 20 checks, single run. AI answers are volatile — studies suggest a large share of brand citations shift month to month — so a single run is a photo, not a movie. The point isn't statistical proof; it's a live demonstration that AI visibility in China diverges from what Western tools show, and that it's measurable. A fuller study (more brands, more engines, weekly tracking) is the obvious next step — which is exactly what the tool is built to do on a schedule.

See your own brand

Want to know whether DeepSeek, Qwen, Kimi and GLM mention your brand — and who they recommend instead? The AI Brand Visibility Monitor runs this for any brand and prompt set, across Chinese and Western engines, on a schedule. Run it once with no API key for a free sample.

It's the only tool that covers the Chinese AI engines — the engines Profound gates to Enterprise and most mid-market GEO tools don't track at all.

Your brand has an AI-search blind spot — and it's in China (DeepSeek, Qwen, Kimi, GLM)

Sami — Sat, 27 Jun 2026 23:53:10 +0000

TL;DR — When someone asks an AI "what's the best running-shoe brand?", that answer is the new shelf placement. There are good tools to track that on ChatGPT/Perplexity/Gemini. There are almost none for the Chinese engines — DeepSeek, Qwen, Kimi, GLM — where 900M+ people now ask instead of searching. I built one that does both in a single run. Try it on Apify.

The shelf moved into the answer

Search is being replaced by answers. People don't scroll ten blue links anymore — they ask an assistant and take the brands it names. So the question every brand now has to answer is no longer "where do I rank on Google?" but "do the AI engines even mention me — and how?"

That discipline has a name now: GEO (Generative Engine Optimization). The tooling is real and growing fast. But there's a hole in it.

The blind spot: the Chinese engines

Every GEO/AI-visibility tool I could find covers the Western engines — ChatGPT, Gemini, and friends. Almost none cover the Chinese ones: DeepSeek, Qwen/Tongyi, Kimi/Moonshot, GLM/Zhipu, where hundreds of millions of people now ask product questions, and where AI already drives a big chunk of product discovery.

If you sell into China — or your competitors do — that's the part of your AI visibility you literally cannot see today. That gap is the whole reason I built this.

What it measures

Give it your brand, the questions your customers actually ask, and the competitors you care about. For every prompt × engine it returns:

Mentioned? — is your brand named in the answer at all
Sentiment — positive / neutral / negative + a −1…1 score (English and Chinese lexicon)
Rank — where you sit vs the competitors you listed
Share of voice — you vs rivals in that specific answer
Cited sources — the URLs the engine surfaced (for search-enabled engines)
Answer snippet — the evidence behind the score

One record per brand × prompt × engine. Export JSON/CSV/Excel or pull it into your BI.

A monitor, not a one-off

A single snapshot of "does DeepSeek mention me" is nearly worthless, because AI answers are volatile — they move week to week. The value is the trend: did you just drop out of Qwen's answer? did a competitor overtake you on DeepSeek this week?

Turn on delta mode and put it on a schedule, and each run returns only what changed since last time — so a quiet week costs almost nothing and a real shift surfaces the moment it happens.

Example output

{
  "brand": "Nike",
  "engine": "deepseek",
  "prompt": "What are the best running shoe brands?",
  "mentioned": true,
  "sentiment": "positive",
  "sentimentScore": 0.5,
  "rank": 2,
  "competitorsMentioned": ["Adidas", "Li-Ning"],
  "shareOfVoice": 0.333,
  "answerSnippet": "For running shoes, the most recommended brands are Adidas, Nike and Li-Ning..."
}

How to run it

Set brand and a handful of prompts (the questions your customers ask).
Pick engines and add your own API key for each in apiKeys — pure pay-as-you-go, no subscription. (Run it with no key and you get a free labeled sample so you can see the output shape first.)
Add competitorBrands, turn on deltaMode, attach a Schedule → a hands-off weekly visibility feed.

Pricing is pay-per-event: $0.25 per visibility check (1 brand × 1 prompt × 1 engine), small add-ons for delta + competitors. You pay your own LLM usage on your own keys. Far below the $270–$2,000/mo enterprise GEO platforms — and the only one that sees the Chinese engines.

→ AI Brand Visibility Monitor on Apify

If you track brand visibility across AI search, which engines/prompts would you most want covered next (Doubao? ERNIE?)? Tell me in the comments and I'll likely ship it.