DEV Community: Alex Spinov

Your AI Agent Re-Reads Every Page It Already Saw. I Measured the 8x Context Tax

Alex Spinov — Sat, 13 Jun 2026 18:16:08 +0000

Turn 1 cost about 300 input tokens. Turn 20 cost 7,000. Same agent, same kind of page, 20 times more expensive for the last step than the first. Nothing was broken. The agent gave the right answer the whole way. It just kept paying for every page it had already read.

If you run a ReAct loop, a LangChain agent, or your own while loop that keeps the full transcript in messages, you are probably paying this. Here is the cumulative billed-input number, measured the same way for both strategies, plus the one counter-argument (prompt caching) that an honest version of this post has to address. There is a 40-line file at the end you can run in five seconds.

What is the context tax in an agent loop?

A naive agent keeps every fetched page in its message history. On turn k it re-sends pages 1 through k as billed input. Walk 20 different pages and the model gets charged for the first page 20 times. The sum is quadratic. A budget layer keeps a bounded window (the current page plus one short rolling summary), so the same walk stays linear. Each page is fine. The repetition is the tax.

I want to be careful here, because the last draft of this post was wrong. So let me show the number first, then the catch.

The numbers (run it yourself)

This is the real stdout of agent_context_budget.py. No network, no I/O, deterministic. The token count is a len // 4 proxy, not a tokenizer, and I will defend that choice below.

agent context tax: cumulative billed INPUT tokens (proxy, synthetic)
page proxy ~= 322 tokens, modeled on real page sizes (not measured here)

--- one session, 20 different growing pages ---
turn |  naive billed |  budget billed
   1 |           322 |            402
   2 |           966 |            804
   5 |         4,830 |          2,010
  10 |        17,774 |          4,084
  15 |        39,984 |          6,414
  20 |        71,844 |          8,744

N=20: naive=71,844  budget=8,744  -> 8.2x, -88%
with ideal prompt caching: naive=15,400  -> 1.8x (raw was 8.2x)

how the gap grows with N (same meter, both cumulative):
   N |     naive |  budget |  raw x | cached x
  10 |    17,774 |   4,084 |   4.4x |     1.4x
  20 |    71,844 |   8,744 |   8.2x |     1.8x
  30 |   164,514 |  13,404 |  12.3x |     2.2x

The headline: 20 pages, 8.2x more billed input for the naive loop, summed across the session. Drop the same agent down to a bounded window and you spend 88% less.

Look at turn 1, though. Budget costs 402, naive costs 322. The window layer is more expensive on a single turn, because the rolling summary is not free. The win only shows up once the transcript starts repeating. If your agent does two fetches and stops, skip all of this. The tax is a distance problem.

Why the naive loop is quadratic (this time it is actually modeled)

Turn k re-sends pages 1..k. So billed input per turn grows by about a page each turn (322, 644, 966, and so on). The running total is roughly page * (1 + 2 + ... + N), the sum of an arithmetic series. That is O(N squared). The budget loop sends one page plus a fixed summary every turn, so its total is about (page + 80) * N. Linear. (The pages are not perfectly equal in my fixture, so the printed naive total runs a little above the round-number formula. The shape is what matters, and the printed numbers are what you reproduce.)

That is the whole shape of it. Roughly the same page size, the same agent, one strategy keeps re-serializing history and one does not. The raw x column climbs 4.4, 8.2, 12.3 because the gap between a parabola and a line widens with N. It is not a constant multiplier. It is a tax that gets worse the longer your agent works.

I am stressing this because my first version of this article claimed "quadratic" in prose while the code was actually linear (one page re-sent N times has no curve). A reviewer caught it. This version walks N different growing pages, so the quadratic is in the code, not in the adjective.

The honest part: doesn't prompt caching fix this?

Yes, someone is already typing it. "Anthropic and others cache the prompt prefix, so the repeated transcript is billed at roughly a tenth. Your naive loop is not 8x, it's barely worse." Fair. If I hid that, this post would deserve the same fate as the last one.

So I modeled it. run_naive_cached bills the already-cached tail at 0.1x (cache read) and writes each new page once at 1.25x (cache write), which is roughly Anthropic's published cache pricing shape. The result is the cached x column above:

N pages	naive raw	naive cached	budget	cached vs budget
10	17,774	5,554	4,084	1.4x
20	71,844	15,400	8,744	1.8x
30	164,514	29,106	13,404	2.2x

Caching cuts the gap hard. It does not close it. Even with a perfect cache, the naive loop still re-reads the whole growing tail every turn, and that cheap-but-not-free read still adds up faster than a bounded window. At 20 pages it is 1.8x. The snowball survives the best case you can give it.

And the best case is generous. Anthropic's prompt cache has a 5-minute TTL. An agent loop with slow tool steps (a fetch, a parse, a model call, another fetch) can easily blow past that between turns, and then the tail falls out of cache and you pay closer to the raw 8.2x. So the real-world number for a tool-heavy agent sits between 1.8x and 8.2x, leaning toward raw the slower your steps are. I will not pretend to know exactly where your loop lands. That depends on your TTL luck and your step latency.

Where our production data actually comes in

I do not have a clean public benchmark of "agent transcript growth across 20 fetches," so I am not going to fake one. What I do have is real page sizes. Across roughly 2,190 production runs on our scrapers, listing and review page bodies after cleaning tend to land in the few-hundred-token range, which is where I set the fixture's page size. The Trustpilot scraper alone has 962 runs, so "a review page is about this big" is something I have actually watched, not guessed.

That is the only place real data touches this post: the size of a page, used as a parameter. I did not measure the multiplier 962 times. The multiplier comes from the model. Be suspicious of anyone who blurs those two.

About that proxy token count

toks(text) is len(text) // 4. Not tiktoken. I did this on purpose, and not to be lazy.

If I pinned a real tokenizer, the exact stdout would shift every time the library version moved, and you could not reproduce my MD5. The number you care about is a ratio (naive over budget), and a constant proxy cancels out of a ratio. Whether a page is 322 proxy-tokens or 410 real ones, naive-over-budget barely moves. What I lose is the right to say "your invoice will read exactly $X." What I keep is a result you can rerun and get bit-for-bit identical. For a structural argument about O(N squared) versus O(N), that is the correct trade.

I ran it twice and diffed the output. Same MD5 both times. If you run the file at the bottom, you should get the same stdout I pasted above.

The fix is smaller than the problem

The budget layer is three ideas, none clever:

Send the current page, not the whole transcript.
Keep one short rolling summary of what came before, capped at a fixed size.
Never let the window grow with the turn count.

That is it. The run_budget function is four lines. The hard part is not the code, it is noticing the bill in the first place, because every quality metric stays green while the cost curve bends upward in the background.

This is the failure class I keep running into with agents: the thing works. Every page is a clean 200. Every answer is correct. The eval dashboard is green. And the only signal that something is wrong is a token bill that grows faster than the task list. It is not a logic bug you can catch in a test. It is a quiet tax on the loop, and quiet taxes are the ones you pay longest, because nothing ever pages you about them.

If you want to see it in your own agent, open the token log and plot billed input per turn. A flat-ish line means you have a window. A line that climbs every turn means you are paying the tax, and the slope is your N.

Where this is wrong

I would rather you trust the limits than the headline.

The rolling summary loses detail. An 80-token summary of fifteen pages is lossy, and if the agent needs an exact fact from page 3 on turn 18, a naive loop still has it verbatim and the budget loop might not. That is a real correctness cost, not a free lunch. The honest framing is a trade: you spend less, you remember less precisely.

The budget can drop a page the agent later wants. My window assumes the current sub-task only needs the current page plus a gist. Some tasks genuinely need to cross-reference page 2 and page 19. For those, a fixed window is the wrong tool and you want retrieval, not truncation.

And the numbers are mine. N=20, that page size, that summary size, ideal-or-zero caching. Your agent, your pages, your TTL will give a different multiplier. The shape (quadratic versus linear) holds. The exact 8.2x does not transfer. Treat it as the direction, not the destination.

This is the third in a small line of notes about giving an agent a web tool that does not quietly hurt you: one gave the agent a fetch tool, one taught it not to trust a 200 OK that was garbage, and this one is about not drowning the context in pages it already read. Each one is a curl that returns 200 and a cost you only see later.

Written by Aleksej Spinov. Numbers in this post are the output of a deterministic synthetic model (agent_context_budget.py, included in full below); the only real-world input is page size, modeled on roughly 2,190 production scraper runs. AI-assisted drafting, human-reviewed and run before publishing.

Follow for the next set of numbers from production runs, and tell me in the comments: what does billed-input-per-turn look like in your own agent log? I read every reply. 👇

The full file

This is agent_context_budget.py exactly as I ran it. Pure stdlib, no dependencies, no network. Save it, run python3 agent_context_budget.py, and you should get the stdout pasted near the top of this post.

#!/usr/bin/env python3
"""agent_context_budget.py - what a naive agent loop pays vs a windowed one.

The fetch already works (#18) and each page is real content (#19). The problem
is the LOOP. A naive ReAct/while agent keeps the FULL transcript in `messages`,
so on turn k it re-sends pages 1..k as billed input. Walk N DIFFERENT pages and
that sum is O(N^2). A budget layer keeps a bounded window - the current page
plus one rolling summary of the past - and the same walk costs O(N).

SAME METER FOR BOTH SIDES. Every function below returns the cumulative billed
INPUT tokens summed over the whole session. Not marginal-on-one-side,
cumulative-on-the-other (that mistake is exactly what made an earlier draft of
this post wrong). Both strategies are charged for everything they actually send
to the model on every turn.

Three strategies over ONE agent log of N different, growing pages:
  run_naive(pages)        - full transcript re-sent every turn          -> O(N^2)
  run_budget(pages)       - current page + one rolling summary          -> O(N)
  run_naive_cached(pages) - naive WITH an ideal prompt cache on the tail

Pure functions. No network, no I/O, deterministic: same fixtures in, same stdout
out (stable MD5). Token count is a deterministic len(text)//4 PROXY, not a real
tokenizer - on purpose, so anyone re-running gets the identical MD5 without
pinning a tiktoken version. The reported numbers are RATIOS, so the proxy
constant cancels out. It is a model, not your invoice.
"""

SUMMARY_TOKENS = 80          # one rolling summary of everything seen so far
CACHE_READ = 0.1             # ideal prompt cache: cached tail billed at ~0.1x
CACHE_WRITE = 1.25           # writing the new tail to cache costs ~1.25x once


def toks(text):
    "Deterministic token PROXY: ~4 chars = 1 token. No tokenizer dependency."
    return len(text) // 4


def run_naive(pages):
    "Full transcript re-sent every turn. billed(k) = sum(tokens 1..k). O(N^2)."
    total, transcript = 0, 0
    for page in pages:
        transcript += toks(page)         # this page joins the transcript...
        total += transcript              # ...and the WHOLE transcript is re-sent
    return total


def run_budget(pages):
    "Bounded window: current page + one rolling summary. billed is O(N)."
    total = 0
    for page in pages:
        total += toks(page) + SUMMARY_TOKENS   # current page + fixed summary
    return total


def run_naive_cached(pages):
    "Naive with an IDEAL prompt cache: old tail billed at CACHE_READ, new at write."
    total, cached_tail = 0, 0
    for page in pages:
        new = toks(page)
        # re-read the whole already-cached tail cheap, then write this page once
        total += cached_tail * CACHE_READ + new * CACHE_WRITE
        cached_tail += new
    return round(total)


# ---- fixture (SYNTHETIC, labelled) ------------------------------------------
# N different pages an agent fetches across one task. Page size (~320 token
# proxy) is modeled on real listing/review page bodies from our production logs
# (the Trustpilot scraper alone has 962 runs), NOT measured per page here. A
# heavier page only widens the gap. Pages differ so the transcript keeps growing.
def make_pages(n):
    return [f"PAGE {i:02d} " + (f"row{i}-" * 256) for i in range(1, n + 1)]


if __name__ == "__main__":
    print("agent context tax: cumulative billed INPUT tokens (proxy, synthetic)")
    print("page proxy ~= %d tokens, modeled on real page sizes (not measured here)\n"
          % toks(make_pages(1)[0]))

    N = 20
    pages = make_pages(N)
    print(f"--- one session, {N} different growing pages ---")
    print(f"{'turn':>4} | {'naive billed':>13} | {'budget billed':>14}")
    nb = tb = 0
    for k, page in enumerate(pages, 1):
        nb = run_naive(pages[:k])
        tb = run_budget(pages[:k])
        if k in (1, 2, 5, 10, 15, 20):
            print(f"{k:>4} | {nb:>13,} | {tb:>14,}")

    naive = run_naive(pages)
    budget = run_budget(pages)
    cached = run_naive_cached(pages)
    print(f"\nN={N}: naive={naive:,}  budget={budget:,}"
          f"  -> {naive/budget:.1f}x, -{100*(1-budget/naive):.0f}%")
    print(f"with ideal prompt caching: naive={cached:,}"
          f"  -> {cached/budget:.1f}x (raw was {naive/budget:.1f}x)\n")

    print("how the gap grows with N (same meter, both cumulative):")
    print(f"{'N':>4} | {'naive':>9} | {'budget':>7} | {'raw x':>6} | {'cached x':>8}")
    for n in (10, 20, 30):
        ps = make_pages(n)
        nv, bd, cd = run_naive(ps), run_budget(ps), run_naive_cached(ps)
        print(f"{n:>4} | {nv:>9,} | {bd:>7,} | {nv/bd:>5.1f}x | {cd/bd:>7.1f}x")

Your AI Agent Trusts a 200 OK. I Logged How Often the Page Was Garbage

Alex Spinov — Fri, 12 Jun 2026 18:09:10 +0000

Yesterday I handed an agent a web_fetch tool. It fetched a page, got back a 200 and a screenful of text, and confidently built a plan on it. The text was a Cloudflare "Just a moment..." screen. The agent never noticed.

That's the failure I want to fix today. Not the fetch. The trust. Your tool returns status=200 and a non-empty string, and your agent treats that string as "what the page said." Most of the time it is. Sometimes it's a challenge wall, an empty shell, or a body cut off mid-stream, and the agent reasons on garbage with full confidence and zero error.

So here's a 40-line gate that sits between the fetch tool and the agent and answers one question: is this blob usable as content at all? It tags every fetch OK / BLOCKED / EMPTY_SHELL / TRUNCATED before your model ever reads it. Real, deterministic output below.

Quick answer: A web-fetch tool's 200 OK + non-empty body does not mean "usable content." That body can be an anti-bot challenge, an empty JS shell, an "access denied" notice, or a truncated stream, all served as 200. sanity_check(text, url, status) runs zero network calls and tags the blob OK / BLOCKED / EMPTY_SHELL / TRUNCATED before your model reads it. Garbage becomes an explicit signal, not a silent input to reasoning.

This is for anyone building agents with web access (LangChain, a ReAct loop, an MCP tool, your own while loop) who has watched the model be confidently wrong and couldn't tell why. The why is often this: the page lied with a 200, and nothing checked.

The artifact first: four verdicts on six blobs

Here's the real output, copy-pasted, not cleaned up. The gate ran against six fixtures: one real captured page (example.com) and five synthetic bodies I hand-wrote to reproduce failure classes I've hit in production. The synthetic ones are labeled synthetic so you don't mistake them for live pulls.

VERDICT      KIND       URL
----------------------------------------------------------------
OK           real       https://example.com
                        → len=353B ratio=0.57
BLOCKED      synthetic  https://shop.example/product/991
                        → soft-block marker 'just a moment...' (status=200)
BLOCKED      synthetic  https://api.example/v2/orders
                        → soft-block marker 'access denied' (status=200)
EMPTY_SHELL  synthetic  https://app.example/dashboard
                        → visible≈0B ratio=0.00 (markup, no content)
TRUNCATED    synthetic  https://blog.example/crawl-bill
                        → no </html> / mid-tag end …'were blunt: most of the spend was on <sp'
EMPTY_SHELL  synthetic  https://example.com/empty
                        → empty body (status=200)
----------------------------------------------------------------
1 of 6 blobs were usable content  ::  {'OK': 1, 'BLOCKED': 2, 'EMPTY_SHELL': 2, 'TRUNCATED': 1}

Every one of those six rows was a 200. One was content. The other five were the kinds of 200 that look fine to a fetch tool and ruin a downstream plan. The gate gives each a name your agent can branch on instead of a string it has to believe.

I ran this on Python 3.13.5. No third-party imports, no network. Run python3 fetch_sanity.py and you get the same bytes. I checked: two consecutive runs hash to the same md5. That matters, and I'll come back to why.

Why "200 and not empty" is the wrong success check

Tool contracts lie by omission. A fetch tool's idea of success is usually two things: the HTTP status was 2xx, and the body wasn't empty. Both can be true while the body is useless.

I run scrapers in production: roughly 2,190 runs across 32 published actors, the Trustpilot one alone at 962 runs. The failure that cost me the most debugging time wasn't a 500 or a timeout. Those are loud; you catch them. It was the 200 that came back with a body that wasn't the page. Four shapes show up again and again:

The challenge wall. Cloudflare's "Just a moment...", Akamai's "Access Denied", a generic "verify you are human", served with status 200, not 403. The bytes are real HTML. They're just not your page.
The empty shell. A single-page app ships <div id="root"></div> and three script tags. The data renders in a browser. Your raw fetch got the skeleton. (Predicting that before you fetch is its own post; see below.)
The truncated body. A size cap, a dropped connection, or a slow stream cut off, and you got the first 8 KB of a 40 KB page. Looks like a page. Ends mid-sentence.
The literal empty 200. Some servers answer a blocked or rate-limited request with 200 and a body of "". I once watched a scraper return empty arrays for days because nobody raised on a soft-blocked request that came back 200-and-nothing.

Here's the agentic twist that makes this worse than in a plain scraper. A scraper that gets garbage produces a bad row, and a human eventually eyeballs the table. An agent that gets garbage feeds it straight into its own next decision (calls another tool, writes a summary, answers the user) with no human in the loop. The silent failure compounds. There's no exception to catch and no retry to trigger, because as far as every layer can tell, the fetch succeeded.

The fix is not smarter prompting. It's a checkpoint that turns a silent 200 into an explicit verdict before the model reasons.

What the gate actually checks

The gate is a heuristic, and I'd rather hand you a blunt one I trust than a clever one I can't reason about. It checks three things, in order, and stops at the first hit.

1. Soft-block markers. A short list of strings that mean "this is a wall, not a page": just a moment..., enable javascript and cookies to continue, attention required, access denied, verify you are human, cf-ray, the captcha vendors. Match any (case-insensitive) and the verdict is BLOCKED, even at status 200, especially at status 200. These are strings I've watched come back from real targets dressed as success.

m = _BLOCK_RE.search(low)
if m:
    return "BLOCKED", f"soft-block marker {m.group(0)!r} (status={status})"

2. Visible-text-to-markup ratio. Strip <script> and <style>, strip the remaining tags, measure what readable text is left versus the size of the whole blob. A real article is mostly words. An empty shell is mostly markup, a ratio near zero. The verdict is EMPTY_SHELL in two cases: the body is literally empty, or it's markup with under ~200 bytes of visible text and a ratio under 0.10. Note the second clause: a short blob that's almost all readable text (a terse JSON reply, a one-line message) has a high ratio, so it stays OK, not EMPTY_SHELL. The shell verdict is specifically for "lots of markup, almost no words."

3. Truncation. If the body opened an <html> tree and never closed it, or ends mid-tag, it got cut off. Verdict TRUNCATED. The reason string even echoes the last 40 characters so you can see where it stopped. …'most of the spend was on <sp' is a body that died inside a <span>.

Nothing tripped? OK. The agent may proceed.

That's the whole decision surface. Notice what it is not doing. It does not validate any field's value: no checksums, no ranges, no cross-field logic. It does not compare this fetch to a previous one or track a schema over time. It does not decide whether you needed a browser. It answers exactly one question, is this a page at all?, and gets out of the way. The narrowness is the feature.

The thresholds are deliberately blunt

200 bytes of visible text. A ratio under 0.10. These aren't fitted to anything. They're "is there clearly almost no content here," with everything above left as OK. Tune them to your traffic: a site that ships a thin-but-real <title> and a 150-byte summary will trip EMPTY_SHELL at these numbers, which might be a false alarm for you. Raise the floor. The point isn't my constants. It's that "is this usable content" is answerable from bytes you already have, before the model spends a token on them.

And the soft-block list is a denylist, so it's never complete. A challenge page with wording I haven't seen sails through as OK. The list catches the vendors I've met across the fleet; it won't catch a custom wall some site rolls tomorrow. More on that in the failure modes.

Wiring it into an agent loop

The gate is a pure function, so it drops in wherever your tool returns. The pattern: fetch, gate, branch.

text, status = my_web_fetch(url)          # your existing tool
verdict, reason = sanity_check(text, url, status)

if verdict == "OK":
    observation = text                    # let the agent reason on it
else:
    observation = f"[fetch unusable: {verdict}] {reason}"   # tell the agent the truth

The second branch is the whole point. Instead of handing the model a challenge page and hoping it notices, you hand it [fetch unusable: BLOCKED] soft-block marker 'just a moment...'. Now the model knows the observation failed and can do something sane: try a different source, escalate to a browser-based fetch, or tell the user it couldn't read the page, instead of confidently summarizing a captcha screen.

That's the move: convert a silent success into a spoken failure. Models are good at reacting to an error they can see. They're terrible at noticing one nobody told them about.

Where this lives in the chain (and the two siblings)

This gate is one checkpoint in a longer pipeline, and it's easy to confuse with neighbors, so here are the seams.

If you want to predict whether a page will even come back as a shell before you fetch it, that's a different tool that reads the raw response shape. I wrote a 30-line probe that tells you if a page needs a browser. That one runs before the fetch. This one runs after: it doesn't switch renderers or decide how to fetch, it just flags the agent that the blob it got back is a skeleton.

And the fetch tool itself, the thing that produced the 200, is the 60-line MCP web_fetch server I built yesterday. That post ends with an honest warning: it "does not beat anti-bot systems" and returns "nothing useful" on a challenged page. This gate is the answer to what do you do with that nothing. You gave the agent eyes; now you teach it not to trust a forgery.

Where this gate is wrong

It's a heuristic. I'd rather tell you the misses than let you find them in an agent that's already in front of a user.

A challenge page in wording I haven't met. The denylist catches Cloudflare, Akamai, the common captchas. A site that rolls its own "please hold" page with novel text will pass as OK. There's no clean way around this short of a model-based classifier, which costs tokens on every fetch: the opposite of the point.
A short page that is genuinely the content. A 120-byte API error rendered as JSON, a terse status page, a stub doc: these can trip EMPTY_SHELL when they're exactly what you asked for. The ratio test can't tell "empty shell" from "small real page." Tune the floor, or skip the ratio check for endpoints you know return short bodies.
A truncated body that happens to close its tags. If the cut-off landed right after a </html> (rare, but possible with a buffered proxy) the truncation check misses it. Length-versus-Content-Length would catch that, but my fetch tool doesn't always have a reliable Content-Length, so I left it out rather than half-implement it.
A truncated body with no markup at all. The truncation check only fires on HTML (it looks for an unclosed <html> tree or a mid-tag cut). A JSON or plain-text response that got chopped mid-array has no tags to read, so it sails through as OK. For JSON endpoints, pair this with a json.loads in a try and treat a parse failure as TRUNCATED yourself.
A blocked page with a 403. This gate is for the 200-shaped lie. If your fetch tool already raises on 4xx/5xx (mine does), those never reach here, which is correct. The gate exists for the failures your status check waves through.

That last point is the design line. The gate doesn't replace your status handling. It catches the class your status handling is structurally blind to: success codes carrying non-content.

Why "no network" isn't a footnote

The gate makes zero network calls on purpose. sanity_check(text, url, status) is a pure function of its inputs: same blob in, same verdict out. That buys three things. Tests pin a fixture to a verdict and never flake. The output above is reproducible, byte for byte (I checked the md5). And a gate that called out to a live anti-bot site to "confirm" a block would add latency, egress, and a second thing that can fail. The blob already arrived. Everything we need to judge it is in the bytes.

Same discipline as every checkpoint I ship: the browser probe, the schema canary, the field sanity checks are all pure functions over data you already have. It's what lets the next person re-run them and get my exact result instead of taking my word.

What I'd do on Monday

Put the gate right after your fetch tool returns, before the result becomes an observation.

Gate every fetch, not just the ones that look wrong. The whole problem is that the bad ones look fine. A 200 with a screen of HTML is exactly what a challenge wall looks like.
Feed the verdict to the agent, not just your logs. [fetch unusable: BLOCKED] in the observation lets the model route around the failure. A line in your logs that the model never sees does not.
Tune the soft-block list to your own traffic. Watch what your targets actually send back as a 200 for a week, and add the strings you see. The list in the file is my fleet's, not yours.

I'll be straight about the limit: this catches the common shapes of a 200 that isn't content. It will not catch a clever, novel wall, and it can't tell a tiny real page from an empty one without your help on the threshold. It's a floor, not a fortress. But most agent loops I've seen don't even have the floor. They hand the model the string and hope.

Here's the open question I haven't solved cleanly. The EMPTY_SHELL check and the soft-block list are both content-shape signals: they look at the blob in isolation. But the strongest signal that a fetch failed is often relative. This page is 200 bytes when the same URL gave 40 KB yesterday, or every URL on this host suddenly returns the same challenge string. That's drift across fetches, and a pure per-blob function can't see it without state. If you've found a cheap way to fold "this looks wrong compared to last time" into a per-call gate without dragging a database into your agent loop, I genuinely want to see it.

What's the worst 200 OK your agent ever believed, and what tipped you off that the page was garbage? 👇

Follow for the next checkpoint from our production runs. I read every comment.

Full script (fetch_sanity.py, stdlib only, no network, the exact file I ran):

#!/usr/bin/env python3
"""fetch_sanity.py — one gate between a web-fetch tool and your agent's reasoning.

A fetch tool can return HTTP 200 and a non-empty body that is still NOT content:
an anti-bot challenge page, an empty JS shell, an "access denied" notice, or a
body that got cut off mid-stream. The agent treats all of it as "the page said
this" and plans on garbage. This function answers ONE question before reasoning:
is the returned blob usable as content at all?

    sanity_check(text, url, status) -> (verdict, reason)
    verdict in {"OK", "BLOCKED", "EMPTY_SHELL", "TRUNCATED"}

Pure function. No network, no I/O, deterministic: same (text, url, status) in,
same verdict out. Run it, diff it, trust it. It is a heuristic, not an oracle —
the "Where it's wrong" section of the post is honest about the misses.
"""
import re

# Soft-block markers: a 200 body that is really a challenge / denial wall.
# Lowercased substring/regex match against the body. Each one is a real string
# I have seen come back with status 200 instead of 403.
BLOCK_MARKERS = [
    r"just a moment\.\.\.",          # Cloudflare interstitial
    r"enable javascript and cookies to continue",
    r"attention required",           # Cloudflare block page title
    r"access denied",
    r"verify you are (?:a )?human",
    r"are you a robot",
    r"complete the security check",
    r"cf-ray",                       # Cloudflare ray id leaks into the body
    r"px-captcha|hcaptcha|g-recaptcha|/recaptcha/",
    r"request unsuccessful\. incapsula",
]
_BLOCK_RE = re.compile("|".join(BLOCK_MARKERS), re.IGNORECASE)

_TAG_RE = re.compile(r"(?is)<(script|style)\b.*?</\1>")
_ANYTAG_RE = re.compile(r"(?s)<[^>]+>")


def _visible_ratio(text):
    """Fraction of the blob that is visible text after stripping script/style/tags.
    A real article is mostly words; an empty JS shell is mostly markup."""
    if not text:
        return 0.0
    stripped = _TAG_RE.sub(" ", text)
    stripped = _ANYTAG_RE.sub(" ", stripped)
    visible = re.sub(r"\s+", " ", stripped).strip()
    return len(visible) / len(text)


def sanity_check(text, url, status):
    """Return (verdict, reason) for one fetched blob. No network calls."""
    body = text or ""
    low = body.lower()

    # 1) BLOCKED — a soft-block / challenge / denial wall served as 200.
    m = _BLOCK_RE.search(low)
    if m:
        return "BLOCKED", f"soft-block marker {m.group(0)!r} (status={status})"

    # 2) EMPTY_SHELL — almost nothing to read. Either a literal empty body
    #    (200 + ""), or markup with the content rendered client-side: a raw
    #    fetch handed the agent a skeleton, not a page.
    ratio = _visible_ratio(body)
    has_markup = "<" in body and ">" in body
    visible_len = int(ratio * len(body))
    if visible_len < 200:
        if not body.strip():
            return "EMPTY_SHELL", f"empty body (status={status})"
        if has_markup and ratio < 0.10:
            return "EMPTY_SHELL", f"visible≈{visible_len}B ratio={ratio:.2f} (markup, no content)"

    # 3) TRUNCATED — body cut off mid-stream: opened an HTML tree but never
    #    closed it, or ends mid-tag / mid-word with no terminal punctuation.
    if has_markup:
        opened_html = "<html" in low
        closed_html = "</html>" in low
        ends_mid_tag = bool(re.search(r"<[a-z][^>]*$", body.rstrip(), re.IGNORECASE))
        if (opened_html and not closed_html) or ends_mid_tag:
            tail = body.rstrip()[-40:].replace("\n", " ")
            return "TRUNCATED", f"no </html> / mid-tag end …{tail!r}"

    # 4) OK — nothing tripped. The agent may reason on this.
    return "OK", f"len={len(body)}B ratio={ratio:.2f}"


# ---------------------------------------------------------------------------
# Fixtures. One REAL captured body (example.com, fetched once and pasted in as a
# byte-for-byte string so this stays offline/deterministic) + five SYNTHETIC
# bodies hand-written to reproduce failure classes I have hit in production.
# Synthetic ones are labeled (synthetic) so nobody mistakes them for a live pull.
# ---------------------------------------------------------------------------

# Real: the actual body of https://example.com (RFC-style sample page, public,
# unchanging). Captured once, hardcoded so the gate needs no network.
EXAMPLE_COM = (
    "<!doctype html><html><head><title>Example Domain</title></head><body>"
    "<div><h1>Example Domain</h1><p>This domain is for use in illustrative "
    "examples in documents. You may use this domain in literature without prior "
    "coordination or asking for permission.</p>"
    "<p><a href=\"https://www.iana.org/domains/example\">More information...</a>"
    "</p></div></body></html>"
)

# Synthetic: a Cloudflare "Just a moment..." interstitial served with status 200.
CF_CHALLENGE = (
    "<!DOCTYPE html><html lang=\"en-US\"><head><title>Just a moment...</title>"
    "<meta http-equiv=\"refresh\" content=\"390\"></head><body>"
    "<div class=\"main-wrapper\"><h1>example.com</h1>"
    "<h2>Checking if the site connection is secure</h2>"
    "<p>example.com needs to review the security of your connection before "
    "proceeding.</p></div>"
    "<!-- cf-ray: 8e2a1f0c9d4e7b21-FRA --></body></html>"
)

# Synthetic: an "Access Denied" wall (Akamai-style) returned as 200.
ACCESS_DENIED = (
    "<html><head><title>Access Denied</title></head><body>"
    "<h1>Access Denied</h1>"
    "<p>You don't have permission to access this resource.</p>"
    "<p>Reference #18.abcd1234.1718200000</p></body></html>"
)

# Synthetic: an empty SPA shell — all markup, the content arrives via JS.
JS_SHELL = (
    "<!doctype html><html><head><meta charset=\"utf-8\">"
    "<link rel=\"stylesheet\" href=\"/static/app.css\"></head><body>"
    "<div id=\"root\"></div>"
    "<script src=\"/static/runtime.js\"></script>"
    "<script src=\"/static/vendor.js\"></script>"
    "<script src=\"/static/main.js\"></script></body></html>"
)

# Synthetic: a real article body that got cut off mid-stream (size cap / dropped
# connection). Opens <html>, never closes it, ends mid-tag.
TRUNCATED_BODY = (
    "<!doctype html><html><head><title>How we cut the crawl bill</title></head>"
    "<body><article><h1>How we cut the crawl bill 82%</h1>"
    "<p>We started by measuring the per-run cost of a headless browser across "
    "every target in the fleet. The first surprise was how often we paid for "
    "Chrome on pages that answered a plain GET in eighty milliseconds. The second "
    "was the cost of the pages that came back empty. We logged each run and "
    "tallied the verdicts, and the numbers were blunt: most of the spend was on "
    "<sp"
)

FIXTURES = [
    ("https://example.com",                 EXAMPLE_COM,     200, "real"),
    ("https://shop.example/product/991",    CF_CHALLENGE,    200, "synthetic"),
    ("https://api.example/v2/orders",       ACCESS_DENIED,   200, "synthetic"),
    ("https://app.example/dashboard",       JS_SHELL,        200, "synthetic"),
    ("https://blog.example/crawl-bill",     TRUNCATED_BODY,  200, "synthetic"),
    ("https://example.com/empty",           "",              200, "synthetic"),
]


if __name__ == "__main__":
    tally = {}
    print(f"{'VERDICT':<12} {'KIND':<10} URL")
    print("-" * 64)
    for url, body, status, kind in FIXTURES:
        verdict, reason = sanity_check(body, url, status)
        tally[verdict] = tally.get(verdict, 0) + 1
        print(f"{verdict:<12} {kind:<10} {url}")
        print(f"{'':<12} {'':<10} → {reason}")
    usable = tally.get("OK", 0)
    n = len(FIXTURES)
    print("-" * 64)
    print(f"{usable} of {n} blobs were usable content  ::  {tally}")

Written by Aleksey Spinov. I run scrapers in production (2,190 runs across 32 published actors, the Trustpilot one at 962) and write up the failures the tutorials skip. The gate, the fixtures, and every verdict above were produced and verified by me on Python 3.13.5; the output shown is the real run, not a mock-up.

AI disclosure: drafted with AI assistance. The code was run locally (stdlib only, no third-party deps, no network); the stdout in this post is the actual output, and the synthetic fixtures are labeled as synthetic throughout.

11 Free AI APIs You Can Use Without Paying OpenAI (2026 Update)

Alex Spinov — Fri, 12 Jun 2026 06:56:28 +0000

You don't need an OpenAI bill to build with LLMs in 2026. There are still eleven providers with a genuinely free tier — real models, real endpoints, no credit card on most — and I pulled the current limits, because half the listicles out there are quoting 2024 numbers that have since been cut.

The short version

Below are 11 LLM APIs with a free tier that still works in June 2026 — each with what's free, what it's best for, and the catch. Most are OpenAI-compatible, so you swap three things (base URL, key, model) and your existing code runs against any of them. One honest warning up front: free limits are moving fast this year (Google in particular has tightened them), so treat every number here as "check the provider's page before you depend on it."

The list at a glance

#	API	What's free (June 2026)	Best for
1	Google Gemini	Gemini 2.5 Flash, large context, no card — but limits cut in 2026, check AI Studio	Big context, broad capability
2	Groq	Llama 3.3 70B: ~30 RPM, ~1,000 req/day	Fast short calls (LPU)
3	Cerebras	~1M tokens/day, 30 RPM, 8K-ctx cap, no card	Very high throughput
4	OpenRouter	25+ `:free` models, ~20 RPM, ~50/day	Model variety, one endpoint
5	GitHub Models	OpenAI/other models for devs, tier limits	Devs already on GitHub
6	Cloudflare Workers AI	10,000 neurons/day at the edge	Edge / serverless apps
7	Mistral	Experiment tier: all models, ~1B tok/mo, no card	EU-hosted prototyping
8	SambaNova Cloud	Fast Llama inference, no card	Fast long-context calls
9	Hugging Face	Serverless Inference, many models	Open models beyond chat
10	Cohere	Free trial key (rate-limited)	RAG: embed + rerank
11	NVIDIA build	Free credits on hosted models	Trying many models fast

One code pattern for most of them

Most of these speak the OpenAI Chat Completions format — Groq, Cerebras, OpenRouter, Mistral, SambaNova, NVIDIA and GitHub Models, and Gemini/Cloudflare/Hugging Face now expose OpenAI-compatible endpoints too. So you don't learn a dozen SDKs — you point the OpenAI client at a different base URL:

from openai import OpenAI

# swap these three lines per provider; everything else stays the same
client = OpenAI(
    base_url="https://api.groq.com/openai/v1",   # provider endpoint
    api_key="YOUR_FREE_KEY",                      # from the provider's console
)
resp = client.chat.completions.create(
    model="llama-3.3-70b-versatile",              # provider's model id
    messages=[{"role": "user", "content": "Say hi in 5 words."}],
)
print(resp.choices[0].message.content)

To move from Groq to Cerebras, change base_url to https://api.cerebras.ai/v1 and the model id. That's the whole migration — and it's also how you build a fallback chain: when one free tier rate-limits you, route to the next. (Cohere is the main exception — it has its own API for embed/rerank.)

The 11, with the real details

1. Google Gemini (AI Studio)

Still a strong free option — Gemini 2.5 Flash, a large context window, and no credit card. The big 2026 caveat: Google has tightened the free limits, and the real cap is now "whatever AI Studio shows for your project" rather than a fixed public number (reports range widely, and extra keys don't add quota). Key from aistudio.google.com.
Catch: free-tier requests may be used to improve Google's models — keep proprietary data off it, and verify your live limit in AI Studio.

2. Groq

Groq runs models on custom LPU hardware and is one of the fastest free options for short calls. Published free limits for llama-3.3-70b-versatile are around 30 RPM and 1,000 requests/day with a per-minute token cap. OpenAI-compatible. Key from console.groq.com.
Catch: the per-minute token cap bites on long prompts.

3. Cerebras

Cerebras is built for speed and has one of the most generous free volumes: ~1,000,000 tokens/day, 30 RPM, no card — across models including Llama 3.3 70B, Qwen3, and GPT-OSS 120B. Throughput is very high (multiple thousand tokens/sec on smaller models). OpenAI-compatible at api.cerebras.ai/v1. Key from cloud.cerebras.ai.
Catch: a free-tier context cap (~8K tokens) across models — fine for chat, tight for long documents.

4. OpenRouter

One endpoint, 25+ models whose id ends in :free (Llama, DeepSeek, Qwen and more). Free limits are modest — roughly 20 RPM and ~50 requests/day on free models; adding ~$10 of credit once raises the free-model daily cap substantially. Endpoint openrouter.ai/api/v1.
Catch: free models get added and removed — pin the id and watch the changelog.

5. GitHub Models

If you have a GitHub account, you have free access to a rotating catalog of models (OpenAI's GPT family and others) for development. Limits depend on the model tier and your account. Auth with a GitHub token.
Catch: it's meant for dev/prototyping, not production traffic; the catalog and limits change.

6. Cloudflare Workers AI

10,000 neurons/day of free inference running at the edge — great when your app already lives on Workers/Pages. Call models like @cf/meta/llama-3.1-8b-instruct; an OpenAI-compatible endpoint is available too.
Catch: "neurons" is its own unit — a heavy model burns the daily budget faster than a small one.

7. Mistral (La Plateforme)

Mistral's Experiment tier gives free, rate-limited access to its models (including larger ones and Codestral) for prototyping — no credit card, just a verified phone number, with monthly token quotas that are generous for development. Key from console.mistral.ai.
Catch: it's an experimentation tier — production is pay-as-you-go per token.

8. SambaNova Cloud

Free, fast Llama inference with no credit card — strong on longer-context calls. OpenAI-compatible. Key from cloud.sambanova.ai.
Catch: which models are available shifts; check the catalog.

9. Hugging Face

The free serverless Inference option lets you call many open models — not just chat, but embeddings, vision, audio, classification. Token from huggingface.co.
Catch: cold starts and per-model limits; not built for steady high QPS.

10. Cohere

A free, rate-limited trial key for command, embed, and rerank. It's here for one specific reason: Cohere's embed + rerank make a genuinely good free RAG backbone, not just another chat endpoint. Key from dashboard.cohere.com.
Catch: trial-tier limits are modest — fine for building, not for serving users.

11. NVIDIA (build.nvidia.com)

Free credits to call a large catalog of hosted models through an OpenAI-compatible endpoint (integrate.api.nvidia.com/v1) — a quick way to try many models without a dozen separate signups.
Catch: it's credits, not a permanent quota — they run out.

How to actually use these (without lying to yourself about limits)

The free-API market splits into three buckets, and mixing them up is how people get a nasty surprise:

Free-quota-style tiers — Groq, Cerebras, Cloudflare, OpenRouter :free. The more durable options — but still verify before you build on them.
Small monthly credits — NVIDIA and similar. Good for trials, not a backend.
Signup trials — short-lived. Don't architect around them.

The practical move: treat free tiers as routing lanes, not one backend. Send fast short calls to Groq/Cerebras, big-context jobs to Gemini, RAG embeddings to Cohere, and let a fallback chain (that one OpenAI-compatible client above) hop to the next lane when one rate-limits you.

Two honest caveats for all of them: some free tiers may use your inputs to improve their models — check each provider's policy and keep proprietary data off them; and these numbers move — what's generous today can be cut next quarter (Google already did in 2026), so verify before you commit.

Which of these are you actually running in production vs just testing? And did I miss a free tier that's been carrying your side projects? Drop it 👇 — I'll re-test and fold it into the next update.

Give Your AI Agent a Web-Fetch Tool: a 60-Line MCP Server (Free, Self-Hosted)

Alex Spinov — Thu, 11 Jun 2026 18:13:53 +0000

Every MCP web-access tutorial I read this month pointed at a paid API.

You don't need one. To let an AI agent read a public web page, sixty lines on the official MCP Python SDK give you a self-hosted web_fetch tool — running on your machine, no key, no per-call bill.

I built it, ran it, and pasted the real terminal output below. The catch isn't the wiring (that part is easy). It's the four defaults the tutorials leave out — the ones that turn a toy into something you'd actually point an agent at.

Quick answer: A Model Context Protocol (MCP) server exposes tools an LLM agent can call. With pip install mcp, one @mcp.tool() function, and mcp.run(), you get a working web_fetch(url) -> clean text tool over stdio in ~60 lines. Self-hosted, free, and returning text instead of raw HTML. The work is in the guardrails: timeout, size cap, and an SSRF check.

This is for anyone building agents or RAG who keeps hitting "give your model live web access — here's our API." If your target is docs, articles, RSS, or JSON endpoints that answer a plain GET, you don't have to pay for that.

The artifact first: what the agent actually receives

Here's the real round-trip (terminal output, reformatted for readability — the raw print() repr is denser). I started the server, connected an MCP client over stdio, asked it to list tools, then called web_fetch on example.com:

=== TOOLS THE AGENT SEES ===
- web_fetch: Fetch a public web page and return clean readable text (no raw HTML).

=== call_tool web_fetch('https://example.com') ===
isError: False
Example Domain Example Domain This domain is for use in documentation examples
without needing permission. Avoid use in operations. Learn more

That's it. The agent asked for a URL and got back readable prose — not a wall of <div> soup. No tags, no <style> block, no nav chrome.

Stack I ran this on: mcp 1.27.2 (pip show mcp, installed 2026-06-11), httpx 0.28.1, Python 3.13.5. The MCP SDK API moves between versions, so I'll flag the parts that matter as we go.

Why free / self-hosted, and why I'm writing this

Here's the context. Pierluigi Vinciguerra runs The Web Scraping Club — one of the most-read voices in this niche. On 2026-06-07 he published a walkthrough titled, roughly, how to give Claude real-time web access with the Decodo MCP. Good post. But Decodo is a paid service. The same week, an HN front-pager pitched a "Bot Browser" MCP server that "saves 90% of tokens." The demand is obvious. The default answer everyone reaches for is a vendor.

For a big chunk of cases, that's overkill.

"Self-hosted" means the server runs as a local process you started. The traffic goes out from your IP, the rate limits are your own, and no third party logs which URLs your agent reads. For internal docs, public APIs, blog posts, changelogs, RSS — a plain GET is all you need, and a vendor in the middle is a cost and a dependency you didn't have to take on.

I'll be straight about the boundary, because this is where honesty matters: this server does not beat anti-bot systems. No headless browser, no fingerprint rotation, no JavaScript execution. Hit a Cloudflare-challenged or JS-rendered page and it returns nothing useful. That's a different tool for a different day. What this does cover is the long, boring, very common tail of sites that just answer.

The server, line by line

pip install mcp httpx and you're ready. The whole thing is one file.

The shape: create a FastMCP instance, decorate a function with @mcp.tool(), and the SDK turns the function's signature and docstring into a tool schema the agent can discover. Run it over stdio with mcp.run().

# server.py — runnable local:  pip install mcp httpx  →  python server.py
import ipaddress, re, socket
from urllib.parse import urlparse
import httpx
from mcp.server.fastmcp import FastMCP

mcp = FastMCP("web-fetch")
MAX_CHARS = 8000  # guardrail: don't blow up the agent's context window

@mcp.tool()
def web_fetch(url: str) -> str:
    """Fetch a public web page and return clean readable text (no raw HTML).
    Use this when you need the current contents of a URL."""
    _is_public_http_url(url)
    headers = {"User-Agent":
               "agent-web-fetch/0.1 (+https://blog.spinov.online; contact you@example.com)"}
    with httpx.Client(headers=headers, timeout=15.0, follow_redirects=True) as client:
        resp = client.get(url)
        resp.raise_for_status()              # 4xx/5xx -> raise, agent sees a real error
    text = _html_to_text(resp.text)
    if len(text) > MAX_CHARS:
        text = text[:MAX_CHARS] + f"\n\n[truncated at {MAX_CHARS} chars]"
    return text

if __name__ == "__main__":
    mcp.run()  # stdio transport by default

The two helpers (_is_public_http_url, _html_to_text) and imports round it out to exactly 60 lines. The full file is at the end. The docstring on web_fetch is not a comment — it becomes the tool description the model reads when deciding whether to call it, which is why the list_tools output above echoes that first line back. Write it for the agent, not for you.

A version note, because this bites people: I ran this on mcp 1.27.2. On that version from mcp.server.fastmcp import FastMCP, the @mcp.tool() decorator, and mcp.run() all exist and behave as shown. The low-level Server API and some helper signatures have shifted across releases — if you're on something older or newer, check pip show mcp and lean on the declarative FastMCP path. It's the most stable surface between versions, and it's what keeps this under sixty lines.

Talking to it from a client

To prove the agent can actually reach the tool, I connected over stdio with the SDK's own client:

# client.py — runnable local
import asyncio, sys
from mcp.client.session import ClientSession
from mcp.client.stdio import StdioServerParameters, stdio_client

async def main():
    params = StdioServerParameters(command=sys.executable, args=["server.py"])
    async with stdio_client(params) as (read, write):
        async with ClientSession(read, write) as session:
            await session.initialize()
            print(await session.list_tools())                       # tool is discoverable
            print(await session.call_tool("web_fetch",
                                          {"url": "https://example.com"}))
asyncio.run(main())

One snag worth saving you: I first wrote command="python", and on a clean venv that raised FileNotFoundError: 'python' — the binary wasn't on PATH, only python3 was. sys.executable points at the interpreter already running, so it just works. Small thing, ten minutes lost. Prefer it over a bare "python".

Don't want to write a client? npx @modelcontextprotocol/inspector python server.py gives you a UI to poke the tool, or you register server.py as a local MCP server in Claude Desktop / Claude Code and call it from there.

The four defaults that aren't decoration

This is the part I actually care about, and the reason I bothered writing instead of just linking the quickstart.

The naive version of this tool is three lines: httpx.get(url).text, return it, done. It demos fine. Then you point a real agent at the open web and it falls over in ways the quickstart never warned you about. These four defaults come straight off our scraping fleet — across roughly 2,190 production runs on 32 published actors (our Trustpilot scraper alone has logged 962 runs), and every one of these earned its place by causing pain when it was missing.

1. A polite, contactable User-Agent. The default httpx/python-requests UA is a fast way to get silently throttled or blocked. A UA that says who you are and how to reach you is the cheapest goodwill there is — and on a few of our actors it was the single line that flipped a site from 403 to 200. Feature → so what: your agent stops getting ghosted by servers that block unknown bots.

2. timeout + raise_for_status(). An agent that hangs on a slow server is worse than one that errors — it freezes the whole tool call with no signal. A 15-second timeout plus raising on 4xx/5xx means a bad URL surfaces as a real error the agent can react to, instead of an empty string it confidently treats as "the page said nothing." Silent garbage is the expensive failure mode; I've watched a scraper return empty arrays for days because nobody raised on a 429.

3. A size cap. MAX_CHARS = 8000 is a guardrail against your agent's context window, not the network. Some pages are enormous. Without a cap, one fetch of a bloated page can eat half your context and your budget with it. Truncating with a visible marker is honest and bounded. Tune the number to your model; the principle doesn't change.

4. Clean text instead of raw HTML. The tool strips tags and returns prose. On example.com — a trivial page — that took the payload from 559 raw HTML characters down to 142 of clean text, about 3.9× smaller (one page; real sites skew far higher because of nav, scripts, and inline styling). Why it matters for your token bill is its own rabbit hole, and I already measured it on real pages in Raw HTML Is a Token Tax — I Measured It. Short version: agents pay for every HTML character they're handed and read almost none of it. Hand them text.

The guardrail people skip: SSRF

One more, and it's the one I'd flag in code review. A web_fetch tool an LLM controls is a request your model can aim anywhere — including http://169.254.169.254/, the cloud metadata endpoint that leaks credentials, or http://localhost:6379 to poke your Redis. That's Server-Side Request Forgery, and an agent can be talked into it by a malicious page telling it to "fetch this URL."

So before any request, the server resolves the host and refuses private, loopback, link-local, and reserved addresses. Real output, same run:

=== call_tool web_fetch('http://169.254.169.254/') ===
isError: True
Error executing tool web_fetch: refusing to fetch a private/internal address

That's the metadata IP getting turned away. The check is ~10 lines and it's not optional if the URL can come from a model. It is not a complete SSRF defense — DNS rebinding and redirect-to-internal are still live concerns, and follow_redirects=True means you'd want to re-check the final hop in anything serious. But refusing the obvious internal targets is the floor, and most toy fetch tools don't even have that.

And the de-tagger is honest about being a toy. The regex _html_to_text is fine for a demo; it is not a real content extractor. For production, swap it for trafilatura or readability-lxml, which actually find the article body and drop boilerplate. I left the regex in so the file stays one dependency and sixty lines — but I'm telling you it's the first thing to replace.

What you've got, and the honest edges

Sixty lines, pip install mcp httpx, and your agent has a web_fetch tool that returns readable text, identifies itself politely, won't hang, won't flood your context, and refuses a direct metadata-IP URL. Free. On your machine. No vendor.

Where it stops, plainly: no JavaScript, no anti-bot evasion, no proxies, a toy extractor by default, and an SSRF guard that's a floor, not a fortress. For the docs/API/article tail, that's plenty. For Cloudflare-walled or JS-heavy targets, you're in headless-browser territory — a separate post.

The source for server.py (exactly 60 lines, the file I ran) is below, verified against mcp 1.27.2.

# server.py — a minimal MCP server that gives an AI agent ONE tool: web_fetch.
# runnable local:  pip install mcp httpx  →  python server.py
# The defaults here (UA, timeout, redirects, raise_for_status, size cap) are the
# same ones I run across our scraping fleet — not decoration, they stop real pain.
import ipaddress
import re
import socket
from urllib.parse import urlparse

import httpx
from mcp.server.fastmcp import FastMCP

mcp = FastMCP("web-fetch")

MAX_CHARS = 8000  # guardrail: don't blow up the agent's context window


def _is_public_http_url(url: str) -> None:
    """Reject non-http(s) and private/loopback targets (a small SSRF guard)."""
    p = urlparse(url)
    if p.scheme not in ("http", "https"):
        raise ValueError("only http/https URLs are allowed")
    host = p.hostname or ""
    try:
        ip = ipaddress.ip_address(socket.gethostbyname(host))
    except (socket.gaierror, ValueError):
        raise ValueError(f"cannot resolve host: {host}")
    if ip.is_private or ip.is_loopback or ip.is_link_local or ip.is_reserved:
        raise ValueError("refusing to fetch a private/internal address")


def _html_to_text(html: str) -> str:
    """Toy de-tag: good enough for a demo, NOT a real extractor.
    For production use trafilatura or readability-lxml instead."""
    html = re.sub(r"<(script|style|noscript)[^>]*>.*?</\1>", " ", html, flags=re.S | re.I)
    text = re.sub(r"<[^>]+>", " ", html)
    return re.sub(r"\s+", " ", text).strip()


@mcp.tool()
def web_fetch(url: str) -> str:
    """Fetch a public web page and return clean readable text (no raw HTML).
    Use this when you need the current contents of a URL."""
    _is_public_http_url(url)
    headers = {
        # Identify yourself. A polite, contactable UA is the cheapest way to
        # not get silently blocked — and it's the right thing to do.
        "User-Agent": "agent-web-fetch/0.1 (+https://blog.spinov.online; contact you@example.com)"
    }
    with httpx.Client(headers=headers, timeout=15.0, follow_redirects=True) as client:
        resp = client.get(url)
        resp.raise_for_status()  # 4xx/5xx -> raise, so the agent sees a real error
    text = _html_to_text(resp.text)
    if len(text) > MAX_CHARS:
        text = text[:MAX_CHARS] + f"\n\n[truncated at {MAX_CHARS} chars]"
    return text


if __name__ == "__main__":
    mcp.run()  # stdio transport by default

I keep coming back to one design question and don't have a clean answer: when you hand an agent a tool, how much should the tool enforce versus how much you trust the model to behave? I put the SSRF check and size cap in the tool because I don't trust prompt-level rules to hold under adversarial input. But that bloats every tool with guardrail code. Where do you draw that line — in the tool, in a sandbox around it, or in the agent's policy?

What's the first tool you'd hand your agent — and what guardrail would you refuse to ship without? 👇

Follow for the next teardown from our production runs. I read every comment.

Written with AI assistance; all code was run and every output above is real terminal output, not generated. Tested on mcp 1.27.2 / Python 3.13.5 on 2026-06-11. Source: the official MCP docs at modelcontextprotocol.io.

Your Scraper Re-Downloads Everything. Most Didn't Change.

Alex Spinov — Tue, 09 Jun 2026 18:11:57 +0000

Your scheduled scraper re-downloaded the whole corpus last night. A few thousand records. About forty of them actually changed since the run before.

It downloaded all of them anyway, because it had no idea which forty.

That's the failure I want to talk about. Not a block, not a crash, not bad data. A scraper that works perfectly and still does an enormous amount of work it didn't need to — because it decides what to fetch after fetching, instead of before.

TL;DR

A full re-scrape is the dumb default. On a scheduled re-run you pay to re-download records that didn't change since last time, and the scraper can't skip them because it learns "did this change" only after the request.
The cheap re-scrape isn't "download faster." It's "don't download what didn't change" — and you decide that from a manifest of what you knew last run, before the first request goes out.
Three levers, in priority order: a trustworthy validator (ETag / Last-Modified) → CONDITIONAL (a 304 transfers zero body); no validator but a stored content hash → FETCH then compare; new URL → FETCH. RFC 7232 is the whole spec.
The trap I hit in production: a weak, per-request-rotating ETag never returns 304. It fakes a 200 every time, so you "saved" nothing and trusted a validator that lies. The planner has to flag it and fall back to content-hash.
The savings number below is a deterministic synthetic manifest you run yourself — not a measurement of any real site. What's real is the exposure: 2,190 production runs across 32 actors, one Trustpilot scraper at 962.

Why I get to talk about re-scrape cost

I run production scrapers. Thirty-two published actors, 2,190 runs logged in production as of this week — that's the live counter on my own Apify dashboard, not a rounded brag. One of them, a Trustpilot review scraper, has run 962 times.

That 962 is the relevant number here. A review scraper isn't a one-shot job. It runs on a schedule, against the same companies, over and over — which means it re-visits the same pages it already saw last week, and the week before. Most of those pages have one new review, or none. Re-pulling the unchanged bulk on every scheduled run is, in my experience, the quiet majority of the work a long-lived scraper does. Not the failures. The redundant success.

Now the honest part. I do not have a clean, published figure for the compute-units, proxy-GB, or wall-clock time of a full re-crawl versus a delta crawl on our real corpus. That number is n/d, and I'm not going to invent one to make a point. What I can give you is the mechanism, on a manifest you can run in two seconds and get the exact count I did. The 2,190 / 962 is the real part — it's the reason I think about this at all.

This is not the other failures in the series

These failures rhyme, and the fixes don't. Draw the boundary hard.

Not this	That post is about	This post is about
Raw HTML token tax	the cost of one fetch (raw HTML → markdown tokens, a polite conditional GET)	how many fetches to make at all on a re-run — work at the level of the whole record set
Corpus near-duplicates	removing duplicates inside data you already collected	not re-collecting what didn't change, before collection
Resume a dead run	finishing an interrupted run without re-doing work	deliberately not re-pulling unchanged records on a successful, scheduled re-run
Yield decay over time	detecting that output is silently rotting vs a baseline	deciding what to re-collect to keep the corpus fresh cheaply
Poisoned data	values that are valid in form but false in fact	the volume of re-collection — nothing about trusting the content

The thin line is with the token-tax post. A conditional GET shows up there too, as politeness on a single request. Here a conditional GET is just one of three levers in a plan over the whole set, and it's not even the center of gravity — the center is SKIP by manifest. The moment a re-scrape post starts reading like "how to send one conditional GET," it has drifted into the token-tax post. The question that keeps it here: for N records on a scheduled re-run, how many do I touch?

New axis: work. Not the cost of one fetch, not deduping, not resume, not detection. The size of the job on a planned repeat.

The decision belongs before the request, not after

Here's the whole reframe. A naive scraper's loop is: fetch the page, then notice it's identical to last time. The noticing is too late — the bytes already crossed the wire, the proxy already burned, the parser already ran.

A planner inverts it. Before any request, it reads last run's manifest — one small row per record with what you knew: {url, etag, last_modified, content_hash} — and assigns a plan. RFC 7232 gives you two of the three levers for free, and they've been in HTTP for years; almost nobody uses them on the re-scrape path.

The priority order:

Trustworthy validator → CONDITIONAL. If last run gave you an ETag or a Last-Modified, send If-None-Match / If-Modified-Since. If the page is unchanged the server answers 304 Not Modified — status line, no body. You confirmed "nothing changed" for the cost of a header round-trip, not a full download.
No validator, but a stored content hash → FETCH, then compare. Some servers give you nothing to precondition on. You still hold last run's body hash. Fetch, hash the new body, and if it matches, stop — don't re-parse, don't re-write downstream, don't re-embed. You paid for the bytes but skipped everything after.
New URL → FETCH. Not in the manifest, so there's nothing to compare. Pull it.

The planner's only job is to assign one of FETCH / SKIP / CONDITIONAL to every record, from the manifest alone, before the loop starts. That plan is the artifact. Everything downstream just executes it.

The trap: a validator that lies

I trusted ETags completely until one source taught me not to.

A conditional GET assumes the validator is stable: same content, same ETag, so an unchanged page returns 304. But RFC 7232 §2.1 explicitly allows weak validators — metadata "that might not change for every change to the representation data … or a desire of the resource owner to group representations by some self-determined set of equivalency" (RFC 7232 §2.1). A weak ETag is written W/"...".

What bit me was worse than weak — it rotated. The server emitted a different ETag on every response for the same unchanged page. So my If-None-Match never matched, the server never returned 304, and every conditional request came back as a fresh 200 with a full body. I'd "optimized" the re-scrape and saved exactly nothing on that source, while believing I had. The data was fine. The plan was a lie.

The fix is to treat the validator as untrustworthy and downgrade: when an ETag is weak or known to rotate, don't plan CONDITIONAL, plan FETCH and compare the content hash after. You lose the 304 savings on that one source, but you stop trusting a number that can't be trusted. The planner below carries that downgrade as an explicit branch — it's the production detail that turns a tutorial into something that survives contact with a real site.

The planner, in pieces

Pure stdlib, no network, no browser, no keys, no random. A deterministic synthetic manifest stands in for last run's stored state, so you get the exact output I did. The transport is irrelevant to the mechanism — the planning is just a decision over a table.

First, the decision for a single record. This is the entire idea:

def plan_for(rec):
    """Return (plan, note) for one manifest record, BEFORE any request."""
    if rec["etag"] is not None and rec["etag_weak"]:
        # A weak/rotating ETag never produces a 304. Don't trust it.
        return "FETCH", "untrustworthy_validator -> hash-compare"
    if rec["etag"] is not None or rec["last_modified"] is not None:
        return "CONDITIONAL", "send If-None-Match / If-Modified-Since"
    if rec["content_hash"] is not None:
        return "FETCH", "no validator -> compare content_hash after fetch"
    return "FETCH", "no prior knowledge"

Read the order. The weak-ETag downgrade is first, on purpose — a record can have an ETag and still be untrustworthy, and if you check "has an ETag" before "is the ETag weak," you plan CONDITIONAL on a validator that lies. Order is the bug surface here.

Then the simulation that proves the savings. It does not hit a network — it models each server's outcome from a fixed rule so the count is reproducible. Unchanged + CONDITIONAL → a 304 (no body). Anything FETCH → a body. Only two records actually changed since last run:

CHANGED_THIS_RUN = {  # the few records whose body really changed
    "https://shop.example.com/p/1002",
    "https://shop.example.com/p/1006",
}

def simulate(plan):
    bodies = 0          # full bodies transferred
    not_modified = 0    # 304s — zero body
    for item in plan:
        changed = item["url"] in CHANGED_THIS_RUN
        if item["plan"] == "CONDITIONAL":
            if changed:
                bodies += 1                 # 200 + new body
            else:
                not_modified += 1           # 304, zero body
        else:                               # FETCH (incl. weak-ETag fallback, new urls)
            bodies += 1
    return bodies, not_modified

The baseline it compares against is the dumb default: a full re-scrape downloads every record's body, every run. So fetches_saved = total − bodies_transferred.

The live run

Twelve records in scope: ten carried over from last run's manifest, two new this run. Seven have a trustworthy validator and get planned CONDITIONAL. Five get FETCH — two with no validator at all, one new-URL pair, and the one weak-ETag trap that got downgraded out of CONDITIONAL. Run the script and you get:

=== RE-SCRAPE PLANNER (deterministic synthetic manifest, not a real site) ===
records in scope          : 12  (10 from manifest + 2 new)
plan decided BEFORE any request:
  CONDITIONAL             : 7  (If-None-Match / If-Modified-Since)
  FETCH                   : 5
    of which weak-ETag fallback: 1  (untrustworthy validator -> hash-compare)
--------------------------------------------------------
simulated run outcomes:
  304 not-modified (no body): 5
  bodies transferred        : 7
--------------------------------------------------------
naive full re-scrape bodies : 12
planner bodies transferred  : 7
fetches_saved               : 5  (5/12 bodies not re-downloaded)
--------------------------------------------------------
per-record plan:
  1001  CONDITIONAL 304 (unchanged, no body)     send If-None-Match / If-Modified-Since
  1002  CONDITIONAL 200 (changed, body transferred) send If-None-Match / If-Modified-Since
  1003  CONDITIONAL 304 (unchanged, no body)     send If-None-Match / If-Modified-Since
  1004  CONDITIONAL 304 (unchanged, no body)     send If-None-Match / If-Modified-Since
  1005  CONDITIONAL 304 (unchanged, no body)     send If-None-Match / If-Modified-Since
  1006  CONDITIONAL 200 (changed, body transferred) send If-None-Match / If-Modified-Since
  1007  CONDITIONAL 304 (unchanged, no body)     send If-None-Match / If-Modified-Since
  1008  FETCH       200 (fetched body)           no validator -> compare content_hash after fetch
  1009  FETCH       200 (fetched body)           no validator -> compare content_hash after fetch
  1010  FETCH       200 (fetched body)           untrustworthy_validator -> hash-compare *TRAP*
  1011  FETCH       200 (fetched body)           new url (not in manifest)
  1012  FETCH       200 (fetched body)           new url (not in manifest)
========================================================
verdict: planned 7 conditional + 5 fetch; transferred 7 bodies vs 12 naive (5 saved).

Read what that output is saying:

CONDITIONAL: 7 and 304 not-modified: 5. Seven records were checked with a header round-trip; five of them came back 304 — confirmed unchanged, zero body transferred. Those five are the win. The naive scraper would have downloaded all five in full.
200 (changed, body transferred) on 1002 and 1006. The two records that actually changed still get their new body — a conditional GET costs nothing when the page did change; you just get a normal 200. The plan never hides a real update.
1010 *TRAP*. The weak-ETag record did not stay CONDITIONAL. The planner downgraded it to FETCH with untrustworthy_validator -> hash-compare, so it transfers the body and compares the hash — instead of trusting a rotating ETag that would have faked a 200 forever.
fetches_saved: 5. Five of twelve bodies not re-downloaded, decided entirely before the first request. On a real corpus the unchanged share is usually far higher than 5/12 — but that's the n/d number I won't fabricate. The 5/12 here is what you can reproduce.

Where this breaks (and I'm not overselling it)

A re-scrape planner is a work-reducer, not magic. The limits are the point.

Content-hash savings happen after the bytes, not before. The CONDITIONAL lever skips the download. The hash-compare lever only skips parsing and downstream work — you still pay for the body. On a source with no validators, you cut CPU and write amplification, not bandwidth. That's a real win, but a smaller one, and worth being honest about.

A moving timestamp in the body breaks naive hashing. If the page embeds a "last viewed" or a server time, the body hash changes on every fetch even when the data didn't. You end up hashing noise. The workaround is to hash a normalized projection — strip the volatile fields first — which is fine until the timestamp is the data you came for.

Weak ETags are one liar; there are others. A server can return a stable strong ETag and still serve changed content (broken cache), or change content without touching Last-Modified. The downgrade catches the rotating case I hit. It does not make validators trustworthy in general — it makes you stop assuming they are.

So treat the planner as what it is: a cheap way to turn "re-download everything" into "touch what plausibly changed, confirm the rest with a header." It won't make a re-scrape free. It makes the dumb default stop being the default.

What to do Monday

Three moves, smallest first:

Persist a manifest. One row per record: url, the ETag and Last-Modified the server gave you, and a hash of the body you stored. It's a tiny JSON or SQLite table. If you don't keep it, every run starts blind and a full re-scrape is your only option — the manifest is what makes a plan possible at all.
Send conditional requests on the re-run path. If-None-Match from the stored ETag, If-Modified-Since from the stored Last-Modified. Honor a 304 as "unchanged, skip." This is in requests and httpx today; it's a few lines, and it's the one change that pays back the most.
Distrust weak and rotating validators. If an ETag starts with W/, or you see it change across two fetches of an unchanged page, downgrade that source to fetch-and-hash. Log the downgrade so you know which sources you can't precondition — a plan that knows what it can't trust beats one that trusts a liar.

You don't need a crawl framework rewrite. You need to stop deciding what to fetch after you've already fetched it.

One thing I haven't solved cleanly: how do you decide a record changed when the source gives you no validator and the body carries a moving timestamp inside it? I hash with the timestamp stripped — but that breaks the moment the timestamp is the data I came to collect, and I don't have a general rule for telling those two cases apart automatically. If you've got a heuristic that holds up in production, I want to hear it. I read every comment.

Follow for the next numbers from the run log. And tell me: what's the worst re-scrape waste you've found in your own pipeline — the job that was re-downloading the most for the least?

Written by Aleksei Spinov — I run production scrapers (2,190 runs across 32 actors; one Trustpilot scraper at 962). Proof: blog.spinov.online and my Apify profile.

AI disclosure: drafted with AI assistance, then edited, fact-checked, and the code run and verified by me. The manifest is synthetic and deterministic; the output above is real stdout from executing the script.

Your AI Agent Is Paying for HTML It Never Reads — I Measured the 7x Token Tax

Alex Spinov — Tue, 09 Jun 2026 07:58:48 +0000

I gave an agent a fetch_page tool, asked it to read one Wikipedia article, and watched that single page cost 48,703 tokens before the model produced a word. The readable text on that page is about 7,300 tokens. I was paying for ~41,000 tokens of <div>, inline CSS, and analytics scripts that never help the model answer anything.

That's the token tax on agent web access, and almost nobody measures it. Here's the number, the 40-line fix, and the honest part — where it doesn't matter.

The short version

When your agent "reads a page", it usually gets raw HTML pasted into the prompt. On three pages I tested, 85–86% of those tokens were markup the model doesn't need to read for meaning. Strip the page to text first and the token bill drops ~7×. The fix is the standard library plus a tokenizer — no API, no paid service.

The measurement

Counted with o200k_base (the tokenizer GPT-4o uses), three live pages of different sizes, raw HTML vs text-only. Measured 2026-06-09 — these are live pages, so your exact numbers will differ:

Page	Raw HTML	Text for the agent	Reduction
Wikipedia: Web scraping (165 KB)	48,703 tok	7,280 tok	6.7× (85% less)
Wikipedia: Large language model (686 KB)	221,622 tok	30,988 tok	7.2× (86% less)
example.com (528 B, control)	152 tok	22 tok	6.9× (86% less)

Three pages is a small sample, not a benchmark. But the ratio barely moved between a 528-byte page and a 686 KB one, which is the interesting part: the markup overhead is roughly proportional, so on the pages I tested the tax shows up everywhere, not just on the big ones.

At GPT-4o input pricing ($2.50 / 1M tokens, OpenAI, checked June 2026), the LLM page alone is $0.55 raw vs $0.078 clean per read. One read. An agent that crawls 200 pages in a loop turns that into real money — and worse, it fills the context window with noise that pushes out the tokens you actually want the model reasoning over.

The fix (stdlib + tiktoken)

import sys, ssl, urllib.request
from html.parser import HTMLParser
import tiktoken

UA = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) Chrome/148.0.0.0 Safari/537.36"
SKIP = {"script", "style", "head", "noscript", "svg", "template"}

class TextOnly(HTMLParser):
    def __init__(self):
        super().__init__(); self.parts = []; self.skipping = 0
    def handle_starttag(self, tag, attrs):
        if tag in SKIP: self.skipping += 1
    def handle_endtag(self, tag):
        if tag in SKIP and self.skipping: self.skipping -= 1
    def handle_data(self, data):
        if not self.skipping and data.strip(): self.parts.append(data.strip())

raw = urllib.request.urlopen(
    urllib.request.Request(sys.argv[1], headers={"User-Agent": UA}), timeout=30
).read().decode("utf-8", "replace")

p = TextOnly(); p.feed(raw); text = "\n".join(p.parts)
enc = tiktoken.encoding_for_model("gpt-4o")          # o200k_base
raw_tok, clean_tok = len(enc.encode(raw)), len(enc.encode(text))
assert clean_tok <= raw_tok and len(text) > 0        # prove the work was done
print(f"{raw_tok:,} -> {clean_tok:,} tokens  ({raw_tok/clean_tok:.1f}x less)")

This is a runnable local excerpt. pip install -U tiktoken (you need a recent version for o200k_base), then python clean.py https://en.wikipedia.org/wiki/Web_scraping. Output on my machine:

48,703 -> 7,280 tokens  (6.7x less)

The full script (with the TLS handling and sanity check below) is in the repo. It's the standard library doing the work HTMLParser was built for, plus tiktoken so you count in the model's units, not characters. No requests, no readability library, no service.

One honest detail from the sanity print: the extracted text starts with Jump to content Main menu Main menu ... Navigation. This is a sanitizer, not a main-content reader — it keeps nav and footer text (more on that below).

One gotcha that cost me ten minutes

I run behind a VPN, and the first fetch died with CERTIFICATE_VERIFY_FAILED. The VPN was intercepting TLS, so the system couldn't chain to a trusted root. urllib hides this: it wraps the ssl.SSLError inside a urllib.error.URLError, so a naive except ssl.SSLError never fires. You catch URLError and look at e.reason:

except urllib.error.URLError as e:
    if not isinstance(e.reason, ssl.SSLError):
        raise
    sys.exit("TLS failed. If you trust this proxy, re-run with --insecure.")

The script fails closed — it won't silently disable verification. Measuring a page over an untrusted MITM proxy is meaningless (you'd be tokenizing whatever the proxy injected), so turning off TLS is an explicit --insecure flag, not a quiet fallback.

When this is NOT worth it

I'd be lying if I said "always strip HTML." It isn't free:

You lose structure. Tables, link targets, alt/title, and <code> boundaries flatten into text. If the agent's job is "extract every row of this table" or "follow these links," text-only throws away the signal. Hand it Markdown for those.
html.parser is not a browser. JS-rendered pages return a near-empty shell — this strips what the server sent, not what a browser paints. SPA targets still need a headless browser first.
It's a sanitizer, not a reader. It keeps menu and footer text (that Jump to content / Main menu above). A readability pass cuts further, at the cost of a dependency and occasionally eating real content. For an agent, "too much text" is cheap; "silently dropped the answer" is expensive — so I over-keep.
The 7× is against raw HTML, the naive default — not against Markdown or a readability pass, which also cut tokens. If you already feed Markdown, your win is smaller.
Numbers depend on the live page, your User-Agent, and locale. Re-measure on your own targets.

So the honest rule: strip to text when the agent is reading for meaning (RAG ingestion, summarization, Q&A). Keep structure when it's extracting specific fields.

Why I bothered

I run production scrapers, and the lesson that transfers to agents is the same one that bit me on data pipelines: the cost isn't in the request, it's in what you carry forward. An agent that pastes raw HTML into every step pays the tax on every step, and the context bloat quietly degrades the reasoning you're paying for twice.

40 lines. One pip install. ~7× fewer tokens on the pages I tested.

How are you feeding pages to your agents right now — raw HTML, Markdown, or a readability pass? And has anyone measured the token difference on their own targets? Drop your numbers 👇

Your Scraper Got Clean Data. The Site Lied to It.

Alex Spinov — Mon, 08 Jun 2026 18:14:03 +0000

Your scraper ran clean. HTTP 200 on every request. The schema validated. Every price sat in a sane range, every date was in the past, every ISBN had thirteen digits. Zero errors, zero retries. You shipped the dataset.

And every value in it was a lie the site fed you on purpose, because it knew you were a bot.

That's the failure I want to talk about. Not a block. Not a captcha. Not a crash. The site looked at your traffic, decided not to fight you, and instead handed back a 200 full of plausible garbage — values engineered to pass every check you have. This is the one corner of data quality where valid and true come apart, and almost every detector people write only measures the first one.

TL;DR

A site that detects a scraper can serve a 200 with a flawless schema and values that pass every sanity rule — and are deliberately fabricated. This is documented, shipping anti-bot behavior, not a hypothetical.
Status codes and sanity checks can't catch it. They answer "is my pipeline correct?" The poisoned row's question is "is the source telling the truth?" No range check answers that.
The fix is grounding: check each row against an independent invariant the source can't fake by making the value look plausible — an ISBN-13 checksum, price * qty == line_total, a real second origin.
The trap I didn't expect: naive cross-source consensus gets fooled too. "Three sources agree" means nothing if all three are mirrors of one poisoned first-party page. Independence is the signal, not the vote count.
The numbers in the demo below are a deterministic synthetic dataset, not a measurement of any real site. What's real is the volume that earns me the right to talk about plausible-but-false data: 2,190 production runs across 32 actors, one Trustpilot scraper at 962.

Why I get to talk about plausible lies

I run production scrapers. Thirty-two published actors, 2,190 runs logged in production as of this week (my own Apify dashboard — that's the live counter, not a rounded brag). One of them, a Trustpilot review scraper, has run 962 times against the same kind of source.

That last one matters more than the total, and here's why. Reviews are the textbook class of data where plausible and true are different problems. A fake review and a real review look identical at the schema level: both have a star rating in 1–5, a date, a body in fluent English, an author handle. Sanity passes on both. The entire job of scraping that surface well is knowing that a clean, well-formed, perfectly typed record can still be fabricated. After 962 runs hitting that wall, "the value looks right and is still false" isn't a thought experiment to me. It's the default assumption.

Now let me be honest about what I don't have. I do not have a clean, published figure for how many poisoned rows we've actually caught in the wild, or what share of any specific site's responses are fake. That number is n/d, and I'm not going to invent it. The 2,190 / 962 is the part that's real — it's the exposure that makes the failure class familiar. The detection mechanism below, I'll show you on a dataset you can run yourself in two seconds, so nothing rests on a number you can't check.

This is not the other failures in the series

If you've read the rest of this series, draw the boundary hard, because these failures rhyme and the fixes don't.

This is not a broken status code. HTTP 200 lying about the shape of a response is the schema canary — that catches a 200 whose structure has drifted. Here the shape is flawless. The canary would report HEALTHY.

This is not a field that violates a sanity rule — a price of $0, a date in the future, a language that doesn't match the country. That post catches values that look wrong. This is the opposite: every field looks right, passes every one of those sanity checks, and the value is still a deliberate lie the source served because it made you as a bot.

It isn't a crash you resume from — there's no crash, the job exits 0. The question those posts answer is "is my pipeline correct?" This one asks something none of them touch: "is the source telling the truth?" And no status code, no schema validator, no range check will ever answer that.

New axis: trust. Not the shape of a response, not a field's plausibility, not a crash. Whether the values themselves are real.

This is shipping, not a thought experiment

The reason this isn't paranoia: the anti-bot industry already does it on purpose. The polite framing is "tarpit" or "decoy content" — when a site fingerprints a crawler, instead of returning a 403 it returns a 200 full of generated content that wastes your time and pollutes your dataset. Cloudflare shipped a feature in 2025 that feeds suspected AI crawlers a maze of AI-generated decoy pages on purpose; their own writeup frames it as serving believable-but-irrelevant data instead of blocking (Cloudflare, AI Labyrinth, 2025). Bruce Schneier has been cataloguing the broader version — deliberate data poisoning aimed at scrapers and the models trained on them (Schneier on Security, 2025).

So the threat model flipped. The old story was "they'll block you." The new story is "they'll let you in and lie to you," because a poisoned dataset that you trust is worth more to them than a blocked request you'll just retry from another IP.

Why status and sanity are silent by design

Here's the uncomfortable part. The poisoned row passes your checks not because your checks are bad, but because they were built to answer a different question.

A status code answers: did the transport succeed? 200 — yes. A schema canary answers: is the structure intact? All keys present, types correct, no nulls — yes. A sanity rule answers: is this value inside the range a real value would fall in? Price is $44.99, qty is 1, date is last month — yes, yes, yes.

Every one of those is a question about form. None of them is a question about correspondence to reality. And an adversary who controls the response can satisfy every form check trivially — that's the whole point of serving a decoy instead of a block. They're not sending you malformed junk that trips a validator. They're sending you a beautifully formed lie.

Sanity catches accidents: the source glitched, a field came back null, a scraper bug doubled a value. It does not catch adversarial fabrication, because fabrication is designed to look sane. Validity is not truth. You can't range-check your way to trust.

What actually catches it: grounding to an invariant

So if you can't trust the value, what can you trust? An invariant the source can't satisfy just by making the number look plausible — something that has to be computed against an independent reference and will break if the value was made up.

Three kinds are cheap and shockingly effective:

A self-checking identifier. An ISBN-13 isn't just thirteen digits. Its last digit is a checksum over the first twelve. A fabricated ISBN that "looks real" almost never satisfies the checksum.
A cross-field arithmetic invariant. price * qty == line_total. Each field can be individually plausible and the math can still be incoherent — which is exactly what happens when values get swapped around.
Real independent corroboration. Not "how many sources cite this," but "how many distinct origins do." Three citations that all resolve to the same first-party domain are one source wearing three hats.

Here's the whole probe. Pure stdlib, no network, no browser, no keys, no random — a deterministic synthetic dataset stands in for the collected rows so you get the exact output I did. The "collection" is a hardcoded list because the mechanism — grounding to an invariant — doesn't depend on the transport one bit.

def isbn13_checksum_ok(isbn13):
    """An ISBN-13 is valid only if sum(d_i * w_i) % 10 == 0, weights alternating
    1, 3. Sanity asks 'is it 13 digits'. This asks 'is it a REAL ISBN'."""
    digits = [int(c) for c in isbn13]
    total = sum(d * (1 if i % 2 == 0 else 3) for i, d in enumerate(digits))
    return total % 10 == 0


def arithmetic_invariant_ok(r, tol=0.01):
    """price * qty must equal line_total. Each field can be individually plausible
    and still violate this — incoherent values don't survive the cross-field math."""
    return abs(r["price"] * r["qty"] - r["line_total"]) <= tol


def registrable_domain(host):
    """Crude eTLD+1: last two labels. 'm.vendor-a.com' -> 'vendor-a.com'.
    In production use the public suffix list."""
    return ".".join(host.split(".")[-2:])


def independent_corroboration_ok(r, min_independent=2):
    """'3 sources agree' means nothing if all 3 are mirrors of one origin.
    Count DISTINCT registrable domains, not citations."""
    distinct = {registrable_domain(h) for h in r["sources_citing"]}
    return len(distinct) >= min_independent

Each row gets graded TRUSTED or POISONED(reason). Note what the checks do not need: they never need to know which field is the lie, or what the "correct" value was. They only need the invariant to hold. That's the property that makes grounding work where sanity can't — it doesn't model the truth, it models a constraint the truth obeys and a lie usually breaks.

The twist: naive consensus gets fooled too

The obvious upgrade, the one most people reach for, is consensus. "Don't trust one source. Cross-check three. If they agree, it's true." It feels bulletproof.

It isn't, and this is the part I want you to take away even if you forget the rest. I ran a row into the probe that three sources corroborate — and it's still poisoned. Because all three "sources" resolve to the same registrable domain: store.vendor-a.com, m.vendor-a.com, cdn.vendor-a.com. One poisoned first-party page, mirrored across a store front, a mobile host, and a CDN. A naive consensus check counts three agreements and votes confidently for the lie.

Consensus measures agreement. Agreement is not independence. If everyone is quoting the same poisoned origin, unanimity is exactly what you'd expect — and exactly what you should distrust. The fix is to count distinct origins, not distinct URLs: collapse each citation to its registrable domain and require at least two that don't trace back to the same place. That's the difference between "three sources said so" and "three independent sources said so," and adversaries live in the gap between those two sentences.

The live run

Twelve "collected" rows. All twelve pass the schema canary (#5) and all twelve pass field sanity (#7) — by construction, so the old detectors stay silent. Three are poisoned, each caught by exactly one grounding rule. Run the script and you get:

=== POISON CHECK (deterministic synthetic dataset, not a measurement of a real site) ===
records collected        : 12
passed schema canary (#5): 12   <- shape is perfect on ALL
passed field sanity (#7) : 12   <- every value looks RIGHT; sanity is silent
--------------------------------------------------------
grounding checks (truth, not validity):
  ISBN-13 checksum        : 1 failed
  price*qty == line_total : 1 failed
  independent corroborat. : 1 failed (3 "sources" all point to one first-party)
--------------------------------------------------------
TRUSTED                  : 9
POISONED                 : 3
  - sku DEMO-0007  reason=isbn13_checksum
  - sku DEMO-0009  reason=arithmetic_invariant
  - sku DEMO-0011  reason=false_consensus_single_origin
========================================================
verdict: 3 clean, well-formed, sanity-passing rows are fabricated.

Read what that output is actually saying:

passed schema canary 12 and passed field sanity 12. All twelve clear every check from the earlier posts in this series. The schema canary and the sanity validator are structurally blind here — not broken, just answering a different question.
ISBN-13 checksum: 1 failed. DEMO-0007 has a thirteen-digit ISBN with a valid 978 prefix. Sanity loves it. The check digit doesn't satisfy the weighted-mod-10 rule, so the number was made up.
price*qty == line_total: 1 failed. DEMO-0009 has a sane price, a positive integer qty, and a positive total — each field passes on its own. 34.99 * 3 is 104.97, not the 79.99 in the row. The values are individually plausible and jointly incoherent.
independent corroborat: 1 failed. DEMO-0011 is the consensus trap: three citations, one origin. The vote says trust it. Independence says don't.
verdict: 3 ... fabricated among nine clean rows. The poison is invisible to status, shape, and sanity, and visible only to grounding.

Where this breaks (and I'm not going to oversell it)

Grounding is not a lie detector. It's narrower than that, and the limits matter more than the wins.

You need an invariant, and not all data has one. A book has an ISBN checksum. A line item has arithmetic. A free-text Trustpilot review has neither. There is no checksum on "the food was cold and the staff were rude." For pure text with no internal constraint and no independent anchor, grounding has nothing to grab — and that's exactly the surface where poisoning is easiest. I can't hand you a 30-line probe for that. I'd be lying if I said I could.

Cross-source corroboration needs sources that are genuinely separate. My domain collapse catches the lazy mirror case. It does not catch a determined adversary who plants the same lie across genuinely distinct domains, or a real ecosystem where everyone honestly syndicates from one upstream feed. Independence is a spectrum, and registrable-domain is a crude proxy for it.

A perfectly consistent fabrication beats this. Grounding catches incoherent lies — a value that breaks a constraint. If the adversary fabricates the ISBN and recomputes a valid checksum, and makes the arithmetic close, the invariant holds and the probe says TRUSTED. This raises the cost of the lie, which is the realistic goal. It does not make lying impossible.

So treat it as what it is: a cheap filter that catches the common, lazy poisoning and forces an adversary to do real, coordinated work to get past it. That's a good trade for 30 lines. It is not trust by itself.

What to do Monday

Three moves, smallest first:

Add every invariant your data already carries. Checksummed identifiers (ISBN, IBAN, EAN, VAT numbers), cross-field arithmetic (price * qty == total, percentages summing to 100), referential constraints (a foreign key that resolves). These are free — the constraint already exists in the domain; you just have to check it. Most pipelines never do.
Count distinct origins, not citations. If you corroborate across sources, collapse each to its registrable domain before you count. "Three sources" that share a domain is one source. The public suffix list does this properly; the two-label hack in the demo is the starter version.
Know which of your columns have no anchor, and flag them. The honest output isn't only "this row is poisoned." It's also "this field has no invariant I can ground it against, so I can't vouch for it." A pipeline that knows what it can't verify is worth more than one that pretends everything is fine.

You don't need a fraud-detection platform for the first pass. You need to use the constraints your own data already ships with — most of us collect them and then never check them.

One thing I haven't solved: how do you ground free text at all? Reviews, descriptions, comments — the highest-value, most-poisoned surface — have no checksum and often no independent anchor. The only handles I've found are weak: distribution drift across a corpus, stylometric oddities, timing clusters. None of them work on a single row, and all of them are gameable. If you've actually caught a fabricated free-text record in production — not a malformed one, a fabricated one — I want to know what signal you used. I read every comment.

Follow for the next numbers from the run log. And tell me: what's the most convincing fake data a source ever fed your scraper — the one that passed every check you had?

Written by Aleksei Spinov — I run production scrapers (2,190 runs across 32 actors; one Trustpilot scraper at 962). Proof: blog.spinov.online and my Apify profile.

AI disclosure: drafted with AI assistance, then edited, fact-checked, and the code run and verified by me. The dataset is synthetic and deterministic; the output above is real stdout from executing the script.

Your Scraper Passes Every Run. It's Still Rotting.

Alex Spinov — Sun, 07 Jun 2026 17:44:25 +0000

A scraper run finished green. Exit 0. Schema valid. Row count looked normal. So did the one before it, and the forty before that.

Then one afternoon you glance at a number you don't usually look at — total rows this month vs the same source last month — and you're collecting noticeably less than you used to. No errors. No traceback. No alert fired, because nothing was ever wrong with any single run.

That's the failure I want to talk about. Not a crash. A slow rot, measured across runs, that every single-run check on earth is blind to.

TL;DR

A scraper can pass every per-run gate (exit 0, schema ok, count plausible) while its rolling yield slides down for weeks.
Single-run checks can't see it. The signal only exists when you compare today's run to your own past, not to a declared total.
The obvious detector — "median of the last K runs" — is a boiling-frog trap. On a slow drift the baseline sinks with the signal, so it never fires. I ran it. Zero warnings while yield dropped 25%.
Fix: a lagged baseline. Compare today against the median of runs from K..2K runs ago — your settled past, not the part already eaten by decay. ~20 lines over your run log. No Grafana, no SRE.
Numbers below are a deterministic synthetic run log, not a claim about our real slide. What's real is the volume that makes such a curve observable: 2,190 production runs, one Trustpilot scraper alone at 962.

Why I trust this surface exists

I've shipped 32 scrapers and they've logged 2,190 runs in production. One of them, a Trustpilot review scraper, has run 962 times against the same source. That's the thing most scraping tutorials don't have: a long line of runs hitting one place over real calendar time.

When you have 962 runs of one source, "yield per run" stops being a single number and becomes a curve. And a curve has a shape. Most of the time the shape is flat and boring. Sometimes it tilts down so gently that no individual run ever looks off — and that's exactly the case nobody writes about, because you only see it if you have the history.

To be honest about the limits up front: I don't have a clean, published figure for how far our real yield slid on any specific site, so I'm not going to invent one. That number is n/d. What I can do is show you the mechanism on a deterministic run log you can execute yourself, and tell you the detector I'd reach for. The 2,190 / 962 is the part that's real — it's the volume that makes the curve visible at all.

This is not the other failures

If you've read the rest of this series, draw the boundary clearly, because the failures rhyme and the fixes don't.

This isn't a bad status code — HTTP 200 lying about a broken response shape is the schema canary. This isn't a wrong field value inside a row that's otherwise valid. It isn't bytes you paid for that came back empty, and it isn't a crash you resume from.

Every run here is green. Exit 0, schema valid, row count plausible for a single run. There is no declared total to check against — you don't know how many rows "should" come back. The decay only exists when you line today's run up against your own past. The rolling yield has been drifting down for weeks, and nothing ever threw.

New axis: time. Not the shape of one response, not a field, not bytes, not a crash. The trend of one source against its own history.

How a healthy run can be a rotting series

Here's the mechanism, because it's quieter than it sounds.

Say your scraper walks 80 pages of a source each run. When the source is healthy, each page hands back about 48 rows — call it a full page minus a little. So a run pulls roughly 3,800 rows. Plausible. Your per-run sanity gate says "fail if rows < 3,000," and it never trips.

Now the source starts thinning. Maybe a soft rate-throttle kicks in and pages return fewer items. Maybe the result set genuinely shrinks. Maybe an A/B test on their side trims the page size. Whatever the cause, each run quietly returns a hair less than the last — say 0.9 fewer rows per page. One run that goes from 48 to 47.1 rows/page looks identical to the one before. Nobody blinks.

Roll that forward. Twenty runs later you're at 30 rows/page. The run still walks 80 pages. Still exits 0. Still has a valid schema on every row. Still clears rows < 3000 — barely. Your per-run gate has no memory, so it can't tell that 2,400 rows used to be 3,800. The frog has been boiling the whole time and the thermometer only ever read "alive."

That's the trap of single-run validation: every check answers "is this run OK by itself?" None of them answers "is this run OK compared to what this source used to give me?"

The detector: ~20 lines over your run log

You don't need a metrics stack for this. You need three things: log the yield of every run, keep the log, and run a baseline check over it.

Here's the whole probe. Pure stdlib, no network, no browser, no paid API — a deterministic synthetic run log stands in for the source so you can run it in seconds and get the exact output I did.

import statistics

# --- knobs ---
K = 7              # baseline window size (median of K runs)
GAP = K            # how far back the window sits (LAGGED, not trailing)
THRESHOLD = 0.15   # WARN if today's yield is >15% below the lagged baseline
MIN_HISTORY = K + GAP

def synth_run_log():
    """Deterministic log of one source over 60 runs. Every run is 'green':
    exit 0, schema valid, plausible single-run row count. Yield slowly decays
    after run 40 (a soft throttle / thinning source) with small jitter."""
    runs = []
    base_yield = 48.0
    jitter = [0.0, -0.6, 0.4, -0.3, 0.5, -0.4, 0.2, -0.5, 0.3, -0.2]
    for i in range(60):
        run_id = i + 1
        decay = 0.0 if run_id <= 40 else (run_id - 40) * 0.9
        y = base_yield - decay + jitter[i % len(jitter)]
        pages = 80
        rows = round(y * pages)
        runs.append({
            "run_id": run_id, "exit_code": 0, "schema_ok": True,
            "pages": pages, "rows": rows,
            "yield_per_page": round(rows / pages, 3),
        })
    return runs

def yield_decay_probe(runs, k=K, gap=GAP, threshold=THRESHOLD):
    """For each run, compare its yield to the median of a LAGGED window:
    runs [idx-k-gap : idx-gap]. The gap is what defeats the boiling-frog trap."""
    first_warn = None
    for idx, run in enumerate(runs):
        if idx < MIN_HISTORY:
            run["verdict"] = "BUILDING_BASELINE"
            run["baseline"] = run["drop"] = None
            continue
        window = [r["yield_per_page"] for r in runs[idx - k - gap:idx - gap]]
        baseline = statistics.median(window)
        drop = (baseline - run["yield_per_page"]) / baseline
        run["baseline"], run["drop"] = round(baseline, 3), round(drop, 3)
        run["verdict"] = "DECAY_WARN" if drop > threshold else "OK"
        if drop > threshold and first_warn is None:
            first_warn = run
    return first_warn

The heart of it is one line:

window = [r["yield_per_page"] for r in runs[idx - k - gap:idx - gap]]

Today's yield isn't compared to the last 7 runs. It's compared to 7 runs that ended 14 to 8 runs ago — your settled past. I'll explain in a second why that gap is the whole game.

Run it and you get:

=== YIELD DECAY PROBE ===
runs in log              : 60
every run exit 0         : True
every run schema ok      : True
baseline window (K)      : 7 runs, lagged by 7 (settled past)
warn threshold           : 15% below baseline
min single-run rows      : 2384
max single-run rows      : 3880
--------------------------------------------------------
FIRST DECAY WARN         : run 48
  baseline yield/page    : 47.8
  this run  yield/page   : 40.3
  drop vs baseline       : 15%
  this run exit code     : 0  (GREEN)
  this run rows          : 3224  (plausible)
--------------------------------------------------------
latest run (run 60)         : yield 29.8/page, rows 2384, exit 0
latest verdict           : DECAY_WARN (drop 25% vs baseline 40.2)
========================================================

Read what that output is actually saying:

every run exit 0 = True, every run schema ok = True. All 60 runs pass every single-run check. There is nothing here a status code or schema validator would catch.
min single-run rows : 2384. Even the worst run pulled 2,384 rows. A rows < 3000 gate would have passed the early decay and only barely caught the late stuff. A rows < 2000 gate never trips at all.
The decay starts at run 41. The probe's first warning fires at run 48 — about halfway down, while the run is fully green: exit 0, 3,224 rows, perfectly plausible. You get the flag weeks before this becomes "we're collecting half of what we used to."
By run 60, yield is down 25% and rows have slid to 2,384. Without the probe, that's still a green run. With it, you'd have known seven runs in.

The part that surprised me: the obvious detector is silent

The first version I'd reach for, and the version most people write, uses a trailing median. Baseline = median of the last K runs. No gap. It feels right.

It's a boiling-frog trap, and I mean that literally. On a slow drift the baseline sinks at the same rate as the signal. Every step from one run to the next is within threshold, because the thing you're comparing against already moved down too. The detector congratulates itself the whole way down.

I didn't take that on faith. I ran the same log through a trailing median (GAP=0):

NAIVE trailing-median (GAP=0):
  warns fired         : 0
  first warn at run   : None
  latest run yield    : 29.8 (started at 48.0, now down 25%)

Zero warnings. Yield fell from 48.0 to 29.8 — a quarter gone — and the "obvious" detector never said a word. It would have caught a cliff. It is structurally blind to a slide.

The fix is the lag. Compare today not against the recent past (which the decay has already infected) but against a settled window further back — runs K..2K ago. That window remembers what healthy looked like. The drop is measured against memory, not against the slowly-poisoned present. Same probe, GAP=7, and it fires at run 48.

If you take one thing from this post: a trend detector whose baseline includes recent data can't see a slow trend. Make the baseline lag.

What this does NOT catch (and where it cries wolf)

I'm not going to oversell 20 lines.

It needs runs. With K + GAP = 14, the probe says BUILDING_BASELINE until you have enough history. Brand-new scraper, sparse schedule — no signal yet. This is a tool for sources you hit repeatedly, which is exactly where slow decay hides.

A genuine cliff also trips it — correctly, but you'll want context. If a source legitimately halves overnight (they really did remove half the listings), the probe fires. That's not a false positive, but it's not decay either; it's a step change. The probe tells you something moved, not why.

Seasonality and legitimate shrinkage will cry wolf. A source that's genuinely quieter on weekends, or a category that's actually emptying out, will look like decay. The probe has no idea your source is supposed to shrink. You'll get warnings you have to read and dismiss. A single global threshold is blunt; per-source thresholds are better, and I haven't built the per-source version into these 20 lines.

It assumes yield is comparable run to run. If your page budget changes between runs, normalize on rows-per-page (as the probe does), not raw rows. If even the per-page meaning drifts, you need a smarter denominator than I've shown here.

So: it's a smoke alarm, not a diagnosis. It earns its 20 lines by catching the one failure that every green log hides — and it will occasionally beep at burnt toast.

What to actually do Monday

Three changes, smallest first:

Log yield per run. Not only the exit code. One number (rows, or rows-per-page if your budget varies) written to a durable run log. If you're not logging it, you can't see the curve, full stop.
Alert on trend, not on an absolute floor. A rows < N gate is a tripwire at one height; decay walks under it. Compare each run to a lagged baseline of its own source.
Make the baseline lag. Trailing windows go blind to slow drift. Median of runs K..2K ago. That's the difference between "no decay detected" and "warn at run 48."

You don't need a metrics platform to start. You need the run log you probably already half-have, plus this probe reading it. Grafana is great once you've decided what to watch. This tells you what to watch before you've stood anything up.

One open question I haven't settled: across 962 runs of one source, how much of the yield wobble is the source genuinely changing vs. our own throttling/proxy behavior leaking into the curve? I can see the curve move; cleanly attributing each dip is harder than I'd like. If you've separated "the source changed" from "my client changed" in a long run history, I'd genuinely like to hear how — I read every comment.

Follow for the next numbers from the run log. And tell me the slowest, sneakiest scraper decay you've watched happen — the one no alert ever caught.

Written by Aleksei Spinov — I run production scrapers (2,190 runs across 32 actors; one Trustpilot scraper at 962). Proof: blog.spinov.online and my Apify profile.

AI disclosure: drafted with AI assistance, then edited, fact-checked, and the code run and verified by me. The run log is synthetic and deterministic; the output above is real stdout from executing the script.

Your Scraper Collected 50 Rows. There Were 4,000.

Alex Spinov — Sat, 06 Jun 2026 18:12:15 +0000

A scraper can pass every check you wrote and still be wrong about the one thing you actually care about: how much it collected.

No exception. No 500. No broken row. Exit code 0, logs green, every field valid. And the set on disk is a quarter of what the site actually has. I have run scrapers in production enough times to stop trusting a green run on its own, and this is the failure that taught me to count.

TL;DR

A paginated source can serve fewer rows than it claims and never throw — page caps, hidden offset limits, infinite scroll that "ends" early.
Your status check (200), schema check (valid row), and byte check (you got data) all pass. None of them counts records.
The tell: declared total vs unique ids collected. Or, when there's no declared total, the page that quietly repeats an earlier page.
Below is a 40-line probe you can run right now. On a source that caps at 1,500 of a declared 4,000, it returned VERDICT: INCOMPLETE (missing 2500 rows).
This is a completeness check, not a correctness check. Different layer, different bug.

What actually goes wrong

You write the loop everyone writes. Walk ?page=1, ?page=2, keep going until a page comes back empty. Stop. Save. Done.

The source has other plans. It says it has 4,000 records — the count is right there in the envelope, or in a "Showing 4,000 results" line in the HTML. But it only ever hands out real data for the first 30 pages. Page 31 doesn't error. It doesn't return empty either. It returns page 1 again. Still HTTP 200. Still 50 valid rows. Your loop has no reason to stop, so it grinds on until its own page budget runs out, collects a pile of rows, and exits clean.

You now have 5,000 rows in hand and feel great about it. Looks like plenty. The catch: only 1,500 are unique. The page cap fed you the same first page over and over, and those duplicates hid the shortfall behind a big-looking row count. That is the exact shape of "50 rows passed every check while 4,000 existed" — the scraper saw a lot of rows and trusted the volume.

This is a completeness check, not a correctness check

Quick scope, because this lands next to three failures I've written about and it is none of them. A bad status code is the schema canary, where HTTP 200 lies and the body is junk. A wrong field inside a valid row is a clean row that's still wrong, a different problem with its own fix. And bytes you paid for that returned nothing is a cost problem; this is a count problem. Here the run is green and every row is correct. What's wrong is the number of rows: you collected fewer than exist, and nothing threw. This check lives between your scraper and the source's own claim about how many records there are. It is not about resume, crashes, ETags, 304s, or whether the data went stale. Just one question: did you get all of it.

That distinction matters because the tools that catch the other three are blind here. A status check sees 200 and is happy. A schema check sees a valid row and is happy. A byte counter sees data flowing and is happy. None of them ever asks "is this all of it." That question needs its own line of code.

Where I keep meeting this

Listing sources. Anything paginated where the platform decides how deep you're allowed to go. The scraper I've leaned on most for this — a Trustpilot review collector — has 962 production runs behind it, and reviews are paginated to the bone. "Showing N of M," page after page, with the platform free to stop serving real pages whenever it wants. That's the genre where the declared count and the collected count drift apart, and where a green run means almost nothing on its own.

I want to be precise about what I'm claiming, because the cheap version of this post would inflate it. I am not going to tell you "page caps cost me X rows on site Y" — I don't keep a clean tally of how many runs hit a silent cap specifically, so I won't invent one. What I'll stand behind: across 2,190 production runs, the failure that scared me most wasn't the loud one. The loud ones page you. This one ships a confident, half-empty dataset into something downstream and waits.

The probe

Here's the whole thing. Pure stdlib, no network, no browser. The mock source lies the way real ones do, so you can watch the probe catch it before you wire it to your own fetch.

import hashlib

PAGE_SIZE = 50
DECLARED_TOTAL = 4000          # what the envelope claims exists
HIDDEN_PAGE_CAP = 30           # server silently refuses real data past this page
PAGE_BUDGET = 100              # every real scraper has a safety budget; so do we
# 30 pages * 50 = 1,500 reachable rows out of a declared 4,000

def mock_api(page):
    """One page, 1-based. The bug: any page past the cap serves page 1 again,
    still HTTP 200 with a valid envelope. No error, no empty page."""
    served = page if page <= HIDDEN_PAGE_CAP else 1   # <-- the silent cap
    start = (served - 1) * PAGE_SIZE
    rows = [{"id": start + i, "name": f"item-{start + i:05d}"}
            for i in range(PAGE_SIZE)]
    return {"total": DECLARED_TOTAL, "page": page, "rows": rows}

def page_fingerprint(rows):
    ids = ",".join(str(r["id"]) for r in rows)
    return hashlib.sha1(ids.encode()).hexdigest()[:12]

def scrape_naive():
    """Walk pages until one looks empty. It never looks empty here, so we
    stop on the page budget and exit clean -- like real code does."""
    collected, first_fp, cap_at_page = [], None, None
    page = 1
    while page <= PAGE_BUDGET:
        rows = mock_api(page)["rows"]
        if not rows:
            break
        fp = page_fingerprint(rows)
        if page == 1:
            first_fp = fp
        elif fp == first_fp and cap_at_page is None:
            cap_at_page = page - 1       # page K repeats page 1 -> cap is K-1
        collected.extend(rows)
        page += 1
    return collected, first_fp, cap_at_page, page - 1

Two checks do the work, and they cover the two cases you actually meet.

Path A — you have a declared total. Compare it to your unique ids, not your raw count. Raw count is the thing the duplicates inflate; unique ids is the thing that tells the truth.

Path B — there is no declared total. Plenty of sources don't give you one. Then the anchor is the fingerprint: the page that repeats an earlier page is exactly where the source quietly looped you. No total needed.

def main():
    collected, first_fp, cap_at_page, pages_walked = scrape_naive()
    unique_ids = len({r["id"] for r in collected})
    declared = DECLARED_TOTAL
    completeness = unique_ids / declared if declared else 1.0

    print("=== COMPLETENESS PROBE ===")
    print(f"declared total (envelope) : {declared}")
    print(f"rows collected (raw)      : {len(collected)}")
    print(f"unique ids collected      : {unique_ids}")
    print(f"pages walked              : {pages_walked}")
    print(f"page-1 fingerprint        : {first_fp}")
    if cap_at_page is not None:
        print(f"page {cap_at_page + 1} repeats page 1 -> "
              f"SILENT PAGE CAP at page {cap_at_page}")
    verdict = "INCOMPLETE" if unique_ids < declared else "OK"
    print(f"completeness ratio        : {unique_ids}/{declared} = {completeness:.3f}")
    print(f"VERDICT                   : {verdict} (missing {declared - unique_ids} rows)")

Run it. This is the captured output from my machine, Python 3.13.5, no edits:

=== COMPLETENESS PROBE ===
declared total (envelope) : 4000
rows collected (raw)      : 5000
unique ids collected      : 1500
pages walked              : 100
page-1 fingerprint        : 323c5cd0274b
page 31 repeats page 1 -> SILENT PAGE CAP at page 30
completeness ratio        : 1500/4000 = 0.375
VERDICT                   : INCOMPLETE (missing 2500 rows)

Read it line by line

rows collected (raw) : 5000 is the trap. Five thousand rows feels like a win. It's the number a naive run brags about.

unique ids collected : 1500 is the truth. The page cap fed back page 1 from page 31 onward, so 3,500 of those 5,000 rows are duplicates. Strip them and you have 1,500.

page 31 repeats page 1 -> SILENT PAGE CAP at page 30 is the second detector earning its place. It found the cap without trusting the declared total at all — useful for every source that won't tell you how many records it has.

completeness ratio : 1500/4000 = 0.375 is the headline. You collected 37.5% of what the source itself says exists. Three-eighths.

VERDICT : INCOMPLETE (missing 2500 rows) is the one boolean you bolt onto your run today. Green exit code, INCOMPLETE verdict. Those two are allowed to disagree, and when they do, the verdict is right.

What to do with this on Monday

Add the unique-id-vs-declared check to your pipeline and fail the run loud when the ratio drops below whatever floor you trust. I'd start strict — anything under 0.95 gets a human — and loosen it once you know a given source's normal drift.

If the source gives no total, keep the fingerprint check. The page that repeats an earlier page is a free signal that the source stopped serving you real data. Cheap to compute, hard to fake.

And stop reporting raw row count as success. Report unique ids against the declared total, or against your own previous high-water mark for that source. Raw count is the number that lies to you the most cheerfully.

One thing I'm still unsure about, and I'll say so plainly: the fingerprint trick assumes the source repeats a whole prior page. Some caps don't loop — they just return a final partial page and stop, or shuffle order so no two pages match exactly. I haven't found one clean detector that covers every flavor of silent cutoff. If you've hit a cap shape that slips past both the unique-id check and the page-repeat check, that's the case I most want to hear about.

Written by Alexey Spinov. I run production scrapers — 2,190 runs across 32 published actors, the Trustpilot collector alone at 962 — and I write up the failures that a green run hides. This post was drafted with AI assistance and edited, fact-checked, and run by me; the probe output above is captured from a real run on my machine, not generated.

Follow for the next batch of numbers from real runs. And tell me in the comments: what's the worst silently-incomplete dataset you've shipped before you noticed? I read every one.

Your Scraper Died at Row 12,000. The Rerun Pattern.

Alex Spinov — Fri, 05 Jun 2026 18:18:31 +0000

My scraper died at row 12,000 of 50,000, three hours in. The crash itself was cheap. A process gets OOM-killed, a quota trips, a machine reboots, it happens. The expensive part came next: I re-ran it. From zero. And paid, in time and in requests, for the 11,999 rows I already had sitting on disk.

That second bill is the one nobody writes code for. This post is the code. It's about 40 lines of stdlib Python that let a crashed job pick up where it died, fetching only the missing rows and writing zero duplicates, plus the real captured output of a run that crashes and a rerun that finishes it cleanly.

To be clear about scope: this is the run after the crash — how to restart a long job so it finishes the work it lost without re-fetching what it already pulled and without writing a row twice. It is not retry/backoff inside a single request (that's a different post of mine), not schema-drift detection (the post where I said "a crash loses the run" — this is the part where you get the run back), not a budget kill-switch that stops a runaway, and explicitly not conditional-GET / ETag / "skip unchanged pages" — that's freshness, a separate question entirely. Just: your job died mid-way, the clock and the bill are still running, how do you resume cheap and clean.

TL;DR

A long scrape that dies at hour 3 of 4 didn't lose one request. It lost the whole run. Retry doesn't help here; resume does.
The fix is three small things: a stable idempotency key per item, a checkpoint cursor written atomically to disk, and an upsert instead of a blind append.
I ran a 5,000-row local job, killed it at row 3,000, and reran it. The rerun fetched only the missing 2,000 and wrote zero duplicates. Final output: 5,000 rows, 5,000 unique. Real captured output below.
Across 2,190 production runs (962 on a single Trustpilot source), long jobs do die mid-way. The cost that bites isn't the crash — it's paying to re-collect everything you already had.
It's stdlib Python. No DB, no framework, no paid API. You can reproduce both runs in about five seconds.

Retry is the wrong layer

Retry fixes a request. It does nothing for a job. That's the whole confusion.

When a scraper flakes, the reflex advice is "add retries with backoff." Good advice, for the layer it lives at. A 429, a connection reset, a slow socket: retry the request a few times with jitter and most transient failures evaporate. I'm a believer; I wrote a whole post on closing resources and retrying inside a run.

But think about what actually happened at row 12,000. The process is gone. The Python interpreter that held your in-memory list of results, your retry counter, your for loop: all of it, evaporated. There is no request to retry, because there is no process left to retry it in. Retry operates inside a run. The thing that died is the run.

So the recovery layer isn't the request. It's the job. And the canonical "just add retries" advice quietly skips that level, because at the request layer everything looks handled.

I bumped into this exact gap once and walked straight past it. In an earlier post about silent schema drift, I argued for making a data-shape check non-fatal, and the reason I gave was: "a crash loses the run. If you blow up on record 12,000 of 50,000, you've thrown away the 11,999 good records you already pulled." True. But I used it only as an argument to not crash this run. I never said what happens on the next run after a crash you didn't prevent. This post is that next run.

What makes a rerun cheap?

A rerun is cheap when it does only the work the first run didn't finish. To get there you need three things, and none of them is fancy.

1. A stable idempotency key, per item. Not the row number. The row number is a lie the moment you skip something: skip 3,000 rows and item 3,001 is now "row 1" of the rerun. Key off something the source gives you: an id, a URL, a SKU. Mine is (source, item_id). So that the second run can ask "do I already have this exact item?" and get a truthful yes/no regardless of order.

2. A checkpoint cursor, written atomically. You want to flush progress to disk as you go, not hold it in memory where a crash takes it with you. The subtle part: writing the cursor itself can crash mid-write, leaving you with a truncated, useless file. The fix is write-to-temp-then-rename: os.replace() is atomic, so the cursor on disk is always either the old complete value or the new complete value, never a half-written one. So that even a crash during a checkpoint can't corrupt your recovery state.

3. An upsert, not a blind append. A naive scraper opens its output and appends every row it scrapes. Rerun it and you get every row twice. The pattern instead reads which keys are already written, and skips them. So that the corpus stays clean no matter how many times the job restarts.

That's it. Stable key, atomic cursor, skip-what-you-have. The cleverness is in not being clever.

The pattern

Pure stdlib. The "scrape" here is a deterministic generator of 5,000 items instead of a network call — on purpose, so you can run it yourself in seconds with no proxies, no keys, no target site. The mechanic being demonstrated (key + checkpoint + upsert + delta rerun) doesn't depend on the transport; swap work_items() for your real fetch and the recovery logic is unchanged.

The output file is line-per-record JSON. That choice matters: each row is durably appended on its own line, so a crash costs you at most one half-written final line — and the loader below skips exactly that.

import os, json

TOTAL = 5000
CRASH_AT = 3000
OUT = "scrape_output.jsonl"
CURSOR = "cursor.json"

def work_items():
    # The "scrape". Stable per-item id, NOT a row counter.
    for i in range(TOTAL):
        yield {"source": "demo", "item_id": i, "payload": f"value-{i:05d}"}

def idem_key(item):
    return f"{item['source']}:{item['item_id']}"   # stable across reruns

def load_done_keys(path):
    # Rebuild what's already written. The output FILE is the source of truth.
    done = set()
    if not os.path.exists(path):
        return done
    with open(path, encoding="utf-8") as f:
        for line in f:
            line = line.strip()
            if not line:
                continue
            try:
                rec = json.loads(line)
            except json.JSONDecodeError:
                continue   # half-written final line from the crash — skip it
            done.add(f"{rec['source']}:{rec['item_id']}")
    return done

def checkpoint(path, last_index):
    # Atomic: temp file + os.replace. A crash mid-write can't truncate it.
    tmp = path + ".tmp"
    with open(tmp, "w", encoding="utf-8") as f:
        json.dump({"last_index": last_index}, f)
        f.flush(); os.fsync(f.fileno())
    os.replace(tmp, path)

The loop ties them together. On a resume, it loads the done-keys first, then walks the same item stream and skips anything already on disk — the upsert — appending only the delta and checkpointing the cursor as it goes:

def run(resume):
    done = load_done_keys(OUT) if resume else set()
    fetched_this_run = duplicates = 0
    with open(OUT, "a", encoding="utf-8") as out:
        for idx, item in enumerate(work_items()):
            key = idem_key(item)
            if key in done:               # already have it -> skip (upsert)
                duplicates += 1            # a blind-append script re-writes here
                continue
            if (not resume) and idx == CRASH_AT:
                raise RuntimeError(f"simulated crash at index {idx}")
            out.write(json.dumps({**item, "result": item["payload"].upper()}) + "\n")
            out.flush()
            done.add(key)
            fetched_this_run += 1
            if idx % 500 == 0:
                checkpoint(CURSOR, idx)

Notice the duplicates += 1 line. On the first run it never fires. On the rerun it fires once for every item already on disk — that counter is the proof that a blind-append version of this script would have written those rows a second time, and this one didn't.

The full runnable file (with the cursor reader, the summary print, and the --resume flag) is at the bottom.

Run it: crash, then resume

Two commands. First run starts fresh and dies at index 3,000. Second run resumes. Here's the actual terminal, copy-pasted, not cleaned up:

########## RUN 1 (fresh, will crash) ##########
Traceback (most recent call last):
  File "resume_demo.py", line 126, in <module>
    run(resume=a.resume)
  File "resume_demo.py", line 99, in run
    raise RuntimeError(f"simulated crash at index {idx}")
RuntimeError: simulated crash at index 3000
exit code: 1

########## state on disk after crash ##########
rows in output:     3000
cursor: {"last_index": 2500}

########## RUN 2 (--resume) ##########
=== RUN 2 (--resume) summary ===
resumed from cursor index : 2500
items already on disk      : 3000
fetched this run           : 2000
duplicate writes avoided   : 3000
final rows in output       : 5000
exit code: 0

And the independent check on the file afterward, because a summary that prints its own numbers is a summary you shouldn't trust:

total lines in output: 5000
unique item_ids: 5000
duplicate lines (should be 0): 0

Read the second run's numbers. fetched this run: 2000, not 5,000. The rerun touched only the rows the crash lost. duplicate writes avoided: 3000 means every row that was already on disk got skipped instead of re-written. final rows: 5000, and unique item_ids: 5000 from the independent count, so the job is genuinely complete with nothing doubled. The crash cost me the last 2,000 rows and exactly nothing else.

One honest wrinkle, because I'd rather point it out than have you spot it. The crash happened at index 3,000, but the cursor on disk said 2500. They disagree by 500. That's not a bug, it's the design. The cursor is checkpointed every 500 rows, but every row is flushed to the output file the instant it's written. So the output file, not the cursor, is the real source of truth: load_done_keys() rebuilds progress from the 3,000 rows actually on disk, and the cursor is just a cheap hint. If I trusted the cursor alone I'd have re-fetched 500 rows I already had. Trusting the durable output instead, I re-fetched zero. Pick the more durable record as your truth.

What this costs you in production

The reason I care about this isn't the demo. It's that long jobs really do die, and I have the run counter to say so.

I run scrapers in production: 2,190 runs across 32 published actors, with one Trustpilot review scraper at 962 runs by itself (that's a lifetime run meter on my Apify profile, knotless_cadence, as of mid-2026; not a controlled study, just a long-running counter). When you run one source nearly a thousand times, you stop asking if a multi-hour job will get interrupted and start assuming it will. OOM on a big page batch, a proxy pool hiccup, a quota wall, a deploy that restarts the worker: any of them ends the process, and the process is the run.

Here's the cost math, on the numbers from my opening. A 50,000-row job that dies at row 12,000. Rerun-from-zero re-pays for all 50,000, and 24% of that work (the 12,000 you'd already done) is pure waste. But flip the crash point. Most jobs die late, not early, because the longer they run the more chances they have to hit something. A job that dies at row 40,000 of 50,000 and reruns from zero re-collects 40,000 rows you already had: you pay 80% of the bill a second time to recover the last 20%. Resume-the-delta pays for 10,000. That's the whole pitch: the later the crash, the more brutal the rerun-from-zero penalty, and the more a stable key plus a durable output saves you.

You might reasonably say: 2,000 rows on one laptop isn't a distributed production crawl. Fair. It isn't. The mechanic is identical, but a single-machine flat file is the simplest place it lives, not the only one — and the next section is exactly where this version stops being enough.

Where this breaks

I'd rather hand you the failure modes than let you find them at row 12,000 of your own job.

The source gives you no stable id. The whole pattern hangs off the key. If items have no id, URL, or natural unique field, you're stuck choosing a worse one: content hash (breaks the instant the content legitimately changes), position (breaks the instant you skip), or fuzzy match (breaks in ways you won't notice for weeks). This is the genuinely hard part, and I haven't solved it cleanly.
One machine, one file. Reading done-keys from a local file is fine for a single worker. Run the same job across a pool of workers and a flat file becomes a race: two workers can both read "not done," both fetch, both write. At that point the done-keys set has to live somewhere shared and atomic — Redis SETNX, a unique constraint in Postgres, an INSERT ... ON CONFLICT DO NOTHING. Same idea, different home for the key.
Flat-file upsert is not a database. load_done_keys() reads the whole output to rebuild the set. At a few thousand rows that's instant. At tens of millions it's a startup cost you'll feel, and the right move is a real keyed store, where "have I seen this key" is an index lookup, not a file scan.
It doesn't help if the source died, not your job. Resume assumes the data is still there to re-fetch. If the site is down, rate-limiting you to a crawl, or has removed the rows since your first pass, resuming cleanly still gets you an incomplete corpus. The pattern recovers your failure, not the world's.

That last one is the boundary I keep relearning. A clean resume is a promise about your bookkeeping, not about the source.

What I'd change on Monday

If you run anything that takes more than a few minutes, do these three before the next long job:

Pick the idempotency key before you write the scraper. It's the one decision the whole recovery story depends on, and it's free to get right up front and expensive to retrofit. If the source has a stable id, use it. If it doesn't, that's a design problem to solve now, not at row 12,000.
Make your output durable per-row and treat it as the source of truth. Append line-per-record and flush. Then "what have I already done" is a question you answer from disk, not from a process that might not exist anymore. The cursor is a hint; the output is the truth.
Make the rerun the default way you finish a job, not the emergency. A job you can stop and resume at any row is a job you can also run in cheap chunks, pause for a deploy, or split across a maintenance window. Resume isn't just crash insurance — it's what makes a long job something you can actually operate.

Open question I haven't solved cleanly, and I'd genuinely like your answer: what's your idempotency key when the source gives you no stable id? Content hash, scroll position, fuzzy match on a few fields — every option I've tried has a failure mode that shows up weeks later in a way that's painful to debug. What do you actually use in prod?

Full script (resume_demo.py, stdlib only — run python3 resume_demo.py then python3 resume_demo.py --resume):

#!/usr/bin/env python3
"""resume_demo.py — resume a crashed job WITHOUT re-fetching or double-writing.

The "scrape" is a deterministic local generator (no network, no browser, no paid
API) — the mechanic shown (idempotency key + atomic checkpoint + upsert +
delta-only rerun) does not depend on the transport, so anyone can reproduce this.

  python3 resume_demo.py            # run 1: crashes at CRASH_AT
  python3 resume_demo.py --resume   # run 2: finishes only the missing delta
"""
import os, sys, json, argparse

TOTAL = 5000                     # items in the whole job
CRASH_AT = 3000                  # run 1 dies right before processing this index
CHECKPOINT_EVERY = 500           # flush the cursor to disk this often
OUT = "scrape_output.jsonl"      # durable, line-per-record output
CURSOR = "cursor.json"           # last-known-good progress marker


def work_items():
    """The 'scrape'. Yields 5,000 records with a STABLE per-item id."""
    for i in range(TOTAL):
        yield {"source": "demo", "item_id": i, "payload": f"value-{i:05d}"}


def idem_key(item):
    """Idempotency key = (source, item_id). Stable across reruns, not position."""
    return f"{item['source']}:{item['item_id']}"


def load_done_keys(path):
    """Rebuild the set of keys already written. The output file is the truth."""
    done = set()
    if not os.path.exists(path):
        return done
    with open(path, "r", encoding="utf-8") as f:
        for line in f:
            line = line.strip()
            if not line:
                continue
            try:
                rec = json.loads(line)
            except json.JSONDecodeError:
                continue  # half-written final line from the crash — skip it
            done.add(f"{rec['source']}:{rec['item_id']}")
    return done


def checkpoint(cursor_path, last_index):
    """Atomic write: temp file + os.replace. Crash mid-write can't truncate it."""
    tmp = cursor_path + ".tmp"
    with open(tmp, "w", encoding="utf-8") as f:
        json.dump({"last_index": last_index}, f)
        f.flush()
        os.fsync(f.fileno())
    os.replace(tmp, cursor_path)


def read_cursor(cursor_path):
    if not os.path.exists(cursor_path):
        return -1
    with open(cursor_path, "r", encoding="utf-8") as f:
        return json.load(f).get("last_index", -1)


def process_one(item):
    """The 'expensive' step. In prod this is the network fetch you pay for."""
    return {**item, "result": item["payload"].upper()}


def run(resume):
    done = load_done_keys(OUT) if resume else set()
    if not resume:
        for p in (OUT, CURSOR):
            if os.path.exists(p):
                os.remove(p)
    resumed_from = read_cursor(CURSOR) if resume else -1

    fetched_this_run = 0
    duplicates = 0
    out = open(OUT, "a", encoding="utf-8")
    try:
        for idx, item in enumerate(work_items()):
            key = idem_key(item)
            if key in done:           # idempotency: already have it, skip
                duplicates += 1        # a blind-append script would re-write here
                continue
            if (not resume) and idx == CRASH_AT:
                raise RuntimeError(f"simulated crash at index {idx}")
            rec = process_one(item)
            out.write(json.dumps(rec) + "\n")
            out.flush()
            done.add(key)
            fetched_this_run += 1
            if idx % CHECKPOINT_EVERY == 0:
                checkpoint(CURSOR, idx)
        checkpoint(CURSOR, TOTAL - 1)
    finally:
        out.close()

    final_rows = len(load_done_keys(OUT))
    label = "RUN 2 (--resume)" if resume else "RUN 1 (fresh)"
    print(f"=== {label} summary ===")
    print(f"resumed from cursor index : {resumed_from}")
    print(f"items already on disk      : {len(done) - fetched_this_run}")
    print(f"fetched this run           : {fetched_this_run}")
    print(f"duplicate writes avoided   : {duplicates}")
    print(f"final rows in output       : {final_rows}")


if __name__ == "__main__":
    ap = argparse.ArgumentParser()
    ap.add_argument("--resume", action="store_true",
                    help="continue an existing output instead of starting fresh")
    a = ap.parse_args()
    run(resume=a.resume)

Written by Aleksey Spinov. I write up the cost and failure math from real production scraping — 2,190 runs and counting. Follow for the next one, and tell me your idempotency key for a source with no stable id — I read every comment.

AI disclosure: drafted with AI assistance; the pattern, the script, and every number in this post were produced and verified by me. The Python here was run locally (stdlib, no third-party deps); the crash, the resume, and the independent file check shown are the real output, not a mock-up.

A 30-Line Probe That Tells You If a Page Needs a Browser

Alex Spinov — Fri, 05 Jun 2026 01:03:40 +0000

Half the "you don't need a browser" takes on my feed this week are right. None of them tell you how to check. They tell you headless Chrome is expensive — true — and then leave you exactly where you started: guessing, per target, whether you can skip it.

You don't have to guess. Whether a page needs a browser is a question you can answer from the raw HTTP response, before you launch anything. Here's a 30-line probe that does it, and the real output from running it on ten named public URLs.

To be clear about scope: this is the decision you make before you start a job — should I launch Chrome at all for this URL — not how to survive headless once it's running, not how much the raw HTML costs you in LLM tokens, not the proxy bandwidth bill. Just: browser, or no browser, on this target.

TL;DR

A page needs a browser only if the data you want isn't in the raw HTTP HTML. You can test that with urllib, no headless required.
The probe reads three cheap signals from the raw response — visible-text size, an embedded JSON/hydration blob, and whether your target text literally appears — and votes NO_BROWSER / JS_REQUIRED / MAYBE.
I ran it on 10 named public URLs. 6 of 10 returned their data without a browser. Two genuinely needed JS, two were borderline (MAYBE — the probe says so on purpose).
It's a heuristic. It will be wrong on scroll-loaded content, data behind auth, and anti-bot walls — the post is honest about exactly where.

Why "launch Chrome just in case" is a tax, not caution

The default a lot of scrapers reach for is browser-by-default: put Playwright or headless Chrome in front of every target because "it's more reliable." It feels safe. It is not free.

I run scrapers in production — 2,190 runs across 32 published actors, the Trustpilot one alone at 962 runs. Here's the part that doesn't show up in any tutorial: a headless instance costs memory, CPU, and cold-start time per run. Multiply that by hundreds of runs and the "just in case" browser is a standing line item — paid on every page, including the pages that would have handed you the data over plain HTTP in 80 milliseconds.

So the default is backwards. It should be HTTP-first, browser-on-fallback. And the thing that decides which path a URL takes shouldn't be a vibe. It should be a measurement.

What actually tells you a page needs a browser

A page needs a browser when the data you want exists only after JavaScript runs. That's it. So the probe asks the inverse question of the raw HTTP HTML: is the data already here?

Three cheap signals, all readable from the bytes urllib gives you:

Visible-text size. Strip <script> and <style>, strip tags, measure what's left. A real article leaves tens of kilobytes of text. An empty SPA shell leaves almost nothing — the body is a <div id="root"> and a bundle.
An embedded data blob. Lots of "JS-heavy" sites actually ship their data inside the first HTML response as JSON: __NEXT_DATA__, __NUXT__, window.__INITIAL_STATE__, or a <script type="application/ld+json">. If that blob is there, you don't need a browser — you need a JSON parser.
The needle. If you know the exact text you're after (a price, a review snippet, a name), the cleanest test is: does that string appear in the raw HTML at all? Present → no browser. Absent → the browser is rendering it in.

The probe

Pure stdlib. No requests, no Selenium, no Playwright — the whole point is to decide before a browser exists. If you can run python3, you can run this.

import sys, re, gzip, argparse
from urllib.request import Request, urlopen

UA = "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 needs_a_browser/1.0"
HYDRATION = (b"__NEXT_DATA__", b"__NUXT__", b"__INITIAL_STATE__",
             b"__APOLLO_STATE__", b"window.__data", b'type="application/ld+json"',
             b'application/json"')

def fetch(url):
    req = Request(url, headers={"User-Agent": UA, "Accept-Encoding": "gzip"})
    with urlopen(req, timeout=20) as r:
        raw = r.read()
        if r.headers.get("Content-Encoding") == "gzip":
            raw = gzip.decompress(raw)
    return raw

def visible_text(html):
    s = re.sub(rb"(?is)<(script|style)\b.*?</\1>", b" ", html)  # drop scripts
    s = re.sub(rb"(?s)<[^>]+>", b" ", s)                        # drop tags
    return re.sub(rb"\s+", b" ", s).strip()

def probe(url, needle=None):
    html = fetch(url)
    text = visible_text(html)
    text_len = len(text)
    has_blob = any(m in html for m in HYDRATION)
    needle_hit = needle is not None and needle.lower().encode() in text.lower()

    if needle is not None:
        verdict = "NO_BROWSER" if needle_hit else "JS_REQUIRED"
    elif text_len < 500 and not has_blob:
        verdict = "JS_REQUIRED"            # empty shell, nothing to parse
    elif text_len >= 2000 or has_blob:
        verdict = "NO_BROWSER"             # data already in the raw HTML
    else:
        verdict = "MAYBE"                  # borderline — the probe says so
    return verdict, f"text={text_len}B blob={has_blob} needle={needle_hit}"

That's the logic. The __main__ block just loops over the URLs you pass, tallies the verdicts, and prints X of N pages returned data without a browser. Full file at the end.

The thresholds (500, 2000) are deliberately blunt. They're not a model fitted to anything — they're "is there clearly nothing here" and "is there clearly a lot here," with an honest gap in the middle called MAYBE. You can tune them. The point isn't the constants, it's that the question is answerable from bytes you already have.

The real run

I pointed it at ten public URLs. Mix of static content, a forum, a couple of deliberate JS controls, and two versions of the same site so you can see the probe flip. Here's the actual output, copy-pasted, not cleaned up:

NO_BROWSER   https://en.wikipedia.org/wiki/Web_scraping
             text=30581B blob=True needle=False
NO_BROWSER   https://news.ycombinator.com/
             text=4050B blob=False needle=False
NO_BROWSER   https://old.reddit.com/r/webscraping/
             text=7185B blob=False needle=False
JS_REQUIRED  https://www.reddit.com/r/webscraping/
             text=37B blob=False needle=False
MAYBE        https://quotes.toscrape.com/
             text=1745B blob=False needle=False
JS_REQUIRED  https://quotes.toscrape.com/js/
             text=98B blob=False needle=False
NO_BROWSER   https://www.python.org/
             text=7064B blob=True needle=False
NO_BROWSER   https://github.com/scrapy/scrapy
             text=5204B blob=True needle=False
MAYBE        https://books.toscrape.com/
             text=1883B blob=False needle=False
NO_BROWSER   https://httpbin.org/html
             text=3596B blob=False needle=False

6 of 10 pages returned data without a browser  ::  {'NO_BROWSER': 6, 'JS_REQUIRED': 2, 'MAYBE': 2}

Six of ten. Most of them didn't need a browser at all. (Re-run it and a live page like the HN front page will report a slightly different byte count — the story list changes — but the verdict holds.)

The two that did are the interesting ones, and they're the same site twice. www.reddit.com/r/webscraping/ came back with 37 bytes of visible text — a shell. old.reddit.com/r/webscraping/ came back with 7,185 bytes of real post titles. Same content, same subreddit; the new front-end renders client-side, the old one ships HTML. If your target is new Reddit, you need a browser or the JSON API. If it's old Reddit, you'd be launching Chrome to read text that was already sitting in the response. That single row is the whole argument for measuring instead of guessing.

The quotes.toscrape.com pair is the cleanest controlled version of the same thing: / ships the quotes as HTML, /js/ builds them in the browser. I'll come back to why / showed up as MAYBE here and not NO_BROWSER — it's the honest edge of this thing.

And Wikipedia, python.org, the Scrapy GitHub page — all blob=True. They look JavaScript-heavy in a browser, but the data is right there in the first response as JSON-LD or __NEXT_DATA__. Launching a browser for those is pure overhead.

When you know what you're looking for, ask directly

The structural signals (text size, blob) are a guess about whether any useful data is present. If you know the specific thing you want, skip the guess. Run it with --needle:

$ python3 needs_a_browser.py "https://quotes.toscrape.com/" --needle "Einstein"
NO_BROWSER   https://quotes.toscrape.com/
             text=1745B blob=False needle=True

$ python3 needs_a_browser.py "https://quotes.toscrape.com/js/" --needle "Einstein"
JS_REQUIRED  https://quotes.toscrape.com/js/
             text=98B blob=False needle=False

Same site, opposite verdicts, and now there's no ambiguity: the word "Einstein" is literally in the raw HTML of /, and literally absent from /js/ until JavaScript runs. Notice / was a MAYBE on structure alone (1,745 bytes — right in the borderline band) but a confident NO_BROWSER once I asked about the actual data. That's the lesson: a needle beats a heuristic. When you can name the field you're scraping, test for the field.

Where this probe is wrong

It's a heuristic. I'd rather tell you its failure modes than let you find them in production.

Scroll / lazy-loaded content. A page can ship a fat, healthy HTML head and still load the rows you want on scroll via XHR. The probe sees a big page, votes NO_BROWSER, and misses that your specific rows arrive later. The needle catches this; the structural signals don't.
Data behind auth or interaction. If the content only appears after a login or a click, an unauthenticated GET can't see it. The probe will read the logged-out shell.
Anti-bot walls. Some targets don't even let a plain urllib request finish. When I pointed this same probe at a Trustpilot review page from a datacenter IP, it didn't return NO_BROWSER or JS_REQUIRED — it threw an ssl handshake timed out, twice, repeatably. The connection got cut at the TLS layer before any HTML came back. That's not a failure of the probe; it's the probe telling you something true. This target won't talk to a bare HTTP client. You're going to a real client (browser and/or residential proxy) regardless of what the HTML would have said — which, for a scraper I've run 962 times in production, is a useful thing to learn in one second instead of one debugging session.
The MAYBE band is real. A page with ~1.5 KB of text and no blob is genuinely ambiguous from bytes alone. The probe doesn't fake confidence there. Treat MAYBE as "fetch one sample with a browser, look, then decide for the batch" — not as a verdict.

That last one is the design choice I care about most. A probe that always answers yes or no is lying part of the time. This one tells you when it doesn't know.

What I'd change on Monday

Flip the default. Don't reach for the browser first; reach for the probe first.

Probe the target before you write the scraper. One run tells you which transport you're building for. It's the cheapest decision in the whole job, and you make it before you've written a line of extraction code.
Prefer the needle to the structural guess. If you know the price, the review text, the SKU you're after, test for that. The structural signals are a fallback for when you don't.
Route MAYBE and ERROR to a human-eyeballed sample, not to a blanket "use Chrome." Launching a browser on every ambiguous URL is just the browser-by-default tax wearing a disguise.

I'll be straight about the limit one more time: 6-of-10 is the result on these ten URLs, not a law about the web. Point the probe at your own targets and you'll get your own number — that's the entire idea. The value isn't my six. It's that you can compute yours in the time it takes to read this paragraph, instead of paying for a Chrome instance on every page that never needed one.

Here's the open question I haven't solved cleanly: the scroll/lazy-load case. The needle catches it if I know a value that's only on a later page of results — but for an open-ended crawl where I don't yet know what's there, structural signals can't distinguish "all the data is here" from "the first screen is here and the rest is one XHR away." If you've found a cheap, no-browser way to detect lazy-loading from the raw response, I'd genuinely like to see it.

Full script (needs_a_browser.py, stdlib only):

#!/usr/bin/env python3
"""needs_a_browser.py — decide if a page needs a browser BEFORE you launch one.
Usage: python3 needs_a_browser.py URL [URL ...] --needle "review"
"""
import sys, re, gzip, argparse
from urllib.request import Request, urlopen

UA = "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 needs_a_browser/1.0"
HYDRATION = (b"__NEXT_DATA__", b"__NUXT__", b"__INITIAL_STATE__",
             b"__APOLLO_STATE__", b"window.__data", b'type="application/ld+json"',
             b'application/json"')

def fetch(url):
    req = Request(url, headers={"User-Agent": UA, "Accept-Encoding": "gzip"})
    with urlopen(req, timeout=20) as r:
        raw = r.read()
        if r.headers.get("Content-Encoding") == "gzip":
            raw = gzip.decompress(raw)
    return raw

def visible_text(html):
    s = re.sub(rb"(?is)<(script|style)\b.*?</\1>", b" ", html)
    s = re.sub(rb"(?s)<[^>]+>", b" ", s)
    return re.sub(rb"\s+", b" ", s).strip()

def probe(url, needle=None):
    html = fetch(url)
    text = visible_text(html)
    text_len = len(text)
    has_blob = any(m in html for m in HYDRATION)
    needle_hit = needle is not None and needle.lower().encode() in text.lower()
    if needle is not None:
        verdict = "NO_BROWSER" if needle_hit else "JS_REQUIRED"
    elif text_len < 500 and not has_blob:
        verdict = "JS_REQUIRED"
    elif text_len >= 2000 or has_blob:
        verdict = "NO_BROWSER"
    else:
        verdict = "MAYBE"
    return verdict, f"text={text_len}B blob={has_blob} needle={needle_hit}"

if __name__ == "__main__":
    ap = argparse.ArgumentParser()
    ap.add_argument("urls", nargs="+")
    ap.add_argument("--needle", default=None)
    a = ap.parse_args()
    tally = {}
    for u in a.urls:
        try:
            v, why = probe(u, a.needle)
        except Exception as e:
            v, why = "ERROR", f"{type(e).__name__}: {e}"
        tally[v] = tally.get(v, 0) + 1
        print(f"{v:<12} {u}\n             {why}")
    n = len(a.urls)
    print(f"\n{tally.get('NO_BROWSER', 0)} of {n} pages returned data without a browser  ::  {tally}")

Written by Aleksey Spinov. I write up the cost and failure math from real production scraping — 2,190 runs and counting. Follow for the next one, and if you've got a clean way to detect lazy-loaded data without a browser, drop it in the comments — I read every one.

AI disclosure: drafted with AI assistance; the probe, the URL list, and every verdict in this post were produced and verified by me. The Python here was run locally (stdlib, no third-party deps); the output shown is the real run, not a mock-up.

You Pay for the Bandwidth That Returns Nothing

Alex Spinov — Thu, 04 Jun 2026 02:05:22 +0000

A proxy invoice that says 24.79 GB · $198.28 reads like you bought 24.79 GB of data. You didn't. You bought 24.79 GB of traffic. Some of it came back with rows. Some came back with a block page, a 404, a CAPTCHA challenge, or a retry of a page that already failed. The meter doesn't care which. It counts the bytes that left the proxy, and it bills all of them at the same rate.

That gap, between bytes you paid for and rows you got back, is where money quietly leaves a healthy run. Not a runaway loop. Not an outage. A run that finished, looked fine in the dashboard, and still spent a third to a half of its bandwidth on responses that returned nothing.

TL;DR

Per-GB billing charges for failed requests, retries, and asset loads — not just rows. ("You pay for bandwidth consumed, whether requests succeed or fail." — Titan Network, 13 Apr 2026.)
In a model of a 100k-row job on a protected target, a low-success datacenter config spent 53% of its bytes returning zero rows; a high-success residential config spent 3%.
$/GB is not cost per row. The cheaper-per-GB pool was cheaper per row here — but the winner flips once success drops below ~9%.
I don't have a dollar billing ledger. The numbers below are a model on published proxy prices. Run it with your own success rate and price.

What the meter actually counts

I run scrapers in production — 2,190 runs across 32 published actors, the Trustpilot one alone at 962 runs. That's the part I can say with a straight face: I've watched a lot of real traffic. What I don't have is a per-run dollar ledger that itemizes every gigabyte. So I'm not going to paste an invoice I don't hold and call it data.

Here's what I can say from watching those logs. The bytes that return nothing aren't tail noise. They're a structural line item. Three things feed it:

Failed responses. A request that gets a 403, a challenge page, or an empty card still pulled bytes over the wire. Usually smaller than a real page. A block page isn't heavy. But it isn't free either, and at scale there are a lot of them.

Retries. Every failed request you re-attempt spends bandwidth again, and the retry often fails again. This is the multiplier most people forget. Titan Network put a number on it: moving success rate from 60% to 95% cuts your total request count by about 63%, because you stop re-issuing the misses ("Web Scraping Cost at Scale," Titan Network, 13 Apr 2026).

Asset and redirect tax. A browser-driven load on a "healthy" page pulls more than the HTML — assets, redirects, sometimes a login bounce. Even your successful traffic carries weight that never becomes a row.

None of that shows up as a problem. The run succeeds. The dashboard is green. The bill is just… higher than the rows would suggest.

A model, not a bill

So I wrote the smallest thing that makes the gap visible. It's stdlib Python, no network, no keys. It takes a job (how many rows you want), a success rate, average response sizes, a retry policy, and a $/GB price — and it tells you what you actually pay per collected row, versus the naive number you'd get if only the row-returning bytes were billed.

The dollar prices are placeholders. I marked them as illustrative in the code and I'll mark them again here: $8/GB is Titan Network's stated average for residential; $1.20/GB stands in for a cheap datacenter-style pool. Residential in 2026 runs roughly $2–$15/GB, with $8 landing in the mid-to-premium band (triangulated across Proxyway's 2026 tests, aimultiple's pricing comparison, and Titan's own figures). Swap in yours.

from dataclasses import dataclass

@dataclass
class RunConfig:
    name: str
    target_rows: int       # rows you actually want
    success_rate: float    # fraction of requests that return a usable row
    row_resp_kb: float     # avg KB of a request that returned a row
    fail_resp_kb: float    # avg KB of a request that returned no row
    asset_overhead: float  # extra byte fraction from assets/redirects
    retries_per_fail: float
    price_per_gb: float    # ILLUSTRATIVE — set yours

def model(cfg):
    requests_for_rows = cfg.target_rows / cfg.success_rate
    failed = requests_for_rows - cfg.target_rows
    retries = failed * cfg.retries_per_fail
    KB_PER_GB = 1024 * 1024

    row_bytes = cfg.target_rows * cfg.row_resp_kb * (1 + cfg.asset_overhead)
    fail_bytes = (failed + retries) * cfg.fail_resp_kb

    total_gb = (row_bytes + fail_bytes) / KB_PER_GB
    returned_gb = row_bytes / KB_PER_GB
    total_cost = total_gb * cfg.price_per_gb

    return {
        "total_gb": total_gb,
        "wasted_share": (total_gb - returned_gb) / total_gb,
        "paid_for_per_returned_gb": total_gb / returned_gb,
        "total_cost": total_cost,
        "effective_cost_per_row": total_cost / cfg.target_rows,
    }

Two configs, same job: collect 100,000 rows from a protected target. One cheap datacenter pool that gets blocked a lot. One pricey residential pool that gets through.

cheap_dc = RunConfig("datacenter (cheap/GB)", 100_000, 0.35, 180, 60, 0.40, 1.5, 1.20)
pricey_res = RunConfig("residential (pricey/GB)", 100_000, 0.95, 180, 60, 0.40, 1.5, 8.00)

Running it:

--- datacenter pool (cheap per GB) ---
  success rate           : 35%
  price (illustrative)   : $1.20/GB
  bandwidth billed       : 50.60 GB
  ... returned rows      : 24.03 GB
  ... returned NOTHING   : 26.57 GB  (53% of the bill)
  paid-for per 1GB data  : 2.11x
  total cost             : $60.72
  naive  cost/row        : $0.288 per 1,000 rows
  EFFECTIVE cost/row     : $0.607 per 1,000 rows

--- residential pool (pricey per GB) ---
  success rate           : 95%
  price (illustrative)   : $8.00/GB
  bandwidth billed       : 24.79 GB
  ... returned rows      : 24.03 GB
  ... returned NOTHING   : 0.75 GB  (3% of the bill)
  paid-for per 1GB data  : 1.03x
  total cost             : $198.28
  naive  cost/row        : $1.923 per 1,000 rows
  EFFECTIVE cost/row     : $1.983 per 1,000 rows

Look at the datacenter run. To collect 24 GB of rows it billed 50.6 GB, so it paid for 2.11× the data it kept. More than half the invoice, 53%, returned nothing. The residential run paid for 1.03×: almost everything it bought, it kept.

That's the whole point in two numbers. Same job, same row sizes. One config converts bandwidth into rows; the other converts about half of it into block pages and retries you still pay for.

So the cheap proxy is the trap, right?

No. And this is where I almost wrote the wrong article.

My first instinct was the clean contrarian line: cheap-per-GB is actually more expensive per row. But the model wouldn't cooperate. At these numbers the cheap datacenter pool costs $0.607 per 1,000 rows and the pricey residential costs $1.983 — the datacenter is 31% the per-row cost. The 6.7× price gap ($1.20 vs $8.00) is just bigger than its waste penalty. The cheap pool wins here, even bleeding 53% of its bytes.

So the honest claim isn't "cheap is a trap." It's narrower and more useful: $/GB and cost-per-row are different numbers, and which proxy is cheaper depends on how hard the target fights back. The waste fraction is a lever on price, not a verdict.

To find where it flips, I held residential at 95% and dropped the datacenter success rate — the way a target gets harder when it tightens its anti-bot:

flip point — datacenter success rate falling on a harder target:
  dc success   35% : 53% of bytes return nothing, $0.607/1k rows -> cheaper: datacenter
  dc success   20% : 70% of bytes return nothing, $0.975/1k rows -> cheaper: datacenter
  dc success   12% : 81% of bytes return nothing, $1.547/1k rows -> cheaper: datacenter
  dc success    9% : 86% of bytes return nothing, $2.024/1k rows -> cheaper: RESIDENTIAL  <-- flip
  dc success    8% : 87% of bytes return nothing, $2.262/1k rows -> cheaper: RESIDENTIAL
  dc success    5% : 92% of bytes return nothing, $3.550/1k rows -> cheaper: RESIDENTIAL

There's the flip, around 9% success. Below it, the cheap pool is wasting so much bandwidth (86% of bytes returning nothing) that even at one-sixth the price it loses on a per-row basis. Above it, cheap wins.

So "the expensive proxy is cheaper" is a regime, not a law. It's true on the targets that beat your cheap pool into the single digits. It's false on the targets your cheap pool handles fine. The only way to know which target you're on is to measure your own success rate and put it in the model — not to pick a proxy by its sticker price per GB.

What I'd change on Monday

Stop pricing proxies by $/GB in isolation. That number is the cost of the traffic, and you don't want traffic. You want rows.

Three things that move the per-row number more than the sticker price:

Log success rate per target, not globally. A 90% average can hide a target sitting at 12%, and that target is eating your bill. The flip lives in the per-target number.
Cap retries per failed request, and watch the multiplier. At 60% success you're issuing ~1.7 requests per row before retries; the retries pile on top. Re-issuing a request that fails the same way twice is just buying the same block page again.
Run the model before you switch pools. A "cheaper" pool that drops your success rate can cost more per row. A "pricey" pool that lifts it can cost less. You can't tell from the price tag.

I'll repeat the limit because it matters: this is a model on published prices, not a measured invoice. I don't have a per-run dollar ledger to show you. What I do have is the shape of the traffic from a lot of production runs — the part that returns nothing is real and it's structural — and a 60-line script that turns your own success rate into a per-row cost. The dollars are yours to fill in.

The honest open question for me: I've been treating fail_resp_kb (the size of a block/challenge response) as a flat 60 KB. On JS-challenge targets a "failed" attempt can pull a full interactive challenge page — heavier than the real data page. If your failures are bigger than your successes, the waste fraction climbs faster than this model shows. I haven't pinned that distribution down per target yet. If you've measured the byte size of your failures versus your successes, I'd genuinely like to see the numbers.

Written by Aleksey Spinov. I write up the cost and failure math from real production scraping — 2,190 runs and counting. Follow for the next one, and if you've metered the bytes a failed request actually costs you, drop the number in the comments — I read every one.

AI disclosure: drafted with AI assistance; all numbers, the model, and its output were produced and verified by me. The Python in this post was run locally (stdlib, no network); the output shown is the real run, not a mock-up.