Three checks that lied to me while building a content extraction API

MariusKa01 — Mon, 20 Jul 2026 11:42:56 +0000

Three times while building a web content extractor, a passing check told me
something that wasn't true. Not a flaky test, not a missed edge case — a green
result that actively asserted the opposite of reality. All three failed the same
way, and once I saw the pattern I couldn't unsee it.

The three

The benchmark said 90% of pages extracted successfully. The success test
asked whether the extractor returned at least 200 characters. A section landing
page — a site homepage, a /technology index — extracts as concatenated
navigation: "HomePoliticsWorldBusiness…". That clears 200 characters comfortably
while amounting to about a dozen words. So pages with no article on them at all
counted as successful extractions. The number was measuring "trafilatura returned
a string", not "we got an article." Re-scoring the same recorded run against a
100-word floor drops it from 90 to 72, and the smallest "successful" body in it
was six words.

The tests said the browser was sandboxed. I moved the service off root and
removed --no-sandbox from the Chromium launch arguments. Tests passed, pages
rendered, everything looked right. But Playwright disables the Chromium sandbox
by default and injects that flag itself, so deleting it from my own argument
list changed nothing. The verification script I wrote to prove the sandbox was on
called launch() exactly the way the application did — inheriting the same
default — and cheerfully confirmed a sandbox that had never once been enabled.
What exposed it was reading the live process list: every running renderer still
carried --no-sandbox, flatly contradicting both the code and the green tests.

The reproduction said CI's failure wasn't real. CI was red while everything
passed locally, so I rebuilt the environment to match: fresh virtualenv, only
pinned requirements, same Python version. 68 tests passed. I concluded the suite
was fine and started removing CI steps to find the culprit. Wrong. Playwright's
browser binaries live in ~/.cache/ms-playwright, shared across virtualenvs — so
my "clean" environment silently reused a Chromium the CI runner didn't have. The
fix was to point PLAYWRIGHT_BROWSERS_PATH at an empty directory and prove the
browser was unavailable before running anything.

The pattern

A check that shares an assumption with the thing it checks cannot detect a
failure in that assumption.

The character threshold shared the extractor's own idea of what counts as
content. The sandbox script shared the application's launch defaults. The "clean"
reproduction shared the machine's browser cache. Each was a mirror held up to the
code, faithfully reporting the code's own opinion back — and I read that
reflection as independent confirmation.

This is not the same as a test being wrong. A wrong test fails when it shouldn't,
and you notice. These passed. They were load-bearing, they were green, and they
were lying.

The rule is cheap to apply: before trusting a check, make it fail on purpose.
Unload the AppArmor profile and confirm the browser stops launching. Empty the
browser cache and confirm the suite breaks. Feed the success test a page of
navigation soup and confirm it says no. If you cannot make a check fail, you have
no evidence that it can — and a check that cannot fail is not measuring anything.

The corollary is what makes it worth writing down: freezing your inputs does not
save you. I had a rigorously frozen benchmark, and it produced a stable, precise,
wrong answer for weeks.

The rest of this is the project those checks were attached to, and what the
numbers look like once they're honest.

What the thing does

Give it a URL, get back clean article text and metadata. That sounds solved —
readability has existed for over a decade — but a realistic spread of the web
breaks it three ways. Content is buried in navigation, ads, cookie banners and
comment widgets. Some pages render their body with JavaScript, so an HTTP fetch
returns an empty shell. And a growing slice sits behind commercial anti-bot
systems that block automated clients outright.

The pipeline tries a fast static extraction first (trafilatura →
readability), and escalates to a real browser (Playwright) only when static
comes back under the floor or the fetch was blocked. Static handles most of it at
a median of ~0.4s; the browser costs ~5s a page, so spending one on a page that
didn't need it is pure waste.

Freezing the benchmark wasn't enough

Before building the browser layer I froze a set of 100 real URLs — news in two
languages, blogs, docs, newsletters, e-commerce, reference — and committed it.
The set never changes. Only the number moves.

That instinct was right. A moving benchmark measures nothing: when the number
shifts you can't tell whether your code improved or the web moved under you. But
freezing the inputs controls exactly one variable, and I had two others loose.

My own code moved. Static-only scored 78%, static-plus-browser 90%, and I
wanted to call that a 12-point win for the browser layer. Going through the runs
URL by URL says otherwise: of the 22 URLs that failed the first run, six were
rescued by the browser — and six more started passing on the static path,
because the static extractor had been refactored in between. Same URLs, different
code. The browser's honest contribution is six points, not twelve.

I never classified what was in the set. I built it carefully and never asked
what each URL actually was. When I finally labelled them, 40 of the 100 were
site homepages or section indexes — theverge.com/, bbc.com/travel,
etsy.com/. Not articles. A homepage has no article on it, so an article
extractor can neither succeed nor fail on one in any meaningful sense.

That unlabelled 40% is what let the broken success test survive. Navigation text
from a homepage cleared the character bar, counted as a win, and nothing in the
set said "this one isn't supposed to produce an article." The two errors propped
each other up: neither was visible while the other held.

The fix wasn't to change the set — it's frozen, that's the point — but to tag the
homepages and report three numbers instead of one. A single blended percentage
was always going to be wrong in both directions: it treats homepages as
extraction failures while counting their menus as successes.

Spike before you build

Before committing to a browser pool — process management, memory limits, a queue,
a circuit breaker, all of it — I ran a 30-minute spike: take 8 of the 22 URLs
that failed static extraction, throw a plain Playwright at them, and measure
how many come back.

Three of eight. Those three were plain JavaScript rendering. Of the other five,
four were commercial anti-bot and one was a plain nginx 429 rate-limit. That set
the ceiling for the whole sprint: the browser layer's job is the JS-rendering
class, not the anti-bot class. Building the pool was justified; building anti-bot
evasion was not.

Spikes like this are cheap, and they keep you honest about what a feature can
actually deliver before you've sunk a week into it.

How to tell which anti-bot system blocked you

When a page fails, why it failed is diagnosable from the response:

Provider	Signal
DataDome	`x-datadome` header, `datadome` cookie
Cloudflare challenge	`cf-mitigated: challenge` header
Akamai Bot Manager	`AkamaiGHost` server header, `_abck` cookie
PerimeterX / HUMAN	`_px*` cookies

The subtlety that cost me a debugging session: a Cloudflare cookie is not a
Cloudflare block. Most of the web sits behind a CDN, so __cf_bm shows up on
pages that serve their content perfectly. My first detector flagged a Substack
post as "protected" while the rendered page carried 22,924 words of real content
(whole-page text, as the spike harness measured it; the extractor's cleaned
article is 5,310). The fix was to make detection content-first: if the page
returned real content, the anti-bot signal is irrelevant. Only a strong signal
(cf-mitigated: challenge) counts before extraction. Get this wrong and you flag
half the internet.

Why I don't bypass anti-bot

I could have reached for stealth plugins, residential proxies, or
undetected-chromedriver. I didn't, for two reasons.

Honesty of the product: returning ANTIBOT_PROTECTED with the provider name —
"this site uses DataDome, we don't support it" — is more useful to a caller than
a vague timeout. They know exactly where the boundary is.

And scope: bypassing commercial anti-bot is an arms race with real legal and
terms-of-service implications. It is a fundamentally different product from clean
text extraction, and mixing them would compromise both. Seven of the 100 URLs
sit behind commercial anti-bot — two article URLs and five homepages — and they
are excluded from the target on purpose.

An explicit boundary is a feature.

Two bugs the resilience test found

I almost shipped without a resilience test. It found two things that would have
bitten in production, weeks later, with nobody watching.

The lazy watchdog. The browser had a watchdog that restarted it when it died —
but only inside the request path. If the browser died and no request came in, it
stayed dead. Worse, restarting on the request path meant every caller ate the
restart latency and, under load, piled up behind a lock. Now a dead browser is a
fast static fallback for callers and a background job for the watchdog.

The page.content() hang. I had a 15-second timeout on page navigation. I did
not have one on page.content(). One e-commerce page passed navigation and then
hung content() for 120+ seconds — and asyncio.wait_for couldn't cancel it,
because Playwright's call was blocked at the C level below Python's event loop.
The only reliable fix was a hard 35-second ceiling in the request handler itself.
No single page can block the server longer than that.

Both rhyme with the rest of this: timeouts you didn't write are timeouts you
don't have, and a component that recovers "eventually, when poked" doesn't
recover.

The result, honestly

Of the 60 URLs that point at an actual article, 54 return a body of at least
100 words — 90%, or 93% setting aside the two behind anti-bot. The other four:
two extract nothing usable (a Rust book chapter, an Amazon product page) and two
time out.

Of the 40 homepages, 11 correctly return nothing — the right answer, not a
failure — and 19 carry enough real prose to extract anyway. Seven URLs across
the set are behind commercial anti-bot, out of scope by design.

The old headline for the same set was also "90%", and it meant something much
weaker: any extraction at all, across all 100 URLs, counting homepage navigation
as success. Same digits, entirely different claim. The smallest body behind
today's number is 103 words. Behind the old one it was six.

I could chase the anti-bot sites with proxies and stealth and maybe claw back a
few points, at the cost of an arms race, legal exposure, and a murkier product.
I'd rather have a number I can defend line by line than a bigger one I can't —
which is, in the end, the same lesson as the three checks.

Code, benchmark, and the full decision log:
https://github.com/MariusKa01/article-extractor-api · live demo:
curl "https://api.lobbyx.net/v1/extract?url=<URL>"

DEV Community: MariusKa01