DEV Community: Sanjay Chauhan

Your scraper says 200 OK. I measured how often it's lying.

Sanjay Chauhan — Mon, 08 Jun 2026 13:57:28 +0000

You write a scraper. It hits a URL, gets back 200 OK, you check resp.status_code, it is 200, so you call save(resp) and move on. The pipeline runs nightly. Everything is green. You trust it, because the whole point of a status code is that it tells you what happened.

Three days later a downstream report looks subtly wrong. A column is empty, or a count is off. You start the long walk back to find out which page quietly handed you a login form instead of the article you asked for. It was a login wall. Or a JavaScript app-shell with no content rendered yet. Or a soft-404 dressed up as a real page. The status code said success, your code believed it, and the junk got stored as data.

In 2026 a 200 OK is not ground truth. It is just as likely to be an anti-bot challenge page, a login wall, a soft-404, or an empty JavaScript shell that never rendered. Status-code retry logic never notices the difference, so the corruption gets stored as data.

I wanted to see this happen on real, named sites rather than argue it in the abstract, so I went and measured it.

What I found

I took three popular Python fetchers (requests, curl_cffi, scrapling), pointed them at a mix of control sites and protected ones, and ran 3 requests each. Then, this is the part that matters, I captured each raw body and labeled what every fetcher actually got back independently, by reading the stored bytes, not by trusting the status line and not by trusting veriscrape. Only after labeling did I compare. Every result was stable 3 of 3.

A "silent failure" here means one specific thing: a 2xx response whose body is junk (a login wall, a JS app-shell, an empty page) that gets reported as success with no signal that anything is wrong. The cleanest, least-disputable case:

discord.com/app and web.telegram.org return 200 with an empty JavaScript app-shell: a mount point and a wall of scripts, zero server-rendered content. Every status-code-only fetcher (requests, curl_cffi, scrapling) stores that husk as a successful page. The HTML loads, the content does not.

This is a category-wide, structural blind spot, not a knock on any one tool. Any fetcher that decides success from the status line stores that 200 as good data, because a 200 with a skeleton in the body is, by every status-code measure, a success.

The independent labeling earned its keep in an unflattering way, and I am keeping that on the record because it is the whole point. An earlier draft of this writeup reported one cell as a competitor's (scrapling) "silent failure" on g2.com. Re-labeling from the captured body showed that was wrong: it was a veriscrape false positive. The real, content-rich G2 homepage had come back (the anti-bot let the fetch through), and veriscrape had mislabeled it as a login wall. I fixed the detector (that homepage now classifies OK) and retracted the claim. That is the thesis turned on its author: the tool exists to flag silently-wrong data, and the discipline has to apply to its own output first. If it cannot survive that, it has no business judging anyone else's fetch.

Why retry logic cannot see it

Here is the shape of almost every fetch-and-store loop I have ever written or reviewed:

resp = fetcher.get(url)
if resp.status_code == 200:
    save(resp)        # looks fine, ships it
else:
    retry(url)        # only fires on 4xx / 5xx

The branch that matters never runs. A login wall is served with 200. A JS shell is served with 200. A DataDome gate can be served with 200. The status code is doing exactly what it is defined to do (the HTTP transaction succeeded), and your code is reading a meaning into it that was never there. So the if is true, save() runs, and the corruption is now in your store. There is no error, no exception, no log line.

You cannot fix this with more retries, because retrying a 200 login wall just gives you the same 200 login wall, stably, 3 of 3. The only way to catch it is to look at the body, the headers, and the cookies, and decide what the response is.

The fix: get the bytes plus a verdict

That decision is what I built. It is a library called veriscrape: a verified-fetch primitive that returns the bytes plus a portable, deterministic trust verdict, so the moment your data is silently wrong you have a signal at the fetch layer instead of a wrong report three days later.

pip install veriscrape

import veriscrape

r = veriscrape.get("https://discord.com/app")

r.verdict      # 'EMPTY_SHELL'
r.cause        # 'js_app_shell'  (or 'datadome', 'login_wall', 'cloudflare_challenge', ...)
r.confidence   # 0.0 to 1.0
r.evidence     # the exact markers matched, for audit
r.ok           # True ONLY when r.verdict is OK

get() is a drop-in for requests.get. It fetches with curl_cffi (browser-like TLS, so you are not labeled on a TLS signal alone), then runs the deterministic classifier over the response. The verdict is one of:

OK  BLOCKED  CHALLENGE  HONEYPOT  SOFT_404  LOGIN_WALL  EMPTY_SHELL  UNVERIFIED

The taxonomy is the whole point. Instead of one boolean status_code == 200, you get a named reason for what the response actually is.

If you already have a fetch stack, you do not have to replace it. Classify what you already pulled, without re-fetching:

from veriscrape.adapters import from_requests, from_response
# from_requests(resp) for a requests.Response
# from_response(...) for httpx, Playwright, or any stack

There is also a Scrapy middleware (VeriscrapeMiddleware) and a CLI (veriscrape check <url>, exit code 0 for OK or UNVERIFIED, 1 on a problem) for pipelines and CI.

How it works under the hood

It is deterministic. No LLM. The verdict is computed from status, headers, cookies, and body, which means it is reproducible and auditable: r.evidence shows you exactly which markers matched, so you can argue with any verdict.

The core rule I keep coming back to is the two-key rule. A vendor fingerprint alone is not a verdict. Server: cloudflare, a cf-ray header, a _px cookie, an x-kpsdk-* header: all of these show up on perfectly normal allowed pages too. If you treat vendor presence as "challenge," you will flag half the internet. So a real verdict needs two keys: the vendor gate and a challenge-or-block-specific marker on a genuine mitigation response. One key without the other is not a verdict.

Coverage today is 14 negative detectors plus the affirmative OK detector. The negatives are 7 anti-bot vendors (Cloudflare, DataDome, Akamai, PerimeterX/HUMAN, Kasada, Imperva/Incapsula, F5 BIG-IP ASM), 3 CAPTCHA gates (reCAPTCHA, Turnstile, hCaptcha), and honeypot, login-wall, soft-404, and empty-shell. The affirmative OK detector is the one that ships a green light: it is the only path to r.ok being True, and it is deliberately the hardest to earn (more on that below).

Every detector ships allowed-page fixtures: real pages from the same vendors that are not challenges. The test suite fails if any of those fixtures trips a verdict. The whole product is the claim "I will not lie to you," so the false-positive gate is the part I care about most.

The honest caveat

Read this part before you adopt anything.

veriscrape abstains over guessing, and the affirmative OK verdict that ships today is built around that. get() returns a positive OK only for a 200 that is a real document (it has a <title>) with substantial server-rendered visible text, the inverse of an empty shell. Anything short, ambiguous, or disqualified comes back UNVERIFIED, not OK, and r.ok is True only on that affirmative OK.

What that means in practice: a padded soft-404, a paywall teaser, a geo or maintenance or age-gate page, a suspended or error page served as a 200, none of these get blessed; they stay UNVERIFIED. The detector keys on affirmative evidence (real document, substantial server-rendered text) and disqualifies long-but-bad pages, because length alone is not proof of content.

That is on purpose. The design rule is abstain over guess: I would rather return UNVERIFIED than emit a confident-but-wrong OK, because a confident-but-wrong success is the exact failure this whole thing exists to prevent. It is the failure mode that costs you three days, and I am not going to reproduce it inside the tool meant to catch it. UNVERIFIED is a real verdict and it is not ok. If that tradeoff does not fit your pipeline, that is fair, and now you know it up front.

Try it, and please try to break it

pip install veriscrape

Repo: https://github.com/san64777/veriscrape (Apache-2.0, Python 3.12+)
The benchmark, dated, with the raw cases and the correction banner for the retracted claim: https://github.com/san64777/veriscrape/blob/main/benchmark/results-2026-06-07.md
The deeper reasoning (why these detectors, why deterministic, the two-key rule): https://github.com/san64777/veriscrape/blob/main/WHY.md

Reproduce it yourself:

uv run --extra benchmark python -m benchmark.run

What I most want is for you to break it. If you find a false positive (a normal page that gets flagged) or a false negative (junk that comes back UNVERIFIED when a detector should have caught it), open an issue with the URL. The detectors are pure functions of (status, headers, body), so a captured response is enough to reproduce and add as a fixture, and the evidence dict will tell us both exactly which marker tripped. That is the conversation I want to have.

I am Sanjay Chauhan. I build reliability and data-integrity primitives for data pipelines. veriscrape is open source under Apache-2.0: https://github.com/san64777/veriscrape . Reach me at san64777@gmail.com.

acroforge: turn a flat PDF into a real fillable AcroForm, with a deterministic core and zero copyleft

Sanjay Chauhan — Mon, 08 Jun 2026 12:13:30 +0000

You generated a "fillable" PDF. It looks perfect in Chrome. Then a user opens it in Firefox and the checkboxes are blank, the text fields show nothing until clicked, and one field renders its value an inch to the left. You did not change anything. Two PDF viewers just disagreed about what your form means.

That is the first trap with PDF forms: a field is not "filled" until it renders the same way across viewers. The PDF spec lets a viewer either trust the appearance stream you baked in or regenerate its own, and the two paths drift apart constantly. If you do not control the appearance stream, you are at the mercy of whichever engine opens the file.

The second trap is licensing. Reach for the well-known Python PDF tools and you hit a wall fast: the strongest ones are AGPL (viral copyleft you cannot ship inside a closed product without consequences), or they are paid, or they are a cloud API you have to send documents to. For a lot of teams, "send our forms to someone else's server" is a non-starter, and "relicense our whole app" is worse.

I wanted a small library that does one job: take a flat, non-interactive PDF and turn it into a real, fillable AcroForm, deterministically, locally, with a permissive license. That is acroforge.

pip install acroforge

Apache-2.0, Python 3.11+, no network calls, no AI, no cloud.

The 5-line core

Three functions. They all take bytes and return bytes, so they compose into any pipeline.

import acroforge as af
from acroforge import FieldSpec, FieldType

fields = [FieldSpec(type=FieldType.TEXT, page=0, rect=(200, 700, 450, 730), name="full_name")]
fillable = af.build(flat_pdf_bytes, fields)          # inject real AcroForm fields
filled   = af.fill(fillable, {"full_name": "Jane Doe"})  # set values by name
final    = af.flatten(filled)                         # bake appearances, lock the form

build injects standards-compliant interactive fields at the coordinates you specify.
fill sets field values by name.
flatten bakes the field appearances into the page content, removes interactivity, and locks the result. The flattened PDF is the one you hand to anyone, because there is no interactive layer left to render differently.

Describing fields

A FieldSpec is a plain pydantic model:

class FieldSpec(BaseModel):
    type: FieldType
    page: int                                  # 0-indexed
    rect: tuple[float, float, float, float]    # (x0, y0, x1, y1) in PDF points
    name: str                                  # AcroForm field name
    options: list[str] | None = None           # radio group members
    maxlen: int | None = None                  # TEXT cap / COMB cell count
    export_value: str | None = None            # checkbox/radio on-value
    confidence: float = 1.0                    # 1.0 = you authored it

The field types cover the forms people actually fill:

TEXT single-line text, optional maxlen to cap length.
COMB the segmented box style, where maxlen is the number of cells. An SSN field is maxlen=9.
CHECKBOX with an export_value for the on-state (default "Yes").
RADIO one FieldSpec per button, sharing a name, each with its own export_value.
SIGNATURE a placeholder signature widget.

Here is text plus a checkbox plus a radio group:

fields = [
    FieldSpec(type=FieldType.TEXT, page=0, rect=(200, 700, 450, 730), name="full_name"),
    FieldSpec(type=FieldType.COMB, page=0, rect=(200, 660, 360, 690), name="ssn", maxlen=9),
    FieldSpec(type=FieldType.CHECKBOX, page=0, rect=(200, 620, 220, 640), name="agree", export_value="Yes"),
    FieldSpec(type=FieldType.RADIO, page=0, rect=(200, 580, 220, 600), name="plan", export_value="basic"),
    FieldSpec(type=FieldType.RADIO, page=0, rect=(260, 580, 280, 600), name="plan", export_value="pro"),
]
fillable = af.build(flat_pdf_bytes, fields)
final = af.flatten(af.fill(fillable, {"full_name": "Jane Doe", "ssn": "123456789", "agree": True, "plan": "pro"}))

The honest two-layer design

acroforge is deliberately split into two layers, and the split is the whole point.

Layer one is the deterministic engine: build, fill, flatten. You tell it exactly where fields go, and it puts them there, fills them, and flattens them reliably. This works on any PDF, vector or scanned, because it does not need to understand the document. It only needs coordinates. This is the part you should depend on.

Layer two is a best-effort detector, and it is labeled best-effort everywhere on purpose. af.detect(pdf) reads the PDF's vector geometry and nearby text labels and guesses where fields belong, returning a FormManifest where every field carries confidence < 1.0 to flag it as a guess. af.make_fillable(pdf) runs detect then build in one step.

manifest = af.detect(pdf)            # guesses, each confidence < 1.0
for f in manifest.fields:
    print(f.type, f.name, f.rect, f.confidence)

fillable = af.make_fillable(pdf)     # detect() then build(), one call

The detector handles underline-style forms (write-on rules become text fields), bordered table and grid forms (cells become text fields, label-aware), vector checkbox squares, and font-glyph checkboxes like the box and check characters. It is vector-only: scanned or image-only PDFs are refused with ScannedPDFError, because there is no OCR and I would rather refuse than pretend.

I make no accuracy promises about detection. It will miss fields and invent spurious ones, and quality varies wildly by document. The intended workflow is: run detect, review the draft manifest, correct it, and hand the corrected specs to the engine. The fuzzy layer bootstraps you; the deterministic layer is what ships. Keeping that boundary sharp is what stops the guessing from contaminating the guaranteed path.

Cross-viewer correctness is the contract

Back to the first trap. The way acroforge avoids the Chrome-vs-Firefox split is by treating cross-viewer rendering as the actual test contract, not an afterthought.

Every field type has golden-image render tests in two engines: pdfium, which is what Chrome uses, and pdf.js, which is what Firefox uses. A change that makes a field render differently in either engine fails CI. Adobe Reader is a manual spot-check on top of that. The rule I hold the library to is simple: a field does not "work" until it renders correctly across viewers, so I test before claiming it.

It has been exercised on real documents, including IRS forms W-9 and 1040 and a 43-page credentialing packet, which is where comb fields, duplicate field names, and multi-page layouts stop being theoretical.

The zero-copyleft story

The runtime dependency tree is strictly permissive, by design and by enforcement:

Package	License	Role
reportlab	BSD	field widget rendering
pypdf	BSD-3-Clause	PDF read / merge / flatten
pdfplumber	MIT	geometry utilities
PyPDFForm	MIT	fill helpers
pydantic	MIT	model validation

No GPL, AGPL, LGPL, or SSPL anywhere in the runtime tree, not even MPL. This is not a promise I am asking you to take on faith. CI enforces it on every push with pip-licenses --fail-on='GPL;AGPL;LGPL;SSPL', so a copyleft dependency cannot sneak in through a transitive bump without turning the build red. You can drop acroforge into a commercial product without a licensing conversation.

read_fields: the inverse of build

read_fields(pdf) ingests the AcroForm fields already in a fillable PDF back into FieldSpecs (real registered fields, so confidence = 1.0). It is the exact inverse of build, which means the two round-trip:

specs = af.read_fields(open("fillable.pdf", "rb").read())   # -> list[FieldSpec]

# copy one form's field layout onto another PDF
af.build(other_pdf, af.read_fields(template_pdf))

You get one FieldSpec per widget with coordinates, type, name, and checkbox/radio on-states recovered. Dropdowns come back as text; pushbuttons are skipped. It is handy for inspecting an existing form, diffing layouts, or lifting a known-good field arrangement onto a new document.

Try it and tell me where it breaks

acroforge 0.2.0 is live on PyPI and the source is on GitHub. It is small, focused, and meant to stay that way.

Repo: https://github.com/san64777/acroforge
PyPI: https://pypi.org/project/acroforge/

If you run it against a form that renders wrong in some viewer, or a layout the detector mangles, that is exactly the report I want. Open an issue with the PDF (or a reproducer) and the viewer. And if it saves you from an AGPL dependency or a cloud round-trip, a star helps other people find it.

pip install acroforge