DEV Community: Gani Mendoza

The Hidden Machinery of Quantum Reality

Gani Mendoza — Sun, 21 Jun 2026 15:20:54 +0000

Why Bohmian Mechanics, Go Programs, AI, and EBP 2.1 Could Help Reopen the Deepest Questions in Physics

There is a quiet crisis in theoretical physics, and it has nothing to do with the equations.

The equations are fine. Quantum mechanics predicts with staggering accuracy. General relativity bends light exactly as calculated. The Standard Model matches experiment after experiment. The mathematics is not the problem.

The problem is what happens between the equations and the claims.

A researcher writes a beautiful paper. The math is correct. The toy model works. A suggestive ratio appears. An analogy crystallizes. And then, in the discussion section, a modest result becomes a bold narrative: classical cosmology is recovered, the problem of time is resolved, spacetime emerges from the quantum.

This is not fraud. It is not even intentional. It is the natural gravity of theoretical work — ideas fall toward overclaiming the way matter falls toward mass.

Two small codebases, written in Go and governed by an epistemic protocol called Elephant Bridge Protocol v2.1, are trying to build a tool against that gravity. They are not trying to solve quantum gravity. They are trying to make it harder to pretend you have solved quantum gravity when you haven't.

One is called Bell–MIPT. It builds toy models connecting Bohmian mechanics to measurement-induced phase transitions in many-body quantum systems.

The other is called BMC — Bohmian Minisuperspace Cosmology. It builds toy models of quantum cosmology using Bohmian guidance in Wheeler–DeWitt minisuperspace.

They share the same philosophy. They share the same protocol. And they share the same radical commitment: no claim may be promoted until its debts are paid.

What Is BMC?

BMC stands for Bohmian Minisuperspace Cosmology. The name is deliberately modest. It is not "Bohmian Quantum Gravity." It is not "The Theory of Everything in Go." It is a cosmology toy model — a wind tunnel, not an airplane.

The physics idea behind it is old and deep.

In quantum cosmology, the universe itself is described by a wavefunction. The Wheeler–DeWitt equation is the quantum constraint that this wavefunction must satisfy — roughly, the quantum version of Einstein's field equations applied to the universe as a whole. But the Wheeler–DeWitt equation has no time variable. The universe, quantum mechanically, is timeless.

This creates a profound puzzle: if the fundamental equation has no time, where does the time we experience come from? How do clocks emerge? How does the classical expanding universe — the Friedmann cosmology of big-bang nucleosynthesis, cosmic microwave background, and accelerating expansion — arise from a static quantum state?

Bohmian mechanics offers one possible answer. Instead of treating the wavefunction as the final word, Bohmian mechanics says there is also an actual configuration of the world. The wavefunction guides this configuration through a velocity law. In quantum cosmology, the configuration is the scale factor of the universe and a matter field. The wavefunction lives on a "minisuperspace" — a drastically simplified version of the full space of all possible geometries.

BMC implements this idea as executable Go code.

The configuration variables are:

α = ln(a)     — the log scale factor
φ             — a homogeneous scalar field

The wavefunction satisfies a toy Wheeler–DeWitt equation:

(-∂²/∂α² + ∂²/∂φ²) Ψ(α, φ) = 0

The Bohmian guidance law says the actual trajectory through minisuperspace follows the phase gradient of the wavefunction:

dα/dλ = ∂S/∂α
dφ/dλ = -∂S/∂φ

where Ψ = R exp(iS).

The quantum potential — the signature Bohmian correction to classical behavior — is:

Q = -1/(2R) (∂²R/∂α² - ∂²R/∂φ²)

BMC computes all of this numerically, generates deterministic JSON reports, and then asks the hardest question: does any of this actually mean what we hope it means?

What Is Bell–MIPT?

The sibling project, Bell–MIPT, asks a different but related question.

John Bell — the physicist who proved that quantum mechanics cannot be explained by local hidden variables — also proposed models for quantum field theory in which actual configurations on a lattice undergo stochastic jumps guided by the universal wavefunction. These are called Bell-type quantum field theories.

Recently, the physics community has become fascinated by measurement-induced phase transitions (MIPT): when you monitor a quantum system, there is a competition between unitary dynamics (which spreads entanglement) and measurement (which suppresses it). At a critical measurement rate, the system undergoes a phase transition between high-entanglement and low-entanglement regimes.

Bell–MIPT asks: if measurement is not fundamental in Bohmian mechanics, could something like measurement-induced dynamics emerge from Bell-type jumps conditioned on the actual environment configuration?

The project builds finite fermionic lattice models, computes Bell jump rates, samples configuration trajectories, and constructs environment-projected conditional vectors. It then checks whether strict environment jumps are associated with large conditional-vector changes — a diagnostic that would suggest a structural similarity between Bell conditioning and monitored quantum dynamics.

The early toy results show a strong signal: environment jumps correlate with fidelity drops in the conditional vector. But the project is ruthlessly honest about what this does not mean:

The toy found a strong environment-correlated conditional-vector update diagnostic.
It did not establish MIPT.
It did not prove a Bell–MIPT bridge.
It did not show Bell jumps are measurements.

That distinction — between a promising diagnostic and a proven result — is the entire point of both projects.

The Architecture: Go as a Laboratory Instrument

Both projects are written in Go. This is not the obvious choice for physics. Python has NumPy, SciPy, and QuTiP. Julia is designed for numerical computing. C++ is the workhorse of high-performance simulation. Mathematica excels at symbolic work.

Go is chosen for a different reason: epistemic safety.

Go compiles quickly, has a small language surface, makes concurrency practical, and produces readable code months later. It does not allow the kind of notebook folklore that accumulates in Python — cells executed out of order, hidden state, environment drift, and results that are hard to reproduce.

A Go program can be treated like a laboratory instrument. It has inputs, outputs, tests, reports, and reproducible behavior. It does not hallucinate. It does not improvise. It does not wake up one morning and decide that a suggestive ratio means the secrets of the universe have been solved.

BMC's Go codebase is organized as a proper research instrument:

cmd/ptw-bmc/main.go           — CLI entry point with 12 subcommands
internal/bmc/model/            — physics model parameters and types
internal/bmc/wave/             — wavefunction evaluation (plane waves, superpositions)
internal/bmc/wdw/              — Wheeler–DeWitt residual checking
internal/bmc/guidance/         — Bohmian velocity and trajectory integration
internal/bmc/qpotential/       — quantum potential computation
internal/bmc/invariant/        — classical-limit recovery checks
internal/bmc/obstruction/      — node detection, clock failure, Q divergence
internal/bmc/nullrun/          — null-model execution
internal/bmc/nullspec/         — null-model specification
internal/bmc/residualrun/      — candidate Friedmann residual computation
internal/bmc/residualaudit/    — residual vs. null comparison audit
internal/bmc/clockdiag/        — clock monotonicity fragility investigation
internal/bmc/clockseg/         — local clock segmentation
internal/bmc/priorart/         — literature and prior-art boundary checks
internal/bmc/audit/            — numerical robustness and convergence auditing
internal/bmc/report/           — deterministic JSON report generation and validation

Each package does one thing. Each subcommand produces a JSON report. Each report is validated against a strict schema. The entire pipeline is deterministic: same inputs, same seed, same report.

The CLI itself reveals the project's priorities:

ptw-bmc run --profile bmc0a-plane --out out/bmc0a_plane.json
ptw-bmc validate --report out/bmc0a_plane.json
ptw-bmc summarize --report out/bmc0a_plane.json
ptw-bmc audit --out out/bmc0a_superposition_robustness.json
ptw-bmc diagnose-clock --out out/bmc0a_clock_fragility.json
ptw-bmc segment-clock --out out/bmc0a_clock_readiness.json
ptw-bmc run-nullmodels --out out/bmc0a_nullrun.json
ptw-bmc run-residuals --out out/bmc0a_local_residual.json
ptw-bmc audit-residuals --out out/bmc0a_residual_audit.json
ptw-bmc prior-art-boundary --out out/bmc0a_prior_art_boundary.json

Notice the verbs: run, validate, audit, diagnose, segment. This is not a physics simulator pretending to be a theory. It is a diagnostic bench that knows its own limitations.

Why EBP 2.1 Is the Heart of Everything

If Go is the skeleton of these projects, Elephant Bridge Protocol v2.1 is their conscience.

EBP 2.1 is an epistemic governance protocol. Its doctrine is seven sentences:

Ideas enter free.
Promotion costs debt.
Debt does not kill.
Debt is forever payable.
New evidence creates new debt.
No final-truth claim may be promoted.
Accounting must never become the work.

Or, even shorter:

Door: wide open.
Throne: guarded by debt.

The protocol exists to protect two things simultaneously. First, the theorist's imagination: any idea may enter as a one-line claim, with no map, no invariant, no Lean formalization, no null model, no review required. Second, the search's epistemic integrity: no idea may be promoted until its unpaid obligations are visible and sufficiently retired.

How Debt Works

When an idea enters, it automatically carries six debts:

Debt	What It Asks
`needMap`	State your domain, codomain, and translation rule
`needInvariant`	State what quantity survives your proposed bridge
`needToyCheck`	Run a finite test — toy success doesn't prove reality, but toy failure kills bad structure early
`needNullModel`	Can a simpler or rival model explain the same thing?
`needObstruction`	File known blockers, no-go results, contradiction risks
`needFaithfulnessReview`	Does the formalization actually match the intended claim?

Debt doesn't kill an idea. It only blocks promotion. An idea from 2024 can sit dormant until 2029, then be repaired and promoted. There is no expiration date and no shame in incompleteness.

But — and this is the crucial move — a promoted idea can become *un*promoted if new evidence creates new debt. A new obstruction, a stronger null model, a faithfulness failure: any of these reinstates debt. The idea remains alive, but promotion is suspended.

Why This Matters for AI-Assisted Research

EBP 2.1 was not designed in the abstract. It was designed because the projects use AI coding agents as collaborators. And AI creates a specific, dangerous new failure mode:

AI can generate theories faster than humans can falsify them.

A language model can write an elegant explanation of something that is not yet true. It can make a toy model sound like a bridge, a bridge sound like a theorem, and a theorem sketch sound like a revolution. It can produce paragraphs that look more mature than the evidence beneath them.

EBP 2.1 is the counterweight. Every claim an AI helps articulate still owes the same debts. Every suggestive result still needs the same null models. Every beautiful narrative still faces the same faithfulness review.

The research loop becomes:

AI proposes.
Go tests.
AI reviews.
Go reports.
Humans judge.
EBP 2.1 prevents promotion until debt is paid.

This is not "AI replaces physicists." It is "AI expands the search space, Go stabilizes the evidence, and EBP keeps the claims honest."

How BMC Uses EBP 2.1 Concretely

BMC doesn't just reference EBP 2.1 in a README. It implements EBP 2.1 in code.

Every JSON report contains an explicit debt ledger:

{
  "ebp_debt": {
    "needMap": "partial",
    "needInvariant": "partial",
    "needToyCheck": "active",
    "needNullModel": "partial",
    "needObstruction": "active",
    "needFaithfulnessReview": "active"
  },
  "promotion_recommendation": "blocked",
  "final_truth_claim": false,
  "toy_analysis_only": true,
  "physics_claim": "minisuperspace_only"
}

The validate subcommand enforces EBP constraints programmatically. If a report claims final_truth_claim: true, validation fails. If the Friedmann residual check is deferred but the promotion gate isn't blocked, validation fails. If faithfulness is contested but promotion is allowed, validation fails.

The project even includes Lean formal contracts — theorem obligations that prove a passing toy report cannot imply full quantum gravity:

theorem no_full_qg_claim_from_toy_report
  (r : BMCReport) :
  reportPassesToyGate r = true ->
  r.toyAnalysisOnly = true := by
  intro h
  simp [reportPassesToyGate] at h
  exact h.left

This theorem is deliberately humble. It proves only that a passing toy report remains toy-only. But that humility is the point.

What BMC Actually Found

BMC progressed through eleven sprints, each building one layer of the diagnostic bench.

Sprints 1–3 implemented the plane-wave control case and superposition profiles. The plane wave (Ψ = exp(i(kα + ωφ))) is the simplest possible wavefunction satisfying the Wheeler–DeWitt constraint. For this state, the quantum potential is exactly zero, the trajectory is a straight line, and everything is classical. This is the regression test — the baseline that must pass before anything interesting is attempted.

The superposition case (Ψ = c₁ exp(i(k₁α + ω₁φ)) + c₂ exp(i(k₂α + ω₂φ))) is where things get interesting. Nodes appear — points where the wavefunction vanishes and the phase becomes undefined. The quantum potential diverges. Trajectories can hit these singularities and stop. The code detects all of this and reports it honestly.

Sprints 4–5 added robustness auditing: convergence analysis, threshold sensitivity, step-size variation, and numerical stability checks.

Sprint 6 specified the Friedmann residual — the diagnostic that would test whether the Bohmian trajectory recovers classical cosmological behavior. Critically, this sprint specified the residual but did not compute it, because the null-model infrastructure didn't exist yet.

Sprint 7 built the null-model scaffold. This is where BMC departs from most toy-model projects. Before computing the Friedmann residual, BMC demanded that null models be defined and ready to run in parallel. The null models include:

Classical FRW: Does the Bohmian trajectory simply reproduce the classical result?
Standard Wheeler–DeWitt: Does adding Bohmian ontology change anything operationally?
Loop Quantum Cosmology correction: Does the quantum potential mimic an LQC-style bounce?
Page-Wootters relational time: Does Bohmian cosmology produce different predictions from other relational-time approaches?

Sprint 8 performed a literature and prior-art audit. The project discovered, unsurprisingly, that Bohmian quantum cosmology in minisuperspace is established territory. Pinto-Neto, Struyve, Fabris, and others have published extensively on Bohmian Wheeler–DeWitt models, trajectories, singularity avoidance, and classical limits. BMC's honest conclusion:

Claim: Bohmian minisuperspace cosmology has been done before.
Status: supported by literature search.

Claim: Our exact EBP-gated software pipeline has been done before.
Status: unknown; needs prior-art audit.

Claim: The toy model is scientifically novel.
Status: not established; do not claim.

Sprints 9–11 built the null-model runner, the candidate local-branch residual runner, and the residual/null comparison audit.

And perhaps the most important finding came from the clock diagnostics: the scalar field φ can fail as a global monotonic clock, even when the trajectory itself remains numerically valid. This is a genuine insight about the structure of relational time in quantum cosmology. A trajectory can be perfectly well-defined while the chosen clock is not globally usable. The model taught its builders caution.

Two Experiments, One Spirit

Bell–MIPT and BMC are two experiments of the same spirit. They ask different physics questions in different domains:

	Bell–MIPT	BMC
Domain	Many-body quantum systems	Quantum cosmology
Physics	Bell-type jumps, entanglement, MIPT	Wheeler–DeWitt, Bohmian guidance, quantum potential
Key question	Can measurement-like dynamics emerge from Bohmian conditioning?	Can classical cosmology emerge from Bohmian guidance in a timeless wavefunction?
Language	Go	Go
Protocol	EBP 2.1	EBP 2.1

But they share the same DNA:

Bohmian mechanics as the ontological starting point. Both take seriously the idea that quantum systems have actual configurations, not just probability distributions. Both ask whether taking that idea seriously leads to new computational and conceptual tools.
Go as the research instrument. Both use Go not because it is the best language for numerical physics, but because it makes the physics visible. The code is the experiment. The JSON report is the lab notebook.
EBP 2.1 as the epistemic immune system. Both use the protocol to prevent the natural tendency of toy models to become grand narratives. Both track debts explicitly. Both require null models before interpreting results. Both block promotion until faithfulness is reviewed.
AI as adversarial collaborator, not authority. Both use AI language models to propose, critique, and review — but never to promote. The AI helps find gaps. The Go code fills them or reports failure. The human judges.
Honesty as the primary output. The most valuable result from both projects is not a positive finding but a diagnostic framework. They tell you where the structure is robust, where it is fragile, and where the debts live.

The Elephant Bridge

The name "Elephant Bridge Protocol" comes from the parable of the blind men and the elephant. Each physicist touches a different part of quantum reality:

Copenhagen touches the operational face: experiments, probabilities, detector clicks.
Bohmian mechanics touches the ontological face: actual configurations, definite trajectories, hidden guidance.
Quantum information touches the entanglement face.
Statistical mechanics touches the phase-transition face.
Quantum field theory touches the creation-and-annihilation face.
General relativity touches the geometric face.

The protocol does not worship one theory. It extracts what each theory touches correctly, then tests whether the pieces can be made to fit. The "bridge" is the claim that two faces touch the same underlying structure. The "elephant" is the unknown whole.

But — and this is where EBP 2.1 earns its keep — a bridge is not a proof. A bridge is a claim, and claims carry debt. You must specify the map. You must name the invariant. You must run the toy check. You must file the null models. You must confront the obstructions. You must face the faithfulness review.

Only then may the bridge be promoted. And even then, it is not truth. It is simply mature enough to carry forward.

The Significance of EBP 2.1

EBP 2.1 matters beyond these two codebases. It matters for anyone working at the intersection of AI and fundamental research.

We are entering an era where AI can generate theoretical physics papers faster than the community can review them. Where language models can produce plausible-sounding arguments for almost any position. Where the bottleneck in science is shifting from generating ideas to evaluating them honestly.

EBP 2.1 is one attempt at an answer. Not a bureaucratic answer — the protocol explicitly says "accounting must never become the work." Not a gatekeeping answer — the door is wide open to any idea. But a conscience answer: every idea is welcome, but every claim must earn its status.

The seven properties that make EBP 2.1 distinctive:

Ideas enter free. No premature formalism. No Lean requirement. No review committee. Write your idea in one sentence and it is alive.
Debt blocks promotion, not life. An unpaid idea is not a dead idea. It is an unfinished idea. The difference matters.
Debt is forever payable. There is no shame in dormancy. An idea from years ago can be repaired and promoted when new tools or evidence arrive.
New evidence creates new debt. Promotion is not permanent. A new obstruction or stronger null model can un-promote an idea without killing it.
Final-truth claims are structurally blocked. Not by social norms or editorial policy, but by validation code. The system will not accept a report that claims to solve quantum gravity.
Lean formalizes promotion, not entry. Lean is not the door through which ideas must pass. It is the strongest instrument for retiring debt once an idea is mature enough to formalize.
The protocol is anti-fragile. Objections, counterexamples, null models, faithfulness failures — all of these create clearer debt. The system gets stronger when attacked.

What Comes Next

Neither project claims victory. That is the point.

BMC's current status is honest:

BMC-0A completed as a bounded toy benchmark.
It shows:
1. Plane-wave controls behave as expected.
2. Superposition creates nontrivial Bohmian trajectories.
3. Nodes and near-node regions are real obstructions.
4. φ is not always a valid global clock.
5. Local relational branches can be segmented.
6. Some local branches are smooth enough for candidate residual specification.
7. Null models are required before interpreting any Friedmann-like behavior.
8. Full BMC remains unpromoted unless stronger gates are paid.

Bell–MIPT's current status is equally honest:

The toy found a strong environment-correlated conditional-vector update diagnostic.
It did not establish MIPT.
It did not prove a Bell–MIPT bridge.
It did not show Bell jumps are measurements.

The next steps for both projects are not grand leaps. They are small, disciplined moves:

Repair current reports with adversarial feedback.
Harden null models.
Run multi-seed and multi-partition robustness checks.
Compute entanglement observables.
Perform matched monitored-dynamics comparisons (Bell–MIPT).
Compute carefully bounded local-branch Friedmann residuals (BMC).

The question is not: can we make the bridge look true?

The question is: can we make it hard for the bridge to survive — and see if it still does?

Why This Matters

Physics has always advanced through strange alliances. Geometry and gravity. Heat and atoms. Symmetry and particles. Information and black holes. Computation and quantum matter.

Perhaps the next alliance will be between ontology, software, and AI.

Bohmian mechanics brings the courage to ask what exists.

Go brings the discipline to make small machines that do not lie.

AI brings the breadth to explore, critique, and connect.

EBP 2.1 brings the guardrails that prevent imagination from pretending it has already won.

Together, they offer a new way to work on old questions. Not by pretending that code can replace physics. Not by letting AI promote speculation. Not by mistaking a toy model for a theory of the universe. But by building executable fragments of understanding — one model, one null test, one adversarial review at a time.

The world described by quantum mechanics is strange. Bohmian mechanics says it may still be real in a deeper, more concrete sense than the textbooks admit. Quantum cosmology says the universe itself might be described by a timeless wavefunction. And somewhere between the equations and the claims, between the door and the throne, there is space for small programs that hold the line between imagination and honesty.

That space is what these projects are building.

It deserves tools.

It deserves tests.

It deserves engineers.

And it deserves a conscience strong enough to let imagination run without letting it lie.

Web Scraping is a Contract

Gani Mendoza — Mon, 01 Jun 2026 18:04:34 +0000

Pithom Labs Scraper introduces a systematic approach to web scraping that treats data extraction as a binding contract rather than a fragile script. Traditional scrapers often fail silently by ingesting corrupted or empty data when website layouts inevitably change. To solve this, we present a specialized engine that utilizes human-guided discovery to establish a baseline of "truth" for a webpage's structure. This baseline, or GoldenSeal, allows the machine to perform runtime assertions and halt execution immediately if the site's data density or lineage shifts. By prioritizing loud failure and forensic evidence over quiet errors, the system ensures that automated pipelines never compromise data integrity. This methodology shifts the focus from evading bot detection to maintaining structural rigor in a constantly evolving digital environment.

Reprint from Medium

Let’s say the quiet part out loud: web scraping is usually held together with hope, CSS selectors, and a cron job that nobody on your engineering team wants to touch.

You build the parser. You map the fields. You run the script. You get a clean CSV or a pristine JSON array, and for a brief, shining moment, you feel invincible. You have conquered the unstructured internet.

And then, inevitably, the site changes.

It rarely breaks in a way that causes your script to crash and burn spectacularly. If it threw a loud, stack-tracing panic, you could fix it. Instead, a React component gets wrapped in three new div tags. A list hydrates half a second later than usual. A "Next" button moves into a different semantic container. A login session quietly expires in the background.

The data pipeline doesn’t explode. It does something infinitely worse: it keeps running. It keeps executing the same obsolete selectors against a mutated DOM. It happily writes empty strings or completely wrong text into your database. Downstream, your analytics dashboard or machine learning model is confidently eating nonsense, manufacturing false confidence at scale.

That is the part of scraping that we chronically understate. The hard problem isn’t figuring out how to extract data once. The hard problem is knowing when the web has shifted under your feet.

Most scraping tools respond to this reality with a brutal arms race. They throw more proxies, more remote headless browser farms, more fingerprint patches, and more opaque infrastructure at the problem, trying to convince the modern web that a faceless machine in a Virginia data center is actually a human being.

At Pithom Labs, we took a different route with our Go-based scraper engine. We stopped treating web scraping like a document parsing exercise, and started treating it like a typed contract with runtime assertions.

If the web is a moving target, your scraper shouldn’t pretend it’s static. It should fail loudly, produce evidence, and refuse to lie to you.

The Optimism of the Modern Scraper

Classic scrapers are optimistic little machines. They operate under a set of foundational assumptions that simply do not map to the reality of the modern internet.

They assume the page will load exactly the same way every time. They assume the selector that worked yesterday will work tomorrow. They assume that if the network request returns an HTTP 200 OK, the payload is probably meaningful.

But websites are not static documents anymore. They are moving, reactive, personalized, occasionally hostile application surfaces. They hydrate dynamically. They load content via asynchronous GraphQL calls. They lazy-load images. They A/B test their layouts. They change their entire markup structure because a frontend engineer decided to refactor a component library on a Tuesday afternoon.

When you point a traditional Python or Node.js script at this environment, you are essentially firing a blindfolded arrow and hoping the target hasn't moved. When the target does move, the script blindly extracts whatever happens to be occupying that coordinate space.

We realized that to fix this, we had to change the fundamental relationship between the scraper and the web page. We couldn't just build a better DOM parser; we had to build a system that understands what it's supposed to be looking at, and aggressively verifies that reality before it writes a single byte of data to disk.

The Baton Pass: Decoupling Discovery from Extraction

A lot of scraping products want to abstract the web away from you. They offer hosted dashboards, remote browser fleets, and managed extraction APIs. This can be useful, but it creates a massive trust problem. You have to hand over your credentials, try to replicate complex browser states on remote machines, debug someone else's infrastructure, and hope the target site doesn’t trigger a Cloudflare CAPTCHA that your headless script has no physical way of solving.

We designed the Pithom Labs Scraper around a radically different philosophy: The desktop is not a limitation. It is the point.

We built the architecture as a strict two-stage, decoupled system. We call the transition between these two stages the Baton Pass.

Stage 1: Human-Guided Discovery

In the first stage, you aren't writing code. You invoke scraper discover from your terminal, which launches a highly visible, headed instance of Google Chrome running directly on your machine.

Because it’s a real browser running locally, you can log in naturally. You can solve the CAPTCHA. You can click past the cookie consent banner. You establish the authorized session exactly as a human user would.

Once you are on the target page, our Omni-Agent Discovery overlay injects into the browser. You visually click the elements you want—titles, prices, detail links, pagination buttons.

Behind the scenes, the scraper isn't just recording dumb CSS paths. It is generating two critical artifacts:

session.json: A durable record of your exact browser cookies, User-Agent, and authentication state.
intent.json: A declarative recipe containing CSS/XPath selectors, semantic hints, structural hashes, and pagination logic.

Stage 2: Headless Extraction

Once you save the intent, the Baton Pass occurs. The human steps away, and the programmatic engine takes over.

You run scraper scrape, and the Go-based engine boots up in headless mode. It reads the session.json to perfectly spoof the authorized user state. It spins up a concurrent render pool using a stealth engine we call Ghost-Walker (which manages Chromedp under the hood to bypass headless detection and preserve JavaScript context).

This decoupling solves the hardest part of scraping—authentication and anti-bot mitigation—by letting a human handle the hard part once, and letting the machine handle the repetition.

But more importantly, the intent.json generated during Stage 1 isn't just a list of selectors. It is a binding contract.

Extraction as a Contract

In traditional software engineering, we use types, interfaces, and assertions to guarantee that our data is shaped correctly. If a function expects an integer and receives a string, it panics. It fails loudly.

Web scraping rarely has this luxury. Because the DOM is fundamentally untyped and fluid, scrapers have historically relied on "vibes-based" extraction. If .product-title > h2 exists, grab it. If it doesn't, write null and keep moving.

We wanted to bring systems-level rigor to DOM extraction. To do this, the intent.json acts as an executable agreement between the discovery phase and the runtime engine.

The GoldenSeal

When you finish Stage 1 discovery, the engine computes something we call the GoldenSeal.

The GoldenSeal is a structural fingerprint of the page at the exact moment you taught the scraper how to read it. It lives at the bottom of your intent.json and looks something like this:

"golden_seal": {
  "sealed_at": "2026-05-29T12:00:00Z",
  "row_count": 20,
  "structural_hash": "sha256:d8e3ab03bc",
  "field_population": {
    "title": 1.0,
    "detail_url": 1.0,
    "description": 1.0,
    "price": 0.95
  }
}

This isn't just metadata. The GoldenSeal establishes the baseline reality of the website. It says: "When the human was looking at this page, there were exactly 20 items. The 'title' field was populated 100% of the time, and the 'price' field was populated 95% of the time."

During headless execution, the engine constantly measures the live DOM against this seal by enforcing Integrity Invariants.

The Density Invariant

The scraper expects each paginated list to maintain a consistent density. If the GoldenSeal expects 20 items per page, and the live execution suddenly extracts 0 items, or 3 items, the engine knows something is wrong.

Traditional scrapers would happily write those 3 items to a CSV and move on to the next page. Our engine trips the Density Invariant. It halts execution immediately, recognizing that either the page hasn't fully hydrated yet (Skeleton DOM), or the site layout has radically changed.

The Lineage Invariant

Even if the scraper finds the correct number of rows, the individual selectors might have drifted. The Lineage Invariant compares runtime field fill-rates against the GoldenSeal.

If the title field was populated 100% of the time during discovery, but during runtime it is only populating 10% of the time, the Lineage Invariant fails. The engine recognizes that it is experiencing Structural Drift. It refuses to continue writing empty columns.

Shift-Left QA: Validating Before We Commit

In a data pipeline, corrupted data is vastly more expensive to fix after it has been written to disk or ingested into a data warehouse. You want to catch the error as far upstream as possible.

To enforce the contract, the Pithom Labs Scraper implements a mechanism we call Shift-Left QA.

When the headless engine begins extracting data from the first page, it does not immediately stream those rows into your output CSV or JSON file. Instead, it buffers the first 5 rows in memory.

It runs these buffered rows through a gauntlet of semantic validations. It checks the Invariants. It verifies that required fields are present. If the site requires clicking into "Detail Pages" for deeper data, it ensures that the detail URLs aren't throwing 404s and that the deep extraction isn't returning blank text (enforcing the detail_skip_tolerance).

If the QA Buffer detects a critical failure—if all the fields are empty, or the data has fundamentally shifted—the run is aborted before a single byte of garbage data touches your output file.

Instead of writing bad data faster, the system stops, records the evidence, and generates a diagnostic bundle.

Failing Loudly: Evidence over Magic

The scariest scraper isn't the one that crashes. The scariest scraper is the one that fails quietly.

When the Pithom Labs Scraper breaks a contract and halts, it doesn't just log a generic error and die. It produces evidence.

It exits with strict, semantic CLI exit codes that programmatic supervisors (like cron jobs or CI/CD pipelines) can actually understand and route:

Exit Code 0: Success. The contract was upheld.
Exit Code 3: Structural Drift. The layout changed or the Density Invariant failed.
Exit Code 4: Integrity Failure. Data quality dropped below tolerance (e.g., detail pages are failing to load).
Exit Code 42: Auth Required. The site returned a 401/403 or redirected to a login screen. The session cookies are dead.

More importantly, upon a critical failure, the engine generates a timestamped diagnostics_YYYYMMDD_HHMMSS/ forensic bundle.

This bundle contains scrape_failure.jsonl (the exact structured events leading up to the crash) and, crucially, failure_snapshot.html—a complete, redacted snapshot of the DOM at the exact moment the scraper realized it was looking at an alien landscape.

You don't have to guess why the scraper failed. You don't have to write custom scripts to reproduce the error. You open the diagnostic snapshot, and you see exactly what the scraper saw: a Cloudflare challenge, a new A/B tested layout, or an expired login redirect.

Engineering for a Hostile Environment

Web scraping is, by definition, the act of writing highly coupled code against an unversioned API that you do not control, built by people who often actively do not want you to be there. It is a uniquely hostile engineering environment.

For too long, the industry's response to this hostility has been to build more complex abstractions—cloud bot farms, proxy rotators, and AI agents that promise to magically understand every DOM structure on the planet.

But magic is inherently un-debuggable. When an AI scraper hallucinates a CSS path, or a remote browser farm gets silently fingerprinted, you are left holding the bag.

We believe that reliable data extraction requires less magic and more engineering rigor.

By starting on the desktop, we inherit your natural trust and authorized access. By decoupling discovery from execution, we isolate the fragile parts of browser automation. And by treating the intent.json as a mathematically verifiable contract—enforced by Invariants and Shift-Left QA—we turn web scraping from a game of whack-a-mole into a predictable, observable system.

The web is going to keep changing. Your selectors are going to break. The goal isn't to build a scraper that never fails. The goal is to build a scraper that never lies.