DEV Community: Gani Mendoza

The Elephant Behind Physics

Gani Mendoza — Sun, 21 Jun 2026 15:20:54 +0000

What if the greatest obstacle to humanity’s most ambitious scientific quest — a single theory explaining everything — is not a missing equation, but a missing protocol for knowing when we’ve finally found one? Purchase the 8-PDF bundle here to find out why.

Consider what it looks like when someone makes the most important discovery of their career and immediately argues against it.

It is 1900. Max Planck, a deeply conservative German physicist with an almost religious commitment to classical theory, has just solved one of the most stubborn problems in science — the precise shape of the spectrum of light emitted by hot objects. For decades, the best minds in physics had failed. Planck succeeded by assuming that energy is not continuous, as every physicist believed, but comes in discrete chunks he called quanta — each chunk proportional to frequency, each carrying energy E = hν, where h is a new constant he had to invent for the purpose.

The formula worked. It fit every measurement. It closed a problem that had embarrassed physics for a generation.

Planck promptly filed an objection against his own result.

Not publicly, not loudly — but in his notes, in his letters, in the careful language of a man who knew what he had and refused to overclaim it. The quantization, he insisted, was a mathematical device. A trick that worked. Not a statement about physical reality. He would spend the next decade trying to derive his own result from classical foundations, attempting to show that the discrete chunks were an artifact of calculation, not a feature of nature. He failed. The quanta were real.

What Planck did in 1900 — filing a faithfulness objection against his own result — is not how we usually tell the story of scientific discovery. We prefer the version with the lightning bolt, the eureka, the lone genius. But the more accurate story is quieter and, it turns out, far more useful: a man who had a protocol for thinking, even if he never named it. A man who knew the difference between this works and this is true.

That distinction is the elephant behind physics. And it has been hiding in plain sight for over a century.

The protocol has a name now. I call it the Elephant Bridge Protocol — EBP v2.1 — and its architecture is disarmingly simple. Ideas enter free. No committee, no approval, no justification required at the door. But promotion to serious candidate — to the equivalent of a claim you are willing to defend — costs debt. You must show your route from A to B. You must state what property survives the crossing. You must run a small test before committing to the large one. You must name at least one simpler explanation you are actively trying to beat. You must check whether known blockers apply. And you must ask, honestly, whether your formalization actually captures what you intended to claim.

Six obligations. None of them particularly onerous. Together they form something remarkable: a system that makes honesty cheaper than overclaiming.

Physics didn’t get careful by being smarter. It got careful by building a protocol that made the cost of vagueness visible before the vagueness became expensive.

You have seen this protocol in operation whether you recognized it or not. A good restaurant has a suggestion board in the kitchen — any cook can pin any idea, no form required, no committee needed. And a good restaurant has a menu, gated by tasting sessions, cost analysis, preparation-time checks. The suggestion board and the menu are intentionally different places with intentionally different rules. The failure mode when they merge is familiar in both directions: either cooks stop suggesting because the barrier is too high, or customers get inconsistent food because the barrier is too low.

Every organization that has ever struggled with the gap between brainstorming and shipping knows this problem. EBP names it, draws the line between the two places explicitly, and gives the gates names so they can be argued about rather than felt vaguely and enforced inconsistently. In a software team it is the distance between a GitHub issue and a production deployment. In a business it is the distance between a whiteboard session and a quarterly commitment. In physics it is the distance between a late-night calculation and a published theory.

The protocol does not care which domain you are in. It cares only about the gap.

Learn about Medium’s values
The most instructive demonstration of EBP in action is not Planck. It is the story of what happened to Isaac Newton.

Newton’s theory of gravity was not wrong. This is the point most people miss. It was promoted — correctly — for over two centuries, within an honest scope: slow-moving objects, weak gravitational fields, no dynamical sources. Within that scope its debt was retired. It predicted planetary orbits, tides, projectile motion, the return of comets.

Then in 1859, a French mathematician named Le Verrier calculated that Mercury’s orbit precesses at a rate Newton’s theory cannot explain. Forty-three arcseconds per century. A number so precise and so reproducible it could not be dismissed. Under EBP, this is not a crisis. It is a ledger entry. Newton is not killed — he is scoped. The debt reopens. The next move becomes visible.

It took fifty-six years for Einstein to make it.

General relativity did not destroy Newton. It contained him — recovered Newton’s predictions exactly in the regime where Newton had always worked, and extended the map into regimes Newton never reached. GR was promoted, with its own open debt explicitly acknowledged: singularities at black hole centers, incompatibility with quantum mechanics at Planck-scale curvatures. No final-truth language. The best currently funded map of classical gravity.

What the EBP ledger shows across three centuries of physics is not a sequence of revolutions — theories overturning each other in dramatic succession. It shows a sequence of honest scopings. Every promoted theory carries the open debt of the questions it cannot yet answer. Every demotion is a narrowing, not a demolition. Newton is not on the trash heap. He is dormant, waiting for anyone who needs to calculate a rocket trajectory.

The ledger never expires. No debt ever dies of old age.

Which brings us to the question the book leaves open — deliberately, in the Socratic tradition.

Physics today carries two fully promoted theories with an unresolved obstruction filed between them. General relativity handles gravity and the large-scale structure of spacetime. Quantum mechanics handles everything else. Both are promoted within their domains. Both have passed every experimental test in their respective regimes with extraordinary precision. And they are, at the deepest mathematical level, structurally incompatible.

The conventional framing of this problem is: find the Theory of Everything. One equation. One framework. The grand unified picture that Einstein spent the last thirty years of his life searching for and never found.

EBP suggests a different question.

What if the Theory of Everything is not a discovery waiting to be made but a bridge waiting to be honestly built — a promoted framework that contains GR and quantum mechanics as limiting cases, carries their open debt forward, retires the obstructions one by one, and makes no final-truth claim it cannot back with a checkable invariant and a finite test?

What if the problem is not that we lack the intelligence for the answer, but that we have been asking for a destination when what we needed was a protocol for recognizing when we have arrived?

The elephant behind physics was never the universe. It was the question of how to think about it honestly.

That question does not belong to physics. It belongs to anyone who has ever stood between a good idea and a premature claim — in a kitchen, in a sprint, in a boardroom, at a desk in Berlin in 1900, staring at a formula that works and knowing, with uncomfortable precision, exactly what it does not yet prove.

The Elephant Behind Physics is out now.

The search for the Theory of Everything is not a problem waiting to be solved by a single mind in a single moment — it is an open ledger, and EBP is our north star for navigating it honestly. This blog documents that search as a living project: open, collaborative, and built with the same protocol it studies — drawing on the community, on AI, on software, and on EBP itself as both method and measure. If that project interests you, kindly follow my blog.

Web Scraping is a Contract

Gani Mendoza — Mon, 01 Jun 2026 18:04:34 +0000

Pithom Labs Scraper introduces a systematic approach to web scraping that treats data extraction as a binding contract rather than a fragile script. Traditional scrapers often fail silently by ingesting corrupted or empty data when website layouts inevitably change. To solve this, we present a specialized engine that utilizes human-guided discovery to establish a baseline of "truth" for a webpage's structure. This baseline, or GoldenSeal, allows the machine to perform runtime assertions and halt execution immediately if the site's data density or lineage shifts. By prioritizing loud failure and forensic evidence over quiet errors, the system ensures that automated pipelines never compromise data integrity. This methodology shifts the focus from evading bot detection to maintaining structural rigor in a constantly evolving digital environment.

Reprint from Medium

Let’s say the quiet part out loud: web scraping is usually held together with hope, CSS selectors, and a cron job that nobody on your engineering team wants to touch.

You build the parser. You map the fields. You run the script. You get a clean CSV or a pristine JSON array, and for a brief, shining moment, you feel invincible. You have conquered the unstructured internet.

And then, inevitably, the site changes.

It rarely breaks in a way that causes your script to crash and burn spectacularly. If it threw a loud, stack-tracing panic, you could fix it. Instead, a React component gets wrapped in three new div tags. A list hydrates half a second later than usual. A "Next" button moves into a different semantic container. A login session quietly expires in the background.

The data pipeline doesn’t explode. It does something infinitely worse: it keeps running. It keeps executing the same obsolete selectors against a mutated DOM. It happily writes empty strings or completely wrong text into your database. Downstream, your analytics dashboard or machine learning model is confidently eating nonsense, manufacturing false confidence at scale.

That is the part of scraping that we chronically understate. The hard problem isn’t figuring out how to extract data once. The hard problem is knowing when the web has shifted under your feet.

Most scraping tools respond to this reality with a brutal arms race. They throw more proxies, more remote headless browser farms, more fingerprint patches, and more opaque infrastructure at the problem, trying to convince the modern web that a faceless machine in a Virginia data center is actually a human being.

At Pithom Labs, we took a different route with our Go-based scraper engine. We stopped treating web scraping like a document parsing exercise, and started treating it like a typed contract with runtime assertions.

If the web is a moving target, your scraper shouldn’t pretend it’s static. It should fail loudly, produce evidence, and refuse to lie to you.

The Optimism of the Modern Scraper

Classic scrapers are optimistic little machines. They operate under a set of foundational assumptions that simply do not map to the reality of the modern internet.

They assume the page will load exactly the same way every time. They assume the selector that worked yesterday will work tomorrow. They assume that if the network request returns an HTTP 200 OK, the payload is probably meaningful.

But websites are not static documents anymore. They are moving, reactive, personalized, occasionally hostile application surfaces. They hydrate dynamically. They load content via asynchronous GraphQL calls. They lazy-load images. They A/B test their layouts. They change their entire markup structure because a frontend engineer decided to refactor a component library on a Tuesday afternoon.

When you point a traditional Python or Node.js script at this environment, you are essentially firing a blindfolded arrow and hoping the target hasn't moved. When the target does move, the script blindly extracts whatever happens to be occupying that coordinate space.

We realized that to fix this, we had to change the fundamental relationship between the scraper and the web page. We couldn't just build a better DOM parser; we had to build a system that understands what it's supposed to be looking at, and aggressively verifies that reality before it writes a single byte of data to disk.

The Baton Pass: Decoupling Discovery from Extraction

A lot of scraping products want to abstract the web away from you. They offer hosted dashboards, remote browser fleets, and managed extraction APIs. This can be useful, but it creates a massive trust problem. You have to hand over your credentials, try to replicate complex browser states on remote machines, debug someone else's infrastructure, and hope the target site doesn’t trigger a Cloudflare CAPTCHA that your headless script has no physical way of solving.

We designed the Pithom Labs Scraper around a radically different philosophy: The desktop is not a limitation. It is the point.

We built the architecture as a strict two-stage, decoupled system. We call the transition between these two stages the Baton Pass.

Stage 1: Human-Guided Discovery

In the first stage, you aren't writing code. You invoke scraper discover from your terminal, which launches a highly visible, headed instance of Google Chrome running directly on your machine.

Because it’s a real browser running locally, you can log in naturally. You can solve the CAPTCHA. You can click past the cookie consent banner. You establish the authorized session exactly as a human user would.

Once you are on the target page, our Omni-Agent Discovery overlay injects into the browser. You visually click the elements you want—titles, prices, detail links, pagination buttons.

Behind the scenes, the scraper isn't just recording dumb CSS paths. It is generating two critical artifacts:

session.json: A durable record of your exact browser cookies, User-Agent, and authentication state.
intent.json: A declarative recipe containing CSS/XPath selectors, semantic hints, structural hashes, and pagination logic.

Stage 2: Headless Extraction

Once you save the intent, the Baton Pass occurs. The human steps away, and the programmatic engine takes over.

You run scraper scrape, and the Go-based engine boots up in headless mode. It reads the session.json to perfectly spoof the authorized user state. It spins up a concurrent render pool using a stealth engine we call Ghost-Walker (which manages Chromedp under the hood to bypass headless detection and preserve JavaScript context).

This decoupling solves the hardest part of scraping—authentication and anti-bot mitigation—by letting a human handle the hard part once, and letting the machine handle the repetition.

But more importantly, the intent.json generated during Stage 1 isn't just a list of selectors. It is a binding contract.

Extraction as a Contract

In traditional software engineering, we use types, interfaces, and assertions to guarantee that our data is shaped correctly. If a function expects an integer and receives a string, it panics. It fails loudly.

Web scraping rarely has this luxury. Because the DOM is fundamentally untyped and fluid, scrapers have historically relied on "vibes-based" extraction. If .product-title > h2 exists, grab it. If it doesn't, write null and keep moving.

We wanted to bring systems-level rigor to DOM extraction. To do this, the intent.json acts as an executable agreement between the discovery phase and the runtime engine.

The GoldenSeal

When you finish Stage 1 discovery, the engine computes something we call the GoldenSeal.

The GoldenSeal is a structural fingerprint of the page at the exact moment you taught the scraper how to read it. It lives at the bottom of your intent.json and looks something like this:

"golden_seal": {
  "sealed_at": "2026-05-29T12:00:00Z",
  "row_count": 20,
  "structural_hash": "sha256:d8e3ab03bc",
  "field_population": {
    "title": 1.0,
    "detail_url": 1.0,
    "description": 1.0,
    "price": 0.95
  }
}

This isn't just metadata. The GoldenSeal establishes the baseline reality of the website. It says: "When the human was looking at this page, there were exactly 20 items. The 'title' field was populated 100% of the time, and the 'price' field was populated 95% of the time."

During headless execution, the engine constantly measures the live DOM against this seal by enforcing Integrity Invariants.

The Density Invariant

The scraper expects each paginated list to maintain a consistent density. If the GoldenSeal expects 20 items per page, and the live execution suddenly extracts 0 items, or 3 items, the engine knows something is wrong.

Traditional scrapers would happily write those 3 items to a CSV and move on to the next page. Our engine trips the Density Invariant. It halts execution immediately, recognizing that either the page hasn't fully hydrated yet (Skeleton DOM), or the site layout has radically changed.

The Lineage Invariant

Even if the scraper finds the correct number of rows, the individual selectors might have drifted. The Lineage Invariant compares runtime field fill-rates against the GoldenSeal.

If the title field was populated 100% of the time during discovery, but during runtime it is only populating 10% of the time, the Lineage Invariant fails. The engine recognizes that it is experiencing Structural Drift. It refuses to continue writing empty columns.

Shift-Left QA: Validating Before We Commit

In a data pipeline, corrupted data is vastly more expensive to fix after it has been written to disk or ingested into a data warehouse. You want to catch the error as far upstream as possible.

To enforce the contract, the Pithom Labs Scraper implements a mechanism we call Shift-Left QA.

When the headless engine begins extracting data from the first page, it does not immediately stream those rows into your output CSV or JSON file. Instead, it buffers the first 5 rows in memory.

It runs these buffered rows through a gauntlet of semantic validations. It checks the Invariants. It verifies that required fields are present. If the site requires clicking into "Detail Pages" for deeper data, it ensures that the detail URLs aren't throwing 404s and that the deep extraction isn't returning blank text (enforcing the detail_skip_tolerance).

If the QA Buffer detects a critical failure—if all the fields are empty, or the data has fundamentally shifted—the run is aborted before a single byte of garbage data touches your output file.

Instead of writing bad data faster, the system stops, records the evidence, and generates a diagnostic bundle.

Failing Loudly: Evidence over Magic

The scariest scraper isn't the one that crashes. The scariest scraper is the one that fails quietly.

When the Pithom Labs Scraper breaks a contract and halts, it doesn't just log a generic error and die. It produces evidence.

It exits with strict, semantic CLI exit codes that programmatic supervisors (like cron jobs or CI/CD pipelines) can actually understand and route:

Exit Code 0: Success. The contract was upheld.
Exit Code 3: Structural Drift. The layout changed or the Density Invariant failed.
Exit Code 4: Integrity Failure. Data quality dropped below tolerance (e.g., detail pages are failing to load).
Exit Code 42: Auth Required. The site returned a 401/403 or redirected to a login screen. The session cookies are dead.

More importantly, upon a critical failure, the engine generates a timestamped diagnostics_YYYYMMDD_HHMMSS/ forensic bundle.

This bundle contains scrape_failure.jsonl (the exact structured events leading up to the crash) and, crucially, failure_snapshot.html—a complete, redacted snapshot of the DOM at the exact moment the scraper realized it was looking at an alien landscape.

You don't have to guess why the scraper failed. You don't have to write custom scripts to reproduce the error. You open the diagnostic snapshot, and you see exactly what the scraper saw: a Cloudflare challenge, a new A/B tested layout, or an expired login redirect.

Engineering for a Hostile Environment

Web scraping is, by definition, the act of writing highly coupled code against an unversioned API that you do not control, built by people who often actively do not want you to be there. It is a uniquely hostile engineering environment.

For too long, the industry's response to this hostility has been to build more complex abstractions—cloud bot farms, proxy rotators, and AI agents that promise to magically understand every DOM structure on the planet.

But magic is inherently un-debuggable. When an AI scraper hallucinates a CSS path, or a remote browser farm gets silently fingerprinted, you are left holding the bag.

We believe that reliable data extraction requires less magic and more engineering rigor.

By starting on the desktop, we inherit your natural trust and authorized access. By decoupling discovery from execution, we isolate the fragile parts of browser automation. And by treating the intent.json as a mathematically verifiable contract—enforced by Invariants and Shift-Left QA—we turn web scraping from a game of whack-a-mole into a predictable, observable system.

The web is going to keep changing. Your selectors are going to break. The goal isn't to build a scraper that never fails. The goal is to build a scraper that never lies.