DEV Community: Ted

You Deleted It. The Crawler Still Sees It.

Ted — Sun, 19 Jul 2026 07:56:19 +0000

I removed two listings from a directory site I run. Not for a small reason — they failed the one promise the directory exists to make. The kind of removal you do immediately and double-check.

I did double-check. Flipped the flag in the database, loaded the site, watched the queries filter them out. Gone from the city pages, gone from the main directory. Removal confirmed.

A day later, both were still on the site.

Not on some cached page or stale CDN edge. In the current, freshly deployed HTML — sitting in the exact lists I had watched them disappear from. I had verified the deletion in one document. The site is two.

A rendered site is two documents

Here's the architecture, because the bug lives in it. The site is a single-page app: the browser downloads a JavaScript bundle, the bundle fetches data, the page assembles itself client-side. Crawlers are historically bad at waiting around for that, so — like a lot of SPA operators — I run a prerender step at build time. A script fetches the same data, bakes real content into each route's static HTML, and that is what robots, link previews, and anyone with broken JavaScript actually receives. The bundle takes over afterward and re-renders everything live.

It works. It's also a decision with a consequence I under-appreciated for months: every fact on the site now exists twice. Once in the live render, computed in the browser from a fresh query. And once in the baked copy, computed at build time by a different script, from a different query, through a different code path.

Two documents. Two authors. One URL.

And the two authors don't consult each other. The browser filters listings with a WHERE allowed = true. The build script — written earlier, for a different purpose — never selected that column. So when I "deleted" the two properties, the browser's query dutifully dropped them, and the build script kept baking both into the static HTML on every deploy, complete with links and ratings. From its perspective, the deletion had never happened.

The deletion was real in one document and fiction in the other. Nothing errored, because nothing was wrong: both documents were internally consistent, both rendered cleanly, both passed every check that looks at documents one at a time.

Three divergences, one day

Once I knew to look for the split, I found it three times in a single day — same site, escalating stakes.

The title fix that wasn't. A page was earning search impressions for a phrase its title didn't contain, so I added the phrase. Checked the page in the browser: there it was, in the tab, in the markup. But client-side titles are set by the bundle after load — the crawler reads the baked <title>, and the baked title still didn't have the phrase. The fix was fully live for every human and invisible to the one reader that determines whether the page ranks. It had been "fixed" in the wrong document.

The labels that didn't propagate. I'd spent that week making the site more honest — replacing a blanket "verified" badge with per-listing labels that say what's actually confirmed versus merely claimed. The app rendered the new labels beautifully. The baked copy went right on telling crawlers every listing was verified, in a header the build script had been stamping onto nine different pages. The honesty overhaul shipped to the browser and not to the record.

The deletion. The first two failures overstate; this one resurrects. A removed listing isn't a stale adjective. It's an entity I had decided, for cause, should not be presented — still being presented, by a document I'd forgotten was the one that counts.

And the failure runs in both directions. While fixing the build script I added a column to its fetch that didn't exist in that table. The data API rejected the query — and the script, hitting an error, quietly returned an empty list, at which point every detail page fell back to a generic template. No build failure. No warning. Just a deploy where the baked document silently regressed while the live one stayed perfect. Two documents means twice the ways to be wrong, and each one's failures are invisible from inside the other.

Why nothing catches this

Every tool I had was pointed at one document or the other. Never at the seam.

The type checker checks the app. The tests exercise the app. The browser — where all manual verification happens, because that's where you look at your site — runs the app. Meanwhile the build logs confirm the prerender ran, which is not the same as confirming what it wrote. The bug wasn't in either document. It was in the fact that there were two.

This is the oldest failure shape in data systems — the same fact stored in two places will drift, given time — wearing a disguise that makes it easy to miss: the two copies here aren't two database rows or two config files. They're two renderings of the same URL, which is precisely why it doesn't feel like duplication. It feels like one page. You'd never think to diff a page against itself.

"Done" names a document

The fix for the specific bugs was mundane: make the build fetch select the flag, filter on it, propagate the labels, correct the title — in the baked layer, where each change should have landed first. The durable fix was changing what I accept as evidence that a change shipped.

Looking at the site in a browser verifies the live document. It verifies nothing about the baked one. So the definition of done, for anything that matters to a crawler — a title, a claim, a listing's existence — is now mechanical: fetch the served HTML the way a robot would, with no JavaScript executed, and confirm the change is in the bytes. curl, grep, done. Ten seconds. It would have caught all four incidents before deploy, and it's the only check that examines the document I kept forgetting existed.

If your site renders twice, you don't have a page. You have a page and its understudy, and the understudy is the one performing for the audience that decides whether anyone finds you. Verify the performer that's actually on stage.

Further reading — adjacent failure modes from the same system: the prerender pipeline that ran on every deploy while writing empty pages, and prose as one more unsynchronized copy of your data. Same lesson at different layers: anything that exists twice, drifts.

Originally published on tedagentic.com.

The Listing Didn't Exist. The Booking Link Worked.

Ted — Sat, 18 Jul 2026 18:55:00 +0000

The property was called something plausible. It had a star rating, a price, a tidy paragraph of description, valid LodgingBusiness structured data, and a green "Verified" badge. It also had a Book Now button wired to a working affiliate link — the kind that pays a commission if someone clicks through and reserves a room.

The property did not exist. Not "closed," not "renamed." It was never real. Some fast, AI-assisted build had generated it whole — name, blurb, rating, amenities — and pointed it at a live payment rail.

It wasn't alone. There was a whole cohort of them, and the giveaway was almost funny: a dozen distinct "hotels" in different cities all sharing the same affiliate link. Different names, different descriptions, one destination URL. A forger who signed every painting with the same signature.

None of them were broken. That's the part worth sitting with. They passed every automated check I could throw at them. Valid types. Valid schema. No 404s, no build errors, no lint warnings. A validator confirmed they were well-formed, and being well-formed was the entire disguise.

Wrong implies there's a real thing being described

I'd spent earlier weeks fixing data that was wrong — a fact gone stale, an entity filed in the wrong city, a corrected value that never reached the page. Wrong is a solved kind of problem. Wrong means there's a true answer and the stored one disagrees. You audit, you compare against a source, you reconcile.

This was different, and it took me a beat to see why. You cannot reconcile a fabrication against a source, because there is no referent. The row doesn't describe a real thing incorrectly. It describes nothing, confidently. There's no fact to check because there's no subject.

So the first reframe: AI-generated seed data isn't a rough draft you refine. It's a forgery you have to detect — and its defining property is that it validates. Every instinct we've built says malformed data is the dangerous data, because malformed data is what breaks things. But malformed data announces itself. The genuinely dangerous record is the one that is perfectly formed and completely false, because nothing in your pipeline is designed to doubt a row that parses.

I deleted them. You don't refactor a forgery.

The quieter version of the same problem

The fake listings were the loud case. Underneath them was a quieter one that had spread much further, and it's the part that actually changed how I think.

Real listings had a policy field — a small piece of data recording whether a given property had actually been confirmed to permit a certain thing on-site. Most of them had never been confirmed. The field was empty.

Here is what the page rendered for an empty field: a green Verified badge. A "Yes" in the amenities grid. A line of structured data asserting the thing was permitted. A confident sentence in the server-rendered copy that a crawler reads before any of the interactive parts load.

Nobody decided to claim these things. There was no bad actor and no bad data. There was one line of code, repeated in spirit across the codebase, that read roughly: if this value is missing, treat it as true. A default. ?? true, or its cousin in four other files.

That is the second reframe, and it's the one I'd tattoo on a new engineer: a default is a claim. When you fill an unknown with the favorable value, you are not being helpful and you are not leaving it blank. You are asserting a fact you do not have. The empty field didn't stay empty on the way to the user. It was quietly promoted to a promise.

Absence rendered as a yes.

The missing value was "I don't know"

Once I saw it in one place I saw it everywhere, and they all traced to the same root cause. The data model had two states where it needed three.

There was true, there was false, and there was null — but null was never a state of its own. It was an input to be resolved, and every layer resolved it the same optimistic way. The card coerced it to yes. The badge coerced it to yes. The structured data coerced it to yes. The prerendered copy a crawler reads coerced it to yes. One missing fact became four confident assertions, none of which had any way to represent the honest answer.

Because there was no honest answer available. "Unverified" wasn't a value the system could hold. And a system that cannot represent "I don't know" does not fall silent about what it doesn't know. It reaches for the most flattering value in range and commits.

The fix was not clever. It was to add the value that was missing.

A real, explicit state for unconfirmed — stored, not inferred.
A three-way rendering instead of two: Yes, No, and Unconfirmed. The unconfirmed state gets its own muted styling — never the green of a confirmed yes, and just as importantly never the red of a confirmed no. "We haven't checked" is a different claim from "this is not allowed," and collapsing them is its own kind of lie, pointed the other direction.
Every layer taught to defer to that state instead of defaulting past it. The card, the badge, the structured data, the static copy the crawler reads — all four now say "unconfirmed" together, or say nothing, rather than four independent optimistic guesses.
Positive claims gated behind an actual confirmed value, so the system asserts a thing is true only when something actually backs it.

Give the model a way to shrug

The lesson generalizes past any one field or any one site. It applies anywhere a system holds facts it can't fully back — which is everywhere, and increasingly so as more of the initial data gets generated rather than entered by someone who checked.

We tend to think of honesty in a system as the absence of false statements — no bugs, no stale rows, a clean audit. But you can pass every audit and still have a system that lies, because the lies aren't in the data. They're in the defaults. They're in what happens to a value you never had.

Real honesty is the presence of "unknown" as a first-class thing: a value you can store, a state you can render, a claim your structured data is allowed to decline to make. If the only options your model offers are yes and no, then every gap in your knowledge silently becomes whichever one flatters you — and the extreme end of that gradient is a hotel that doesn't exist with a working button to book it.

Give the model an explicit way to say "I don't know." Otherwise it will nod at everything, and you won't find out where until someone audits the room and discovers half of it was painted by the same forger.

Further reading — the adjacent failure modes: why AI tools produce confident silent errors on production sites, and why a fallback chain hides missing data by design. This post is the layer under both: why the gap gets filled with a lie in the first place.

Originally published on tedagentic.com.

Logging Into the Higgsfield CLI on a Server With No Browser

Ted — Wed, 08 Jul 2026 04:08:43 +0000

I wanted the Higgsfield CLI on my server — a terminal front-end to a pile of image and video generation models, which makes it scriptable and cron-able in a way the web UI never will be. The install was one line. Then it asked me to log in, and a specific, recurring failure showed up that has nothing to do with Higgsfield and everything to do with a word that means two different things at once.

The word is localhost.

The server is headless: no monitor, no browser, reached only over SSH from a laptop. That detail is the whole story.

The flow that assumes a lap

Higgsfield authenticates with OAuth 2.0 PKCE, the way most modern CLIs do, and the polite version goes like this: the CLI opens a small HTTP listener on localhost:<port>, prints a login URL, and waits. You open the URL in a browser, approve, and the identity provider (Higgsfield fronts theirs with Clerk) redirects your browser to http://localhost:<port>/callback?code=.... The listener catches that request, reads the code out of it, trades it for a token, and you're in.

On a laptop this is seamless because the browser and the listener are the same machine. localhost resolves to the same place for both halves of the handshake. The whole design quietly depends on that.

On a headless server it falls apart, because the two halves are now on different machines. The listener is on the server. The browser is on my laptop. When Clerk redirects the browser to http://localhost:<port>/callback, the browser dutifully connects to the laptop's localhost — where nothing is listening — and shows:

Unable to connect

The login succeeded. The callback failed. The code was issued — it just landed on the wrong machine's front door.

A loopback OAuth flow assumes the browser and the listener share a host. That assumption is invisible on a laptop and load-bearing on a headless box.

The tunnel, and the wall the spec put in front of it

The reflex fix is an SSH tunnel. Forward the callback port from the laptop to the server, so the laptop's localhost:<port> actually reaches the server's listener:

ssh -L 8799:localhost:8799 you@server

Sound. It works. But I hit a wall trying to pin the port, and the wall is worth understanding because it's not a bug — it's the security model doing its job.

higgsfield auth login has a --port flag, so I told it to use 8799 to match my tunnel. Clerk rejected the whole login:

error: invalid_request
The 'redirect_uri' parameter does not match any of the
Client's pre-registered redirect URIs.

Here's the thing PKCE doesn't remove: the redirect URI allowlist. PKCE lets a CLI skip shipping a client secret — the code challenge proves the token request came from whoever started the login — but the provider still refuses to send a code to any callback URL the app didn't register in advance. That allowlist is what stops a malicious page from pointing your authenticated redirect at its server. Higgsfield registered a specific handful of ports with Clerk, and those are the only ones that work. 8799 wasn't in the club.

The redirect URI is a registered constant, not a free parameter. You don't get to pick the port; you get to pick from the ports the app's developer already blessed. Drop the --port flag and the CLI falls back to its own default and it works — but now your tunnel has to match a port you don't fully control, and if that port is already busy on the server (mine collided with another service on the default), the CLI silently falls back again to a different registered port, and your tunnel is aimed at the wrong one.

The tunnel isn't wrong. It's just more moving parts than the problem actually needs.

The callback is just data

The realization that made this easy: the callback is not a handshake, it's a message. It's a plain HTTP GET whose entire payload is visible in the URL — ?code=<the code>&state=<the anti-forgery token>. The listener doesn't care who delivers that message. It cares that the message arrives with a state matching the one it issued.

So I stopped trying to route the browser to the listener at all. I ran higgsfield auth login on the server (letting it pick its own registered port — it landed on 8766), opened the printed URL in my laptop browser, approved, and let the redirect fail — Unable to connect, as expected. Then I copied the dead URL straight out of the browser's address bar:

http://localhost:8766/callback?code=OWVH...F1L&state=kg6P...TCY

That URL is the whole message. The listener is sitting on the server's own localhost, exactly where the CLI is watching. So I replayed it there:

curl "http://localhost:8766/callback?code=OWVH...F1L&state=kg6P...TCY"

The listener saw a request on its port, checked the state, matched it, exchanged the code for a token, and printed Successfully authenticated. No tunnel. The browser did the one thing only a browser with my logged-in Higgsfield session could do — prove I'm me — and then I hand-carried the resulting code the last hop the browser couldn't make.

Two things make this safe to lean on, and both are worth saying out loud:

The state parameter is the check that makes replay legitimate. The listener issued that random token at the start and refuses any callback that doesn't echo it. I'm not bypassing a security control by replaying the URL — I'm satisfying the exact one the flow was built around.
The code is short-lived and single-use. An OAuth authorization code is a password with a fuse — good for one exchange, expiring in a minute or two. Copy it, replay it promptly, and don't paste it anywhere it'll be logged. It's not a token you're storing; it's a token you're spending immediately.

What I'd carry to any headless login

localhost is a per-machine word. Any auth flow that redirects to localhost is quietly assuming the browser and the service are the same box. On a headless server they aren't, and that mismatch — not the tool — is your bug.
Redirect URIs are an allowlist, not a setting. PKCE drops the client secret but keeps the registered-callback rule. If pinning a port throws invalid_request, you're fighting the security model, not a config typo. Use the app's own default port.
A loopback callback is replayable data. It's a URL carrying a code and a state. You don't need to route the browser to the listener; you can let the redirect die and deliver the URL yourself with curl on the machine that's listening.
state is what makes hand-delivery honest. It's the flow's own anti-forgery check. Replaying a URL that carries the right state isn't a workaround around the security — it's the security working.

The tunnel is the textbook answer, and some days you'll want it. But when a login on a headless machine tells you it can't connect to localhost, remember that it already succeeded — the code exists — and the last hop is just a URL you're allowed to carry across yourself. This worked cleanly for Higgsfield; it'll work for any CLI whose login is a loopback redirect, which by now is most of them.

Both Delivery Paths Went Silent. Neither Was Broken.

Ted — Sat, 04 Jul 2026 18:25:22 +0000

My home server sends me a stream of scheduled reports over Telegram every morning — system health, data pulls, monitoring summaries, a wake-up alarm. Around forty-seven cron jobs, delivered through two separate messaging agents, deliberately independent so that one agent failing can't silence the mornings.

Today the mornings went silent anyway. Both agents. Nothing at 3 AM, nothing at 7, nothing at 8:30. My first assumption was the agents — they're the newest, most complicated part of the stack, and when two things fail at once you suspect the thing they share. I was right about the reasoning and wrong about the layer.

Neither agent was broken

Both gateway services were green — one had been running uninterrupted for a day, the other for longer. But the job telemetry told the real story: 695 failed job executions since 01:03, and every single one carried the same last line:

Temporary failure in name resolution

The machine had lost the internet. Not Wi-Fi flapping, not a DNS misconfiguration — even direct queries to 1.1.1.1 got nothing, and the LAN link never dropped. The router's upstream was simply dead, from 01:03 to 15:43. Fourteen and a half hours, spanning every scheduled send of the day.

So the redundancy I'd built was real — two agents, two codepaths, two processes — and also cosmetic. Both paths terminate at the same wall socket. I had redundancy at the application layer and a single point of failure one layer down, which is the kind of thing that's obvious the moment it bites and invisible every day before that.

Redundancy that shares a dependency is a single point of failure with extra steps. Worth auditing for: my two "independent" delivery agents shared the WAN, the DNS resolver, and the power strip.

The network came back. Nothing else did.

Here's the part that actually earned this post.

At 15:43 the internet returned. The every-minute and every-15-minute jobs recovered on their own — their next tick came, succeeded, done. By evening they were all green without anyone touching anything.

The daily jobs did not come back. The morning report, the data pulls, the summaries — each of them runs once a day, each had fired exactly once into a dead network, and each would not try again until tomorrow. Cron kept perfect time through the whole outage. It just doesn't keep score.

That's the property I'd never had to think about before: cron remembers nothing. A missed tick doesn't queue, doesn't retry, doesn't log a debt. It evaporates. cron is a metronome, not a ledger — and every recovery guide you'll read is about getting the daemon running again, not about the runs that vanished while it was already running fine.

So recovery was manual: work out which of the day's jobs had failed inside the window, rerun seven of them by hand — in dependency order, with their exact original command strings so the monitoring would attribute the reruns to the right jobs — and skip the 3 AM alarm, because an alarm at 7 PM isn't late, it's wrong.

Doing that once is fine. Planning to do it every time the ISP hiccups is choosing to be the retry mechanism yourself.

The fix: give cron a memory

The fix is a small reconciler that runs every 15 minutes and closes the loop cron leaves open. The idea is stolen from control loops everywhere: compare desired state (what the crontab says should have run by now today) against actual state (what the job telemetry says actually succeeded), and converge.

Every job here already runs through a telemetry wrapper that appends one JSON line per execution — script, start, end, exit code. That log, it turns out, is the missing ledger. The crontab is the promise; the event log is the record; the reconciler is just the diff:

for sched, script, cmd in sorted(due_today()):
    runs = [e for e in events_today
            if e["script"] == script and e["ts_start"] >= sched - 60]
    if any(e["exit"] == 0 for e in runs):
        continue          # promise kept
    rerun(cmd)            # promise broken — run the verbatim crontab line, once

The design decisions matter more than the loop:

Gate on the network, quietly. The first check is a TCP connect to the delivery endpoint. If it fails, the reconciler exits 0 and says nothing — "network down" is a state, not an incident. It will still be here in 15 minutes.
Only catch up jobs that run once a day or less. Anything on an interval self-heals on its next tick; rerunning it is pure noise. Recovery effort should match the job's natural cadence.
One attempt per job per day. If the rerun also fails, the job is genuinely broken and the existing failure watchdog owns it. A reconciler without an attempt cap is a retry storm with a nicer name.
Rerun the verbatim crontab line — wrapper, redirects, everything. The telemetry keys runs by command; a rerun that doesn't match the original string shows up as a new mystery job instead of clearing the red one.
Denylist the jobs where the time is the payload. The wake-up alarm doesn't get caught up. Late data is still data; a late alarm is a bug.
Reuse the schedule parser the dashboard already has. I nearly wrote a second little cron-expression parser for this and stopped: two parsers means two opinions about what "should have run" means, drifting independently. This was every copy of a fact is a liability again, wearing a parser costume — the schedule semantics are a fact, and I almost gave them a second copy.

One accepted gap, named honestly in the code: the reconciler trusts exit codes. A job that exits 0 but fails to deliver its message is invisible to it. The exit code is the contract; scripts that lie about success are a different bug with a different fix.

What I'd carry to any scheduled system

Cron remembers nothing. If a run matters, something other than cron has to remember that it should have happened. A schedule is a promise, not a record.
Audit redundancy for shared dependencies. Two of anything that terminate in the same cable is one of that thing.
Per-execution telemetry turns out to be the ledger. I added the wrapper months ago for a dashboard. It ended up being the thing that made automatic recovery possible — you can't reconcile against a record you never kept.
Recovery must distinguish "late" from "wrong." Some payloads age gracefully; some are only valid at their scheduled moment. Encode that, or your catch-up mechanism becomes a noise generator.

Total build: about 150 lines, one evening, on top of telemetry that already existed. The next 14-hour outage ends with one Telegram message — network back, reran six missed jobs — instead of a silent morning and an evening of forensics.

Two Pages Disagreed About the Same Law. Both Were Wrong.

Ted — Sat, 04 Jul 2026 14:04:07 +0000

Last week I audited my travel site's core dataset — legal-status data for 213 countries and every US state — and found the same fact stored in four places, drifting independently. I fixed all four copies, fixed the silent join that kept corrections from rendering, and made the headline counts derive from the data at build time. Audit closed. Confidence high.

Then I asked the agent to check one blog post — the flagship guide that covers the same topic as the dataset. Just to confirm it matched.

It didn't. And the ways it didn't match turned out to be worse than anything in the original audit, because this time the wrong facts weren't sitting in a data file waiting for a diff. They were sitting in sentences.

The page that argued with itself

The flagship guide is half dynamic, half hand-written. The card grids and comparison tables pull from the data file I'd just fixed — so when the audit shipped, those healed automatically. A Central European country that fully legalized in January showed up in the "Legal" cards the moment the corrected data deployed.

Three scrolls down, the hand-written FAQ on the same page said that same country was "decriminalized — not legal," and used it as the textbook example of the difference.

One page. Two answers. The derived half was right, the asserted half was months behind, and no test could ever catch it because both halves were rendering exactly what they were told to render. There was also a link reading "See all 82 countries" pointing at a page that now lists 155.

A page that's half-derived and half-asserted will eventually contradict itself. It's not a risk, it's a schedule — the same schedule the four data copies were on, just running through paragraphs instead of rows.

Two pages, opposite lies

That was the warm-up. The real find was a Southeast Asian country that famously opened up in 2022 and has been walking it back since.

My blog post about it, written in March, said: legal grey zone, walk into any shop, no paperwork needed, here's what to buy.

My country-profile page about it, updated later, said: fully re-criminalized, every shop closed, zero tolerance, tourists face arrest and four-digit fines — treat it like the strictest country on Earth.

Same site. Same country. Same month. Opposite advice — and here's the part that matters: both were wrong.

The verified reality (current primary sources, not model memory) sat exactly between them: the country had gone medical-only in mid-2025 — thousands of licensed shops still open, but buying now requires a prescription any tourist can get in a fifteen-minute consultation at the shop.

So one page was encouraging illegal behaviour. The other was denying a legal pathway that actually existed. The specifics of the law are just the evidence; either page, followed faithfully, failed the reader.

If I had noticed the contradiction without checking externally, the obvious move would have been to pick the newer page and sync the older one to it. That would have replaced a dangerous lie with a merely costly one and called it consistency.

When two of your own copies disagree, neither one is the tiebreaker. Your copies can be wrong in opposite directions, and averaging them or trusting the newer one just launders the error. The only tiebreaker is outside the building.

Internal consensus isn't verification.

The fabricated statute

After that, I had the agent sweep every post on the site — about fifty-five of them — for anything shaped like a legal claim: statuses, ages, quantities, penalties, years.

Most survived. One didn't: a travel-safety post claimed a neighboring state had "decriminalized possession under three ounces in 2023," complete with a specific first-offense fine.

That state never decriminalized anything. It is, per my own freshly-audited dataset, in the strictest tier in the region — any possession is a criminal offense. The claim wasn't stale; there was no year in which it was true. It was invented, presumably by whatever wrote the first draft, and it survived every read-through since because it had everything plausibility needs: a number, a year, a dollar amount.

In the dataset audit, this failure mode — confident filler for unknown values — was at least constrained by a schema. A status column can only hold one of five values, and thirty identical suspicious values in a row is a visible pattern. Prose has no schema. A fabricated statute sits in a paragraph looking exactly like a researched one, and no uniformity tell will ever expose it. The same sweep also caught a legal age wrong by three years — duplicated, consistently, in two places, which is exactly why duplication felt like confirmation.

Prose is a copy too

The last audit's lesson was "one fact, four copies." That count was wrong.

Every sentence on your site that states a fact is another copy of that fact. Unversioned, untyped, invisible to schema checks, excluded from every reconciliation you'll ever write — and, unlike the database, it's the copy your readers and the search engines actually consume. My four-copy inventory missed the largest and least maintained store of facts on the entire site: the writing.

What actually worked, if you want to reproduce the sweep:

Grep for claim-shaped text. Jurisdiction names, numbers with units, permission verbs ("legal," "decriminalized," "banned"), years, fine amounts. Crude, but claims are surprisingly greppable.
Route every hit against the audited structured data. The dataset you just verified is now your answer key for the prose.
When your own sources disagree, go external. Primary sources settle it — not recency, not confidence, not which page looks more authoritative.
Fix toward derivation. The parts of pages that rendered from the data file healed themselves the moment the data was fixed. Every hand-written sentence needed a human-shaped repair. The ratio of derived-to-asserted facts on a page is the ratio of self-healing to permanent liability.

What I'd carry to any site

Prose is a copy too. Count it in your inventory. It's the biggest copy you have and the only one with no schema to defend it.
When two copies disagree, neither is the tiebreaker. Verify outside, or you're just choosing which error to standardize on.
Half-derived pages contradict themselves on a schedule. Either derive the fact everywhere it appears, or accept that every hand-written restatement of it is a new maintenance obligation.
Plausibility protects fabrications in prose better than in data. A schema makes invented values look uniform; a paragraph makes them look researched.

The dataset audit took one long session. This follow-up took another — and found the errors with the highest actual cost to a reader, in the layer the first audit had implicitly certified as fine. The data was never the product. The sentences were.

The Database Was Fixed Months Ago. The Website Disagreed.

Ted — Fri, 03 Jul 2026 04:31:21 +0000

I run a travel site whose core dataset is legal-status data: whether cannabis is recreational, medical-only, decriminalized, or illegal in 213 countries and all 50 US states. Getting one of those entries wrong isn't a typo. Someone plans a trip around it.

The problem is that this kind of data rots quietly. Laws moved constantly through 2025 and 2026: one US state opened its first dispensaries three weeks before I ran this audit, a Caribbean nation decriminalized this year, and one country famous for legalizing reversed course almost entirely. A dataset that was accurate when written is a dataset that is wrong now — you just don't know where.

So I gave an AI agent (Claude Code, in my case) a blunt instruction: go through every state and every country, compare each stored status against current law, and tell me what's wrong. No edits until I approve.

I expected a list of stale entries. I got that — plus two other categories of wrong I wasn't expecting, and they were both worse. I thought I was auditing facts. It took three kinds of wrong to see I was auditing a system.

Way #1 to be wrong: the facts moved

The boring, expected kind. Around a dozen entries where the law simply changed after the data was written.

Two details from this pile worth keeping:

Recency cuts both ways. The most out-of-date entry wasn't old — it described a legal regime that ended three weeks earlier.
Some "stale" data was never right. One US state was labeled decriminalized-only. It has had a medical program since 2013. That error sat on the page through every previous review, because nobody audits the entries that look plausible.

Fine. This is why you audit. If the story ended here it wouldn't be worth writing up.

Way #2 to be wrong: invented by a default

Around thirty small countries and territories — Pacific micro-states, Caribbean islands, a handful of French and Dutch territories — all showed the same status: Medical.

Almost none of them has any medical program. Several are among the strictest prohibition jurisdictions on Earth.

The tell was the uniformity. These entries had been bulk-generated at some point, and whoever — or whatever — generated them needed a value for places it knew nothing about. It picked "Medical." Plausible-sounding, middle-of-the-road, and wrong in the one direction that matters: permissive.

Think about the asymmetry. If you label a legal place "Illegal," the cost is a missed trip and maybe an annoyed email. If you label a strict-prohibition island "Medical," the cost is someone packing something they should never have packed. A default that errs permissive isn't a data bug, it's a liability generator.

The rule I took from this: defaults are policy. If a generator has to fill in an unknown, it must fill in the strictest value the schema allows — and ideally flag it as unverified. "Unknown" should never round to "kind of allowed."

Way #3 to be wrong: fixed, but the page never heard about it

This is the one that made the audit worth it.

The database had correct rows for five Caribbean nations — statuses corrected months ago in an earlier data pass. The live page still showed the old, wrong values for three of them. Not stale data. Correct data that never rendered.

The page builds its country list by merging a static fallback array with the database at runtime. The match was by URL slug or exact name:

const dbCountry = dbCountries.find(
  (db) => db.slug === staticCountry.slug ||
          db.name.toLowerCase() === staticCountry.name.toLowerCase()
);

Now look at the actual rows:

Database says	Static entry says	Match?
Saint Lucia	St Lucia	❌
Antigua and Barbuda	Antigua & Barbuda	❌
Saint Vincent	St Vincent & the Grenadines	❌

"Saint" vs "St". "and" vs "&". Same countries, different spellings, zero matches. When the lookup missed, the code did the reasonable-looking thing — fell back to the static entry. No error. No log line. No way to notice unless you compared what was stored with what was displayed, country by country.

The database corrections had been shipped, reviewed, and celebrated. The page disagreed for months, silently.

The fix was two parts. First, normalize both sides before comparing — lowercase, strip accents, treat "&" as "and" and "St" as "Saint" — plus an alias map for the two names normalization can't bridge. Second, and more importantly, a test: run every database row against every static entry and print the pairs. Exactly the five broken pairs matched, and nothing else did. A join you haven't tested isn't a join, it's a hope.

The real disease: one fact, four copies

Digging into all this surfaced the structural problem underneath. The site held each country's legal status in four places:

A static fallback array inside the page component
The database (source of truth, in theory)
A "rich profiles" data file used for detail rendering
The site-search index, with its own copy of every status chip

Nobody ever decided "let's keep four copies." Each copy was added for a locally reasonable reason — a fallback for resilience, an index for speed, a profile file for richer fields. That's how it always happens. And with four copies and no reconciliation, drift isn't a risk. It's a schedule.

The audit method that actually worked, if you want to reproduce it:

Dump every source. All four copies, into flat files.
Simulate the merge. Compute what each entry displays, not what any store contains. The displayed value is the only value.
Diff against verified current law. Web-verify each contested claim against primary sources — more on that below.
Fix all copies in one pass, plus the join, plus the tests, and make any headline counts ("legal in N countries") derived from the data at build time, never hand-written. Hardcoded counts are just a fifth copy.

Don't let the auditor grade its own answer key

One more thing, because it's the part people skip: during verification, the agent's own recollection of the law was flat-out wrong at least once. It "remembered" that a certain country had enacted a medical bill. Web search showed the bill was introduced and never passed — the database's boring "illegal" was right, and the confident model memory was the error.

Which means the audit loop has to be: model proposes, primary sources dispose. An LLM auditing data against its own training memory isn't an audit. It's the same failure mode as the template default in Way #2 — confident filler — wearing a reviewer's badge.

What I'd carry to any dataset

Audit what renders, not what's stored. Simulate the merge. The user never sees your database.
Defaults are policy. Unknown must round to the strictest value, never the most plausible one.
Every copy of a fact is a liability. Count your copies. You have more than you think, and the search index is always one of them.
A fix without a render test isn't a fix. The Caribbean corrections were "done" for months. Done means displayed.
Verify against sources, not model memory — especially when the model is the one doing the verifying.

Total cost of the whole thing — audit, verification, code fixes, database corrections, tests, deploy — was one long session with the agent doing the dumps, diffs, and searches. The only genuinely human part was the judgment calls: what "medical" should even mean to a traveler. Everything else was exactly the kind of work agents should be doing: patient, exhaustive, and immune to the plausibility that let these errors sit unnoticed for months.

One Page Ranked #6 and #20 at the Same Time. The Average Said #13.

Ted — Wed, 01 Jul 2026 14:22:38 +0000

A page I run sat at average position 13 in Search Console. That's the worst place a page can be — not page one, not far enough down to ignore. The kind of number you glance at, sigh, and scroll past.

I almost did. Then one question stopped me — the one an average can never answer: was that number describing one ranking, or two? I pulled the query-level breakdown to find out, and the "13" fell apart.

The average was two numbers wearing a trenchcoat

The page is built for one category of listing — call it Category A. For the A queries, it ranks 5 to 9. Top of page one, healthy click-through, doing its job.

But the same page also showed up for a second, adjacent category — Category B — at position 20. Page two. A handful of clicks on hundreds of impressions.

Lay the two query populations side by side and the problem is obvious at a glance:

Category A          Category B
─────────           ─────────
pos  5              pos 20
pos  6              pos 19
pos  7              pos 22
pos  6              pos 21
pos  8              pos 20
─────────           ─────────
avg ≈ 6.4           avg ≈ 20.4

Page average reported: ≈ 13
  a rank it never held

Average those two clusters together, weighted by impressions, and you get 13. A number that describes neither reality. The page wasn't "mediocre everywhere." It was excellent at one thing and irrelevant at another, and the mean smeared them into a single grey lie.

This is the trap with average position: it's an average. A page that ranks #3 for everything and a page that ranks #6 and #20 for two different intents can report the identical number. One is finished; the other has a structural problem the headline metric will never show you. You only see it when you split by query.

Why a Category-A page was chasing Category-B queries

The next question was: why is an A page surfacing for B at all?

The answer was in its own title tag. The page title named both categories — "A & B." I'd written it that way months earlier thinking it would catch more traffic. Instead it did the opposite of what I wanted.

A title tag is a magnet for intent. Tell Google your page is about B, and Google will test it against B searches — even when the page body is really about A, even when you have a different, dedicated page for B. So the engine dutifully ranked my A page at position 20 for B queries, pulling in impressions it could never convert and dragging the page's average into the swamp.

Advertising a category you can't win for isn't free reach. It's a tax.

The part that actually stung: I was competing with myself

When I pulled every page of mine that ranked for those Category-B queries, three URLs came back. The A page at ~20. A general directory page at ~23. And the actual B-specific article at ~15.

Three of my own pages, splitting authority for one query cluster, and none of them winning. Classic self-cannibalization. Google had three candidates from the same site and couldn't decide which deserved the spot, so it ranked all three poorly and gave the position to someone else entirely.

I'd manufactured my own competition. The dedicated B article — the page that should own that intent — was being undercut by two other pages of mine that mentioned B in passing.

The fix was subtraction

The instinct with SEO is to add: more words, more keywords, more content, more links. This fix was the reverse. I edited the A page's title and meta to remove every mention of Category B. No new content. No new links. Just deleting the words that were inviting the wrong queries.

The goal:

The A page stops matching B searches, stops ranking #20 for them, and concentrates its signals on the cluster it already wins (the 5–9 range).
Category B gets handed cleanly to the one page meant to own it, instead of being split three ways.
The "average position 13" should resolve upward — not because anything ranked better, but because I stopped averaging in the queries the page was never going to win.

Then I pushed the change and submitted the URL for recrawl so the engine sees the new title sooner rather than waiting on its own schedule.

What I'm taking from this

Never trust an averaged metric to tell you what to do. Average position, average CTR, average anything — they hide bimodal distributions, and bimodal is exactly where the interesting problems live. The page-level number told me "meh." The query-level split told me "two different problems wearing one URL." The whole diagnosis lived one dimension down.

This is also why I have the query breakdown wired to pull on demand rather than reading dashboards by hand. The signal that mattered here wasn't in any summary view — it was in the shape of the distribution, and you only see the shape when something queries the raw rows and looks for the split. The page average will tell you a page is sick. It will almost never tell you that the page is two pages, one of them healthy, fighting over a title tag.

I'll know in a couple of weeks whether the subtraction worked. The recheck is already scheduled to pull the same queries and compare. But the lesson holds regardless of the numbers: when a page looks evenly bad, check whether it's actually unevenly good.

I Ignored Bing for Months. It Ranked My Pages Backwards from Google.

Ted — Thu, 25 Jun 2026 02:42:54 +0000

I'd spent months optimizing for one search engine. Not on purpose — Google is just where I looked. Search Console was wired into my morning reports, the numbers showed up in Telegram every day, and that was "search performance." Bing Webmaster Tools existed in a browser tab I never opened.

When I finally opened it, the data didn't agree with Google. It actively contradicted it.

On the site, two pages cover the same subject. One is a dated long-form article — the "everything about this topic, updated for 2026" piece. The other is an interactive map that renders the same underlying data geographically: pick a place, see its status. They target the same cluster of queries.

On Google, the article wins decisively — roughly five times the impressions of the map, and a better average position. I'd internalized that as settled: Google had picked the article as the answer, end of story.

On Bing, the map wins. The article is barely present. The map page alone accounts for 56% of every impression the site gets on Bing. The article that dominates Google sits near the bottom of the list.

Same two pages. Opposite order. Two engines looking at identical content and disagreeing about which one deserves to rank.

Page	Google	Bing
Long-form article	Wins	Loses
Interactive map	Loses	Wins

Why an engine "picks" a page

This isn't randomness, and it isn't a quirk of one crawl. The two engines reward different things for the same informational query.

Google rewards the article. Its helpful-content systems lean hard toward substantive, dated, written pages for informational intent. A long article with depth, a clear publish date, and a "2026" freshness signal reads to Google as the authoritative answer to "tell me about this topic." The map page — mostly an interactive widget with thin surrounding text — reads as a tool, not an answer. Google buries it.

Bing rewards the tool. Bing is more literal. When the query itself implies a lookup — anything phrased as a map or a by-location question — Bing matches the page whose entire purpose is that lookup. The map page's title, structure, and intent line up with the query word-for-word, so Bing puts it first. The article, to Bing, is just a wall of text adjacent to the thing the user actually asked for.

Neither is wrong. They're optimizing for different definitions of "best result," and my two pages happened to be perfect representatives of each definition. So each engine crowned a different king.

The inversion showed up at the sub-page level too, even more starkly: one regional reference page ranks on the first page of Bing and sits at position 69 on Google — effectively invisible. Same page. The gap there isn't content; it's authority. Those head terms on Google are owned by encyclopedias, government sites, and AI summaries, and a young page doesn't crack them. Bing's bar is lower, so the same page surfaces. Watching only Google, I'd have called that page a failure. It isn't — it's just winning on the engine I never checked.

The zero-click trap

Here's the part that nearly fooled me twice.

The map page's top Bing queries — the high-volume, head-of-cluster ones — rank around positions 6 to 9 and convert almost no clicks. My first read was "impressions without clicks, so this is worthless ranking."

That might be the wrong conclusion. One likely explanation — and I want to be careful to call it a hypothesis, not a proven cause — is that some of those impressions are being answered inline by an AI assistant: the model pulls the page into its grounding set, writes the answer, and the user never clicks because they already got what they came for. I've written before about why AI citation is becoming the metric that replaces the click, and a high-impression, zero-click, mid-position query is exactly the shape that pattern leaves behind.

Here's the honest limit, though: I can't confirm it from the data I have. The search engine's webmaster API reports impressions, clicks, and positions — it does not expose whether an AI surface cited the page. So "these zero-click impressions are AI citations" is an inference from the shape of the numbers, not something I measured. It's plausible, it fits, and it's the explanation I'd bet on — but it's a bet, and treating it as confirmed would be the exact confidence gap I keep warning about. What I can say for certain is narrower and still useful: zero clicks at position 6–9 does not mean zero value, and reading it as failure would be a mistake.

So "zero clicks" on those queries isn't dead weight. It's the citation surface. The clicks I do get come from longer-tail queries lower in the cluster, ranking higher, where no AI summary intercepts them. Two completely different value mechanisms inside the same page, and you can only tell them apart if you stop treating clicks as the only outcome that counts. (If the GEO/AEO/citation vocabulary is new, I untangled the three terms here.)

What watching one engine hid

I would not have seen any of this from Google alone. Not the inversion, not the page that's invisible on one engine and page-one on the other, not the citation pattern. Google's data is internally consistent and tells a clean story — the article is the answer — and that story is true on Google and false everywhere else.

That's the part that actually unsettled me, and it's bigger than Bing. For years my working definition was Google = search. One dashboard, one set of numbers, one verdict on whether a page worked. But search has quietly become plural — Google, Bing, AI Overviews, Perplexity, ChatGPT, Gemini — and they don't agree, because they don't rank the same way or even reward the same kind of page. My mental model wasn't wrong so much as incomplete: I was reading one instrument and calling it the weather. The inversion was just the first thing that didn't fit, because it was the first time I looked at a second instrument.

The fix was boring and is the actual point: pipe the second engine into the same place as the first. Bing exposes a Webmaster API with the same kind of search-performance data Google's does — totals, top pages, top queries, positions — behind a single API key you generate in the tool's settings. I wired Google Search Console into a daily Telegram briefing months ago; adding Bing was the same shape of work pointed at a different endpoint. Now both engines land in the same thread, weekly, side by side — a short "this week vs last week" and a six-month trend so I can watch the curve. (That curve, for the record, has compounded every month from a near-zero start — visibility I had no idea was accruing because I wasn't looking.)

One gotcha worth saving someone an hour: the related IndexNow endpoint — the protocol that lets you ping Bing and others the instant a URL changes — returned 403 Forbidden for every request until I added a User-Agent header. curl and the browser don't care; the API does. No error message explains it. If your IndexNow submissions are silently rejected, send a User-Agent.

The takeaway

"Rank" isn't one number. The same page can be a winner and a loser at the same moment depending on who's asking, and the two biggest engines can disagree completely about content they're both looking at.

The practical move once you see the split is not to force the pages to fight — I'm not stripping the article of the terms it wins on Google to help the map, or vice versa. They've each won an engine. The right move is to let them, cross-link them so they reinforce instead of compete, and keep both engines on the dashboard so the next divergence isn't invisible for months.

But the durable lesson isn't about Bing at all. It's that treating one engine as the whole of search means your picture is incomplete by definition — and you won't know what you're missing until you instrument the engines you've been ignoring. Mine had a story in it the whole time. I just wasn't looking.

Originally published on tedagentic.com.

Chrome Showed the Data. Firefox Showed Nothing. The API Was Being Blocked.

Ted — Wed, 17 Jun 2026 15:26:31 +0000

I was looking at the same page in two browsers side by side. In Chrome: a full list of results, images, working buttons. In Firefox: a different, smaller set of placeholder cards with no images and no links — and a banner that read "0 results found."

Same URL. Same deployed build. No "works on my machine" excuse, because it wasn't working on the other machine — it was the same machine, two windows.

That contradiction is the whole story, and the cause was a class of bug that is almost impossible to catch in development: the browser was blocking my own API call, and I'd been developing in the one browser that didn't.

How the page is built

The page renders a list of results that come from a database. The pattern is the common one for a single-page app: the page ships, then a bit of JavaScript runs and fetches the data from a hosted Postgres/API service (Supabase, in this case) at its own domain — something like xxxx.supabase.co. The results come back, React renders the cards.

If that fetch returns nothing, the component falls back to a hardcoded placeholder list so the page isn't blank. That fallback is what Firefox was showing. So the real question was narrow: why did the fetch return nothing in Firefox but everything in Chrome?

The diagnosis

The server was fine. I could hit the API directly from the command line and get the full result set, HTTP 200. The database, the credentials, the query — all correct. Chrome proved it too: it ran the exact same JavaScript against the exact same endpoint and got the data.

The difference was the browser, and specifically what the browser allowed. Firefox ships with Enhanced Tracking Protection, and a lot of people add uBlock Origin or Privacy Badger on top. Those tools block requests to domains they classify as third-party trackers. The relevant move is in the browser's network panel: the request to xxxx.supabase.co wasn't failing with an error from the server — it was being blocked before it ever left, by the privacy layer.

From my code's point of view, the fetch just throws. data comes back empty. The component does what I told it to do with no data: show the fallback. No crash, no red console error a casual glance would notice. The page "works." It just works with nothing in it.

The whole bug fits in two columns:

   Chrome                      Firefox + tracking protection

   Page                        Page
    ↓                           ↓
   Fetch supabase.co           Fetch supabase.co
    ↓                           ↓
   Data arrives                Blocked
    ↓                           ↓
   Render results              Fallback cards
                                ↓
                               0 results

Same code reaching the same fork, and the third step decides everything.

Why this is so easy to ship

Here's the uncomfortable part. I build in Chrome. Most developers build in Chrome. Chrome, with no aggressive privacy extensions, happily makes the cross-origin call, so the page is perfect every single time I look at it. The failure only appears in an environment I never use during development.

And it gets worse, because the measurement is blocked too. My click tracking — the thing that tells me whether people are using the page — went through the same API domain. So the users who couldn't see the content also couldn't be counted not using it. The bug hides in the browser I don't test, and it erases its own evidence in the analytics. It's invisible twice.

The only reason I found it is that someone opened the page in Firefox and said "this looks broken."

Fix one: stop the fallback being a dead end

The first thing I did was cheap and defensive. The placeholder fallback had no links — a blocked user landed on dead cards and had nowhere to go. So I made every fallback card and every "quick pick" link to the real directory page. Now, whatever blocks the fetch, the visitor still has a working path forward.

This is worth doing regardless, but be honest about what it is: a safety net, not a fix. The user is still being denied the real content. They're just no longer stranded. To actually fix it, the call has to stop being blockable.

Fix two: make the call first-party

It's worth being precise about what I'm fixing here, because it's easy to mislabel. This isn't a Supabase problem — Supabase did everything right. It's a browser trust-boundary problem. The browser draws a line between "the site I'm on" and "some other domain," and privacy filters police that line. My data was on the wrong side of it.

So the fix isn't to change the database. It's to move the request to the trusted side of the line — to change what the browser sees:

   Before                       After

   mysite.com                   mysite.com
    ↓                            ↓
   xxxx.supabase.co             mysite.com/sb-api
   (third-party — policed)      (first-party — trusted)

Same data, same backend. The only thing that changed is whose domain the browser thinks it's talking to. You do that with a reverse proxy: route the API through your own domain so the browser only ever sees a same-origin request to the site it's already on. On Vercel that's a rewrite in vercel.json:

{
  "source": "/sb-api/:path*",
  "destination": "https://xxxx.supabase.co/:path*"
}

Then point the client at the first-party path instead of the vendor's domain:

const SUPABASE_URL =
  import.meta.env.PROD && typeof window !== "undefined"
    ? `${window.location.origin}/sb-api`
    : RAW_SUPABASE_URL; // dev + build-time stay on the direct URL

Now the browser makes a request to mysite.com/sb-api/..., which Vercel quietly forwards to the real API. To Firefox's tracking protection it's a call to the site you're already visiting — first-party, not a tracker, not blocked. As a bonus, the browser no longer has to negotiate cross-origin access at all.

I verified it the same way I found the bug — by hitting the new first-party URL and watching the real data come back through it — then opened the page in Firefox and got the full list, images and all, identical to Chrome.

The gotchas, because there are always gotchas

A reverse proxy in front of your whole data layer is a real change. Three things to check before you ship it:

Websockets don't ride through a simple rewrite. If you use the service's realtime/subscription features, a path rewrite won't proxy the socket cleanly. I got to skip this because the app uses none — but check first, or you'll trade a Firefox bug for a realtime outage.
Pin your auth storage key. Client libraries often derive the key they store the session under from the API URL. Change the URL and the key changes, and everyone silently gets logged out. Set the storage key explicitly to the original value so swapping the URL is invisible to existing sessions.
Keep server-side and build-time calls on the direct URL. Anything running outside the browser — a build step, prerendering, server code — has no ad blocker and no same-origin to honor. Sending it through the proxy is pointless and can be circular. Gate the rewrite to the browser only.

The crawler is just another blocked client

There's a tidy way to think about all of this. A browser with tracking protection is a client that won't run part of your page. A search crawler is also a client that won't run part of your page — it may not execute your fetch at all. Both end up looking at the empty shell.

So the durable answer isn't only the proxy; it's not making the page's existence depend on a fetch that some clients will never complete. I also prerender the real results into the page's static HTML at build time, so the meaningful content is in the document before any JavaScript — for crawlers, for blocked browsers, for anyone whose script never runs. The proxy fixes the live experience; prerendering makes sure there's something real there even when nothing runs at all.

What I took from it

A page that's perfect in Chrome can be empty in Firefox. If your data comes from a client-side call to a third-party domain, privacy filters will block it for a real slice of users, and you'll never see it in the browser you develop in.
Silent failure is the dangerous kind. A blocked fetch doesn't crash — it just renders your empty state. Make the empty state recoverable, but don't mistake that for a fix.
The blocked users are uncounted users. If your analytics and your content share an API domain, the people who can't see the page also don't show up as not seeing it. Test in Firefox with tracking protection on, and with an ad blocker — that's your blind spot, on purpose.
First-party proxy beats third-party fetch. Route vendor APIs through your own domain so the browser sees same-origin. Mind websockets, pin the auth key, keep build/server calls direct.
Treat crawlers and blocked browsers as the same problem. Anything that won't run your JavaScript needs the real content in the HTML already.

Underneath the specifics, this was the same shape as most of the worst bugs I find:

Human validation passed
        ↓
System validation failed
        ↓
Find the hidden assumption
        ↓
Fix the assumption, not the symptom

The hidden assumption here was four words long — "if it works in Chrome" — and the real version was "it works in the browser I happened to develop in." The fix wasn't really a proxy. The proxy was just how I retired the assumption.

The page had been broken for months for an entire category of visitors. It looked flawless the whole time — in the one browser I happened to use.

My Redirects Worked in the Browser. Googlebot Saw Soft 404s.

Ted — Sun, 14 Jun 2026 11:06:53 +0000

Google Search Console flagged six URLs on a client site as Soft 404. Every one of them returned a clean 200 OK when I curled it. So how does a page that loads fine get reported as "not found"?

That contradiction is the whole story, and the answer turned out to be a category of bug I'd been shipping without realizing it.

What "Soft 404" actually means

A hard 404 is honest: the server returns a 404 status, Google drops the URL, everyone moves on. A soft 404 is when the server returns 200 OK but the content looks like an error or an empty page to Google. The status line says "here's your page," the body says "there's nothing here." Google trusts the body.

On a single-page app, there's a very common way to produce exactly that.

The setup

The site is a React/Vite SPA, deployed on Vercel, with a prerender layer that injects real HTML for SEO so the crawler doesn't have to run JavaScript. Routes that are prerendered get proper content. Routes that aren't fall through to Vercel's catch-all, which serves the app shell — effectively the homepage HTML — at whatever URL was requested.

Hold onto that last sentence. It's the trap.

The fingerprint

All six soft-404s shared a pattern: they were old, renamed blog slugs. A few were year-suffixed posts I'd renamed (think /blog/some-guide-2025 → /blog/some-guide-2026). Two were older duplicates of a post that now lives at a cleaner URL. One was the same path on the www host instead of the apex.

Old URLs that should redirect somewhere. And in the codebase, three of them did have redirects. That's what made this confusing — I had redirects, and Google was still calling the pages broken.

The trap: the redirects were client-side

The three "redirected" routes used the SPA router's redirect component — the React-Router <Navigate> element. In a browser, it works perfectly: the app boots, the router matches the old path, and the user is bounced to the new URL before they notice.

But look at the order of operations from Googlebot's point of view:

Request /blog/some-guide-2025.
The route isn't prerendered, so Vercel serves the app shell — homepage-ish HTML, status 200.
Then the JavaScript would run and redirect — but the crawler has already been handed a 200 with the wrong content.

The redirect lives inside the JavaScript. The crawler's verdict is formed before the JavaScript runs. So Google sees a 200 response whose body is the homepage, served at a URL that's supposed to be a specific blog post — a page that resolves to nothing meaningful. Soft 404.

The other two had no redirect at all and went straight to the shell fallback — same outcome by a more direct route.

A client-side redirect is invisible to a crawler that judges the first response. If you want Google to treat a URL as moved, the move has to happen in the response itself, before any JavaScript: a real server-side 3xx.

The fix

The fix is to do the redirect at the edge, not in the app. On Vercel that's a redirects entry in vercel.json:

{
  "source": "/blog/old-slug",
  "destination": "/blog/new-slug",
  "permanent": true
}

"permanent": true emits a 308 (Vercel's permanent redirect), which Google treats the same as a 301. Now the first response Googlebot gets is "this moved, permanently, here" — no shell, no JavaScript, no ambiguity. The www variant was already handled by an apex-redirect rule, so once the apex path resolved correctly, the www one chained into it.

I added server-side redirects for the two that had none, pointed both old duplicates at the canonical post, and the client-side <Navigate> routes became harmless fallbacks behind the real ones.

The twist: most of them were already fixed

Here's the part that saved me a pile of unnecessary work — and that I almost skipped.

Before "fixing" the three that already had server-side redirects, I checked when Google last crawled them. The URL Inspection API returns lastCrawlTime. Every one of those dates was weeks before the server-side redirects had shipped. The pages weren't broken anymore. Google's report was a snapshot from the last time it looked, and it simply hadn't looked again.

GSC statuses are not live. They're the result of the most recent crawl, which can be a month stale on a low-traffic site. Before you re-fix something the report calls broken, check lastCrawlTime — you may be debugging a problem that no longer exists.

Submission is not a fix

The instinct, once the redirects were in, was to "submit the pages to Google." But you don't submit a redirect source — it's not content, it's a signpost. The Indexing API is for telling Google a real page changed. For redirected URLs, the correct lever is GSC → "Validate Fix" on the soft-404 report, which re-queues the crawl. The only thing worth submitting is the redirect targets — the live pages — so Google freshens those.

Resubmitting a URL Google already crawled doesn't change Google's mind. It just asks the same question again.

The other bucket: "crawled, not indexed"

Separately, a larger batch of pages on the same site had been sitting in "Crawled – currently not indexed" — and around two dozen of them cleared into the index over the same period. It's tempting to lump that together with the soft-404 fix, but it's a different problem with a different cause.

Soft 404 is a technical verdict: wrong response, fix the response. "Crawled, not indexed" is a quality verdict: Google fetched a real page and chose not to index it. You don't move that with redirects or resubmission — you move it with better content (the prerender layer putting real HTML in front of the crawler) and internal links that give the page a reason to matter. Those recoveries were the delayed payoff of content and linking work from weeks earlier, not anything I did that day.

Keeping the two buckets separate matters, because the fixes don't transfer. A 308 will never rescue a thin page, and a content rewrite will never fix a client-side redirect.

What I took from it

A 200 status doesn't mean Google sees a real page. Soft 404 is the body contradicting the status line.
On an SPA, any route that isn't prerendered falls through to the app shell — a 200 full of the wrong content. That's a soft-404 factory.
Client-side redirects don't exist as far as a crawler is concerned. Anything Google should treat as moved needs a server-side 3xx, before the JavaScript.
GSC reports are stale snapshots. Check lastCrawlTime before re-fixing.
Submission re-asks the question; it doesn't change the answer. Use "Validate Fix" for redirects; submit only live targets.

The redirects had been working the entire time — in the one place that couldn't see them.

Migrating Off OpenClaw Without Downtime — and the Offset That Made Hermes Look Dead

Ted — Sun, 14 Jun 2026 10:19:10 +0000

A while back I wrote a comparison of OpenClaw and Hermes — two open-source, self-hosted AI agents I run on the same bare-metal box, both wired to Telegram. The verdict was that they're complementary: OpenClaw as the dependable gateway for scheduled delivery, Hermes as the agent that builds context over time. Run both, I said. Don't choose.

Then I actually lived with both for a few more weeks, and the verdict started to move.

OpenClaw is gateway-first: a messaging hub that runs plugins, models, and cron delivery on a schedule you define. Hermes is agent-first: the messaging is just how you reach it, and the point is an agent that accumulates knowledge and needs less steering each time. On paper that made OpenClaw the better runtime for my scheduled jobs.

The migration started for a plainer reason than "one is better": I'd come to trust Hermes more with anything long-running. When I handed my OpenClaw setup a task with real weight, it would acknowledge it and then go quiet — the agent loop hitting a timeout and dropping the work before it finished. That isn't a verdict on the project; it's a mismatch between how I had it configured and the load I was putting on it. But trust is trust, and mine had moved.

Hermes, on the same machine, on a free OpenRouter model, just finishes. It holds the task, retries, and delivers — because the persistence is in the harness, not the model. That's the whole thing: the model matters less than whether the system around it gives up.

So I decided to migrate. But not the way you'd think.

The rule: don't turn anything off

The constraint I set for myself was that OpenClaw keeps running, untouched, the entire time. No big-bang switch. Two reasons:

It's my fallback. If the new path has a blind spot I haven't seen, I don't want to discover it by losing notifications.
I want to compare. Running both in parallel is the only way to see where each one lags before I commit. You don't get that from a one-way cutover.

So the plan became: deliver everything through Hermes in parallel with OpenClaw for a couple of weeks, watch them side by side, then retire OpenClaw's delivery one job at a time.

The part I underestimated: there wasn't one delivery path. There were four.

My notification stack is a pile of cron jobs — site monitors, system-health checks, morning summaries, scheduled reports. The important thing about them is that the agent never touches them. They're pure notifiers: each one computes a result and pushes it straight to Telegram. The agent only gets involved when I actually ask it something.

I assumed they all delivered the same way. They didn't. When I went looking, there were four different mechanisms across ~39 delivery points:

Scripts shelling out to the OpenClaw CLI
Node monitors hitting the Telegram Bot API directly
A shared bash helper used by a few jobs
One Python script posting straight to the API

There was no single switch to flip. Which is exactly why a clean cutover would have been a bad idea.

The mirror pattern

Instead of rerouting anything, I made the new delivery purely additive. One control script — hermes_mirror.sh — and every delivery point gets one extra line that calls it in the background:

bash jobs: hermes_mirror.sh "$MSG" &
Python jobs: subprocess.Popen([...])
Node jobs: child_process.spawn(...)

The helper waits a moment, then sends the same message through Hermes. The original OpenClaw send is never modified, never wrapped, never moved. If the mirror errors or hangs, the existing delivery already happened — it cannot be affected. And because everything funnels through that one script, there's a single kill switch: set ENABLED=0 and all mirroring stops at once. One offset value, one log, one place to reason about.

That additive property is the whole safety story. I wasn't migrating the system. I was running a second, shadow system next to it and watching.

The offset that made it look broken

Here's the part that actually taught me something.

I didn't want the two copies of every notification landing in the same second — that's noisy and makes them hard to tell apart. So I set the mirror to wait three minutes before sending. Comfortable separation. Seemed obviously correct.

Then I ran the first real test. Fired a notification. It showed up on the old bot. The new one… nothing. I sat there watching an empty chat, fully convinced the Hermes path had failed.

It hadn't. It was running exactly as designed — three minutes behind. The copy landed right on schedule, 180 seconds later, while I was already digging through logs looking for a bug that didn't exist.

The lesson stuck: in a parallel migration, "late" reads as "broken." A safety margin I added to keep things tidy became a false failure signal, because the only way I had to judge the new system was did the message show up when I expected it. Three minutes of silence and your brain files it under "dead."

But the delay itself wasn't the mistake. The mistake was adding a delay without adding any signal that the delay was intentional. If a mirrored notification is meant to land three minutes later, something should say so — a log line, a status ping, anything that separates "received, waiting" from "never arrived." Without that, silence and failure are indistinguishable. A safety mechanism with no observability doesn't make a system safer; it makes it ambiguous — and ambiguity, in a migration, is its own kind of failure.

I cut the offset to 20 seconds. Long enough that the two copies never collide, short enough that the new one clearly arrives with the original instead of looking like it got lost. Same mechanism, completely different read on whether it's working. (The better fix is still on the list: have the mirror log an explicit "received, holding 20s" line, so the wait is something you can see, not infer.)

The principles that fell out of it

By the time the offset was sorted, the migration had quietly handed me its own rules:

Never modify the primary delivery path first. The thing that works keeps working, untouched.
Add the new path alongside the old — purely additive. It can fail without taking anything down with it.
Keep one global kill switch. A single place to stop everything the moment it misbehaves.
Make delayed delivery visibly delayed, not silently delayed. A wait with no signal is indistinguishable from a failure.
Remove the old path only after watching a full execution cycle — including the jobs that only fire weekly.

None of them are clever. They're just what's left after you stop assuming the replacement works and start making it prove it.

The gotcha in automating the edit

Wiring one line into ~20 Python scripts, I wrote a small injector instead of editing each by hand. It dropped the mirror call in as the first line of each delivery function. Clean — except one script imported its dependency inside the function, below where my line now sat. So the injected call referenced a module that wasn't imported yet. Instant NameError, only in that one file.

Blind codemods across a heterogeneous pile of scripts will find the one file that doesn't match your mental model. Compile-check everything after, not just the ones you expect to fail.

The thing I expected to be hard and wasn't

I assumed I'd have to port OpenClaw's scheduled agent jobs — the ones where it actually reasons, not just delivers. Turned out they were all already disabled, superseded long ago by plain deterministic scripts. The only thing OpenClaw was still genuinely doing for me on a schedule was acting as the mailman. The agentic work I thought I'd be migrating didn't exist anymore. The real dependency was just delivery.

Where it stands

Both agents now deliver the full stream in parallel, Hermes trailing each original by 20 seconds. OpenClaw is untouched and still my fallback. The plan is to watch a complete cycle — including the jobs that only fire weekly — and then start removing OpenClaw's send per-job, one at a time, only once I've seen each one deliver cleanly through Hermes.

The migration isn't "done" — and that's the point. The mirror pattern turned it from a decision into an observation exercise. I never had to ask whether Hermes was ready; I could watch it prove it, one notification at a time, with the old system holding the floor the whole way.

That's the part I'll carry to the next one. When you're replacing a live system you don't trust yet, the move isn't to cut over and hope. It's to run the replacement in the open, right next to the thing it's replacing, and let it earn the handoff — visibly, on a schedule you can watch, with nothing torn out until it has.

GEO: What Generative Engine Optimization Actually Means for Your Content

Ted — Thu, 11 Jun 2026 08:30:10 +0000

Search didn't break. It restructured. The page that ranked #1 for an informational query now sits below an AI-generated summary that pulls from it — and the user never scrolls down.

That's the GEO problem in one sentence.

What Changed in the SERP

Old SERP for most informational queries:

┌─────────────────────────────────────┐
│  [Ads]                              │
│  1. example.com/answer              │
│  2. another-site.com/guide          │
│  3. wikipedia.org/wiki/topic        │
│  4. ...                             │
└─────────────────────────────────────┘

New SERP for the same query:

┌─────────────────────────────────────┐
│  [AI Overview]                      │
│   └─ synthesized answer             │
│   └─ 3-5 citation links (collapsed) │
│                                     │
│  [People Also Ask]                  │
│   └─ expands inline                 │
│                                     │
│  1. example.com/answer  ← you       │
│  2. another-site.com/guide          │
└─────────────────────────────────────┘

The user gets an answer before they see your link. CTR drops. Impressions stay the same or grow. That gap is the GEO signal.

On one site I run, impressions on an informational page grew 4x over three months while CTR dropped below 0.5% — the query cluster started triggering AI Overviews during that period. The page didn't lose rankings. It lost the click.

SEO vs GEO — What Each Optimizes For

Dimension	SEO	GEO
Target	Ranking position	Citation in AI summary
Signal	Backlinks, authority, E-E-A-T	Structure, clarity, entity match
Win condition	User clicks your link	AI quotes your content
Metric	Clicks, CTR	Impressions → brand recall
Risk	Algorithm update	AI answers the query fully

GEO and SEO aren't competing strategies. GEO is what happens to SEO when the SERP adds a layer above organic.

Which Query Types Are Most Exposed

Not every query gets an AI Overview. The exposure depends on query type:

HIGH GEO EXPOSURE
─────────────────────────────────────────
  Informational   "what is X"
                  "how does X work"
                  "X explained"
                  "X rules / laws / limits"

MEDIUM GEO EXPOSURE
─────────────────────────────────────────
  Comparison      "X vs Y"
                  "best X for Y"
                  "X alternatives"

LOW GEO EXPOSURE
─────────────────────────────────────────
  Transactional   "buy X"
                  "X near me"
                  "book X in [city]"
                  "X price"

Transactional and local queries rarely trigger AI Overviews. Google still routes those to maps, commerce, and organic listings. That's where clicks live.

What GEO-Optimized Content Looks Like

AI systems appear to favor content that answers cleanly and early. The structure matters more than the word count.

Not this:

Introduction paragraph...
Background on the topic...
History and context...
What experts say...
[Actual answer buried at paragraph 6]

This:

[Direct answer in first 2 sentences]

## Key points
- Fact with number
- Fact with source signal ("as of 2026...")
- Specific limit, rule, or definition

[Supporting detail below]

The AI Overview pulls from the top of the page. If your answer is buried, a cleaner competitor gets cited instead — even if you outrank them.

Other signals that increase citation likelihood:

Defined terms in headings (##)
Numbered lists with specific values (not vague)
Dates and freshness markers
Tables for comparison content
No filler text between the H1 and the answer

The Downstream Intent Play

Here's the strategic shift GEO forces: stop trying to own queries where AI answers the whole thing. Own what comes next.

User query: "what are the cannabis limits in Colorado"
     │
     ▼
AI Overview answers it fully
     │
     ▼
User now knows the limit — and wants to act on it
     │
     ├──► "420-friendly hotels in Colorado"      ← transactional
     ├──► "cannabis tours in Denver"             ← transactional
     └──► "where to buy in [city]"               ← local

The informational query feeds intent. The downstream query converts. GEO-aware content strategy maps the full path — not just the definition page, but every action a user might take after getting the answer.

If your site only has the definition page, AI Overview cuts the journey short. If you have the definition page and the action pages linked from it, you capture both the citation signal and the conversion traffic.

Practical GEO Checklist

Content structure:

[ ] Direct answer in the first 2 sentences
[ ] No more than one paragraph before the first heading
[ ] Key facts in lists or tables, not buried in prose
[ ] Dates and specifics (not "recently" — use the year)
[ ] Headings match the exact phrasing of common questions

Internal linking:

[ ] Every informational page links to a downstream action page
[ ] Transactional pages don't depend on informational traffic alone
[ ] Related guides section at the bottom of every post

Monitoring:

[ ] Track impressions vs clicks separately — divergence signals AI capture
[ ] Flag pages where impressions grow but CTR drops below 1%
[ ] Check which queries trigger AI Overviews for your target keywords

The Honest Summary

GEO doesn't replace SEO work. Rankings still matter for non-AI-captured queries, and being cited in an AI Overview still requires ranking well enough to be in Google's index pull.

What GEO changes is the goal for different content types. Informational pages: optimize for citation and downstream linking, not for direct clicks. Transactional pages: optimize for clicks, those are still yours to win.

The operators who will get hurt are the ones with sites built entirely on informational content with no action layer underneath. The ones who built the full funnel — definition → comparison → booking — are fine.

Build the full funnel.