DEV Community: Ted

Clicks Rose 73%. I Shut the Site Down.

Ted — Tue, 25 Aug 2026 13:21:04 +0000

In June I fixed a real problem on a small editorial site I run.

The site was a client-side app — the kind where the server sends a nearly empty HTML shell and the browser assembles the actual page afterward. That works fine for people. It works badly for search crawlers, because the description a search engine shows underneath your link in the results comes from the HTML the server sent. Every page on this site was sending the same generic one. Fifty-odd pages, one description, all identical.

So I built a step into the deploy that renders each page's real title and description into the file the server hands out. Crawlers started seeing per-page text instead of one repeated boilerplate line.

It worked. I can show you that it worked.

The numbers said yes

Two things get measured here. Impressions are how many times one of your pages appeared in someone's search results. Clicks are how many times someone actually clicked through. Click-through rate is the second divided by the first — of the people who saw you, what fraction picked you.

The fix was aimed squarely at click-through rate. A better description should win more of the people already seeing you, without changing where you rank.

Comparing the 28 days before the fix to the 28 days ending this week:

	Clicks	CTR
Before the fix	11	0.47%
Now	19	0.56%

Click-through rate up about 19%. Clicks up 73%.

That is the shape you want. The intervention targeted a specific mechanism, the mechanism moved, and it moved in the predicted direction. If I'd put that on a dashboard and walked away, I'd have called it a win and gone looking for the next thing to fix.

Two months later I removed the site from all of my monitoring and retired it.

The number I wasn't watching

Here's the third column, the one that isn't on anybody's dashboard.

Not how many clicks did the site get — how many distinct pages earned anything at all.

	Pages with ≥1 impression	Pages with ≥1 click
Before the fix	13	5
A month later	12	5
Now	10	5

Ten. Out of about fifty.

And the same five pages have earned every single click for three consecutive months. Not five pages of a rotating cast — the identical five. The top five pages account for 91% of all impressions the site receives.

So both readings are true at once. The clicks went up 73%. The surface producing those clicks contracted by nearly a quarter. I got better at converting a shrinking audience.

How an aggregate hides a shrinking surface

This isn't a paradox, it's arithmetic, and it's worth being precise about because the same trap sits under a lot of small-system metrics.

Clicks is a sum over pages. Any sum can rise while its terms are disappearing, as long as the survivors grow faster than the losses. On a big site this washes out — thousands of pages, and one page's fate is noise. On a site where five pages produce 91% of everything, the total is those five pages. It stops being a measure of the site and becomes a measure of a handful of URLs.

The distinction that matters: clicks measured how well my best pages were doing. Coverage measured whether the site could still get new pages in front of anyone. Those are different questions, and only the second one tells you whether there's a business here.

My fix improved the first. It was never capable of improving the second — better descriptions help pages that already appear in results. It does nothing for pages that never appear at all. When I audited this site in June, nineteen of its pages had been crawled and passed over, and eighteen had never been crawled at all. Better descriptions were never going to reach either group. Two months on, the count of pages earning anything is lower than when I started, so whatever happened to those thirty-seven, it wasn't a breakthrough.

I shipped a change that could only help the pages that needed help least. And the metric I'd chosen to judge it by was the one metric that couldn't tell me that.

The detail that ended the argument

One more number, because it's the one that made the decision obvious rather than merely defensible.

The site's single best query is a person's name — a historical figure the site has a long article about. Over the last month that query put the page in front of people 454 times, at an average position of 8.6. That's the bottom of the first page of results. Respectable. Hard-won.

It earned one click. A click-through rate of 0.22%.

At position eight or nine you'd expect somewhere between 1.5% and 3%. Getting a fifth of the floor isn't a snippet problem, and it isn't something a better description fixes. It's what happens when you rank underneath a Wikipedia entry and one of those boxed summaries search engines assemble on the right-hand side. The question gets answered on the results page. The user's need is met before your link is a candidate. You are technically visible and functionally absent.

Which reframes the whole asset. The queries this site ranks for are queries that get answered without anyone leaving the search results. That is not a fixable snippet. That is the market telling you what your content is worth to it.

The part I got wrong

I laid all of this out and then offered to set up a checkpoint: re-run the exact comparison in a month, deliver it automatically, decide then with fresh numbers.

The answer I got back was that eight months of no movement was already the measurement.

That's correct, and I should have said it myself instead of proposing more data collection. The trend had twelve months in it. Best month in the entire year: 16 clicks — this month, the one that supposedly shows the fix working. Second best: 15, back in March, three months before the fix existed. The whole twelve-month range is 4 to 16. Impressions had peaked in February and were down 49% from that peak. Every one of those facts was already in front of me when I offered the checkpoint.

There's a failure mode here that looks like rigor and isn't. When a trend is genuinely ambiguous, more measurement is the right call. When a trend has been flat for eight months across every framing you can apply to it, proposing another month of measurement isn't rigor — it's deferring a decision while appearing thorough. The tell is whether you can describe, in advance, a result that would change your mind. I couldn't. I'd already concluded the only remaining lever was a multi-month authority build that wasn't going to happen. Wanting one more datapoint was me not wanting to say so.

What I'd watch instead

For anything small enough that a handful of pages dominates the totals, the aggregate is close to useless as a health signal. It tells you how your winners are doing, and your winners are usually fine — that's what makes them winners.

The two numbers that actually carried information:

Distinct pages earning any impression, tracked over time. Rising means the system still has reach. Falling means it's consolidating toward a few survivors, which is what dying looks like from the inside before it looks like anything on a revenue chart.

Whether anything outside the incumbent set has broken through. Not "did clicks grow" but "did a page that wasn't earning before start earning." On this site the answer was no, for three straight months, while clicks climbed 73%.

If both of those are flat or falling, a rising total isn't recovery. It's a smaller thing being measured more favourably. I had that data the whole time, in the same account, one query away. I just wasn't asking for it, because the number I was asking for kept coming back green.

Every Page Returned 200. There Was No Database.

Ted — Mon, 24 Aug 2026 15:23:46 +0000

My travel site was completely non-functional for about four hours today and never once went down.

That is not wordplay. Every route returned HTTP 200 the entire time. Time to first byte was normal. The HTML came back full of content — property names, headings, structured data, the lot. An uptime monitor pointed at any page on the site would have recorded a flawless afternoon.

Nobody could book anything, because there was nothing to book.

What actually broke

The site's backend runs on a managed platform — a hosted wrapper around Postgres that bundles the database, authentication, file storage and serverless functions into one project. The workspace was on the free tier. Its allowance ran out and the project was paused.

There was no option to settle a balance and carry on. The dashboard offered exactly one action: upgrade to a $25/month plan. That is worth separating from the usual story about an operator forgetting to pay a bill, because it is a different failure — the free tier had been running production for nine months, and the notice that it would stop was the stopping itself.

The first thing I checked was DNS, and that told me how complete this was:

$ dig +short @1.1.1.1 <project>.supabase.co
$ dig +short @8.8.8.8 <project>.supabase.co
$ dig +short @1.1.1.1 supabase.co
76.76.21.21

Two independent resolvers, no answer. The parent domain resolved fine, so this was not a DNS outage — the subdomain had been withdrawn. NXDOMAIN, not a pause page.

One exhausted allowance took four subsystems with it at once: the database, every login, every uploaded image, and the edge function that records outbound clicks. They felt like separate concerns right up until they shared a fate.

What a visitor saw

Here is the part that has stayed with me.

The site is a React SPA with a prerender step: at build time, every route is rendered to static HTML with its data baked in. That HTML is what a crawler gets, and it is why curl looked reassuring:

$ curl -s https://<site>/hotels | grep -o "Patterson Inn"
Patterson Inn

The content was right there. So for a few minutes I believed the damage was limited to logged-in features.

Then I rendered the page in a real browser instead of reading its source. React hydrated, fired its query at a host that no longer existed, got nothing, and re-rendered the component with an empty array.

Found 0 verified stays.

Every filter read zero. The listings area showed skeleton placeholders. The page was beautifully styled, fully responsive, instantly loaded, and contained nothing whatsoever.

I checked the one thing that actually pays for the site:

$ curl -s https://<site>/hotels | grep -c "affiliate"
0

Not a single outbound booking link exists in the static HTML. Those URLs come from the database at render time. So there was no accidental fallback, no degraded-but-working state. Just a confident, empty shop.

Why nothing alerted me

I found out because I happened to open the platform dashboard for an unrelated reason.

Think about what a conventional check would have needed to notice. HTTP status? 200. Response time? Fine. Does the HTML contain expected text? Yes — the prerendered content was intact. Is the page non-empty? Very. Does the site load? Beautifully.

Every signal a normal monitor collects was healthy, because every one of them measures whether the server answered, not whether the answer meant anything. The failure lived entirely in the gap between those two questions.

That gap is not exotic. Any site that renders static shells and fills them from an API has it. The more thorough your prerendering, the wider it gets — because the static layer keeps looking healthy long after the dynamic layer has died.

The crawler got a better experience than the customer

There is a genuine consolation, and it is a strange one.

Googlebot does not execute JavaScript on the schedule a visitor's browser does. It was served the prerendered HTML — 623 routes of intact content, correct structured data, working internal links. Throughout the outage, the search engine's view of the site was perfectly healthy.

So the prerendering protected the index and abandoned the customer. For a nine-month-old domain still being assessed, that is the more expensive of the two to lose, and I would not have chosen differently. But it is worth being clear about what was protected and what was not, because "the prerendering saved us" is only half true and the other half was every booking that afternoon.

Resolving is not serving

One more thing, because it cost me the backup.

Before paying, I set a script running that polled DNS and would export everything the moment the host came back — the idea being that if the alert came at 3am, the data would already be on my disk.

It fired at 15:42. It captured nothing.

DNS came back roughly forty minutes before the service did. My readiness check probed the API root, got a 401, and treated that as life — a 401 is a valid HTTP response from a live server, after all. It then requested every table and received 521 for each one, which is the CDN saying it cannot reach the origin. The script dutifully reported sixteen tables as "not exported" and exited pleased with itself.

The fix is to probe something that only works when the thing actually works:

# before: any response means alive
code=$(curl -s -o /dev/null -w '%{http_code}' "$HOST/rest/v1/")
[ "$code" != "000" ] && break

# after: a real table, and only 200 counts
code=$(curl -s -o /dev/null -w '%{http_code}' \
  "$HOST/rest/v1/hotels?select=name&limit=1" -H "apikey: $KEY")
[ "$code" = "200" ] && break

Same mistake as the monitoring, one layer down. I asked "did something answer?" when the question was "did the thing I need work?"

The re-run pulled 91MB: 220 countries, 143 dispensaries, 58 properties, and 563 storage objects, none of which failed.

The reframe

Uptime is a measurement of whether your server responds. For a static site those are the same thing, which is why the convention exists and why it goes unexamined. For a data-driven site they are different questions, and the difference is the entire business.

A 500 would have been better. A 500 pages an on-call rota, trips a monitor, sends an email. What I had instead was a site that passed every automated check while converting nothing, and would have kept doing it for as long as I did not personally look.

So the health check I actually needed was never about the server. It was about the data:

Fetch the listings endpoint. Assert the count is greater than zero.

Two lines. It would have caught this within a minute, and it is the only check in this entire incident that would have.

The corollary is a build I have not made yet: the prerender already pulls every listing at build time, so the client could fall back to that baked-in data when a live fetch fails, instead of rendering zeros. Stale listings still sell. Zero listings sell nothing, and look immaculate doing it.

Two Pages Ranked for It. Neither Answered It.

Ted — Sun, 23 Aug 2026 17:29:03 +0000

A page on a site I run collected 4,347 impressions over twelve months and produced zero clicks. Average position 8.1 — first page, most of the time.

Zero clicks at position 8 is a strange number. Position 40 with no clicks is just losing. Position 8 with no clicks means Google repeatedly decided you belonged in front of people, and every one of them chose something else.

The obvious read is a CTR problem: bad title, weak description, rewrite it and move on. That read is wrong often enough that it's worth knowing what to check first.

Start with the queries, not the page

Search Console will give you the queries per URL if you filter by page and pull the query dimension. That's the step people skip, because the page-level table is right there on the first screen and the query breakdown is two clicks deeper.

Every query hitting my country-level page looked like this:

amsterdam coffeeshop rules tourists 2026        19 impressions   pos 9.6
amsterdam coffeeshops tourists allowed 2026     12 impressions   pos 8.2
amsterdam cannabis laws tourists 2026           12 impressions   pos 8.9
can tourists buy cannabis in amsterdam ...      14 impressions   pos 6.7
amsterdam coffeeshop rules tourists 2026 offic   6 impressions   pos 9.8

Not one of them was about the country. All of them were about one city inside it.

I already had a dedicated page for that city. So I checked which of my URLs ranked for the single biggest query:

--- pages ranking for "amsterdam coffeeshop rules tourists 2026"
   /blog/amsterdam-coffeeshop-guide-2026     0 cl   35 impr  pos 7.6
   /world/europe/netherlands                 0 cl   19 impr  pos 9.6
   www./world/europe/netherlands             0 cl    4 impr  pos 9.2

Three of my own URLs, one query, zero clicks between them. The third is a www duplicate from a redirect that returns 307 instead of 301 — a separate bug, and a story for another day.

The cannibalisation was real, but it wasn't the disease

The country page had a hand-written title override. Someone — me — had noticed the city got more search volume than the country and retitled the country page to chase it. So a page about a nation's legal framework went to market wearing a city guide's title, against my own actual city guide.

That's textbook keyword cannibalisation and it's worth fixing. The country page now carries country intent: legal status, the national tolerance policy, possession limits, how different cities diverge. The city guide keeps the city questions.

But splitting them doesn't explain zero clicks. Two pages splitting a query should produce some clicks, just fewer than one strong page. Zero is a different signal.

Read the queries again

Look at what those searches actually contain. Not the nouns — the modifiers.

2026. official. allowed. tourists 2026 official.

Those aren't people asking how coffeeshops work. Those are people checking whether a specific thing happened. There had been a long-running proposal to bar non-residents from buying cannabis in Amsterdam — it was back in the news in February 2026, and it was dropped from the city's coalition agreement on 3 June 2026.

So I searched my own two pages for the word "ban."

Zero occurrences. On either one.

I had two pages ranking on the first page of Google for a question neither of them answered. Google put me in the room because my pages were about the right subject. Every searcher then picked one of the seven results that actually said whether the thing happened.

Impressions measure adjacency. Clicks measure answers.

That's the reframe I walked away with.

An impression is Google saying "this page is topically near the query." A click is a human saying "this page looks like it contains my answer." They are not two points on the same scale, and treating a click-through rate as a headline-writing score conceals the case where the headline is honest and the page simply doesn't have the answer in it.

Zero clicks at position 8 isn't "your snippet is unappealing." It's right neighbourhood, wrong door.

Which means the fix isn't in the <title>. If I had done the obvious thing — rewritten the title to be punchier — I'd have made the page more attractive at describing something the searcher didn't ask about, and the number wouldn't have moved. Then I'd have concluded the query was just low-intent and stopped looking.

What actually shipped

Two changes, in this order:

Separate the intents. The country page answers country questions; the city page answers city questions. They stop bidding against each other.
Answer the question that's being asked. The city guide now leads with the outcome — the residents-only rule was debated for years, was back on the agenda in February 2026, and was left out of the June coalition agreement; the shops stay open; the tourist tax went up instead.

While I was in there I found a second version of the same failure. The page said "no consumption on the street." The actual rule is a €100 on-the-spot fine in four named districts, in force since May 2023 and actively enforced. I had turned a specific, checkable, useful fact into a vague one — which is its own kind of not-answering.

The check worth stealing

For any URL with lots of impressions and few clicks:

Pull the queries for that URL, not the page-level average.
Ask whether those queries match what the page is about. If they don't, you have an intent mismatch, not a CTR problem.
Query the same terms against all your URLs. If more than one comes back, you're splitting your own vote.
Then do the dumb thing: search your page's text for the key noun in the top queries. If it isn't there, no title rewrite is going to save you.

Step 4 took ten seconds and explained a year of zero clicks.

The part I'd want a reviewer to push back on

I can't prove the fix worked yet, and I'm suspicious of posts that skip that. Retitling resets how a page is understood, and it usually gets worse before it gets better. The honest position is that I've corrected a page that was demonstrably failing to answer the question it ranked for, and the measurement comes later.

What I'm confident about is the diagnosis, because it's not a theory about ranking — it's a text search. The word wasn't on the page.

The Commit Was Named After the Wrong Half

Ted — Thu, 20 Aug 2026 00:44:55 +0000

Two weeks ago I shipped a fix to my travel site with a commit message that named the wrong cause.

The headline change was real work: 220 country pages had never been prerendered, so every crawler that asked for one got the raw single-page-app shell — identical bytes for all 220, canonical pointing at the homepage. Google had filed them as duplicates of each other and of the front page, which is exactly what they looked like. Fixing it meant pulling a 2,800-line data structure out of a React component (it carried icon references, so no build script could import it), and the component went from 4,343 lines to 1,417. Call it 4,600 lines changed across nine files.

While wiring that up I found a second bug, unrelated except by proximity, and fixed it in the same commit almost as an afterthought. It was a lookup: one function building a URL from the wrong source. Maybe fifteen lines.

I have two weeks of search data now. The afterthought did most of the work.

What the numbers actually said

The obvious way to check whether a fix worked is to look at the thing you fixed and see if it went up. It went up. I nearly stopped there.

The problem is that everything went up that week — the whole site was 35% ahead, and I hadn't shipped anything to most of it. So "it went up" was worth nothing on its own. I needed a control: some part of the site with no structural changes in the same window, to measure the tide against the boat.

The blog section was the control — untouched in those weeks. It ran 19% ahead. That's the tide.

Against that baseline, comparing the eleven days before the deploy to the nine days after:

The 220 newly prerendered country pages: +41%
The hub page that links to them: +171%

The hub also moved from an average position in the high teens into the low teens. And here's the detail that killed my original story: 142 of those 220 country pages were already earning impressions before the fix. Only eight pages showed up for the first time afterward.

So the pages I'd spent two days rescuing from duplicate hell were, mostly, not in duplicate hell. They were doing fine. The page that improved dramatically was the one I'd changed almost by accident.

The fifteen lines

Here's what the afterthought fixed.

The hub page renders lists of countries, each one a link. It built those links from the database: take the row's region, take the row's slug, join them into a path. Straightforward.

But the router doesn't resolve URLs against the database. It resolves them against a static data file. And ten countries disagreed between the two.

Some disagreed on spelling. The database said united-kingdom, the router knew uk. The database said saint-lucia, the router knew st-lucia.

Some disagreed on something more interesting: which continent a country is in. The database filed Georgia, Armenia and Azerbaijan under Asia. The static file has them in Europe. Both are defensible — that's a genuinely contested boundary, and the two sources had been populated by different people at different times, each making a reasonable call.

So the hub was emitting links to ten URLs that did not exist.

A dead link that returns 200

If those links had 404'd, I'd have found them in a week. Every link checker I run would have flagged them, and a 404 in a crawl report is unambiguous.

They didn't 404. The route pattern /world/[continent]/[country] matched fine — asia is a plausible continent string and georgia is a plausible country string. The page rendered. It just had no country to render, so it fell through to the generic world-map copy. Status 200. Real HTML. A <title>, a canonical tag pointing at itself.

That is not a broken link. That is a duplicate factory: a URL that manufactures a near-empty page, declares itself canonical, and gets vouched for by an inbound link from an indexed hub page. Ten of them, all generating the same generic content, all pointing at themselves, all endorsed by the one page in that section with the most authority.

And nothing in my toolchain could see it. The link checker asks "does this return an error?" — no. The typechecker asks "is this a valid string?" — yes. The build asks "did anything throw?" — no. The sitemap didn't list these URLs, so a sitemap audit wouldn't surface them either. The only signal was in the crawler's opinion of the section as a whole, which is not a signal you can grep for.

I've written a lot recently about facts stored in multiple places drifting apart. This is the same failure with a different payload: the identifiers drifted, not the facts. And identifier drift is sneakier, because a wrong fact eventually reads wrong to a human, while a wrong slug produces a page that looks perfectly fine — just not the page anyone meant.

When two systems disagree about what a thing is called, you don't get an error. You get a phantom.

The fix, and what makes it durable

The lookup now resolves the same way the app itself does — same normalization, same alias table — and if a database row has no corresponding page, it renders as plain text instead of a link. That last part matters more than the matching: the failure mode is now "no link," not "link to a URL we hope exists."

I checked the ten offenders live before writing this. /world/asia/georgia now serves the real Georgia page with a canonical pointing to /world/europe/georgia. The united-kingdom and saint-lucia spellings resolve and canonicalise to uk and st-lucia. The phantom path is closed.

The build also now refuses to run if the data module loads zero entries, with an error message that says why: every country page would deploy as an unprerendered duplicate of the homepage. That's the pattern I keep landing on — the check that matters isn't the one that verifies success, it's the one that makes the specific silent failure loud.

What I won't claim

Two things I can't support, and I'd rather say so than let a tidy story stand.

About two days of the improvement predates the deploy. The hub started climbing on the 6th; the commit landed on the 8th. The steepest single-day jump is the day after the deploy, and the position improvement lines up, but a chunk of that curve started before I touched anything and I have no explanation for it.

There was no confirmed algorithm update to attribute the rest to. Third-party trackers spiked at the start of the month and the forums called it an update, but Google logged no ranking incident in that window and has confirmed nothing. So the site-wide lift is real, measurable, and unexplained.

Also worth saying: this is visibility, not traffic. The hub took 1,891 impressions last week and 10 clicks. It's a section where the answer gets consumed on the results page, and making more of it eligible to be seen hasn't changed that. I could also only see about 62 of those 1,891 impressions at the query level — the rest are anonymized — so I know that it rose, not what it rose on.

What I'd carry to any site

Name the commit after the mechanism, not the effort. I named mine after the part that took two days. The part that took fifteen minutes moved the number. Effort is not evidence.
A link that returns 200 can still be broken. Any route with a wildcard segment will happily serve a page for a parameter that means nothing. Those aren't dead ends, they're duplicate generators, and no link checker will tell you.
Build links from the same source the router resolves. If your URLs are constructed from data and interpreted by code, those two need one shared answer — or they will disagree, quietly, in the ten cases nobody thought about.
Prefer "no link" to "probably a link." Failing to render a link is a visible, harmless absence. Rendering a link to a URL that resolves to nothing is an invisible, compounding cost.
Always measure against a control. Something you didn't touch, in the same window. Without it you cannot tell your fix from the weather, and the weather that week was worth 19%.
Publish the part you can't explain. Two days of my improvement have no cause I can name. That's not a hole in the analysis; it's the honest shape of it.

Three of the First Four Alerts Were the Question's Fault

Ted — Thu, 20 Aug 2026 00:24:52 +0000

Last week I turned my data audit into a build step: a check that runs before anything else and fails the build when the database and any static copy of my travel site's legal-status data disagree. It ended the era of the site contradicting itself.

It did nothing about the site agreeing with itself on something false.

That's not a hypothetical. The most expensive error the whole project found was a country whose law changed in January while every copy on my site — database, data files, search index — kept saying the old thing in perfect unison. Internal consistency was the camouflage. No diff between my own sources could ever have caught it, because every internal source was equally behind the world.

A build gate proves agreement. Agreement is not truth. Something has to look outside.

You can't diff against the world, but you can sample it

The naive version of "look outside" is another audit — a human session checking primary sources jurisdiction by jurisdiction. I've done three of those now, and I know exactly what they're worth: they're correct the day they ship and they decay from that morning on. Laws don't change on my audit schedule.

So the outside check became what the inside check became: a scheduled job. Once a week, a script asks a web-connected model — one that searches and cites, not one answering from training memory — for the current legal status of about fourteen jurisdictions, and compares each answer to the corresponding database row.

Fourteen, not all 271, because the selection is doing the real work:

A hot list is checked every single run: the highest-traffic pages plus the jurisdictions with active legislative motion — the places where being a month stale costs the most.
Everything else sits on a rotating cursor: eight per run, round-robin, so every row on the site gets sampled roughly twice a year without any run costing more than a few cents. The whole thing runs on about seven cents a week.

Two rules were non-negotiable, both inherited from the build gate:

It never writes. A model's answer is not data — it's a tripwire. When the model and the database disagree, the script sends me a flag that says, in so many words: verify this against primary sources, do not trust it. Every fabricated-but-plausible fact I've written about in this series was an AI-generated value that skipped exactly this step. The pipeline is model → flag → human → primary sources → fix. The model never touches the last three.

A run that can't check anything is a failure, not a quiet pass. If the database is unreachable or every query errors, the job exits nonzero and my monitoring treats it like any other broken cron. "Couldn't verify" reported as success is the most expensive bug a verification step can have — same rule, new layer.

Each flag also alerts exactly once per claimed value. A definitional disagreement I've decided not to act on shouldn't renag me weekly; an alert channel that repeats itself gets muted, and a muted tripwire is decoration.

First run: four alerts, three of them mine

The first live run checked thirteen jurisdictions and raised four flags. Three of them said the same alarming thing: states my database grades as medical — real, functioning patient programs — were, according to the model, illegal.

All three flags were wrong. And the model hadn't failed. My question had.

I'd phrased the prompt the way I think about the site: "what does an ordinary adult visitor face here?" For a medical-only state, the honest answer to that question is prohibition — a visitor has no local patient card and no way to get one. The model answered the question I asked, correctly. My database answers a different question: what is the jurisdiction's regime. Medical program: exists. Same territory, same statutes, two defensible grades — because grading needs a rubric, and my dataset and my prompt were using different ones.

This is the part I'd generalize furthest: a comparison is only as strong as the rubric both sides share. Last month I resolved twenty-six "conflicts" between two of my own data stores by discovering they meant different things by the same word, and the fix was writing the definition down at the top of the data files. The first run of the sampler was the same bug at a new boundary — this time between the dataset and the question being asked about it. The fix was the same too: the prompt now carries the dataset's own written definition, verbatim rules included, so the model grades with the rubric the data was graded by. Re-ran the three states: all came back medical, matching. The flags dissolved.

A sensor doesn't measure the world. It measures the world through its question. Calibrate the question first, or every reading is an artifact.

The fourth flag was real — and it beat me at my own rule

One flag survived the prompt fix. A small New England state sat in my database as medical. The model said decriminalized, and cited its reason: the state removed jail for small-amount possession back in 2017 — a civil violation, a hundred-dollar fine, no cell.

Here's what makes this one uncomfortable. My dataset's written definition includes a tiebreaker for exactly this situation: when a jurisdiction is both medical and decriminalized, the status that describes possession by an ordinary person wins. I wrote that rule. I applied it across an entire continent's worth of countries a week earlier. And I had still left this state graded by its program instead of by its possession law — the value predated the rule, and nothing had forced it through.

The model wasn't smarter than the dataset. It was more consistent with the dataset's own rules than I had been. That's what a good tripwire does: it doesn't need to out-think you, it just needs to apply your stated policy without your habits.

The flag went through the full pipeline: verified against primary sources (the 2017 statute is current law; the fine schedule is unchanged), then fixed in the database and the static file in the same commit — because the build gate from last week fails if the two move separately. The page's headline counts recomputed themselves at build time, since they derive from the data. One wrong prompt and one wrong row, both found and fixed on day one, for the price of a coffee's loose change per year.

What I'd carry to any dataset that describes the world

Consistency checks certify agreement; something else must certify truth. A fact wrong in every copy at once is invisible to every internal comparison you will ever write.
Sample the world on a schedule, weighted by cost of staleness. Hot set every run, full rotation over months. You don't need to check everything weekly — you need everything checked eventually and the expensive things checked often.
A web-connected model is a tripwire, not an oracle. Its disagreement is a signal to open primary sources. Give it no write path — not because it's usually wrong, but because "usually" is doing unpriced work in that sentence.
The prompt must carry the dataset's rubric. If your data grades by a written definition, the question you ask the model must include that definition, or you'll spend your alert budget on vocabulary mismatches.
Expect the first flags to audit the sensor. Three of my four were the question's fault. That's not a failed launch; that's what launching a measurement instrument looks like.
Alert once per finding. A check that nags gets muted, and a muted check is worse than none, because you believe you have coverage.

This series has been one long inventory of places the same fact can be wrong: four data copies, then the prose, then the agreement between copies, and now the gap between all of them and the world. Each layer's check certifies something the next layer can't see. The build gate catches my copies lying to each other, minutes after it happens. The sampler catches all of them being wrong together — and its first real catch was a fact my own rulebook said I should have already fixed.

The Audit Became a Build Step

Ted — Wed, 19 Aug 2026 19:54:25 +0000

I have audited my travel site's legal-status dataset twice. The first audit found the same fact stored in four places, drifting independently. The second found the drift had a fifth home — the prose — and ended with me writing that a half-derived page contradicting itself "is not a risk, it's a schedule."

Six weeks later the schedule delivered again. A country's status was wrong in one file and right in another. A different country sat in the wrong continent in the database, which quietly broke the links a component built from it — two dead links on a live page, generated from data everyone had already audited.

Both audits were real work and both were correct the day they shipped. That's the problem. An audit certifies a moment. The dataset has four writers — an admin panel, an agent editing files, me in a code editor, the occasional migration — and nothing in the system held the agreement after the auditor left. Every writer produces output that is valid on its own. The typechecker is happy, the build is green, and the site contradicts itself, because validity is checked per file and truth lives across files.

So the third audit is not a session. It's a script, and it runs before anything else in the build.

What the check actually does

The site's legal facts live in a Postgres database and four static TypeScript files — one for US states, one feeding the country detail pages, one feeding the maps, one feeding search autocomplete. The check pulls the live tables over the public read-only API, loads the four static files, and compares every value that appears in more than one place.

Two design decisions matter more than the diffing.

It loads the app's real modules, not a regex scrape of them. The check bundles the actual TypeScript files and imports them, then reuses the app's own slug transforms, name normalization, and merge logic. This sounds like overkill until you try the alternative: a scraper with its own parsing and its own slug rules validates a site that doesn't exist. Early on, my ad-hoc name normalizer stripped a cedilla differently than the app's did, and a country appeared "missing from the database" when it was there all along. The only comparison worth automating is the one the rendering code would make.

Not every mismatch is a failure. A contradiction between two sources a reader can reach — the database says one status, the search dropdown says another — is an error and kills the build. A country that exists in the static files but has no database row is a warning: nothing disagrees yet, but that page is running unmanaged, and edits made in the admin panel will never reach it. Errors are lies; warnings are blind spots. Conflating them either makes the check ignorable or makes it a liar itself.

Two smaller rules earned their place fast:

Failing to run is a failure. If the database is unreachable, the check exits nonzero and prints "this is not a pass." A verification step that silently passes when it can't verify is worse than no step — it launders outages into green builds.
The baseline must shrink. Day one produced 28 known conflicts, and blocking every deploy until all were resolved would have gotten the check deleted within a week. So known differences live in a baseline file — but every baseline entry that stops matching reality gets reported too. A baseline that only suppresses becomes a second copy of the lie. Mine went 28 → 26 → 0 in three days, and an empty baseline is the contract: any future disagreement, anywhere, fails the build.

The check audited my beliefs, not just my data

Building it forced me to read the actual data flow, and the data flow was not what I believed. For months I'd treated the database as admin-only bookkeeping — I'd even written it down as fact. Wrong. A merge function I'd forgotten about overrides the static file with the database row on every country detail page. The database had been winning on the most-read pages of the site the whole time, while the maps and search kept rendering the static values.

That's the four-way split in one sentence: the same fact had different authorities on different pages of the same site. No amount of auditing the values fixes that; I had to know which copy wins where before "disagree" even meant anything. The check now encodes that knowledge — it's the only place the whole routing of truth is written down and executed.

26 conflicts, one actual disagreement

The day-one errors split cleanly. Two were the dead links — wrong continent values in the database, five-minute fix. The other 26 looked like 26 separate factual disputes about 26 countries, and they were nothing of the sort. One store said "Medical" 51 times where the other said illegal or decriminalized, because the two stores meant different things by the word. One counted any statute with the word "medical" in it — including countries whose entire program is licensing crops for export, where no resident can be prescribed anything. The other described what a person there can actually do.

Arbitrating that country-by-country would have produced 26 judgment calls to re-litigate forever. Instead the fix was a definition, written at the top of the data files where the next editor — human or agent — can't miss it:

// STATUS FIELD — what the value means (settled, do not re-litigate)
//
// `status` describes WHAT AN ORDINARY VISITOR FACES, not what the
// statute book contains. Export-only licensing is NOT "Medical".
// "Decriminalized" means no CUSTODIAL penalty. When two statuses
// both apply, the one describing possession by an ordinary
// person wins.

One definition resolved all 26. And it settled which side to trust with evidence rather than vibes: the only two countries that disagreed in the opposite direction had both changed status in the real world years ago — the database had tracked the changes, the static file hadn't. The store that was right about the hard cases got believed about the rest.

The definition keeps paying. Later, seeding new rows, I hit two countries with nearly identical laws on paper — small-amount possession punished without prison in one, punished "administratively" in the other. The first is a fine or community service: decriminalized. The second allows fifteen days of detention: illegal, because fifteen days in a cell is custodial no matter what the statute calls it. Without the written rule, those two rows would have been decided by whichever mood I was in that day.

Closing the blind spot the check can't see

A comparison check has a structural limit: it can only check facts that exist in both stores. Sixty-three countries had pages but no database row — no contradiction possible, no protection either, and any admin edit to them silently went nowhere. So the last phase was seeding every missing row, region by region, each value verified against current primary sources before it went in.

That seeding was itself an audit — the last manual one — and it caught three statuses the site had carried wrong for months: a Balkan country whose medical program took effect this January (the top search result "contradicting" this was dated 2021 — it didn't refute the change, it predated it), an Indian Ocean island that quietly amended its drug law in 2022, and a South American country that replaced jail with counselling for small amounts back in 2022. Three more corrections that no diff between my own sources could ever have surfaced, because every internal copy agreed on the stale value.

The blind spot is now closed: every country on the site has a row, every row is compared on every build, and the check's first fully green run reported both stores in complete agreement for the first time in the site's history.

What I'd carry to any system with more than one copy of the truth

An audit is a snapshot; a build gate is a contract. If a fact has multiple writers, the agreement between copies needs an enforcer that outlives the session that created it.
Compare with the app's own logic or don't bother. A checker with its own parsing validates an imaginary site.
Errors are contradictions; warnings are blind spots. A reader-reachable disagreement fails the build. A gap gets counted, loudly, until it's closed.
A check that can't run must fail. "Couldn't verify" reported as success is the most expensive bug a verification step can have.
Baselines must shrink. Report stale exceptions as hard as new failures, and drive the file to zero. An empty baseline is the only version that means anything.
When two copies chronically disagree, look for a missing definition before arbitrating cases. Twenty-six disputes that share a shape are one dispute. Write the rule down where the data lives, and the next hundred edits inherit it.

Two audits taught me where the copies were. The check is what finally made them agree — and the next time they don't, I won't find out in an audit. I'll find out in a red build, minutes after it happens.

Every Page Was Exactly the Same Size. That Was the Bug Report.

Ted — Sun, 09 Aug 2026 08:00:17 +0000

Search Console told me ten pages on a site I run were duplicates of each other. That report is usually noise — it fires on trailing slashes, on query strings, on pagination. I opened it expecting to close it.

The site is a React single-page app that prerenders static HTML at build time, so crawlers get real content instead of an empty <div>. I fetched the ten flagged URLs to see what the crawler was seeing.

/section/a/first-page      200   6025B
/section/a/second-page     200   6025B
/section/b/third-page      200   6025B
/section/c/fourth-page     200   6025B

Ten different URLs. Ten identical byte counts.

Not similar. Identical — down to the byte, because they were the same file. Every one of them was the homepage, served under a different URL, carrying a <link rel="canonical"> pointing at the homepage. Each of those pages was telling Google, in the crawler's first pass, I am not a page. I am the homepage.

Nothing had failed

Here is what makes this hard to catch, and it is the whole point of this post.

The routes returned 200. The app has a catch-all rewrite, so any path serves the shell and the router sorts it out in the browser. A human visiting one of those URLs saw the correct page, fully rendered, with the correct title in the tab — because after hydration the client-side code sets all of that properly.

The build exited 0. It printed a success line and a route count, and the route count was a number nobody had a reason to question.

The prerender step had a try/catch around the part that loads the page data. When that load failed, the catch logged a warning and returned an empty array. Every downstream route was generated by looping over that array, so an empty array produced zero pages — quietly, with a success exit code.

And the pages that were missing didn't 404. They fell through to the catch-all and served the shell. The failure covered its own tracks.

So: no exception, no red build, no broken link, nothing wrong on screen. Four independent layers, each doing something individually reasonable, and the composite result was a few hundred pages that existed for humans and did not exist for crawlers.

A fallback converts a failure into a plausible lie

This is the reframe I keep coming back to.

A fallback exists to stop a failure from being loud. That is its entire job, and most of the time it is the right job. But the output of a fallback is not an error message — it is a value. A plausible one. Something shaped exactly like a correct answer.

Which means you cannot find this class of bug by looking for failures, because no failure occurred. Your logs are clean by construction. Your monitors are green by construction. The system did not break; it substituted.

So the question stops being what went wrong and becomes something much more mechanical:

What should be different here, and is it?

Uniformity is the signal. Ten pages that should have ten different sizes had one size. That is not a subtle statistical hint — it is a screaming anomaly, and it is invisible unless you go looking for sameness specifically.

Once I started asking that question instead of hunting for errors, the same afternoon turned up three more.

Four bugs, one technique

Identical outputs where inputs differed. The byte counts above. Ten inputs, one output. That found the missing pages.

One half worked and the other didn't, in the same file. Two sections of the same build enriched their pages from different sources — one fetched over HTTP, one needed a bundler to read a local data file. The HTTP one produced 156 links on the live page. The bundler one produced zero. Same file, same build, same run, one worked and one didn't. That is a controlled experiment I got for free, and it isolated the cause to the bundler dependency in about a minute. Without the working half I would have been guessing at the whole pipeline.

A set difference nobody had ever computed. One page linked out to every item in a database table. The router resolved those links against a separate static file. Nobody had ever asked whether those two lists agreed. I subtracted one from the other: ten links pointed at pages that could not exist. Not because of a typo — because the two sources disagreed about spelling in six cases and about categorisation in four. Every one of those links had been manufacturing a fresh duplicate page, from an indexed page, for months.

Arithmetic against a stale premise. A migration script sitting in the repo predicted the table would end up with a particular distribution of values: eleven of one kind, a hundred and seven of another. The live table had twenty-one and ninety-seven. That ten-row gap was the entire finding — a later audit had deliberately moved ten rows, and running the older script would have reverted it. I didn't need to understand either script's reasoning to know something was wrong. The numbers didn't match, and mismatched numbers are a question.

None of those four is clever. All of them are comparisons. That is the only skill involved.

The one that actually mattered

The worst thing I found that day had nothing to do with prerendering, and it was hiding behind the same mechanism.

A classification field in the database had five possible values. It had gained its fifth some months earlier. One function mapped that field to a human-readable label with a switch, and that switch had cases for four of the five. The fifth hit the default branch and came out as the most severe of the other four.

The pages affected were three of the highest-traffic pages on the site. Each of them was displaying a confident, specific, wrong classification. Not a blank. Not a placeholder. The opposite of the truth, stated plainly, in the page title.

Two things kept it invisible. First, the fallback again — the output was a real label, so nothing looked empty or broken. Second, and worse: a different function, the one that picks the colour for the status badge, did have a case for the fifth value. So the page rendered the correct colour next to the incorrect word. Anyone glancing at it saw a styled, complete, confident page.

The tell was in the type signature. The function declared a return type listing all five labels. It could only ever return four. A function whose return type names a value it cannot produce is telling you, in writing, that a case is missing — and TypeScript will not flag it, because returning a subset of a union is perfectly legal.

That is a cheap thing to grep for and I had never thought to.

What I'd actually do differently

Two changes, and one of them is not a technique.

Delete the fallback where the fallback is a lie. The prerender's try/catch now throws, and the build refuses to run if it loads fewer pages than expected. A build that silently drops a few hundred pages and exits 0 is worse than a build that fails, because the failing one costs you ten minutes and the silent one costs you months. That is the change I'd make first in any system: find the places where degraded output is indistinguishable from correct output, and make them loud.

Compare things that should match. Not as a debugging step — as a routine check. Do the links on this page point at pages that exist? Does the sitemap match what the build emitted? Does the static file agree with the database table? Every one of those is a set difference you can compute in a few lines, and every one of them found something real for me. They're boring, they're fast, and they don't require knowing what you're looking for in advance, which is the entire point when the bug produces plausible output.

The non-technique: stop trusting the fact that nothing looks wrong. For this whole class of problem, "nothing looks wrong" is not evidence of health. It's the expected symptom.

The short version

You cannot detect a failure that didn't happen. Every fallback in your system is a place where a failure was quietly converted into a plausible value, and plausible values do not show up in logs, monitors, or a glance at the page.

What they do show up in is comparison. Two things that should differ and don't. Two lists that should match and don't. A number that should be one number and is another.

Ten files with the same byte count is not a subtle clue. It's a bug report — as long as somebody thinks to measure the thing that was never supposed to be uniform.

The Insecure Default Was the Configuration

Ted — Fri, 07 Aug 2026 08:00:52 +0000

I was moving a pile of automation scripts into version control — years of accumulated cron jobs that had only ever existed on one machine — and ran a secret scan before the first commit. It found this, in two files:

SERP_API_KEY = os.environ.get("SERP_API_KEY", "a1b2c3d4…")

A live API key, sitting in source, as the default argument. The fix is so obvious it barely counts as a decision. Delete the literal, put the value in an env file, read it from there. Ten minutes.

I did the ten minutes. Then, before committing, I checked one more thing: what actually sets SERP_API_KEY when these run?

Nothing does. Not the crontab, which has no env line for it. Not the wrapper the jobs run through, which passes the environment along untouched. Not a shell profile — cron does not read one.

So that hardcoded literal was not a fallback. It was the configuration. Every scheduled run of those jobs for months had used the second argument, because the first was never populated. What I had just deleted as a bad habit was the only reason the jobs worked.

Two halves that look alike

os.environ.get(KEY, default) reads as a preference and a safety net. Use the environment; if it is not there, fall back to this.

The shape is honest about the mechanism and silent about the reality. It does not tell you which half runs. Both branches are valid code, both produce a working value, and the expression looks identical whether the variable is set on every machine or on none of them. process.env.KEY || 'literal' has the same problem. So does every config layer with a default: the fallback is invisible precisely when it is doing all the work.

The asymmetry that makes this dangerous is that the two failure modes arrive at different times. If the environment variable is set and I delete the default, nothing happens — the code was already taking the first branch. If it is not set and I delete the default, the job dies. Both edits look the same in a diff. Both feel like the same small act of hygiene.

And the death is quiet. These were scheduled jobs. They fail at 09:10 on a Monday into a log file, and the next thing that reads that log is me, whenever I next think to look.

The question that separates them

Not "is there a hardcoded secret here." I could see that. The scanner saw it.

The question is: which branch is running in production right now?

For an environment variable that is one command, and it has to be run in the same context the job runs in — not in my shell, where my profile has been sourced and everything looks populated. My interactive shell is the least representative environment on the machine. Cron gets a nearly empty one.

Once I asked that, the fix changed shape. It was no longer "delete the literal." It was "make the environment actually carry the value, then delete the literal, then prove the job still resolves it." The last step is the one I would have skipped. It is also the only one that distinguishes a security fix from an outage.

The scanner was too clever

There is a second half to this, and it is worse.

My scan looked for high-entropy strings — long runs of hex and base64, the shape of an API key. It caught both hardcoded keys immediately. It also caught a third one in a different file, and I felt good about the coverage.

Then, reading around an unrelated line in a config block, I found this a few lines below:

EMAIL    = "me@example.com"
PASSWORD = "…"

A real account login, in plaintext, used to authenticate against a hosted database. Fifteen characters. My pattern required thirty-two.

That is not a tuning problem I can fix by lowering the threshold — at fifteen characters and lower, everything matches. It is a category error. Entropy-based secret detection is built for machine-generated credentials, and a password is human-generated by definition. It is short because a person types it. It contains a word because a person remembers it. Every property that makes it a bad password makes it invisible to a scanner looking for randomness.

The credential with the widest blast radius on the machine — an account login, plausibly reused elsewhere — was the one my tooling was structurally incapable of finding. I found it by reading.

So the scan needs two passes with different logic: one for shape, matching entropy and known key prefixes, and one for name, matching PASSWORD, SECRET, TOKEN, PASSWD as assignment targets regardless of what is on the right-hand side. The second pass is dumber and catches what the first cannot.

Not everything that looks like a secret is one

The mirror of that failure showed up in the same commit, and it is worth naming because over-correcting here has its own cost.

Two more long random-looking strings survived my scan, and I left both in source deliberately.

One was a verification key for a search-engine ping protocol. That protocol works by having you host the key as a text file at your own domain root, so a crawler can confirm you control the site. The key is published by design. I confirmed it in one request: fetching the key file at the domain returned 200. Stripping it would have been theatre.

The other was a JSON Web Token for a database client. JWTs are three base64 segments joined by dots, and the middle one is the payload — you can decode it without any credential at all. Its role field said anon: the public client key, meant to ship to browsers, protected by row-level policies rather than by being secret. Also fine to commit.

Both of those look exactly like the thing I was hunting. The difference is not visible in the string. You get it by asking what the value is for — and for a JWT you can simply read the answer out of the token.

Blindly stripping every high-entropy string would have broken two working systems in the name of security. That is the same failure as leaving the real password in, just pointed the other way: acting on the shape of a value instead of its function.

What I actually changed

The refactor that shipped is unremarkable, which is the point:

Each script reads its credential from a single env file, using the same loader the rest of the codebase already used. Consistency here matters more than elegance; a second loader is a second thing to get wrong.
Every one raises a clear error naming the missing variable, instead of falling back. If it cannot find the value, I want the job dead and loud, not running on a stale literal.
I verified each one by executing it and watching it resolve a real value. Not a syntax check. A syntax check would have passed on all three, including the one where I had used os.environ without importing os.

That last one is the small joke at the end. Having reasoned carefully about which branch runs in production, I introduced a NameError and nearly shipped it, because ast.parse validates grammar and knows nothing about names. The check that caught it was running the thing.

The short version

A default value is a claim about what happens when the real source is missing. It is not a claim that the real source exists.

Before you delete one as a bad habit, find out which half has been load-bearing. If the answer is "the literal," you are not tightening security — you are removing the configuration, and you will find out at 09:10 on a Monday, in a log file, if you find out at all.

And run two scans. One for what secrets look like, one for what they are called. The scanner that finds your API keys will walk straight past your password.

Your Analytics Can Only See As Far As Your Login Does

Ted — Sat, 01 Aug 2026 04:34:28 +0000

I run a directory site. People arrive, look at listings, and some of them click through to an external booking page. Those outbound clicks are the only thing on the site that correlates with money, so they are logged: what was clicked, from which page, on what device, and — if the person happens to be signed in — which account.

Out of 1,372 recorded clicks, six carried an account.

Six. Not six percent. Six clicks, across eight months, belonging to four distinct people.

That number reads as a broken integration. Something in the chain that attaches identity to an event is dropping it, and 99.56% of the time nobody notices because the click still gets recorded and the dashboard still fills in. So I went looking for the break.

The chain was fine

The click handler passes through a hook, the hook calls an edge function, the edge function reads the Authorization header and resolves it to a user. The client library attaches the current session's token automatically. There was no header override anywhere, session persistence was on, and the storage key was pinned so it could not rotate out from under itself.

The requests also pass through a same-origin proxy on the way to the backend, which is the kind of thing that quietly eats headers. I tested it directly: the auth endpoint received the token, parsed it, and rejected it for the correct reason. Headers survive the hop.

Every layer did what it was written to do. Which meant the six were not survivors of a lossy process. They were the entire population of something.

The six were not random

Here is the part that ended the investigation.

I pulled the six clicks and joined them to the accounts that made them, and looked at the gap between when the account was created and when the click happened.

1.7 minutes
1.8 minutes
2.8 minutes
7.4 minutes
11.6 minutes
14.9 minutes

Six clicks. All of them inside a quarter of an hour of the account existing at all.

That is not the signature of a lossy pipeline. A pipeline that drops identity 99.56% of the time drops it uniformly — you would see attributed clicks scattered across the whole timeline, at all sorts of distances from signup. Instead every attributed click sits in one narrow window, and the window has an obvious shape: it is the period during which somebody was still signed in from having just signed up.

Nothing was failing. The tracking captured identity every single time identity was present. It was present for about ten minutes per person, ever.

Nobody chose this

The site is deliberately open. Nothing is gated. You never have to make an account to see anything, and the only reasons to make one are to save a listing or leave a review — both optional, both rare. That was a good decision, made on its own merits, about content access.

But an auth model is not only a content-access decision. It also silently sets the maximum duration over which your analytics can associate two events with the same person.

Nobody wrote that down. There is no config value for it. It emerged from "the site should be open to everyone" and it landed at roughly ten minutes, and it will stay at roughly ten minutes no matter how good the analytics get, because the analytics were never the constraint.

You cannot measure a relationship that outlives your identifier. Whatever your system uses to say "this is the same person as before" — a session, a token, a cookie, a device id — has a lifespan, and every question your analytics can answer must fit inside it. Ask a longer question and you do not get a wrong answer. You get an empty one, which is worse, because empty looks like evidence of absence.

Why it presents as a bug

The reason this costs a day rather than a minute is that the symptom shows up in the wrong place.

The number that looks wrong is in the analytics table. So you debug the analytics: the handler, the function, the header, the proxy. All of that code is fine, which means every check comes back clean, which feels like you are narrowing in on something subtle when in fact you are auditing a layer that has no defect in it at all.

The defect is not in the layer that produced the symptom. It is in a design decision one layer down and several months back, which is working exactly as intended and has no idea it is also an analytics constraint. Instrumentation problems present as instrumentation. Architecture problems also present as instrumentation.

The tell was the distribution, not the magnitude. 0.44% tells you almost nothing — it is consistent with a lossy pipeline, a rare event, or a design ceiling. The shape of when those six occurred distinguishes between them in one query. When a number looks broken, the useful question is rarely "how big is it" and almost always "how is it distributed".

The fix is a design change

More tracking would not have helped, because nothing was going untracked. The change is to stop borrowing identity from the session.

So every tracked click now also carries a first-party identifier generated once in the browser's local storage, and signup copies that identifier onto the new account. Joining the two recovers the whole visit: what somebody clicked before they had an account, and — the part that actually matters — what they clicked days afterwards.

That is a different kind of thing from a session. It is not authentication and it grants nothing. It is one value whose only job is to outlive the login, because the login was never going to live long enough to answer the question.

What it still cannot do

Worth being precise about, because an identifier like this invites more confidence than it earns:

It identifies a browser, not a person. Cleared storage, a second device, a private window — each one starts fresh. Somebody who browses on a phone and books on a laptop is two rows and always will be.
It undercounts, never over. It can fail to connect two events that belonged to the same person. It cannot merge two people into one. Given a choice about which direction to be wrong in, that is the right one.
It only works going forward. Every existing account predates it, so their history stays blank. There is no backfill, because the data was never captured — and inventing a join key retroactively would produce exactly the kind of confident wrong answer this was meant to eliminate.

None of that makes it a weak instrument. It makes it an instrument with a stated range, which is the only kind worth having. The version I replaced also had a range. It just did not mention it.

The general shape

Every system that counts things has a horizon, and the horizon is usually set somewhere else entirely — in an auth model, a retention policy, a cookie lifetime, a log rotation window. None of those are described as analytics decisions. All of them are.

When a metric comes back far lower than it should be, there are three possibilities, and they look identical at a glance: the pipeline is lossy, the event is genuinely rare, or you asked a question longer than your identifier lives. The first is a bug and you should fix it. The second is a finding and you should accept it. The third is neither — it is a boundary, and no amount of work inside the analytics layer will move it.

Check the distribution before you check the code. If every observation you do have clusters inside one window, you have not found a leak. You have found the edge of what your system was ever able to see.

Your System Cannot Say I Don't Know

Ted — Fri, 31 Jul 2026 08:00:44 +0000

Four things I have been told by my own systems in the last three months:

0 clicks on outbound links, on a page I could watch people click through.
Clean from a typechecker, on a codebase with two type errors in it.
Position 13 for a page that has never once ranked 13th.
200 OK from an endpoint that did nothing at all.

None of these was a bug in the ordinary sense. Nothing crashed, nothing threw, nothing appeared in a log with the word ERROR next to it. Each was a real value, produced by a real system, working exactly as written.

And every one of them was a statement about the world that the system had no business making, because in each case the thing that was supposed to do the measuring had not run.

The overloaded value

Pick any layer of your stack and there is a single value doing two incompatible jobs.

A database query returns no rows. Does that mean the table has no matching rows, or that the query never reached the table? Both are null.

A process exits 0. Does that mean it checked everything and found nothing wrong, or that it checked nothing? Both are zero.

A counter reads 0. Is that "we measured, and it was none", or "nothing was ever counted"? Both are 0.

A command produces no output. Silence is what success looks like. It is also what a program that did nothing looks like.

This is not a language design mistake anyone can point at and fix. It is a consequence of representing absence at all: the moment you have a value meaning "nothing here", it will be produced both by nothing being here and by the lookup not happening. Every layer inherits it. Every layer passes it upward, where it stops looking like an absence and starts looking like data.

The asymmetry that keeps it hidden

Here is what makes this specifically dangerous rather than merely annoying.

In all four of my examples, the incorrect reading is the reassuring one.

Zero errors. All checks passed. No such user. Nothing to report.

A false alarm gets investigated within the hour, because it is uncomfortable and somebody wants it to go away. A false all-clear gets investigated never, because it is exactly what you hoped to see and there is nothing to chase.

So the failure mode is not just that the two states are indistinguishable. It is that the wrong one is load-bearing for your peace of mind, and the entire purpose of the signal is to let you stop looking.

That is why these survive for months rather than hours. A green check is not a piece of information you evaluate. It is permission to move on.

Four layers, one shape

The instrument. A dashboard showed zero outbound clicks. The tracking worked correctly — for one specific kind of link. Every other link on the site was a plain anchor tag the counter had never been taught about. So "0" was accurate about what it measured and silent about the fact that it was measuring almost nothing. The number was not wrong. Its scope was, and a number does not carry its scope. (Zero Is Not a Measurement)

The compiler. I ran a typechecker four times before shipping, and it exited clean every time. The project used a config that points at other configs and contains no source files of its own — a normal, widely-generated layout. Given that config, the tool typechecks an empty set, finds nothing wrong with it, and exits zero. It was not being lenient. It was being thorough about nothing. (It Passed Because It Never Looked)

The statistic. A page reported an average search position of 13. It had never ranked 13th. It ranked around 6 for one group of queries and around 20 for another, and the mean of a bimodal distribution is a number describing a state that does not exist. Unlike the others this one is not even a failure — the average is computed correctly, from complete data, by a system with no faults. It still describes nothing real. (One Page Ranked #6 and #20 at the Same Time)

The database. A password reset endpoint looked users up through an API layer that could not reach the table it needed. The call failed every time it ran. But only the result was captured and the error was discarded, so the failure arrived as null, and null was read as "no such user" — which routed into a branch specified to return success while doing nothing, because revealing whether an account exists is a security leak. (A Discarded Error Becomes an Answer)

Four different layers. One shape: an operation that did not happen, represented identically to an operation that happened and found nothing.

The question that separates them

There is one question that distinguishes a real signal from a decorative one, and it takes about a minute to ask:

If this were broken, what would I see?

If the honest answer is "exactly what I am seeing now", you do not have a signal. You have a value that happens to be reassuring.

Run it against the four:

If the click tracker were counting nothing, the dashboard would read 0. It read 0.
If the typechecker were examining no files, it would exit clean. It exited clean.
If the average were hiding a split, it would return a plausible middle number. It returned 13.
If the user lookup were failing completely, the endpoint would return 200 and send nothing. It returned 200 and sent nothing.

Four for four. In each case, "healthy" and "not running" produce byte-identical output, and I had been reading that output as evidence for months.

The follow-up question is the actionable one: what would have to change for these to be distinguishable? That is usually a small piece of work. Print how many files were checked. Report the count of rows the query considered, not just the result. Show a distribution alongside the mean. Return a different status code when the lookup itself failed.

None of that is clever. It is just refusing to let one value carry two meanings.

Unknown is a state you have to store

The fix that generalises is to stop modelling two states where there are three.

Not found and not found. Found, not found, and could not determine.

That third state has to survive as far as the decision. It is not enough to log it — logging is what you do with something you have decided not to act on. It has to reach the branch, and the branch has to treat it differently, because "I could not check whether this person has an account" and "this person has no account" call for opposite behaviour. One is a 500 that pages you. The other is a normal response.

In practice that means a handful of unglamorous habits:

Bind your errors even when you do not think you need them. Discarding an error does not make it not happen. It converts it into whatever your success path does with a missing value, which is usually to proceed confidently.
Make failure reachable, then check that it is. Break the check on purpose and watch it go red. If it stays green, you have not learned that your code is clean; you have learned that your check does not run.
Require the failure to be specific. A nonzero exit says something failed. It does not say this check found this fault. Assert on the content, not just the status, or an empty run and a passing run remain interchangeable.
Distinguish "no data" from "data says zero" in anything you display. They are different claims and a dashboard that renders them identically is lying quietly, every day, to whoever is looking at it.

Why this is getting harder

The volume of code arriving in my repositories that I did not type by hand has gone up sharply, and it is very good code by every measure my tooling can apply. It compiles. It is typed correctly. It follows the conventions of the file it sits in.

What it is not reliably doing is knowing things. It produces the plausible value confidently, and the plausible value is frequently right, which is exactly what makes the exceptions invisible. Generated code and swallowed errors have the same signature: something that looks like knowledge, occupying the place where knowledge should be, with no marker distinguishing it from the real thing.

Which means the burden shifts. It is no longer enough to check that the code is well-formed — every tool I have already does that, extremely well. The open question is whether the things it asserts are true, and almost nothing in a standard pipeline is built to ask.

The short version

Software very rarely fabricates information out of nothing. What it does constantly, at every layer, is take I could not find out and quietly round it to the nearest confident-looking answer — usually zero, usually null, usually success.

Then it hands you that answer with exactly the same face it uses for a real one.

A Second Check Is a Second Source of Truth

Ted — Thu, 30 Jul 2026 08:00:45 +0000

Signup on a site I run asked people to prove they owned their email address, and then asked them again.

First a six-digit code, emailed and typed back in. Only after that does the account get created. Then, on creation, the auth service sent its own confirmation email with a link in it.

Two emails. One address. Same question asked twice.

It read as thorough. It was actually a fork in the data, and it had already put a real user somewhere the system has no sensible answer for.

The state nobody designed

Someone signed up. They received the code, entered it correctly, and the account was created. The app signed them in immediately, as it is written to do.

They never clicked the second link. Why would they? They had just typed a code out of their inbox and were now looking at a logged-in page. From where they sat, the job was done.

So the record says: account exists, password set, session valid, signed in — and email_confirmed_at is null.

That row is not describing a half-finished signup. It is describing a completed one, plus a leftover flag from a check that nobody needed and nothing enforced. But a flag called email confirmed saying no is going to be read by something, eventually. Any support tooling, any audit, any future feature that gates on confirmation, will look at that user and conclude they never proved their address.

They did. Six weeks earlier. There is a used code in another table with their name on it.

Two records, one fact

This is the reframe that took me embarrassingly long.

I had been thinking of the second check as redundancy — belt and braces, no harm in it. It is not redundancy. It is a second place where the same fact is recorded, and two records of one fact will eventually disagree.

The fact is "this person controls this address." My OTP table knows it, in the form of a code marked used. The auth service knows it, in the form of a timestamp. Nothing keeps the two in sync, because they were never designed as one thing — one is application logic, the other is a platform default that was left on.

The moment a user satisfies one and not the other, the fact has two values. And when a fact has two values, every consumer downstream has to pick which one to believe, usually without knowing there is a choice being made.

The one that wasn't enforcing anything

There is a version of this where the second check is the real one and the first is a convenience. That would at least be coherent.

It was not that. The link enforced nothing.

The user was signed in before the confirmation email could possibly have been opened, because the app calls sign-in directly after creating the account. Sign-in did not consult the flag. Nothing in the product consulted the flag. The account worked identically whether the link was clicked or not.

So the second check could not block anything, could not gate anything, and could not fail in any way the user would notice. Its entire observable effect was to set a column that then contradicted the other record of the same fact.

That is the test worth applying: if a check cannot refuse anything, it is not a check. It is a log entry with an opinion.

Why it was on at all

Nobody chose this. The confirmation email is the platform default, and it is a sensible default — for an app whose signup is "create account, we'll email you a link." That is the flow it was designed for.

We had replaced that flow with our own and never turned the old one off. The original code even said so, in a comment:

// Create the user account with auto-confirm (since we verified email via OTP)

The intent was recorded. The setting it referred to was never changed. So the comment described a system that did not exist, sitting directly above the line that created the problem, for months.

I have a lot of sympathy for that comment. It is what happens when you build the replacement and reasonably assume the thing it replaces has stepped aside.

Fixing it, and the row already broken

Turning off the platform confirmation was one setting. New signups now get exactly one email, containing exactly one code, and land signed in with a single coherent record of the fact.

The stranded user needed a separate decision, and it is a more interesting one than it looks. The options were:

Leave them flagged unverified, which is false.
Set the flag to now, which is also false — it says they proved ownership today, which they did not.
Set it to the moment they actually entered the code.

I took the third. Their confirmation timestamp is now thirty-three seconds before their account was created, which looks like a data error and is in fact the most accurate statement available: they proved they owned that address, then the account was made.

I would rather have a timestamp that looks odd and is true than one that looks tidy and is not. Anyone who investigates that row will find the corresponding used code sitting right there in the other table, and the sequence will make sense.

What I would ask earlier next time

When you replace a platform flow with your own, the work is not finished when yours works. It is finished when the platform's version is off.

Until then you do not have a verified email address. You have two claims about a verified email address, maintained by different systems, agreeing by coincidence — and coincidences hold right up until a user does something entirely reasonable, like reading their inbox once instead of twice.

Your Toolchain Checks Grammar, Not Facts

Ted — Wed, 29 Jul 2026 08:00:21 +0000

I went looking for one wrong email address and found six wrong domains.

They had been in production for months, across six different server functions. Nothing had ever complained, because there was nothing in the stack capable of complaining. Every one of them was valid code.

What was actually there

Three distinct hosts, none of which I own.

The first was the .com version of my own brand, which I have never owned — the site runs on a different TLD. Somewhere along the way, generated code decided the .com was the obvious address and used it anyway: in a contact-form reply telling users to email support there, and in admin links inside security alerts.

The second was the .io version. Same idea, different guess. This one was worse, because it was not only a link:

from: "Alerts <alerts@brand.io>",

That is the sender on a batch of subscriber notifications. Transactional email providers verify sending domains. You cannot send from a domain you have not proven you control. So every one of those emails failed at the provider, silently, from the day the code shipped.

The third was the provider's own shared test domain — the address used in every tutorial and quickstart:

from: "App <noreply@resend.dev>",

That one was on the password reset email. Shared test domains are heavily restricted, typically to your own account address. So the one email in the system that a locked-out user depends on was going out from the least deliverable sender available.

Why none of it was caught

Consider what each stage of a normal pipeline actually knows.

The typechecker knows "alerts@brand.io" is a string, and a string is what the field wants. Correct, and finished.

The bundler knows the file parses. Correct, and finished.

The tests know the function returns the shape they asserted. If they mock the mail client — and they do, because nobody sends real email in CI — they cannot see the sender at all.

The deploy knows the function compiled and uploaded.

Not one of these stages holds a fact about which domains I own. That is not an oversight in any of them. Type systems check the shape of values, not their correspondence to the world. There is no type that means "a domain this organisation controls."

So the whole apparatus is checking grammar. The sentence is well-formed. Nobody is checking whether it is true.

Generated code fails where the plausible answer is wrong

This is the part that has changed for me.

When I write a domain by hand, I am reading it off something — a dashboard, a DNS record, an invoice. When a model writes one, it is producing the most plausible token sequence given the context. For a brand named X, x.com is overwhelmingly the most plausible domain. For an email provider's sender field, the address from the documentation is the most plausible value.

Both guesses are excellent guesses. They are what a reasonable person would assume. They are also wrong, and wrong in a way that is invisible in review, because support@brand.com looks correct on the page unless you already know the answer.

That predicts where to look. These errors cluster wherever the correct value is guessable: brand domains, support addresses, affiliate and partner IDs, API senders, dashboard URLs. Anywhere there is an obvious-looking answer, the obvious-looking answer is what you get, and it is right often enough that the exceptions never get checked.

It is not a hallucination in the dramatic sense. Nothing is invented. Something real is substituted for something true.

Declare your external surface

The fix took about twenty minutes. Extract every URL host and email domain from the source, compare against a list of hosts you have declared, fail on anything else.

✗ 12 undeclared domain reference(s) in git ref abc1234:

  resend.dev  (5 references)
      functions/send-password-reset/index.ts:184
      functions/verify-code/index.ts:155
      … and 3 more

  brand.io   (3 references)
  brand.com  (3 references)

Run against the commit from before I started fixing things by hand, it reported all eleven references I had found — plus one more I had missed entirely. A page was setting its canonical URL to a domain I do not own, which meant it had been telling search engines that the authoritative copy of that page lives on someone else's property.

I found six by reading. The script found seven, in about a second, and will keep finding them.

Two details matter more than the regex:

The allowlist belongs in the repository, not in configuration. The failure mode is generated code introducing a plausible host. If the allowlist lives somewhere an assistant can quietly extend, it stops being a check and becomes a formality. In the repo, widening it is a diff, and a diff gets looked at.

Where it runs is not obvious. A pre-push hook is the tight loop, and it only sees commits that pass through your machine. If you use a hosted agent that commits and pushes on its own — mine does — a local hook is blind precisely where most of the generated code enters. So the same script also runs on a schedule against the remote branch, and alerts only when the set of offending hosts changes, so a known-unfixed finding does not decay into noise.

What it does not do

It catches wrong constants. It does not catch wrong behaviour.

The same week, I fixed a lookup that had been failing on every call for months because it queried a schema the API layer does not expose. Valid code, real function, correct arguments, and the error was being discarded so the failure read as an ordinary result. No domain check would ever see that. It is a different class and it needs a different tool.

I would rather say that plainly than let a script imply more coverage than it has. A guard that quietly under-delivers is the thing we are trying to get away from.

The general version

There is a category of error that generated code produces far more readily than handwritten code, and that conventional tooling is structurally unable to detect: values that are well-formed and untrue.

You cannot review your way out of it, because the values look right. You cannot typecheck your way out of it, because they are the right type. What you can do is write down the small set of facts your project depends on — which domains are yours, which accounts, which endpoints — and make a machine compare the code against that list every time.

The list is the interesting artifact. Not because it is clever, but because until you sit down to write it, nobody has ever actually enumerated what your codebase is allowed to talk to.