DEV Community: Ted

Every Page Was Exactly the Same Size. That Was the Bug Report.

Ted — Sun, 09 Aug 2026 08:00:17 +0000

Search Console told me ten pages on a site I run were duplicates of each other. That report is usually noise — it fires on trailing slashes, on query strings, on pagination. I opened it expecting to close it.

The site is a React single-page app that prerenders static HTML at build time, so crawlers get real content instead of an empty <div>. I fetched the ten flagged URLs to see what the crawler was seeing.

/section/a/first-page      200   6025B
/section/a/second-page     200   6025B
/section/b/third-page      200   6025B
/section/c/fourth-page     200   6025B

Ten different URLs. Ten identical byte counts.

Not similar. Identical — down to the byte, because they were the same file. Every one of them was the homepage, served under a different URL, carrying a <link rel="canonical"> pointing at the homepage. Each of those pages was telling Google, in the crawler's first pass, I am not a page. I am the homepage.

Nothing had failed

Here is what makes this hard to catch, and it is the whole point of this post.

The routes returned 200. The app has a catch-all rewrite, so any path serves the shell and the router sorts it out in the browser. A human visiting one of those URLs saw the correct page, fully rendered, with the correct title in the tab — because after hydration the client-side code sets all of that properly.

The build exited 0. It printed a success line and a route count, and the route count was a number nobody had a reason to question.

The prerender step had a try/catch around the part that loads the page data. When that load failed, the catch logged a warning and returned an empty array. Every downstream route was generated by looping over that array, so an empty array produced zero pages — quietly, with a success exit code.

And the pages that were missing didn't 404. They fell through to the catch-all and served the shell. The failure covered its own tracks.

So: no exception, no red build, no broken link, nothing wrong on screen. Four independent layers, each doing something individually reasonable, and the composite result was a few hundred pages that existed for humans and did not exist for crawlers.

A fallback converts a failure into a plausible lie

This is the reframe I keep coming back to.

A fallback exists to stop a failure from being loud. That is its entire job, and most of the time it is the right job. But the output of a fallback is not an error message — it is a value. A plausible one. Something shaped exactly like a correct answer.

Which means you cannot find this class of bug by looking for failures, because no failure occurred. Your logs are clean by construction. Your monitors are green by construction. The system did not break; it substituted.

So the question stops being what went wrong and becomes something much more mechanical:

What should be different here, and is it?

Uniformity is the signal. Ten pages that should have ten different sizes had one size. That is not a subtle statistical hint — it is a screaming anomaly, and it is invisible unless you go looking for sameness specifically.

Once I started asking that question instead of hunting for errors, the same afternoon turned up three more.

Four bugs, one technique

Identical outputs where inputs differed. The byte counts above. Ten inputs, one output. That found the missing pages.

One half worked and the other didn't, in the same file. Two sections of the same build enriched their pages from different sources — one fetched over HTTP, one needed a bundler to read a local data file. The HTTP one produced 156 links on the live page. The bundler one produced zero. Same file, same build, same run, one worked and one didn't. That is a controlled experiment I got for free, and it isolated the cause to the bundler dependency in about a minute. Without the working half I would have been guessing at the whole pipeline.

A set difference nobody had ever computed. One page linked out to every item in a database table. The router resolved those links against a separate static file. Nobody had ever asked whether those two lists agreed. I subtracted one from the other: ten links pointed at pages that could not exist. Not because of a typo — because the two sources disagreed about spelling in six cases and about categorisation in four. Every one of those links had been manufacturing a fresh duplicate page, from an indexed page, for months.

Arithmetic against a stale premise. A migration script sitting in the repo predicted the table would end up with a particular distribution of values: eleven of one kind, a hundred and seven of another. The live table had twenty-one and ninety-seven. That ten-row gap was the entire finding — a later audit had deliberately moved ten rows, and running the older script would have reverted it. I didn't need to understand either script's reasoning to know something was wrong. The numbers didn't match, and mismatched numbers are a question.

None of those four is clever. All of them are comparisons. That is the only skill involved.

The one that actually mattered

The worst thing I found that day had nothing to do with prerendering, and it was hiding behind the same mechanism.

A classification field in the database had five possible values. It had gained its fifth some months earlier. One function mapped that field to a human-readable label with a switch, and that switch had cases for four of the five. The fifth hit the default branch and came out as the most severe of the other four.

The pages affected were three of the highest-traffic pages on the site. Each of them was displaying a confident, specific, wrong classification. Not a blank. Not a placeholder. The opposite of the truth, stated plainly, in the page title.

Two things kept it invisible. First, the fallback again — the output was a real label, so nothing looked empty or broken. Second, and worse: a different function, the one that picks the colour for the status badge, did have a case for the fifth value. So the page rendered the correct colour next to the incorrect word. Anyone glancing at it saw a styled, complete, confident page.

The tell was in the type signature. The function declared a return type listing all five labels. It could only ever return four. A function whose return type names a value it cannot produce is telling you, in writing, that a case is missing — and TypeScript will not flag it, because returning a subset of a union is perfectly legal.

That is a cheap thing to grep for and I had never thought to.

What I'd actually do differently

Two changes, and one of them is not a technique.

Delete the fallback where the fallback is a lie. The prerender's try/catch now throws, and the build refuses to run if it loads fewer pages than expected. A build that silently drops a few hundred pages and exits 0 is worse than a build that fails, because the failing one costs you ten minutes and the silent one costs you months. That is the change I'd make first in any system: find the places where degraded output is indistinguishable from correct output, and make them loud.

Compare things that should match. Not as a debugging step — as a routine check. Do the links on this page point at pages that exist? Does the sitemap match what the build emitted? Does the static file agree with the database table? Every one of those is a set difference you can compute in a few lines, and every one of them found something real for me. They're boring, they're fast, and they don't require knowing what you're looking for in advance, which is the entire point when the bug produces plausible output.

The non-technique: stop trusting the fact that nothing looks wrong. For this whole class of problem, "nothing looks wrong" is not evidence of health. It's the expected symptom.

The short version

You cannot detect a failure that didn't happen. Every fallback in your system is a place where a failure was quietly converted into a plausible value, and plausible values do not show up in logs, monitors, or a glance at the page.

What they do show up in is comparison. Two things that should differ and don't. Two lists that should match and don't. A number that should be one number and is another.

Ten files with the same byte count is not a subtle clue. It's a bug report — as long as somebody thinks to measure the thing that was never supposed to be uniform.

The Insecure Default Was the Configuration

Ted — Fri, 07 Aug 2026 08:00:52 +0000

I was moving a pile of automation scripts into version control — years of accumulated cron jobs that had only ever existed on one machine — and ran a secret scan before the first commit. It found this, in two files:

SERP_API_KEY = os.environ.get("SERP_API_KEY", "a1b2c3d4…")

A live API key, sitting in source, as the default argument. The fix is so obvious it barely counts as a decision. Delete the literal, put the value in an env file, read it from there. Ten minutes.

I did the ten minutes. Then, before committing, I checked one more thing: what actually sets SERP_API_KEY when these run?

Nothing does. Not the crontab, which has no env line for it. Not the wrapper the jobs run through, which passes the environment along untouched. Not a shell profile — cron does not read one.

So that hardcoded literal was not a fallback. It was the configuration. Every scheduled run of those jobs for months had used the second argument, because the first was never populated. What I had just deleted as a bad habit was the only reason the jobs worked.

Two halves that look alike

os.environ.get(KEY, default) reads as a preference and a safety net. Use the environment; if it is not there, fall back to this.

The shape is honest about the mechanism and silent about the reality. It does not tell you which half runs. Both branches are valid code, both produce a working value, and the expression looks identical whether the variable is set on every machine or on none of them. process.env.KEY || 'literal' has the same problem. So does every config layer with a default: the fallback is invisible precisely when it is doing all the work.

The asymmetry that makes this dangerous is that the two failure modes arrive at different times. If the environment variable is set and I delete the default, nothing happens — the code was already taking the first branch. If it is not set and I delete the default, the job dies. Both edits look the same in a diff. Both feel like the same small act of hygiene.

And the death is quiet. These were scheduled jobs. They fail at 09:10 on a Monday into a log file, and the next thing that reads that log is me, whenever I next think to look.

The question that separates them

Not "is there a hardcoded secret here." I could see that. The scanner saw it.

The question is: which branch is running in production right now?

For an environment variable that is one command, and it has to be run in the same context the job runs in — not in my shell, where my profile has been sourced and everything looks populated. My interactive shell is the least representative environment on the machine. Cron gets a nearly empty one.

Once I asked that, the fix changed shape. It was no longer "delete the literal." It was "make the environment actually carry the value, then delete the literal, then prove the job still resolves it." The last step is the one I would have skipped. It is also the only one that distinguishes a security fix from an outage.

The scanner was too clever

There is a second half to this, and it is worse.

My scan looked for high-entropy strings — long runs of hex and base64, the shape of an API key. It caught both hardcoded keys immediately. It also caught a third one in a different file, and I felt good about the coverage.

Then, reading around an unrelated line in a config block, I found this a few lines below:

EMAIL    = "me@example.com"
PASSWORD = "…"

A real account login, in plaintext, used to authenticate against a hosted database. Fifteen characters. My pattern required thirty-two.

That is not a tuning problem I can fix by lowering the threshold — at fifteen characters and lower, everything matches. It is a category error. Entropy-based secret detection is built for machine-generated credentials, and a password is human-generated by definition. It is short because a person types it. It contains a word because a person remembers it. Every property that makes it a bad password makes it invisible to a scanner looking for randomness.

The credential with the widest blast radius on the machine — an account login, plausibly reused elsewhere — was the one my tooling was structurally incapable of finding. I found it by reading.

So the scan needs two passes with different logic: one for shape, matching entropy and known key prefixes, and one for name, matching PASSWORD, SECRET, TOKEN, PASSWD as assignment targets regardless of what is on the right-hand side. The second pass is dumber and catches what the first cannot.

Not everything that looks like a secret is one

The mirror of that failure showed up in the same commit, and it is worth naming because over-correcting here has its own cost.

Two more long random-looking strings survived my scan, and I left both in source deliberately.

One was a verification key for a search-engine ping protocol. That protocol works by having you host the key as a text file at your own domain root, so a crawler can confirm you control the site. The key is published by design. I confirmed it in one request: fetching the key file at the domain returned 200. Stripping it would have been theatre.

The other was a JSON Web Token for a database client. JWTs are three base64 segments joined by dots, and the middle one is the payload — you can decode it without any credential at all. Its role field said anon: the public client key, meant to ship to browsers, protected by row-level policies rather than by being secret. Also fine to commit.

Both of those look exactly like the thing I was hunting. The difference is not visible in the string. You get it by asking what the value is for — and for a JWT you can simply read the answer out of the token.

Blindly stripping every high-entropy string would have broken two working systems in the name of security. That is the same failure as leaving the real password in, just pointed the other way: acting on the shape of a value instead of its function.

What I actually changed

The refactor that shipped is unremarkable, which is the point:

Each script reads its credential from a single env file, using the same loader the rest of the codebase already used. Consistency here matters more than elegance; a second loader is a second thing to get wrong.
Every one raises a clear error naming the missing variable, instead of falling back. If it cannot find the value, I want the job dead and loud, not running on a stale literal.
I verified each one by executing it and watching it resolve a real value. Not a syntax check. A syntax check would have passed on all three, including the one where I had used os.environ without importing os.

That last one is the small joke at the end. Having reasoned carefully about which branch runs in production, I introduced a NameError and nearly shipped it, because ast.parse validates grammar and knows nothing about names. The check that caught it was running the thing.

The short version

A default value is a claim about what happens when the real source is missing. It is not a claim that the real source exists.

Before you delete one as a bad habit, find out which half has been load-bearing. If the answer is "the literal," you are not tightening security — you are removing the configuration, and you will find out at 09:10 on a Monday, in a log file, if you find out at all.

And run two scans. One for what secrets look like, one for what they are called. The scanner that finds your API keys will walk straight past your password.

Your Analytics Can Only See As Far As Your Login Does

Ted — Sat, 01 Aug 2026 04:34:28 +0000

I run a directory site. People arrive, look at listings, and some of them click through to an external booking page. Those outbound clicks are the only thing on the site that correlates with money, so they are logged: what was clicked, from which page, on what device, and — if the person happens to be signed in — which account.

Out of 1,372 recorded clicks, six carried an account.

Six. Not six percent. Six clicks, across eight months, belonging to four distinct people.

That number reads as a broken integration. Something in the chain that attaches identity to an event is dropping it, and 99.56% of the time nobody notices because the click still gets recorded and the dashboard still fills in. So I went looking for the break.

The chain was fine

The click handler passes through a hook, the hook calls an edge function, the edge function reads the Authorization header and resolves it to a user. The client library attaches the current session's token automatically. There was no header override anywhere, session persistence was on, and the storage key was pinned so it could not rotate out from under itself.

The requests also pass through a same-origin proxy on the way to the backend, which is the kind of thing that quietly eats headers. I tested it directly: the auth endpoint received the token, parsed it, and rejected it for the correct reason. Headers survive the hop.

Every layer did what it was written to do. Which meant the six were not survivors of a lossy process. They were the entire population of something.

The six were not random

Here is the part that ended the investigation.

I pulled the six clicks and joined them to the accounts that made them, and looked at the gap between when the account was created and when the click happened.

1.7 minutes
1.8 minutes
2.8 minutes
7.4 minutes
11.6 minutes
14.9 minutes

Six clicks. All of them inside a quarter of an hour of the account existing at all.

That is not the signature of a lossy pipeline. A pipeline that drops identity 99.56% of the time drops it uniformly — you would see attributed clicks scattered across the whole timeline, at all sorts of distances from signup. Instead every attributed click sits in one narrow window, and the window has an obvious shape: it is the period during which somebody was still signed in from having just signed up.

Nothing was failing. The tracking captured identity every single time identity was present. It was present for about ten minutes per person, ever.

Nobody chose this

The site is deliberately open. Nothing is gated. You never have to make an account to see anything, and the only reasons to make one are to save a listing or leave a review — both optional, both rare. That was a good decision, made on its own merits, about content access.

But an auth model is not only a content-access decision. It also silently sets the maximum duration over which your analytics can associate two events with the same person.

Nobody wrote that down. There is no config value for it. It emerged from "the site should be open to everyone" and it landed at roughly ten minutes, and it will stay at roughly ten minutes no matter how good the analytics get, because the analytics were never the constraint.

You cannot measure a relationship that outlives your identifier. Whatever your system uses to say "this is the same person as before" — a session, a token, a cookie, a device id — has a lifespan, and every question your analytics can answer must fit inside it. Ask a longer question and you do not get a wrong answer. You get an empty one, which is worse, because empty looks like evidence of absence.

Why it presents as a bug

The reason this costs a day rather than a minute is that the symptom shows up in the wrong place.

The number that looks wrong is in the analytics table. So you debug the analytics: the handler, the function, the header, the proxy. All of that code is fine, which means every check comes back clean, which feels like you are narrowing in on something subtle when in fact you are auditing a layer that has no defect in it at all.

The defect is not in the layer that produced the symptom. It is in a design decision one layer down and several months back, which is working exactly as intended and has no idea it is also an analytics constraint. Instrumentation problems present as instrumentation. Architecture problems also present as instrumentation.

The tell was the distribution, not the magnitude. 0.44% tells you almost nothing — it is consistent with a lossy pipeline, a rare event, or a design ceiling. The shape of when those six occurred distinguishes between them in one query. When a number looks broken, the useful question is rarely "how big is it" and almost always "how is it distributed".

The fix is a design change

More tracking would not have helped, because nothing was going untracked. The change is to stop borrowing identity from the session.

So every tracked click now also carries a first-party identifier generated once in the browser's local storage, and signup copies that identifier onto the new account. Joining the two recovers the whole visit: what somebody clicked before they had an account, and — the part that actually matters — what they clicked days afterwards.

That is a different kind of thing from a session. It is not authentication and it grants nothing. It is one value whose only job is to outlive the login, because the login was never going to live long enough to answer the question.

What it still cannot do

Worth being precise about, because an identifier like this invites more confidence than it earns:

It identifies a browser, not a person. Cleared storage, a second device, a private window — each one starts fresh. Somebody who browses on a phone and books on a laptop is two rows and always will be.
It undercounts, never over. It can fail to connect two events that belonged to the same person. It cannot merge two people into one. Given a choice about which direction to be wrong in, that is the right one.
It only works going forward. Every existing account predates it, so their history stays blank. There is no backfill, because the data was never captured — and inventing a join key retroactively would produce exactly the kind of confident wrong answer this was meant to eliminate.

None of that makes it a weak instrument. It makes it an instrument with a stated range, which is the only kind worth having. The version I replaced also had a range. It just did not mention it.

The general shape

Every system that counts things has a horizon, and the horizon is usually set somewhere else entirely — in an auth model, a retention policy, a cookie lifetime, a log rotation window. None of those are described as analytics decisions. All of them are.

When a metric comes back far lower than it should be, there are three possibilities, and they look identical at a glance: the pipeline is lossy, the event is genuinely rare, or you asked a question longer than your identifier lives. The first is a bug and you should fix it. The second is a finding and you should accept it. The third is neither — it is a boundary, and no amount of work inside the analytics layer will move it.

Check the distribution before you check the code. If every observation you do have clusters inside one window, you have not found a leak. You have found the edge of what your system was ever able to see.

Your System Cannot Say I Don't Know

Ted — Fri, 31 Jul 2026 08:00:44 +0000

Four things I have been told by my own systems in the last three months:

0 clicks on outbound links, on a page I could watch people click through.
Clean from a typechecker, on a codebase with two type errors in it.
Position 13 for a page that has never once ranked 13th.
200 OK from an endpoint that did nothing at all.

None of these was a bug in the ordinary sense. Nothing crashed, nothing threw, nothing appeared in a log with the word ERROR next to it. Each was a real value, produced by a real system, working exactly as written.

And every one of them was a statement about the world that the system had no business making, because in each case the thing that was supposed to do the measuring had not run.

The overloaded value

Pick any layer of your stack and there is a single value doing two incompatible jobs.

A database query returns no rows. Does that mean the table has no matching rows, or that the query never reached the table? Both are null.

A process exits 0. Does that mean it checked everything and found nothing wrong, or that it checked nothing? Both are zero.

A counter reads 0. Is that "we measured, and it was none", or "nothing was ever counted"? Both are 0.

A command produces no output. Silence is what success looks like. It is also what a program that did nothing looks like.

This is not a language design mistake anyone can point at and fix. It is a consequence of representing absence at all: the moment you have a value meaning "nothing here", it will be produced both by nothing being here and by the lookup not happening. Every layer inherits it. Every layer passes it upward, where it stops looking like an absence and starts looking like data.

The asymmetry that keeps it hidden

Here is what makes this specifically dangerous rather than merely annoying.

In all four of my examples, the incorrect reading is the reassuring one.

Zero errors. All checks passed. No such user. Nothing to report.

A false alarm gets investigated within the hour, because it is uncomfortable and somebody wants it to go away. A false all-clear gets investigated never, because it is exactly what you hoped to see and there is nothing to chase.

So the failure mode is not just that the two states are indistinguishable. It is that the wrong one is load-bearing for your peace of mind, and the entire purpose of the signal is to let you stop looking.

That is why these survive for months rather than hours. A green check is not a piece of information you evaluate. It is permission to move on.

Four layers, one shape

The instrument. A dashboard showed zero outbound clicks. The tracking worked correctly — for one specific kind of link. Every other link on the site was a plain anchor tag the counter had never been taught about. So "0" was accurate about what it measured and silent about the fact that it was measuring almost nothing. The number was not wrong. Its scope was, and a number does not carry its scope. (Zero Is Not a Measurement)

The compiler. I ran a typechecker four times before shipping, and it exited clean every time. The project used a config that points at other configs and contains no source files of its own — a normal, widely-generated layout. Given that config, the tool typechecks an empty set, finds nothing wrong with it, and exits zero. It was not being lenient. It was being thorough about nothing. (It Passed Because It Never Looked)

The statistic. A page reported an average search position of 13. It had never ranked 13th. It ranked around 6 for one group of queries and around 20 for another, and the mean of a bimodal distribution is a number describing a state that does not exist. Unlike the others this one is not even a failure — the average is computed correctly, from complete data, by a system with no faults. It still describes nothing real. (One Page Ranked #6 and #20 at the Same Time)

The database. A password reset endpoint looked users up through an API layer that could not reach the table it needed. The call failed every time it ran. But only the result was captured and the error was discarded, so the failure arrived as null, and null was read as "no such user" — which routed into a branch specified to return success while doing nothing, because revealing whether an account exists is a security leak. (A Discarded Error Becomes an Answer)

Four different layers. One shape: an operation that did not happen, represented identically to an operation that happened and found nothing.

The question that separates them

There is one question that distinguishes a real signal from a decorative one, and it takes about a minute to ask:

If this were broken, what would I see?

If the honest answer is "exactly what I am seeing now", you do not have a signal. You have a value that happens to be reassuring.

Run it against the four:

If the click tracker were counting nothing, the dashboard would read 0. It read 0.
If the typechecker were examining no files, it would exit clean. It exited clean.
If the average were hiding a split, it would return a plausible middle number. It returned 13.
If the user lookup were failing completely, the endpoint would return 200 and send nothing. It returned 200 and sent nothing.

Four for four. In each case, "healthy" and "not running" produce byte-identical output, and I had been reading that output as evidence for months.

The follow-up question is the actionable one: what would have to change for these to be distinguishable? That is usually a small piece of work. Print how many files were checked. Report the count of rows the query considered, not just the result. Show a distribution alongside the mean. Return a different status code when the lookup itself failed.

None of that is clever. It is just refusing to let one value carry two meanings.

Unknown is a state you have to store

The fix that generalises is to stop modelling two states where there are three.

Not found and not found. Found, not found, and could not determine.

That third state has to survive as far as the decision. It is not enough to log it — logging is what you do with something you have decided not to act on. It has to reach the branch, and the branch has to treat it differently, because "I could not check whether this person has an account" and "this person has no account" call for opposite behaviour. One is a 500 that pages you. The other is a normal response.

In practice that means a handful of unglamorous habits:

Bind your errors even when you do not think you need them. Discarding an error does not make it not happen. It converts it into whatever your success path does with a missing value, which is usually to proceed confidently.
Make failure reachable, then check that it is. Break the check on purpose and watch it go red. If it stays green, you have not learned that your code is clean; you have learned that your check does not run.
Require the failure to be specific. A nonzero exit says something failed. It does not say this check found this fault. Assert on the content, not just the status, or an empty run and a passing run remain interchangeable.
Distinguish "no data" from "data says zero" in anything you display. They are different claims and a dashboard that renders them identically is lying quietly, every day, to whoever is looking at it.

Why this is getting harder

The volume of code arriving in my repositories that I did not type by hand has gone up sharply, and it is very good code by every measure my tooling can apply. It compiles. It is typed correctly. It follows the conventions of the file it sits in.

What it is not reliably doing is knowing things. It produces the plausible value confidently, and the plausible value is frequently right, which is exactly what makes the exceptions invisible. Generated code and swallowed errors have the same signature: something that looks like knowledge, occupying the place where knowledge should be, with no marker distinguishing it from the real thing.

Which means the burden shifts. It is no longer enough to check that the code is well-formed — every tool I have already does that, extremely well. The open question is whether the things it asserts are true, and almost nothing in a standard pipeline is built to ask.

The short version

Software very rarely fabricates information out of nothing. What it does constantly, at every layer, is take I could not find out and quietly round it to the nearest confident-looking answer — usually zero, usually null, usually success.

Then it hands you that answer with exactly the same face it uses for a real one.

A Second Check Is a Second Source of Truth

Ted — Thu, 30 Jul 2026 08:00:45 +0000

Signup on a site I run asked people to prove they owned their email address, and then asked them again.

First a six-digit code, emailed and typed back in. Only after that does the account get created. Then, on creation, the auth service sent its own confirmation email with a link in it.

Two emails. One address. Same question asked twice.

It read as thorough. It was actually a fork in the data, and it had already put a real user somewhere the system has no sensible answer for.

The state nobody designed

Someone signed up. They received the code, entered it correctly, and the account was created. The app signed them in immediately, as it is written to do.

They never clicked the second link. Why would they? They had just typed a code out of their inbox and were now looking at a logged-in page. From where they sat, the job was done.

So the record says: account exists, password set, session valid, signed in — and email_confirmed_at is null.

That row is not describing a half-finished signup. It is describing a completed one, plus a leftover flag from a check that nobody needed and nothing enforced. But a flag called email confirmed saying no is going to be read by something, eventually. Any support tooling, any audit, any future feature that gates on confirmation, will look at that user and conclude they never proved their address.

They did. Six weeks earlier. There is a used code in another table with their name on it.

Two records, one fact

This is the reframe that took me embarrassingly long.

I had been thinking of the second check as redundancy — belt and braces, no harm in it. It is not redundancy. It is a second place where the same fact is recorded, and two records of one fact will eventually disagree.

The fact is "this person controls this address." My OTP table knows it, in the form of a code marked used. The auth service knows it, in the form of a timestamp. Nothing keeps the two in sync, because they were never designed as one thing — one is application logic, the other is a platform default that was left on.

The moment a user satisfies one and not the other, the fact has two values. And when a fact has two values, every consumer downstream has to pick which one to believe, usually without knowing there is a choice being made.

The one that wasn't enforcing anything

There is a version of this where the second check is the real one and the first is a convenience. That would at least be coherent.

It was not that. The link enforced nothing.

The user was signed in before the confirmation email could possibly have been opened, because the app calls sign-in directly after creating the account. Sign-in did not consult the flag. Nothing in the product consulted the flag. The account worked identically whether the link was clicked or not.

So the second check could not block anything, could not gate anything, and could not fail in any way the user would notice. Its entire observable effect was to set a column that then contradicted the other record of the same fact.

That is the test worth applying: if a check cannot refuse anything, it is not a check. It is a log entry with an opinion.

Why it was on at all

Nobody chose this. The confirmation email is the platform default, and it is a sensible default — for an app whose signup is "create account, we'll email you a link." That is the flow it was designed for.

We had replaced that flow with our own and never turned the old one off. The original code even said so, in a comment:

// Create the user account with auto-confirm (since we verified email via OTP)

The intent was recorded. The setting it referred to was never changed. So the comment described a system that did not exist, sitting directly above the line that created the problem, for months.

I have a lot of sympathy for that comment. It is what happens when you build the replacement and reasonably assume the thing it replaces has stepped aside.

Fixing it, and the row already broken

Turning off the platform confirmation was one setting. New signups now get exactly one email, containing exactly one code, and land signed in with a single coherent record of the fact.

The stranded user needed a separate decision, and it is a more interesting one than it looks. The options were:

Leave them flagged unverified, which is false.
Set the flag to now, which is also false — it says they proved ownership today, which they did not.
Set it to the moment they actually entered the code.

I took the third. Their confirmation timestamp is now thirty-three seconds before their account was created, which looks like a data error and is in fact the most accurate statement available: they proved they owned that address, then the account was made.

I would rather have a timestamp that looks odd and is true than one that looks tidy and is not. Anyone who investigates that row will find the corresponding used code sitting right there in the other table, and the sequence will make sense.

What I would ask earlier next time

When you replace a platform flow with your own, the work is not finished when yours works. It is finished when the platform's version is off.

Until then you do not have a verified email address. You have two claims about a verified email address, maintained by different systems, agreeing by coincidence — and coincidences hold right up until a user does something entirely reasonable, like reading their inbox once instead of twice.

Your Toolchain Checks Grammar, Not Facts

Ted — Wed, 29 Jul 2026 08:00:21 +0000

I went looking for one wrong email address and found six wrong domains.

They had been in production for months, across six different server functions. Nothing had ever complained, because there was nothing in the stack capable of complaining. Every one of them was valid code.

What was actually there

Three distinct hosts, none of which I own.

The first was the .com version of my own brand, which I have never owned — the site runs on a different TLD. Somewhere along the way, generated code decided the .com was the obvious address and used it anyway: in a contact-form reply telling users to email support there, and in admin links inside security alerts.

The second was the .io version. Same idea, different guess. This one was worse, because it was not only a link:

from: "Alerts <alerts@brand.io>",

That is the sender on a batch of subscriber notifications. Transactional email providers verify sending domains. You cannot send from a domain you have not proven you control. So every one of those emails failed at the provider, silently, from the day the code shipped.

The third was the provider's own shared test domain — the address used in every tutorial and quickstart:

from: "App <noreply@resend.dev>",

That one was on the password reset email. Shared test domains are heavily restricted, typically to your own account address. So the one email in the system that a locked-out user depends on was going out from the least deliverable sender available.

Why none of it was caught

Consider what each stage of a normal pipeline actually knows.

The typechecker knows "alerts@brand.io" is a string, and a string is what the field wants. Correct, and finished.

The bundler knows the file parses. Correct, and finished.

The tests know the function returns the shape they asserted. If they mock the mail client — and they do, because nobody sends real email in CI — they cannot see the sender at all.

The deploy knows the function compiled and uploaded.

Not one of these stages holds a fact about which domains I own. That is not an oversight in any of them. Type systems check the shape of values, not their correspondence to the world. There is no type that means "a domain this organisation controls."

So the whole apparatus is checking grammar. The sentence is well-formed. Nobody is checking whether it is true.

Generated code fails where the plausible answer is wrong

This is the part that has changed for me.

When I write a domain by hand, I am reading it off something — a dashboard, a DNS record, an invoice. When a model writes one, it is producing the most plausible token sequence given the context. For a brand named X, x.com is overwhelmingly the most plausible domain. For an email provider's sender field, the address from the documentation is the most plausible value.

Both guesses are excellent guesses. They are what a reasonable person would assume. They are also wrong, and wrong in a way that is invisible in review, because support@brand.com looks correct on the page unless you already know the answer.

That predicts where to look. These errors cluster wherever the correct value is guessable: brand domains, support addresses, affiliate and partner IDs, API senders, dashboard URLs. Anywhere there is an obvious-looking answer, the obvious-looking answer is what you get, and it is right often enough that the exceptions never get checked.

It is not a hallucination in the dramatic sense. Nothing is invented. Something real is substituted for something true.

Declare your external surface

The fix took about twenty minutes. Extract every URL host and email domain from the source, compare against a list of hosts you have declared, fail on anything else.

✗ 12 undeclared domain reference(s) in git ref abc1234:

  resend.dev  (5 references)
      functions/send-password-reset/index.ts:184
      functions/verify-code/index.ts:155
      … and 3 more

  brand.io   (3 references)
  brand.com  (3 references)

Run against the commit from before I started fixing things by hand, it reported all eleven references I had found — plus one more I had missed entirely. A page was setting its canonical URL to a domain I do not own, which meant it had been telling search engines that the authoritative copy of that page lives on someone else's property.

I found six by reading. The script found seven, in about a second, and will keep finding them.

Two details matter more than the regex:

The allowlist belongs in the repository, not in configuration. The failure mode is generated code introducing a plausible host. If the allowlist lives somewhere an assistant can quietly extend, it stops being a check and becomes a formality. In the repo, widening it is a diff, and a diff gets looked at.

Where it runs is not obvious. A pre-push hook is the tight loop, and it only sees commits that pass through your machine. If you use a hosted agent that commits and pushes on its own — mine does — a local hook is blind precisely where most of the generated code enters. So the same script also runs on a schedule against the remote branch, and alerts only when the set of offending hosts changes, so a known-unfixed finding does not decay into noise.

What it does not do

It catches wrong constants. It does not catch wrong behaviour.

The same week, I fixed a lookup that had been failing on every call for months because it queried a schema the API layer does not expose. Valid code, real function, correct arguments, and the error was being discarded so the failure read as an ordinary result. No domain check would ever see that. It is a different class and it needs a different tool.

I would rather say that plainly than let a script imply more coverage than it has. A guard that quietly under-delivers is the thing we are trying to get away from.

The general version

There is a category of error that generated code produces far more readily than handwritten code, and that conventional tooling is structurally unable to detect: values that are well-formed and untrue.

You cannot review your way out of it, because the values look right. You cannot typecheck your way out of it, because they are the right type. What you can do is write down the small set of facts your project depends on — which domains are yours, which accounts, which endpoints — and make a machine compare the code against that list every time.

The list is the interesting artifact. Not because it is clever, but because until you sit down to write it, nobody has ever actually enumerated what your codebase is allowed to talk to.

A Discarded Error Becomes an Answer

Ted — Tue, 28 Jul 2026 20:03:11 +0000

A user told me password reset was broken. They clicked the link, entered their address, got the confirmation message, and no code arrived. They tried again. They checked spam. Nothing.

The endpoint was returning HTTP 200 with a friendly message every single time.

The send was never reached

The function writes the code to the database before it emails it. So the first useful question was not "did the email fail" but "is there a row". There wasn't — not for either attempt.

That moves the fault upstream of anything to do with email, which is where I had been looking. Something returned early.

Here is the line that did it:

const { data: existingUser } = await supabase
  .schema("auth")
  .from("users")
  .select("id")
  .eq("email", normalizedEmail)
  .maybeSingle();

const userExists = !!existingUser;

That call can never succeed. It reaches for the auth schema through the REST layer, and the REST layer does not expose auth — only public and one other. The request fails every time it is made.

You cannot tell from the code, because the error was never bound. Only data was destructured. The failure had nowhere to go, so it went nowhere, and existingUser came out null.

Three states, one variable

null here is doing an enormous amount of work it was never asked to do.

There are three possible worlds:

The lookup ran and found a user.
The lookup ran and found nobody.
The lookup could not run.

The code models two of them. World three collapses into world two, because both produce null, and !!null is false either way. An infrastructure failure and a factual answer about a person become the same value.

And once they are the same value, the next line acts on it with total confidence:

if (!userExists) {
  // Don't reveal whether the account exists.
  return json({ success: true, message: "If an account exists, a reset code will be sent" });
}

The privacy feature was the camouflage

This is the part I keep turning over.

That branch is correct. It is there on purpose. If a password reset endpoint answers differently for a registered address than an unregistered one, it becomes an oracle: anyone can feed it addresses and learn who has an account. Refusing to distinguish the two cases is standard practice and it is the right call.

But look at what it does in combination with the swallowed error.

The endpoint's job, in that branch, is to be indistinguishable from success while doing nothing. That is the specification. So when a total backend failure routed into it, the failure inherited a response designed from the ground up to look exactly like the working path — to the user, to the client code, to logs, to monitoring.

A deliberate decision to reveal less became the reason nobody could see the outage. The feature was not bypassed or defeated. It worked perfectly, on the wrong input.

I do not think there is a clever fix for that tension. Anti-enumeration branches have to be quiet. What has to change is what is allowed to reach them.

The same shape, three more times that day

Once I knew what I was looking for, it was not one bug.

The same lookup appeared in the function that completes the reset. It needed the user's id to set the new password, got null, and returned "user not found". So even if I had fixed the sending half, a correct code would still have failed to change anything. Two functions, one root cause, and fixing either alone would have looked like the fix didn't work.

A third copy guarded invitations — "does this person already have an account?" It always answered no, so the guard never fired.

The rate limiter counted rows in a table that does not exist. Same pattern: query fails, error discarded, data is null, null?.length || 0 is 0, zero is below the threshold, no limit applied. A rate limiter that had never once limited anything, reporting healthy by returning a number.

Silence is not evidence

The common thread is not carelessness about errors. It is that in each case, an absence was allowed to mean something.

No row means no user. No count means no traffic. No output means no problems. Each of those readings is reasonable, and each is only valid if you already know the check ran. Nothing in the code established that, so the inference ran on an assumption that was never checked.

That is worth separating from the ordinary advice to handle your errors. The problem is not an unhandled exception crashing something. Nothing crashed. The problem is that a failure was converted into a confident, plausible, wrong claim about the world, and then acted on.

What actually changed

The lookup now goes through a database function that can read the table it needs, and — more to the point — the error is bound and checked:

const { data: userId, error: lookupError } = await supabase
  .rpc("auth_user_id_by_email", { p_email: normalizedEmail });

if (lookupError) {
  console.error("lookup failed:", lookupError);
  return json({ error: "Could not process reset request" }, { status: 500 });
}

if (!userId) {
  return json({ success: true, message: "If an account exists, a reset code will be sent" });
}

Same two branches as before, plus a third that was always necessary and always missing. The rule it encodes is small enough to state in one line:

A lookup that could not run is not a verdict about the user.

It is a 500. It is our problem, not a fact about them. The quiet branch is still quiet, but only genuine answers are allowed to reach it.

The rate limiter got the same treatment — a failed count now returns an error rather than a permissive zero. If you cannot tell whether someone is over the limit, "0" is a guess wearing a number's clothing.

It used to work

The part I find hardest to sit with is that this was not a thing nobody ever got right. It worked for months. Here is what the broken lookup replaced:

// Check if user exists
const { data: userData, error: userError } = await supabase.auth.admin.listUsers();
const userExists = userData?.users?.some(user => user.email === email);

That version goes through the admin API, which can read the table. It loads every user and scans them in memory, which is genuinely wasteful and gets worse as you grow. Replacing it was a reasonable thing to want.

The commit that replaced it left a comment saying exactly what it was for:

// Targeted user lookup (O(1)) instead of listing all users

That comment is honest and the intent behind it is good. The new query really is O(1). It is also a query against a schema the REST layer does not expose, so it returns nothing, forever — and because the error was dropped, "nothing" got read as "no such user" and routed into the branch designed to look like success.

An optimisation replaced a slow correct answer with a fast wrong one, and every downstream signal said fine.

That reframes what the failure actually is. Not neglect, not inexperience — a deliberate improvement to a working system, applied without a way to notice it had stopped working. The old code was scanning every user on every request and that was visible: it would have shown up in timings, in cost, in anything watching. The replacement was instant, because doing nothing is very fast.

Nobody reported it in the ten weeks it was broken. They tried, got a reassuring message, and left. I had no signal at all, because the endpoint was reporting success the entire time.

It was found because one person mentioned it in passing, and because a row that should have existed didn't.

If you have an endpoint that deliberately returns the same response whether or not it did anything, go and check that it does the thing. It is the one place in your system where working and broken were designed to look identical, and that design is doing its job for both.

It Passed Because It Never Looked

Ted — Mon, 27 Jul 2026 06:17:34 +0000

I ran the typechecker. It passed. I ran the build. It passed. I pushed, and a page that carries a large share of the site's traffic started rendering an error screen instead of content.

The error was not subtle:

error TS2350: Only a void function can be called with the 'new' keyword.

That is about as blunt as TypeScript gets. It is not an edge case, not a version-specific quirk, not something that needs strict mode to surface. It is the compiler saying you tried to new something that cannot be constructed.

So why did four consecutive clean runs of npx tsc --noEmit tell me everything was fine?

The command examined nothing

The project's tsconfig.json looks like this:

{
  "compilerOptions": { "...": "..." },
  "files": [],
  "references": [
    { "path": "./tsconfig.app.json" },
    { "path": "./tsconfig.node.json" }
  ]
}

This is a solution-style config. It contains no source files of its own — "files": [] says so explicitly. It exists to point at two other configs, one for application code and one for build tooling. The pattern is normal and sensible; it lets you apply different rules to your app than to your config files.

The catch is what plain tsc does with it. Given a config with project references, tsc does not descend into the referenced projects. It typechecks what the config directly includes, which here is nothing at all, finds no errors in that empty set, and exits zero.

The command that actually checks the application code is:

npx tsc -p tsconfig.app.json --noEmit

Run that against the same commit and both errors appear immediately, with file and line numbers.

This is not an exotic misconfiguration I inflicted on myself. It is the shape that scaffolding tools generate by default for React and TypeScript projects. A very large number of repositories have this exact layout, and in every one of them the reflexive npx tsc --noEmit is a no-op that looks like a pass.

The build does not save you either. Modern bundlers transpile TypeScript by stripping the types — they never check them. A clean build tells you the syntax parsed. It says nothing about whether the types are coherent.

The bug itself was a dead import

The underlying mistake is almost funny in isolation. The file imported a set of icons from an icon library, and the list included one named Map:

import {
  MapPin, ExternalLink, Info, CheckCircle2,
  Store, ListChecks, ArrowRight, Map, ShieldCheck,
} from "lucide-react";

That import shadows the global Map constructor for the entire module. So when I later wrote what I thought was an ordinary lookup table:

const websiteBySlug = new Map(
  rows.filter(r => r.website).map(r => [r.slug, r.website])
);

…I was not constructing a Map. I was trying to call new on a React component that renders an SVG. Hence: only a void function can be called with the new keyword.

The detail that stuck with me: that icon was never rendered anywhere in the file. It was a leftover from an earlier edit, an unused import with no visible effect on the page. Its only remaining function in the codebase was to shadow a global constructor and lie in wait. Deleting the line was the entire fix.

If you use an icon library, it is worth knowing which of its exports collide with JavaScript globals. Map, Set, Image, Text, Menu, Link, History, Option, Search, Frame — all common icon names, all real globals.

Green and zero are not the same kind of wrong

I have written before about a dashboard reading zero when nothing was counting — a null result produced by the instrument rather than the world.

A false green is worse, and the reason is behavioural rather than technical.

A zero is uncomfortable. It contradicts what you hoped for, so it invites a second look. Even when you accept it, you accept it as a finding, and findings get revisited.

A green result closes the question. It is the outcome you wanted, arriving in the form you expected, and it terminates the investigation by design. That is its whole purpose. Nobody audits a passing check, because a passing check is the thing you run instead of auditing.

So the two possible meanings of my clean exit code —

everything was examined and nothing was wrong
nothing was examined

— are indistinguishable from outside, and only one of them prompts any further curiosity. I got the second one four times and read it as the first, because that is what green is for.

The tell was there

There was evidence, and I walked past it repeatedly.

npx tsc --noEmit produced no output whatsoever on a codebase of several hundred routes. Not a warning, not a note, not a summary line. Just an instant return to the prompt.

That should have registered. A typechecker that finishes instantly on a large project and says nothing has either done something impressive or done nothing at all, and the second is far more likely. I read the silence as cleanliness. Silence and success produce identical terminal output, which is precisely the problem.

Verify the verifier

Make your check fail on purpose. Introduce a deliberate error — a genuinely broken line — and confirm the tool actually complains. If it stays green, you have not discovered a robust codebase. You have discovered that your check does not run.

I first wrote that as a setup step: do it once when you create the project, and again if you inherit a repository or change build tooling. FromZeroToShip pushed back on the word once, and their counterexample is sharper than my own.

Their scanner had an exclusion rule that quietly stopped excluding what it was meant to. Six files were scored wrong for weeks and nobody opened the report, because it was green. They had already run the deliberate-break test months earlier and it had passed — the check was genuinely falsifiable on the day they tested it. Then the pattern it matched widened, the failure condition stopped being reachable, and "no errors" stayed true for the wrong reason.

That is my ending reached by a different road. Mine was never reachable. Theirs stopped being reachable. Neither transition emits anything at the moment it happens, so a one-time proof has an expiry date you cannot read from the outside.

Which means the deliberate break should not be a habit. It should be a job. Break each guard on every run, and require it to go red before its verdict counts.

And require it to go red for its own reason. A nonzero exit is not enough — that only says something failed, not that this check found this fault. My own failure signature was not a wrong answer, it was no answer: exit code 0 with empty output, byte-identical to exit code 0 on a genuinely clean codebase. An assertion on the content of the complaint — given this broken line, I expect TS2350 in this file — separates them immediately, because an empty check has nothing to say. Silence only becomes evidence once something is required to speak.

The generalisation goes past TypeScript. Any verification step can degrade into theatre: a test suite where a path filter stopped matching, a linter whose config no longer resolves, an integration check pointed at a stale environment. In each case the pipeline goes green, the ritual is observed, and nothing is being verified.

An unrun check is not neutral. It is worse than having no check at all, because no check leaves you appropriately nervous, and a broken one sells you confidence you have not earned.

I shipped a crash to a production page with four green checkmarks behind me. The compiler had the answer the entire time. I just never asked it anything.

Zero Is Not a Measurement

Ted — Tue, 21 Jul 2026 13:19:11 +0000

The affiliate dashboard said zero.

Not a small number — zero. One booking platform account had been quietly accumulating commissions for months. The other, for a different product category on the same site, had never recorded anything at all.

The conclusion writes itself. That category gets a fraction of the traffic. The audience is narrower. The intent is weaker. Some channels just don't convert, and the honest thing to do is stop pretending otherwise and put the effort somewhere with a pulse.

I'd been reasoning that way for a while. It's a reasonable read of a zero.

Then I actually looked at the links.

Nothing was counting

There were ten of them, all pointing at the booking platform's public product pages. Clean URLs. They worked — click one and you land exactly where you should, on the right product, ready to book.

Not one of them carried a partner ID.

The platform had no way to know those visitors came from my site. No parameter, no identifier, nothing to attribute a booking to. Every click for months had arrived as anonymous traffic. The commission wasn't small. It was structurally impossible.

The detail that makes this worth writing down is what came next. I checked whether the platform's integration was set up at all — and it was. Their analytics script was sitting in the page head, correctly installed, carrying the right partner ID. Someone had done that step properly and reasonably concluded the integration was finished.

It wasn't finished. Those are two different systems doing two different jobs. The script powers the vendor's widget analytics. Attribution comes from the parameter on the outbound link. Installing one tells you nothing about the other, and the dashboard that would have revealed the gap was the same dashboard reading zero.

So the setup looked complete, the links looked fine, the clicks were real, and the earnings were structurally guaranteed to be nothing.

The money wasn't the loss

At the volume involved, the forgone commission was trivial. Pennies, honestly.

What I actually lost was the ability to learn anything. For as long as that channel was uninstrumented, a click that converted and a click that went nowhere produced identical output: silence. There was no experiment I could have run, no amount of patience, no traffic threshold that would have separated "this doesn't work" from "nothing is watching."

And I had been drawing conclusions from it the whole time. Every judgment I'd made about that category — that the demand was weak, that it wasn't worth building out — came from a system that was incapable of reporting success. The zero wasn't evidence. It was a measurement that had never been shown to work.

Then it happened again, four hours later

Same day, unrelated system.

I was auditing internal links in a build output, trying to work out why a particular page had almost no search visibility. I searched the generated HTML for links pointing to it and got zero results. Nothing on the entire site linked to this page.

That's a real diagnosis with a real fix, and I wrote it up as one. Orphaned page. No inbound links. That's why it's invisible.

The search pattern was wrong. The build writes links as absolute URLs — full domain and all — and I'd searched for the relative form. When I corrected it, the page had nineteen inbound links. More than any comparable page on the site. It was one of the best-linked pages I had.

I'd already built a recommendation on top of that zero before I checked it.

What a zero actually is

Here's the thing I keep relearning.

A zero is not an observation about the world. It's a claim your instrument makes about the world — and a broken instrument makes that claim with exactly as much confidence as a working one.

"Nothing happened" and "nothing was recorded" are indistinguishable from inside the data. There is no field that separates them. Both render as the same empty result set, the same flat line, the same blank dashboard panel.

This is what makes null results uniquely treacherous compared to wrong ones. A wrong number tends to announce itself eventually — it contradicts something, it fails a sanity check, someone says that can't be right. A zero contradicts nothing. It's perfectly consistent with a world in which the thing you're measuring simply doesn't happen. It slots neatly into whatever story you already believed, which in both my cases was a story I found quite easy to accept.

Worse, a zero is actionable in a way that invites you to stop looking. It says: this channel is dead, this page is orphaned, move on. It closes the question. Both of my zeros came with a ready-made next step attached, and in both cases the next step was work I'd have done for nothing.

The habit that catches it

The fix is embarrassingly cheap, which is why it's worth making automatic: every zero needs a positive control.

Before you believe a null result, point the same instrument at something you already know should be non-zero. Not a different instrument — the same one, same query, same pattern, same code path. Then check that it returns something.

If I'd counted inbound links to a page I knew was heavily linked and gotten zero for that too, the broken pattern would have surfaced in about ten seconds. The failure was entirely visible; I just never gave it a chance to show itself, because I only ever asked the question where I didn't already know the answer.

The affiliate case is the same shape. One deliberate test click through the full path — link, redirect, dashboard — would have made the silence audible immediately. That test is five minutes and you do it once, at setup, when you still remember what the system is supposed to do.

The general rule I'd write on the wall: any measurement pipeline that has never once produced a non-zero reading has not been tested. It's not reporting an absence. It's reporting nothing at all, and you have no basis to tell those apart.

Sometimes you can't manufacture a positive. The event is genuinely rare, or producing one costs real money, or the only way to get one is to write fake data into production and hope you remember to take it out. That situation is common enough that pretending otherwise would make the rest of this advice easy to dismiss.

When it happens, the answer isn't to shrug and trust the zero anyway. It's to downgrade what the zero entitles you to claim. An uncontrolled null is not a negative result — it's an unknown, and it should sit in your head as I have no reading rather than the reading is nothing. Those two states feel identical and justify completely different decisions. One of them tells you to shut the channel down; the other tells you that you still have work to do before you're allowed an opinion.

What I changed

The links carry their partner ID now, and I confirmed the format against the platform's own link generator rather than guessing the parameter name — because a wrong parameter would have failed in precisely the same silent way, and I'd have congratulated myself on a fix that changed nothing.

The bigger change is procedural. Whenever a check comes back empty now, I ask one question before I do anything with it: what would this look like if my measurement were broken?

If the answer is "exactly like this" — and with a zero, the answer is almost always "exactly like this" — then I haven't found anything yet. I've just found a place where I need to look harder.

You Deleted It. The Crawler Still Sees It.

Ted — Sun, 19 Jul 2026 07:56:19 +0000

I removed two listings from a directory site I run. Not for a small reason — they failed the one promise the directory exists to make. The kind of removal you do immediately and double-check.

I did double-check. Flipped the flag in the database, loaded the site, watched the queries filter them out. Gone from the city pages, gone from the main directory. Removal confirmed.

A day later, both were still on the site.

Not on some cached page or stale CDN edge. In the current, freshly deployed HTML — sitting in the exact lists I had watched them disappear from. I had verified the deletion in one document. The site is two.

A rendered site is two documents

Here's the architecture, because the bug lives in it. The site is a single-page app: the browser downloads a JavaScript bundle, the bundle fetches data, the page assembles itself client-side. Crawlers are historically bad at waiting around for that, so — like a lot of SPA operators — I run a prerender step at build time. A script fetches the same data, bakes real content into each route's static HTML, and that is what robots, link previews, and anyone with broken JavaScript actually receives. The bundle takes over afterward and re-renders everything live.

It works. It's also a decision with a consequence I under-appreciated for months: every fact on the site now exists twice. Once in the live render, computed in the browser from a fresh query. And once in the baked copy, computed at build time by a different script, from a different query, through a different code path.

Two documents. Two authors. One URL.

And the two authors don't consult each other. The browser filters listings with a WHERE allowed = true. The build script — written earlier, for a different purpose — never selected that column. So when I "deleted" the two properties, the browser's query dutifully dropped them, and the build script kept baking both into the static HTML on every deploy, complete with links and ratings. From its perspective, the deletion had never happened.

The deletion was real in one document and fiction in the other. Nothing errored, because nothing was wrong: both documents were internally consistent, both rendered cleanly, both passed every check that looks at documents one at a time.

Three divergences, one day

Once I knew to look for the split, I found it three times in a single day — same site, escalating stakes.

The title fix that wasn't. A page was earning search impressions for a phrase its title didn't contain, so I added the phrase. Checked the page in the browser: there it was, in the tab, in the markup. But client-side titles are set by the bundle after load — the crawler reads the baked <title>, and the baked title still didn't have the phrase. The fix was fully live for every human and invisible to the one reader that determines whether the page ranks. It had been "fixed" in the wrong document.

The labels that didn't propagate. I'd spent that week making the site more honest — replacing a blanket "verified" badge with per-listing labels that say what's actually confirmed versus merely claimed. The app rendered the new labels beautifully. The baked copy went right on telling crawlers every listing was verified, in a header the build script had been stamping onto nine different pages. The honesty overhaul shipped to the browser and not to the record.

The deletion. The first two failures overstate; this one resurrects. A removed listing isn't a stale adjective. It's an entity I had decided, for cause, should not be presented — still being presented, by a document I'd forgotten was the one that counts.

And the failure runs in both directions. While fixing the build script I added a column to its fetch that didn't exist in that table. The data API rejected the query — and the script, hitting an error, quietly returned an empty list, at which point every detail page fell back to a generic template. No build failure. No warning. Just a deploy where the baked document silently regressed while the live one stayed perfect. Two documents means twice the ways to be wrong, and each one's failures are invisible from inside the other.

Why nothing catches this

Every tool I had was pointed at one document or the other. Never at the seam.

The type checker checks the app. The tests exercise the app. The browser — where all manual verification happens, because that's where you look at your site — runs the app. Meanwhile the build logs confirm the prerender ran, which is not the same as confirming what it wrote. The bug wasn't in either document. It was in the fact that there were two.

This is the oldest failure shape in data systems — the same fact stored in two places will drift, given time — wearing a disguise that makes it easy to miss: the two copies here aren't two database rows or two config files. They're two renderings of the same URL, which is precisely why it doesn't feel like duplication. It feels like one page. You'd never think to diff a page against itself.

"Done" names a document

The fix for the specific bugs was mundane: make the build fetch select the flag, filter on it, propagate the labels, correct the title — in the baked layer, where each change should have landed first. The durable fix was changing what I accept as evidence that a change shipped.

Looking at the site in a browser verifies the live document. It verifies nothing about the baked one. So the definition of done, for anything that matters to a crawler — a title, a claim, a listing's existence — is now mechanical: fetch the served HTML the way a robot would, with no JavaScript executed, and confirm the change is in the bytes. curl, grep, done. Ten seconds. It would have caught all four incidents before deploy, and it's the only check that examines the document I kept forgetting existed.

If your site renders twice, you don't have a page. You have a page and its understudy, and the understudy is the one performing for the audience that decides whether anyone finds you. Verify the performer that's actually on stage.

Further reading — adjacent failure modes from the same system: the prerender pipeline that ran on every deploy while writing empty pages, and prose as one more unsynchronized copy of your data. Same lesson at different layers: anything that exists twice, drifts.

Originally published on tedagentic.com.

The Listing Didn't Exist. The Booking Link Worked.

Ted — Sat, 18 Jul 2026 18:55:00 +0000

The property was called something plausible. It had a star rating, a price, a tidy paragraph of description, valid LodgingBusiness structured data, and a green "Verified" badge. It also had a Book Now button wired to a working affiliate link — the kind that pays a commission if someone clicks through and reserves a room.

The property did not exist. Not "closed," not "renamed." It was never real. Some fast, AI-assisted build had generated it whole — name, blurb, rating, amenities — and pointed it at a live payment rail.

It wasn't alone. There was a whole cohort of them, and the giveaway was almost funny: a dozen distinct "hotels" in different cities all sharing the same affiliate link. Different names, different descriptions, one destination URL. A forger who signed every painting with the same signature.

None of them were broken. That's the part worth sitting with. They passed every automated check I could throw at them. Valid types. Valid schema. No 404s, no build errors, no lint warnings. A validator confirmed they were well-formed, and being well-formed was the entire disguise.

Wrong implies there's a real thing being described

I'd spent earlier weeks fixing data that was wrong — a fact gone stale, an entity filed in the wrong city, a corrected value that never reached the page. Wrong is a solved kind of problem. Wrong means there's a true answer and the stored one disagrees. You audit, you compare against a source, you reconcile.

This was different, and it took me a beat to see why. You cannot reconcile a fabrication against a source, because there is no referent. The row doesn't describe a real thing incorrectly. It describes nothing, confidently. There's no fact to check because there's no subject.

So the first reframe: AI-generated seed data isn't a rough draft you refine. It's a forgery you have to detect — and its defining property is that it validates. Every instinct we've built says malformed data is the dangerous data, because malformed data is what breaks things. But malformed data announces itself. The genuinely dangerous record is the one that is perfectly formed and completely false, because nothing in your pipeline is designed to doubt a row that parses.

I deleted them. You don't refactor a forgery.

The quieter version of the same problem

The fake listings were the loud case. Underneath them was a quieter one that had spread much further, and it's the part that actually changed how I think.

Real listings had a policy field — a small piece of data recording whether a given property had actually been confirmed to permit a certain thing on-site. Most of them had never been confirmed. The field was empty.

Here is what the page rendered for an empty field: a green Verified badge. A "Yes" in the amenities grid. A line of structured data asserting the thing was permitted. A confident sentence in the server-rendered copy that a crawler reads before any of the interactive parts load.

Nobody decided to claim these things. There was no bad actor and no bad data. There was one line of code, repeated in spirit across the codebase, that read roughly: if this value is missing, treat it as true. A default. ?? true, or its cousin in four other files.

That is the second reframe, and it's the one I'd tattoo on a new engineer: a default is a claim. When you fill an unknown with the favorable value, you are not being helpful and you are not leaving it blank. You are asserting a fact you do not have. The empty field didn't stay empty on the way to the user. It was quietly promoted to a promise.

Absence rendered as a yes.

The missing value was "I don't know"

Once I saw it in one place I saw it everywhere, and they all traced to the same root cause. The data model had two states where it needed three.

There was true, there was false, and there was null — but null was never a state of its own. It was an input to be resolved, and every layer resolved it the same optimistic way. The card coerced it to yes. The badge coerced it to yes. The structured data coerced it to yes. The prerendered copy a crawler reads coerced it to yes. One missing fact became four confident assertions, none of which had any way to represent the honest answer.

Because there was no honest answer available. "Unverified" wasn't a value the system could hold. And a system that cannot represent "I don't know" does not fall silent about what it doesn't know. It reaches for the most flattering value in range and commits.

The fix was not clever. It was to add the value that was missing.

A real, explicit state for unconfirmed — stored, not inferred.
A three-way rendering instead of two: Yes, No, and Unconfirmed. The unconfirmed state gets its own muted styling — never the green of a confirmed yes, and just as importantly never the red of a confirmed no. "We haven't checked" is a different claim from "this is not allowed," and collapsing them is its own kind of lie, pointed the other direction.
Every layer taught to defer to that state instead of defaulting past it. The card, the badge, the structured data, the static copy the crawler reads — all four now say "unconfirmed" together, or say nothing, rather than four independent optimistic guesses.
Positive claims gated behind an actual confirmed value, so the system asserts a thing is true only when something actually backs it.

Give the model a way to shrug

The lesson generalizes past any one field or any one site. It applies anywhere a system holds facts it can't fully back — which is everywhere, and increasingly so as more of the initial data gets generated rather than entered by someone who checked.

We tend to think of honesty in a system as the absence of false statements — no bugs, no stale rows, a clean audit. But you can pass every audit and still have a system that lies, because the lies aren't in the data. They're in the defaults. They're in what happens to a value you never had.

Real honesty is the presence of "unknown" as a first-class thing: a value you can store, a state you can render, a claim your structured data is allowed to decline to make. If the only options your model offers are yes and no, then every gap in your knowledge silently becomes whichever one flatters you — and the extreme end of that gradient is a hotel that doesn't exist with a working button to book it.

Give the model an explicit way to say "I don't know." Otherwise it will nod at everything, and you won't find out where until someone audits the room and discovers half of it was painted by the same forger.

Further reading — the adjacent failure modes: why AI tools produce confident silent errors on production sites, and why a fallback chain hides missing data by design. This post is the layer under both: why the gap gets filled with a lie in the first place.

Originally published on tedagentic.com.

Logging Into the Higgsfield CLI on a Server With No Browser

Ted — Wed, 08 Jul 2026 04:08:43 +0000

I wanted the Higgsfield CLI on my server — a terminal front-end to a pile of image and video generation models, which makes it scriptable and cron-able in a way the web UI never will be. The install was one line. Then it asked me to log in, and a specific, recurring failure showed up that has nothing to do with Higgsfield and everything to do with a word that means two different things at once.

The word is localhost.

The server is headless: no monitor, no browser, reached only over SSH from a laptop. That detail is the whole story.

The flow that assumes a lap

Higgsfield authenticates with OAuth 2.0 PKCE, the way most modern CLIs do, and the polite version goes like this: the CLI opens a small HTTP listener on localhost:<port>, prints a login URL, and waits. You open the URL in a browser, approve, and the identity provider (Higgsfield fronts theirs with Clerk) redirects your browser to http://localhost:<port>/callback?code=.... The listener catches that request, reads the code out of it, trades it for a token, and you're in.

On a laptop this is seamless because the browser and the listener are the same machine. localhost resolves to the same place for both halves of the handshake. The whole design quietly depends on that.

On a headless server it falls apart, because the two halves are now on different machines. The listener is on the server. The browser is on my laptop. When Clerk redirects the browser to http://localhost:<port>/callback, the browser dutifully connects to the laptop's localhost — where nothing is listening — and shows:

Unable to connect

The login succeeded. The callback failed. The code was issued — it just landed on the wrong machine's front door.

A loopback OAuth flow assumes the browser and the listener share a host. That assumption is invisible on a laptop and load-bearing on a headless box.

The tunnel, and the wall the spec put in front of it

The reflex fix is an SSH tunnel. Forward the callback port from the laptop to the server, so the laptop's localhost:<port> actually reaches the server's listener:

ssh -L 8799:localhost:8799 you@server

Sound. It works. But I hit a wall trying to pin the port, and the wall is worth understanding because it's not a bug — it's the security model doing its job.

higgsfield auth login has a --port flag, so I told it to use 8799 to match my tunnel. Clerk rejected the whole login:

error: invalid_request
The 'redirect_uri' parameter does not match any of the
Client's pre-registered redirect URIs.

Here's the thing PKCE doesn't remove: the redirect URI allowlist. PKCE lets a CLI skip shipping a client secret — the code challenge proves the token request came from whoever started the login — but the provider still refuses to send a code to any callback URL the app didn't register in advance. That allowlist is what stops a malicious page from pointing your authenticated redirect at its server. Higgsfield registered a specific handful of ports with Clerk, and those are the only ones that work. 8799 wasn't in the club.

The redirect URI is a registered constant, not a free parameter. You don't get to pick the port; you get to pick from the ports the app's developer already blessed. Drop the --port flag and the CLI falls back to its own default and it works — but now your tunnel has to match a port you don't fully control, and if that port is already busy on the server (mine collided with another service on the default), the CLI silently falls back again to a different registered port, and your tunnel is aimed at the wrong one.

The tunnel isn't wrong. It's just more moving parts than the problem actually needs.

The callback is just data

The realization that made this easy: the callback is not a handshake, it's a message. It's a plain HTTP GET whose entire payload is visible in the URL — ?code=<the code>&state=<the anti-forgery token>. The listener doesn't care who delivers that message. It cares that the message arrives with a state matching the one it issued.

So I stopped trying to route the browser to the listener at all. I ran higgsfield auth login on the server (letting it pick its own registered port — it landed on 8766), opened the printed URL in my laptop browser, approved, and let the redirect fail — Unable to connect, as expected. Then I copied the dead URL straight out of the browser's address bar:

http://localhost:8766/callback?code=OWVH...F1L&state=kg6P...TCY

That URL is the whole message. The listener is sitting on the server's own localhost, exactly where the CLI is watching. So I replayed it there:

curl "http://localhost:8766/callback?code=OWVH...F1L&state=kg6P...TCY"

The listener saw a request on its port, checked the state, matched it, exchanged the code for a token, and printed Successfully authenticated. No tunnel. The browser did the one thing only a browser with my logged-in Higgsfield session could do — prove I'm me — and then I hand-carried the resulting code the last hop the browser couldn't make.

Two things make this safe to lean on, and both are worth saying out loud:

The state parameter is the check that makes replay legitimate. The listener issued that random token at the start and refuses any callback that doesn't echo it. I'm not bypassing a security control by replaying the URL — I'm satisfying the exact one the flow was built around.
The code is short-lived and single-use. An OAuth authorization code is a password with a fuse — good for one exchange, expiring in a minute or two. Copy it, replay it promptly, and don't paste it anywhere it'll be logged. It's not a token you're storing; it's a token you're spending immediately.

What I'd carry to any headless login

localhost is a per-machine word. Any auth flow that redirects to localhost is quietly assuming the browser and the service are the same box. On a headless server they aren't, and that mismatch — not the tool — is your bug.
Redirect URIs are an allowlist, not a setting. PKCE drops the client secret but keeps the registered-callback rule. If pinning a port throws invalid_request, you're fighting the security model, not a config typo. Use the app's own default port.
A loopback callback is replayable data. It's a URL carrying a code and a state. You don't need to route the browser to the listener; you can let the redirect die and deliver the URL yourself with curl on the machine that's listening.
state is what makes hand-delivery honest. It's the flow's own anti-forgery check. Replaying a URL that carries the right state isn't a workaround around the security — it's the security working.

The tunnel is the textbook answer, and some days you'll want it. But when a login on a headless machine tells you it can't connect to localhost, remember that it already succeeded — the code exists — and the last hop is just a URL you're allowed to carry across yourself. This worked cleanly for Higgsfield; it'll work for any CLI whose login is a loopback redirect, which by now is most of them.