I wrote a commit message that says SEO.tsx "sets documentElement.lang" and "emits hreflang links". I opened a Spanish post in the browser, looked at the document, and there it was: documentElement.lang === "es", and three <link rel="alternate" hreflang> tags for es, en, and x-default. Right language, right pairs, right default.
The commit message was accurate. The site was also serving <html lang="en"> on every Spanish post, with zero hreflang annotations.
Both of those sentences are true. They are true of different surfaces.
The annotation that was there and wasn't
This site is a React 19 SPA on Vercel with a build-time prerender step. The prerender writes the <head> into the static HTML so that crawlers get titles and meta tags without executing anything. SEO.tsx is a React component. It runs in the browser, after React mounts, and it does exactly what its commit message claims.
scripts/prerender.mjs writes the HTML the server actually sends. At that commit:
$ git show 86569b0:scripts/prerender.mjs | grep -c hreflang
0
The prerender rewrote only the <head> element. It never touched the <html> tag at all, so the lang attribute stayed at whatever index.html had baked in, which was en. And it emitted no alternates, because nothing in it knew the translation table existed.
I confirmed it with curl on all four paired URLs — English original and Spanish translation, both directions. Then I tried again with a Googlebot user-agent, in case Vercel was doing something clever for crawlers. Byte-identical response. There was no second delivery path. The sitemap had no xhtml:link alternates either.
There is a trap in this one worth naming, because it is what lets the mistake survive for a while. Every page on the site serves this:
<link rel="alternate" type="application/rss+xml" ...>
Grep the served HTML for alternate and you find a hit. If you are checking quickly, and you are already fairly sure the feature works because you watched it work, a hit is enough. The string you searched for was present. It was the feed.
The practical consequence is narrower than it sounds. Google renders JavaScript, so for Google this was a delay rather than a loss. The exposure was to engines that do not render, which saw Spanish posts declared as English with no indication that a translation existed.
Fixed on 21 August. prerender.mjs now sets lang on the <html> element and emits the alternates, reading src/config/translations.json — the same file src/config/translations.ts reads. The table lives in JSON because a .mjs build script cannot import a .ts module, and keeping two copies of it is precisely how the two halves would drift apart again.
Verified after: lang="es" on the two Spanish posts, lang="en" on the two English ones, three hreflang links on each of the four paired posts, and zero on an unpaired post like /tailwind-css — which is the correct number, since an hreflang pointing at a translation that doesn't exist is a false claim.
I did not verify the wrong thing here. I verified a real thing on the surface that was easier to reach.
The slot no test was watching
The second one isn't about checking the wrong surface. There was no surface to check.
Every post carries two ad units: one in-article, one at the foot of the post. On 20 August at 22:48 I shipped this line:
setStatus(ins.getAttribute('data-ad-status') ?? 'no-response');
Three seconds after mount, if AdSense had not written a verdict onto the element, the code invented one. The CSS collapsed the container on no-response, so the slot went display: none — while AdSense was still measuring it. An ad cannot be placed into a hidden, zero-width box. The guess made itself come true.
That was live for about ten hours. The fix, at 08:46 the next morning, removed the timeout entirely: only AdSense's own verdict changes the state, and a missing verdict means "pending", which reserves space instead of collapsing it.
That installed the exact mirror of the bug it fixed.
In production, the end-of-post unit reaches data-adsbygoogle-status="done" and then never receives data-ad-status at all. Under the new rule, "never" is indistinguishable from "not yet". It sat in pending permanently. Measured in a real browser on 23 August, on /portfolio-audit-20-problems, both slots at the same moment:
slot 9344511662 (end of post)
data-adsbygoogle-status "done"
data-ad-status null
iframes 0
height 303px
display block
"Advertisement" label visible
slot 2970675007 (in-article)
data-adsbygoogle-status "done"
data-ad-status "unfilled"
height 0px
display none
Same page, same load. The asymmetry was in my state machine, not in AdSense.
So for two and a half days, every post on this site ended with an empty rectangle labeled "Advertisement" — 303 pixels tall on the page I measured. Nothing was red. No test failed. The build passed, the page rendered.
The real fix: the grace period now starts only once AdSense itself reports done, and the observer stays attached afterward so a late verdict still wins. It also reads the attribute once immediately on attach, because a MutationObserver tells you nothing about what happened before you called observe(). Verified after: both slots 0px, display: none.
Green because nothing is asking
The two failures are not the same shape.
In the hreflang case there was a check and it was watching the browser DOM instead of the response body. Wrong surface, honest mistake, findable.
In the ad case there was no check. Not a weak one — none. And from outside, "no check" and "check passed" produce the identical signal: a build that goes green. Absence of a failing test looks exactly like presence of a passing one when all you can see is the color. A missing assertion doesn't announce itself the way a broken one does.
That is why the empty box lasted two and a half days and the hreflang bug lasted until I happened to run curl.
The thread this came from
I'm writing this because of a comment. On the dev.to cross-post of "How I Actually Code with Claude Code", Heinrich Neb put it better than I can: a check can be green and blind. His example was a test asserting that a benchmark number appeared in his server's output — where the number was a hardcoded string. The check guarded the sentence, not the measurement. It stayed green for weeks. Debashish Ghosal, in the same thread, suggested writing an outcome note onto each plan file after execution, which is a different attack on the same problem: make the record say what happened, not what was intended.
I don't have a general solution. I'm not going to end this with a testing philosophy, because I'd be inventing one on the spot to make the post feel finished.
What I actually have is a habit: open a browser, find the specific element, and measure it. Not the framework's idea of the element — the element. curl the URL instead of trusting the component that writes the tag. Read the height off the box instead of trusting the state that decides the height.
That is how both of these turned up: one at a time, by hand, because I happened to look. It would not have found a third.
I write up the things I break and fix at codewithgabo.com.
Top comments (7)
Being quoted is an honor; being quoted in a post this rigorous is a better one. Your line is going into my collection next to Marco's and Glen's: "Absence of a failing test looks exactly like presence of a passing one when all you can see is the color." That's the cleanest statement of the second failure mode I've read - and it's why counting checks is never enough; you have to count the surfaces nobody is asking about.
Your RSS-alternate near-hit deserves its own name, because it's a third shape: the known-good doppelganger. A grep-check needs a negative control against NEIGHBORING strings, not just against absence - "matches hreflang-alternate, does NOT match feed-alternate." We got bitten by the mirror image this week: an over-eager filter matched "readme" inside readme-generator.go and would have silently eaten a real source file; the probe that caught it was exactly a doppelganger case. Same discipline, both directions.
On "it would not have found a third": the countable version of your habit is an inventory. List the surfaces your consumers actually read - response body, sitemap, feed, headers - and give each one curl-assertion. That's maybe six assertions, and it's the perfect first content for the test runner you said comes first. And your MutationObserver detail - read the attribute once on attach, because the observer knows nothing about what happened before observe() - is a reusable gem that deserves to be more famous than the bug it fixed.
Built it. Twenty-eight assertions, and
npm testexists now, which itdidn't when you asked.
It failed on the first run.
og:image was the raw upload. Seven of eighteen covers are portrait or
1.75:1 — one is 2160x2700 — so those cards were being framed by Twitter's
rules rather than mine, and the eighteen images came to 10.10 MB. Cropped
to the card frame: 840 KB. Nothing had ever checked that surface,
including me, and it is read by more consumers than the ones I was
worried about.
The recursive part: I fixed it in the React component first, then
realized the og:image a crawler reads is the one the prerenderer bakes
into the served HTML, not the one React writes afterward. Same bug as the
post, one week later, inside the fix for it.
Your doppelganger control went in literally, and writing it forced a
second assertion I would not have thought to make: the feed link must be
present. Otherwise the negative control goes vacuous the day someone
removes the feed — the pattern stops matching the neighbour because the
neighbour is gone, and the test keeps passing for the wrong reason.
I added one more in the same spirit. Before any assertion that
distinguishes a prerendered page from the SPA shell, the suite asserts it
can tell them apart at all: the shell carries the shell title, a post
does not. If that ever stops discriminating, every test after it is green
and blind in your original sense.
On "it would not have found a third" — the inventory is the answer, and
the honest version is that the count is now something I can grow instead
of a habit I have to remember. Six surfaces in your list, eight in the
file, one bug on first contact.
The un-checkable ones are still un-checkable. Nothing in there would
catch a comment claiming the ad lands a third of the way in when the code
cuts at three blocks. But that category is smaller than I claimed, and it
got smaller because you made me count.
Twenty-eight assertions, a first run that caught 10.10 MB of og:image nobody had ever weighed, and a recursive bug inside the fix for the bug - this is the best possible outcome of a comment section, and the numbers are yours, not mine. Two things you built deserve names. The discriminator assertion - proving the suite can tell shell from page at all, before any assertion relies on the difference - converges exactly with a standing rule of ours: a checker only runs after three gates (red on the unsolved state, green on the solution, red again on the known-bad). Your version adds the gate we didn't have words for: "can this test distinguish anything?" That's going into our house vocabulary with your name on it. And "same bug, one week later, inside the fix for it" is a failure class worth publishing on its own: the fix inherits the blind spot of the thing it fixes. One question, since you now have an inventory instead of a habit: when did your suite last fail? If the answer drifts toward "never," you know which of our shared diseases that is.
"When did your suite last fail?" — two days ago, on its first run, and
again today.
Today's is the better one. I built the route-level byte budget you asked
about in the other thread, and it caught 2.6 MB of static PNGs sitting in
the gap between my two existing checks: not bundled, not from the CMS,
invisible to both. So the answer to "when did it last fail" is "when I
gave it a surface nobody was asking about," which I think is your point
rather than an escape from it.
Your three gates are stricter than what I have and I'm taking them. Red on
the unsolved state is the one I skipped — I wrote the discriminator
assertion, but I never checked that any of the others go red when the
thing they guard is broken. Which means I don't actually know they can
fail. That's the same question one level down, and I hadn't asked it.
"The fix inherits the blind spot of the thing it fixes" is a better name
than anything I had for it. I hit it twice now: the og:image fix in the
component instead of the prerenderer, and then Resource Timing as the
instrument for the byte budget — the obvious tool, which reports zero for
cross-origin responses without Timing-Allow-Origin. Every image here is
cross-origin. The check would have passed by measuring nothing.
"When I gave it a surface nobody was asking about" is the answer, and it is a better one than the question deserved. Twice in two days is not a suite that passes - it is a suite that is still finding the edge of what it can see.
On the gate you skipped: red on the unsolved state is the cheap one and the one everybody skips, including me until it bit. But I would add the ordering, because I got it wrong today. I checked that my new guard could go red before I checked that it went green on the fixed state - and the green run was the one that mattered, because it told me the guard was not simply always-red. Both directions in the same sitting, or you have half a control. An always-red check gets deleted within the month, and the deletion looks like housekeeping.
"The fix inherits the blind spot of the thing it fixes" - I got the same shape from a different angle this week, and it is worse than inheritance. I built a guard against invisible Unicode characters: zero-width spaces, soft hyphens, the things that silently break a regex-based check. Its first run went red on itself - the script contained the very characters it searches for, written literally into its own patterns. 26 of them. Then I wired the same check into the publishing path and did it again: 17 more.
Twice in one day, same self-reference, and only the run caught it. I had read that file twice.
So the version I would now put next to your sentence: the fix inherits the blind spot, and a check that examines a property tends to have that property. Anything that searches for a pattern contains the pattern. Anything that measures noise makes noise. The first run of a new guard should be pointed at its own source before anything else.
This is a great example of the same failure mode from a different angle: the check was green because it was observing a surface that was easier to reach, not the surface where the invariant actually lived.
The hreflang case especially resonates with my recent work on security verification. A test can exercise real code, produce the expected result, and still prove the wrong property. The ad case is even more interesting because there wasn't a failed check to debug — there simply wasn't a check capable of observing the failure.
I really like your closing distinction: measure the element, not the framework's idea of the element. That's essentially the same principle I'm trying to apply with mutation testing: don't trust that the test claims to protect the boundary; deliberately break the boundary and see whether anything goes red.
Great write-up. 🔐
Mutation testing is the right frame, and I think it names what my suite is
still missing. I have assertions that pass. I have not once broken the
thing they guard to watch them go red. So I know they agree with the
current state and I don't know they can disagree with anything.
The ad case is the one that stays with me for the reason you name: there
was no failed check to debug. Debugging assumes a signal. The whole
difficulty is that the absence of a signal and a healthy signal are the
same observation from outside.