DEV Community: dustin chu

Passing locally proves nothing: four ways production disagreed in one week

dustin chu — Tue, 25 Aug 2026 05:08:20 +0000

Last week I moved a site to a new domain, which meant redirecting 41 legacy URLs.

I did the responsible thing: I didn't wait for launch to test them. wrangler pages dev
runs the same redirect engine Cloudflare Pages runs, so passing locally should mean passing in
production.

That "should" broke four times in one week.

Four unrelated causes, and not one of them was what the symptom suggested. That's the
reason this is worth writing down — not "remember to test in production," which is useless
advice, but the fact that every one of these lied about what was wrong.

One: a rule that matched locally and not in production

Two legacy URLs contained parentheses:

/2017/06/20/java swing -教育訓練程式(第二版)/

In _redirects I covered both forms — the literal ( and the encoded %28. Verified locally,
row by row: 43 rules, all passing.

After launch, the two %28 rules did nothing. 404.

The observed production behaviour: with a literal ( in the rule, a client sending a literal
parenthesis gets a 301. A client sending %28 gets nothing, regardless of how the rule is
written. Locally, both forms matched.

I deleted the two dead rules. A rule that never fires in production is worse than no rule —
it tells the next person reading the redirect table (me, in three months) that the case is
handled.

⚠️ Worth noting: clicking the link by hand cannot find this, because browsers send the
literal parenthesis. Only row-by-row automated checking hits it.

Two: all 41 failed, and the site was fine

After launch I pointed the verification script at the live domain. All 41 rows failed, every
one a 403.

It looked exactly like the whole site was down.

The site was fine. Cloudflare was blocking my script's User-Agent — Python's stdlib default,
Python-urllib/3.x.

curl default UA       → 301  ✅
python urllib default → 403  ❌
custom tool UA        → 301  ✅
browser UA            → 301  ✅

Same URL. Change the UA, 403 becomes 301.

This class of failure is the dangerous one, because the symptom is indistinguishable from a
real disaster. Had I believed the screen, my next move would have been rolling back the
deploy or rewriting the redirect table — both of which break something that worked.

My checks now always send an identifying UA: <tool>/<version> (<contact email>). Not to
disguise anything — to avoid looking like noise, and so anyone who wants to block me can reach
me first.

Three: layout shift that only existed locally

After fixing cumulative layout shift on article pages, local kept reporting 0.212. Nothing I
changed moved it.

Production measured 0.

The images live on imgur, and imgur's hotlink protection rejects a localhost referer:

Referer: http://127.0.0.1:8788/  → 403
Referer: https://wisplu.com/     → 200
no Referer                       → 200

Images fail, the browser renders alt text instead, the box is a different size than the space
reserved — layout shift. That 0.212 was manufactured entirely by the local environment.

This one is the most uncomfortable of the four: I nearly restructured something to fix a
problem that did not exist.

Four: production, but measured too early

Not a local/production gap, but the same species — the verification method itself lying.

I checked immediately after a deploy finished and the homepage still showed old content. I
assumed the build hadn't picked up the change and started digging through the build pipeline.

I had simply hit an edge node that hadn't updated yet. Same URL with a cache-buster: new.
Same URL forty seconds later: new.

A false negative makes you act just as surely as a true positive — and act in the wrong
direction.

What the four have in common

The causes were rule matching, UA filtering, hotlink protection, and cache propagation — four
different layers. But the shape of the error is identical:

A difference between your test environment and the real one does not present itself as a
difference. It presents itself as a bug.

No hotlink protection locally doesn't look like "no hotlink protection here." It looks like
CLS 0.212. A blocked UA doesn't look like a blocked UA. It looks like 41 failing redirects.
They disguise themselves as failures you recognise, so you fix them the way you'd fix those.

What I do now

1. Local verification is fast feedback, never the verdict.
Its job is catching obvious breakage before deploy, not proving production will pass. Those are
very different claims.

2. Every automated check sends an identifying User-Agent.
This has saved me elsewhere too: Wikidata's query service also rejects default UAs, and it
doesn't return an error code — it returns something that isn't JSON, so the parser explodes and
it looks like a bug in your code.

3. When a check fails, ask "is it broken, or did I measure it wrong?"
Concretely: re-test along a different axis. Change the UA, change the referer, add a
cache-buster, wait and retry. If changing an axis fixes it, your method was broken, not the
thing.

4. Delete anything that doesn't work in production.
Dead rules accumulate into false confidence.

The most counterintuitive part: in three of the four, my first instinct was to go fix the
thing — and all three would have been the wrong fix.

The real value of automated verification isn't that it tells you something is wrong. It's that
it tells you row by row, which makes "41 of 41 failing" and "2 of 41 failing" visibly
different problems. All-failing is usually your method. Scattered failures are usually the
thing. That distinction is worth more than any individual check result.

My agent inflated its own state file to 49MB. Every check passed for three days.

dustin chu — Tue, 25 Aug 2026 01:35:24 +0000

I run an agent that wakes every four hours, does one thing, and writes its state back to
STATUS.md. That file is its entire memory across sessions — without it, every wake starts
from nothing.

Yesterday a write failed on a missing substring. I opened the file:

893,828 lines
49,414,452 bytes

49MB. It should be about 13KB.

Worse: it had been that way for three days, during which I read it, wrote to it, and
verified it daily. Every check passed.

The one character

The offending code replaces a section:

old = s[s.index('## Next') : s.index('### Index status')]
s = s.replace(old, new_section)

Looks fine. But what if ### Index status appears before ## Next in the file?

Then the second index returns a smaller number than the first, and a Python slice whose start
exceeds its stop doesn't raise. It returns an empty string.

So the line becomes:

s = s.replace('', new_section)

Which does something people rarely have cause to remember:

>>> 'abc'.replace('', '-')
'-a-b-c-'

replace('') inserts the replacement between every character.

A 13KB file with a 1KB section inserted this way becomes roughly 12.6MB. My agent then woke
every four hours and did the same thing again to the already-broken file. Three days later:
49MB.

That's not the interesting part

The bug is dumb and the story ends there. What's worth writing about is that I inspected that
file every day and the inspection kept saying fine.

Because replace('', x) deletes nothing. Every original character is still present. They're
just separated by thousands of copies of the same section.

So:

grep 'Last heartbeat' STATUS.md   # found
sed -n '5,8p' STATUS.md           # looks about right
grep -c 'published' STATUS.md     # returns a number

Every check passed, because everything I was looking for really was there.

After each write I habitually sed the first few lines to confirm the update landed. Those
lines had updated. I never looked at the whole file, because I thought I knew what it looked
like.

This is the same failure as my last post

Last time I wrote that a local test passing proves nothing about production, and the conclusion
was: a difference between your test environment and the real one doesn't present as a
difference — it presents as a bug.

What lied to me this time wasn't the environment. It was the granularity I chose to check at.

I was checking whether content was present. This failure doesn't remove content — it dilutes
it. The metric I picked is structurally blind to this class of fault.

That's harder to defend against than an environment gap, because at least you know your test
environment differs from production. Check granularity is something you choose yourself, and
you'll naturally choose to check for failures you can imagine.

What I changed

1. Check the size after writing, not just the content.

wc -c STATUS.md

One command. File size tells you almost nothing about whether content is correct, and it is
extremely sensitive to whether structure is broken. 13KB becoming 7MB is instantly visible.
I missed it for three days because it never occurred to me to look.

Cheap, coarse, unlyable signals are badly undervalued. A precise check tells you "the thing
you asked about is fine." A coarse one tells you "something is off" — and only the second kind
catches the failure you didn't imagine.

2. Assert the ordering before slicing.

a, b = s.index(A), s.index(B)
assert 0 <= a < b, f'range inverted: {a} → {b}'
old = s[a:b]
assert old, 'empty slice — stop'

The second assert matters most. An empty first argument to replace is essentially always a
bug; no legitimate use needs that behaviour.

3. Assert the file's shape, not just its contents.

I now check that a section heading like ## Next appears exactly once. During the breakage
it appeared 12 times — an earlier and far clearer signal than size, and I simply wasn't looking
for it.

One last thing, and it's the part that stings.

This system's entire premise is autonomous operation. It checks its own state every four hours,
writes logs, verifies its output. And it destroyed its own core memory file for three days
without noticing.

It wasn't failing to check. It checked diligently — it just checked the places it expected to
break.

If you're building something like this, add one rule: periodically verify the things that
can't possibly be wrong. The parts that can break are already being watched. What bites you
is whatever you're confident doesn't need watching.