On Nov 5, 2026, at 9:33 in the morning, we pushed to main. By 9:35 every request to since.dev answered 500. Every page, every visitor. The tests had passed.
The same day, a different commit went in with a failing test in it. The tests had "passed" for that one too.
Neither was a flaky test. Both did exactly what they were told. They were just answering a different question than the one I thought I was asking.
1. A small, nice change
The change behind the first one was honestly kind of nice. We wanted two things to happen before any page on the site renders:
- A coding agent that asks for a page with
Accept: text/markdowngets that page's Markdown instead of the HTML. Agents read Markdown better, and they shouldn't have to scrape our layout to get the words. - A link preview bot that asks for a scan result gets a card with that repository's own title and counts, so a shared scan unfurls as itself.
That logic lives in one small file that runs in front of every request. Like any reasonable person who doesn't want one giant file, I kept the helpers somewhere else in the project and imported two of them.
The unit tests imported the file and checked both behaviors. Markdown for agents, cards for bots, plain pages for people. Green.
2. Then it ran where it actually runs
Our host runs that one file on its own. It doesn't bundle it with what it imports. So in production, the two helpers it imported simply weren't there. The file failed to load. And since it sits in front of every request, every request failed with it.
The site wasn't slow, or broken on one page. It was a wall of 500s for everyone, from 9:35 to 9:43 MST.
The test runner never saw it, because the test runner resolves imports for you. That's a feature. It's a big part of why tests are pleasant to write. But it meant the test loaded the file in a world where ./lib/helper existed, and production loaded it in a world where it didn't.
Here's the same thing in miniature. One file that imports one helper, run two ways:
// site/lib/helper.mjs
export const greet = () => "hello from helper";
// site/middleware.mjs
import { greet } from "./lib/helper.mjs";
console.log(greet());
Run inside the project, then copied alone into another folder the way a host might load it (Node 24, in an empty temp folder):
cd site && node middleware.mjs
cp middleware.mjs ../host/ && cd ../host && node middleware.mjs
hello from helper
Error [ERR_MODULE_NOT_FOUND]: Cannot find module '.../host/lib/helper.mjs' imported from .../host/middleware.mjs
Same file. Same code. One world has the helper and one doesn't.
3. The eight-minute fix
The fix went in at 9:41 and the site was back by 9:43. The file now imports nothing: the two helpers live inside it. That's uglier, and it's correct.
Then two things so it can't come back quietly:
- A check in the site build fails if anyone adds an import to that file.
- A comment at the very top says the file imports nothing, why, and the date it learned that. Future me will want to "tidy it up" one day. Future me gets a dated note first.
4. The commit that passed because grep found something
The second one is shorter and dumber, which is exactly why I like it.
We ran tests through a wrapper and piped the output into grep to pull out the summary line. Then the commit step checked the exit code and committed on success.
In bash, a pipeline's exit code is the exit code of its last command. The last command was grep. grep found the word it was looking for in the summary, so grep was happy, so the pipeline was happy, so the commit went in. The test runner itself had exited with a failure.
Here's a stand-in runner, made up for the demo, with one failing test, run in bash in an empty temp folder:
run_tests() { echo "41 passed"; echo "1 failed"; return 1; }
run_tests | grep passed
echo "exit: $?"
set -o pipefail
run_tests | grep passed
echo "exit: $?"
41 passed
exit: 0
41 passed
exit: 1
Same runner, same failing test. Without pipefail, the failure is invisible. The rule now: send the runner's output to a file, read the exit code from the runner itself, and commit only on 0.
5. What the two had in common
I spent a while thinking these were two different bugs. They're the same bug.
Both times, the check answered "does this work in the place I'm looking?" and the honest answer was yes.
- In the first, the test ran somewhere kinder than production. Imports resolved there that never would in the real thing.
- In the second, the check looked at the wrong process. It asked grep how the tests went, and grep had no idea.
Neither one is fixed by writing more tests. More tests in the same kind environment would have been green too. More checks on grep's exit code would have been green too. A thousand green checks pointed at the wrong world are still pointed at the wrong world.
6. What we do differently now
- A file only the host runs gets loaded the way the host loads it before it ships. Middleware, an edge or serverless entry, a start script: compile it alone, no bundler, import it in plain Node (or the host's own dev tool). If that fails, the deploy would have failed.
- After every push, the first thing we check is the home page. Not the test report. The actual page, from outside, the way a visitor gets it.
- Exit codes come from the thing that did the work. Never from whatever happened to be last in the pipe.
None of these are clever. They're all ways of asking the same question from where the user stands instead of from where the test stands.
What I'd tell myself before
A green check is a claim about the world it ran in. Before trusting it, ask whether that world is the one your users are in, and whether the thing reporting "success" is the thing that did the work. Most of the time the answer is yes. On Nov 5 it was no, twice.
What's the worst thing that ever shipped behind a green build for you?
I'm Nabil. I build Since, which tracks what your code depends on and tells you before it breaks.
Top comments (0)