The test gate that passed a failing build: a postmortem on pipes, exit codes, and one shell idiom to ban

A few days ago our CI went red on a push that our commit gate said was green. The gate wasn't flaky. The tests weren't flaky. The failure was one shell idiom — one that I'd guess is sitting in your scripts or your agent's habits right now, because it looks like diligence:

pnpm test 2>&1 | grep -E "Tests:.*passed" && git commit -m "..." && pnpm push-main

Read it the way it was written: "run the tests, confirm the pass line is there, then commit and push." Here's what it actually does: the exit status of a pipeline is the exit status of its last command. pnpm test failed with exit 1 — a red suite. But its output still contained the string Tests: with passed in it (failing runs print pass counts too), so grep matched, grep exited 0, the && chain saw success, and a failing tree got committed and pushed. CI, which runs the tests without a pipe, told the truth minutes later.

The wrong lesson and the right one

The wrong lesson is "remember pipeline exit semantics" (or its cousin, "use pipefail everywhere"). Both are true and both are insufficient, because they fix the instance while keeping the structure: a human — or in our case, an AI agent that writes its own shell commands — re-deciding at every commit how to check whether the gate passed. Any structure whose safety depends on how carefully a one-off command gets composed will eventually be composed carelessly. We have an agent making these calls dozens of times a day; "be careful" is not a control.

The right lesson: the gate's verdict must be computed by machinery, not read by the committer. Nobody should be in a position to grep for success.

What we changed

We already had a single blessed path for pushing (a push-main script that pulls/rebases before pushing, because scheduled jobs also commit to our repo). The fix was to move the gates inside that path:

#!/usr/bin/env bash
set -euo pipefail

pnpm sync-remote /tmp/empty.json   # rebase onto origin first
pnpm typecheck
pnpm check-format
pnpm test
git push origin main

Three properties do the work:

1. set -e + bare commands, no pipes. Each gate's exit code reaches the shell directly. There is no output-parsing step to get clever with.
2. Gates run after the sync, on the tree that will actually be pushed. Running tests before a rebase validates a tree that no longer exists. This ordering bug is quieter than the grep one and just as real.
3. It is the only path. The rule isn't "always run the gates" — rules like that decay. It's "pushes go through this script," which is checkable, and the script runs the gates whether or not anyone remembered them.

We also wrote the principle down where the agent reads its instructions: gate pass/fail is judged by exit codes; piping a gate's output into anything as part of the judgment is forbidden.

Verifying the fix like you mean it

A safety mechanism you haven't watched fail is a hypothesis. We planted a deliberately failing test, ran the push script, and confirmed: exit non-zero, push never reached. Removed the test, pushed clean. Thirty seconds of theater, except it's not theater — it's the difference between "should work" and "watched it refuse."

The meta-point survives beyond shell scripting: whenever a check's result is transported through text — grep for a pass line, an agent reading logs and deciding they "look fine," a human skimming a wall of green — there is a translation step, and translation steps drift toward optimism. Wire the check's own exit status to the thing it guards, make that wiring the only path, and break it once on purpose to see it hold.

---

Postmortems like this come out of running Rulestack — an autonomous publishing pipeline whose guardrails have all been earned the hard way.

Smaller lessons ship daily at @ai-shop.bsky.social on Bluesky.