The Test Wasn't Wrong. It Was Answering an Older Question.

I shipped a change I had reviewed, simulated, and deployed. An hour later my acceptance suite was red.

The obvious reading is that the change broke something. I've made that reading correct plenty of times, so I started where anyone starts: at the failure. And the failure was, in a narrow sense, true — an assertion had stopped holding. It just wasn't an assertion about the code as it existed that afternoon. It was an assertion about the code as it existed three weeks earlier, frozen into a file nobody had touched since.

This post is about that gap, because red is not one thing. In my suite — and in most suites I've seen — a red run can mean at least three very different things, and only one of them is "you broke it."

Three kinds of red

Here's the taxonomy I now use before I touch anything:

1. A missing artefact. The suite opens a fixture by name and the file isn't there. The stack trace ends in FileNotFoundError naming the fixture, not a line of product code. This is an environment problem wearing a test failure's clothes.
2. A stale expectation. The test asserts a concrete value or a concrete shape that the product used to produce. The product changed on purpose; the assertion didn't. The test is answering a question about a version of the code that no longer exists.
3. A real regression. Behaviour that should still hold, doesn't.

All three show up as a red X with a line number. The difference lives in what the assertion says and where it dies. So the first thing I do now is not "find the bug" — it's classify. Which of the three am I looking at?

Classifying wrong costs the most when it costs the least to check. A FileNotFoundError takes thirty seconds to confirm. A stale expectation takes reading one file. A real regression takes an afternoon. But if you skip the first two checks because you "know" your change was fine, you'll eventually skip them on the day the third one is true.

The oracle that pinned a number

My suite has a behaviour oracle: a small module that calls the shipped functions and asserts specific outputs. It exists so that a refactor that changes behaviour without changing structure still fails. (The numbers below are stand-ins, not my actual defaults.)

That's a good idea with a sharp edge. The oracle held a literal from the previous version — a default parameter value that fed into a scoring function. When I changed that default deliberately (the old one was wrong for the data I now have), the oracle's assertion failed on correct output.

The failure message was honest: expected 0.20, got 0.35. Nothing in that message says "the expectation is three weeks old." The number just looks wrong, so the reflex is to look at the code that produced the new number.

What makes this class insidious is that the oracle is supposed to be rigid. I don't want a test that shrugs and says "close enough." I want it to pin behaviour so drift is visible. The cost of that rigidity is that every intentional behaviour change requires an explicit rebaseline of the expectation — and if you forget, you get red on green code.

The check that requires exactly one thing to change

The second red flag was structural. One of my integrity checks doesn't assert a value; it asserts a diff. Specifically: exactly one function may differ from a reference snapshot, and a short list of protected functions must be byte-identical.

That is a genuinely good guard against drive-by edits — the thing it catches is real. But it has an implicit assumption baked in: that history stopped at the snapshot. It doesn't know that two documented follow-up changes landed afterwards. Each of those changed additional functions, so the diff is now "one function plus the ones I meant to change," and the check says red.

changed = ['open_trade', 'score_signal', 'load_config']
expected = ['open_trade']
=> FAIL: protected_same=False

Reading that output, "the change broke something" is a perfectly reasonable first hypothesis. It's also wrong. The message is not lying; it's comparing against a museum piece and reporting the difference.

The tell: the failure lists functions you know you changed on purpose, and a protected one that moved as a direct consequence. If every entry in the diff has a commit message, you're not looking at a regression. You're looking at a baseline that stopped being maintained.

The driver that called the old signature

The third one is my favourite, because it points at the wrong file.

The suite's own driver called a function and then indexed the result:

trade = scanner.open_trade(signal, size)
confidence = trade["confidence"]

Then a gate I'd added made that function return None when the signal didn't qualify. Which is the correct behaviour — a gated signal shouldn't produce a trade. But now the driver does None["confidence"], and the traceback says:

TypeError: 'NoneType' object is not subscriptable
  at scanner.py line ...

The traceback names the product file, so that's where people look. The actual defect is in the test: it assumes a call always returns an object. The test was written before the gate existed, and the gate made an impossible path possible.

The general shape: a test that encodes an old contract fails when the contract changes, and blames the callee. If the caller is the test, the fix belongs in the test. If the caller is production code, the fix belongs in production code. The traceback never tells you which — only the call site does.

The dangerous fix

Every one of these three failures has the same tempting remedy: make it green.

For the first two, "make it green" means editing the expectation. Sometimes that's right. But the version of it I've caught myself doing is worse: bending the product back into the shape the test expects. Reverting a correct gate because a caller crashes. Putting the old default back because an oracle asserted it. That's how a suite stops being a safety net and becomes a museum with an admission policy.

A test suite is a model of the behaviour you want. When the model and the code disagree, one of them is stale. Deciding which one is the entire job, and it's a judgement call that shouldn't be made by whoever is closest to the red X.

Rebaselining is a decision, not an edit

When all three flags trace to stale expectations, the correct action is a rebaseline. It's also the action that quietly makes your suite weaker, so I don't do it silently anymore.

The concrete loss in my case: the side-effect audit only inspects functions that appear in the diff. Fold a function into a new baseline and it drops out of the audit — including, if you're careless, a brand-new function that opens files or calls things it shouldn't. The suite goes green by forgetting to look. That's a real trade, and it belongs to the person who owns the risk, not to the agent who wants a clean run.

What I do instead:

• Say what's being weakened. Not "rebaselined, all green" — but "these four assertions now describe the new behaviour, and this function is no longer audited."
• Simulate the whole rebaseline on a copy. A partial simulation is worse than none: it can pass the cheap checks and miss the one that reads the live directory.
• Move every anchor in one edit. A manifest here, a hardcoded digest there, a driver call somewhere else — fix one and the others still fail, and you'll spend an hour convinced the fix didn't work.
• Stop the background writer. My suite has a hermetic check that requires certain files to be byte-identical before and after. A running service writes to them every few seconds, so the check reads zero — which looks exactly like a defect. Pause the writer for the run, then confirm it's back.
• Re-run against the deployed artefact. Green against my working copy proves my copy is self-consistent. Green against the file that actually runs proves something about the machine.

What I check now, in order

1. Read where it died, not just that it died. A file-not-found, a value mismatch, and a TypeError are three different investigations.
2. Ask what the assertion is about. Does it describe behaviour I still want? If yes, the code is suspect. If it describes last month's behaviour, the test is.
3. Check whether the test's caller is the test. Tracebacks blame the callee; contracts live at the call site.
4. Never edit the oracle to get green without saying so. If I change an expectation, I write down what guarantee I just gave up.

Four things I'd tell my past self

1. Red is a category, not a verdict. Missing fixture, stale expectation, real regression. Classify before you debug.
2. A baseline is a snapshot of an argument, not of the truth. It records what you believed when you froze it. If you keep changing the product and never the snapshot, the snapshot becomes an opinion about history.
3. Structural tests have a hidden expiry date. "Exactly one function changed" is true exactly once.
4. The cheapest wrong fix is bending the product to satisfy the test. It works immediately, and it removes the thing that was protecting you.

The failure that taught me this cost an afternoon and produced no bug fix at all — the code was right the whole time. Which is its own kind of lesson: sometimes the most valuable outcome of a red suite is the sentence "nothing is broken, my model is out of date," said out loud, on the record, before anyone touches the code.

---

Curious how other people handle this. Do you treat a stale assertion as a bug in the test, a bug in the process, or just debt? And has anyone found a rebaseline workflow that doesn't quietly shrink what the suite covers? Drop it in the comments.

Part of my Building in Public series — previously: the pipeline that went silent for five hours, the scheduled job that died with 'Operation not permitted', and the exit rule that counted the wrong thing.

Comments

[Abhinav Sharma]

The failure message tells you the two numbers disagree, never which side moved. "expected 0.20, got 0.35" reads identical whether the code regressed or the pin is three weeks old. Do you rebaseline the oracle in the same commit as the behavior change, or does it lag a release?