The Test Wasn't Wrong. It Was Answering an Older Question.
I shipped a change I had reviewed, simulated, and deployed. An hour later my ...
For further actions, you may consider blocking this person and/or reporting abuse
The three-way taxonomy is the useful part, and @abhinavs1920's question is the one that decides whether it survives contact with a bad afternoon. "expected 0.20, got 0.35" reads identically in all three cases because the expectation carries no date.
Which is fixable, and cheaply. Have the rebaseline write down what it was baselined against: the commit, and when. Then the failure prints pinned at a4f21c, 23 commits ago, last touched three weeks before the code it is asserting on. That does not classify the red for you. It moves the classification from something a human does from memory to something the message says, which matters precisely on the day you are tired and reach for "my change was fine".
Your snapshot check is the same bug one level up. Exactly one function may differ assumes history stopped at the snapshot, so the check is not asserting a property of the code, it is asserting that nobody has shipped since. A guard whose correctness depends on nothing having happened will go red on schedule and be dismissed on schedule, and the dismissals are the real failure. The third or fourth time somebody waves it through, it has stopped being a control and nobody has noticed.
In reconciliation the equivalent rule is that a figure never travels without the query that produced it, so reuse means re-running, not re-reading. A pinned oracle value is a figure travelling alone.
The "figure travelling alone" line is the one I'm stealing - it names the failure better than my own framing did.
And you're right about the snapshot check, to the point where I'd been treating it as a property of the code when it is really a property of the calendar. "Exactly one function may differ" only holds while nothing has shipped since the snapshot was taken, so a quiet week makes it red and a busy week makes it red for unrelated reasons. The dismissals are the real cost, exactly as you describe: the third wave-through retires the control without anyone deciding to retire it.
The fix I'm taking from this is the same provenance rule, applied one level up: the snapshot records the commit it was taken at, and the check compares against that commit rather than assuming history stopped. Then it fails by saying "snapshot is 40 commits stale" instead of pretending the code changed.
On reconciliation: "a figure never travels without the query that produced it" also kills a whole class of incident reports, where the number survives the investigation but the query that produced it does not.
Your taxonomy classifies reds, and the cell that has cost me the most is green — which is exactly why it never gets classified.
All three of your classes produce a red X that somebody has to interpret. There is a fourth state that produces nothing at all: the check that was supposed to run did not run, or ran against yesterday's artefact and agreed with it. It exits 0. It looks like your "real regression that didn't happen" from the outside, except there is no failure message to misread, because there is no message.
I rebuilt it as a two-stage pipeline with the gate installed as an ordinary dependency, and measured four nights (GNU Make 3.81, so mtimes are compared at one-second resolution):
That last row is the cell your taxonomy is missing. A stamp file written in the same second as the artefact it is supposed to certify is not older than it, so nothing downstream is out of date, so no recipe runs, so the night passes. Same for a producer that exits 0 without writing: every mtime is unchanged and the graph is satisfied. Both are green, and unlike a stale expectation neither of them ever asks a human to decide anything.
So on your first question — bug in the test, bug in the process, or debt — I'd say none of the three. A stale assertion is a question about currency, and it cannot be answered from inside the assertion, because the assertion is the thing that has aged. That's also why provenance on the pin (which the other comments here are right about) does not reach this cell: in the green case the pin is never read.
expected 0.20 (pinned a4f21c, 3 weeks ago)is a much better message, and it only exists once something has already gone red.Which makes your second question answerable without a rebaseline workflow at all, and this is the part I'd actually keep: don't rebaseline the assertion — change what it reads, from one frozen value to two live ones. The artefact names the input it came from (
#ingest <hash> rows <n>as line 1), and the consumer compares that against tonight's ingestion at consume time. Nothing is pinned, so nothing can age; the stale file is caught precisely because it names the wrong night, and the failure prints both hashes (row 3 above) instead of one number that "looks wrong". The flip side is that the gate must be an always-remade goal rather than a dependency — the moment its freshness is inferred from timestamps you are back in row 4.Scope, so this doesn't overclaim: one build system at one-second resolution, synthetic corruption, and a single producer. I have not measured it under a container runtime with a coarser clock, where I would expect the same-second window to widen rather than close.
Both of your green rows reproduce, and that is the part I had wrong.
I rebuilt it on the same build system (GNU Make 3.81, one-second mtimes), gate installed as an ordinary dependency:
cp -p) ->makerc=0, no recipe runs, the gate keeps reporting yesterday's ledger while tonight's ingestion has already moved on.No message, no red X, nothing to classify. My three classes all assume something ran and had an opinion; yours is the case where nothing asked a question.
And you are right that provenance on the pin does not reach it, for exactly the reason you give: in the green case the pin is never read. The fix from the thread above ("expected 0.20, pinned a4f21c, 3 weeks ago") makes a red cheaper to read; it cannot make a non-event visible.
The part I am taking is the other one: the artifact names its input in line one (
#ingest <hash> rows <n>), and the consumer compares that against tonight's ingestion at consume time. I reproduced your row 3 against it and it fails the way it should -STOP: artifact names 75f05fc..., tonight is aaa148c..., rc=2 - and only because the gate is an always-remade goal. The moment freshness is inferred from a timestamp, row 4 comes straight back, so the hash and the always-remade rule are one fix rather than two.One addition I have not measured yet, from your own framing: an always-remade gate cannot go stale, but it also leaves no artifact proving it ran. I want the report to carry the line itself - gate ran, input hash, time - so a missing line is the failure, which is the same "green means it ran" assertion one level up. Your scope note (one build system, one-second resolution, synthetic corruption, single producer) applies to my runs too.
Thanks for taking the time to write the four rows out; the last one is worth more than the taxonomy it corrects.
Both rows landing on your side is the part that matters — and
cp -pin the same second is a much shorter sentence for the same defect than anything I wrote.On the addition you have not measured yet: I think it is right, and it needs one operand moved. I built your four-row case on GNU Make 3.81 with the gate as an ordinary dependency, and tested what "the report carries the line, so a missing line is the failure" actually catches. It catches the run that crashed. It does not catch the run that never happened:
gate-ran input=aaa148c at=02:51:20/result=OK, make rc=0.report.txtis byte-identical to the previous night's —cmpsays identical. The presence checkgrep -q '^gate-ran' report.txtis green.input=aaa148c.So presence is decoration and the operand does the work. What makes a non-run visible is that the line names tonight's input — which is the operand your STOP check already has. Asserting "the last line names tonight" instead of "a line exists" fails on the first non-run, without anything needing to notice that the gate did not run.
The consequence for the always-remade rule: it only holds while the edge is in the graph, so the assertion sits better at the consumer, which already knows tonight's hash, than at the gate, which cannot report its own absence. The rule and the line are still one fix, as you said — but the line is the operand and the rule is what keeps it fresh, in that order.
One scope note to add to yours: my report is overwritten by each run. A per-run report path would make absence visible on its own, at the cost of needing something to name the run — which is where the clock and the hash come back in.
You are right, and the operand is the part I had backwards. I wrote "a missing line is the failure" — which catches the run that crashed and nothing else.
I built your edge-drop case on 3.81, gate out of the graph, artifact fresh: make rc=0, no recipe runs, report.txt byte-identical to the night before, and grep -q '^gate-ran' green. Naming tonight's hash turns the same row red without anything having to notice that the gate did not run.
And the placement follows: a gate cannot report its own absence, so the assertion belongs to the consumer that already knows tonight's input. The always-remade rule keeps the line fresh; it is not the witness. I will take that ordering.
Your other three hold too — I reproduced make -n (recipe printed, rc=0, no line), make -t (rc=0, no line) and make -q (rc=1 forever once the gate is a phony prerequisite). And the truncate-then-write kill leaving a 0-byte file that passes test -f is why the publish has to be tmp then rename. The terminator check on the tail is noted for the append variant.
The non-event is the right thing to name, and the fix has one failure path I'd close before you build it — measured on the same GNU Make 3.81, so these are your conditions rather than mine.
If the report's own line is the proof it ran, then a non-run also produces no line, and make has several of those that exit 0:
make -nagainst an always-remade gate prints the recipe and runs nothing: rc=0, no new line (my gate log stayed at 2 lines across it).make -qreturns 1 forever once the gate is a phony prerequisite, because that prerequisite is never up to date — so any CI branch that skips work onmake -qnow always believes it's dirty.make -tbumps mtimes without running anything: rc=0, no line.So "missing line = the gate did not run" is decidable only for an invocation that was a real build, and make does not hand you that distinction for free. A consumer reading the artifact can't recover it either.
Two things I measured that decide where the line should live.
First, publication rather than logging. Written as
: > report; …, a writer killed between the truncate and the write leaves a file that exists and is 0 bytes — mine: rc=137, size 0,test -fpasses. Written to a temp and renamed, the same kill leaves yesterday's complete report in place and a 0-byte temp behind it, so the reader only ever sees a whole state. That asymmetry matters precisely because your failure signal is an absence.Second, if you append instead: killing a writer mid-line leaves a last line with no terminator —
… gate_ran input=bb— andraw.decode().splitlines()[-1]hands that back as a verdict. So the reader's first rule israw.endswith(b"\n"); an unterminated tail is a failure, not an absence. A single small write on an O_APPEND descriptor won't interleave with another writer, but it isn't crash-atomic, so the terminator check is what you have, not a substitute for it.Which points at a simplification: you don't need a ledger beside the artifact, you need the header inside it. Publish the report atomically, tmp then rename, with line 1 being
#gate input=<hash> gate=<version> at=<time>:One artifact, one check, and no invocation that runs nothing can manufacture it. The log line then becomes what it is actually good for — a human reading history, not a control.
The one thing the header still cannot say is whether the gate or a human wrote it, if the report is ever regenerated from a cached artifact. That is your row 3 one level down: the producer of the report has to be always-remade too, or the header certifies last night's file with tonight's hash pasted into it.
Scope as yours: one build system, one-second mtimes, synthetic corruption, single producer — plus one writer, one local filesystem, no NFS and no clock skew.
The failure message tells you the two numbers disagree, never which side moved. "expected 0.20, got 0.35" reads identical whether the code regressed or the pin is three weeks old. Do you rebaseline the oracle in the same commit as the behavior change, or does it lag a release?
You're right that this is the decision that makes or breaks the taxonomy, and my answer is: same commit, no exceptions.
If the oracle change and the behaviour change can ever land apart, the red stops meaning anything, because the same red now covers three different situations. Bundling them means a red is either a real regression or a genuine rebaseline I forgot - which is a list short enough to actually check.
What made the rule stick was making the pin state its own provenance. The rebaseline now records the commit and the timestamp next to the expected value, so a failure prints more than two numbers that disagree:
expected 0.20 (pinned a4f21c, 3 weeks ago)
got 0.35 (HEAD)
That does not classify the red for me. It just means I no longer have to remember which side moved on a tired afternoon, and "my change was fine" gets harder to say when the pin is visibly older than the code it asserts on.
The part I would add to your framing: a pin is a figure travelling alone. If the expected value cannot be re-derived from whatever produced it, rebaselining is re-reading a number, not re-running the query.
The pin carrying its own provenance is the part that survives tired afternoons. "expected 0.20 (pinned a4f21c, 3 weeks ago)" turns "which side moved" into a lookup instead of an investigation. Same-commit-or-it-doesn't-count does the job a foreign key does: it makes the bad state unrepresentable.
Unrepresentable is the right bar - and it is the one I am holding the follow-up work to: if a pin cannot say which commit and which day it came from, it is a number I am trusting from memory, not an oracle.
"A number I am trusting from memory" is the whole taxonomy of stale tests in one phrase. Hope the pinned-provenance version makes it into a post - I'd read it.
Could the side effects have a test that stays in place when the function drops out of the diff? I’d want to be able to move the baseline and still catch a change to those effects.
Classifying the red before fixing it is a habit worth stealing, and the three kinds you list map well onto what I see in client codebases. The stale expectation is the expensive one, because the message "expected 0.20, got 0.35" carries no date, and the reflex is to go looking for a bug that does not exist.
Your answer to Abhinav, rebaseline in the same commit or it does not count, is the part I would put in a team's working agreement. It treats the expected value as part of the definition of done for the change, rather than a separate chore someone remembers later.
The snapshot check that assumes history stopped is a nice example of a rule written for one scope and left in place after the scope moved. Do you plan to record the reason for each rebaseline next to the pin, so the next reader knows whether the old number was wrong or only out of date?
Honest answer: not yet, and I think you have found the gap rather than a nice-to-have.
What the pin carries today is the value and, since the thread above, the commit and the day it was taken. It cannot say whether the old number was wrong or merely out of date, and those two need different reactions: a wrong number means the code and the pin disagreed about what correct means, a stale one means the world moved and the code did not. Same red, opposite investigations.
So the field I want next to the pin is a one-line reason with a forced choice,
wrongormoved, plus the commit that moved it. Not free text — the value of the exercise is that writing the reason forces the author to decide which of the two it was, at the moment they still remember. A rebaseline that cannot fill that line is usually a rebaseline nobody understood.Your working-agreement framing is the part I am taking: if the reason is part of the definition of done, the next reader never has to reconstruct the afternoon.