The verification step for this project cannot run where the source lives. The toolchain is unusable on that filesystem, so the pipeline mirrors the tree onto a Linux volume and builds there, then runs the control suites against what it built.
That day the log read:
BUILD exit=0 seconds=0
Then thirteen control suites, every one green, and each one run twice with byte-identical output.
Nothing in that paragraph is a lie. All of it is worthless.
rsync preserves timestamps, and that is the whole bug
The mirror step copies the source tree with timestamps preserved, which is the sane default for a mirror. The build tool then does what build tools do: for each artifact it compares the modification time against the inputs, then skips anything whose inputs are not newer.
The inputs arrived carrying their original timestamps. The artifacts in the target directory were from the previous run, and were therefore newer than the freshly copied sources.
$ stat -c '%Y %n' src/lib.rs target/debug/libcore.rlib
1757800000 src/lib.rs # copied today, timestamp from when it was written
1757830000 target/debug/libcore.rlib # built yesterday evening, later
So the build had nothing to do. It did nothing and exited 0 in zero seconds. Correctly. It was asked whether anything was newer and it answered honestly.
Every other check in the pipeline was structurally blind
This is the part worth keeping. The pipeline was not short of checks. It was short of a check that could see this particular thing.
| signal | what it reported | could it see a stale binary |
|---|---|---|
| build exit code | 0 | no, doing nothing is a success |
| the test suites | 13/13 green | no, they ran and passed, on old code |
| run-twice determinism control | byte-identical | no, see below |
| build duration | seconds=0 |
yes, and nothing was reading it |
The determinism control deserves its own sentence. It runs each suite twice and compares the output byte for byte, to catch a check that depends on ordering or a clock or an address. A stale binary is the most deterministic thing in the building. It has not changed and it will not change, so it produces identical output on every run, forever. The control was not weakened by the staleness. It was confirmed by it.
A check that asks "did this produce the same answer twice" cannot ask "is this the right thing to be asking".
The only honest signal was the one nobody read
seconds=0 was in the log the entire time. It was printed and it was committed, and it meant "I compiled nothing", which given that the whole point of the step was to compile the new source is a full description of the failure.
It was not a field anything checked. It was decoration next to exit=0, which is the field everyone reads.
Touching the sources after the mirror makes the number tell the truth:
BUILD exit=0 seconds=12
What the repair actually is
The immediate fix is one line in the mirror script: after copying, set every source file's mtime to now. That restores the invariant the build tool assumes, which is that a file's timestamp reflects when it arrived in this tree.
The real repair is to stop asking a timestamp a question about content. A timestamp is metadata that a copy tool decided to carry over. It is not a statement about what is inside the file. Build systems that hash their inputs do not have this failure mode, and the timestamp shortcut is an optimisation whose failure is silent and green.
Short of adopting one of those, the cheap version is to make the pipeline assert what it assumes:
assert: the build consumed the sources it was given
either seconds > 0
or every artifact mtime > the mirror's own start time
One line, and the class of failure becomes loud instead of quiet.
Two things I keep
A fast incremental build is a claim about timestamps, not about code. The speed you are enjoying is exactly the check you are skipping, and when the timestamps are wrong the speed is total.
Determinism controls are blind to staleness by construction. "Identical twice" is what a frozen artifact does best, so a green determinism check on a stale input is not a weakened signal, it is the signal working perfectly on the wrong subject. Pair every "did it agree with itself" check with one "is this the current thing" check, because neither implies the other.
Top comments (0)