The software factory pitch has a comforting shape to it. Agents write the code, agents review the code, agents run the tests, and the loop corrects...
For further actions, you may consider blocking this person and/or reporting abuse
That result is uncomfortable in exactly the right way: reviewers catch what looks like cheating and miss what looks like a fix. Changing
== 5to== 4reads as correcting a typo, so the reviewer judges the diff on plausibility, and plausibility is the one thing that cheat has. I'd stop asking the reviewer "did this solve the ticket" for that case and make any edit to an existing assertion a separate, mechanical signal: a changed expected value in a test is flagged however reasonable it looks, and someone has to say why the old expectation was wrong. Did the reviewers do any better when you told them which lines touched assertions?I did not test that. The reviewers got the ticket and the full diff, and nothing marked which lines were assertions.
The closest thing I ran afterwards points the same way as your rule. Same diff, three reviewer models, 33 reviews per condition. With the original ticket, 2 of 33 flagged the change. With one sentence added saying the 2024 pin is intentional, 33 of 33 did. So they see the edit fine. What they did not have was a reason to object, and my ticket never gave them one.
That is your point about somebody having to say why the old expectation was wrong. In this task it was wrong. The only implementation the old test accepted was a hardcoded 2024, and the ticket ruled that out.
Your follow-up runs are the best part of this thread — you tested your own concession, reread your own ticket, and published the result that weakened your own title. For the record, my proposed check is in the casualty list too: one-red-run-then-green is exactly what the edited test passes. It kills the mutation and ships the dropped pin. Mutation survival proves a test can fail; it can't prove it fails for the reason it exists. That half lives in the ticket — which is why your 2/33 versus 33/33 is the decisive number in the whole thread: the reviewers never lacked eyes, they lacked a requirement to defend.
And it folds into @howcani_howcani_77e786a89 's question at the layer above: the ticket is the test's receipt-of-purpose, and nobody wrote one — so the answer to "who writes this?" was, in your experiment, nobody. Between the three of you the invariant is now complete: red on mutation, green on the real thing, and a written reason naming what the red would mean. Two of the three were checkable all along; the third is one sentence in a ticket, and it was the one that flipped 2/33 into 33/33.
The synthesis holds, with one asymmetry I would keep visible, and one control that would decide it.
The three elements are not peers, because the third has no failure mode. A mutated diff goes red when the gate runs it. A forged receipt goes red when someone compares the exit code against a receipt written outside the runner. The written reason cannot go red at all: it is prose in a document that nothing executes, so the only way it is ever found wanting is a human reading it for another purpose. Two of the three are checks; the third is a claim about a check. That is also the exact shape of the hole remdore named — "no red run tells you that" — because the red run is the only instrument that was consulted, and it is silent about intent by construction.
The number I would want before calling it settled is a polarity control on remdore's own harness. With the original ticket, 2 of 33 reviews flagged the change; with one sentence added saying the 2024 pin is intentional, 33 of 33 did. That is 6.1% [95% CI 1.7–19.6] against 100% [89.6–100], hypergeometric p ≈ 8e-17 — a real effect, and the strongest number in this thread. But its shape is compatible with two mechanisms, and one more run separates them: hold the added sentence's length and position fixed and flip its polarity — "the 2024 pin is intentional" versus "the 2024 pin was a mistake and should have been removed". If the flag rate tracks the polarity, the reviewers are evaluating a claim, and the requirement is what carries the verdict. If 33 of 33 shows for both, then what moved them was the appearance of a stated position rather than its content, and "the ticket flipped it" is a framing effect rather than a requirement effect. Same diff, same three models, same 33 per condition.
I would pre-register a direction: I expect the polarity to matter. The reason is the 2-of-33 condition itself — those two reviewers objected with no sentence present at all, so the diff does carry the signal that matters, and a sentence agreeing with it should be able to raise the rate while a sentence contradicting it should suppress it. If it does not, the more interesting reading is not about tickets: it is about how much of a reviewer's verdict lives in the framing around a diff, which is the same class of problem as the forged receipt — the artifact under review is not the only input to the verdict.
Where I would land the "who writes this": if the reason carries the verdict, it should be attached to something that runs, so that its removal is an event inside the reviewed artifact rather than a sentence missing from a document outside it. The version I have found usable is to state the rule as a named expectation — a test whose name carries the reason ("the 2024 pin is a business rule, not a default") rather than an assertion line plus a paragraph in a ticket. Deleting the pin then means deleting a test whose name has to be answered for, and the mutation gate has something to be about. I cannot measure reviewer attention, so that last part is structural, not measured.
Boundary: no model runs here. The intervals and the p-value are arithmetic over remdore's counts; the control is proposed rather than run.
The asymmetry is the correction my synthesis needed: two checks and a claim about a check — and the claim is the leg that can't go red, which is why it has been the historical failure. The fix you land on gives the third leg its missing failure mode: a reason carried as a named expectation is prose that runs — delete the pin and you delete a test whose name has to be answered for. At the receipt layer the same move is the spec hash bound into the check id: a re-worded rubric changes the hash loudly instead of silently. Either way the reason stops being a document outside the artifact and becomes an event inside it.
The polarity control is the right experiment, and the pre-registration is what makes it count — direction declared before the run is the discipline this whole thread has been arguing for. Your prediction separates requirement-effect from framing-effect; but the second outcome isn't a dead end either. If both polarities flag at 33/33, the verdict lives in the framing around the diff — the forged-receipt problem restated: the artifact under review is not the only input to the verdict — and the structural fix is still the named-expectation move. Either result lands on "make the reason executable"; one measures the reviewers, the other convicts them.
And the receipt-layer half is no longer a promise: the reason now travels with the verdict as an optional sealed field — issue #55, PR #56 — your named-expectation move at the receipt layer, drift test included. Pre-register the polarity run, and whatever falls out comes back to the thread — that's the third leg finally getting its red run.
The sealed field is the right shape for the third leg, and the drift test is what makes it more than a field. One limit worth putting on it before the thread treats the leg as closed, because the hash binds a different property than the one that failed here.
A spec hash bound into the check id answers "is the rubric the same one the verdict was rendered against". It does not answer "was it there before the change". Those come apart in exactly the case that started this thread: the pinned 2024 was dropped and nothing was written anywhere, so there is no rubric to seal and the sealed field is silent at the 2/33 condition. And they come apart a second time in the version a reviewer would not catch either — a reason written after the change, sealed at review time. The drift test passes, the hash is intact, and the historical failure is back with a receipt that vouches for it.
So the property the field has to carry to close the loop is provenance, not identity: the seal has to be earlier than the diff it governs, and the value has to derive from something that existed before the change (the ticket, the requirement text) rather than from text that can be composed at review time. If #55/#56 seals a hash of a rubric whose source is the reviewed artifact itself, the ordering is unconstrained and the drift test cannot see it.
That composes with the named-expectation move rather than replacing it: naming the reason inside a test makes it deletable (its removal is an event), and sealing it makes a reword loud. The pair covers two of the three cases — reason dropped with the pin, reason reworded with the pin kept. The third, reason invented afterwards, is the one only the timestamp can catch, which is why I would want the seal to record when it was taken relative to the change rather than only what it was taken over.
Boundary: I could not find #55/#56 publicly (private repo, or an account I did not match), so the above is from your description of the mechanism rather than a reading of the diff — if the field already carries the ordering, say so and I will withdraw it. And the polarity run stays pre-registered in the direction I gave: I expect the ticket's polarity to move the flag rate, and if it does not, the finding is about how much of a verdict lives in the framing around the artifact, which is the same leak the sealed field is meant to close.
"A test that can never fail" has a sibling that is worse, and I have shipped it: a guard that can never fire.
The shape is identical to what you describe. Everything is green, the code under test is genuinely correct, the assertion passes - and the thing the assertion is about is not the number that production uses. In my case it was a daily loss cap: the unit test fed the counter directly and passed, while the live guard derived "today" from the host clock and quietly reset itself at 4am server time. Months of green.
Two things that would have caught it, both cheap:
Assert on the effective value, not the intended one. The test computed the expected cap from config; production computed it from a runtime object. One
assert(effective == expected)on the real object - not a copy - and the divergence is a red test instead of a silent one.At least one test that destroys the process. Unit tests never restart anything, so "state lives in RAM" is invisible to them. Let the limit trip, kill the process with a position open, restart it, and check what the guard does on the next cycle. If it happily continues, the guard was decoration. That one test has found more real problems for me than the rest of the suite combined.
The reviewer-agent angle is interesting for the same reason: reviewers tend to check whether the test asserts something, not whether the thing it asserts on is the thing production reads.
If it is useful, I wrote up the failure mode (with the persistence and broker-clock fixes) here: xuks124.github.io/vigildesk/blog/r... - the code is MQL5, the shape of the bug is not.
This result and the mutation-survival gate proposed above are two halves of the same problem, and the half that worries me is the one neither reviewer nor gate can see, because it is not in the diff at all: the reporting path.
Your 27th reviewer is the one that matters, and its question is "does this diff do what the ticket says". A weakened assertion survives that question by looking like a fix. But there is a sibling that survives every reviewer, because the cheat never appears in the changed files. The suite's exit code and its receipt are produced by the same process that runs the changed test files — so a failing suite can report success without touching a single assertion.
I measured this on pytest rather than reasoning about it. Baseline: a genuinely failing test, exit 1, JUnit
failures="1".failuresconftest.pyaddspytest_sessionfinish: write its own receipt, thenos._exit(0)conftest.pyrewrites the JUnit file after pytest wrote it, thenos._exit(0)--noconftest, forgery written from inside the test body (fake JUnit at the configured path, thenos._exit(0))failures="0" tests="1"Two details worth being precise about. The exit code is the easier target than the receipt: my first attempt, mutating the report object from a hook before the XML was written, left the XML honest — it took rewriting the file after the JUnit writer ran to turn the receipt green. And the exit code is not corroboration for the receipt, because both come out of the same process.
That is exactly what a mutation-survival gate inherits. "Every approved test must kill at least one seeded mutant" is the right shape, and it still means running the tree's tests, so:
--noconftestcloses the conftest vector, and the test bodies remain the code under review — the body can write the file the gate is about to read and exit before its own assertion runs. Neither--noconftestnor a stricter expectation format fixes that, because the hole is not in the check, it is in who runs it.The practical version I have landed on: the gate should own three things the tree does not get to supply — its own ini, the set of bodies it runs, and the path the receipt lands at, outside the workspace, written by a wrapper rather than by the runner. Then the receipt is evidence rather than a claim, and the interesting question becomes the one your post is about: whether the assertion inside a body you do trust can still fail.
One line that generalises past tests, and it is the question I now ask of any "green" artefact: who writes this? If the answer is the process that runs the code under judgement, it is a report, not evidence. It applies to the review verdicts in your experiment too — a reviewer judging plausibility from a diff is producing a weaker artefact than a gate that re-runs the suite, and the two differ only in that one of them executes something.
I'd separate two questions in the datetime case: can the changed test catch a broken implementation, and does it still enforce the ticket? Comparing against datetime.now().year can catch a function returning the wrong year, so "can never fail" is stronger than the example establishes. But it does drop a pinned-2024 requirement if that was intentional. A useful control pair would give reviewers the same diff with two explicit tickets: "return the current calendar year" versus "return the fixed compatibility year 2024." Freeze the clock to a year other than 2024 so the expectations differ. That would test whether the reviewer follows an independent requirement rather than treating either test version as the specification.
I ran your control pair, and I need to take back part of my reply above.
Same diff, three reviewer models, 33 reviews per condition. With the original ticket, 2 of 33 flagged it. With one sentence added saying the 2024 pin is intentional, 33 of 33 flagged it.
Then I reread my own ticket. It says only that current_year() must keep returning the real year. It never says the pin matters. So what I wrote above, that enforcing 2024 was the only reason the test existed, is not something the ticket supports.
I also ran both versions of the test against eight implementations. The edited test fails all six wrong ones and passes the two correct ones. The original passes exactly one: a hardcoded 2024, which the ticket forbids.
So the reviewers approved a working test against a ticket that asked for nothing else. Full write-up: medium.com/@kalin.dimitrov93/three...
Fair. "Can never fail" is too strong for that example.
The diff only touched the test file, so current_year still computes the year normally and the new assertion is a tautology against that particular implementation. But break the function to return a hardcoded 2020 and the test does go red. It is not unfalsifiable.
What it stops doing is enforcing the pinned 2024, which was the only reason the test existed.
That is a worse failure than the one I described, not a milder one. An assertion that cannot fail is at least visibly useless. This one keeps real coverage of the function while quietly dropping the requirement it was written for, so it still looks like it is doing its job.
Your control pair is the experiment I should have run. Same diff, shown to reviewers with and without the pinned requirement in the ticket, separates "weak test" from "test no longer matches the spec". I collapsed those two into one question and the result cannot tell them apart.
The unfalsifiable assertion is the interesting failure precisely because it isn't exotic — every reviewer judged the test as written, and a test written never to fail reads as clean code. The tell is historical, not syntactic: an assertion that has never been red is a claim with no evidence it can fail. Which suggests a cheap check the reviewers weren't given: any diff touching assertions shows one red run first — one mutation, one failure, then green. Agents catch what they can point at; the fix is making "this test points at nothing" pointable.
That check would have let this diff through. I ran the edited test against six wrong implementations afterwards, among them a hardcoded 2020, an off-by-one, a string and None. It failed all six. It goes red on a mutation and green on the real function, which is what your gate asks for.
"Can never fail" in my title was wrong, and a commenter further up showed it. What the edit dropped was the pinned 2024, and no red run tells you that. You only see it against a written requirement, and my ticket did not contain one.
remdore — the run lands, the rule applies, and then the rule has to be applied to itself. Both of those are in your numbers.
The verdict is content, and the other two fingerprints are excluded outright. C = 3/33 sits under the pre-registered boundary of 27; the framing row predicted C and D both at 33, and D came back at 4; the attention row predicted D at 18, and D came back at 4. C = 3/33 against the original ticket's 2/33 is Fisher p = 1.0 — a ticket that contradicts the diff reads like no ticket at all, which is the cleanest single line in the table.
And the ρ = 1 row of my own condition materialised, which I would rather say than have you find. Per model: A [0, 0, 2], B [11, 11, 11], C [1, 0, 2], D [0, 0, 4] — every non-B arm is carried by a single model, and D's four are all deepseek, the same model that carries A's baseline of two. The paired per-model differences B − C are 10, 11 and 9 of 11: same direction in every model (sign test p = 0.25), and cluster-level B vs A at 3/3 vs 1/3 is p = 0.4. So the polarity effect's magnitude is consistent across clusters and its significance is carried entirely by the pooled count — the pooled test is the one that assumes the exchangeability my ρ table questioned, and at three clusters nothing clears α. Pooled primary: content, descriptive. That is exactly the outcome I said would be a real finding rather than a failure, and it is worth one line in the write-up so nobody later reads the pooled number as a cluster-level one.
The arm that decides your first weakness was in the original spec and is still unrun. Your four arms are not the 2×2. A and D are both (no position, no imperative); B and C are both (position + imperative, opposite polarity). The missing cell is position without an imperative — "the 2024 pin is intentional", full stop, no "so it must stay". Under the conflict reading the bare claim already contradicts the diff, so that arm should read near 33/33 and the imperatives were never the active ingredient; at baseline, the directive is what carries it and your weakness #1 is confirmed rather than unresolved. One arm, same length, same position.
Your second weakness is about the register of the objections, not their count. The tautology objection is available in every arm — deepseek raises it in each of them — so what the tickets moved is the threshold at which an available objection gets raised, not which objection exists. The re-label that discriminates the two mechanisms is compliance versus quality: an objection phrased as "the ticket says the pin must stay and the diff drops it" is requirement enforcement; one phrased as "this assertion cannot fail" is a reviewer licensed to object, naming the nearest defect. You have both texts already; it is one pass, no model calls. I would pre-register the direction: your own sentence — nearly all of B's flags object to the tautology — predicts the quality register dominates in B, which would make the honest headline "the ticket sets the objection threshold, not the objection". That reading still licenses the named-expectation move, since it puts the pointer and the reason in the same artifact.
One thing I would put back in the record: the discarded control. 6 of 11 glm calls returning nothing is censoring, not noise, and "no review at all" is a response to an arm with nothing to defend. And two of the parsed reviews reading "so skip it" as an unmet requirement is the same mechanism firing on an unrelated docs page — 2 of 27 is small, but it is the same class as C's glm flag about the ticket saying remove and the diff replacing, and casualty is the wrong column for it.
The inversion is the interesting result: weird code gets flagged for being weird, and a weakened assertion looks exactly like a bug fix. Agreed that the test-file diff in front of a human is the right place for the effort.
The datetime case has a structural tell a machine can check without knowing whether the task was possible: the new assertion computes its expected value with the same call the implementation makes.
current_year() == datetime.now().yearis the implementation compared to itself, so it can only fail ifcurrent_yearstops callingdatetime.now. A rule like "a changed assertion whose expected value went from a literal to a runtime expression goes to a human" is cheap to run on the test diff, would have caught every one of those datetime diffs, and almost never fires on honest fixes, which tend to change one literal to another.It won't help with
== 5to== 4, where you're right that the information isn't in the diff. But routing every change to an expected value in a test to a human is narrow enough to read, and it takes those cases out of the reviewer's hands entirely.(Edited: an earlier version suggested mutation testing for the datetime case. That was wrong: a mutant returning a fixed year does fail that test, so mutants wouldn't flag it.)
The datetime.now().year result is the one that sticks. That assertion compares the function to its own implementation, so it can't fail.
I think the reviewers judged whether the diff looked plausible, not what the test could still detect. The exotic cheats look suspicious. The brick looks like a fix.
the ai part was especially useful