From September 23 to October 2, a customer kept asking our AI app builder for changes that never held.
They had built a sizeable section of their ...
For further actions, you may consider blocking this person and/or reporting abuse
The passing destructive refine that preserved names but lost bodies makes a valuable permanent regression fixture. I would keep it separate from threshold calibration and add a few legitimate shrinking refactors as controls, so tightening the smoke alarm does not merely fit that one incident.
Your retry change is the stronger invariant: every rejected attempt must recover from the original accepted ancestor, including a failure on the second repair. An end-to-end test that records each retry's parent version and asserts the published version never changes after exhausted retries would protect that behavior even if the disappearance heuristic evolves.
Thanks. I checked both against the test suite, and each one found something.
On the first: our only shrink control empties every function body, keeps every name, and asserts the check lets it through. It was written to prove the rule needs both conditions. It is also the very failure you describe, filed as a pass. Your version is better: real shrinks as controls (dead code removed, a feature dropped on request), and the refine that got through pinned as a known miss, asserting it passes today with a comment saying why. Whoever tightens the rule then sees which side each test lands on. Telling those cases apart will take another signal than names and total size, not a tighter threshold.
On the second: the exhausted-retries test checks the call count, the failed status and that nothing gets published. It checks the base only on the first retry; the second retry's base is never asserted. Your test closes that; I'm adding it.
One nuance on the invariant: only destroyed attempts restart from the accepted version. An attempt rejected for a forgotten rule still retries on its own output, since that's the cheap fix. What keeps it safe is that the anchor never moves: every attempt is measured against the last accepted version, not against the base it was handed, so drift can't compound past the threshold.
That is a fair correction: I overstated the restart invariant. The execution base can remain the rejected output for a rule fix; the comparison anchor must remain the last accepted version. Those are separate properties.
A mixed-failure sequence would make the distinction explicit: first reject an otherwise intact attempt for a forgotten rule, then produce a destructive attempt during that correction. Assert both the base chosen for each retry and the unchanged comparison anchor. That would test the transition between the two recovery paths, rather than only repeated failures of one kind, while keeping legitimate shrink cases separate from the known miss.
Reading the loop again to answer you, I had the base part slightly wrong too. Bases don't chain. The base is chosen once, from the first rejected output: that output itself for a rule fix, the last accepted version if it came back destroyed or missing a field. Every retry starts from that same base; only the error message moves forward.
So in your sequence, the second retry starts again from the intact first attempt, with the destruction message as its only feedback. The forgotten rule is no longer mentioned anywhere. It fails safe (nothing ships unless it passes both the validator and the check), but it can spend the last retry fixing the wrong thing. That is a better reason to write your test than the one I had: assert the base of each retry, the fixed anchor, and that the last feedback still carries every error seen so far. Today that last assertion would fail.
That clarifies the loop: a fixed execution base and a fixed comparison anchor do not imply that feedback preserves all unresolved constraints. The mixed-failure fixture now has a sharper purpose than the restart behavior I initially described.
I would accumulate unresolved errors, then remove an error only after its corresponding check passes on the current candidate. Keeping every historical error forever can request a fix that already succeeded. The last retry should therefore receive the current complete failure set, with the original base and comparison anchor asserted independently.
Both of your tests went in that evening: the known miss pinned next to three legitimate shrinks, and the mixed sequence in both orders. The loop now carries every validation error seen. Pruning is where I'd push back, because the reverse order breaks it.
First attempt destroyed: the base becomes the accepted version, which doesn't have the change yet. Retry 1 keeps the app but forgets a rule. The destruction check passes on that candidate, so your rule drops the destruction error. But retry 2 doesn't start from that candidate, it starts again from the accepted version, and the destruction error was the only place the user's request still appeared. The last retry hears about the rule and nothing else, hands back the old app untouched, and that passes every check: the previous version, published as a success. That's exactly what the second order pins.
Removing an error once it passes on the current candidate is right when bases chain, because the verified fix is in what the next retry edits. With a fixed base, a check passing on a discarded candidate says nothing about what the next retry receives.
You're right that the list isn't all repairs, though. In the first order, the base already has the change, and the destruction error still asks for the app "with only this change applied". For that base it's a constraint, and a literal model could apply the change twice. Sorting errors by whether the base has them, not by whether the last candidate passed, is probably the better version of your idea.
Your last line, about the old model honoring a contract the code never enforced, is what we hit in late September with the same model. We ran GPT-6 Luna on our news enricher's golden set before switching. Every answer passed the format checks. But it read one clause of our scoring rubric literally: no explicit "today" in the article meant the news wasn't new, so real events scored low. Our current model reads the same clause generously. It got caught only because we compare row by row with the current model and count high-scoring rows the new one buries, with a bar of zero. Will the canary be judged on restore rate alone, or will you replay old refines through the new model first?
Both, and the replay isn't new: I run one before every switch. It missed this one for the reason you give: it judged each output on its own, and on its own each one was a valid app. All 111 losses were published, so the pipeline called each one a success. A replay is only as good as its row-by-row comparison. Yours is the score; ours is now the check, and since it sits in the pipeline, every replay goes through it. The count that matters is yours: rows where the current model's edit passed the check and the user kept building on it, and the new model's gets flagged. Your buried rows, with the user's next action as the label.
Your question also made me see that the canary can't be judged on restores alone anymore. Since the fix, a destroyed output is rejected before it ships and never becomes a retry's base; if the retries destroy too, the generation fails. A model as destructive as this one would show up as rejected attempts and failed generations, not as restores. Judged on restores, that canary could look clean. We log every rejection, so that count goes next to the restore rate.
The limit: the check counts what disappears. A model that keeps every function and changes what one of them does passes it, and that's your case. There, the user's next action is still the only label we have. So the canary stays, on a slice that includes big apps with long histories, where this one hid.
With a bar of zero, how do you tell the rows the new model buries from the ones the current model buries on its own when you rerun it?
Rejections next to restores makes sense. With the check in place, a clean restore rate could just mean it's working. On your question: before comparing anything we run the current model twice the same morning. On that set the two runs agreed row for row. So whatever Luna buried wasn't noise from our side, and a bar of zero was fair. Luna wouldn't take temperature 0, which is why it got three runs. Each one buried about the same number of rows.
Running the current model against itself first is what makes a bar of zero honest, and it's an easy step to skip. We don't set a temperature at all: every call, on every model, runs at the provider's default. So on our side a single replay run is a draw, and the current model has to agree with itself before anything Luna buries means something.
One thing I'm curious about in your three runs: were they the same rows each time, or about the same count spread over different articles? Same rows would point at the clause itself. Different rows would mean the literal reading is a coin the model flips per article, and a golden set your size could miss it on a lucky run.
Mostly the same rows. A core survived every prompt version we tried that day, like a trial readout the article itself dated six days earlier and a placement a few weeks old, and a few of those are labels I'd call borderline myself. Around that core, repeat runs moved by about three rows either way. So on our set it was a stable reading of the clause, with some coin flipping at the edges.
"Coin flipping at the edges" is a good name for it. And the core says more than the count: if it survived every prompt version you tried, it isn't the wording, it's how the model reads "new". With a few of those labels borderline even to you, Luna's reading may be the one you'd write down if you had to define it. On those rows a bar of zero measures a change, not an error. Your answer also tells me how to read our next replay. The check is a fixed rule, but at the provider's default temperature the output it judges isn't, so an edit close to the thresholds can land on either side from one run to the next. Edits flagged on every run are our core; edits flagged on only some are edges, and they shouldn't count the same.
The line about the validator only judging the output stuck with me. Counting surviving function names and size is a clever invariant, and it is cheap. In my world that is the same lesson as verifying LLM outputs against something outside the pipeline itself.
Agreed, with a twist: the reference here wasn't outside the pipeline. It was the input, which the pipeline already had and never looked at. The validator read the output; nothing compared it to what came in. The outside part came later, from users: their restores are what told us where to put the thresholds.
The output collapsing from 70,000 characters to 510 after the GPT-6 Luna swap is a brutal way to surface a latent bug. Your point lands: the validator checked the output but never what the edit kept, and I hit that same blind spot grading agent edits, so I now diff each result against its parent. How did you set the preservation threshold without flagging a legitimate rewrite?
Short answer: it still flags some. I chose which mistake to make.
First, the rule needs two signals: a field is flagged only when it keeps less than 30% of its parent's function names and less than 40% of its size. Ordinary edits rarely lose both. A refactor that renames everything keeps its size; a cleanup keeps its names. That's also why we count instead of diffing: a refactor makes a huge diff and loses nothing.
Second, the numbers came from what users did next. Across 5,960 production refines, an edit restored right away counted as rejected, one the user kept building on as accepted. Then we read the candidate pairs against those labels. Moving the size cut from 40% to 50% flagged more accepted edits and not a single extra rejected one. Going to 50/50 caught 2 more rejected edits and cost about 20 more accepted ones per model. So we stopped at 30/40.
What does lose both is a real rewrite, and those still get flagged: in a sample of 40 flagged edits users had kept, about one in four was a rewrite they had asked for. No exception for them, on purpose: the only component that could declare "this one is intentional" is the model we're guarding against. Starting over means clearing the history, so there's no parent to compare with. The failure message doesn't say so yet; that's still open.
Where your diff probably beats our counts is the one miss we know of: names kept, bodies gone.How do you keep it from firing on refactors?