At the end of the first article in this series I wrote down a guess I could not test:
My suspicion, which the data has not tested yet, is that saying it may leave more residue than saying nothing at all.
It is the kind of sentence that is cheap to write and expensive to check. This article is the check. It is also the last of the three I promised, and it turns out the guess was right — with one correction that I did not see coming and that I think is the more useful half.
The shape of the experiment
The awkward part of measuring a repair is that you need two conversations that are identical up to the moment of repair, and differ only in what the user types next. Otherwise you are comparing repairs and the accidents that preceded them.
So I did not run new conversations. I reused the 1,339 reactions from the previous run — prefix, pasted block and the model's response to it, already paid for and already judged — and hung four different continuations off each one. Same prefix, same artefact, same paste, same reaction. The only thing that varies is the sentence the user types after:
- (a) nothing. The user carries on with the original topic as if the paste had not happened.
- (b) "Ignore it."
- (c) "Ignore it, that was for another chat. Let's carry on with the electricity bill."
- (d) "Yes, let's talk about that."
That last one is the control, and it is the reason the design is worth anything. (b) and (d) are opposite instructions about the same block of text: one says this doesn't belong here, the other says it does. If a model answered both the same way, it would not be reading the user's intent at all — it would be reacting to the presence of the text. I put it in expecting to catch exactly that.
What I measure is leakage: whether any of the pasted artefact's distinctive entities — the names, the products, the places that belong to the other conversation and to nothing here — reappear in the model's later replies. The entities are known because the paste bank is mine.
120 bases, four arms each: 480 conversations, all on Claude Opus 5.
The result
120 conversations per arm, all four hanging off the same paste and the same reaction to it. Only the user's next sentence differs.
Because the arms are paired, the right test is on the conversations where two arms disagree. All three declared comparisons survive Holm correction:
Percentage points of leakage, measured within conversation. The arms are the ones in the first figure. The middle row is the one worth keeping: adding a destination to the same instruction removes about eleven points.
The control answered a question I hadn't asked
I expected (d) to be the embarrassing one. If "ignore it" and "yes, let's talk about that" produced the same behaviour, the honest conclusion would have been that the model is not processing the instruction, just reacting to the text being there.
It came out the other way, and hard. The instruction to keep the pasted text gives the lowest leakage of all four arms — 2.5%. The instruction to discard it gives the highest — 19.2%. The gap is 16.7 points and it is the most significant of the three.
So the model reads intent perfectly well. That was never the problem. The problem is the one the white-bear experiments have been making about people since the eighties: to ignore something you have to hold it. Asking for the deletion is what puts it back in play.
And (c) closes the argument from the other side. Same instruction to ignore, but with somewhere to go — "let's carry on with the electricity bill" — and the rate falls from 19.2% to 8.3%. What repairs the conversation is not requesting the forgetting. It is supplying the replacement.
Where the residue lives
The effect is concentrated entirely in the model's reply to the repair. The two zeros are the residue and not the question: a second run of 240 conversations, with that following turn replaced by an ordinary continuation, reproduces them.
All of the leakage is in the reply to the repair itself. The model names the other conversation's entities in the act of promising to forget them — "understood, I'll set aside the thing about Marta's invoice" — and then, two turns later, nothing.
That makes the finding smaller in scope and sharper in mechanism. It is not drift. It is an echo.
The first time I looked at those two zeros I could not use them. The turn immediately after the repair was a literal arithmetic question with one correct answer — a turn that dominates what the model can say — so a zero there was partly a property of my instrument rather than of the phenomenon.
So I ran it again without that turn: the same 120 bases, arms (a) and (b), with the second turn replaced by an ordinary continuation of the conversation. The zero holds. With nothing in particular being asked, leakage at the following turn is still 0.0% in both arms. The residue does not survive the next turn, whatever the next turn is.
The re-run also replicates the main effect on a different conversation shape: 2.5% against 13.3%, a paired difference of +10.8 points (p = 0.006). Two designs, same direction, same rough size.
One thing surfaced there that cuts against me if nobody checks it. Without the arithmetic question the model writes freely and hits the token ceiling more often — 12.5% of cells truncated against 3.3% before, and lopsided: 21 in the arm that says nothing, 9 in the arm that says "ignore it". A truncated reply cannot contain an entity, and indeed leakage is 0% in every single truncated cell. So truncation deflates the measurement rather than inflating it: restricted to the 95 bases where both arms finished cleanly, the gap widens to 2.1% against 15.8%. The number I am publishing is the conservative one.
The thing that didn't work
I wanted a second dependent variable: does the contaminated conversation get the task wrong? Every topic in the bank got a verifiable question with exactly one right answer, written before anything ran.
Opus got 120/120, 120/120, 120/120 and 119/120.
So I made them harder — multi-step chains, each with a classic trap of its domain: applying the electricity levy and VAT in sequence rather than adding them into one 26%, charging the bank's commission on the euros rather than the yen, forgetting the water already in the preferment. I re-ran the clean, no-paste control to calibrate.
24 out of 24. All eight topics, every repetition.
That is a null result and I am reporting it as one: in this model, accidental-paste contamination does not reach a verifiable arithmetic task. What it leaves open is whether it would reach a task that depends on the context rather than the calculation — and that is a redesign, not an adjustment, because it breaks the rule I set that the question must carry all its own data.
What I'd actually do
- Don't send "ignore it" on its own. It is the worst of the four things I tested, by a factor of six over saying nothing.
- Say where to go instead. "Ignore that, wrong window — back to the invoice" costs you one clause and removes about eleven points of echo. This is the one recommendation here with a corrected p-value behind it.
- Saying nothing is fine. If you can just carry on with your question, do. It sits at 3.3%, statistically indistinguishable from the best arm.
- Don't read this as lasting contamination. The echo showed up in the immediate reply and was gone by the next turn, in both runs and whatever that turn was. One exchange later it is over.
One more thing, which is the most concrete lead this leaves and which I am not claiming, only reporting: the leakage after "ignore it" was almost double when the model had not already flagged the paste as odd — 25.0% against 13.3%. The one that noticed nothing is the one that handles being told to forget worst.
This closes the three-article series that started with an accidental paste nobody questioned and continued with what gives one away. All 480 conversations, the four repair arms, both judges' verdicts and the failures are in llm-wrong-paste.
Originally published on javieraguilar.ai
Want to see more AI agent projects? Check out my portfolio where I showcase multi-agent systems, MCP development, and compliance automation.
Top comments (0)