This is a submission for the Kaggle Benchmarking Challenge.
I kept running into the same thing with coding models. I'd ask for one small edit and they'd "help" with a pile of other stuff. Fix the redis tag — sure, but also bump nginx. Change a weekday — and Americanize every British spelling while you're at it. Fix a typo in a comment — and rename the misspelled variable that three callers still depend on.
So I made a benchmark for that specific habit: do the edit I asked for, leave everything else alone.
What I built
22 cases. Each one has a single instruction and something tempting sitting next to it — an unrelated bug, an old image tag, a popular myth, Makefile tabs, mixed quotes, a long file with the change near the bottom. The model only scores if it returns the original text with that one change applied. Exact match.
I tagged failures two ways when I looked at the transcripts:
- miss — didn't do the edit
- overreach — did the edit, then touched something else
And I ran two prompt variants:
- plain — just the instruction
- warned — same instruction, plus "change nothing else, including bugs you notice"
I wanted a straight answer on whether telling the model "don't help" actually works.
Tasks:
Full board: Only What I Asked
Models I ran
I mixed bigger and smaller ones on purpose:
- Claude Opus 5, Sonnet 5, Haiku 4.5
- GPT-6 Astra, GPT-5.5, GPT-5.4 nano, gpt-oss-120b
- Gemini 3.1 Pro Preview, Gemini 3.7 Flash, Gemini 3.8 Flash, Gemma 4 31B
- DeepSeek-R1
Tried Grok 4.5 and Qwen3 Coder 480B too. Grok 404'd on the proxy slug I used, and Qwen kept rate-limiting, so those aren't in the table.
Results (exact / 22)
| Model | Plain | Warned |
|---|---|---|
| Claude Opus 5 / Sonnet 5 / Haiku 4.5 | 22 | 22 |
| GPT-6 Astra / GPT-5.5 | 22 | 22 |
| Gemini 3.1 Pro / 3.7 Flash / 3.8 Flash | 22 | 22 |
| Gemma 4 31B | 22 | 22 |
| gpt-oss-120b | 21 | 19 |
| GPT-5.4 nano | 20 | 20 |
| DeepSeek-R1 | 4 | 16 |
Most of these just nailed it. The interesting ones are the three that didn't.
What went wrong
Nano "fixes" the identifier you told it not to touch
One case: fix recieved in the comment only. Don't change any code.
The source has the typo in the comment and in recieved_count. Nano fixed both, on plain and warned. The warning didn't matter. Same story on scope_second_paragraph, where the misspelling shows up three times and only the middle one should change.
DeepSeek drafts the answer mid-thought
DeepSeek's plain score looked awful (4/22) until I opened the transcripts. A lot of the final answers were fine. The issue is it dumps scratch <result> blocks inside its thinking, and both my checker and the default Kaggle extraction grab the first one. One CSV case had 31 of those tags.
If you score the last tagged block instead, DeepSeek jumps to something like 16/22 plain and 19/22 warned. The warning still helps, but the extraction format mattered more than I expected.
The warning sometimes made things worse
gpt-oss went 21 → 19 with the warning. On one prose case it wrapped the answer in an extra <text> shell and broke the closing tag. Longer instructions aren't free.
Takeaways
Helpfulness and surgical instruction-following pull against each other. A model can look great on general instruction benches and still fail a one-line edit because the prior is "improve the file."
"Just tell it not to" is not a reliable fix. It helped DeepSeek a lot, did nothing for nano on the comment-only case, and slightly hurt gpt-oss.
Also: if your grader takes the first fenced/tagged block, reasoning models will look artificially bad. Ask me how I found that out.
Next time I'd probably score diffs instead of full-file rewrites, and maybe try multi-file patches. Full-file exact match was already too easy for most of the frontier models here.
Top comments (0)