A blind "try again" fixed the bug on the first extra attempt. Telling the model exactly what it got wrong did not help at all.
My local model wrote a next-permutation function that failed 3 of 5 test cases. I showed it the broken code, the exact inputs it got wrong, and what it returned versus what was expected, twice. Both times it sent back the identical broken function, character for character. A separate run where I gave the same model zero information beyond "that didn't work, try again" fixed the bug completely on the first attempt.
I wrote this with AI assistance, and I fact-checked and edited every claim myself.
That result is the opposite of what I expected going in. The obvious assumption behind every auto-retry loop I have built is that showing the model its own failure is strictly better than a blind retry, since more information should never hurt. On this run, with this model, it did not just fail to help. It produced a worse outcome than giving the model nothing at all.
The setup, in one line: llama3.1:8b run locally through Ollama, 12 algorithmic coding tasks, one shared baseline attempt per task, then up to 2 extra attempts under each of 3 conditions (no retry, blind retry, error-aware retry). $0 total cost, no paid API calls. Full raw results, grading harness, and reference implementations: linked at the end.
TL;DR
- 9 of 12 tasks passed the baseline attempt perfectly. No retries fired on those; they are reported but carry no signal.
- Of the 3 tasks that failed the baseline, error-aware retry's final score was worse than or tied with blind retry's final score on 2 of 3.
- On one task, error-aware retry's two follow-up attempts were byte-identical to its own buggy baseline code. Shown the exact failing inputs and outputs, the model echoed the same bug back twice and changed nothing.
- Retry is not monotonic. On a second task, the model's third attempt scored worse than its second attempt, within the same condition.
- Error-aware retry won clearly exactly once, out of three opportunities.
The question this raises, stated as something you could go test yourself: if a cheap retry loop has a fixed attempt budget and no independent verification step, does showing the agent its own error ever reliably beat a blind retry, or is the real value of "self-correction" prompts concentrated in cases the first retry would have fixed anyway?
Method
Each of the 12 tasks got exactly one baseline attempt, generated with llama3.1:8b at Ollama's default temperature of 0.7, zero system prompt, one natural-language prompt describing the function to write. If the baseline passed every test, all three conditions were recorded as that single result and no retries ran, which is what happened on 9 of the 12 tasks. If the baseline failed at least one test, the same baseline branched into three independently tracked conditions:
- No retry: the baseline score stands as final.
- Blind retry: the identical prompt sent again, with no information about what failed, up to 2 extra times, stopping early on a perfect score.
- Error-aware retry: the model's own previous code shown back to it, with an auto-generated list of exactly which inputs failed and what the function returned versus what was expected, up to 2 extra times.
Sharing one baseline across all three conditions, instead of generating three independent first attempts, holds the starting point constant. A difference in final score is then attributable to the retry condition, not to ordinary attempt-to-attempt variance in a fresh first try.
FOLLOWUP_TEMPLATE = """Your previous solution had a bug. Here is your code:
{previous_code}
It failed these cases:
{failure_detail}
Write a corrected version of the function."""
That template, plus the specific failing inputs and actual-versus-expected outputs, is the entire difference between the blind and error-aware conditions. Everything else, including the model, temperature, and task prompt, is identical.
The 12 tasks are classic algorithmic problems: next permutation, Roman numeral conversion, a decode-ways dynamic-programming count, valid parenthesis strings with wildcards, spiral matrix traversal, and others, each with 4 to 6 test inputs chosen to include at least one edge case. An earlier, easier task set of ordinary list and string utilities produced a ceiling effect: 11 of 12 solved perfectly on the first attempt, leaving nothing to analyze. That set was discarded, not reported as a null result, the same lesson I had already documented from an earlier LLM-judge experiment in this series.
Grading pulled the first fenced code block out of each response, ran it in an isolated python3 -I subprocess with a 15-second timeout, and compared the output against a hand-verified reference implementation for each test input. A pass required an exact match or the correct exception type.
Results
3 of 12 tasks failed the baseline and produced real retry data. The other 9 passed baseline perfectly and are not analyzed further; the three conditions are identical by construction when there is nothing to retry.
| Task | Baseline | Blind retry path | Blind final | Error-aware path | Error-aware final |
|---|---|---|---|---|---|
| next_greater_permutation | 2/5 | 2 → 5 | 5/5 | 2 → 2 → 2 | 2/5 |
| valid_parenthesis_with_star | 4/6 | 4 → 3 → 4 | 4/6 | 4 → 4 → 5 | 5/6 |
| decode_ways | 0/6 | 0 → 3 → 2 | 2/6 | 0 → 2 → 2 | 2/6 |
On next_greater_permutation, the baseline bug was a one-line slicing mistake that reversed the wrong part of the list. Blind retry, given nothing but "try again," produced a different implementation on its first extra attempt and fixed the bug completely. Error-aware retry, shown the exact failing cases and the specific wrong output for each, returned the same buggy function on attempt two, then the same buggy function again on attempt three. I diffed the three responses directly: the error-aware attempts are character-for-character identical to the baseline.
On decode_ways, the baseline crashed every single test with an IndexError, an off-by-one after prepending a sentinel character to the input string. Blind retry's second attempt rewrote the whole approach as a memoized recursion and reached 3 of 6, its best result in the sequence, then its third attempt abandoned that approach for a third, more broken one and dropped to 2 of 6. More attempts did not mean more correct, even within a single condition. Error-aware retry patched the crash on its second attempt but shifted the same off-by-one into a silent wrong-answer bug instead of a crash, then made a cosmetic, functionally identical edit on its third attempt. Final score: also 2 of 6, tied with blind retry, for a different reason.
valid_parenthesis_with_star is the one case where error-aware retry won outright: final score 5 of 6 versus blind retry's 4 of 6, the only task in this run where seeing the actual failing test genuinely moved the needle. Blind retry's own path shows a regression below baseline on its second attempt (4 → 3) before recovering to a tie with baseline on its third, never beating it.
Taken together, across the 3 tasks that actually failed, error-aware retry's final scores were worse than or equal to blind retry's final scores on 2 of 3, and better on 1 of 3. Neither strategy dominates the other.
What I take away from this
The failure that stands out is not that error-aware retry sometimes loses. It is that on next_greater_permutation, a loop with a fixed retry budget and no independent check would have shipped the identical broken function twice in a row and called it done, because nothing in the setup checks whether a "corrected" attempt actually changed anything before accepting it. Telling an agent exactly what it got wrong is not the same as the agent using that information. In this run, the model treated the error detail as something to acknowledge rather than something to act on, and produced the same answer back.
The decode_ways result is the other half of the lesson: retry is not a monotonic climb toward correctness even when you do give the model room to try again. A third attempt that abandons a partially-working second attempt for something worse is a realistic failure mode of any "keep trying until the budget runs out" loop, not an edge case.
My own practical change from this: a retry loop I would actually trust needs a cheap verification step after each attempt, comparing the new output against the previous one, not just against a pass/fail threshold, so a loop can tell the difference between "tried something different and it helped," "tried something different and it didn't," and "returned the exact same thing." None of the three conditions here had that check, and it is the first thing I would add before using error-aware retry in anything real.
Scope, stated plainly
This is one model, llama3.1:8b, on one set of 12 tasks, with one specific error-aware-retry prompt template, run once each with no repeated trials per condition. It is not a benchmark of this model's general coding ability and it does not claim to generalize to other models, other task difficulties, or other ways of phrasing a retry prompt. All 12 tasks are reported, including the 9 that produced no signal, and all 3 failing tasks' full attempt sequences are reported, including the 2 of 3 where error-aware retry did not win.
Reproducibility: full tasks, reference solutions, grading harness, raw results, failure writeups, and the chart above, at github.com/priyansh18/medium-articles/tree/d74f0b7/retry-made-it-worse.
If you run a retry loop with a fixed attempt budget and no verification step between attempts, has it ever actually caught a case where the "corrected" output was identical to the broken one, or is that failure mode just invisible until someone goes and diffs the responses?
Top comments (0)