TL;DR: I ran a small replication of the paper "Prompt Repetition Improves Non-Reasoning LLMs": 100 MMLU questions, one non-reasoning model, prompt sent once versus twice. Baseline 59%, repetition 61%. Probably not statistically significant, and that turned out to be the interesting part. The gap between the paper's headline numbers and my +2% taught me more about transformer attention than the trick itself.
Why I tried this
The claim sounds like a joke: paste your prompt twice and a non-reasoning LLM gets better. No chain-of-thought, no fine-tuning, no extra instructions. The paper, Prompt Repetition Improves Non-Reasoning LLMs from Google Research, tested Gemini, GPT, Claude, and Deepseek models across seven benchmarks and found repetition won 47 of 70 model-benchmark combinations with zero losses, including improvements as large as 21% to 97% on long-context tasks like NameIndex, where the model has to pull one item out of a long list.
A trick that cheap deserves a replication, and I wanted to know two things. Does it show up on ordinary short questions? And if it does, why does duplicating text change anything at all inside a transformer?
The setup
I kept it deliberately small and fixed everything I could:
-
Dataset: 100 multiple-choice questions sampled from MMLU science and math subjects (high school and college math, physics, chemistry, biology, abstract algebra, astronomy). Four options, one correct answer. The dataset was built once with a fixed seed and frozen to
questions.jsonso both conditions saw identical questions. -
Model:
llama-3.1-8b-instantvia the Groq API. A small, fast, genuinely non-reasoning model, which is the population the paper is about. - Conditions: the baseline sends the question once with "Answer with one letter." The repetition condition sends the exact same block twice (n = 2), nothing else changed.
# baseline # repetition (n = 2)
<Question> <Question>
A. ... A. ...
B. ... B. ...
C. ... C. ...
D. ... D. ...
Answer with one letter Answer with one letter
<Question>
A. ...
...
Answer with one letter
Results
| Method | Correct | Accuracy |
|---|---|---|
| Baseline (n = 1) | 59 / 100 | 59% |
| Repeated prompt (n = 2) | 61 / 100 | 61% |
Two more questions correct. Before calling that a win, the honest check: for paired correct/incorrect outcomes on the same questions, the right test is McNemar's, which looks only at the questions where the two conditions disagree. With 100 questions and a swing this small, the improvement is likely not statistically significant. I am not claiming the effect; I am reporting what 100 questions showed.
So the replication "worked" in the least satisfying way possible: a positive direction, too small to trust on its own. The useful part was figuring out why the effect should be small here when the paper's numbers are so large.
Why my effect was small and the paper's was big
The paper's dramatic gains come from long-context retrieval tasks. Give a model a list of 50 names and ask for the 25th, and attention has to hold position across hundreds of tokens. Attention weight spreads thin across a long sequence, the signal for any single token gets diluted, and the model misses. Repeating the prompt puts a second copy of every important token in the context, doubling the chances that the answer-relevant tokens get attended to. On tasks that stress attention, that rescue is worth a lot.
My questions were short. A four-option MCQ barely stresses attention at all; the model can already see everything clearly. Repetition amplifies a signal that was never weak, so there is not much headroom to gain. The effect size is not a property of the trick, it is a property of how starved the task is for attention. That is the actual takeaway, and it is also a testable prediction: rerun this on long-context questions and the gap should open up.
Why repetition changes anything at all
This is the part that got me to actually read about attention instead of nodding along to diagrams.
When the model predicts its answer token, attention is distributed across every prompt token. A key concept might get a weight of, say, 0.07. With the prompt repeated, each copy might get 0.05 and 0.04, but the combined signal is stronger than before. Nothing about the model changed; the input just gave the mechanism two chances to find the same information.
The KV cache view says the same thing: during inference every prompt token's key and value vectors sit in the cache, and repetition puts two copies of the relevant vectors in there. Retrieval by attention becomes more reliable when the target exists twice. Prompt repetition is signal amplification you perform from outside the model, which is also a decent one-line explanation of why prompt engineering works at all.
The paper itself adds a framing I had not considered: causal masking. In a causal LLM, tokens can only attend backwards, so a token early in your question never gets to see the tokens that come after it. In the second copy of the prompt, every token effectively has the full query behind it. Repetition is the cheapest possible workaround for one-directional attention.
Limitations, plainly
- 100 questions is small, and the headline result is within noise.
- Short MCQs only, which the previous section argues is the worst case for this trick.
- One model, one repetition factor (n = 2).
What I would run next
The obvious follow-ups, roughly in order of what I am most curious about: long-context tasks like the paper's NameIndex setup where the prediction says the effect should be large, repetition factors n = 3 to 5 to find where returns diminish, and a comparison against chain-of-thought on the same frozen dataset, since both tricks spend extra tokens to buy accuracy.
Repo with the dataset builder, notebook, and raw per-question CSVs:
CoffeeAurCode
/
Prompt-Repetition-Improves-Non-Reasoning-LLMs
I tried replicating the paper “Prompt Repetition Improves Non-Reasoning LLMs.”
Prompt Repetition Improves Non-Reasoning LLMs
A Small Replication Study
This repository contains a small experiment replicating the core idea from the paper:
Prompt Repetition Improves Non-Reasoning LLMs
The goal of this experiment was to understand whether repeating a prompt improves accuracy on a custom multiple-choice dataset and to explore why such a simple trick can affect transformer behavior.
Rather than focusing on achieving large gains, the purpose of this project was to investigate the mechanics of prompt repetition and understand what it reveals about transformer attention.
Motivation
Recent work has shown that repeating a prompt multiple times can significantly improve the performance of non-reasoning LLMs on certain tasks.
Example reported improvements from the paper: 21% -> 97% accuracy
on tasks requiring long context reasoning such as NameIndex.
This raises interesting questions:
- Why does prompt repetition help?
- Does it work on simple MCQ tasks?
- What does this reveal about…
If you have run prompt repetition on anything with real context length, I would like to know whether the effect showed up for you, and how large it was. And if you see a flaw in the setup that could explain the +2% besides the trick itself, that is exactly the kind of comment I am hoping for.
Top comments (0)