A model that writes clean prose starts emitting broken Python the moment you set repetition_penalty=1.3. Indentation collapses, brackets go unmatched, and it refuses to write return twice in the same function. Nothing in your prompt changed. What changed is that you turned on a sampling penalty that was designed to stop prose from looping — and code is nothing but legal repetition.
This is the trap with repetition penalty and its cousins frequency_penalty and presence_penalty. They look like interchangeable "stop the model repeating itself" knobs. They are three different math operations on the logits, one of them has a sign quirk that surprises people, and all three fight directly against structured output.
TL;DR
- repetition_penalty (Hugging Face, llama.cpp) is multiplicative on the logit and scale-free: it divides positive logits and multiplies negative ones by the penalty. Above ~1.2 it visibly degrades code, JSON, and any format with mandatory repeated tokens.
- frequency_penalty and presence_penalty (OpenAI Chat Completions) are additive subtractions from the logit — frequency scales with how many times a token appeared, presence is a flat one-time hit.
- The Anthropic Messages API for Claude exposes none of these — no repetition, frequency, or presence penalty. That is a deliberate design choice, and it's usually the right one.
- Penalties operate on tokens, not concepts, so they punish
(indentation),,,", andreturnexactly as hard as a genuinely looping phrase. - For repetition problems, prefer
no_repeat_ngram_sizeor a DRY sampler over a blunt global penalty, or just lower the temperature and use min-p.
What does repetition_penalty actually do to the logits?
It rescales every logit for tokens that already appeared in the context, before softmax. The Hugging Face / CTRL formulation (Keskar et al., 2019) is not a subtraction — it's a conditional multiply:
# Hugging Face-style repetition_penalty, per generated token
def apply_repetition_penalty(logits, generated_ids, penalty):
for tok in set(generated_ids):
if logits[tok] > 0:
logits[tok] /= penalty # shrink toward 0
else:
logits[tok] *= penalty # push more negative
return logits
Note the branch. For a positive logit, dividing by penalty > 1 moves it toward zero (less likely). For a negative logit, you multiply, which moves it further from zero — more negative, also less likely. The sign flip is there precisely so the penalty always reduces probability regardless of the logit's sign.
That branch is also the source of endless confusion. The penalty is scale-dependent: a logit of 8.0 with penalty=1.3 becomes 6.15 (a 1.85 drop); a logit of 0.5 becomes 0.38 (a 0.12 drop). The same penalty value hits confident tokens harder in absolute terms but leaves the relative ordering warped in ways that depend on the raw logit magnitudes of that specific model. A "safe" repetition_penalty on Llama is not a safe one on Qwen, because their logit scales differ.
Why does repetition_penalty break code and JSON?
Because the penalty acts on token IDs with no idea what they mean, and code is built from a small set of tokens that must recur. Python indentation is the same whitespace token on every line. JSON is a stream of ", :, ,, {, }. A function that returns in three branches emits the return token three times, correctly.
Once those tokens land in the context window, repetition_penalty discounts them on every subsequent step. By the tenth line of a function, the model has been told — repeatedly — that the indentation token, the comma, and the closing brace are all "overused." So it starts reaching for alternatives that don't exist in valid syntax: a different quote character, a dropped comma, three spaces instead of four. You get output that is locally plausible and globally uncompilable.
The effect scales with how much you've already generated. Short completions look fine; the corruption shows up in longer files, which is exactly when you can't eyeball it. This is why teams ship a repetition_penalty=1.15 default that "worked in testing" and then get bug reports about malformed JSON in production — the test prompts were short.
frequency_penalty vs presence_penalty: what's the difference?
They're both additive penalties on the logit, and they differ only in how they count. Given count[tok] = how many times a token has appeared:
# OpenAI-style additive penalties
logits[tok] -= count[tok] * frequency_penalty # grows with repetition
logits[tok] -= (count[tok] > 0) * presence_penalty # flat, one-time hit
frequency_penalty scales linearly with occurrences — the fourth use of a word is penalized four times as hard as the first. presence_penalty is binary: appear once and you take a fixed hit, appear ten more times and it doesn't grow. Both range from -2.0 to 2.0 in the OpenAI API; negative values encourage repetition, which is occasionally useful for forcing a fixed refrain or schema.
Because they subtract a fixed amount rather than rescaling, additive penalties are more predictable across models than multiplicative repetition_penalty — the penalty is in logit units you can reason about, independent of the base logit's magnitude. They still punish structural tokens, so a high frequency_penalty will damage JSON and code too, just more gently and more legibly. As a rule, keep frequency_penalty under ~0.3 for any structured output, and reach for presence_penalty when you want "cover new topics" behavior without compounding.
Why does Claude have no repetition penalty at all?
The Anthropic Messages API exposes temperature, top_p, top_k, and stop_sequences — and no repetition, frequency, or presence penalty. This is not an oversight. Penalty knobs are a patch for models whose base sampling distribution degenerates into loops (the "neural text degeneration" problem Holtzman et al. described in 2019). A well-tuned modern model with good sampling defaults rarely needs them, and exposing them invites exactly the code-corruption footgun above.
If Claude Opus 4.x starts looping — which is uncommon — the fix is upstream: tighten the prompt, add a stop_sequences entry, drop the temperature, or give it an explicit instruction not to repeat a section. You do not get a blunt logit hammer, and in practice you don't miss it. GPT-5 via the Chat Completions API still exposes frequency_penalty and presence_penalty; if you're building a provider-agnostic layer, treat those as OpenAI-only params and don't try to emulate them for Claude.
What should I use instead of a global penalty?
Match the tool to the failure. A global penalty is the wrong instrument for most repetition problems because it's context-length-blind and token-blind.
no_repeat_ngram_size hard-bans any n-gram from occurring twice. Set it to 3 and the model can never emit the same 3-token sequence again. This is a scalpel compared to repetition_penalty's hammer: single tokens like indentation and commas are untouched, only actual repeated phrases get blocked. The risk is over-blocking legitimate n-grams (for i in, } else {), so keep the n large enough that real code patterns survive.
DRY (Don't Repeat Yourself) sampler, available in llama.cpp and text-generation-webui, penalizes tokens that would extend an existing repeated sequence, with a length-scaled multiplier and an allowlist for sequences you expect to recur. It targets runaway loops specifically while leaving normal structure alone — the best general-purpose option for open models prone to looping.
Just lower the temperature and add min-p. A surprising share of "the model keeps repeating" reports are actually high-temperature sampling wandering into a low-probability loop it can't escape. Dropping temperature to 0.7 with min_p=0.05 fixes the degeneration without penalizing any structural token. Try this before you touch any penalty.
Here's a defensible config split by task:
# Prose / brainstorming — light penalty is fine
{"temperature": 0.9, "frequency_penalty": 0.3, "presence_penalty": 0.3}
# Code / JSON / structured output — no global penalty
{"temperature": 0.2, "top_p": 0.95, "frequency_penalty": 0.0,
"presence_penalty": 0.0, "no_repeat_ngram_size": 0} # rely on low temp
# Open model looping badly — DRY beats repetition_penalty
{"temperature": 0.7, "min_p": 0.05, "dry_multiplier": 0.8,
"dry_base": 1.75, "dry_allowed_length": 2}
The one-paragraph answer
repetition_penalty corrupts code generation because it is a multiplicative, token-level penalty applied to every token that already appeared in the context — and code is built from a tiny alphabet of tokens (indentation, commas, quotes, keywords, brackets) that must legally repeat. Above roughly 1.2, the model gets told its own indentation and punctuation are "overused" and starts substituting invalid alternatives, producing longer completions that silently fail to parse. It differs from OpenAI's additive frequency_penalty (scales with count) and presence_penalty (flat one-time hit), which are gentler but share the same structural-token problem, and from Claude, which exposes no penalty at all by design. For structured output, disable global penalties entirely and control repetition with low temperature plus min-p, no_repeat_ngram_size, or a DRY sampler — instruments that target repeated phrases instead of punishing the syntax your output depends on.
Top comments (0)