Why Repetition Penalty Breaks JSON and Code Generation

A batch job of mine emitted arrays of 200 objects. The first ~30 were perfect. Then user_id became userId. Then userld. Then an object closed with ' instead of " and the whole parse died. Same model, same prompt, same seed family — the corruption always started deep into the array and got worse monotonically.

The cause was one line copied from a chat preset: frequency_penalty=0.3.

Repetition penalty is the single most misapplied sampling parameter in production LLM serving. It was designed to stop degenerate loops in open-ended prose, and in structured output it does something close to the opposite of what you want: it applies its strongest downward pressure to the tokens your grammar requires you to repeat.

TL;DR

• Repetition penalty scales with token count, and structural tokens have the highest counts. In JSON, ", :, ,, {, } and repeated field names dominate the output distribution — so they absorb almost all the penalty. Content tokens, which appear once, absorb almost none.
• frequency_penalty is unbounded. OpenAI-style penalties subtract frequency_penalty * count from the logit. After 60 quote characters at 0.3, that's an 18-logit subtraction — an effective ban on ".
• Low temperature amplifies it. Penalties run on raw logits before temperature, so a penalty of δ changes the probability ratio by exp(-δ/T). At temperature=0.2, a 0.5 penalty behaves like a 2.5 penalty. Structured-output configs use low temperature, which is exactly where the penalty bites hardest.
• HF repetition_penalty is multiplicative and asymmetric (score/p if positive, score*p if negative) and by default includes prompt tokens — so your few-shot examples and inline schema are pre-penalized before generation starts.
• Fix: set frequency_penalty=0, presence_penalty=0, repetition_penalty=1.0 for any JSON, code, or tool-call output. Kill real loops with a DRY sampler or a streaming n-gram detector instead.

Why does repetition penalty break JSON and code generation?

Because the penalty's ranking of "most repetitive tokens" is nearly identical to the ranking of "most syntactically mandatory tokens."

Token frequency in structured output is brutally Zipfian, and the head of that distribution is pure syntax. Serialize an array of objects with six fields and the token " appears roughly 12 × n times for n elements. Every field name repeats n times. Indentation tokens in Python repeat once per line. self, =, (, ), return dominate a class body.

Meanwhile the actual content — the values you care about — appears once or twice each and receives essentially zero penalty.

So the penalty gradient points away from syntax and toward novelty. The model, forced to pick something, picks the nearest unpenalized neighbor of the token it wanted:

• " → ' or a Unicode smart quote
• user_id → userId → userID → usr_id
• four-space indent → three-space or two-space indent
• } → nothing, and the object never closes

None of these are hallucinations in the usual sense. They're the arithmetic consequence of subtracting a growing constant from the correct token's logit.

How do the two penalty formulas actually differ?

They differ in a way that matters: one is additive and unbounded, the other is multiplicative and confidence-proportional.

# OpenAI-style (also vLLM's presence_penalty / frequency_penalty).
# Additive, applied over *output* tokens only. Unbounded in count.
def openai_penalty(logits, counts, presence=0.0, frequency=0.0):
    return logits - frequency * counts - presence * (counts > 0)

# HuggingFace-style repetition_penalty (Keskar et al., CTRL).
# Multiplicative and ASYMMETRIC. Applied over prompt + output tokens.
def hf_repetition_penalty(logits, seen_ids, penalty=1.1):
    s = logits[seen_ids]
    logits[seen_ids] = torch.where(s < 0, s * penalty, s / penalty)
    return logits

The additive form has no ceiling. frequency_penalty=0.3 sounds tiny. It is tiny at count 1. At count 60 it is a 18-nat subtraction, which in a softmax is a factor of e^-18 ≈ 1.5e-8. The quote character is gone. Longer output = stronger corruption, which is exactly the "gets worse deep into the array" signature.

The multiplicative form is confidence-proportional, which is subtler and arguably nastier. Dividing a logit by 1.1 removes 9% of its magnitude — so a token the model is certain about (logit 20 → 18.2) loses 1.8 nats, while a marginal token (logit 2 → 1.8) loses 0.2. The penalty is strongest precisely where the model is most confident, which in structured output is always the mandatory next character.

The asymmetry compounds it: negative logits get multiplied by the penalty, pushing them further down. A token with logit −5 becomes −5.5. So the operation isn't a uniform shift in log-space at all; it's a stretch away from zero in both directions, and its effect depends on where the model's logit scale happens to sit — which varies by layer norm, by model, and by temperature calibration.

Also note the scope difference. In vLLM, presence_penalty and frequency_penalty count output tokens; repetition_penalty counts prompt + output tokens. In HF generate(), repetition_penalty applies to everything in input_ids. If your prompt contains the JSON schema, a few-shot example, or retrieved documents that mention your field names, those field names start the generation already penalized.

Why does low temperature make repetition penalty worse?

Because penalties are applied to raw logits before the temperature warper, so temperature divides the penalty too — and dividing by a number less than 1 makes it bigger.

In both HF LogitsProcessorList and vLLM's sampler, the order is: penalties → temperature → top-k/top-p. The final probability ratio between a penalized token a and an unpenalized competitor b is:

p_a / p_b  =  exp( (z_a - z_b) / T ) * exp( -δ / T )

The penalty's contribution is exp(-δ/T). At T=1.0, δ=0.5 is a 1.6× handicap. At T=0.2 — the sort of value everyone uses for extraction and codegen — the same δ=0.5 is a 12× handicap. Low temperature does not "stabilize" the output against penalties; it multiplies their effect by 1/T.

There's a second-order effect too. Because penalties run before nucleus truncation, a heavily penalized token can fall out of the top-p mass entirely. Once it's masked, it has probability zero. Raising temperature afterward cannot bring it back — the token isn't merely improbable, it's been removed from the candidate set.

Which failure modes should I look for?

Four signatures, in rough order of how often they show up:

1. Key drift in repeated objects. The same schema field rendered differently across array elements — user_id / userId / user id. Distinctive because early elements are fine.
2. Quote and delimiter substitution. ' or ’ in place of "; missing closing } or ]. Produces hard parse failures at the tail of long outputs.
3. Indentation collapse in code. Python that starts at 4 spaces and drifts to 2, or a for body that silently dedents. The whitespace token has the highest count of any token in the file.
4. Premature EOS. Every legal continuation is penalized; EOS has appeared zero times and so is penalized zero. Its relative probability rises with output length. Truncated-but-syntactically-plausible output is the hardest version of this to catch.

If you're debugging a "the model gets worse the longer it writes" report, check the penalty parameters before you touch the prompt.

Does structured output or constrained decoding fix it?

No — it hides it, and that's worse.

Grammar-constrained decoding (outlines, XGrammar, llguidance, response_format: json_schema) masks all tokens that would violate the schema, then samples from what's left. So the penalty can no longer produce invalid JSON. What it can still do is bias which valid token wins:

• Enum values drift toward whichever member hasn't been emitted yet.
• With additionalProperties or a union of key names, the model picks the unused key.
• Numeric digits get skewed — digits are individually low-count and the mask leaves several legal, so the penalty tilts the choice between them.
• With optional fields, the mask permits closing the object early, and every non-EOS continuation is penalized. Fields go missing.

You get 100% parse success and quietly wrong values. That failure mode survives every schema-validation test you have.

What should I use instead of repetition penalty?

Turn it off for structured output, and attack real loops with something that understands sequences rather than counts.

The DRY sampler (llama.cpp, exllamav2, text-generation-webui) penalizes based on the length of the verbatim suffix the model is about to extend, not on raw token counts:

penalty = multiplier * base ** (match_length - allowed_length)

It only fires when the model is genuinely about to repeat a long span, and it takes sequence breakers (newline, ", :, ,) that reset matching at structural boundaries. Typical settings: dry_multiplier=0.8, dry_base=1.75, dry_allowed_length=2. For code, raise dry_allowed_length to 4–6 — for i in range( is a legitimate repeat.

Or detect loops outside the sampler. This is what I ship, because it's deterministic and doesn't perturb the distribution at all:

def loop_detected(text, n=12, threshold=3):
    """Stop when any n-gram of characters repeats `threshold` times."""
    tail = text[-4000:]
    grams = {}
    for i in range(len(tail) - n):
        g = tail[i:i + n]
        grams[g] = grams.get(g, 0) + 1
        if grams[g] >= threshold:
            return True
    return False

Run it on the streaming buffer, abort, and retry with a nudged prompt. A retry costs less than a corrupted 200-object payload downstream.

Note also that Anthropic's API doesn't expose frequency or presence penalties at all — Claude Opus 4.x and Sonnet 4.x take temperature, top_p, and top_k. So this class of bug is confined to OpenAI-compatible endpoints, vLLM/SGLang/TGI deployments, and local runtimes. The risk there is that gateways, framework defaults, and copy-pasted chat presets inject nonzero penalties without you noticing.

What's the safe configuration?

from vllm import SamplingParams

# Structured output / codegen: penalties OFF, no exceptions.
structured = SamplingParams(
    temperature=0.0,
    top_p=1.0,
    repetition_penalty=1.0,   # multiplicative — 1.0 is the no-op
    frequency_penalty=0.0,    # additive — 0.0 is the no-op
    presence_penalty=0.0,
    max_tokens=4096,
)

# Open-ended prose, if you must. Presence over frequency: bounded, count-independent.
prose = SamplingParams(
    temperature=0.8,
    top_p=0.95,
    presence_penalty=0.3,     # fires once per token, then stops
    frequency_penalty=0.0,    # grows without bound — leave it alone
)

Two assertions worth putting in CI: any request whose response_format is a JSON schema, or whose route is tagged codegen, must have all three penalty knobs at their no-op values. And log the effective sampling params per request — most of these incidents trace back to a default set three layers up the stack by someone configuring a chat UI.

The short answer

Repetition penalty breaks JSON and code generation because it penalizes tokens by how often they've appeared, and in structured output the most frequent tokens are the mandatory ones: quotes, braces, commas, repeated field names, indentation. OpenAI-style frequency_penalty subtracts penalty × count with no ceiling, so corruption grows with output length; HuggingFace-style repetition_penalty scales with logit magnitude, so it hits the model's most confident predictions hardest and also covers prompt tokens by default. Because penalties are applied before the temperature warper, a low temperature multiplies their effect by 1/T. Constrained decoding masks the syntax errors but converts them into silently wrong enum values, skewed numbers, and dropped optional fields. Set repetition_penalty=1.0, frequency_penalty=0.0, and presence_penalty=0.0 for every structured or code output path, and handle genuine degenerate loops with a DRY sampler or a streaming n-gram detector instead.