frequency_penalty Broke My JSON: 47 of 88 Long Invoices Lost Items

The invoice had 34 line items. The JSON my extractor returned had 19. It parsed. It passed schema validation. Nobody noticed for two weeks.

The cause was one line in a shared client config: frequency_penalty=0.3. I had added it to stop a summarizer from looping. It was also quietly applied to my invoice extraction endpoint, and frequency_penalty breaks JSON output in a way that gets worse with every item in the list. Short invoices were fine. Long ones fell apart.

Here's the mechanism, the math, and what I changed.

TL;DR

• frequency_penalty subtracts alpha * count from the logit of every token that has already appeared in the output. The penalty has no idea what JSON is.
• JSON repeats the same tokens constantly: quotes, colons, commas, braces, and every key name once per array item. Those counts grow linearly with array length.
• Eventually the penalized "continue" token (,) loses to the never-used "stop" token (]). The model closes the array early. The output is valid JSON with missing items.
• Constrained decoding (JSON schema / grammar mode) fixes renamed keys but does not fix truncation, because ] is a legal token at every item boundary.
• Fix: set frequency_penalty=0, presence_penalty=0, repetition_penalty=1.0 for any structured output request, and validate item counts against something independent.

What actually happened?

Setup: a vLLM server running an open-weights 7B instruct model, called through the OpenAI-compatible API. One endpoint summarized support threads. Another extracted line items from invoices into this shape:

{"items": [{"sku": "A-1001", "description": "Hex bolt M8", "qty": 200, "unit_price": 0.14}]}

The summarizer had a habit of repeating the same sentence three times at the end of long threads. I added this to the shared client wrapper:

DEFAULT_SAMPLING = {
    "temperature": 0.2,
    "frequency_penalty": 0.3,
    "presence_penalty": 0.1,
}

The looping stopped. I moved on.

Two weeks later, accounting flagged an invoice where our extracted total was way below the printed total. I pulled every extraction from that window and compared item counts against a separate OCR line count. These are my numbers from that batch:

Invoice size	Invoices	Wrong item count or invalid JSON
1-10 items	290	1
11-25 items	122	9
26+ items	88	47

Of the 47 long-invoice failures, most were valid JSON with the array closed early. A smaller group had keys that drifted mid-array: "sku" became "SKU" or "sku_id", or "unit_price" turned into "price". Those failed Pydantic validation, so at least they were loud. The truncated ones were silent.

The failure rate tracking array length was the clue. Prompt problems don't usually care how long your output is. A per-token counter does.

How does frequency_penalty actually work?

frequency_penalty is a logit adjustment applied at every decoding step: for each token already in the generated output, subtract alpha times the number of times it has appeared. presence_penalty subtracts a flat amount once a token has appeared at all.

The OpenAI-style formula, which vLLM implements for these two parameters:

logit[t] = logit[t] - frequency_penalty * count[t] - presence_penalty * (count[t] > 0)

Two details matter here:

1. It operates on tokens, not words or meaning. It doesn't know , is syntax. It just sees a token with a high count.
2. The count never resets. Item 30 is penalized for every comma in items 1 through 29.

For prose that's the point. You want the model to stop reaching for the same phrase. For JSON it's poison, because JSON is supposed to repeat. An array of 30 objects with 4 keys repeats each key 30 times and each separator over a hundred times.

Why does the model close the array early?

At the end of every array item, the model makes one choice: emit , and keep going, or emit ] and stop. The penalty tilts that choice further toward ] on every item.

, has been emitted many times. ] has been emitted zero times, because the array hasn't closed yet. So , carries a large penalty and ] carries none. At some item, the gap the model "wanted" between them gets erased.

Here's a toy simulation. These logits are made up to show the shape of the curve. They are not measured from any model:

import math

def p_continue(n_items, base_gap=9.0, alpha=0.3, commas_per_item=4):
    """Probability of ',' vs ']' after n_items, two-token toy model."""
    comma_count = n_items * commas_per_item
    gap = base_gap - alpha * comma_count  # ']' has count 0, so no penalty
    return 1 / (1 + math.exp(-gap))

for n in [1, 5, 8, 10, 15, 20]:
    print(n, round(p_continue(n), 3))

1 1.0
5 0.953
8 0.354
10 0.047
15 0.0
20 0.0

Look at the shape. Nothing happens for a while, then the probability drops off a cliff over two or three items. That matches what I saw: invoices were either complete or badly truncated, rarely off by one.

Real models have larger logit gaps for syntax than my toy, and tokenizers often merge ", or }, into single tokens, which spreads the count across several token IDs. That's why my real failures started in the 20s instead of at item 8. The curve is the same. Only where it starts moves.

Why do the keys get renamed?

Key drift is the same math applied to key tokens. By item 25, the token for sku has a count of 25. With alpha=0.3 that's a 7.5 logit penalty, which is roughly a 1,800x hit to its relative probability. Meanwhile SKU, sku, and sku_id are different token IDs with a count of zero.

The model still "knows" the key is sku. It just picks the closest unpenalized spelling. You get a valid-looking object with a key your schema has never seen.

Values suffer too. A column where every row says "EA" or "USD" gets penalized into "each" or "usd". Digits get counted, so long numbers with repeated digits are under pressure as well.

Does structured output or guided decoding fix this?

Partly. Constrained decoding (vLLM's guided JSON, response_format with a JSON schema, grammar-based samplers) masks out any token the schema doesn't allow. That kills key renaming, because "SKU" isn't a legal token sequence when the schema says "sku".

It does not fix truncation. At the end of every array item, both , and ] are legal under the grammar. The mask lets both through, and the penalty still decides between them. If your schema has no minItems, a 19-item array is just as valid as a 34-item one.

So the order of fixes is: penalty off first, constraints second.

What about repetition_penalty in vLLM and Hugging Face?

repetition_penalty is a different knob with a different failure mode, and it's arguably worse for structured output. Instead of subtracting a count, it scales any token that has already appeared: positive logits get divided by the penalty, negative logits get multiplied by it. It's binary per token, not count-based.

The catch: in both Hugging Face generate and vLLM, repetition_penalty looks at the prompt tokens too, not just the output. If your system prompt contains an example JSON object, every key and every brace in it is penalized before the model writes its first token. A value of 1.15 sounds tiny. On a token with logit 20, it's a drop to about 17.4, a 2.6 logit hit on the exact tokens your schema needs.

If you put a schema example in your prompt and run with repetition_penalty > 1, you are penalizing the model for following your example.

How do I fix frequency_penalty breaking JSON?

Stop sharing sampling parameters between prose and structured endpoints. Here's what I run now:

PROSE_SAMPLING = {
    "temperature": 0.7,
    "frequency_penalty": 0.3,
    "presence_penalty": 0.1,
}

STRUCTURED_SAMPLING = {
    "temperature": 0.0,
    "frequency_penalty": 0.0,
    "presence_penalty": 0.0,
    "extra_body": {"repetition_penalty": 1.0},  # vLLM-specific, explicit on purpose
}

I set the zeros explicitly instead of omitting them. If someone changes server-side defaults or a wrapper adds a penalty later, the explicit value wins.

Then, the checks that would have caught this in a day instead of two weeks:

1. Validate counts against an independent signal. For invoices: sum of qty * unit_price should match the printed total within rounding. For any list extraction, compare against a cheap heuristic count (OCR rows, regex hits, bullet count).
2. Add minItems or an expected count to the schema when you know it. If the upstream system knows there are 34 rows, say so.
3. Log finish_reason and output token count per request. Truncation from the penalty shows finish_reason: "stop", not "length", which is exactly why it hides. A sudden drop in tokens per item is the tell.
4. Write a length sweep test. Generate synthetic inputs with 5, 10, 20, 40, 80 items and assert the output count matches. Run it whenever sampling config changes. My failures only showed up past 25 items; a test suite of 3-item fixtures would never catch them.

The summarizer looping problem I originally added the penalty for? It went away after I lowered max_tokens for that endpoint and added a stop sequence. The penalty was the wrong tool even there.

Why does frequency_penalty break JSON output?

frequency_penalty breaks JSON output because it subtracts alpha * count from every token already generated, and JSON is built from tokens that must repeat: quotes, colons, commas, braces, and the same key names in every array item. As an array grows, the continue token , piles up penalty while the closing token ] has none, so the model ends the array early and returns valid but incomplete JSON. Key tokens get the same treatment and drift to unpenalized spellings like SKU or sku_id. Constrained decoding blocks the renamed keys but not the early ]. Set frequency_penalty, presence_penalty, and repetition_penalty to neutral values on every structured output request, and check extracted item counts against a source that doesn't come from the model.

---

Written by the developer behind Preterview, an interview prep platform.