LLM JSON output: one setting dropped valid JSON from 75% to 4%

Originally published at tensorust.com.

TL;DR: On October 17, 2026 I asked three small language models for JSON 432 times, in a real Chrome tab, with the exact sampling settings my browser chat app ships. Llama 3.2 1B returned schema-valid JSON 75% of the time with plain greedy decoding. Turning on one anti-repetition setting, no_repeat_ngram_size: 3, dropped that to 4%. With all the chat settings on, it was 0 out of 24. The model wasn't failing at JSON. It was banned from writing it.

Short answer for the search. To get reliable JSON output from an LLM: turn off no_repeat_ngram_size and keep repetition_penalty at 1.0 for any structured call, put the exact schema in the prompt, and parse with a forgiving ladder (strip code fences, cut the first {...} out of the prose, repair small syntax slips) before you validate against the schema. If your provider offers constrained decoding (OpenAI Structured Outputs, llama.cpp GBNF grammars, Outlines), use it, and it makes most of this article unnecessary.

If you have a broken reply in front of you right now, paste it into the free LLM JSON repair tool. It runs the same four-step ladder as this benchmark, in your browser, and tells you which step fixed it. The rest of this piece is the measurement behind those rules, with every number from runs on October 17 between 4:12 and 4:22 PM ET.

One sampling knob at a time on top of greedy decoding. Transformers.js 4.2.0 on WebGPU in real Chrome, Oct 17 2026, 4:12 to 4:22 PM ET.

The setup: 432 replies from three models in a real browser

The models come from BEINGS, my chatroom where every AI creature runs inside the visitor's own browser, so inference costs me nothing. I wrote up how that works in Run an LLM in the browser: WebGPU did 63 tokens/s. The creatures only ever needed one line of chat. The obvious next feature is letting them return structured actions (a mood, a target, a reply), which means JSON. Before building that, I wanted to know how often these models can actually produce it.

Three models, all through Transformers.js 4.2.0 on WebGPU with 4-bit weights, the same pinned build BEINGS loads:

• SmolLM2 360M Instruct
• Qwen2.5 0.5B Instruct
• Llama 3.2 1B Instruct (q4f16)

Three tasks, eight inputs each: tag a chat message with a speaker, a mood from four choices and a reply; pull a name, email and phone out of a message; turn a shopping note into a list of items with integer quantities. Every input has a known right answer, so I could grade three things separately: does it parse, does it match the schema, and is it correct.

The script (json_bench.mjs) attaches to a Chrome window that was already open, opens its own tab on the BEINGS site so the cached models load in about a second, and runs every prompt through two prompt styles and two sampling setups. These are the two setups, copied from the script:

const SAMPLING = {
  chat: { do_sample: true, temperature: 0.75, top_p: 0.9, top_k: 40,
          repetition_penalty: 1.12, no_repeat_ngram_size: 3 }, // = beings llm-worker.js
  greedy: { do_sample: false, repetition_penalty: 1.0, no_repeat_ngram_size: 0 },
};

The chat line is not something I made up for the test. It's what the live BEINGS worker sends today, because small models in a chatroom love to repeat themselves, and those two penalties are the standard cure:

// public/llm-worker.js (BEINGS, live)
out = await pipe(msg.messages, {
  max_new_tokens: msg.maxTokens,
  do_sample: true,
  temperature: msg.temperature,
  top_p: 0.9,
  top_k: 40,
  repetition_penalty: 1.12,
  no_repeat_ngram_size: 3,
  return_full_text: false,
});

The "plain" prompt describes the JSON in a sentence. The "example" prompt gives the schema plus one input and output pair, and says no code fences. 3 models × 3 tasks × 8 inputs × 2 prompts × 2 setups is 288 replies, capped at 160 new tokens each. Then a second run of 144 changed one setting at a time, which is where the headline came from.

The chat settings scored 0 out of 72

With the chat settings and the example prompt, not one reply from any of the three models matched the schema: 0 of 72. With greedy decoding and the same prompt, 25 of 72 did, and Llama alone went from 0 to 18 of 24.

So I split the chat settings apart and turned on one knob at a time, on top of greedy decoding, with the example prompt (24 replies per bar, 4:21 to 4:22 PM ET):

Setting added to greedy	Llama 3.2 1B	SmolLM2 360M
Nothing (greedy)	18/24 (75%)	7/24 (29%)
temperature: 0.75 sampling	19/24 (79%)	9/24 (38%)
repetition_penalty: 1.12	16/24 (67%)	9/24 (38%)
no_repeat_ngram_size: 3	1/24 (4%)	0/24 (0%)
All chat settings together	0/24	0/24

Temperature was harmless here; it even nudged Llama up by one. The repetition penalty cost a little. The n-gram ban destroyed it on its own.

Why: no_repeat_ngram_size bans the tokens your schema needs

no_repeat_ngram_size: 3 means "never produce any sequence of 3 tokens that already appeared". Here's the check in Transformers.js (logits_process.js, trimmed). Any token that would complete a repeated 3-gram gets its score set to minus infinity, so it can never be picked:

_call(input_ids, logits) {
  for (let i = 0; i < input_ids.length; ++i) {
    const batch_logits_data = logits[i].data;
    const bannedTokens = this.calcBannedNgramTokens(input_ids[i]);
    for (const token of bannedTokens) {
      batch_logits_data[token] = -Infinity;
    }
  }
  return logits;
}

The catch is what input_ids holds. In the generate loop (modeling_utils.js) it starts as the whole prompt, and new tokens are appended to it:

const all_input_ids = input_ids.tolist();
// ...each step:
const next_tokens_scores = prepared_logits_processor(all_input_ids, logits);

Now think about JSON. A list of items is {"name": ..., "qty": ...}, {"name": ..., "qty": ...}: the same few tokens over and over by design. And the example prompt already wrote "name": " and "qty": once, before the model said a word. Every one of those 3-token runs is banned from the first token of the reply.

A model can't refuse to answer, so it does the next most likely thing: it swerves. These are real, unedited Llama replies from the ablation run:

Unedited replies from the ablation run, Oct 17 2026, 4:21 PM ET. Red marks what the n-gram ban forced.

" qty" with a leading space. ".name". 'name' in single quotes. "quantity" instead of "qty". "type" instead of "name". Every one is the shortest detour around a banned 3-gram. Some still parse as JSON, which is the dangerous part: a parser says yes, and your code reads item.qty as undefined.

The model wasn't bad at JSON. The sampler was banning the exact tokens the schema required, and the example in the prompt made it worse.

The repetition penalty also counts your prompt

repetition_penalty is the gentler cousin, introduced in the CTRL paper (Keskar et al., 2019). It divides the score of every token that already appeared by the penalty, here 1.12. The Transformers.js source says it plainly in the comment above RepetitionPenaltyLogitsProcessor: "for decoder-only models like most LLMs, the considered tokens include the prompt."

So every quote mark, colon and key name in your schema is slightly less likely to appear again. At 1.12 that's a nudge, not a ban, which matches the data: Llama dropped from 18 to 16 of 24. I'd still set it to 1.0 for structured calls. There's nothing to gain from penalising the exact punctuation you asked for.

The Hugging Face generation docs describe both settings as general quality knobs, and for chat they are. My guess is a lot of apps set them once in a shared config, the way BEINGS did, and every structured call quietly inherits them. Worth a grep in your codebase today.

A parse ladder rescues 43 more replies, not all

Even with good settings, small models wrap JSON in prose and code fences. JSON.parse throws on all of it. So the analysis runs every reply down a four-step ladder and records the first step that worked:

Llama 3.2 1B, chat settings, plain prompt, Oct 17 2026 4:19 PM ET. Error messages are the real JSON.parse output at each step.

Here's the ladder from json_analyze.mjs (trimmed). The repair tool runs a copy of this exact code:

export function extractJson(t) {   // first balanced {...} or [...], string-aware
  const start = t.search(/[{[]/);
  if (start < 0) return null;
  const stack = []; let inStr = false, esc = false;
  for (let i = start; i < t.length; i++) {
    const c = t[i];
    if (inStr) { if (esc) esc = false; else if (c === "\\") esc = true; else if (c === '"') inStr = false; continue; }
    if (c === '"') inStr = true;
    else if (c === "{" || c === "[") stack.push(c === "{" ? "}" : "]");
    else if (c === "}" || c === "]") { stack.pop(); if (!stack.length) return { text: t.slice(start, i + 1), closed: true }; }
  }
  return { text: t.slice(start), closed: false, open: stack, inStr };
}

export function ladder(raw) {
  let v = tryParse(raw);                         // 1. as is
  if (!v) v = tryParse(stripFences(raw));        // 2. strip ```
{% endraw %}
json fences
  if (!v) { const e = extractJson(stripFences(raw)); if (e && e.closed) v = tryParse(e.text); } // 3. cut out {...}
  if (!v) { const e = extractJson(stripFences(raw)); if (e) v = tryParse(repair(e.text)); }       // 4. repair
  return v;
}

{% raw %}

repair() does five small things: drops // comments, quotes bare keys, swaps single quotes for double quotes (only when the text has no double quotes, so "can't" survives), maps Python's True/False/None, and removes trailing commas. If the reply was cut off, it closes whatever brackets were left open. The string-aware scan matters: a naive "find the last }" breaks the moment a value contains a brace.

Across all 288 replies from the main run, strict JSON.parse accepted 84 (29%). The ladder raised that to 127 (44%). For Llama with greedy decoding and the plain prompt it took 14 of 24 to 24 of 24:

json_analyze.mjs over the main run, Oct 17 2026. Parse only, before the schema check.

That last number is the case for always having a ladder. The bigger point is in the gap between 44% and 100%. The ladder fixes how JSON is wrapped. It can't fix replies that never contained JSON.

Why JSON.parse threw: cut-offs and no JSON at all

For every reply that failed strict parsing, the analysis logged the first cause. Of 204 failures, code fences and "Here is your JSON:" prose (the two things everyone writes a regex for) were only 43. The two biggest causes were things no parser can fix:

288 replies from 3 in-browser models, Oct 17 2026.

• Cut off at the token limit (62). Mostly not long answers but loops: Qwen repeating "jars", "jars", "jars" until it ran out of tokens.
• No JSON at all (61). Prose, a Python function that "extracts the contact info", an apology, or the input echoed back.

Qwen2.5 0.5B deserves its own line: 0 of 96 replies parsed as is, 4 after the ladder, and 0 matched the schema, in every configuration. Greedy replies mostly echoed the user's message and then looped ("another meeting another meeting another meeting..."). In my earlier browser benchmark the same 4-bit WebGPU build also produced the most garbage chat lines. I wouldn't read this as a verdict on Qwen2.5 in general; it might be this quantized build. But it's a reminder to test the exact file you ship.

Valid JSON is not the right JSON

Schema-valid and correct are different numbers. Llama's best setup (greedy + example) gave 18 schema-valid replies but 16 correct ones. Two ways it went wrong:

• The example leaks. Asked for the contact details in "Call me back on [phone]. Thanks, Tom Okafor", Llama returned {"name": "Dana Reyes", "email": "[e-mail]", ...}: the name and email from the prompt's example, with Tom's phone number. Across the 288 example-prompt replies, 21 contained a value lifted from the example (Dana, "chaos lord", lemons and so on). With the plain prompt it was 1 in 144.
• Types drift. "qty": "1" as a string, a missing "email" key instead of null. A schema validator such as any JSON Schema library catches these; JSON.parse never will.

The example prompt was still the better prompt for Llama (16 correct vs 12 with the plain prompt). For SmolLM2 it was worse (3 vs 7). Tiny models copy the example rather than learn from it, which fits what I saw in the raw text. If you use an example, make its values obviously fake ("NAME", 0) so a leak is easy to catch.

The JSON spec is tiny, and that is the trap: being syntactically valid is cheap. The work is in the checks after the parse.

What about GPT, Claude and big models?

This was 360M to 1B parameter models running on one consumer GPU in a browser tab, so the failure rates are higher than you'd see from a frontier API. I'd still bet the sampler finding carries over to anyone running open models themselves through Hugging Face style generate code, llama.cpp or vLLM: the n-gram ban is a mechanical rule, and it doesn't care how smart the model is. A bigger model just finds cleverer detours, which might be worse because they look right.

Where you can, skip the guessing entirely. Constrained decoding restricts each next token to ones that keep the output valid against a grammar or schema, so a reply can't be malformed by construction. That's what OpenAI's Structured Outputs, llama.cpp grammars and Outlines do. If you can't, a ladder plus a schema check plus one retry is the practical floor. For Python there's a solid open-source repair library, json_repair, that covers more cases than my five rules.

The same "measure it on your real traffic" rule saved me once before, with an LLM cache that would have served the wrong answer: Stop paying twice for the same LLM answer.

Do this in your project today

1. Grep for no_repeat_ngram_size and repetition_penalty. If a structured call inherits either from a shared config, override them: no_repeat_ngram_size: 0, repetition_penalty: 1.0.
2. Use greedy decoding (or low temperature) for JSON. Temperature didn't hurt here, but greedy makes failures repeatable, so you can fix them.
3. Parse with a ladder. As is, strip fences, extract the first balanced object, repair. Log which step was needed; a rising step-4 rate is an early warning.
4. Validate against the schema after parsing, and check types, not just keys.
5. Retry once on failure with the error in the prompt. The two biggest failure types (loops and no JSON) are the ones a parser can't fix.
6. Make example values obviously fake, and test for leaks.
7. Raise the token limit for lists. A cut-off reply is the one case where repair should close the brackets, but you'll silently lose items.

To try your own replies against the same ladder, use the LLM JSON repair tool. It has four real broken replies from this run preloaded.

The free LLM JSON repair tool running the same ladder, Oct 17 2026.

The numbers

Real output of the analysis script, Oct 17 2026, 4:22 PM ET.

Measure (Oct 17 2026, 4:12 to 4:22 PM ET)	Number
Replies generated and graded	432 (288 main + 144 ablation)
Llama 3.2 1B schema-valid, greedy + example	18/24 (75%)
Same, with only no_repeat_ngram_size: 3 added	1/24 (4%)
All three models, chat settings + example prompt	0/72
Parsed as is, all 288 main replies	84 (29%)
Parsed after the 4-step ladder	127 (44%)
Llama greedy + plain prompt after the ladder	24/24
Replies that leaked example values	21 of 288 example-prompt replies
Qwen2.5 0.5B (q4f16, WebGPU) schema-valid	0/96
Llama average time per JSON reply (greedy + example)	658 ms, 43 tokens

One honest note on the run: Chrome's WebGPU device was lost twice (once 18 replies into Llama's first pass, once at the start of SmolLM2's ablation). I reran Llama's batch from scratch and the script reloads the model and retries the reply on a lost device, so every number above comes from a complete set.

Next for BEINGS: creature actions as JSON, with a separate greedy, penalty-free config for structured calls while chat keeps its anti-repetition settings. I'll report whether the 75% holds on real room traffic.

Question for you: have you found a sampling or decoding setting that quietly broke structured output in your stack? I'd like to know which ones bite outside the browser.

---

Got a broken reply right now? Paste it into the free LLM JSON repair tool. Need AI built into an app you already have? Fixed prices here.