DEV Community: jidonglab

Why Repetition Penalty Breaks JSON and Code Generation

jidonglab — Tue, 04 Aug 2026 21:15:05 +0000

A batch job of mine emitted arrays of 200 objects. The first ~30 were perfect. Then user_id became userId. Then userld. Then an object closed with ' instead of " and the whole parse died. Same model, same prompt, same seed family — the corruption always started deep into the array and got worse monotonically.

The cause was one line copied from a chat preset: frequency_penalty=0.3.

Repetition penalty is the single most misapplied sampling parameter in production LLM serving. It was designed to stop degenerate loops in open-ended prose, and in structured output it does something close to the opposite of what you want: it applies its strongest downward pressure to the tokens your grammar requires you to repeat.

TL;DR

Repetition penalty scales with token count, and structural tokens have the highest counts. In JSON, ", :, ,, {, } and repeated field names dominate the output distribution — so they absorb almost all the penalty. Content tokens, which appear once, absorb almost none.
frequency_penalty is unbounded. OpenAI-style penalties subtract frequency_penalty * count from the logit. After 60 quote characters at 0.3, that's an 18-logit subtraction — an effective ban on ".
Low temperature amplifies it. Penalties run on raw logits before temperature, so a penalty of δ changes the probability ratio by exp(-δ/T). At temperature=0.2, a 0.5 penalty behaves like a 2.5 penalty. Structured-output configs use low temperature, which is exactly where the penalty bites hardest.
HF repetition_penalty is multiplicative and asymmetric (score/p if positive, score*p if negative) and by default includes prompt tokens — so your few-shot examples and inline schema are pre-penalized before generation starts.
Fix: set frequency_penalty=0, presence_penalty=0, repetition_penalty=1.0 for any JSON, code, or tool-call output. Kill real loops with a DRY sampler or a streaming n-gram detector instead.

Why does repetition penalty break JSON and code generation?

Because the penalty's ranking of "most repetitive tokens" is nearly identical to the ranking of "most syntactically mandatory tokens."

Token frequency in structured output is brutally Zipfian, and the head of that distribution is pure syntax. Serialize an array of objects with six fields and the token " appears roughly 12 × n times for n elements. Every field name repeats n times. Indentation tokens in Python repeat once per line. self, =, (, ), return dominate a class body.

Meanwhile the actual content — the values you care about — appears once or twice each and receives essentially zero penalty.

So the penalty gradient points away from syntax and toward novelty. The model, forced to pick something, picks the nearest unpenalized neighbor of the token it wanted:

" → ' or a Unicode smart quote
user_id → userId → userID → usr_id
four-space indent → three-space or two-space indent
} → nothing, and the object never closes

None of these are hallucinations in the usual sense. They're the arithmetic consequence of subtracting a growing constant from the correct token's logit.

How do the two penalty formulas actually differ?

They differ in a way that matters: one is additive and unbounded, the other is multiplicative and confidence-proportional.

# OpenAI-style (also vLLM's presence_penalty / frequency_penalty).
# Additive, applied over *output* tokens only. Unbounded in count.
def openai_penalty(logits, counts, presence=0.0, frequency=0.0):
    return logits - frequency * counts - presence * (counts > 0)

# HuggingFace-style repetition_penalty (Keskar et al., CTRL).
# Multiplicative and ASYMMETRIC. Applied over prompt + output tokens.
def hf_repetition_penalty(logits, seen_ids, penalty=1.1):
    s = logits[seen_ids]
    logits[seen_ids] = torch.where(s < 0, s * penalty, s / penalty)
    return logits

The additive form has no ceiling. frequency_penalty=0.3 sounds tiny. It is tiny at count 1. At count 60 it is a 18-nat subtraction, which in a softmax is a factor of e^-18 ≈ 1.5e-8. The quote character is gone. Longer output = stronger corruption, which is exactly the "gets worse deep into the array" signature.

The multiplicative form is confidence-proportional, which is subtler and arguably nastier. Dividing a logit by 1.1 removes 9% of its magnitude — so a token the model is certain about (logit 20 → 18.2) loses 1.8 nats, while a marginal token (logit 2 → 1.8) loses 0.2. The penalty is strongest precisely where the model is most confident, which in structured output is always the mandatory next character.

The asymmetry compounds it: negative logits get multiplied by the penalty, pushing them further down. A token with logit −5 becomes −5.5. So the operation isn't a uniform shift in log-space at all; it's a stretch away from zero in both directions, and its effect depends on where the model's logit scale happens to sit — which varies by layer norm, by model, and by temperature calibration.

Also note the scope difference. In vLLM, presence_penalty and frequency_penalty count output tokens; repetition_penalty counts prompt + output tokens. In HF generate(), repetition_penalty applies to everything in input_ids. If your prompt contains the JSON schema, a few-shot example, or retrieved documents that mention your field names, those field names start the generation already penalized.

Why does low temperature make repetition penalty worse?

Because penalties are applied to raw logits before the temperature warper, so temperature divides the penalty too — and dividing by a number less than 1 makes it bigger.

In both HF LogitsProcessorList and vLLM's sampler, the order is: penalties → temperature → top-k/top-p. The final probability ratio between a penalized token a and an unpenalized competitor b is:

p_a / p_b  =  exp( (z_a - z_b) / T ) * exp( -δ / T )

The penalty's contribution is exp(-δ/T). At T=1.0, δ=0.5 is a 1.6× handicap. At T=0.2 — the sort of value everyone uses for extraction and codegen — the same δ=0.5 is a 12× handicap. Low temperature does not "stabilize" the output against penalties; it multiplies their effect by 1/T.

There's a second-order effect too. Because penalties run before nucleus truncation, a heavily penalized token can fall out of the top-p mass entirely. Once it's masked, it has probability zero. Raising temperature afterward cannot bring it back — the token isn't merely improbable, it's been removed from the candidate set.

Which failure modes should I look for?

Four signatures, in rough order of how often they show up:

Key drift in repeated objects. The same schema field rendered differently across array elements — user_id / userId / user id. Distinctive because early elements are fine.
Quote and delimiter substitution. ' or ’ in place of "; missing closing } or ]. Produces hard parse failures at the tail of long outputs.
Indentation collapse in code. Python that starts at 4 spaces and drifts to 2, or a for body that silently dedents. The whitespace token has the highest count of any token in the file.
Premature EOS. Every legal continuation is penalized; EOS has appeared zero times and so is penalized zero. Its relative probability rises with output length. Truncated-but-syntactically-plausible output is the hardest version of this to catch.

If you're debugging a "the model gets worse the longer it writes" report, check the penalty parameters before you touch the prompt.

Does structured output or constrained decoding fix it?

No — it hides it, and that's worse.

Grammar-constrained decoding (outlines, XGrammar, llguidance, response_format: json_schema) masks all tokens that would violate the schema, then samples from what's left. So the penalty can no longer produce invalid JSON. What it can still do is bias which valid token wins:

Enum values drift toward whichever member hasn't been emitted yet.
With additionalProperties or a union of key names, the model picks the unused key.
Numeric digits get skewed — digits are individually low-count and the mask leaves several legal, so the penalty tilts the choice between them.
With optional fields, the mask permits closing the object early, and every non-EOS continuation is penalized. Fields go missing.

You get 100% parse success and quietly wrong values. That failure mode survives every schema-validation test you have.

What should I use instead of repetition penalty?

Turn it off for structured output, and attack real loops with something that understands sequences rather than counts.

The DRY sampler (llama.cpp, exllamav2, text-generation-webui) penalizes based on the length of the verbatim suffix the model is about to extend, not on raw token counts:

penalty = multiplier * base ** (match_length - allowed_length)

It only fires when the model is genuinely about to repeat a long span, and it takes sequence breakers (newline, ", :, ,) that reset matching at structural boundaries. Typical settings: dry_multiplier=0.8, dry_base=1.75, dry_allowed_length=2. For code, raise dry_allowed_length to 4–6 — for i in range( is a legitimate repeat.

Or detect loops outside the sampler. This is what I ship, because it's deterministic and doesn't perturb the distribution at all:

def loop_detected(text, n=12, threshold=3):
    """Stop when any n-gram of characters repeats `threshold` times."""
    tail = text[-4000:]
    grams = {}
    for i in range(len(tail) - n):
        g = tail[i:i + n]
        grams[g] = grams.get(g, 0) + 1
        if grams[g] >= threshold:
            return True
    return False

Run it on the streaming buffer, abort, and retry with a nudged prompt. A retry costs less than a corrupted 200-object payload downstream.

Note also that Anthropic's API doesn't expose frequency or presence penalties at all — Claude Opus 4.x and Sonnet 4.x take temperature, top_p, and top_k. So this class of bug is confined to OpenAI-compatible endpoints, vLLM/SGLang/TGI deployments, and local runtimes. The risk there is that gateways, framework defaults, and copy-pasted chat presets inject nonzero penalties without you noticing.

What's the safe configuration?

from vllm import SamplingParams

# Structured output / codegen: penalties OFF, no exceptions.
structured = SamplingParams(
    temperature=0.0,
    top_p=1.0,
    repetition_penalty=1.0,   # multiplicative — 1.0 is the no-op
    frequency_penalty=0.0,    # additive — 0.0 is the no-op
    presence_penalty=0.0,
    max_tokens=4096,
)

# Open-ended prose, if you must. Presence over frequency: bounded, count-independent.
prose = SamplingParams(
    temperature=0.8,
    top_p=0.95,
    presence_penalty=0.3,     # fires once per token, then stops
    frequency_penalty=0.0,    # grows without bound — leave it alone
)

Two assertions worth putting in CI: any request whose response_format is a JSON schema, or whose route is tagged codegen, must have all three penalty knobs at their no-op values. And log the effective sampling params per request — most of these incidents trace back to a default set three layers up the stack by someone configuring a chat UI.

The short answer

Repetition penalty breaks JSON and code generation because it penalizes tokens by how often they've appeared, and in structured output the most frequent tokens are the mandatory ones: quotes, braces, commas, repeated field names, indentation. OpenAI-style frequency_penalty subtracts penalty × count with no ceiling, so corruption grows with output length; HuggingFace-style repetition_penalty scales with logit magnitude, so it hits the model's most confident predictions hardest and also covers prompt tokens by default. Because penalties are applied before the temperature warper, a low temperature multiplies their effect by 1/T. Constrained decoding masks the syntax errors but converts them into silently wrong enum values, skewed numbers, and dropped optional fields. Set repetition_penalty=1.0, frequency_penalty=0.0, and presence_penalty=0.0 for every structured or code output path, and handle genuine degenerate loops with a DRY sampler or a streaming n-gram detector instead.

RoPE Scaling: Why Raising rope_theta Breaks Short Context

jidonglab — Mon, 03 Aug 2026 21:12:23 +0000

You edit one line in config.json — rope_theta: 10000.0 becomes 1000000.0 — restart vLLM with --max-model-len 131072, and your needle-in-a-haystack test at 64k goes green. Then your function-calling eval, which averages 900 tokens per prompt, drops a couple of points. Nothing else changed. No weights moved. No sampling params changed.

That is not noise, and it is not a bug in your serving stack. RoPE scaling buys long-range positional coverage by spending short-range positional resolution, and the exchange rate is computable from the config you just edited.

TL;DR

RoPE gives dimension pair i of each head a wavelength λ_i = 2π · base^(2i/d). Raising rope_theta stretches every wavelength except i=0, and stretches the high-index dims the most.
Stretching wavelengths moves whole bands of dimensions out of the range where they can discriminate nearby positions. With d=128, going from base 1e4 to 1e6 drops the number of dimension pairs with sub-2048-token wavelength from 40 to 27 — a third of your short-range positional bandwidth, gone.
Linear position interpolation (dividing positions by s) is worse for short context: it compresses the high-frequency dims that encode adjacency.
YaRN (NTK-by-parts) fixes this by only interpolating the dims that never completed enough rotations during training, leaving the fast dims alone, plus an attention temperature of 0.1·ln(s) + 1 to re-sharpen logits.
Static YaRN applies the scaling to every request, including your 400-token ones. Enable it only when you actually need the length, or use dynamic scaling and accept that it complicates prefix-cache reuse.

What does rope_theta actually control?

Rotary position embedding splits each head's d-dimensional query and key into d/2 2-D pairs and rotates pair i at position m by angle m · θ_i, where θ_i = base^(-2i/d). The dot product between a query at m and a key at n then depends only on m - n. That relative property is why RoPE extrapolates at all.

The useful way to read that formula is as a wavelength per dimension pair: λ_i = 2π / θ_i = 2π · base^(2i/d). With d = 128 and base = 10000:

dim pair	wavelength (tokens)
0	6.3
16	63
32	628
48	6,283
63	~57,000

Each head carries a bank of positional "clocks" spanning four orders of magnitude. Fast dims resolve adjacency — which of these two tokens came first. Slow dims resolve document-scale position — beginning versus middle. A dim is only informative over roughly half a wavelength; past that it wraps and starts aliasing.

Why does raising rope_theta break short-context accuracy?

Because base scaling is not uniform, and the dims it hurts are the ones doing mid-range work.

Note that λ_0 = 2π · base^0 = 2π regardless of base. The very fastest dim is untouched. The multiplier on λ_i is (new/old)^(2i/d), so it grows with i. Going from 1e4 to 1e6 at d=128:

dim 16: 63 → 199 tokens (3.2×)
dim 32: 628 → 6,283 tokens (10×)
dim 48: 6,283 → 198,700 tokens (31.6×)

Dim 32 used to be the clock for "roughly where in this 600-token span are we." Now it takes 6,000 tokens to complete a rotation, so inside a 900-token prompt it barely moves. Its contribution to the attention logit is nearly constant across the whole prompt — a bias, not a signal.

Count the dims that still complete a full rotation inside 2048 tokens. Solve 2π · base^(i/64) < 2048:

base 1e4: i < 40 → 40 of 64 pairs
base 1e6: i < 27 → 27 of 64 pairs

You deleted a third of the positional bandwidth available at typical prompt lengths, and the model's attention heads were trained against the old allocation. The heads that learned to read dim 32-40 for paragraph-level ordering now read a nearly-DC signal. That is the mechanism behind the eval drop, and it shows up hardest on tasks with position-sensitive structure: multi-item ordering, "the third tool result", diff-style comparisons, long system prompts with numbered rules.

Fine-tuning after the base change largely repairs this — the model reallocates which dims it trusts. Llama 3.1 shipping with rope_theta = 500000 and Qwen with 1e6 are trained that way, not patched at inference time. What does not work is editing the number on a checkpoint trained at 10000 and expecting short prompts to behave.

Why doesn't linear position interpolation fix it either?

Position interpolation divides positions by the scale factor s before rotation, so a 32k sequence maps into the 8k range the model saw in training. Every angle shrinks by s.

That is strictly worse for short context than base scaling. Base scaling leaves the fast dims alone; PI squeezes them the most in relative terms. Adjacent tokens that were separated by a rotation of 2π/6.3 now differ by 2π/(6.3·s). At s=4 the model must resolve token order from a quarter of the angular separation it trained on. PI works, but only with fine-tuning, and it is why naive PI reports degraded local-detail behavior.

NTK-aware scaling was the first fix: instead of scaling positions, scale the base so that the highest-frequency dims are almost untouched and only the slow dims get effectively interpolated. Better zero-shot, but it is still a blunt instrument — every dim is adjusted by a smooth function of i, including the ones that needed nothing.

What does YaRN do differently?

YaRN makes the interpolate-or-extrapolate decision per dimension, based on how many rotations that dim completed inside the original training context.

Define r_i = L_orig / λ_i, the rotation count. If r_i > β (default 32), the dim has seen many full periods during training; it can extrapolate safely, so leave it alone. If r_i < α (default 1), the dim never completed even one rotation, so every long-context position is genuinely out of distribution; interpolate it fully by s. Between them, ramp linearly.

import math

def yarn_inv_freq(dim=128, base=10000.0, scale=4.0, L_orig=8192,
                  beta_fast=32, beta_slow=1):
    """Per-dimension-pair inverse frequencies under YaRN (NTK-by-parts)."""
    exps = [2 * i / dim for i in range(dim // 2)]
    extrap = [base ** -e for e in exps]           # untouched
    interp = [(base ** -e) / scale for e in exps]  # stretched by s

    def dim_at_rotations(r):
        # index whose wavelength completes exactly r rotations in L_orig
        return dim * math.log(L_orig / (r * 2 * math.pi)) / (2 * math.log(base))

    low = math.floor(dim_at_rotations(beta_fast))   # fast side
    high = math.ceil(dim_at_rotations(beta_slow))   # slow side

    out = []
    for i, (e, f) in enumerate(zip(extrap, interp)):
        ramp = 0.0 if i <= low else 1.0 if i >= high else (i - low) / (high - low)
        out.append(e * (1 - ramp) + f * ramp)
    return out

# attention temperature: multiply cos/sin tables by this
attn_factor = 0.1 * math.log(4.0) + 1.0   # ~1.139 at s=4

Run the boundaries for d=128, base=1e4, L_orig=8192: low = 25, high = 50. Dims 0-25 are left exactly as trained. Dims 50-63 are fully interpolated. Only the 24 dims in between get a blend. Your adjacency clocks survive untouched — that is the whole trick.

The second half of YaRN is the attention temperature. Longer sequences mean more keys competing in the softmax, which raises attention entropy and flattens the distribution. YaRN counters it by scaling logits, implemented for free by multiplying the cos/sin tables by sqrt(1/t) = 0.1·ln(s) + 1. At s=4 that is a 1.139× sharpening. It costs nothing at runtime and it is the difference between "works" and "works well" in the paper's perplexity numbers.

Config side, HF and vLLM take the same shape:

{
  "rope_scaling": {
    "rope_type": "yarn",
    "factor": 4.0,
    "original_max_position_embeddings": 32768,
    "beta_fast": 32,
    "beta_slow": 1
  },
  "max_position_embeddings": 131072
}

vllm serve Qwen/Qwen3-8B \
  --max-model-len 131072 \
  --rope-scaling '{"rope_type":"yarn","factor":4.0,"original_max_position_embeddings":32768}'

Does static YaRN hurt short prompts too?

Yes, and this is the part people miss. Static YaRN applies the same interpolation to every request. A 400-token prompt gets its slow dims compressed by 4× for no reason — you paid the long-context tax on traffic that never needed it. Qwen's own docs say as much: turn YaRN on only when long inputs are actually expected.

Dynamic scaling computes s = max(1, L_current / L_orig) per sequence, so short prompts run the identity transform. The cost is that the frequency table changes as a sequence grows, meaning keys already in the cache were rotated under a different s than new ones. Implementations paper over this in various ways, but it complicates clean prefix-cache reuse, which is exactly the optimization you want on long prompts. If you serve mixed traffic, the cheap answer is two deployments: an unscaled one for the short high-QPS path, a YaRN one for the long path, routed on token count.

How do I test whether my RoPE change hurt short context?

Needle tests are not enough — they measure retrieval, not positional resolution, and they only probe the long regime. Bucket your eval by prompt length and report deltas per bucket, with 0-2k as its own line. Then add at least one task that requires ordering rather than presence: reproduce the k-th item from a list, identify which of two near-identical spans came first, or apply numbered rules from a long system prompt in order. Those are the tasks that read the mid-frequency dims you just stretched.

For hosted models — Claude Opus 4.x, Sonnet 4.x, GPT-5.x — none of this is a knob you own. The provider ships one trained configuration. The corollary still applies though: keep prompts as short as the task allows, because you are always somewhere on this trade-off curve, whoever set the dial.

So why does raising rope_theta break short-context accuracy?

Because rope_theta sets a bank of positional wavelengths, λ_i = 2π · base^(2i/d), and raising it stretches the mid and high dims far more than the fast ones. Dimensions that used to resolve position within a few hundred tokens now need thousands, so inside an ordinary prompt they emit a near-constant signal instead of a positional one — at d=128, base 1e4 → 1e6 cuts the dims with sub-2048-token wavelength from 40 to 27. The attention heads were trained against the old allocation and were never retrained for the new one. Fine-tuning at the new base fixes it; an inference-time config edit does not. If you need the length without retraining, use YaRN, which interpolates only the dims that never completed a full rotation during training and leaves the adjacency clocks alone — and turn it on per request rather than globally.

Claude Prompt Caching: Why cache_read_input_tokens Stays 0

jidonglab — Mon, 03 Aug 2026 09:11:03 +0000

A support agent I reviewed ran 40-turn conversations against Claude Opus 4.8 with a 12,000-token system prompt and eight tool definitions. Every request carried cache_control. The team had "enabled prompt caching" three months earlier and moved on.

cache_read_input_tokens was 0 on every single request. Every turn re-billed the full 12K prefix at full price, plus a 1.25x cache-write premium on top. They were paying roughly 25% more than if they had never touched caching at all.

The cause was a single line: f"Current date: {datetime.now():%Y-%m-%d %H:%M}" in the system prompt header. Claude prompt caching is a byte-exact prefix match, and that timestamp sat at byte 200 of a 12,000-token prefix.

TL;DR

Claude prompt caching is a byte-exact prefix match. Render order is tools → system → messages. Any byte change at position N invalidates every cache breakpoint at position ≥ N. A timestamp in your system prompt makes the whole request uncacheable.
Invalidation is tiered, not all-or-nothing. Changing tool_choice, toggling thinking, or adding an image preserves the tools+system cache. Only tool-definition edits and model switches force a full rebuild.
Breakpoints look back at most 20 content blocks. An agentic turn that appends more than 20 tool_use/tool_result blocks silently misses the previous turn's cache. Place an intermediate breakpoint roughly every 15 blocks.
The minimum cacheable prefix is model-dependent and not monotonic — 512 tokens on Claude Opus 5, but 4096 on Opus 4.6 and Haiku 4.5. Below it you get no error, just cache_creation_input_tokens: 0.
Reads cost ~0.1x base input; writes cost 1.25x (5-minute TTL) or 2x (1-hour TTL). Five-minute TTL breaks even at two requests; one-hour needs three.

Why is cache_read_input_tokens zero?

Because something in your prefix changed between requests, and the cache key is derived from the exact rendered bytes up to each breakpoint.

The API renders your request in a fixed order: tools, then system, then messages. A cache_control marker on the last system block therefore caches tools and system together. It does not cache them independently — there is one linear byte stream, and the breakpoint is a position in it.

This makes the design question simple: does your prompt-building code emit stable content strictly before volatile content? Everything else is detail.

Grep your prompt assembly path for these. Each one silently kills caching:

Pattern	Why it breaks
`datetime.now()` / `Date.now()` in system prompt	Prefix differs every request
`uuid4()` or request IDs early in content	Every request is a unique prefix
`json.dumps(d)` without `sort_keys=True`	Non-deterministic key order → different bytes
Iterating a `set` to build tool descriptions	Non-deterministic ordering
`tools=build_tools(user)` varying per user	Tools render at position 0; nothing caches across users
`if flag: system += "..."`	Each flag combination is a distinct prefix

The fix is always the same shape: make it deterministic, move it after the last breakpoint, or delete it.

import json

def build_request(user_msg, session_ctx, tools, history):
    return {
        "model": "claude-opus-5",
        "max_tokens": 16000,
        # Position 0. Sorted, frozen, identical across every user and session.
        "tools": sorted(tools, key=lambda t: t["name"]),
        "system": [
            {
                "type": "text",
                # No f-strings. No dates. No user IDs. Byte-identical forever.
                "text": FROZEN_SYSTEM_PROMPT,
                "cache_control": {"type": "ephemeral"},
            }
        ],
        "messages": [
            *history,
            {
                "role": "user",
                "content": [
                    # Volatile content lives here, AFTER the breakpoint.
                    {"type": "text", "text": f"<context>{json.dumps(session_ctx, sort_keys=True)}</context>"},
                    {"type": "text", "text": user_msg},
                ],
            },
        ],
    }

Verify with the response, not by inspection:

u = response.usage
assert u.cache_read_input_tokens > 0, (
    f"cache miss: read={u.cache_read_input_tokens} "
    f"write={u.cache_creation_input_tokens} uncached={u.input_tokens}"
)

One trap in reading usage: input_tokens is the uncached remainder only. Total prompt size is input_tokens + cache_creation_input_tokens + cache_read_input_tokens. If your agent ran for an hour and input_tokens shows 4K, that is not the whole prompt — check the sum.

What actually invalidates a Claude prompt cache?

Not everything. The API has three cache tiers, and a change only invalidates its own tier and everything below it. This is the part most teams get wrong in the conservative direction — they avoid harmless per-request changes and then break the cache with something structural.

Change	Tools cache	System cache	Messages cache
Tool definitions (add/remove/reorder)	❌	❌	❌
Model switch	❌	❌	❌
`speed`, web-search or citations toggle	✅	❌	❌
System prompt content	✅	❌	❌
`tool_choice`, images, `thinking` on/off	✅	✅	❌
Message content	✅	✅	❌

Two practical consequences:

You can flip tool_choice per request or toggle thinking without losing the tools+system cache. Don't build elaborate machinery to keep those stable.

You cannot swap the tool set for "modes." A mode switch that adds three tools and removes two invalidates position 0 and re-bills the entire conversation. If you need dynamic capability, use tool search (schemas are appended, preserving the prefix) rather than rebuilding tools.

The same logic applies to sub-agents and compaction calls. A fork that rebuilds system/tools/model with any difference from its parent misses the parent's cache entirely. Copy the parent's three fields verbatim and append fork-specific content at the end.

Two of these rows have an escape hatch, and they're gated separately. For system-prompt changes: on Claude Opus 5, Opus 4.8, Fable 5, and Mythos 5, append a {"role": "system", "content": "..."} message to messages[] instead of editing top-level system. No beta header required. The instruction lands after the cached history, so the prefix survives, and unlike a <system-reminder> stuffed into a user turn, it's a non-spoofable operator channel.

messages = [
    *history,
    {"role": "user", "content": user_msg},
    {"role": "system", "content": "Terse mode enabled — keep responses under 40 words."},
]

Constraints: it must follow a user message (or an assistant message ending in server-tool use), it can't be messages[0], and it must be last or followed by an assistant turn. Unsupported models — including Claude Sonnet 5 — return a 400; catch it and fall back to a user-turn reminder block.

Why does my agent lose the cache after a long tool-calling turn?

Because each breakpoint walks backward at most 20 content blocks looking for a prior cache entry. Beyond that, it stops looking and writes a fresh entry.

This is the failure mode that survives every other fix, and it's specific to agentic loops. A single assistant turn that fires twelve parallel tool calls produces twelve tool_use blocks plus twelve tool_result blocks — 24 blocks in one turn. The next request's breakpoint scans back 20, never reaches the previously cached position, and misses. No error, no warning, just a cache-write charge where you expected a read.

The fix is cheap: place an intermediate breakpoint roughly every 15 blocks in long turns. You get four breakpoints per request, so budget them — one on the frozen tools+system prefix, and up to three floating through recent history.

Why doesn't a 3,000-token prompt cache at all?

Because the minimum cacheable prefix is model-dependent, and it isn't monotonic across generations:

Model	Minimum prefix
Claude Opus 5, Fable 5, Mythos 5	512 tokens
Opus 4.8, Sonnet 5, Sonnet 4.6, Sonnet 4.5	1024 tokens
Opus 4.7, Haiku 3.5	2048 tokens
Opus 4.6, Opus 4.5, Haiku 4.5	4096 tokens

A 3,000-token prompt caches fine on Claude Opus 5, Opus 4.8, and Sonnet 5 — and silently doesn't on Opus 4.6 or Haiku 4.5. There's no error. cache_creation_input_tokens comes back 0 and you conclude caching is broken.

Claude Opus 5 halved the Opus 4.8 minimum from 1024 to 512, so prompts you previously wrote off as uncacheable are worth re-checking with no code change. These minimums now apply on every platform where the model is available — the old Bedrock override for Fable 5 was removed.

Should I use the 1-hour TTL?

Only if traffic is bursty with gaps longer than five minutes. The doubled write cost is real.

Cache reads cost about 0.1x the base input price. Cache writes cost 1.25x for the default 5-minute TTL and 2x for the 1-hour TTL ({"type": "ephemeral", "ttl": "1h"}).

Run the break-even:

5-minute TTL: 1.25x write + 0.1x read = 1.35x for two requests, versus 2.0x uncached. Profitable at request two.
1-hour TTL: 2.0x write + 0.1x + 0.1x = 2.2x for three requests, versus 3.0x uncached. Needs three.

If requests arrive more often than every five minutes, the default TTL is kept warm by real traffic and the 1-hour option is pure overpayment. Reach for it when you have long idle gaps you'd otherwise re-warm across.

On pre-warming: send a max_tokens: 0 request at startup. The API runs prefill, writes the cache at your breakpoint, and returns immediately with content: [], stop_reason: "max_tokens", and zero output tokens billed. Put the cache_control on the block shared with the real request — the system prompt — not on the placeholder user message, and not via top-level auto-caching, which would key the entry to the placeholder. It's rejected with stream: true, thinking.type: "enabled", output_config.format, forced tool_choice, and inside Batches.

Skip pre-warming when traffic is continuous, the prefix is small, or you'd be speculatively warming many distinct prefixes at 1.25x each.

Why do parallel requests all miss the cache?

Because a cache entry becomes readable only once the first response begins streaming. Fire N identical-prefix requests simultaneously and all N pay full price — none can read what the others are still writing.

For fan-out over a shared document or shared few-shot block: send one request, await the first streamed token (not the full response), then release the remaining N−1. They'll read the entry the first one just wrote. On a 50K-token shared prefix with 20 parallel branches, that ordering is the difference between 20 full-price prefills and one.

The short answer

cache_read_input_tokens stays at zero because Claude prompt caching matches the exact bytes of your rendered prefix — tools, then system, then messages — and a single differing byte at position N invalidates every breakpoint after it. In practice the culprit is one of five things: volatile content (a timestamp, UUID, or unsorted JSON) sitting ahead of your breakpoint; a tool set or model that changes mid-conversation; an agentic turn appending more than 20 content blocks and blowing past the lookback window; a prefix below the model's minimum (512 to 4096 tokens depending on the model); or parallel requests racing to write the same entry. Diff the rendered prompt bytes between two consecutive requests and the answer shows up in seconds. Then assert on cache_read_input_tokens in your integration tests, because this failure is silent and it costs 1.35x forever.

Attention Sinks: Why Evicting Token 0 Wrecks Sliding-Window KV

jidonglab — Sun, 02 Aug 2026 21:07:51 +0000

You ship a KV cache eviction policy: keep the last 4k tokens, drop the rest. Perplexity on your 3k-token eval set is unchanged. You roll it out, and long sessions start producing fluent nonsense — grammatical, on-topic-ish, completely detached from the conversation. The degradation begins exactly when the sliding window passes the start of the sequence.

You didn't hit a context-length limit. You evicted the model's attention sinks, and the softmax had nowhere to put the mass it is mathematically obligated to spend.

TL;DR

Softmax attention weights must sum to 1, so a head with nothing relevant to read still has to distribute full attention mass somewhere. Trained transformers dump that mass on the first few tokens, whose value vectors have near-zero norm — a learned no-op.
Drop those tokens from the KV cache and the mass gets redistributed onto real tokens with real value vectors, injecting spurious content into the residual stream. Perplexity blows up within a few hundred tokens.
Keeping just 4 initial tokens plus a rolling window restores stable streaming quality (StreamingLLM, Xiao et al. 2023). The fix is cheap; the failure is catastrophic.
Position IDs must be assigned by cache index, not original text index. Get this wrong and RoPE sees a hole where the evicted tokens were.
Sink tokens carry massive activations, so they also break per-tensor KV quantization, H2O-style importance eviction, and any agent loop that trims the top of its own transcript.

What are attention sinks in a transformer?

An attention sink is a token position that soaks up a large fraction of attention probability across many heads and layers, regardless of what the query is asking about. In practice it's the first token of the sequence — BOS if you use one, or whatever token happens to be at index 0 — plus the two or three positions after it.

This is not a bug in training. It's the only way a softmax head can express "nothing here is relevant."

Consider one head, one query. Scores s_i = q·k_i / sqrt(d), weights a = softmax(s), output o = Σ a_i v_i. There is no a_i = 0 for all i state. The head must spend its full unit of probability mass. If the honest answer for this query is "skip this operation," the head needs a target whose value vector is approximately zero — a token it can attend to hard while contributing nothing to the residual stream.

Trained models converge on the same solution: make v_0 ≈ 0, make k_0 easy to score highly, and route all idle mass there. Measure it and you'll see heads in mid-to-late layers putting the majority of their attention on the first handful of positions on most queries.

Why does dropping token 0 break sliding-window attention?

Because the remaining weights get rescaled by 1 / (1 - a_sink), and everything left in the window has a non-trivial value vector.

Say a head puts 0.80 of its mass on the sink and spreads 0.20 over 4,000 real tokens. Evict the sink and renormalize: every real token's weight is multiplied by 5x. The head that was supposed to be a no-op now writes a 5x-amplified average of whatever happens to sit in the window into the residual stream.

That output feeds the next layer's queries and keys. The corruption compounds depth-wise, then step-wise as the poisoned states get cached. The output stays locally fluent — the language modeling head still produces well-formed tokens — while the actual retrieval and instruction-following signal drowns.

The tell is that quality collapses at a position, not at a length. A 40k-token session with a 4k window fails; a 4k-token session with a 4k window is fine. Same model, same window, same prompt style.

Why do sinks land on the first tokens specifically?

Causal masking. Position 0 is the only position visible to every subsequent query in every layer. If the model needs a globally reachable dump site, that's the only candidate that always exists.

Two consequences follow:

Sinks are positional, not semantic. Cut the first sentence off your prompt and the new first token becomes the sink. The model doesn't care that it's "The" instead of <|begin_of_text|> — but the KV entry it learned to lean on is gone, and a fresh prefill has to rebuild the role from a token that wasn't trained for it. Fresh prefill mostly recovers; mid-stream eviction does not, because the surviving cache entries were computed against the original sink.
Sinks carry massive activations. The hidden states at sink positions have norms orders of magnitude above the rest of the sequence. This is the same phenomenon that forces outlier handling in activation quantization, and it means a sink token is the single most expensive thing in your cache to quantize naively.

How do you evict KV cache without losing the sink?

Reserve the first N entries permanently, roll the rest, and re-derive positions from cache slot index. N = 4 is the standard choice.

class SinkKVCache:
    """Rolling KV cache that pins the first `n_sink` tokens.

    Positions are re-derived from cache slot index, not the original
    text index. RoPE must see a contiguous 0..len-1 run, otherwise the
    evicted span shows up as a gap in relative distance.
    """

    def __init__(self, n_sink=4, window=4096):
        self.n_sink = n_sink
        self.window = window
        self.k = None   # [B, H_kv, S, D]
        self.v = None

    def append(self, k_new, v_new):
        self.k = k_new if self.k is None else torch.cat([self.k, k_new], dim=2)
        self.v = v_new if self.v is None else torch.cat([self.v, v_new], dim=2)

        cap = self.n_sink + self.window
        s = self.k.shape[2]
        if s > cap:
            keep_tail = cap - self.n_sink
            self.k = torch.cat([self.k[:, :, :self.n_sink], self.k[:, :, -keep_tail:]], dim=2)
            self.v = torch.cat([self.v[:, :, :self.n_sink], self.v[:, :, -keep_tail:]], dim=2)
        return self.k, self.v

    def query_position(self):
        # The next query's position = current cache length, NOT the
        # number of tokens actually generated so far.
        return self.k.shape[2]

The query_position detail is the part people get wrong. If you evicted 30k tokens and keep using absolute text positions, RoPE rotates the incoming query as if it were 34k positions away from the sink while the surviving neighbors sit at 30k+. Every relative distance in the window is now wrong, and you're extrapolating far past the trained range for no reason.

Two other requirements that bite in real serving code:

Cache keys with rotation already applied are not relocatable. If you store post-RoPE keys (most implementations do), you cannot re-index them after eviction without un-rotating. Either store pre-rotation keys for the tail, or accept that only contiguous suffix eviction is safe. Pinning a prefix plus a contiguous suffix satisfies this; arbitrary token-level eviction does not.
Paged/block allocators need a pinned block. In a block-based KV allocator, the sink lives in block 0. Make it non-evictable explicitly, or your LRU policy will reclaim it precisely because nothing "recently wrote" to it.

How do I check whether my model has attention sinks?

Read the attention weights directly. One forward pass with output_attentions=True is enough to see the structure.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

name = "meta-llama/Llama-3.1-8B"
tok = AutoTokenizer.from_pretrained(name)
model = AutoModelForCausalLM.from_pretrained(
    name, torch_dtype=torch.bfloat16, device_map="auto",
    attn_implementation="eager",  # required to get attention probs back
)

ids = tok("The mitochondrion is the powerhouse of the cell. " * 40,
          return_tensors="pt").to(model.device)
with torch.no_grad():
    out = model(**ids, output_attentions=True)

for layer_idx in (0, 8, 16, 24, 31):
    attn = out.attentions[layer_idx][0]       # [H, S, S]
    last_q = attn[:, -1, :]                   # final query row per head
    sink_mass = last_q[:, :4].sum(-1)         # mass on first 4 positions
    print(f"L{layer_idx:2d}  median sink mass {sink_mass.median():.3f}  "
          f"max {sink_mass.max():.3f}  heads>0.5: {(sink_mass > 0.5).sum().item()}")

Layer 0 is usually clean. From roughly the second layer onward you'll see a large share of heads with most of their final-row mass parked on the first four positions, on a prompt with zero informational reason to look there. That's your sink budget. Also print the value-vector norms at those positions — they'll be conspicuously small relative to the sequence median, which is the "no-op" half of the mechanism.

Where else do attention sinks bite?

Importance-based eviction. H2O-style "heavy hitter" policies score tokens by accumulated attention and usually retain the sink for free, since it's the heaviest hitter by construction. Policies that score by recency, semantic salience, or embedding similarity will happily throw it away. If you built a custom eviction heuristic, check that the sink survives it.
KV cache quantization. Sink keys and values sit far outside the distribution of the rest of the cache. Per-tensor scales get dragged toward the outlier and everything else loses precision. Keep the pinned sink entries in fp16/bf16 — it's 4 tokens, the memory cost rounds to zero.
Trained-in sliding windows are different. Models trained with a rolling buffer from the start (Mistral-style sliding window attention, or interleaved local/global layer stacks) learned to operate without a permanently visible token 0 in their local layers. Retrofitting a window onto a model trained with full attention is where this fails. Don't assume the architecture rescues you; check the training config.
Learned sink logits. Several recent open-weight models, including OpenAI's gpt-oss family, add a learned per-head scalar to the softmax denominator — an explicit "attend to nothing" slot that costs no KV entry. If your model has it, the head has a real no-op and the positional-sink pressure drops. Make sure your inference stack actually implements that extra denominator term; silently ignoring it renormalizes every attention distribution in the model.
Hosted APIs are not exposed to this. With Claude Opus 4.x or GPT-5.x you don't own the KV cache, and every request re-prefills whatever prefix you send, so a token 0 always exists. The analogous cost when you trim the top of an agent transcript is prefix-cache invalidation and lost instructions, not sink loss. Different problem, different fix — don't apply this article's reasoning to your API context-window trimmer.

So why does evicting token 0 wreck sliding-window attention?

Because softmax attention weights are forced to sum to 1, and trained transformers solve the "nothing here is relevant" case by dumping that mandatory mass onto the first few tokens, whose value vectors are near zero. Those positions are attention sinks: high attention, no content. Evict them mid-stream and the same mass is redistributed onto real tokens with real value vectors — scaled up by 1/(1 - a_sink), often a 5x amplification — which writes garbage into the residual stream and compounds across layers and decode steps. Pin the first 4 KV entries, roll the rest, assign positions by cache slot index rather than original text index, and keep those pinned entries out of your quantizer. The fix costs 4 tokens of cache; skipping it costs the whole session.

Why temperature=0 Still Gives Different Answers: Batch Invariance

jidonglab — Sun, 02 Aug 2026 09:05:45 +0000

Set temperature=0, send the same prompt twice, get two different answers. Most engineers blame "GPU nondeterminism" and move on. That explanation is wrong, and being wrong about it means you'll never fix it.

The real cause is batch invariance — or rather, the lack of it. Your request's logits depend on what other requests happened to be in the same batch, because the kernel picked a different reduction order for a different batch shape. Nothing about your request changed. The floating-point arithmetic did.

This is now visible at the API layer too: on Claude Opus 5, Sonnet 5, and Opus 4.7/4.8, temperature is no longer an accepted parameter at all — sending it returns a 400. Anthropic's own migration guidance says the quiet part out loud: temperature = 0 never guaranteed identical outputs on prior models either.

TL;DR

Greedy decoding is deterministic; your serving stack is not. argmax over identical logits always returns the same token. The logits aren't identical run to run.
Floating-point addition is non-associative. (a+b)+c != a+(b+c) in fp16/bf16. Every matmul, RMSNorm, and softmax is a reduction, and the summation order is chosen by kernel heuristics keyed to batch shape.
Continuous batching makes batch shape depend on server load. Your request gets batched with whatever else arrived that millisecond, so the reduction tree — and the last bits of every logit — changes with unrelated traffic.
A 1e-6 logit perturbation is harmless until two tokens are nearly tied. Then argmax flips, and autoregressive feedback amplifies one flipped token into a completely different answer.
The fix is batch-invariant kernels (fixed split sizes, no atomics, one config per op) at a real throughput cost — or, for hosted APIs, accepting nondeterminism and testing semantically instead of byte-wise.

Why does temperature=0 still produce different outputs?

Because "deterministic sampling" and "deterministic inference" are different claims. temperature=0 makes token selection a pure function of the logit vector. It says nothing about whether the logit vector is reproducible.

Consider what the forward pass actually is: a very long chain of reductions. A single logit is a dot product over the hidden dimension. RMSNorm sums squares across the row. Attention normalizes over the KV sequence. Every one of these sums thousands of floating-point values.

And floating-point addition is not associative:

import numpy as np

x = (np.random.randn(4096) * 0.02).astype(np.float32)

seq   = np.float32(0.0)
for v in x: seq += v                     # strictly sequential

tree4 = x.reshape(4, -1).sum(axis=1, dtype=np.float32).sum(dtype=np.float32)
tree8 = x.reshape(8, -1).sum(axis=1, dtype=np.float32).sum(dtype=np.float32)

print(f"{seq:.10f}  {tree4:.10f}  {tree8:.10f}")
print("4-way vs 8-way differ:", tree4 != tree8)

Same numbers, same values, three answers differing in the last few bits. The only thing that changed was how the sum was partitioned. That partition is exactly what a GPU kernel picks at launch time.

What is batch invariance in LLM inference?

A kernel is batch-invariant if the result computed for a given row is bitwise identical regardless of what other rows are in the batch. Most production LLM kernels are not.

The reason is performance tuning. To saturate a GPU, a matmul kernel splits work across streaming multiprocessors. When the batch is small, there aren't enough output tiles to fill the device, so the kernel splits along the reduction dimension instead (split-K) and combines partial sums afterward. Larger batches don't need the split. Which strategy runs — and how many splits — is chosen by a heuristic or an autotuner reading the tensor shapes.

Same for attention decode kernels: FlashDecoding-style implementations partition the KV cache into chunks, run online softmax per chunk, then rescale and merge. The number of KV splits is commonly derived from batch size, head count, and sequence length so the kernel fills all SMs. Different batch → different split count → different merge order → different last bits.

Note what's not on this list: race conditions. Individual kernels are typically run-to-run deterministic for fixed inputs and fixed launch config. Atomics-based split-K reductions are a real second source of nondeterminism, but they're the smaller problem. The dominant one is that your batch composition is set by other people's traffic.

Additional shape-dependent sources worth knowing about:

Prefix caching. A cache hit reuses KV computed under a different chunking than a cache miss recomputes. Same prompt, different numerics, before you've sampled a single token.
Chunked prefill boundaries. Where a long prompt gets sliced depends on scheduler state.
Tensor parallelism degree. A different TP size means a different all-reduce tree.
MoE routing. Expert assignment and per-expert batching shift with batch composition, which is why MoE models feel noticeably flakier than dense ones.

How does a 1e-6 logit difference change the whole answer?

It usually doesn't. Argmax is robust when the top-1 margin is comfortable. The failure mode is narrow but structural.

Define the margin at step t as logit[top1] - logit[top2]. Numerical noise on the order of 1e-6 relative flips the decision only when the margin falls below that noise floor. On confident tokens — punctuation, the second half of a word, a memorized fact — the margin is enormous. On genuinely uncertain tokens — a synonym choice, a formatting decision, whether to open a code block — the margin can be near zero.

Two properties make this bite in production:

Ties are not rare over long outputs. A per-token flip probability of even 0.1% compounds to roughly a 40% chance of at least one divergence across 500 tokens.
Divergence is absorbing. One different token becomes part of the context for every subsequent token. There's no reconvergence mechanism. A single flipped connective can send an agent down a different tool call.

That's the whole story: a last-bit arithmetic difference, amplified by argmax at a near-tie, amplified again by autoregressive feedback.

How do I measure nondeterminism in my own stack?

Don't argue about it — quantify it. The metric you want is divergence rate under load, plus the index of first divergence. This probe works against any endpoint, hosted or self-hosted:

import asyncio, hashlib
from collections import Counter

async def one(client, prompt, model):
    r = await client.messages.create(
        model=model, max_tokens=512,
        messages=[{"role": "user", "content": prompt}],
    )
    return "".join(b.text for b in r.content if b.type == "text")

async def probe(client, prompt, model, n=64, concurrency=16):
    sem = asyncio.Semaphore(concurrency)
    async def guarded():
        async with sem:
            return await one(client, prompt, model)
    outs = await asyncio.gather(*(guarded() for _ in range(n)))

    counts = Counter(hashlib.sha256(o.encode()).hexdigest()[:12] for o in outs)
    base = outs[0]
    firsts = [
        next((i for i, (a, b) in enumerate(zip(base, o)) if a != b), None)
        for o in outs[1:]
    ]
    firsts = [f for f in firsts if f is not None]

    print(f"unique outputs : {len(counts)}/{n}")
    print(f"modal share    : {counts.most_common(1)[0][1] / n:.1%}")
    if firsts:
        firsts.sort()
        print(f"first divergence char (min/median): {firsts[0]} / {firsts[len(firsts)//2]}")

Run it twice: once at concurrency=1, once at concurrency=32. If unique-output count climbs with concurrency, you have a batch-invariance problem, not a sampling problem. On a self-hosted stack, also run it with prefix caching on and off — the delta tells you how much of your instability is cache-boundary numerics.

How do I make LLM inference reproducible?

If you control the server, you have three tiers of fix, in increasing order of cost.

Tier 1 — pin the shape. Batch size 1, one replica, fixed TP degree, prefix caching off, CUDA graphs off. This is what your reproducibility test suite should run against, not production.

vllm serve <model> \
  --max-num-seqs 1 \
  --tensor-parallel-size 1 \
  --no-enable-prefix-caching \
  --enforce-eager

Throughput will be terrible. That's fine — this configuration exists to make regression diffs meaningful, not to serve users.

Tier 2 — batch-invariant kernels. Replace the shape-sensitive ops with versions whose reduction structure is fixed. The core rule is to make the split a function of a constant chunk size rather than a target split count: process the reduction dimension in fixed 256-element chunks and combine in a fixed order, so a row's reduction tree is identical whether it ships alone or with 63 neighbors. Concretely that means data-parallel RMSNorm (one block per row, no cross-block reduce), a single matmul tile configuration with no atomic split-K, and an attention kernel with a fixed KV chunk size that treats cached and freshly-computed KV identically.

Thinking Machines published this analysis and a batch_invariant_ops library in late 2025; vLLM has since grown a batch-invariant mode behind an environment flag (check your build — the flag name and coverage have moved). The honest tradeoff: the naive deterministic path was substantially slower, and an optimized attention kernel recovered much of the gap without fully closing it. You are trading throughput for bitwise reproducibility. Buy it deliberately, for eval and RL-rollout paths, not for your whole fleet.

Tier 3 — accept it, and change what you assert. Which brings us to hosted APIs.

What about hosted APIs like Claude and GPT-5?

You have zero control over batch composition on a hosted endpoint, so bitwise reproducibility is not on the menu. The API surface now reflects this. Frontier Claude models (Opus 5, Sonnet 5, Opus 4.8/4.7) reject temperature, top_p, and top_k outright — the documented guidance is to steer behavior with prompting instead. Anthropic's API has never exposed a seed. OpenAI's seed plus system_fingerprint is explicitly best-effort, and the fingerprint exists precisely to tell you when the backend changed underneath you.

So stop writing tests that hash the completion. Write these instead:

Structured output, field-level assertions. Constrain the response to a schema and assert on the fields that matter. Prose varies; {"risk": "high"} shouldn't.
Divergence rate as an SLO, not a bug. Measure it, set a threshold, alert when it moves. A jump in divergence rate is a genuine signal that something changed on the provider side.
Never cache on output hash. Key caches on the request, not the response.
Budget for it in evals. Run n samples per item and report a confidence interval. A 1-point accuracy difference between two prompts, measured at n=1, is noise.

The short answer

temperature=0 isn't deterministic because greedy decoding only guarantees the same token for the same logits — and your logits are not reproducible. Floating-point addition is non-associative, GPU kernels choose their reduction order from batch shape, and continuous batching makes that shape depend on unrelated concurrent traffic. The result is a last-bit perturbation that flips argmax whenever two candidate tokens are nearly tied, and autoregressive decoding turns one flipped token into a completely different response. Fixing it requires batch-invariant kernels with fixed reduction structure, at a real throughput cost — an investment that makes sense for evals, regression tests, and RL rollouts. Everywhere else, and on every hosted API, the right move is to measure divergence rate and assert on meaning instead of bytes.

Speculative Decoding: Why 80% Acceptance Still Loses at Batch 64

jidonglab — Sat, 01 Aug 2026 21:03:06 +0000

A 1B draft in front of a 70B target. 80% token acceptance. Four speculative tokens per step. Your dashboard says the draft model is doing its job. Then you raise concurrency from 8 to 64 and throughput drops below the non-speculative baseline you were trying to beat.

Nothing broke. Speculative decoding is a latency optimization that you pay for in FLOPs, and at batch 64 you ran out of free FLOPs to spend. Acceptance rate was never the number that decided this.

TL;DR

Acceptance rate is exactly 1 - TV(p, q), the total variation distance between target and draft distributions at that position. It is not "how often the draft is right."
Expected tokens per step is a geometric sum, not γ × α. At α = 0.8 and γ = 4 you get ~3.36 tokens per verify step, not 3.2 accepted plus wishful thinking.
Per-position acceptance decays because the draft conditions on its own drift. The 5th draft token is worth ~0.1 tokens of expected output while costing a full draft forward pass. γ = 3–4 is usually the peak.
Speculative decoding trades memory bandwidth for compute. It wins only while decode is memory-bound. Batch B with γ speculative tokens puts B × (γ+1) tokens in flight; past roughly 300 on an H100 you are compute-bound and rejected tokens become pure waste.
Measure tokens_out / target_forward_FLOPs, not acceptance. If that ratio drops, turn it off for that traffic shape.

If you call Claude Opus 4.5 or GPT-5.x over an API, this is your provider's problem. If you run vLLM, SGLang, or TensorRT-LLM yourself, it's yours.

What is the speculative decoding acceptance rate, exactly?

The acceptance rate is 1 - TV(p, q) where p is the target distribution and q is the draft distribution at the same position. That identity is exact, not an approximation.

The standard scheme (Leviathan et al., Chen et al., 2023) samples x ~ q(x) and accepts with probability min(1, p(x)/q(x)). Integrate over the draft's own sampling:

P(accept) = Σ_x q(x) · min(1, p(x)/q(x))
          = Σ_x min(q(x), p(x))
          = 1 - TV(p, q)

On rejection you resample from the normalized residual max(0, p - q). That correction is what makes the output distribution identical to the target's — speculative decoding is not an approximation, and you should refuse any implementation that claims a speed win by relaxing it.

Two consequences people get wrong:

Greedy is a special case, not the easy case. At temperature 0 both distributions collapse to one-hot, so acceptance equals top-1 agreement. Nothing about the tail matters. At T = 1 you are matching the whole distribution, and a draft that agrees on argmax 85% of the time can still have TV = 0.35. Acceptance is not monotone in temperature. Measure it per temperature; don't extrapolate from your greedy eval.

Sampler mismatch silently destroys both properties. If your stack applies top-p to the target but not the draft, or applies them in a different order relative to temperature, you get a different q than the acceptance test assumes. Acceptance falls, and the distributional guarantee is gone. This is the single most common real bug I see in hand-rolled speculative loops.

Why doesn't 80% acceptance mean a 5x speedup?

Because a run of accepted tokens terminates on the first rejection, so expected output per step is a geometric sum, and because the draft passes aren't free.

Let α_i be acceptance at draft position i. The target's verify pass always emits at least one token: either the correction at the first rejection, or the bonus token from the target's final position when all γ drafts are accepted. So:

def expected_tokens(alphas):
    """E[tokens emitted per verify step]. The leading 1.0 is the token the
    target always produces: correction on rejection, or bonus on full accept."""
    e, run = 1.0, 1.0
    for a in alphas:
        run *= a
        e += run
    return e

def speedup(alphas, c):
    """c = draft forward cost / target forward cost, in the memory-bound regime."""
    return expected_tokens(alphas) / (len(alphas) * c + 1.0)

With uniform α = 0.8, γ = 4: 1 + .8 + .64 + .512 + .4096 = 3.36 tokens per step. With a draft costing 10% of a target pass, the step costs 1.4 target-equivalents. Net: 2.40x, not 5x.

Forgetting the bonus token is a real regression, not a rounding error. It costs you a full token on every fully-accepted block — at α = 0.8, γ = 4 that is 41% of blocks and roughly 12% of total throughput.

Why does per-position acceptance decay make γ = 5 pointless?

Because the draft conditions on tokens it generated itself, so it drifts further from the target with each speculative step. Acceptance is a decaying sequence, and the marginal token requires every prior one to be accepted.

A realistic measured profile for a small draft against a large target looks like [0.80, 0.72, 0.66, 0.60, 0.55]. Run the model:

γ	E[tokens]	step cost (c=0.1)	speedup
2	2.376	1.2	1.98x
3	2.756	1.3	2.12x
4	2.984	1.4	2.13x
5	3.109	1.5	2.07x

The 4th draft token contributes 0.8 × 0.72 × 0.66 × 0.60 = 0.228 expected tokens for 0.1 target-equivalents of cost. The 5th contributes 0.125 and loses money. γ = 4 is the peak and γ = 3 is within noise of it, which means the cheaper config is the better config once you account for the extra KV memory the longer draft window holds.

One genuinely good piece of news: acceptance is bursty, not i.i.d. Long runs of easy tokens (JSON scaffolding, indentation, copied spans from the prompt) alternate with hard, high-entropy decisions. Σ α^i is convex in α, so by Jensen's inequality a bursty α with mean 0.7 beats a flat α of 0.7. Your average acceptance understates your real throughput. Log the distribution of accepted-run lengths, not the mean.

Why does speculative decoding lose at batch 64?

Because it converts a memory-bandwidth problem into a compute problem, and at batch 64 you no longer have spare compute.

Single-token decode is bandwidth-bound. You stream every weight from HBM to produce one token per sequence. Verifying γ+1 tokens in one forward pass reads the same weights, so the extra positions are nearly free. That is the entire trick — not "the draft is smart," but "the target's verify pass was already idle on FLOPs."

Now the roofline. An H100 SXM does roughly 990 TFLOP/s dense BF16 against about 3.35 TB/s of HBM3, so the crossover is near 300 FLOP per byte. A BF16 GEMM with B rows does about B FLOP per weight byte. You need on the order of 300 tokens in flight to saturate compute.

Speculative decoding multiplies tokens in flight by γ+1:

Batch 8, γ = 4 → 40 tokens in flight. Deep in the bandwidth-bound regime. Free lunch.
Batch 64, γ = 4 → 320 tokens in flight. Past the crossover. Verification now costs ~5x the FLOPs of a plain decode step, and at α = 0.8 roughly a third of those positions are discarded.

Continuous batching already gave you the throughput win that speculation was faking. Stacking them means paying twice for the same thing. GQA, MoE routing, and FP8 all move the crossover, so measure yours — but the shape holds everywhere.

The practical rule: speculative decoding is for low-concurrency, latency-sensitive traffic. Interactive coding agents, single-user local inference, tail-latency SLOs. It is not for batch summarization jobs.

How do you configure this in vLLM?

Keep the draft small enough that c stays under ~0.15, and gate speculation on concurrency.

from vllm import LLM

# Draft-model speculation: good for low-concurrency interactive serving.
llm = LLM(
    model="meta-llama/Llama-3.1-70B-Instruct",
    tensor_parallel_size=4,
    max_num_seqs=16,                 # keep tokens-in-flight under the roofline
    speculative_config={
        "model": "meta-llama/Llama-3.2-1B-Instruct",
        "num_speculative_tokens": 3,  # peak, not max
    },
)

# N-gram speculation: no draft model, no extra weights, no extra KV.
llm = LLM(
    model="meta-llama/Llama-3.1-70B-Instruct",
    speculative_config={
        "method": "ngram",
        "num_speculative_tokens": 5,
        "prompt_lookup_max": 4,
    },
)

Flag names have moved across vLLM versions (the old --speculative-model family folded into speculative_config); check yours. What matters is that max_num_seqs × (num_speculative_tokens + 1) stays on the bandwidth-bound side of your hardware's crossover.

When should you use n-gram speculation instead of a draft model?

When the output copies substantially from the input. Then the best draft model is no model at all.

N-gram (prompt-lookup) speculation matches the last few generated tokens against the prompt and proposes the continuation verbatim. Zero draft parameters, zero draft KV, c ≈ 0. On workloads where the output quotes the input, acceptance runs long and the cost model has no denominator to speak of:

RAG answers that quote retrieved passages
Code edits that reproduce most of the original file
Structured extraction echoing field values from a document
Any "rewrite this with X changed" task

On free-form generation it does nothing, and because c ≈ 0 it costs nothing when it fails. That asymmetry makes it the default I'd reach for first. Only when it demonstrably doesn't fire do I add a draft model.

EAGLE-style methods sit between the two: they draft in the target's feature space and verify a tree of candidate branches instead of a chain, which raises acceptance for a given γ. Note the cost-model implication — tree verification puts even more tokens in flight per step, so it hits the compute roofline at a lower batch size than chain speculation does. Higher acceptance, narrower operating window.

Failure modes worth a runbook entry

KV rollback off-by-one. Rejected positions leave KV entries that must be truncated before the next step. Off-by-one here produces fluent, subtly wrong output that no unit test catches.
Draft/target tokenizer mismatch. Different vocabularies make p(x)/q(x) meaningless. Use a draft from the same family, or do explicit vocab mapping.
Acceptance measured on the wrong traffic. Chat evals overstate acceptance for agentic tool-call traffic, where outputs are short, schema-constrained, and high-entropy at exactly the tokens that matter.
TTFT regression. Draft weights and draft KV shrink the KV budget for real sequences, cutting how many requests fit and pushing queueing delay up. Watch TTFT, not just inter-token latency.

So does speculative decoding actually help?

Speculative decoding helps when decode is memory-bound and hurts when it is compute-bound, and acceptance rate tells you almost nothing about which side you're on. The acceptance rate is exactly 1 - TV(p, q); expected output per verify step is the geometric sum 1 + Σ Π α_i, which caps out near γ = 3–4 once per-position decay is accounted for; and the real speedup is that sum divided by γc + 1. The decisive variable is tokens in flight: batch × (γ+1). Below your hardware's roofline crossover — roughly 300 on an H100 — the extra verified positions ride along for free and you get a genuine 2–2.5x on inter-token latency. Above it, you are spending real FLOPs on tokens you will throw away, and 80% acceptance at batch 64 loses to plain continuous batching. Run speculation on your low-concurrency latency tier, use n-gram speculation on copy-heavy workloads where it's nearly free, and instrument tokens_out / target_forward_FLOPs so the regression shows up before your throughput graph does.

Why JSON Schema Field Order Breaks Structured Output Accuracy

jidonglab — Sat, 01 Aug 2026 09:00:10 +0000

Someone on your team reorders a Pydantic model so the API response reads better — label first, rationale last. No prompt change, no model change, no temperature change. The classification eval drops. The rationales still look great; they just now describe a decision the model had already made.

That is not a coincidence and it is not prompt superstition. With strict structured outputs, JSON Schema field order is a decoding constraint. The grammar compiler turns your property order into a finite-state machine, and that machine assigns probability zero to any token that would start the wrong key. The model cannot reorder. It cannot think first and answer second unless your schema lets it.

TL;DR

JSON Schema field order determines generation order under constrained decoding (OpenAI strict structured outputs, vLLM guided_json, Outlines/XGrammar/llguidance). Objects are emitted in declaration order because tracking arbitrary property order needs 2^k FSM states for k required keys.
Autoregressive models compute in the token stream. A field declared before the answer is scratchpad; a field declared after it is post-hoc rationalization that cannot influence the answer at all.
Put evidence → rationale → label → confidence. Never label first. The only cost is time-to-first-useful-token, not accuracy.
Forced key tokens are free prompt real estate: step_by_step_reasoning injects that phrase into the KV cache right before the model writes the value. Their logprobs are meaningless — don't compute confidence over forced spans.
Design enum values to differ at the first token (urgent/normal/low, not p0/p1/p2) so the masked, renormalized logprobs at that position give you a usable class posterior.

Why does JSON Schema field order change structured output accuracy?

Because a decoder-only model has exactly one place to do intermediate computation: the tokens it has already emitted. There is no hidden scratch buffer that persists across steps. Everything the model "works out" has to exist as tokens in the prefix, or it does not exist.

So when the schema forces this:

{"label": "p0", "rationale": "The stack trace shows ..."}

the token for p0 is sampled from a context that contains the ticket, the system prompt, and the seven characters {"label":. That is it. Whatever the rationale says afterwards was conditioned on p0, not the other way around. You did not get reasoning. You got a language model doing what it is extremely good at — writing a persuasive defense of a position already on the page.

Flip the order and the same tokens become causal:

{"evidence": ["OOM at 03:12 on the payments pod"], "rationale": "...", "label": "p0"}

Now every token of evidence and rationale is in the prefix that conditions label. This is chain-of-thought, just wearing a JSON hat.

The part people miss: with an unconstrained model you might get away with a bad order, because a strong model will sometimes ignore your schema and emit keys in the order it prefers. Strict mode removes that escape hatch. The constraint is enforced at the logit level.

How does constrained decoding actually enforce the order?

The schema is compiled into a grammar, the grammar into a state machine over the token vocabulary. At each step the engine computes the set of tokens that keep the output on a valid path, masks everything else to -inf, and renormalizes:

# What every guided-decoding backend does, stripped down.
mask = fsm.allowed_token_mask(state)     # bool[vocab_size]
logits[~mask] = float("-inf")
probs = softmax(logits)                  # renormalized over the legal set only
tok = sample(probs)
state = fsm.advance(state, tok)

Consider the step right after {. The model's unconstrained top token might be "reason with high probability — it wants to think first. If your schema declared label first, that token is masked. The mass gets redistributed over legal tokens, and the model emits "label. You never see the preference; you only see the degraded downstream answer.

Why does declaration order win? Because permitting properties in any order is expensive. To allow k required properties in arbitrary order, the automaton must remember which subset has already been emitted — that is 2^k states, before you nest anything. Some engines support unordered objects; the hosted strict modes generally pin declaration order instead, and it is the sane default. Which means your schema file is a program, and property order is control flow.

Two more strict-mode consequences worth knowing:

OpenAI's strict mode requires every property in required and additionalProperties: false. If a field has no evidence in the input, the model must still emit something — that is where placeholder hallucinations come from. Model genuine optionality as a nullable union ("type": ["string", "null"]), not by omitting the key.
The grammar is compiled and cached per schema. The first request with a brand-new schema pays extra latency. Generating schemas dynamically per request throws that cache away every time.

What does the fix look like in code?

Same fields, same descriptions, same model. Only the order changes.

from typing import Literal
from pydantic import BaseModel, Field
from openai import OpenAI

client = OpenAI()

# WRONG — the decoder commits to `label` before writing one token of analysis.
class TriageBad(BaseModel):
    label: Literal["urgent", "normal", "low"]
    confidence: float
    rationale: str

# RIGHT — evidence and rationale are in the prefix that conditions `label`.
class TriageGood(BaseModel):
    evidence: list[str] = Field(
        ..., description="Verbatim quotes from the ticket. No paraphrase."
    )
    ruled_out: str = Field(
        ..., description="Which severity you considered and rejected, and why."
    )
    label: Literal["urgent", "normal", "low"]
    confidence: float

resp = client.responses.parse(
    model="gpt-5.1",
    input=[{"role": "user", "content": ticket_text}],
    text_format=TriageGood,
)
out = resp.output_parsed

Note ruled_out. Under strict mode the model must fill it, so you have effectively forced a contrastive step into the decode path. You can attach compute to a task by declaring a field, and remove it by deleting one. That is a stronger lever than most prompt edits.

The self-hosted equivalent is the same idea through vLLM:

llm.generate(
    prompt,
    sampling_params=SamplingParams(
        max_tokens=512,
        guided_decoding=GuidedDecodingParams(json=TriageGood.model_json_schema()),
    ),
)

model_json_schema() preserves Pydantic field order, and XGrammar walks the properties map in that order. Reorder the class, reorder the decode.

Does the same thing happen with Claude tool use?

Yes, with a softer mechanism and the same fix. Claude Sonnet 4.5 and Opus 4.x tool use is schema-guided rather than compiled to a hard token-level grammar the way OpenAI strict mode is, so Claude has more freedom to deviate. But your input_schema is serialized into the request in property order, and the model overwhelmingly writes keys in the order it read them. Answer-first schemas produce answer-first generation, which produces post-hoc rationales.

tools = [{
    "name": "triage_ticket",
    "description": "Record a triage decision for one support ticket.",
    "input_schema": {
        "type": "object",
        # Order is the plan. Cheap observations first, commitments last.
        "properties": {
            "evidence":   {"type": "array", "items": {"type": "string"}},
            "ruled_out":  {"type": "string"},
            "label":      {"type": "string", "enum": ["urgent", "normal", "low"]},
            "confidence": {"type": "number"},
        },
        "required": ["evidence", "ruled_out", "label", "confidence"],
    },
}]

With Claude there is a better option available: turn on extended thinking and let the reasoning happen in thinking blocks before the tool call, then keep the tool input lean. That gives you a real scratchpad with no schema contortions, and the tool result stays clean for downstream consumers. Use in-schema reasoning fields when you are on a non-thinking configuration or when you need the reasoning persisted as structured data.

Why are forced key tokens worth designing?

Because they are prompt tokens you get for free, positioned exactly where they matter. The model does not choose "rationale" — the FSM does — but those tokens still enter the KV cache and still condition everything after them. Renaming a field from notes to contradicting_evidence_from_the_ticket inserts that phrase directly before the value is generated. It is the highest-leverage prompt edit that costs nothing at the call site.

The flip side: never treat logprobs over forced spans as signal. The key tokens, the colons, the braces, the closing quotes are all masked down to a single legal option, so their logprob is ~0 by construction. Averaging token logprobs across a structured output to get "confidence" mostly measures how much punctuation your schema has.

Where logprobs are meaningful is the first token that distinguishes one legal continuation from another. For a Literal["urgent","normal","low"] enum, the FSM has already forced the opening quote, so the next token's renormalized distribution is a genuine posterior over your classes. That only works if the values diverge at the first token. p0/p1/p2 all start with the same p token, so the informative step is buried one position deeper and may split unevenly across the vocabulary. Pick lexically distinct enum values and you get a calibratable classifier out of a generative call.

Also drop the self-reported confidence: float. A model writing 0.9 after its own label is producing a plausible-looking number, not a probability. The enum logprobs are the real thing.

How do you measure this in your own stack?

Change one variable. Duplicate the schema, permute only the property order, hold everything else — same model snapshot, same prompt, same temperature, same seed if available — and run both over the same N examples.

Then analyze it as a paired experiment, not two independent accuracy numbers. Look only at the examples where the two orders disagree, and test whether the disagreements are lopsided (McNemar's test on the discordant pairs). Paired analysis needs far fewer examples to reach significance than comparing two marginal accuracies, which matters because per-call eval noise on a few hundred items is large enough to swallow a real effect.

The cost of reasoning-first is real but narrow: you cannot stream the answer early, and you pay for the extra tokens. Bound the scratchpad fields (maxItems on evidence, an explicit "one sentence" in the description) and the tax stays small. Latency to the field you care about is the trade — accuracy is not.

The short answer

JSON Schema field order breaks structured output accuracy because constrained decoding compiles your property order into a state machine that emits keys in declaration order and masks every other token to zero probability. A decoder-only model can only compute in tokens it has already written, so any field declared before the answer becomes usable scratchpad and any field declared after it is a post-hoc justification with zero causal effect on the answer. Declare evidence and rationale first, the label and any derived numbers last, model optional fields as nullable unions rather than omissions, name fields so the forced key tokens act as instructions, and read confidence from the enum's first divergent token instead of asking the model for a number. Reordering four lines of a Pydantic model is the cheapest accuracy fix in a structured-output pipeline.

Why INT4 Weight-Only Quantization Doesn't Speed Up Prefill

jidonglab — Fri, 31 Jul 2026 20:57:34 +0000

You benchmark a 70B model with batch_size=1, one prompt, one stream. FP16 gives you 18 tokens/sec. You swap in an AWQ INT4 checkpoint and get 55 tokens/sec. Three times faster, same GPU, ~1 point of accuracy lost. You ship it.

Then production traffic arrives: 8k-token RAG prompts, 40 concurrent users. Time-to-first-token gets worse than the FP16 build, and your throughput at high concurrency is flat or slightly down. Nothing is broken. INT4 weight-only quantization did exactly what it does — it reduced bytes moved, and prefill was never bottlenecked on bytes.

TL;DR

INT4 weight-only quantization (W4A16) reduces memory traffic, not FLOPs. The kernel dequantizes INT4 back to FP16 and runs the same FP16 tensor-core MMA. Peak compute is unchanged.
Decode is memory-bandwidth-bound, so cutting weight bytes 4x gives close to a 4x speedup at low concurrency. Prefill is compute-bound, so it gets nothing — and pays the dequantization overhead.
The crossover is arithmetic: on an H100 SXM (~990 TFLOPS dense BF16, ~3.35 TB/s HBM3), the roofline ridge point is ~295 FLOP/byte. BF16 weights need ~295 concurrent decode tokens to become compute-bound; INT4 weights need only ~74. Above that, INT4's advantage is gone.
Chunked prefill makes it worse: mixing prefill chunks into decode batches pushes every batch into the compute-bound regime, which is precisely where W4A16 stops paying.
If you are compute-bound, quantize activations too — FP8 (W8A8) on Hopper, or W4A8 — or accept INT4 purely as a capacity play (more KV cache, fewer GPUs, no tensor-parallel comms).

Why does INT4 weight-only quantization speed up decode but not prefill?

Because decode and prefill sit on opposite sides of the roofline, and INT4 weight-only quantization only moves the memory axis.

A transformer's linear layers are the whole story here. During decode with B concurrent sequences, each weight matrix [K, N] is read from HBM once and used for B token-vectors:

FLOPs: 2 · B · K · N
Bytes: K · N · b where b = bytes per weight (2 for BF16, 0.5 for INT4)

Arithmetic intensity is therefore 2B / b FLOP/byte. At B = 1, BF16 gives you 1 FLOP per byte. An H100 wants ~295. You are running the tensor cores at roughly a third of a percent of peak and burning the entire kernel time on HBM reads. Shrink the weights 4x and the kernel gets ~4x faster — the math never had to change.

Prefill runs the same weight matrix against S tokens at once. With S = 2048, intensity is 2·2048/2 ≈ 2048 FLOP/byte — an order of magnitude past the ridge point. The weight read is amortized to nothing. You are limited by tensor-core throughput, and W4A16 does not raise tensor-core throughput. It lowers it slightly, because you now spend cycles unpacking nibbles, applying per-group scales and zero-points, and writing FP16 into the MMA path.

That is the sentence to remember: W4A16 buys bytes, not FLOPs.

What batch size kills the INT4 advantage?

Roughly the roofline ridge point scaled by bytes-per-weight. On an H100 SXM it is about 74 concurrent decode tokens for INT4 versus ~295 for BF16.

# Where does a weight-stationary GEMM stop being memory-bound?
# intensity = 2*B*K*N / (K*N*bytes_per_weight) = 2B / bytes_per_weight

SPECS = {                      # dense tensor-core FLOPS, HBM bandwidth
    "H100-SXM": (990e12, 3.35e12),   # BF16
    "A100-80G": (312e12, 2.039e12),  # BF16
}

def crossover(gpu, bytes_per_weight):
    flops, bw = SPECS[gpu]
    ridge = flops / bw                       # FLOP/byte the GPU wants
    return ridge * bytes_per_weight / 2      # concurrent tokens B

for gpu in SPECS:
    for name, b in [("BF16", 2), ("FP8", 1), ("INT4", 0.5)]:
        print(f"{gpu:10s} {name:5s} weights -> compute-bound at B ~= {crossover(gpu, b):5.0f}")

# H100-SXM  BF16  weights -> compute-bound at B ~=   296
# H100-SXM  FP8   weights -> compute-bound at B ~=   148
# H100-SXM  INT4  weights -> compute-bound at B ~=    74
# A100-80G  BF16  weights -> compute-bound at B ~=   153
# A100-80G  INT4  weights -> compute-bound at B ~=    38

This model ignores activation traffic and assumes large K, N — fine for a 70B's 8192×28672 MLP projections, less fine for tiny models. Treat it as the right order of magnitude, not a promise.

The practical reading: quantization lowers the concurrency at which you become compute-bound. That is counterintuitive until you see the formula. Fewer bytes per FLOP means you hit the compute wall sooner. So the regime where INT4 gives its headline speedup — small B — is exactly the regime a well-utilized production server tries to leave.

If your serving fleet runs at 60+ concurrent decodes on H100, you have already crossed the line. Your INT4 build is doing the same FLOPs as FP16 plus dequantization.

Why does prefill sometimes get slower after quantizing to INT4?

Three reasons, in descending order of impact.

Dequantization is real work in the inner loop. A W4A16 kernel loads packed nibbles, unpacks, multiplies by a per-group FP16 scale (group size 128 is the common default), subtracts a zero-point, then feeds FP16 MMA. In the memory-bound regime that work hides behind HBM latency for free. In the compute-bound regime there is nothing to hide behind — it lands directly on the critical path.

Kernel maturity and shape sensitivity. Marlin-class kernels closed most of the large-M gap for GPTQ/AWQ weights, but they still tune for specific tile shapes. Feed them a 6k-token prefill with an odd hidden size and you can fall off the fast path into a slower fallback. FP16 cuBLAS/CUTLASS GEMMs have no such cliff.

No fused epilogue advantage. FP8 paths on Hopper get native tensor-core support and can keep activations in 8-bit through the layer. W4A16 has to materialize FP16 activations regardless, so it saves nothing on the activation side of prefill, where activations are large.

Net effect in practice: TTFT moves the wrong way by a modest but measurable amount, while inter-token latency improves substantially. If your SLO is dominated by TTFT on long RAG prompts, INT4 is a regression dressed up as an optimization.

Does chunked prefill make INT4 weight-only quantization worse?

Yes, structurally. Chunked prefill exists to stop long prompts from stalling decode, and it works by slicing a prompt into chunks and co-scheduling them with in-flight decode tokens in a single batched forward pass.

That merged batch has a token count of B_decode + chunk_size. With a 512-token chunk, every batch is at B ≥ 512 — far past the ~74-token crossover. The GEMMs are compute-bound essentially all the time, so the weight-byte savings buy nothing, and the dequantization overhead is paid on every step.

You can still win here, but the win comes from the freed HBM (bigger KV cache → more concurrency → better throughput), not from faster matmuls.

Is INT4 still worth it if it doesn't speed up prefill?

Often yes — as a capacity decision rather than a latency one. Three concrete payoffs:

KV cache headroom. A 70B model at BF16 needs ~140 GB of weights; at INT4, ~35 GB. On a single 80GB H100 that is the difference between "doesn't fit" and "40+ GB of KV cache." More KV cache means higher concurrency, which means better throughput — a second-order effect, but usually the largest one.
Fewer GPUs, no tensor-parallel comms. Dropping from TP=4 to TP=2 (or TP=1) removes all-reduce traffic from every layer. On nodes without NVLink this can dominate everything else in the decode loop.
Cold-start and multi-model density. Loading 35 GB instead of 140 GB matters when you swap models or autoscale.

None of those are "prefill is faster." Be honest about which one you are buying.

What should I run instead when I'm compute-bound?

Quantize the activations, not just the weights.

# vLLM, H100 fleet, high concurrency, long RAG prompts.
# FP8 (W8A8) uses native Hopper tensor cores: halves weight bytes AND
# roughly doubles compute peak vs BF16.
model: meta-llama/Llama-3.3-70B-Instruct
quantization: fp8              # W8A8, native on Hopper/Blackwell
kv_cache_dtype: fp8_e4m3       # also halves KV traffic during decode
max_model_len: 32768
enable_chunked_prefill: true
max_num_batched_tokens: 2048   # tune against TTFT vs ITL SLOs
gpu_memory_utilization: 0.92

Rules of thumb:

Interactive, low concurrency, one user per GPU (local inference, agents with B ≈ 1–8): INT4 weight-only is the right call. This is where the 3x single-stream number is real.
High-concurrency serving on Hopper/Blackwell: FP8 W8A8. You get the byte reduction and the compute-peak increase, which is the only way to speed up prefill.
You need INT4 for capacity but serve at high concurrency: look at W4A8 schemes, which keep 4-bit weights for memory but use 8-bit tensor cores for the math.
Pre-Hopper (A100): no native FP8. INT8 W8A8 with SmoothQuant-style outlier handling is the compute-side option; INT4 remains capacity-only.

How do I measure this on my own workload?

Stop benchmarking with batch_size=1. Sweep concurrency and separate the two phases:

# Sweep concurrency; watch TTFT (prefill) and ITL (decode) separately.
for c in 1 4 16 64 128; do
  vllm bench serve \
    --model $MODEL --dataset-name random \
    --random-input-len 4096 --random-output-len 256 \
    --max-concurrency $c --num-prompts $((c * 8))
done

Log TTFT p50/p99 and inter-token latency p50/p99 as separate series, then plot each against concurrency for both checkpoints. The signature of the failure mode is unmistakable: the INT4 ITL curve sits well below FP16 at low concurrency and converges to it somewhere in the 50–100 range, while the INT4 TTFT curve sits above FP16 everywhere. If you only ever look at end-to-end throughput at one concurrency level, both effects average into a single number that tells you nothing.

The short answer

INT4 weight-only quantization doesn't speed up prefill because prefill is compute-bound and W4A16 only reduces memory traffic — the kernel still dequantizes to FP16 and runs the same tensor-core matmuls, so it does the same FLOPs plus unpacking overhead. Decode at low concurrency has an arithmetic intensity near 1 FLOP/byte against a GPU that wants ~295, so removing 75% of the weight bytes gives a near-linear speedup there; prefill runs at hundreds or thousands of FLOP/byte, where those bytes were already free. The crossover is around 74 concurrent decode tokens on an H100, and chunked prefill pushes every batch past it. Use INT4 for single-stream latency and for memory capacity; use FP8 or another activation-quantized scheme when you need prefill and high-concurrency throughput to actually get faster.

DPO Likelihood Displacement: Why Chosen Responses Get Rarer

jidonglab — Fri, 31 Jul 2026 08:55:07 +0000

You run DPO on a clean preference set. The margin goes up, the win rate on your pairwise eval goes up, and then you scroll back through the training logs: logps/chosen started at -142 and ended at -231. The response you were rewarding is now less likely than it was before training. Not relatively — absolutely.

This is DPO likelihood displacement, and it is not a bug in your data loader. It is a direct consequence of what the DPO objective optimizes, and it is the mechanism behind the most annoying DPO failure mode in practice: you train on "helpful and safe" versus "harmful," and you ship a model that refuses everything.

TL;DR

DPO only constrains the *margin* between chosen and rejected log-probs. Both can fall together and the loss still decreases — nothing in the objective anchors log π(y_w) upward.
The exact condition: log π(y_w) decreases when ⟨∇log π(y_w), ∇log π(y_l)⟩ > ‖∇log π(y_w)‖² — i.e. when the rejected response's gradient is aligned with the chosen one's and larger. Similar-looking pairs are the danger zone.
Displaced probability mass doesn't vanish. It flows to whatever is geometrically adjacent — usually a short refusal, a hedge, or an empty answer, none of which appear in your dataset.
Detect it by logging logps/chosen in absolute terms (not just the margin) plus a fixed "escape probe" string, and watch whether the probe's log-prob rises.
Fix it with an NLL anchor on chosen (rpo_alpha in TRL), by filtering high-similarity pairs (CHES), and by making sure your reference model actually assigns decent probability to the chosen responses.

What is DPO likelihood displacement?

Likelihood displacement is when DPO training reduces the absolute log-probability of the preferred response y_w while still increasing the preference margin, pushing the freed probability mass onto completions that were never in the training data.

The margin looks great. rewards/margins climbs monotonically. rewards/chosen — which in TRL is β(log π_θ(y_w) − log π_ref(y_w)) — goes negative and stays there. That negative number is the whole story: the policy assigns less mass to the chosen response than the reference did.

Why does DPO push down the probability of the chosen response?

Because the DPO loss is a function of a difference, and a difference is invariant to shifting both terms down.

L_DPO = −log σ( β·[log π_θ(y_w|x) − log π_ref(y_w|x)]
              − β·[log π_θ(y_l|x) − log π_ref(y_l|x)] )

∇_θ L_DPO = −β · σ(−β·Δ) · [ ∇_θ log π_θ(y_w|x) − ∇_θ log π_θ(y_l|x) ]

The update direction is g_w − g_l with equal weight on both terms. Lowering log π(y_l) by 3 nats and lowering log π(y_w) by 1 nat is a perfectly good descent step. The reference model appears only inside Δ as a constant offset; it shifts when the sigmoid saturates, not which direction the gradient points.

Now do the first-order analysis. Under gradient flow, the rate of change of the chosen log-prob is the inner product of its own gradient with the update direction:

d/dt log π_θ(y_w) ∝ ⟨ g_w , g_w − g_l ⟩ = ‖g_w‖² − ⟨g_w, g_l⟩

So log π(y_w) decreases exactly when ⟨g_w, g_l⟩ > ‖g_w‖². Two things drive that: the gradients must point in similar directions, and ‖g_l‖ must be large relative to ‖g_w‖. Both are common. Preference pairs are usually near-duplicates — same prompt, same topic, often the same opening 40 tokens — so g_w and g_l are highly aligned. And if the rejected response is already low-probability under the policy, its gradient norm is large. You get a step that suppresses y_l hard and drags y_w down with it.

Look at the last layer to see why alignment is so high. For unembedding matrix W, the gradient of a token's log-prob is an outer product (e_token − p) hᵀ, where h is the final hidden state. The inner product of two such gradients factorizes into a token-geometry term times a hidden-state similarity term. When chosen and rejected differ only by near-synonyms — "Never" vs "No", "I'd suggest" vs "You should" — the token vectors are close and the hidden states are nearly identical. The suppression signal on the rejected token bleeds straight onto the chosen token through the shared softmax normalizer.

Where does the displaced probability mass go?

Softmax conserves mass. If you push down the rejected continuation and the chosen one comes down with it, that mass lands on whatever the model considers the next-nearest neighbor in output space — and the objective has no term describing that region at all.

In practice the recipient is almost always a short, generic, high-prior string: a refusal, "I don't have enough information," a truncated one-liner. These sit at high probability under the base model and are geometrically far from both members of a typical preference pair, so they absorb displaced mass without ever contributing to the loss.

Why does DPO on safety data cause over-refusal?

Because safety preference sets are the worst case for the condition above. The chosen response is typically a nuanced, careful, non-refusing answer; the rejected one is a harmful answer to the same prompt. They share the prompt, the topic, and often the framing — maximum gradient alignment. The rejected response is far off-policy after SFT, so it has a large gradient norm.

Result: DPO suppresses the harmful answer, drags the careful answer down with it, and the mass lands on a blanket refusal that was never labeled as preferred. Razin et al.'s work on unintentional unalignment documents this in both directions — the same mechanism can also move mass toward unsafe outputs when the pair geometry flips. The lesson generalizes past safety: any dataset where chosen and rejected are minimally-edited variants of each other is a displacement machine.

How do I detect likelihood displacement in a training run?

Log absolute log-probs, not just the margin, and add a probe string that isn't in your data. Three columns tell you almost everything:

`logps/chosen`	`logps/rejected`	Reading
↑	↓	Healthy. This is what you want.
↓ slightly	↓↓ steeply	Displacement. Margin is a lie.
↓↓	↓↓↓	Severe. Check probe mass immediately.
↑	↑	Reference/policy mismatch — verify `π_ref` is your actual SFT checkpoint.

The probe is the part people skip. Pick a string you never want to see more of — a canned refusal — and track its log-prob under the policy on a fixed set of benign prompts:

from transformers import TrainerCallback
import torch

PROBE = "I'm sorry, but I can't help with that."

class EscapeMassProbe(TrainerCallback):
    def __init__(self, tok, prompts, every=50):
        self.tok, self.prompts, self.every = tok, prompts, every

    @torch.no_grad()
    def on_step_end(self, args, state, control, model=None, **kw):
        if state.global_step % self.every:
            return
        total = 0.0
        for p in self.prompts:
            ids = self.tok(p + PROBE, return_tensors="pt").to(model.device)
            n_probe = len(self.tok(PROBE).input_ids)
            logits = model(**ids).logits[:, :-1].log_softmax(-1)
            tgt = ids.input_ids[:, 1:]
            lp = logits.gather(-1, tgt.unsqueeze(-1)).squeeze(-1)
            total += lp[:, -n_probe:].sum().item()   # log p(PROBE | prompt)
        state.log_history.append(
            {"step": state.global_step, "probe/refusal_logp": total / len(self.prompts)}
        )

If probe/refusal_logp climbs while logps/chosen falls, you are watching mass move in real time. That number correlates with over-refusal on your held-out benign set far earlier than any win-rate eval will show it.

How do I fix DPO likelihood displacement?

Four interventions, roughly in order of effort-to-payoff.

1. Anchor the chosen response with an NLL term. This is the single highest-value change. Adding λ · −log π_θ(y_w|x) to the loss gives the optimizer an explicit reason to keep log π(y_w) high instead of only maximizing the margin. TRL exposes it as rpo_alpha; Llama 3's post-training used the same idea, and CPO/RPO variants formalize it.

from trl import DPOConfig, DPOTrainer

cfg = DPOConfig(
    beta=0.1,
    rpo_alpha=1.0,            # NLL anchor on chosen — the fix that matters most
    loss_type="sigmoid",
    learning_rate=5e-7,       # DPO wants an order of magnitude less than SFT
    max_length=2048,
    max_prompt_length=1024,
    logging_steps=10,
)

trainer = DPOTrainer(
    model=policy, ref_model=sft_checkpoint,   # must be the SFT model, not the base
    args=cfg, train_dataset=pairs,
    processing_class=tokenizer,
    callbacks=[EscapeMassProbe(tokenizer, benign_prompts)],
)

2. Filter high-similarity pairs before training. The gradient condition says similar pairs are the problem, so measure similarity in the space that actually drives the gradient — final hidden states, not text. CHES (centered hidden embedding similarity) is a cheap proxy for ⟨g_w, g_l⟩:

@torch.no_grad()
def ches(model, tok, prompt, y_w, y_l):
    def resp_embed(y):
        ids = tok(prompt + y, return_tensors="pt").to(model.device)
        n = len(tok(y).input_ids)
        h = model(**ids, output_hidden_states=True).hidden_states[-1][0]
        return h[-n:].sum(0)                     # sum over response tokens
    hw, hl = resp_embed(y_w), resp_embed(y_l)
    mu = (hw + hl) / 2                           # center within the pair
    hw, hl = hw - mu, hl - mu
    return torch.cosine_similarity(hw, hl, dim=0).item()

pairs = [p for p in pairs if ches(ref, tok, p["prompt"], p["chosen"], p["rejected"]) < 0.75]

Dropping the top-similarity slice — the pairs whose chosen and rejected are minimal edits of each other — removes most of the displacement pressure while keeping the pairs that carry real preference signal. Text-level edit distance is a weaker but zero-cost first pass.

3. Check your reference model. If chosen responses were distilled from a stronger model, log π_ref(y_w) is already low, ‖g_w‖ is large in a direction the policy can't cheaply follow, and the optimizer takes the easy route of crushing y_l. Run one SFT epoch on the chosen responses first and use that as π_ref. This is why "SFT on chosen, then DPO" is the standard recipe and not a ritual.

4. Raise β, carefully. Larger β saturates the sigmoid sooner, so the effective step shrinks once the margin is achieved, limiting how far both log-probs can drift. It also slows learning. Treat it as a damper, not a cure — β from 0.1 to 0.3 is a reasonable sweep before you conclude the data is the problem.

Does this happen with PPO or GRPO too?

Not the same way. Policy-gradient methods increase log π(y) directly for sampled completions with positive advantage — there's no paired difference term that can be satisfied by lowering both sides. Sampling is on-policy, so the mass that gets redistributed lands near the current policy rather than on some distant refusal mode, and the explicit KL penalty against the reference bounds total drift. You trade the displacement failure mode for reward hacking and a much heavier training loop, which is a real trade, not a free win.

The short answer

DPO likelihood displacement happens because the DPO loss constrains only the difference between chosen and rejected log-probabilities, never their absolute level — so gradient descent is free to satisfy the objective by pushing both down, which it does whenever the rejected response's gradient is aligned with and larger than the chosen response's (⟨g_w, g_l⟩ > ‖g_w‖²), the common case for near-duplicate preference pairs. The vacated probability mass moves to whatever is geometrically adjacent and high-prior, typically a generic refusal that never appears in your training set — which is why safety-focused DPO runs so often produce over-refusing models with excellent margin curves. Log logps/chosen in absolute terms alongside a refusal probe, add an NLL anchor on the chosen response (rpo_alpha=1.0), filter out high-CHES pairs, and make sure your reference model is the SFT checkpoint that already fits the chosen responses.

ColBERT Late Interaction: Why MaxSim Beats One Vector Per Chunk

jidonglab — Thu, 30 Jul 2026 20:52:50 +0000

A single dense vector per chunk asks your encoder to do something absurd: compress everything anyone might ever ask about a 250-token passage into one point in 1024-dimensional space, before it has seen the question. It works fine until your query mentions a part number, a drug interaction, or a clause that occupies 8 of those 250 tokens. Then the pooled vector — dominated by the other 242 — sits nowhere near the query, and your retriever silently returns the wrong chunk.

ColBERT late interaction attacks this by refusing to pool. It keeps one embedding per token and scores with MaxSim. Here's the mechanism, the real storage bill, and the four things that break in production.

TL;DR

ColBERT late interaction stores one vector per token instead of one per chunk, and scores with MaxSim: every query token independently finds its best-matching document token, and those maxima are summed.
It's "late" because query and document never meet inside the transformer — document embeddings are precomputed offline, unlike a cross-encoder. You get term-level evidence at bi-encoder indexing cost.
The win is out-of-domain robustness. Pooling destroys rare-entity signal; MaxSim preserves it, so late interaction degrades far more gracefully on domains you never fine-tuned on.
Storage is ~30x a single-vector index uncompressed, but ColBERTv2-style residual compression (centroid + 1–2 bits/dim) cuts it to roughly 4x. That's the difference between shippable and not.
MaxSim scores are unnormalized sums over query tokens. They are not comparable across queries, so fixed thresholds and naive score fusion are broken by construction.

What is late interaction in ColBERT, and how does MaxSim score a document?

Late interaction encodes the query and the document separately into sets of token vectors, then computes their similarity with a cheap, non-neural operator afterwards. The operator is MaxSim:

S(q, d) = Σ_{i=1..|q|}  max_{j=1..|d|}  q_i · d_j

Both sides are L2-normalized, so each dot product is a cosine in [-1, 1]. Query token i scans every document token, takes its single best match, and contributes that number. Nothing else about the document matters to that query token.

Compare the three regimes:

Bi-encoder: score = pooled_q · pooled_d. One dot product. Interaction happens after total compression — too late, too lossy.
Cross-encoder: query and document go through the transformer together, full attention between every pair. Best quality, but you cannot precompute anything, so it only works on a top-k shortlist.
Late interaction: interaction after encoding but before pooling. Document vectors are precomputable; you still get per-term matching.

The scoring itself is a small matmul:

import torch

def maxsim(Q, D, doc_mask):
    """
    Q:        [nq, dim]           L2-normalized query token embeddings
                                  (includes [MASK] expansion tokens)
    D:        [ndocs, dlen, dim]  L2-normalized doc token embeddings
    doc_mask: [ndocs, dlen]       True for real tokens, False for padding
    returns:  [ndocs]             MaxSim scores
    """
    # pairwise cosine between every query token and every doc token
    sim = torch.einsum("qh,nlh->nql", Q, D)          # [ndocs, nq, dlen]

    # padding must never win a max
    sim = sim.masked_fill(~doc_mask[:, None, :], -1e4)

    return sim.max(dim=-1).values.sum(dim=-1)        # max over doc, sum over query

For a 32-token query, a 220-token document and dim=128, that's a 32x220x128 matmul per candidate — nothing. The cost of late interaction is never the arithmetic. It's the memory traffic to get those document vectors into registers, which is why the entire engineering story is about compression and candidate pruning.

Why does MaxSim beat a single dense vector out of domain?

Because pooling is a lossy summary chosen before the question is known, and MaxSim isn't.

Take a chunk that covers three things: a product's warranty terms, its power requirements, and a shipping note. A mean-pooled vector lands at the centroid of those three topics — close to none of them. A query about power requirements has to be near that centroid to retrieve the chunk, and it isn't. Under MaxSim, only the ~15 tokens describing power draw need to match. The other 200 tokens contribute nothing and cost nothing.

This is why late interaction behaves like a soft, contextualized BM25. It has the term-matching precision of lexical search — a rare token in the query lands on a rare token in the document — but the tokens are contextual embeddings, so synonyms and morphological variants still match. You get lexical robustness without lexical brittleness.

The practical consequence: on domains you never trained on, a fine-tuned single-vector bi-encoder falls apart in a way late interaction does not. Single-vector models learn what to keep during pooling for their training distribution. Shift the distribution and they keep the wrong things. ColBERT-style models have much less to unlearn, because the decision of what matters is deferred to query time. If you have no labeled data for your corpus, this is the strongest argument for the architecture.

What does per-token indexing actually cost in storage?

Do the arithmetic before you fall in love with the quality numbers.

DIM        = 128     # ColBERT output dim (projected down from the encoder's 768)
DOC_MAXLEN = 220     # tokens kept per chunk; everything past this is invisible
NBITS      = 2       # residual bits per dimension (ColBERTv2-style compression)

single_vector = 1024 * 2                       # 1024-dim fp16 = 2,048 B / chunk
uncompressed  = DOC_MAXLEN * DIM * 2           # fp16 per token = 56,320 B / chunk
compressed    = DOC_MAXLEN * (DIM * NBITS / 8  # residual = 32 B
                              + 4)             # centroid id = 4 B  -> 7,920 B / chunk

print(uncompressed / single_vector)   # ~27x
print(compressed  / single_vector)    # ~3.9x

Naively, one vector per token is ~30x the footprint of a single-vector index. Nobody ships that at scale.

ColBERTv2's residual compression is what makes it tractable. Run k-means over all token embeddings in the corpus to get a centroid codebook. For each token, store the nearest centroid's id plus the residual (token minus centroid) quantized to 1 or 2 bits per dimension. Because contextual token embeddings cluster hard — the same word in the same sense lands in the same place — the residuals are small and survive brutal quantization. At 2 bits you're at ~36 bytes per token; at 1 bit, ~20 bytes. That's ~4x a single-vector index, not 30x, and it's the whole reason the architecture is deployable.

Note the second-order effect: the codebook itself gives you a candidate generation mechanism for free.

How do you retrieve candidates without scoring every document?

You don't scan the corpus. You probe the centroid index with each query token separately, and let the union of their neighborhoods define the candidate set.

The PLAID-style pipeline runs in stages:

Centroid probe. Each of the ~32 query token embeddings does an ANN lookup against the centroid codebook (tens of thousands of centroids, not billions of tokens). Collect the document ids that touch those centroids.
Centroid-only scoring. Approximate MaxSim using only centroids, never decompressing residuals. This is cheap and prunes aggressively.
Full scoring. Decompress residuals for the survivors and compute exact MaxSim.

Each stage narrows by roughly an order of magnitude. Stage 3 touches a tiny fraction of the corpus, which is what keeps latency in the same ballpark as HNSW despite the vector count.

Also worth knowing: token pooling. Adjacent or near-duplicate document token embeddings can be clustered and collapsed before indexing, cutting vector count meaningfully with modest quality loss. It's the first knob to reach for when your index doesn't fit.

What breaks in production?

MaxSim scores are unnormalized sums, so thresholds are meaningless. The score is a sum over |q| terms. A 40-token query scores structurally higher than a 6-token query on the same document. There is no fixed cutoff that means "relevant." Two consequences: never hard-threshold raw MaxSim, and never feed raw MaxSim into a weighted score fusion with BM25 — use rank-based fusion, or divide by query length first. Teams port a score > 0.7 filter from cosine-similarity search and quietly retrieve nothing for short queries.

Query padding is query expansion, not padding. ColBERT pads queries to a fixed query_maxlen (32 is the usual default) with [MASK] tokens, and those masks are encoded, kept, and scored. They learn to act as contextual expansion terms. So query_maxlen is a retrieval-behavior knob, not a buffer size. Doubling it doesn't just cost memory; it changes what the model retrieves. Tune it, and be aware very short queries get proportionally more expansion.

Long documents win. Max over more tokens means more chances at a spuriously high match. Late interaction has a length bias toward long chunks, which matters if your corpus mixes 50-token FAQ entries with 400-token manual sections. Keep chunk lengths roughly uniform, or the length distribution becomes part of your ranking function. (ColBERT filters punctuation from document embeddings for a related reason — junk tokens that can win a max are pure noise.)

doc_maxlen truncation is silent. Tokens past the limit are simply not indexed. Set doc_maxlen to 180 with 400-token chunks and half of every chunk is unretrievable, with no error and no log line. Measure your chunk token-length distribution against doc_maxlen before you trust any eval.

When should you not use ColBERT late interaction?

When your chunks are short and single-topic, your queries are in-domain, and you have labeled pairs to fine-tune on. A well-tuned single-vector retriever plus a cross-encoder reranker on the top 50 will match it, costs a quarter of the storage, and runs on any vector database you already operate. Late interaction pays for itself specifically when you have domain shift and no training data, heterogeneous multi-topic chunks, or entity-heavy queries where a single missing term should sink a result.

The infrastructure constraint is real too: this needs an engine that understands multi-vector indexing and residual codebooks. That's a narrower set of options than "anything that speaks HNSW," and it's a genuine operational cost, not a footnote.

So why does MaxSim beat one vector per chunk?

ColBERT late interaction wins because it defers the compression decision until the query arrives. A single-vector bi-encoder must decide what a passage "means" during indexing, and mean-pooling 250 contextual token embeddings destroys exactly the rare, specific signal that distinguishes one chunk from its near-duplicates. MaxSim keeps every token vector and lets each query token pick its own best evidence, giving lexical-grade precision with contextual matching — which is why it degrades so much more gracefully out of domain. You pay for it in index size (~30x raw, ~4x after centroid-plus-residual compression) and in a retrieval pipeline that must prune through a centroid codebook rather than a plain ANN index. Take the trade when you have domain shift and no labeled data; skip it when a fine-tuned bi-encoder plus a reranker already covers your queries.

Surface Form Competition: Why Log-Prob Answer Scoring Fails

jidonglab — Thu, 30 Jul 2026 08:50:31 +0000

Your eval harness scores four candidate answers by summing token log-probabilities and picks the highest. The model "knows" the answer is a nuclear power plant, but the harness returns coal — because the right answer is four tokens longer and every extra token multiplies in another probability less than one. Nothing is broken. The scoring function is doing exactly what you asked. It's just measuring the wrong thing.

This failure has a name: surface form competition. If you rank answer options by log P(option | prompt), you are ranking strings, not meanings, and strings compete with each other in ways that have nothing to do with whether the model knows the answer.

TL;DR

Log-prob answer scoring conflates three things: whether the answer is right, how long its surface form is, and how a-priori likely that string was anyway.
Length bias is the easy half. Normalize by bytes, not tokens — token counts are tokenizer-dependent, so per-token normalization is not comparable across models or even across options.
Surface form competition is the hard half. Probability mass splits across paraphrases (computer / a computer / the computer), so no length normalization can rescue a correct answer whose mass is spread thin.
PMI / domain-conditional normalization is the standard fix: score log P(a | question) − log P(a | domain premise) to subtract out the option's a-priori plausibility.
Switching to A/B/C/D doesn't dodge the problem — it trades length bias for token-prior and position bias. Fix that with contextual calibration and cyclic permutation, not by hoping it averages out.

What is surface form competition in LLM evaluation?

Surface form competition is what happens when one meaning has many valid spellings and each spelling gets its own slice of the probability mass. An autoregressive LM must put mass on strings. If the correct concept can be written as a nuclear power plant, nuclear power, nuclear plants, or nuclear energy, each of those gets a fraction, and each competes against a single-form distractor like coal that hogs all of its own mass.

The model can be perfectly calibrated about the concept and still rank the wrong string first. This is a measurement artifact of your scoring function, not a knowledge failure — and it means an eval regression can appear when nothing about the model's understanding changed.

Length bias rides along on top of it. Under the chain rule:

log P(a | q) = Σ_i log P(a_i | q, a_<i)

Every additional token adds a negative term. If the model's average per-token log-probability in this context is roughly −H, the sequence score scales like −H · L. Longer answers lose by construction. This is why lm-evaluation-harness reports both acc and acc_norm, and why on datasets with long, uneven endings (HellaSwag is the canonical case) the two numbers are visibly different — sometimes by enough to reorder a leaderboard.

Does normalizing by length fix it?

It fixes the length half, and only if you normalize by the right unit.

Do not divide by token count. Tokenization is model-specific: the same answer string is 3 tokens under one tokenizer and 6 under another, and within a single option set, an answer full of rare proper nouns fragments far more than a common phrase. Per-token normalization silently rewards options that tokenize efficiently.

Divide by byte length (len(option.encode("utf-8"))). Bytes are tokenizer-invariant, so the same normalized score is comparable across models — this is exactly what acc_norm does.

What byte normalization does not fix is mass splitting. If a computer and the computer and computer each take a third of the concept's mass, dividing each by its own length leaves all three below a distractor that never had competition. You need a second correction.

What does PMI normalization actually do?

It divides out the option's prior. Instead of ranking by log P(a | q), rank by a pointwise-mutual-information-style score:

score(a) = log P(a | q) − log P(a | domain_premise)

The second term is the option's log-probability under a content-free premise from the same domain — "Answer:" for a QA task, "The sentence continues:" for a completion task. Subtracting it asks a sharper question: how much did seeing the question raise this string's likelihood? A string that was already probable (short, frequent, generic) gets docked; a string that only became probable because of the question gets rewarded.

Use a domain-conditional premise rather than the fully unconditional P(a). Unconditional normalization over-corrects toward rare, weird strings, because rarity alone maximizes the ratio. Keeping the premise inside the task's domain and format cancels the domain prior without handing the win to nonsense.

Here's all four scorers, computed from one forward pass per (prompt, option) pair:

import torch
import torch.nn.functional as F

@torch.no_grad()
def seq_logprob(model, tok, prompt: str, cont: str):
    """Sum of log P(cont | prompt) plus its token and byte lengths."""
    p_ids = tok(prompt, return_tensors="pt").input_ids.to(model.device)
    full_ids = tok(prompt + cont, return_tensors="pt").input_ids.to(model.device)
    n_p = p_ids.shape[-1]

    # Boundary check: BPE can merge the last prompt char with the first
    # continuation char, which shifts every index below by one.
    assert torch.equal(full_ids[0, :n_p], p_ids[0]), "tokenizer merged across the boundary"

    logits = model(full_ids).logits                       # [1, T, V]
    logprobs = F.log_softmax(logits[:, :-1].float(), dim=-1)
    targets = full_ids[:, 1:]                             # next-token targets
    tok_lp = logprobs.gather(-1, targets.unsqueeze(-1)).squeeze(-1)[0]

    cont_lp = tok_lp[n_p - 1:]        # predictions for the continuation tokens
    return cont_lp.sum().item(), cont_lp.numel(), len(cont.encode("utf-8"))


def score_option(model, tok, question: str, option: str, premise: str = "Answer:"):
    lp, n_tok, n_bytes = seq_logprob(model, tok, question, option)
    lp_prior, _, _ = seq_logprob(model, tok, premise, option)
    return {
        "raw":       lp,               # length-biased; do not ship this
        "per_token": lp / n_tok,       # tokenizer-dependent; not comparable
        "per_byte":  lp / n_bytes,     # this is acc_norm
        "pmi_dc":    lp - lp_prior,    # domain-conditional PMI
    }

Two details that quietly corrupt results if you skip them:

Own the whitespace. Strip trailing spaces from the prompt and put the leading space on the continuation (" a nuclear power plant"). A prompt ending in a space makes the model predict a token that almost never starts a word, and it tanks the score of every option equally except the ones where the merge happens to work out.
Assert the prefix property. The assert above is not paranoia. Byte-pair merges across the prompt/continuation boundary shift n_p and make you score the wrong slice — silently, with plausible-looking numbers.

Why doesn't the A/B/C/D format avoid this?

Because it swaps one bias for two. Reformatting as "answer with a single letter" makes every option exactly one token long, which kills length bias and mass splitting outright. What you get instead:

Token prior bias. The model has a baseline preference over " A", " B", " C", " D" before it reads anything. That prior is often far from uniform.
Position bias. Move the correct answer to a different slot and the prediction changes. This is the same class of artifact as position bias in LLM-as-judge setups, and it does not average out on small eval sets.
Symbol binding failure. Weaker or base models often "know" the content but can't reliably map it to the letter, which reads as a knowledge failure and isn't one.

The fix for the prior is contextual calibration: measure the model's letter distribution on a content-free input, then apply a diagonal affine correction that flattens it.

import numpy as np

LETTERS = ["A", "B", "C", "D"]

def letter_probs(logprob_fn, prompt: str) -> np.ndarray:
    """Softmax restricted to the four letter tokens."""
    lps = np.array([logprob_fn(prompt, letter) for letter in LETTERS])
    e = np.exp(lps - lps.max())
    return e / e.sum()

def calibrated_pick(logprob_fn, render, question, options,
                    content_free=("N/A", "", "[MASK]")):
    p = letter_probs(logprob_fn, render(question, options))
    # Same template, same options, no question content.
    p_cf = np.mean([letter_probs(logprob_fn, render(cf, options))
                    for cf in content_free], axis=0)
    W = 1.0 / p_cf                       # diag(p_cf)^-1, b = 0
    return LETTERS[int(np.argmax(W * p))]

The calibration set matters: use the same rendered template and the same option strings, with only the question replaced. You are estimating the template's bias, not the model's general letter preference.

If your provider doesn't return token log-probabilities, this method isn't available directly. OpenAI-style APIs expose logprobs / top_logprobs and you can pin the output to the letter set with logit_bias. Anthropic's Messages API does not return token log-probs, so for Claude Opus 4.x or Sonnet 4.x you get the equivalent debiasing behaviorally: run each item under cyclic permutations of the option order and take a majority vote. Four permutations for a four-way item costs 4× the calls, and it removes both position bias and letter-prior bias without needing scores at all. Structured output (a single-field schema constrained to the letter set) keeps parsing clean.

Which scorer should you actually ship?

Setup	Scorer	Why
Base model, cloze/completion format	per-byte + PMI-DC	Length and prior both bite; report both, they disagree informatively
Base model, uneven-length options	PMI-DC	Length norm alone leaves mass splitting untouched
Instruction-tuned model, letter format	contextual calibration + cyclic permutation	No length bias; prior and position bias dominate
No log-prob access (Claude, most hosted chat APIs)	cyclic permutation + majority vote	Behavioral equivalent of calibration

Two operating rules. First, report the scorer alongside the number — "72% on ARC-Challenge" is not a claim until you say whether that's acc, acc_norm, or PMI. Second, fix the scorer before you fix the model: a scoring change can move a benchmark several points, which is comfortably larger than most real gains you're chasing, and you don't want to spend a week attributing one to the other.

Failure-mode checklist

Prompt ends in a space → whitespace collides with the continuation's leading token.
Prompt is not a token-level prefix of prompt+continuation → your slice indices are off by one.
Normalizing by token count → results not comparable across tokenizers.
Unconditional PMI instead of domain-conditional → rare, malformed options win.
Letter format with no calibration → you're partly measuring the model's fondness for " C".
Fixed option order across the whole eval set → position bias baked into the headline number.

The short answer

Log-prob answer scoring fails because of surface form competition: log P(option | prompt) measures the likelihood of a string, which bundles together correctness, length, and the string's a-priori frequency. Longer answers accumulate more negative log terms, and correct answers with many valid paraphrases have their probability mass split across them, so a short single-form distractor can outrank an answer the model actually knows. Normalize by byte length to remove the length term, subtract a domain-conditional prior (PMI) to remove the frequency term, and if you switch to A/B/C/D letters, apply contextual calibration and permute the option order — because that format doesn't eliminate the bias, it just moves it somewhere less visible.

Activation Outliers: Why W8A8 INT8 Quantization Needs SmoothQuant

jidonglab — Wed, 29 Jul 2026 20:47:01 +0000

Take a 7B+ decoder, dump the input activations to down_proj in layer 20, and look at the per-channel absolute max. Most of the 11008 channels sit around 0.3–1.0. Six or seven of them sit at 40–90. Same channel indices, every token, every prompt, across most layers.

Now quantize that tensor to INT8 with one scale: s = 70 / 127 ≈ 0.55. Your typical activation of 0.5 rounds to 1. Your 0.4 rounds to 1. Your 0.2 rounds to 0. You just replaced a 4096-dimensional vector with a 3-bit vector plus a few giant spikes, and your model now writes fluent nonsense while perplexity looks "only a bit worse."

That is the whole story of why W8A8 INT8 quantization fails on large language models. It is not a rounding-mode problem, and no amount of better calibration search fixes it. Activation outliers are structural, and the fix is an algebraic one.

TL;DR

Activation outliers are systematic: a handful of hidden channels carry magnitudes one to two orders of magnitude above the rest, at the same indices across tokens and layers. LLM.int8() reported them emerging consistently around the ~6.7B parameter mark.
Per-tensor INT8 activation quantization collapses because one channel sets the scale for all of them. Per-token scaling does not help — the outlier is inside every row.
You cannot quantize activations per input channel in an INT8 GEMM: the channel axis is the reduction axis, so the scale does not factor out of the int32 accumulator.
SmoothQuant exploits Y = (X diag(s)⁻¹)(diag(s) W) to migrate range from activations into weights per channel, then folds diag(s)⁻¹ into the preceding RMSNorm/LayerNorm affine. Zero runtime cost, mathematically exact.
If your decode is memory-bound at small batch, skip activation quantization entirely and use weight-only 4-bit (AWQ/GPTQ). W8A8 only pays off when you are compute-bound: large-batch serving and prefill.

Why do activation outliers break per-tensor INT8 quantization?

Because INT8 has 256 levels of uniform resolution and outliers force nearly all of them to be spent on values that almost never occur.

Weights are easy to quantize. Per output channel, weight distributions are close to Gaussian with a tame dynamic range — INT8 per-channel weight quantization is essentially free in accuracy. Activations are not. The residual stream in a trained transformer develops a small set of channels that behave like fixed, high-magnitude biases. They correlate with the "massive activation" / attention-sink phenomenon: the model uses a few dimensions as a near-constant reference signal rather than as content.

The damage math is simple. For symmetric absmax quantization, the step size is Δ = max|X| / 127. Signal-to-noise for the non-outlier bulk scales with E|x| / Δ. Push max|X| up 100x and you lose ~6.6 bits of effective precision on the values that actually carry the token's meaning.

Clipping the outliers is not an escape hatch. Those channels are load-bearing; clip them and you break the sink behavior, which shows up as degraded long-context retrieval and unstable attention at exactly the point where quantization was supposed to be invisible.

Why can't you just quantize activations per channel?

Because of where the scale sits relative to the summation. This is the part most people get wrong, and it is pure linear algebra.

For Y = X W with X ∈ ℝ^{T×Cin}, W ∈ ℝ^{Cin×Cout}:

Y[t, o] = Σ_c  X[t, c] * W[c, o]

A per-token activation scale a[t] and a per-output-channel weight scale b[o] both pull straight out of the sum:

Y[t, o] ≈ a[t] * b[o] * Σ_c  Xq[t, c] * Wq[c, o]

That is why INT8 kernels are happy with per-token × per-out-channel: the int32 accumulator is computed once, then rescaled once in the epilogue.

A per-input-channel activation scale g[c] does not pull out:

Y[t, o] = Σ_c  g[c] * Xq[t, c] * W[c, o]

The scale is trapped inside the reduction. Applying it would mean dequantizing before accumulation — which is exactly what tensor cores refuse to do, and what would erase the throughput win you quantized for in the first place.

This asymmetry also explains why weight-only 4-bit quantization gets away with group-wise scales along Cin (group size 128 is standard in AWQ/GPTQ kernels). W4A16 kernels dequantize weights to FP16 and accumulate in FP16/FP32, so per-group scales are applied before the multiply-accumulate. INT8 W8A8 has no such freedom.

So per-channel is where the outliers are, and per-channel is the one axis you cannot use. That is the trap.

What does SmoothQuant actually do?

It moves the problem to an axis you can scale. For a diagonal diag(s) with positive entries:

Y = X W = (X diag(s)⁻¹) (diag(s) W)

Divide activation channel c by s[c], multiply weight row c by s[c]. Exact, not an approximation. Choose s so both operands become quantizable:

s[c] = max|X[:, c]|^α  /  max|W[c, :]|^(1-α)

α = 0.5 splits the difficulty evenly; higher α (0.75–0.85) is what the paper needed for the most outlier-heavy models like OPT-175B. SmoothQuant reports near-lossless W8A8 across OPT/BLOOM-class models with roughly 1.5x latency improvement and half the memory versus FP16.

The runtime trick is the good part: diag(s)⁻¹ never executes as a kernel. In a pre-norm transformer, the tensor feeding q/k/v and gate/up is an RMSNorm output, and RMSNorm's affine weight is an elementwise per-channel multiply after normalization. Divide that weight by s and the scaling is free and exact. The normalization statistic is unaffected because you changed the affine, not the input.

Two structural constraints follow directly:

Every linear consuming the same norm output must share one s. q_proj, k_proj, v_proj all read the same tensor, so you take the elementwise max over their weight ranges.
down_proj's input has no preceding norm — it comes from act(gate) * up. Fold s⁻¹ into up_proj's output channels instead. That is safe only because up_proj's output feeds nothing but the elementwise multiply. Never fold into a projection whose output also lands on a residual add.

How do you profile outliers and fold the scales?

Collect per-channel absmax over a calibration set, then apply the fold in place:

import collections
import torch

@torch.no_grad()
def collect_channel_absmax(model, batches, targets=("q_proj", "gate_proj", "down_proj")):
    """Per-input-channel absmax of the activations entering each target Linear."""
    stats = collections.defaultdict(lambda: None)
    hooks = []

    def make_hook(name):
        def hook(_mod, inp, _out):
            x = inp[0].detach()
            m = x.abs().reshape(-1, x.shape[-1]).amax(dim=0).float()
            stats[name] = m if stats[name] is None else torch.maximum(stats[name], m)
        return hook

    for name, mod in model.named_modules():
        if isinstance(mod, torch.nn.Linear) and any(t in name for t in targets):
            hooks.append(mod.register_forward_hook(make_hook(name)))
    for b in batches:
        model(**b)
    for h in hooks:
        h.remove()
    return dict(stats)


def outlier_ratio(absmax, x_sample):
    """absmax / p99.9 — above ~10 means per-tensor INT8 will not survive."""
    p999 = torch.quantile(x_sample.abs().flatten().float(), 0.999)
    return (absmax.max() / p999).item()


@torch.no_grad()
def smooth_norm_linears(norm, linears, act_absmax, alpha=0.5):
    """Fold diag(s)^-1 into the norm affine, diag(s) into the linear weights."""
    w_absmax = torch.stack([
        l.weight.abs().amax(dim=0) for l in linears   # [Cin] per linear
    ]).amax(dim=0).clamp(min=1e-5)

    s = (act_absmax.clamp(min=1e-5).pow(alpha) / w_absmax.pow(1 - alpha)).clamp(min=1e-5)
    s = s.to(norm.weight.dtype).to(norm.weight.device)

    norm.weight.div_(s)
    if getattr(norm, "bias", None) is not None:
        norm.bias.div_(s)
    for l in linears:
        l.weight.mul_(s.view(1, -1))

Two things this code makes concrete. First, act_absmax is a calibration artifact — if your calibration data does not cover the real input distribution, you underestimate s on some channels and those channels clip in production. Use a few hundred sequences drawn from actual traffic, including your longest prompts and your tool-call-heavy turns, not just WikiText. Second, verify layer-by-layer output MSE after folding; a silent nan from a zero-range weight channel is the classic bug, which is why every term is clamped.

How do you pick alpha without wrecking the weights?

Search it per layer against output error, not globally against perplexity.

α is a dial between two failure modes. Too low and activations stay spiky. Too high and you dump so much range onto weights that per-channel INT8 weights start clipping, which is worse because weight error is systematic rather than per-token noise. Sweep α ∈ {0.5, 0.6, 0.7, 0.8, 0.85} per decoder layer and keep the value minimizing ‖Y_fp16 − Y_int8‖² on calibration activations. This is the same objective AWQ optimizes for weight-only quantization, where activation magnitude is used to identify salient weight channels and scale them up before rounding.

Do not tune on perplexity. Perplexity is dominated by high-frequency tokens and hides precision loss beautifully. Outlier damage shows up first in exact-copy behavior: long identifiers, JSON field names, base64, digit sequences, multi-hop retrieval from mid-context. Build your quantization eval out of those.

When is W8A8 worth it at all?

Only when you are compute-bound. This is the decision most teams skip.

Small-batch decode is memory-bandwidth-bound on weight loads. Quantizing activations buys nothing; the GEMMs are skinny. Use W4A16 (AWQ/GPTQ, group 128) and get a near-4x cut in weight traffic.
Prefill and large-batch serving are compute-bound and arithmetic-intensive. Here INT8/FP8 tensor-core throughput is the point, and W8A8 earns its complexity.
KV cache is a separate axis problem with its own asymmetry between keys and values; do not assume a W8A8 recipe transfers to it.
MoE routers should stay in higher precision. They are tiny, and a rounding flip in router logits changes expert assignment, which is a discrete, non-recoverable error.

Does FP8 make activation outliers a non-issue?

Mostly, and for a specific reason: E4M3 spends four bits on the exponent, so its dynamic range spans several orders of magnitude. An outlier channel 100x above the bulk no longer forces the bulk toward zero — it just costs you mantissa precision. That is why per-tensor FP8 on Hopper-class hardware behaves so much better than per-tensor INT8, and why the standard FP8 recipe (per-token activations, per-channel weights) is usually near-lossless without any smoothing pass.

Blocked formats push this further and, in effect, solve the problem in hardware. MXFP4 attaches an E8M0 scale to every block of 32 values; NVFP4 uses blocks of 16 with an E4M3 scale. Those blocks run along the reduction axis — precisely the per-channel scaling that INT8 GEMMs cannot express, made legal because the hardware applies the block scale inside the MMA pipeline. The outlier problem was never about bit width. It was about scale granularity on the wrong axis.

Even so, 4-bit blocked formats still want a smoothing/rotation pass (SmoothQuant-style scaling or a Hadamard rotation à la QuaRot/SpinQuant) to spread outlier energy before rounding. The axis constraint relaxes; the distribution problem does not fully disappear.

Direct answer

W8A8 INT8 quantization needs SmoothQuant because large language models concentrate activation magnitude in a few fixed hidden channels, and the channel axis is the one axis an INT8 GEMM cannot scale — a per-input-channel scale sits inside the int32 accumulator's reduction, so it cannot be factored out the way per-token and per-output-channel scales can. One outlier channel therefore sets the scale for the entire tensor and quantizes the meaningful values into two or three effective bits. SmoothQuant sidesteps the constraint algebraically: divide activations by a per-channel s, multiply the corresponding weight rows by s, and fold the division into the preceding RMSNorm affine so it costs nothing at inference. Tune α per layer against output MSE, calibrate on real traffic, and check whether you are compute-bound before doing any of it — if you are bandwidth-bound at small batch, weight-only 4-bit is the better trade, and on FP8 or NVFP4 hardware the wide exponent and block-level scaling absorb most of the outlier damage for you.