Left Padding vs Right Padding: Why Batched LLM Generation Breaks

You batch eight prompts through model.generate() to speed up an eval. The longest prompt produces the same output it did at batch_size=1. The shortest one produces an empty string. The rest produce text that is plausible but subtly worse. Your accuracy drops four points and you start blaming the model.

The model is fine. You padded on the wrong side. Right padding breaks batched LLM generation in decoder-only models in a way that the attention mask cannot repair, and it fails loudest on your shortest sequences — which are usually the ones you least suspect.

TL;DR

• In decoder-only generation, generate() reads next-token logits from the last tensor position (logits[:, -1, :]). With right padding, that position is a pad token, so you sample the continuation of <pad>, not of your prompt.
• The attention mask stops tokens from attending to pads. It does not stop pads from being the query that produces your output logits. Masking is not a fix.
• Right padding is correct in training and in log-likelihood scoring because you never read logits at a pad position — labels there are -100. Generation is the only phase that reads the last slot unconditionally.
• Fix: tokenizer.padding_side = "left", always pass attention_mask, and let the model derive position_ids from attention_mask.cumsum(-1) - 1. Use max_new_tokens, never max_length.
• Engines like vLLM and TGI never hit this because they run ragged batches (cu_seqlens / paged block tables) with no intra-sequence padding at all.

Why does right padding break batched LLM generation?

Because generation is positional, not semantic. A causal LM produces one logit vector per input position, and the decoding loop wants the distribution for "what comes after the prompt." It takes that from the final slice of the sequence dimension. With a right-padded batch:

row 0 (long):  [t0 t1 t2 t3 t4]          -> logits[:, -1] is after t4   ✅
row 1 (short): [t0 t1 <pad> <pad> <pad>] -> logits[:, -1] is after <pad> ❌

Row 1 samples a token conditioned on a pad. If your pad_token_id is eos_token_id — the default workaround everyone applies to Llama- and Mistral-family tokenizers — the model has just been shown an end-of-turn marker and will very reasonably emit another one. That is your empty generation.

It gets worse on step two. The new token is appended at the end of the tensor, so row 1's context becomes [t0 t1 <pad> <pad> <pad> x0]. The pads are still masked, so x0 attends to t0, t1, x0 — but x0 was itself sampled from garbage, and every subsequent token inherits it. There is no recovery: a single bad first token derails the whole continuation.

HuggingFace transformers actually warns about this:

A decoder-only architecture is being used, but right-padding was detected! For correct generation results, please set padding_side='left' when initializing the tokenizer.

It is a warning, not an error, and it scrolls past in eval logs.

Doesn't the attention mask handle the padding?

No, and this is the part people get wrong. The attention mask is a key-side mask. It zeroes out pad columns so real tokens cannot attend to pads. It says nothing about which rows (queries) you read.

Padding correctness in a transformer needs three separate things:

1. Keys: pads must be excluded from attention — the mask does this.
2. Positions: real tokens must get contiguous position indices — position_ids does this.
3. Readout: you must read logits at a real token position — only padding side does this.

Right padding gets (1) and (2) right and (3) catastrophically wrong. Left padding gets all three right, because after left padding the last position is always the last real prompt token, for every row in the batch, regardless of length.

That uniformity has a nice side effect: with left padding, outputs[:, input_ids.shape[1]:] slices off exactly the prompt for every row. With right padding, the prompt boundary is per-row and you have to reconstruct it from the mask.

Why is right padding fine during training but not generation?

The asymmetry is entirely about which logits you consume.

In supervised fine-tuning you set labels to -100 at pad positions. Cross-entropy skips them. The pad positions still get computed — wasted FLOPs — but their outputs never touch the loss. Causal masking guarantees real tokens can't see future pads. So the gradient is identical to an unpadded run (up to float non-associativity).

Log-likelihood scoring behaves the same way. Harnesses that score multiple-choice options gather logits at specific real-token indices, so right padding is harmless there too.

The rule, stated once: right padding is safe exactly when you never read a logit at a padded position. Training and scoring satisfy that. generate() does not, because it reads slot -1 unconditionally.

Do position_ids matter if the model uses RoPE?

Yes, but less than you'd guess for RoPE models — and enormously for learned absolute embeddings.

RoPE rotates queries and keys by an angle proportional to position, and the resulting attention logit for a query at position $m$ and key at position $n$ depends only on $m - n$. Add a constant offset to every position in a row and the relative geometry is unchanged. So if you left-pad and forget position_ids, a RoPE model with a per-row constant shift is mathematically nearly neutral — you'll see small numerical drift from different rotation angles, not a broken output.

Models with learned absolute position embeddings (GPT-2, OPT) have no such invariance. The embedding at index 3 is a trained vector; shifting your prompt by 40 pad slots means you look up 40 different vectors. Outputs degrade immediately.

Two more reasons not to rely on RoPE's forgiveness:

• Sliding-window attention. Window boundaries are defined over positions. If an implementation derives the window from tensor index rather than position id, left pads eat into the window and clip real context on your short rows.
• Position-dependent scaling. YaRN/NTK-style long-context scaling makes rotation behavior a function of absolute position bucket. Large offsets push short prompts into a regime they weren't tuned for.

Here is what a correct manual decode loop looks like — this is also what prepare_inputs_for_generation does for you:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

MODEL = "meta-llama/Llama-3.1-8B-Instruct"
tok = AutoTokenizer.from_pretrained(MODEL)
tok.padding_side = "left"                     # non-negotiable for generation
if tok.pad_token is None:
    tok.pad_token = tok.eos_token             # then rely on the mask, not the id

model = AutoModelForCausalLM.from_pretrained(
    MODEL, torch_dtype=torch.bfloat16, device_map="cuda"
)

prompts = ["Explain KV cache eviction.", "Hi."]   # deliberately lopsided
batch = tok([tok.apply_chat_template([{"role": "user", "content": p}],
                                     tokenize=False, add_generation_prompt=True)
             for p in prompts],
            return_tensors="pt", padding=True, add_special_tokens=False).to("cuda")

# Positions must count only real tokens: 0,0,0,0,1,2 -> pads get a dummy index.
position_ids = batch["attention_mask"].long().cumsum(-1) - 1
position_ids.masked_fill_(batch["attention_mask"] == 0, 1)

out = model.generate(
    **batch,
    position_ids=position_ids,
    max_new_tokens=256,          # NOT max_length: that counts pads
    do_sample=False,
)
completions = tok.batch_decode(out[:, batch["input_ids"].shape[1]:],
                               skip_special_tokens=True)

Two traps embedded in that snippet. First, max_length counts the padded prompt, so a heavily padded row silently gets fewer new tokens than a short one — pass max_new_tokens. Second, add_special_tokens=False after apply_chat_template; otherwise you get a duplicate BOS, which is its own separate accuracy leak.

What does left padding cost you?

Memory and a little throughput, not correctness.

Pad tokens are still materialized in the KV cache. A batch of 32 where one prompt is 4k tokens and the rest are 200 pads to 4k, so you allocate roughly 32×4k cache slots to hold ~4k+31×200 real ones. At batch_size=32 on a 70B-class model that is real HBM. The mitigation is length bucketing: sort by token count and batch similar lengths together. That alone often buys more throughput than raising the batch size.

Why doesn't vLLM have this problem?

Because it doesn't pad. vLLM's paged attention keeps a per-sequence block table and every sequence is exactly its own length. FlashAttention's varlen kernels take a cu_seqlens prefix-sum and treat the batch as one ragged buffer. There is no pad token in the KV cache, no shared sequence-length axis, and no ambiguity about where the last real token is — the scheduler tracks it per sequence.

This is why "the same model scores worse in my HF eval loop than through my vLLM server" is such a common report. Both are running the same weights. One of them is asking the model to continue a pad token.

Note that left padding still won't give you bitwise-identical logits to batch_size=1. Reduction order in batched GEMMs and attention changes with batch shape, so greedy decoding can diverge on a near-tie. That is float non-associativity, a completely separate phenomenon from the padding bug — don't let one mask the other.

How do I verify my batching is correct?

One test, thirty seconds, and it catches every variant of this bug:

solo = [model.generate(**tok(p, return_tensors="pt").to("cuda"),
                       max_new_tokens=64, do_sample=False)
        for p in prompts]
# vs. the batched call above, greedy, same prompts

Decode both and diff. Correct batching gives you outputs that agree for many tokens and diverge only at genuine near-ties. A padding bug gives you divergence at token zero on the shortest prompt — and only on the shortest prompt. That signature (fails short, passes long, scales with length spread inside the batch) is diagnostic.

Add it as a unit test with a deliberately lopsided batch: one 2k-token prompt and one three-word prompt. Uniform-length test batches hide the bug perfectly.

Left padding vs right padding: the direct answer

Batched LLM generation breaks with right padding because decoder-only decoding reads next-token logits from the last position of the sequence tensor, and right padding puts a pad token there for every row shorter than the batch maximum. The attention mask only prevents attention to pads; it cannot stop a pad from being the query whose logits you sample, so the first generated token for short rows is conditioned on padding — frequently eos, hence empty outputs — and every later token inherits that corruption. Left padding guarantees the final position is a real token for all rows, and combined with an attention_mask and position_ids derived from attention_mask.cumsum(-1) - 1, it is exactly equivalent to unbatched generation up to float non-associativity. Right padding remains correct for training and log-likelihood scoring, where pad-position logits are never consumed. Set padding_side="left" for anything that calls generate(), use max_new_tokens instead of max_length, bucket by length to reclaim the wasted KV cache, and prefer a ragged-batch engine like vLLM if you want the problem to be structurally impossible.