SFT Loss Masking: Why Your Fine-Tuned LLM Never Emits EOS

Your SFT run converged. Loss curve is clean, eval loss tracks train loss, the model answers the question correctly. Then it keeps going — invents a follow-up user turn, answers that too, and grinds to max_tokens every single time.

The model learned your task. It never learned to stop. In almost every case I've debugged, the cause is SFT loss masking: the end-of-turn token was present in input_ids but set to -100 in labels, so it contributed exactly zero gradient.

TL;DR

• SFT loss masking bugs delete the EOS signal. The most common one: setting tokenizer.pad_token = tokenizer.eos_token, then masking labels with labels[labels == pad_token_id] = -100. That masks every real end-of-turn token too.
• Your loss curve cannot detect this. EOS is 1 token out of ~600 in a typical assistant response. Removing it moves mean cross-entropy by ~0.001 — far under run-to-run noise.
• Truncation is the second cause, and it's biased. max_seq_length chops the trailing turn-end token off exactly the longest examples, teaching "long answer → no stop."
• Half the time the weights are fine and the config is wrong. Trained on <|eot_id|> but shipped a generation_config.eos_token_id that only lists <|end_of_text|> → the model emits a stop token nobody is listening for.
• Diagnose with one number: P(turn-end token) at the final position of a held-out reference response. Healthy checkpoints put nearly all mass there. Broken ones put ~none.

Why does a fine-tuned LLM never emit EOS?

Because emitting EOS is a learned behavior like any other, and gradient only flows through positions where labels != -100. Instruction tuning masks the prompt so the model isn't trained to generate user turns. The turn-end token sits at the boundary between "assistant content" and "everything after," which is precisely where off-by-one errors and pad-token collisions live.

Modern chat models have two distinct stop tokens. On Llama 3, <|end_of_text|> (128001) ends a document; <|eot_id|> (128009) ends a turn. Qwen's chat template ends turns with <|im_end|>, not <|endoftext|>. Chat-tuned behavior depends on the turn-end token, and that's the one templates append and buggy collators eat.

There's an opposite failure worth naming, because people confuse the two. Train on the full sequence with no masking at all and the model learns to produce user turns — it stops, then immediately opens a new <|start_header_id|>user<|end_header_id|> block and talks to itself. Self-conversation means "no masking." Endless single-turn rambling means "EOS masked."

Why does pad_token = eos_token silently break SFT loss masking?

Because the label mask is built by comparing token IDs, and after that assignment the pad ID and the EOS ID are the same integer. Every genuine turn-end token in your dataset matches the pad filter and gets deleted from the loss.

Here's the pattern, which appears in a huge number of training scripts:

# The bug. tok.pad_token was None, so someone did this:
tok.pad_token = tok.eos_token          # pad_token_id == eos_token_id  (e.g. 128009)

batch = tok(texts, padding=True, truncation=True,
            max_length=2048, return_tensors="pt")
labels = batch["input_ids"].clone()
labels[labels == tok.pad_token_id] = -100   # <-- also masks every real EOS
batch["labels"] = labels

Fix it by masking on position, not on token identity. The attention mask already knows which positions are padding:

# The fix: pad positions come from attention_mask, never from token id.
labels = batch["input_ids"].clone()
labels[batch["attention_mask"] == 0] = -100
batch["labels"] = labels

A real padding token is cleaner still — most modern checkpoints ship reserved specials you can claim (<|finetune_right_pad_id|> on Llama 3.x, <|reserved_special_token_N|> elsewhere) so pad_token_id != eos_token_id and this class of bug becomes impossible. Do not add a brand-new token to the vocabulary just for padding; that forces an embedding resize with its own failure modes.

For prompt masking, prefer the tokenizer over string matching. apply_chat_template(..., return_assistant_tokens_mask=True) gives you exact assistant spans — but only if the template contains a {% generation %} block. Many community templates don't, and you get an all-zero mask, which trains on nothing. Check it. Response-template string matching (find the <|start_header_id|>assistant marker, mask everything before) is the fallback, and it fails silently when your template string doesn't tokenize identically in context.

The other half of that fallback is the slice bound:

# Off-by-one: exclusive end drops the turn-end token from supervision.
labels[start:end] = input_ids[start:end]        # end == index of <|eot_id|>  -> wrong
labels[start:end + 1] = input_ids[start:end + 1]  # include the turn-end token -> right

Why doesn't the training loss show the bug?

Because EOS is one token in several hundred, and mean cross-entropy averages it away. Take a 600-token assistant response with mean per-token loss 0.85 and a well-learned turn-end token at loss 0.05. Include it: (0.85 × 599 + 0.05) / 600 ≈ 0.8487. Exclude it: 0.85. The difference is 0.0013 — an order of magnitude below the seed-to-seed variance of the same run.

This is why the bug survives review. Every dashboard signal is green. The metric that would catch it — per-token loss at the final position of each response — is not on anyone's dashboard.

The same argument explains why the bug is hard to fix by training harder. Even with correct masking, the stop decision gets one gradient contribution per example. If 15% of your examples have a corrupted or missing turn-end token, you're not training a slightly worse stop behavior — you're training a genuinely ambiguous one, and at sampling temperature the model will happily take the "keep going" branch.

How do you check whether EOS is actually in your labels?

Two checks. First, before training, decode what you're supervising. Second, after training, measure the stop probability directly.

# 1) Pre-flight: does the supervised region end with the turn-end token?
EOT = tok.convert_tokens_to_ids("<|eot_id|>")
n_missing = n_trunc = 0
for ex in dataset.select(range(500)):
    ids, labels = ex["input_ids"], ex["labels"]
    sup = [t for t in labels if t != -100]
    if not sup or sup[-1] != EOT:
        n_missing += 1
    if len(ids) >= MAX_LEN:
        n_trunc += 1
print(f"missing EOS in labels: {n_missing}/500   truncated: {n_trunc}/500")
print("supervised tail:", tok.decode(sup[-16:]))   # eyeball it once

# 2) Post-train: probability the model assigns to stopping at the true end.
import torch
EOT = tok.convert_tokens_to_ids("<|eot_id|>")

def p_stop(messages):
    ids = tok.apply_chat_template(messages, tokenize=True, return_tensors="pt").to(model.device)
    ids = ids[:, :-1]                      # drop the gold turn-end; model must predict it
    with torch.no_grad():
        logits = model(ids).logits[0, -1].float()
    return torch.softmax(logits, -1)[EOT].item()

Run p_stop over 50 held-out reference responses and look at the distribution, not the mean. In healthy chat checkpoints this sits close to 1.0 for nearly every example. In checkpoints with a masking bug I've seen it collapse to the 1e-3 range — the token is not merely deprioritized, it's effectively unreachable under nucleus sampling, which is why the symptom is 100% reproducible rather than intermittent.

Why does truncation teach the model to ramble?

Because truncation removes the turn-end token from exactly the examples where stopping is hardest to learn. truncation=True, max_length=2048 cuts the tail off the longest responses in your dataset. Those are the multi-step derivations, the long code blocks, the detailed reports — and every one of them now ends mid-sentence with no stop supervision.

The model generalizes the correlation it was shown: short answers end, long answers don't. Which is the exact behavior you observe in production, on the exact prompts that matter.

Filter instead of truncate. Drop examples that exceed max_length (and log how many), or raise the limit. A dropped example teaches nothing; a truncated one teaches something false. If you're packing sequences, make sure the packer preserves the turn-end token as the document separator and supervises it — a packer that strips separators reintroduces the same hole.

Is it the model or the generation config?

Check the config first — it's a one-line fix and it's wrong surprisingly often. The model can emit <|eot_id|> perfectly while the runtime ignores it, because generation_config.eos_token_id was inherited from the base model and lists only <|end_of_text|>.

from transformers import GenerationConfig
gc = GenerationConfig.from_pretrained(CKPT)
print(gc.eos_token_id)     # want BOTH: [128001, 128009] for Llama 3 chat

# vLLM: stop_token_ids does not inherit from your fine-tune's chat template.
params = SamplingParams(max_tokens=1024,
                        stop_token_ids=[tok.convert_tokens_to_ids("<|eot_id|>"),
                                        tok.eos_token_id])

The discriminator is trivial: generate with max_tokens=200 and inspect raw token IDs, not decoded text (skip_special_tokens=True hides the evidence). If 128009 appears in the output and generation continued past it, your weights are fine and your config is broken. If it never appears, you have a masking or truncation bug.

Papering over this with stop strings is a trap. It hides the symptom while leaving the distribution broken, and the damage resurfaces where no stop string helps: run-on JSON in structured output, an extra hallucinated turn appended after a tool call, and RL rollouts that never terminate and burn your entire generation budget on garbage tails.

The short answer

A fine-tuned LLM never emits EOS when the end-of-turn token exists in input_ids but not in labels — most often because pad_token = eos_token collided with an ID-based label mask, because a slice bound was exclusive, or because max_length truncation cut the token off the longest examples. Mean cross-entropy cannot show you this: one token in six hundred moves the loss by ~0.001. Verify SFT loss masking directly — decode the supervised tail of a few hundred examples and confirm it ends with your turn-end token, then measure P(turn-end) at the final position of held-out responses after training. If the model does emit the token and generation continues anyway, the weights are fine and your generation_config.eos_token_id or vLLM stop_token_ids is missing it.