A fine-tuning job that fails almost always fails at file validation, before a single gradient step. The message names a line number and a field, and the fault is nearly always one of six things: invalid JSON on one line, a missing or misnamed key, an unexpected role, an empty assistant message, an example over the per-example token limit, or too few examples.
Where jobs actually fail
| Stage | Description |
|---|---|
| Upload | File too large, wrong purpose or MIME type, or a truncated upload. Fails in seconds. |
| Validation | Structural checks on every line. Where most failures happen, and the only stage that gives you a line number. Free — no compute is billed. |
| Queue | Not a failure, though it looks like one. Jobs can sit for a long time; check the job status field before assuming anything is wrong. |
| Training | Infrastructure faults and out-of-memory conditions. Rare on hosted services, common when you run it yourself — see CUDA out of memory. |
| Completed and worse | Not an error at all, and the most expensive outcome. The last section is about this. |
Reading the validation message
Validation errors are usually specific, and the specificity is wasted if you skim them. Wording varies by provider; the recurring shapes are:
- A line number and a field. Line numbers are one-based and refer to the file as uploaded. If you edited the file after uploading, you are reading the wrong line.
- Invalid JSON on line N. Nearly always a trailing comma, a real newline inside a string, unescaped quotes, or a pretty-printed object spanning several lines. JSONL is one complete object per line; a formatter that pretty-printed the file has destroyed it.
- Missing or unexpected key. The conversational format expects a
messagesarray; some services also accept a prompt-completion form. Mixing the two formats in one file fails, and so does using the format the documentation deprecated. - Invalid role. Roles are drawn from a small allowed set.
bot,ai,humanandAssistantwith a capital letter all fail. - Example exceeds the token limit. There is a maximum per training example, separate from the model’s context window, and examples over it are either rejected or silently truncated — truncation being the worse outcome, since it trains on a cut-off answer.
- Too few examples. Services impose a minimum, often around ten. Below it the job will not start.
Minimum counts, per-example token limits and file size caps differ by provider and change. Read them from the current documentation for the service you are using; the linter below takes them as parameters rather than hard-coding values that would go stale.
The linter to run first
Every check below is one a hosted validator performs, and running them locally turns a queued rejection into a five-second failure with all the faults listed at once rather than the first one only.
#!/usr/bin/env python3
"""Lint a chat-format JSONL fine-tuning file. Reports everything, not
the first fault. Pass a real tokeniser for accurate token counts."""
import json, sys
from collections import Counter
ROLES = {"system", "user", "assistant", "tool"}
MAX_TOKENS = 16384 # per example; check YOUR provider's current limit
MIN_EXAMPLES = 10
def lint(path, count=lambda s: len(s) // 4): # crude default; replace it
problems, labels, lengths, seen = [], Counter(), [], set()
n = 0
with open(path, encoding="utf-8") as f:
for i, line in enumerate(f, 1):
if not line.strip():
problems.append(f"{i}: blank line"); continue
if i == 1 and line.startswith("\ufeff"):
problems.append("1: file starts with a UTF-8 BOM")
try:
ex = json.loads(line)
except json.JSONDecodeError as e:
problems.append(f"{i}: invalid JSON: {e.msg} at col {e.colno}")
continue
n += 1
msgs = ex.get("messages")
if not isinstance(msgs, list) or not msgs:
problems.append(f"{i}: missing or empty 'messages'"); continue
if extra := set(ex) - {"messages", "tools", "parallel_tool_calls"}:
problems.append(f"{i}: unexpected top-level keys: {sorted(extra)}")
roles = [m.get("role") for m in msgs]
for r in roles:
if r not in ROLES:
problems.append(f"{i}: invalid role {r!r}")
if roles[-1] != "assistant":
problems.append(f"{i}: last message is {roles[-1]!r}, "
f"not 'assistant' — nothing to learn from")
if not any(r == "assistant" for r in roles):
problems.append(f"{i}: no assistant message")
for j, m in enumerate(msgs):
c = m.get("content")
if m.get("role") == "assistant" and not (c or "").strip() \
and not m.get("tool_calls"):
problems.append(f"{i}: assistant message {j} is empty")
if c is not None and not isinstance(c, (str, list)):
problems.append(f"{i}: message {j} content is {type(c).__name__}")
text = " ".join(m.get("content") or "" for m in msgs
if isinstance(m.get("content"), str))
t = count(text)
lengths.append(t)
if t > MAX_TOKENS:
problems.append(f"{i}: ~{t} tokens, over the {MAX_TOKENS} limit")
key = json.dumps(msgs, sort_keys=True)
if key in seen:
problems.append(f"{i}: exact duplicate of an earlier example")
seen.add(key)
last = msgs[-1].get("content")
if isinstance(last, str):
labels[last.strip()[:60]] += 1
if n < MIN_EXAMPLES:
problems.append(f"only {n} examples; the minimum is {MIN_EXAMPLES}")
print(f"{n} examples, {len(problems)} problems")
for p in problems[:50]:
print(" ", p)
if lengths:
lengths.sort()
print(f"tokens: min {lengths[0]} median {lengths[len(lengths)//2]} "
f"max {lengths[-1]}")
print("most frequent final messages (label balance):")
for label, c in labels.most_common(5):
print(f" {c:>6} ({c/max(n,1):5.1%}) {label!r}")
return 1 if problems else 0
if __name__ == "__main__":
sys.exit(lint(sys.argv[1]))
Two of those checks are worth calling out because no hosted validator performs them. The duplicate check catches the copy-paste that silently overweights an example. And the label distribution printed at the end is the subject of the next section.
The format that trains on the wrong thing
A file that validates can still be teaching something you did not intend, and the two ways that happens are both about which tokens the loss is computed over.
Multi-turn examples. In a conversation with four assistant turns, most services compute loss over every assistant message by default. If you built the file from production transcripts and only the final answer was reviewed, you are training on three unreviewed answers as well as the good one. Where a service supports a per-message weight, set the earlier assistant turns to zero. Where it does not, split the conversation into separate examples so that each one ends at the turn you actually want learned.
# One conversation, four assistant turns, only the last one reviewed.
# Option A — weights, where supported:
{"messages": [
{"role": "user", "content": "..."},
{"role": "assistant", "content": "...", "weight": 0},
{"role": "user", "content": "..."},
{"role": "assistant", "content": "the reviewed answer", "weight": 1}
]}
# Option B — truncate the conversation so it ends at the good turn.
# The field name and its support vary by provider; check before relying
# on it, and fall back to option B if it is silently ignored.
The system prompt. Whatever appears in the training examples is what the model is adapted to. Three common mistakes: no system message in the file but one in production, a different system message in each, or a very long system message repeated in every example — which inflates the token count you are billed for training on, sometimes by more than the content itself. If the system prompt is fixed, the argument for including it is that it matches serving; the argument against is cost. Pick one deliberately and keep the file and production consistent either way.
Both faults share a signature after training: the model behaves well on inputs that resemble the file exactly and poorly on anything else, which reads as overfitting and is really a formatting mismatch.
The failure that is not an error
A file can pass every validator and still be the reason the job was not worth running. Class imbalance is the usual culprit: if 88% of your examples end in the same label, a model that always predicts that label scores 88% and has learned nothing you wanted. It will look acceptable on an aggregate accuracy number and be useless on the cases you built it for.
- Print the distribution before training, not after. The linter does this. If the majority class is above roughly 70%, deal with it first — class imbalance covers rebalancing, and dataset balancing covers doing it without throwing data away.
- Check for leakage into the validation split. Near- duplicates split across train and validation produce a validation loss that looks excellent and means nothing. Deduplicate before splitting, not after.
- Check formatting consistency. If half the assistant responses end with a full stop and half do not, or half are JSON and half are prose, you are training the inconsistency in.
- Check that the system prompt matches production. A model fine-tuned with one system prompt and served with another has been trained on a distribution it will not see.
When the job runs and the model is worse
The job succeeded, the loss curve looks fine, and the model is worse than the base model on your task. This is common enough to expect, and it is usually one of four things: too few examples for the behaviour you are trying to teach, too many epochs on a small set so the model memorised rather than generalised, a learning rate too high, or the task being one that fine-tuning does not address at all — fine-tuning teaches format and style far more readily than it teaches facts.
Two habits make this recoverable. Hold out a test set the training job never saw and evaluate the fine-tuned model against the base model on the same set, with the same prompt. And check the validation loss curve for the point where it stopped falling while training loss kept falling — that gap is overfitting, and the fix is fewer epochs or more data. Fine-tuning failures and evaluating a fine-tuned model cover both, and whether to fine-tune at all is worth re-reading if the answer keeps being no.
Top comments (0)