A generic fine-tuning playbook, written after doing it wrong several times

Every fine-tuning guide I read before my first serious attempt was a tutorial about knobs: learning rates, LoRA ranks, quantization settings. None of them covered the part that actually decides whether the project succeeds, which happens before and after the training run, not during it. This is the playbook I now follow for any model on any task. It is deliberately generic: the same sequence has carried me through vision-language models on driving data, small text models for domain QA, and RL-style preference tuning, on hardware ranging from a single consumer GPU to rented cloud boxes.

Step 0: try not to fine-tune

Fine-tuning is the most expensive intervention in the stack, so it goes last. The ladder, cheapest first:

1. Better prompting. A system prompt with three good few-shot examples routinely closes half the gap that people reach for fine-tuning to close. It costs an afternoon.
2. Retrieval. If the failure is missing knowledge rather than missing behavior, RAG beats weights. Knowledge changes; your fine-tune will not.
3. A bigger or different base model. Run the comparison honestly: a zero-shot larger model against your imagined fine-tuned smaller one. I have watched an untouched open-weights model beat an in-domain fine-tune on the fine-tune's own benchmark. It happens more often than leaderboards suggest.
4. Fine-tune. Only when the behavior you need is demonstrably not in the base model and cannot be retrieved or prompted in: output format compliance, domain-specific reasoning patterns, a persona that must survive thousands of turns, latency budgets that force a small model.

Write down, in one sentence, which failure of steps 1 to 3 justifies the fine-tune. If you cannot write that sentence, stop. That sentence also becomes your evaluation target later, which is the real reason to write it.

Step 1: build the evaluation before the dataset

This ordering feels backwards and is the single highest-leverage decision in the playbook.

Before collecting training data, build a held-out evaluation that measures the sentence from step 0, and run the base model through it. That number is your baseline, and it does three jobs. It tells you the true size of the gap. It occasionally kills the project on the spot because the base model was already good enough and nobody had measured it. And it validates the harness itself: an evaluation that has never scored a known model is untested code that emits numbers.

Two rules for the eval that I no longer break:

• The test split is designed before training, and nothing from training may touch it. Not for hyperparameter selection, not for checkpoint picking, not once. Use a validation split for those. If your splits share source documents, scenes, or sessions with the training set, you are measuring memorization and calling it generalization.
• Run a sanity control. Delete the input and measure again. If a vision model scores far above chance with the images removed, your benchmark leaks answers through priors and question phrasing, and every score it has produced is inflated. The same control exists for text: shuffle the context, drop the retrieved passages, feed the question alone. Cheap to run, devastating when it fires, and better fired at you than at a reviewer.

Step 2: data is the model

The dataset decides what you get. The recipe decides only how efficiently you get it.

• A few thousand excellent examples beat a hundred thousand scraped ones for behavior tuning. For format compliance and persona, hundreds can be enough with LoRA.
• Deduplicate against your evaluation. Near-duplicates leak. Exact-match dedup is not sufficient; hash at the level the data actually repeats (documents, scenes, sessions).
• Audit a random hundred by hand. Not the first hundred, a random hundred. The first hundred were curated by whoever built the file; the random hundred tell you the truth about label noise. Every bad label teaches the model confidently.
• Match the training distribution to the inference distribution. If production inputs will be messy, OCR-damaged, or truncated, train on that, not on the clean version.

Expect data work to consume more than half the project's wall-clock. When it does not, that is usually a sign it was skipped, and the bill arrives later, denominated in GPU-hours.

Step 3: choose the lightest recipe that can express the change

• LoRA or QLoRA is the default. Behavior shaping, format compliance, domain adaptation, personas: adapters handle all of it at a fraction of the memory, and the artifact is small enough to version, ship, stack, and roll back.
• Full fine-tuning is for when the change is deep: new modalities, new tokenizers, or when adapters measurably plateau below target. Prove the plateau before paying for the parameters.
• Preference or RL methods (DPO, GRPO and relatives) are for objectives that supervised examples cannot express: relative quality, verifiable rewards, multi-step outcomes. They are also where silent failure lives, so they come with an extra rule: track a metric that measures the actual objective, not a proxy. I once watched token accuracy sit at 99 percent while the policy learned the opposite of the intended behavior. The proxy was fine; the behavior was not.

One decision that outranks the recipe choice: change one variable at a time. A run that changes base model, dataset, rank, and learning rate simultaneously produces a result that cannot be attributed to anything. Reviewers catch this in papers; production catches it in incidents. If you must move fast, move fast serially.

Step 4: engineer the run like it will be interrupted, because it will be

Training runs die. Disks unmount, drivers hiccup, a colleague's job lands on your GPU, spot instances vanish. The runs that survive share the same boring infrastructure:

• Checkpoint on a fixed cadence and verify resume actually works before the long run, not during the outage. A checkpoint you have never resumed from is a hope, not a checkpoint.
• Log to an experiment tracker, not a terminal. Scalars, config, git commit, environment. The question you will ask in three weeks is "what exactly produced this file", and scrollback does not answer it.
• Emit progress with the failure count first. A silent job that is working and a silent job that is failing on every batch look identical from outside. If the error count is climbing, you want to see it in the first minute, not after the run completes.
• Pin the environment. Container images beat requirements files; requirements files beat memory. Half of my hardest debugging sessions were two libraries disagreeing about something as small as position ids, and the fix was environmental, not algorithmic.

Step 5: evaluate like an adversary, then decide

When training finishes, resist the demo. Run the same held-out evaluation from step 1, and read three numbers together, never one:

1. Target metric versus base model. Did the gap from step 0 actually close, with the improvement larger than your seed-to-seed noise? If you have not measured seed noise, run the fine-tune twice before believing any margin smaller than a point or two. Single-seed margins evaporate embarrassingly often.
2. Regression suite. What did the model lose? General capability regressions are the default outcome of narrow fine-tuning, not the exception. A small fixed battery of out-of-domain checks is enough to catch the worst of it.
3. Train-versus-held-out gap. Large gap: you memorized; get more data or regularize. No gap and no improvement: capacity or recipe; move up the ladder from step 3.

Then make the deployment decision with the same honesty as step 0: is the fine-tuned model better than the best non-fine-tuned alternative, at the quality, latency, and cost that production actually needs? Sometimes the answer is no, and the fine-tune becomes a lesson rather than a deployment. That outcome is not a failure of the playbook. Discovering it for the price of one training run instead of one production incident is the playbook working.

The one-page version

1. Prompt, retrieve, or upgrade the base model first; fine-tune last, and write the sentence that justifies it.
2. Build the eval and score the base model before touching training data. Design the splits before training. Run the deleted-input control.
3. Spend most of the project on data: dedup against eval, audit random samples, match production distribution.
4. Default to adapters; escalate only on proven plateaus; one variable per run.
5. Checkpoint, track, and emit errors-first progress; pin the environment.
6. Judge with three numbers: target delta versus noise, regressions, generalization gap. Then decide like you have not already sunk the cost, because the model does not care that you did.