Every LLM team hits the same fork eventually, either keep steering the prompt or start training the model. Here's how to read the signals before you burn a sprint finding out the hard way.
Originally published at overmindlab.ai.
Prompt engineering changes what you send the model. Fine-tuning changes the model's weights. Start with prompts because they are fast, cheap and reversible. Fine-tune when the eval stops improving, the task is narrow, and you have a few thousand real examples of the output you want.
Most teams building LLM applications start the same way. They write a system prompt, test it against a few examples, and ship. Prompt engineering is cheap, fast, and reversible. Fine-tuning used to be something only teams with a training pipeline and a GPU budget could afford. Things have changed but the order of operations hasn't.
Both techniques steer model behaviour. Prompting changes what you say to the model. Fine-tuning changes the model itself.
Prompt engineering vs fine-tuning, side by side
| Prompt engineering | Fine-tuning | |
|---|---|---|
| What changes | The input | The model's weights |
| Where it lives | The context window | The model |
| Cost | Seconds, no infrastructure | GPU time and a training pipeline |
| Data needed | None, or a handful of examples | Thousands of labelled runs |
| Reversibility | Instant | Only by retraining |
| Survives a model swap | Yes | No |
| Ceiling | Plateaus quickly | High, bounded by data quality |
Why prompting is the right first move
Prompt engineering means editing what enters the context window. No adjustment of the model's weights. You're changing the input and reading the output.
For a new agent, that's almost always the correct starting point. You learn what the model can already do before you spend anything on training. Most of the behaviour changes you want, prompting will give you.
How to tell which regime you're in
This isn't a one-time decision so much as a loop. You stay in prompt-engineering mode until the eval curve goes flat, then check whether you actually have what fine-tuning needs, a narrow task and real data.
Ship prompt changes until the eval goes flat, then fine-tune.
| Where you are | What you're seeing | What to do next |
|---|---|---|
| New behaviour needed | A gap between what the agent does and what it should do | Ship a prompt change |
| Eval still moving | Each change shifts the score | Keep prompting |
| Eval flat for 3-4 changes, task is narrow, a few thousand labelled runs in hand | Plateau, and you have the ingredients | Fine-tune |
| Eval flat, but the task is broad or the labelled runs aren't there | Plateau without the ingredients | Build the dataset first |
Where prompting hits its ceiling
Prompting always runs out of road eventually, and the symptoms show up in predictable order:
- The plateau. Each prompt tweak moves the eval less.
- Whack-a-mole. The instruction that fixes today's failure mode causes tomorrow's.
- Prompt bloat. The system prompt keeps growing, slows the model down, and gets partially ignored.
- Paying rent. You're re-sending the same correction on every single call.
At that point, prompting stops being cheap. Engineering hours are the priciest line item in most LLM pipelines, and it's easy to burn a month nudging a prompt that plateaued weeks ago.
When to fine-tune
Fine-tuning updates a model's weights on examples of the behaviour you want until the model does it by default, without being told every time. It's worth trying when these conditions are met:
- The task is narrow. Fine-tuning sharpens a model on one job, not general capability.
- You have real data. A few thousand labelled runs, ideally sampled from production traffic rather than synthetic examples.
- Cost or latency matters. A smaller, fine-tuned model can outperform a much larger general-purpose one carrying a bloated prompt, at a fraction of the inference cost.
Deciding whether to fine-tune an open-weight model or keep renting a frontier one? See open-weight vs frontier models.
Where RAG fits
Retrieval-augmented generation (RAG) is the third option, and it solves a different problem. Prompting and fine-tuning both shape how the model behaves. RAG changes what the model knows at the moment it answers, by fetching documents and putting them in the context window.
Use RAG when the answer depends on facts that change, such as product docs, policies or customer records. Use fine-tuning when the model already has the knowledge it needs but gets the format, tone or decision wrong. Fine-tuning is a poor way to teach facts that change next week, because every change means another training run.
Many production systems use all three. RAG supplies the facts, a short prompt supplies the per-request instructions, and a fine-tuned model supplies the behavior.
Can you use prompt engineering and fine-tuning together?
Yes, and most teams should. A fine-tuned model still takes a prompt. The difference is what the prompt has to carry. Before fine-tuning, the system prompt holds every rule and every correction, and you pay to resend it on every call. After fine-tuning, the behavior lives in the weights and the prompt can shrink to the instructions that change per request.
Keep your eval set fixed across both stages. Score the prompted baseline, train, then score the fine-tuned model on the same set. That is the only fair way to tell whether the training run was worth it. How to freeze the eval set before you train.
Prompt, RAG or fine-tune? Worked examples
| Problem | Start with | Why |
|---|---|---|
| Replies are too long or the wrong tone | Prompt engineering | One instruction fixes it, and you can undo it instantly |
| The agent needs this week's pricing or policy | RAG | The facts change faster than you could retrain |
| A classification or extraction step runs on every request | Fine-tuning | Narrow task, high volume, and production gives you the examples |
| The system prompt keeps growing and each fix breaks something else | Fine-tuning | You have hit the prompting ceiling |
| A new agent with no traffic yet | Prompt engineering | You have no data to fine-tune on |
How Overmind handles both
On the prompt side, the Overmind Optimiser has your coding agent rewrite prompts, tool descriptions and logic as real git diffs, replays each candidate over your dataset and scores it with your evaluators. You apply the winning diff yourself. On the fine-tuning side, you build a dataset from production traces, fine-tune an open-weight base model with LoRA or full supervised training, and benchmark the result with the same evaluators before it serves traffic. The weights are downloadable.
FAQ
What is the difference between prompt engineering and fine-tuning?
Prompt engineering changes the input you send to a model and leaves the model untouched. Fine-tuning trains the model on examples of the behavior you want, which changes its weights so it behaves that way without being told. Prompting is instant and reversible. Fine-tuning takes data and GPU time, and you undo it only by retraining.
Is fine-tuning better than prompt engineering?
Neither is better in general. Prompting is the right first move for almost every new agent. Fine-tuning wins on narrow, high-volume tasks once prompting has plateaued and you have real examples to train on. A smaller fine-tuned model can then beat a larger prompted one at a fraction of the inference cost.
How much data do you need to fine-tune?
A few thousand labeled examples is a sensible starting point, ideally sampled from production traffic rather than written synthetically. Quality matters more than volume. Types of fine-tuning covers which method fits which amount of data.
What is prompt tuning?
Prompt tuning is not the same as prompt engineering. It trains a small set of soft prompt vectors that are prepended to the input, while the model's own weights stay frozen. It sits between the two approaches. It needs training data like fine-tuning, but changes far fewer parameters.
Is fine-tuning still worth it?
Yes, for narrow tasks at volume. Frontier models keep getting better at everything, but most production agents do a handful of jobs over and over. A small model trained on those jobs can match or beat a much larger one on them, at a fraction of the cost per call, and you own the weights.
Does fine-tuning replace the prompt?
No. A fine-tuned model still takes a prompt, but a much shorter one. The rules you used to repeat on every call move into the weights.
Overmind is the model training platform for AI teams. It turns your production traces into specialised models you own. Get started.
Top comments (0)