TL;DR: Most teams reach for fine-tuning when they should be tightening their retrieval stack. Anthropic does not currently offer general-purpose fine-tuning for Claude - prompt caching, RAG, and context engineering carry it. OpenAI's fine-tuning is useful for narrow style and format control, less useful for domain knowledge injection. Open-weight fine-tuning (Llama 3, Mistral, Qwen) unlocks real domain adaptation but adds infra cost most teams underestimate. When in doubt: RAG first, fine-tune only when RAG has a measurable ceiling.
Fine-tuning is the most misused word in production LLM engineering right now. Half the teams asking for it need a better retrieval stack. A quarter need better prompts. The remaining quarter have a real fine-tuning problem and need to pick the right target model, and that choice depends on constraints most decision-makers do not see coming.
This is the version from the trenches. We ship production ML systems across Claude, OpenAI, Llama, Mistral, and Gemini stacks, most of them for teams that hire us specifically because they hit the ceiling of prompt engineering. The comparison below is what we actually recommend on scoping calls in 2026.
When to fine-tune vs when to RAG vs when to prompt-engineer
The decision tree is simple but rarely applied cleanly. Fine-tuning teaches a model style, format, and tone at a level prompt engineering cannot match consistently. RAG teaches a model what your domain knowledge is at a level fine-tuning cannot match economically. Prompt engineering is the baseline that determines whether either of the other two is worth trying.
Rules of thumb we use in production:
You want the model to sound like your brand voice or follow a very specific output format on every response → fine-tune
You want the model to answer using your product docs, support tickets, contracts, or any body of proprietary text → RAG
You want the model to reason over a specific analytical framework or make decisions using your rubric → prompt engineering, possibly with structured outputs
You want to shrink the model to fit on your infra or reduce inference cost → fine-tune a smaller open-weight model on high-quality outputs from a bigger one (distillation)
The model gets facts wrong on production tasks → almost always a RAG problem, not a fine-tuning problem
The mistake we see repeatedly is teams trying to inject domain knowledge through fine-tuning. It works, technically, but the economics rarely favour it. Fine-tuning a knowledge cutoff into a model means retraining every time your knowledge changes. RAG updates the moment you update the source. On changing knowledge, RAG always wins on total cost of ownership.
Claude and the Anthropic stack
Anthropic does not currently expose general-purpose fine-tuning for Claude the way OpenAI does. This surprises teams when they first plan a Claude project. The reasoning from Anthropic is roughly that most fine-tuning use cases are better served by prompt engineering plus RAG in Claude specifically because Claude handles very long contexts well and follows instructions in ways that reduce the need for fine-tuning.
In practice, that is largely correct for the workloads we run. Claude Opus and Sonnet with a well-designed system prompt, prompt caching for the invariant parts of the context, and a properly built retrieval stack cover most of what teams reach for fine-tuning to solve. The one gap is heavy structural output constraints where you need the model to always emit a specific schema, in which case tool use and structured output modes carry it.
Prompt caching is the piece most teams underuse. If your system prompt plus retrieved context is 40k tokens and stays largely stable across a session, prompt caching drops the per-request cost by 90% on the cached portion. Combined with the extended thinking mode on newer Claude models, this is often the pattern that turns a project from unaffordable to production-ready.
For teams that do need Claude-specific adaptation beyond prompt engineering, Anthropic's enterprise offerings include prompt-engineering support and evaluation frameworks. That is not fine-tuning in the technical sense, but for most production use cases it produces the same outcome cheaper.
OpenAI and GPT fine-tuning
OpenAI offers fine-tuning across the GPT-4o, GPT-4o-mini, and GPT-3.5-turbo families. The API is mature, the pricing is transparent, and the fine-tuning cost has come down over the past two years to a level where it is genuinely economical for narrow style and format tuning.
Where OpenAI fine-tuning shines: consistent output style across thousands of requests, structural format enforcement that goes beyond what tool-use covers, and voice or tone matching a brand's specific characteristics with minimum drift. A hundred to a few hundred well-labelled examples typically suffice for style and format.
Where OpenAI fine-tuning is oversold: domain knowledge injection. Fine-tuning a GPT-4o on your product docs feels like teaching the model your business. In practice, the fine-tuned model develops confident wrong answers on facts adjacent to but not in the training data. RAG produces measurable improvements on the same task with source-attribution as a free by-product. If a customer support agent needs to cite the specific doc it used, RAG makes that trivial and fine-tuning makes that impossible.
One useful pattern with OpenAI fine-tuning that works: distil a mini or Nano model from GPT-4o outputs to run high-volume inference at a fraction of the cost. This is the case where fine-tuning economics dominate. The distilled model is not as capable as GPT-4o on hard tasks, but it is fast and cheap on the narrow task you distilled it for.
Open-weight fine-tuning: Llama, Mistral, Qwen
Fine-tuning a Llama 3 8B or 70B model, a Mistral Small or Large, or a Qwen 2.5 variant is the option that gives the most control and the most infrastructure cost. LoRA and QLoRA make training cheap - a single 80GB A100 can fine-tune Llama 3 8B in a few hours - but inference infrastructure is where the total cost of ownership lives.
Cases where open-weight fine-tuning wins: data residency requirements that forbid US-cloud LLM providers, on-premise deployment for regulated industries, ultra-high-volume inference where API pricing becomes prohibitive, and specialised domains where you need to teach the model concepts that pre-training did not include (specific medical, legal, or scientific specialties).
Cases where it does not win: teams without ML engineering resources for the deployment. The training is the easy part. The inference stack - VLLM or SGLang for serving, request batching, monitoring, model versioning, drift detection - is where the total cost lives, and most teams underestimate it by an order of magnitude.
Our recommendation: default to Claude or GPT via API for most use cases. Move to open-weight fine-tuning when specific constraints force it - compliance, scale, or capability gaps. Do not move to open-weight because it feels like more control. Control comes with cost.
RAG done right beats fine-tuning done poorly
Retrieval-augmented generation is where most production LLM value actually lives. Teams get it wrong in predictable ways.
The most common mistake: chunking documents by fixed character count and embedding each chunk. This produces retrieval that returns bits of context out of sequence, half a sentence at a time. The model then hallucinates the connective tissue. Good chunking respects semantic boundaries - paragraphs, sections, complete thoughts - and often uses overlapping chunks so no boundary is a knowledge cliff.
The second most common: using pure vector search when the query has strong keyword signal. Hybrid search - vector similarity plus BM25 keyword score with a fusion step - beats vector alone on most real workloads. Cohere's Rerank or Voyage's rerank models add another meaningful lift for a small marginal cost.
The third: no evaluation loop. A RAG system that works on your ten test queries and quietly fails on the ninetieth production query is worse than useless. Build a evaluation dataset of two hundred to a thousand real user queries and their correct answers, and re-run it on every retrieval stack change. Without this, the team is optimising in the dark.
Multi-model orchestration: when to route
In our video-generation and content pipelines, we run multi-model orchestration - route each stage of the pipeline to the model that owns that step. Brief interpretation to Claude for reasoning, storyboard to a vision model, image generation to the model with the best fidelity on the current subject, product-fidelity cleanup to a specialist model. Total cost per output is lower and quality per stage is higher than running one model end-to-end.
This pattern applies beyond video. Any pipeline with distinct steps benefits from routing. The overhead is a thin orchestration layer that tracks which model handled which step and lets you swap models per step as new versions land.
Cost math that matters in production
The pricing conversation on scoping calls always ends up in the same place. Let us do the math cleanly with realistic numbers.
Cost per one million tokens (input) in 2026 - approximate:
Claude Haiku 4.5: cheapest tier, fast, good for high-volume routine tasks
Claude Sonnet: mid-tier, the workhorse for most production work
Claude Opus: highest capability, expensive, use for hard reasoning stages
GPT-4o-mini: comparable to Sonnet, sometimes slightly cheaper
GPT-4o: comparable to Opus, priced accordingly
Llama 3 70B on Together AI or Groq: cheaper than Sonnet at API tier, more expensive if you self-host at low volume
Prompt caching changes the math for Claude specifically. If your invariant context is 30k tokens and you serve 1000 requests per day, the cached tier drops the cost by 90% on that portion. This alone can be the difference between a project being economical and not.
Fine-tuning has a training cost (one-off) and an inference cost (per-request, higher than base model). For narrow style tasks with high volume, the inference math works out favourably. For domain knowledge with modest volume, RAG on a base model always wins on total cost.
What we actually recommend
For most production LLM projects we scope in 2026, the recommendation is: Claude Sonnet or GPT-4o via API, prompt caching where invariants exist, a proper RAG stack for domain knowledge, and evaluation infrastructure from day one. Fine-tuning enters the conversation only when a specific ceiling is hit and there is clear evidence that fine-tuning would move it.
For compliance-constrained workloads: Llama 3 or Mistral with LoRA fine-tuning, self-hosted, is the answer. The engineering cost is real and needs to be part of the project scope.
For narrow style and format control at scale: OpenAI fine-tuning is the tightest option. A hundred to five hundred labelled examples, a single training run, and consistent output.
For anything scoring, decisioning, or touching unit economics: LLMs stay in supporting roles - drafting, summarising, explaining - and the decision layer sits on auditable ML. This is not an LLM story at all, and we cover it in the iGaming retention playbook linked above.
Building a production LLM system and unsure whether fine-tuning is the right lever? Book a call. We scope honestly - if RAG or prompt engineering would deliver the same outcome cheaper, we tell you before quoting fine-tuning work. See /services/ml for the full stack we work with.
Originally published at 2pizza.team. We build AI and automation systems for small teams - fixed price, two to six weeks. See the work.
Top comments (0)