You can fine tune an open source LLM on your own data to bend a general-purpose model into a task‑specific specialist that often beats prompting and retrieval‑augmented generation on narrow, repetitive work. At techpotions, we’ve run this play on everything from internal support tickets to medical‑coding rulesets. The pattern rarely changes: define the job, prep a dataset crystal‑clear enough that the model can’t dodge it, pick a tuning method that matches your compute budget, and obsess over a quantitative score that actually captures success. Here’s the full playbook.
When fine‑tuning actually pays off
Fine‑tuning is not a better RAG; it’s a different tool. Use it when the task is so tight that a prompt can’t reliably capture the desired behaviour, or when you need the model to internalize a large body of proprietary style, logic, or classification patterns that would blow out a prompt’s context window.
- Prompt engineering works for one‑off transformations, light formatting, and simple extraction.
- RAG shines when facts change faster than you retrain, or when you need up‑to‑the‑minute references.
- Fine‑tuning dominates when the pattern is stable, the training data is plentiful, and latency + cost matter — no prompt injection, no retrieval step, just a raw model call that already thinks the way you need it to.
A litmus test: if you find yourself writing system prompts longer than 200 words just to steer the model, or if you’re post‑processing outputs with regex and heuristic fixes, you’re already funding the technical debt that fine‑tuning can clear.
How to fine‑tune an open source LLM on your own data
1. Nail the task definition before touching a GPU
If you can’t write a labeler’s guide, you can’t fine‑tune. The model will only be as consistent as the examples you give it. We force every project into a single sentence: “Given X, produce Y, exactly like Z.”
For example, when we recently tuned Llama‑3.1‑8B to classify incoming legal briefs by jurisdiction and urgency, the spec was: “Return JSON with keys jurisdiction (one of civil/criminal/administrative) and urgency (high/medium/low), never explain.” That rigid output contract let us automate evaluation later.
2. Build a dataset that teaches, not just memorizes
A dataset built by copy‑pasting existing outputs teaches the model to mimic, not to think. We aim for 500–2000 curated examples, each an input‑output pair that forces the model to apply the rule, not recall the answer.
| Property | What to do |
|---|---|
| Coverage | Include edge cases, empty inputs, and contradictory‑seeming prompts that the rule still resolves. |
| Consistency | No two examples should contradict the task definition. Run a script that samples your data and checks for label mismatches. |
| Format | Use a clean instruction‑response format (Alpaca, ChatML, or the model’s native tokeniser template). |
| Diversity | Vary sentence length, punctuation, and typos — real world data is messy. |
| Separation | Hold out a fixed 10–15% for testing. Never train on it. |
A small naming convention: we store everything as a Hugging Face Dataset and version it in git‑LFS. Reproducibility is free.
3. Pick the right tuning lever: LoRA, QLoRA, or full fine‑tune
LoRA is where you start unless you have a clear reason not to. It adds small trainable adapters while freezing the base weights, letting a single 24 GB GPU handle 7–8B parameter models without breaking a sweat.
# Minimal LoRA config for Llama‑3.1‑8B using PEFT
from peft import LoraConfig, get_peft_model
lora_config = LoraConfig(
r=16, # rank – higher captures more nuance, costs more VRAM
lora_alpha=32, # scaling factor
target_modules=["q_proj", "v_proj"], # attention projection layers
lora_dropout=0.05,
bias="none",
task_type="CAUSAL_LM"
)
model = get_peft_model(base_model, lora_config)
| Method | VRAM needed (8B model) | Quality plateaus | When to choose |
|---|---|---|---|
| LoRA (r=16) | ~14 GB | After ~2 epochs | Default starting point |
| QLoRA (4‑bit) | ~8 GB | Same as LoRA with slightly noisier convergence | Consumer GPUs, large batch experiments |
| Full fine‑tune | ~140 GB (bf16) | Highest possible fidelity to the dataset | If you need every fraction of a point on a hard metric and have the budget |
Full fine‑tuning only makes sense when the data volume is in the tens of thousands and the task requires the model to unlearn something fundamental. We’ve done full runs on 70B models for enterprise generative‑AI deployments, but for most teams, QLoRA on a rented A100 gives 95 % of the result at 5 % of the cost.
4. Train with a loss that matches your output shape
Language‑modeling loss (cross‑entropy on every token) will dilute your signal if the output is short and structured. When we fine‑tuned Mistral‑7B v0.3 for a closed‑domain QA system, we masked the loss so it only computed over the assistant’s answer tokens, completely ignoring the prompt tokens. That one change lifted our exact‑match score by 11 points.
# Attention mask trick: mark user tokens as -100 so they are ignored in loss
def tokenize_function(examples):
tokenized = tokenizer.apply_chat_template(
examples["messages"], tokenize=True, add_generation_prompt=False
)
# The template already marks assistant tokens with a different role; many libraries
# let you pass a `loss_on_response_only` flag – enable it.
return tokenized
Use libraries like TRL’s SFTTrainer with loss_on_response_only=True. It works out of the box for Alpaca‑style datasets.
5. Evaluate: if you can’t measure it, you didn’t improve it
The model’s own loss curve is a liar. It will tell you the model is memorizing the training set beautifully while generating nonsense on unseen data. You need a test‑set metric that directly mirrors the business goal.
- For extraction or structure tasks: exact‑match on a JSON field, or Levenshtein distance with a tolerance.
- For classification: macro‑F1 across the target classes, weighted by business impact.
- For generation: a judging pipeline (gpt‑4 or a simpler model) that scores adherence to the spec on a 1‑5 scale. We built an auto‑eval loop that calls the fine‑tuned model and the baseline with the same test prompts and runs a deterministic check; anything fuzzy gets flagged for human review.
A rule we will not compromise: if fine‑tuning doesn’t beat a 5‑shot prompted baseline by at least 10 % relative improvement on your key metric, the model isn’t ready. We’ve shipped AI products that started at 72 % accuracy and left the oven only at 93 %. That gap is the entire value of the exercise.
Connecting it to your stack
Once you’ve fine‑tuned an open source LLM on your own data, you can deploy it through vLLM, TGI, or a simple FastAPI container — no prompt‑engineering loop required. At techpotions, we instrument these deployments with logging that captures input‑output pairs for the next training cycle, creating a flywheel where the model gets smarter the more it’s used.
If you are staring at a proprietary dataset and a task that keeps you up at night, start a conversation — we’ll help you decide whether fine‑tuning is the right bet and, if it is, how to get to a provable win without burning a quarter of your runway.
FAQ
Do I need a huge dataset to fine‑tune an open source LLM on my own data?
Not if you use parameter‑efficient methods. We’ve seen compelling results with as few as 300 carefully curated examples when combined with LoRA and a strong base model. Quality and consistency matter far more than quantity.
How do I prevent the model from forgetting everything else it knew?
LoRA and QLoRA inherently limit catastrophic forgetting because only adapters are updated. If you do full fine‑tuning, you can mix in a small portion of general‑domain data (1‑5 % of the batch) to keep the model grounded. Monitor performance on a handful of generic prompts before and after training.
Can I fine‑tune a model and still use RAG together?
Absolutely. Fine‑tune the model for a specific output format or reasoning style, then feed retrieved documents into the fine‑tuned prompt template. The combination often outperforms either technique alone, especially when the language must conform to strict regulatory phrasing.
Top comments (0)