Fine tuning remains the most reliable path to align an open-source model with proprietary data, tone, and task structure. While retrieval-augmented generation covers dynamic facts, fine tuning hardcodes behavior, reduces prompt length, and cuts latency. This guide walks through the practical decisions that separate experimental notebooks from production pipelines: when to tune, which method to use, how to curate data, and where to run inference economically.
When to Fine Tune
Retrieval-augmented generation (RAG) is the right tool when answers depend on a changing knowledge base. Fine tuning is the better investment when you need to enforce a consistent output format, internalize proprietary logic, or eliminate lengthy few-shot prompts that consume tokens on every request. A simple heuristic: if you are embedding hundreds of examples in the context window to steer style, move that knowledge into the weights instead.
Fine Tuning Methods
Full fine-tuning updates every parameter. It produces the strongest alignment but requires significant GPU memory and risks catastrophic forgetting. Most production teams should start with parameter-efficient fine-tuning (PEFT).
LoRA injects trainable low-rank matrices into attention layers while freezing the base model. It cuts memory use by roughly an order of magnitude and lets you swap adapters at serving time.
QLoRA pushes efficiency further by loading the base weights in 4-bit precision and using paged optimizers. A single 48 GB GPU can tune a 70B parameter model, which makes architectures like Llama 3.3 70B accessible without a cluster.
After supervised fine-tuning (SFT), you can run preference alignment. RLHF requires a reward model and is complex to stabilize. Direct Preference Optimization (DPO) reuses the same dataset format, pairs chosen and rejected completions, and often reaches comparable alignment with less engineering overhead.
Dataset Construction
Model performance is bounded by data quality, not data volume. A few hundred meticulously cleaned examples usually outperform tens of thousands of noisy records.
Structure each example as a conversation array with roles: system, user, and assistant. Keep system prompts identical across the dataset if you want the model to learn a fixed persona. Remove duplicates, strip PII, and verify that outputs match your tone guidelines.
Split aggressively. Reserve 10 to 20 percent of examples for validation and do not let training examples leak into eval. If your eval loss plateaus while training loss drops, your dataset is either too small or already memorized.
Training Code with LoRA and QLoRA
The following snippet fine-tunes a Llama 3.3 70B base model with QLoRA using Hugging Face TRL and PEFT. The same pattern works for other popular bases such as Qwen 3 32B.
import os
import torch
from datasets import load_dataset
from transformers import (
AutoModelForCausalLM,
AutoTokenizer,
TrainingArguments,
BitsAndBytesConfig,
)
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training
from trl import SFTTrainer
model_id = "meta-llama/Llama-3.3-70B"
bnb_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_compute_dtype=torch.bfloat16,
bnb_4bit_quant_type="nf4",
bnb_4bit_use_double_quant=True,
)
model = AutoModelForCausalLM.from_pretrained(
model_id,
quantization_config=bnb_config,
device_map="auto",
torch_dtype=torch.bfloat16,
)
model = prepare_model_for_kbit_training(model)
peft_config = LoraConfig(
r=64,
lora_alpha=16,
target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
lora_dropout=0.05,
bias="none",
task_type="CAUSAL_LM",
)
model = get_peft_model(model, peft_config)
tokenizer = AutoTokenizer.from_pretrained(model_id)
tokenizer.pad_token = tokenizer.eos_token
dataset = load_dataset("json", data_files="train.jsonl", split="train")
training_args = TrainingArguments(
output_dir="./lora-output",
per_device_train_batch_size=1,
gradient_accumulation_steps=4,
num_train_epochs=3,
learning_rate=2e-4,
bf16=True,
logging_steps=10,
optim="paged_adamw_8bit",
save_strategy="epoch",
)
trainer = SFTTrainer(
model=model,
tokenizer=tokenizer,
train_dataset=dataset,
max_seq_length=2048,
args=training_args,
)
trainer.train()
model.save_pretrained("my-adapter")
tokenizer.save_pretrained("my-adapter")
After training, merge the adapter back into the base weights if you need a single consolidated checkpoint for serving. Keep the adapter separate if you plan to serve multiple specializations behind one base model.
Evaluation
Training loss is a poor proxy for usefulness. Build a held-out evaluation set that contains tricky inputs representative of production traffic. Score outputs with exact match, embedding similarity, or a rubric-based LLM judge.
If you use an LLM judge, select a high-capability reasoning model. Oxlo.ai hosts options such as DeepSeek R1 671B MoE and Kimi K2.6 that you can call through the same OpenAI-compatible endpoint to grade completions at scale.
Watch for overfitting. When eval loss diverges upward while training loss continues to fall, reduce epochs, increase dropout, or add regularization.
Deploying to Production Inference
Fine tuning changes the model, but it does not change the economics of serving. Most providers bill by the token, so long system prompts, tool definitions, and multi-turn agent traces raise costs unpredictably. Specialized agents often make this worse by carrying larger contexts.
Oxlo.ai offers a different structure. Its request-based pricing charges one flat cost per API call regardless of prompt length. For long-context and agentic workloads, this can be significantly cheaper than token-based alternatives. The platform is fully OpenAI SDK compatible, requires no code changes beyond the base URL, and delivers no cold starts on popular models.
You can use Oxlo.ai for base model fallbacks, LLM-as-a-judge pipelines, or the final inference layer:
import os
import openai
client = openai.OpenAI(
base_url="https://api.oxlo.ai/v1",
api_key=os.environ["OXLO_API_KEY"],
)
response = client.chat.completions.create(
model="llama-3.3-70b",
messages=[
{"role": "system", "content": "You are a specialist trained on internal legal contracts."},
{"role": "user", "content": "Summarize the indemnification clause."},
],
stream=True,
)
for chunk in response:
print(chunk.choices[0].delta.content or "", end="")
For teams running multiple fine-tuned variants, Oxlo.ai supports 45+ open-source and proprietary models, including Qwen 3 32B for multilingual agent workflows and DeepSeek V4 Flash for contexts up to 1M tokens. See https://oxlo.ai/pricing for plan details.
Conclusion
Fine tuning is not a single script. It is a pipeline of data curation, efficient training, honest evaluation, and economical serving. Start with QLoRA on a small, high-quality dataset, measure rigorously, and only scale up once eval metrics improve.
For the serving layer, choose infrastructure that aligns costs with business value. Oxlo.ai's request-based pricing and OpenAI-compatible API remove the tax on long prompts, making it a strong option for production inference of specialized models.
Top comments (0)