DEV Community: finetuning

Fine-Tuning LLMs with LoRA & QLoRA: Complete Production Guide

Phil IPS — Mon, 12 Oct 2026 21:48:24 +0000

Fine-Tuning LLMs with LoRA & QLoRA: Complete Practical Guide

Master parameter-efficient fine-tuning to adapt any open-source LLM to your specific domain, style, or task.

Introduction

Full fine-tuning requires massive compute. LoRA (Low-Rank Adaptation) and QLoRA (Quantized LoRA) change the game — fine-tune 7B-70B models on consumer GPUs.

Why LoRA/QLoRA?

Method	Trainable Params	VRAM (7B)	Quality	Hardware
Full FT	100% (7B)	~48 GB	Best	A100/H100
LoRA (r=64)	~0.5% (35M)	~16 GB	Excellent	RTX 3090/4090
QLoRA (r=64)	~0.5% (35M)	~8 GB	Excellent	RTX 3060 12GB+
LoRA (r=16)	~0.1% (7M)	~12 GB	Good	RTX 3060 12GB
QLoRA (r=16)	~0.1% (7M)	~6 GB	Good	RTX 3060 8GB

QLoRA = LoRA + 4-bit quantization + paged optimizers + gradient checkpointing

Prerequisites

# Core dependencies
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121
pip install transformers accelerate peft bitsandbytes trl datasets

# For quantization
pip install auto-gptq optimum

# Optional: Weights & Biases for logging
pip install wandb

Understanding LoRA

The Math

Instead of updating all weights $W \in \mathbb{R}^{d \times d}$, LoRA learns two small matrices:

$$
W' = W + \Delta W = W + BA
$$

Where:

$B \in \mathbb{R}^{d \times r}$ (down-projection)
$A \in \mathbb{R}^{r \times d}$ (up-projection)
$r \ll d$ (rank, typically 8-64)

Only $A$ and $B$ are trained. $W$ stays frozen.

Target Modules

Which layers to adapt? Research shows:

# For LLaMA/Mistral/Gemma architecture
TARGET_MODULES = [
    "q_proj", "k_proj", "v_proj", "o_proj",  # Attention
    "gate_proj", "up_proj", "down_proj"       # MLP
]

# For smaller models or less VRAM, use attention only
TARGET_MODULES_ATTENTION = ["q_proj", "k_proj", "v_proj", "o_proj"]

Data Preparation

Format: Instruction Tuning (Alpaca/ShareGPT)

// alpaca_format.json
[
  {
    "instruction": "Write a Python function to calculate fibonacci numbers",
    "input": "",
    "output": "def fibonacci(n):\n    if n <= 1:\n        return n\n    return fibonacci(n-1) + fibonacci(n-2)"
  },
  {
    "instruction": "Explain the concept of recursion",
    "input": "In the context of computer science",
    "output": "Recursion is a programming technique where a function calls itself..."
  }
]

Format: Conversation (ChatML)

// chatml_format.json
[
  {
    "messages": [
      {"role": "system", "content": "You are a helpful coding assistant."},
      {"role": "user", "content": "How do I handle errors in Go?"},
      {"role": "assistant", "content": "In Go, errors are values..."}
    ]
  }
]

Data Processing Pipeline

# data_prep.py
from datasets import Dataset, load_dataset
from transformers import AutoTokenizer
import json

def load_and_format_data(file_path: str, format: str = "alpaca") -> Dataset:
    with open(file_path) as f:
        data = json.load(f)

    if format == "alpaca":
        def format_alpaca(example):
            if example["input"]:
                prompt = f"### Instruction:\n{example['instruction']}\n\n### Input:\n{example['input']}\n\n### Response:\n{example['output']}"
            else:
                prompt = f"### Instruction:\n{example['instruction']}\n\n### Response:\n{example['output']}"
            return {"text": prompt}

        dataset = Dataset.from_list(data).map(format_alpaca)

    elif format == "chatml":
        def format_chatml(example):
            messages = example["messages"]
            # Apply chat template
            text = tokenizer.apply_chat_template(messages, tokenize=False)
            return {"text": text}

        dataset = Dataset.from_list(data).map(format_chatml)

    return dataset

def tokenize_dataset(dataset: Dataset, tokenizer, max_length: int = 2048):
    def tokenize(example):
        return tokenizer(
            example["text"],
            truncation=True,
            max_length=max_length,
            padding="max_length",
            return_tensors="pt"
        )

    return dataset.map(tokenize, batched=True, remove_columns=dataset.column_names)

# Train/val split
dataset = load_and_format_data("train.json")
dataset = dataset.train_test_split(test_size=0.1, seed=42)
train_dataset = dataset["train"]
eval_dataset = dataset["test"]

LoRA Fine-Tuning with Hugging Face TRL

Complete Training Script

# train_lora.py
import torch
from transformers import (
    AutoModelForCausalLM,
    AutoTokenizer,
    TrainingArguments,
    BitsAndBytesConfig
)
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training
from trl import SFTTrainer
from datasets import load_dataset
import os

# ============ CONFIGURATION ============
MODEL_ID = "meta-llama/Llama-3.2-3B-Instruct"  # Or mistralai/Mistral-7B-Instruct-v0.3
DATASET_PATH = "train.json"
OUTPUT_DIR = "./llama-3.2-3b-lora-finetuned"
MAX_LENGTH = 2048
BATCH_SIZE = 2
GRAD_ACCUM = 8  # Effective batch = 16
LEARNING_RATE = 2e-4
NUM_EPOCHS = 3
LORA_R = 64
LORA_ALPHA = 16
LORA_DROPOUT = 0.1

# ============ QUANTIZATION CONFIG (QLoRA) ============
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",        # NormalFloat 4-bit
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,   # Double quantization
)

# ============ LoRA CONFIG ============
lora_config = LoraConfig(
    r=LORA_R,
    lora_alpha=LORA_ALPHA,
    target_modules=[
        "q_proj", "k_proj", "v_proj", "o_proj",
        "gate_proj", "up_proj", "down_proj"
    ],
    lora_dropout=LORA_DROPOUT,
    bias="none",
    task_type="CAUSAL_LM",
    # For QLoRA: modules_to_save=["embed_tokens", "lm_head"]  # If extending vocab
)

# ============ LOAD MODEL & TOKENIZER ============
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID, trust_remote_code=True)
tokenizer.pad_token = tokenizer.eos_token
tokenizer.padding_side = "right"

model = AutoModelForCausalLM.from_pretrained(
    MODEL_ID,
    quantization_config=bnb_config,
    device_map="auto",
    trust_remote_code=True,
    torch_dtype=torch.bfloat16,
    attn_implementation="flash_attention_2",  # Requires flash-attn
)

# Prepare for k-bit training
model = prepare_model_for_kbit_training(model)
model = get_peft_model(model, lora_config)
model.print_trainable_parameters()
# Example output: trainable params: 35,651,584 || all params: 3,200,000,000 || trainable%: 1.11%

# ============ LOAD DATA ============
def formatting_func(example):
    return example["text"]

dataset = load_dataset("json", data_files=DATASET_PATH, split="train")
dataset = dataset.train_test_split(test_size=0.1, seed=42)

# ============ TRAINING ARGS ============
training_args = TrainingArguments(
    output_dir=OUTPUT_DIR,
    num_train_epochs=NUM_EPOCHS,
    per_device_train_batch_size=BATCH_SIZE,
    per_device_eval_batch_size=BATCH_SIZE,
    gradient_accumulation_steps=GRAD_ACCUM,
    learning_rate=LEARNING_RATE,
    lr_scheduler_type="cosine",
    warmup_ratio=0.03,
    logging_steps=10,
    eval_strategy="steps",
    eval_steps=100,
    save_strategy="steps",
    save_steps=100,
    save_total_limit=3,
    load_best_model_at_end=True,
    metric_for_best_model="eval_loss",
    bf16=True,
    tf32=True,
    gradient_checkpointing=True,
    gradient_checkpointing_kwargs={"use_reentrant": False},
    optim="paged_adamw_32bit",  # QLoRA optimizer
    report_to="wandb",  # or "tensorboard" or "none"
    run_name=f"llama-3.2-3b-lora-r{LORA_R}",
    seed=42,
    ddp_find_unused_parameters=False,
)

# ============ TRAINER ============
trainer = SFTTrainer(
    model=model,
    args=training_args,
    train_dataset=dataset["train"],
    eval_dataset=dataset["test"],
    formatting_func=formatting_func,
    max_seq_length=MAX_LENGTH,
    tokenizer=tokenizer,
    packing=True,  # Efficient packing of sequences
)

# ============ TRAIN ============
trainer.train()

# Save final model
trainer.save_model()
tokenizer.save_pretrained(OUTPUT_DIR)

print(f"Model saved to {OUTPUT_DIR}")

Launch Training

# Single GPU
python train_lora.py

# Multi-GPU with Accelerate
accelerate config  # First time setup
accelerate launch train_lora.py

# With DeepSpeed ZeRO-3 (for larger models)
accelerate launch --config_file ds_zero3.yaml train_lora.py

QLoRA Specific Optimizations

Paged Optimizers

# Automatically used with optim="paged_adamw_32bit"
# Moves optimizer states to CPU when not in use
# Enables larger models on same VRAM

Gradient Checkpointing

# In TrainingArguments:
gradient_checkpointing=True,
gradient_checkpointing_kwargs={"use_reentrant": False},

# Saves ~30% VRAM at cost of ~20% slower training

Flash Attention 2

# Install
pip install flash-attn --no-build-isolation

# Use in model loading
attn_implementation="flash_attention_2"

# 2-3x faster, less memory

Monitoring Training

Weights & Biases

# In TrainingArguments
report_to="wandb",
run_name="my-lora-run",

# Before training
import wandb
wandb.init(project="llm-finetuning", name="llama-3b-lora-r64")

TensorBoard

# Start tensorboard
tensorboard --logdir ./llama-3.2-3b-lora-finetuned/runs

# In TrainingArguments
report_to="tensorboard",

Key Metrics to Watch

Metric	Healthy Range	Warning Signs
train_loss	Decreasing steadily	Plateauing early, diverging
eval_loss	Decreasing, close to train	Increasing (overfitting)
learning_rate	Cosine decay to 0	Stuck, not decaying
grad_norm	< 1.0	> 10 (exploding gradients)
GPU memory	Stable	Growing (memory leak)

Merging & Exporting

Merge LoRA Weights

# merge_lora.py
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

BASE_MODEL = "meta-llama/Llama-3.2-3B-Instruct"
LORA_PATH = "./llama-3.2-3b-lora-finetuned"
OUTPUT_PATH = "./llama-3.2-3b-merged"

# Load base model in fp16/bf16 (not 4-bit)
base_model = AutoModelForCausalLM.from_pretrained(
    BASE_MODEL,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True,
)

# Load LoRA adapter
model = PeftModel.from_pretrained(base_model, LORA_PATH)

# Merge and unload
merged_model = model.merge_and_unload()

# Save merged model
merged_model.save_pretrained(OUTPUT_PATH)
tokenizer = AutoTokenizer.from_pretrained(BASE_MODEL)
tokenizer.save_pretrained(OUTPUT_PATH)

print(f"Merged model saved to {OUTPUT_PATH}")

Export to GGUF (for llama.cpp/Ollama)

# Install llama.cpp with Python bindings
pip install llama-cpp-python

# Convert using llama.cpp convert script
python llama.cpp/convert_hf_to_gguf.py \
  ./llama-3.2-3b-merged \
  --outfile ./llama-3.2-3b-finetuned.Q4_K_M.gguf \
  --outtype q4_k_m

# Or use llama.cpp quantize directly
./llama.cpp/quantize ./llama-3.2-3b-merged/model.fp16.gguf \
  ./llama-3.2-3b-finetuned.Q4_K_M.gguf q4_k_m

Create Ollama Modelfile

# Modelfile
FROM ./llama-3.2-3b-finetuned.Q4_K_M.gguf

TEMPLATE "{{ .System }}\n{{ .Prompt }}\n{{ .Response }}"

PARAMETER temperature 0.7
PARAMETER top_p 0.9
PARAMETER num_ctx 8192

# Create Ollama model
ollama create my-finetuned-model -f Modelfile

# Test
ollama run my-finetuned-model "Write a Python function for binary search"

Advanced: LoRA Hyperparameter Tuning

Rank (r) Selection

# Experiment with different ranks
RANK_CONFIGS = [
    {"r": 8, "alpha": 16},    # Minimal, fast, less capacity
    {"r": 16, "alpha": 32},   # Good balance
    {"r": 32, "alpha": 64},   # More capacity
    {"r": 64, "alpha": 16},   # High capacity (alpha often = r/4 to r/2)
    {"r": 128, "alpha": 32},  # Maximum capacity
]

Target Module Ablation

# Test different target module sets
TARGET_CONFIGS = {
    "attention_only": ["q_proj", "k_proj", "v_proj", "o_proj"],
    "mlp_only": ["gate_proj", "up_proj", "down_proj"],
    "all_linear": ["q_proj", "k_proj", "v_proj", "o_proj", 
                   "gate_proj", "up_proj", "down_proj"],
    "attention_mlp": ["q_proj", "v_proj", "gate_proj", "up_proj"],  # Key ones
}

Learning Rate Schedule

# For LoRA, higher LR often works (2e-4 to 1e-3)
# But depends on dataset size:
# Small dataset (<1k): 1e-4 to 5e-4
# Medium (1k-10k): 2e-4
# Large (>10k): 1e-4 to 2e-4

Evaluation & Testing

Quantitative Evaluation

# eval_model.py
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
import torch
from datasets import load_dataset
import json

def evaluate_model(model_path, test_data_path, base_model_id=None):
    if base_model_id:
        base = AutoModelForCausalLM.from_pretrained(
            base_model_id, torch_dtype=torch.bfloat16, device_map="auto"
        )
        model = PeftModel.from_pretrained(base, model_path)
    else:
        model = AutoModelForCausalLM.from_pretrained(
            model_path, torch_dtype=torch.bfloat16, device_map="auto"
        )

    tokenizer = AutoTokenizer.from_pretrained(model_path)

    # Load test data
    with open(test_data_path) as f:
        test_data = json.load(f)

    results = []
    for item in test_data[:50]:  # Sample 50
        prompt = f"### Instruction:\n{item['instruction']}\n\n### Response:\n"
        inputs = tokenizer(prompt, return_tensors="pt").to("cuda")

        with torch.no_grad():
            outputs = model.generate(
                **inputs,
                max_new_tokens=256,
                temperature=0.7,
                do_sample=True,
                pad_token_id=tokenizer.eos_token_id
            )

        response = tokenizer.decode(outputs[0][inputs.input_ids.shape[1]:], skip_special_tokens=True)
        results.append({
            "instruction": item["instruction"],
            "expected": item["output"],
            "generated": response
        })

    # Save for human evaluation
    with open("eval_results.json", "w") as f:
        json.dump(results, f, indent=2)

    return results

Qualitative Evaluation (Human/LLM Judge)

# judge_eval.py
from openai import OpenAI
import json

client = OpenAI()

def llm_judge(instruction, expected, generated):
    prompt = f"""Rate the quality of the generated response compared to expected.

Instruction: {instruction}
Expected: {expected}
Generated: {generated}

Rate 1-5 on:
1. Accuracy
2. Completeness  
3. Clarity
4. Relevance

Return JSON: {{"accuracy": 1-5, "completeness": 1-5, "clarity": 1-5, "relevance": 1-5, "reasoning": "..."}}"""

    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": prompt}],
        temperature=0
    )
    return json.loads(response.content)

# Run evaluation
with open("eval_results.json") as f:
    results = json.load(f)

scores = {"accuracy": [], "completeness": [], "clarity": [], "relevance": []}
for r in results:
    eval = llm_judge(r["instruction"], r["expected"], r["generated"])
    for k in scores:
        scores[k].append(eval[k])

for k, v in scores.items():
    print(f"{k}: {sum(v)/len(v):.2f}")

Common Issues & Solutions

Issue	Cause	Solution
OOM	Model too large	Increase gradient_accumulation, reduce batch_size, use QLoRA, gradient_checkpointing
Loss not decreasing	LR too low/high	Try 2e-4, check data format, verify tokenization
Overfitting	Too many epochs, small data	Reduce epochs, add dropout, more data, weight decay
Gibberish output	Wrong chat template	Match training format exactly
NaN loss	Unstable training	Reduce LR, add gradient clipping, check data
Slow training	No Flash Attention	Install flash-attn, use bf16

Cost Analysis

Training Cost (Estimates)

Model	Method	GPU	Time (1k samples, 3 epochs)	Cost (Cloud)
3B	LoRA	RTX 3090 (24GB)	~30 min	$0.50
3B	QLoRA	RTX 3060 (12GB)	~45 min	$0.30
7B	LoRA	RTX 3090 (24GB)	~1 hr	$1.00
7B	QLoRA	RTX 3090 (24GB)	~1.5 hr	$0.80
13B	QLoRA	2× RTX 3090	~3 hr	$3.00
70B	QLoRA	4× A100 (80GB)	~8 hr	$40+

Local training on owned hardware = $0 marginal cost

Best Practices Summary

Data

[ ] Quality > Quantity: 1k high-quality > 100k noisy
[ ] Deduplicate: Remove exact/near duplicates
[ ] Balance: Cover all target domains evenly
[ ] Format consistency: Match training format exactly at inference

Training

[ ] Start small: Test with r=16, 1 epoch first
[ ] Monitor eval_loss: Stop when it plateaus/increases
[ ] Save checkpoints: Every 100-500 steps
[ ] Use cosine LR: With warmup (3-5% of steps)
[ ] Pack sequences: packing=True in SFTTrainer

LoRA Config

[ ] r=64, alpha=16 for most cases
[ ] Target all linear layers for best quality
[ ] Dropout 0.05-0.1 for regularization
[ ] Don't quantize LoRA weights (keep in bf16/fp16)

Deployment

[ ] Merge weights for inference speed
[ ] Quantize merged model to GGUF Q4_K_M
[ ] Test inference with same prompt format
[ ] Benchmark latency before production

Complete Example: Domain Adaptation

# Complete workflow for adapting LLaMA-3.2-3B to medical QA
# 1. Collect 2k high-quality medical QA pairs
# 2. Format as Alpaca
# 3. Train with QLoRA r=64, 3 epochs
# 4. Evaluate on held-out medical questions
# 5. Merge, quantize to GGUF, deploy via Ollama

# Expected: 90%+ accuracy on medical specialty vs 60% base model

Resources

Fine-tuning is no longer reserved for big tech. With QLoRA, a $300 GPU can customize a 7B model to beat GPT-3.5 on your specific task. Start small, iterate fast, deploy locally.

Fine-Tuning LLMs with LoRA & QLoRA: Complete Practical Guide

Phil IPS — Mon, 12 Oct 2026 17:31:02 +0000

Fine-Tuning LLMs with LoRA & QLoRA: Complete Practical Guide

Master parameter-efficient fine-tuning to adapt any open-source LLM to your specific domain, style, or task.

Introduction

Full fine-tuning requires massive compute. LoRA (Low-Rank Adaptation) and QLoRA (Quantized LoRA) change the game — fine-tune 7B-70B models on consumer GPUs.

Why LoRA/QLoRA?

Method	Trainable Params	VRAM (7B)	Quality	Hardware
Full FT	100% (7B)	~48 GB	Best	A100/H100
LoRA (r=64)	~0.5% (35M)	~16 GB	Excellent	RTX 3090/4090
QLoRA (r=64)	~0.5% (35M)	~8 GB	Excellent	RTX 3060 12GB+
LoRA (r=16)	~0.1% (7M)	~12 GB	Good	RTX 3060 12GB
QLoRA (r=16)	~0.1% (7M)	~6 GB	Good	RTX 3060 8GB

QLoRA = LoRA + 4-bit quantization + paged optimizers + gradient checkpointing

Prerequisites

# Core dependencies
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121
pip install transformers accelerate peft bitsandbytes trl datasets

# For quantization
pip install auto-gptq optimum

# Optional: Weights & Biases for logging
pip install wandb

Understanding LoRA

The Math

Instead of updating all weights $W \in \mathbb{R}^{d \times d}$, LoRA learns two small matrices:

$$
W' = W + \Delta W = W + BA
$$

Where:

$B \in \mathbb{R}^{d \times r}$ (down-projection)
$A \in \mathbb{R}^{r \times d}$ (up-projection)
$r \ll d$ (rank, typically 8-64)

Only $A$ and $B$ are trained. $W$ stays frozen.

Target Modules

Which layers to adapt? Research shows:

# For LLaMA/Mistral/Gemma architecture
TARGET_MODULES = [
    "q_proj", "k_proj", "v_proj", "o_proj",  # Attention
    "gate_proj", "up_proj", "down_proj"       # MLP
]

# For smaller models or less VRAM, use attention only
TARGET_MODULES_ATTENTION = ["q_proj", "k_proj", "v_proj", "o_proj"]

Data Preparation

Format: Instruction Tuning (Alpaca/ShareGPT)

// alpaca_format.json
[
  {
    "instruction": "Write a Python function to calculate fibonacci numbers",
    "input": "",
    "output": "def fibonacci(n):\n    if n <= 1:\n        return n\n    return fibonacci(n-1) + fibonacci(n-2)"
  },
  {
    "instruction": "Explain the concept of recursion",
    "input": "In the context of computer science",
    "output": "Recursion is a programming technique where a function calls itself..."
  }
]

Format: Conversation (ChatML)

// chatml_format.json
[
  {
    "messages": [
      {"role": "system", "content": "You are a helpful coding assistant."},
      {"role": "user", "content": "How do I handle errors in Go?"},
      {"role": "assistant", "content": "In Go, errors are values..."}
    ]
  }
]

Data Processing Pipeline

# data_prep.py
from datasets import Dataset, load_dataset
from transformers import AutoTokenizer
import json

def load_and_format_data(file_path: str, format: str = "alpaca") -> Dataset:
    with open(file_path) as f:
        data = json.load(f)

    if format == "alpaca":
        def format_alpaca(example):
            if example["input"]:
                prompt = f"### Instruction:\n{example['instruction']}\n\n### Input:\n{example['input']}\n\n### Response:\n{example['output']}"
            else:
                prompt = f"### Instruction:\n{example['instruction']}\n\n### Response:\n{example['output']}"
            return {"text": prompt}

        dataset = Dataset.from_list(data).map(format_alpaca)

    elif format == "chatml":
        def format_chatml(example):
            messages = example["messages"]
            # Apply chat template
            text = tokenizer.apply_chat_template(messages, tokenize=False)
            return {"text": text}

        dataset = Dataset.from_list(data).map(format_chatml)

    return dataset

def tokenize_dataset(dataset: Dataset, tokenizer, max_length: int = 2048):
    def tokenize(example):
        return tokenizer(
            example["text"],
            truncation=True,
            max_length=max_length,
            padding="max_length",
            return_tensors="pt"
        )

    return dataset.map(tokenize, batched=True, remove_columns=dataset.column_names)

# Train/val split
dataset = load_and_format_data("train.json")
dataset = dataset.train_test_split(test_size=0.1, seed=42)
train_dataset = dataset["train"]
eval_dataset = dataset["test"]

LoRA Fine-Tuning with Hugging Face TRL

Complete Training Script

# train_lora.py
import torch
from transformers import (
    AutoModelForCausalLM,
    AutoTokenizer,
    TrainingArguments,
    BitsAndBytesConfig
)
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training
from trl import SFTTrainer
from datasets import load_dataset
import os

# ============ CONFIGURATION ============
MODEL_ID = "meta-llama/Llama-3.2-3B-Instruct"  # Or mistralai/Mistral-7B-Instruct-v0.3
DATASET_PATH = "train.json"
OUTPUT_DIR = "./llama-3.2-3b-lora-finetuned"
MAX_LENGTH = 2048
BATCH_SIZE = 2
GRAD_ACCUM = 8  # Effective batch = 16
LEARNING_RATE = 2e-4
NUM_EPOCHS = 3
LORA_R = 64
LORA_ALPHA = 16
LORA_DROPOUT = 0.1

# ============ QUANTIZATION CONFIG (QLoRA) ============
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",        # NormalFloat 4-bit
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,   # Double quantization
)

# ============ LoRA CONFIG ============
lora_config = LoraConfig(
    r=LORA_R,
    lora_alpha=LORA_ALPHA,
    target_modules=[
        "q_proj", "k_proj", "v_proj", "o_proj",
        "gate_proj", "up_proj", "down_proj"
    ],
    lora_dropout=LORA_DROPOUT,
    bias="none",
    task_type="CAUSAL_LM",
    # For QLoRA: modules_to_save=["embed_tokens", "lm_head"]  # If extending vocab
)

# ============ LOAD MODEL & TOKENIZER ============
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID, trust_remote_code=True)
tokenizer.pad_token = tokenizer.eos_token
tokenizer.padding_side = "right"

model = AutoModelForCausalLM.from_pretrained(
    MODEL_ID,
    quantization_config=bnb_config,
    device_map="auto",
    trust_remote_code=True,
    torch_dtype=torch.bfloat16,
    attn_implementation="flash_attention_2",  # Requires flash-attn
)

# Prepare for k-bit training
model = prepare_model_for_kbit_training(model)
model = get_peft_model(model, lora_config)
model.print_trainable_parameters()
# Example output: trainable params: 35,651,584 || all params: 3,200,000,000 || trainable%: 1.11%

# ============ LOAD DATA ============
def formatting_func(example):
    return example["text"]

dataset = load_dataset("json", data_files=DATASET_PATH, split="train")
dataset = dataset.train_test_split(test_size=0.1, seed=42)

# ============ TRAINING ARGS ============
training_args = TrainingArguments(
    output_dir=OUTPUT_DIR,
    num_train_epochs=NUM_EPOCHS,
    per_device_train_batch_size=BATCH_SIZE,
    per_device_eval_batch_size=BATCH_SIZE,
    gradient_accumulation_steps=GRAD_ACCUM,
    learning_rate=LEARNING_RATE,
    lr_scheduler_type="cosine",
    warmup_ratio=0.03,
    logging_steps=10,
    eval_strategy="steps",
    eval_steps=100,
    save_strategy="steps",
    save_steps=100,
    save_total_limit=3,
    load_best_model_at_end=True,
    metric_for_best_model="eval_loss",
    bf16=True,
    tf32=True,
    gradient_checkpointing=True,
    gradient_checkpointing_kwargs={"use_reentrant": False},
    optim="paged_adamw_32bit",  # QLoRA optimizer
    report_to="wandb",  # or "tensorboard" or "none"
    run_name=f"llama-3.2-3b-lora-r{LORA_R}",
    seed=42,
    ddp_find_unused_parameters=False,
)

# ============ TRAINER ============
trainer = SFTTrainer(
    model=model,
    args=training_args,
    train_dataset=dataset["train"],
    eval_dataset=dataset["test"],
    formatting_func=formatting_func,
    max_seq_length=MAX_LENGTH,
    tokenizer=tokenizer,
    packing=True,  # Efficient packing of sequences
)

# ============ TRAIN ============
trainer.train()

# Save final model
trainer.save_model()
tokenizer.save_pretrained(OUTPUT_DIR)

print(f"Model saved to {OUTPUT_DIR}")

Launch Training

# Single GPU
python train_lora.py

# Multi-GPU with Accelerate
accelerate config  # First time setup
accelerate launch train_lora.py

# With DeepSpeed ZeRO-3 (for larger models)
accelerate launch --config_file ds_zero3.yaml train_lora.py

QLoRA Specific Optimizations

Paged Optimizers

# Automatically used with optim="paged_adamw_32bit"
# Moves optimizer states to CPU when not in use
# Enables larger models on same VRAM

Gradient Checkpointing

# In TrainingArguments:
gradient_checkpointing=True,
gradient_checkpointing_kwargs={"use_reentrant": False},

# Saves ~30% VRAM at cost of ~20% slower training

Flash Attention 2

# Install
pip install flash-attn --no-build-isolation

# Use in model loading
attn_implementation="flash_attention_2"

# 2-3x faster, less memory

Monitoring Training

Weights & Biases

# In TrainingArguments
report_to="wandb",
run_name="my-lora-run",

# Before training
import wandb
wandb.init(project="llm-finetuning", name="llama-3b-lora-r64")

TensorBoard

# Start tensorboard
tensorboard --logdir ./llama-3.2-3b-lora-finetuned/runs

# In TrainingArguments
report_to="tensorboard",

Key Metrics to Watch

Metric	Healthy Range	Warning Signs
train_loss	Decreasing steadily	Plateauing early, diverging
eval_loss	Decreasing, close to train	Increasing (overfitting)
learning_rate	Cosine decay to 0	Stuck, not decaying
grad_norm	< 1.0	> 10 (exploding gradients)
GPU memory	Stable	Growing (memory leak)

Merging & Exporting

Merge LoRA Weights

# merge_lora.py
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

BASE_MODEL = "meta-llama/Llama-3.2-3B-Instruct"
LORA_PATH = "./llama-3.2-3b-lora-finetuned"
OUTPUT_PATH = "./llama-3.2-3b-merged"

# Load base model in fp16/bf16 (not 4-bit)
base_model = AutoModelForCausalLM.from_pretrained(
    BASE_MODEL,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True,
)

# Load LoRA adapter
model = PeftModel.from_pretrained(base_model, LORA_PATH)

# Merge and unload
merged_model = model.merge_and_unload()

# Save merged model
merged_model.save_pretrained(OUTPUT_PATH)
tokenizer = AutoTokenizer.from_pretrained(BASE_MODEL)
tokenizer.save_pretrained(OUTPUT_PATH)

print(f"Merged model saved to {OUTPUT_PATH}")

Export to GGUF (for llama.cpp/Ollama)

# Install llama.cpp with Python bindings
pip install llama-cpp-python

# Convert using llama.cpp convert script
python llama.cpp/convert_hf_to_gguf.py \
  ./llama-3.2-3b-merged \
  --outfile ./llama-3.2-3b-finetuned.Q4_K_M.gguf \
  --outtype q4_k_m

# Or use llama.cpp quantize directly
./llama.cpp/quantize ./llama-3.2-3b-merged/model.fp16.gguf \
  ./llama-3.2-3b-finetuned.Q4_K_M.gguf q4_k_m

Create Ollama Modelfile

# Modelfile
FROM ./llama-3.2-3b-finetuned.Q4_K_M.gguf

TEMPLATE "{{ .System }}\n{{ .Prompt }}\n{{ .Response }}"

PARAMETER temperature 0.7
PARAMETER top_p 0.9
PARAMETER num_ctx 8192

# Create Ollama model
ollama create my-finetuned-model -f Modelfile

# Test
ollama run my-finetuned-model "Write a Python function for binary search"

Advanced: LoRA Hyperparameter Tuning

Rank (r) Selection

# Experiment with different ranks
RANK_CONFIGS = [
    {"r": 8, "alpha": 16},    # Minimal, fast, less capacity
    {"r": 16, "alpha": 32},   # Good balance
    {"r": 32, "alpha": 64},   # More capacity
    {"r": 64, "alpha": 16},   # High capacity (alpha often = r/4 to r/2)
    {"r": 128, "alpha": 32},  # Maximum capacity
]

Target Module Ablation

# Test different target module sets
TARGET_CONFIGS = {
    "attention_only": ["q_proj", "k_proj", "v_proj", "o_proj"],
    "mlp_only": ["gate_proj", "up_proj", "down_proj"],
    "all_linear": ["q_proj", "k_proj", "v_proj", "o_proj", 
                   "gate_proj", "up_proj", "down_proj"],
    "attention_mlp": ["q_proj", "v_proj", "gate_proj", "up_proj"],  # Key ones
}

Learning Rate Schedule

# For LoRA, higher LR often works (2e-4 to 1e-3)
# But depends on dataset size:
# Small dataset (<1k): 1e-4 to 5e-4
# Medium (1k-10k): 2e-4
# Large (>10k): 1e-4 to 2e-4

Evaluation & Testing

Quantitative Evaluation

# eval_model.py
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
import torch
from datasets import load_dataset
import json

def evaluate_model(model_path, test_data_path, base_model_id=None):
    if base_model_id:
        base = AutoModelForCausalLM.from_pretrained(
            base_model_id, torch_dtype=torch.bfloat16, device_map="auto"
        )
        model = PeftModel.from_pretrained(base, model_path)
    else:
        model = AutoModelForCausalLM.from_pretrained(
            model_path, torch_dtype=torch.bfloat16, device_map="auto"
        )

    tokenizer = AutoTokenizer.from_pretrained(model_path)

    # Load test data
    with open(test_data_path) as f:
        test_data = json.load(f)

    results = []
    for item in test_data[:50]:  # Sample 50
        prompt = f"### Instruction:\n{item['instruction']}\n\n### Response:\n"
        inputs = tokenizer(prompt, return_tensors="pt").to("cuda")

        with torch.no_grad():
            outputs = model.generate(
                **inputs,
                max_new_tokens=256,
                temperature=0.7,
                do_sample=True,
                pad_token_id=tokenizer.eos_token_id
            )

        response = tokenizer.decode(outputs[0][inputs.input_ids.shape[1]:], skip_special_tokens=True)
        results.append({
            "instruction": item["instruction"],
            "expected": item["output"],
            "generated": response
        })

    # Save for human evaluation
    with open("eval_results.json", "w") as f:
        json.dump(results, f, indent=2)

    return results

Qualitative Evaluation (Human/LLM Judge)

# judge_eval.py
from openai import OpenAI
import json

client = OpenAI()

def llm_judge(instruction, expected, generated):
    prompt = f"""Rate the quality of the generated response compared to expected.

Instruction: {instruction}
Expected: {expected}
Generated: {generated}

Rate 1-5 on:
1. Accuracy
2. Completeness  
3. Clarity
4. Relevance

Return JSON: {{"accuracy": 1-5, "completeness": 1-5, "clarity": 1-5, "relevance": 1-5, "reasoning": "..."}}"""

    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": prompt}],
        temperature=0
    )
    return json.loads(response.content)

# Run evaluation
with open("eval_results.json") as f:
    results = json.load(f)

scores = {"accuracy": [], "completeness": [], "clarity": [], "relevance": []}
for r in results:
    eval = llm_judge(r["instruction"], r["expected"], r["generated"])
    for k in scores:
        scores[k].append(eval[k])

for k, v in scores.items():
    print(f"{k}: {sum(v)/len(v):.2f}")

Common Issues & Solutions

Issue	Cause	Solution
OOM	Model too large	Increase gradient_accumulation, reduce batch_size, use QLoRA, gradient_checkpointing
Loss not decreasing	LR too low/high	Try 2e-4, check data format, verify tokenization
Overfitting	Too many epochs, small data	Reduce epochs, add dropout, more data, weight decay
Gibberish output	Wrong chat template	Match training format exactly
NaN loss	Unstable training	Reduce LR, add gradient clipping, check data
Slow training	No Flash Attention	Install flash-attn, use bf16

Cost Analysis

Training Cost (Estimates)

Model	Method	GPU	Time (1k samples, 3 epochs)	Cost (Cloud)
3B	LoRA	RTX 3090 (24GB)	~30 min	$0.50
3B	QLoRA	RTX 3060 (12GB)	~45 min	$0.30
7B	LoRA	RTX 3090 (24GB)	~1 hr	$1.00
7B	QLoRA	RTX 3090 (24GB)	~1.5 hr	$0.80
13B	QLoRA	2× RTX 3090	~3 hr	$3.00
70B	QLoRA	4× A100 (80GB)	~8 hr	$40+

Local training on owned hardware = $0 marginal cost

Best Practices Summary

Data

[ ] Quality > Quantity: 1k high-quality > 100k noisy
[ ] Deduplicate: Remove exact/near duplicates
[ ] Balance: Cover all target domains evenly
[ ] Format consistency: Match training format exactly at inference

Training

[ ] Start small: Test with r=16, 1 epoch first
[ ] Monitor eval_loss: Stop when it plateaus/increases
[ ] Save checkpoints: Every 100-500 steps
[ ] Use cosine LR: With warmup (3-5% of steps)
[ ] Pack sequences: packing=True in SFTTrainer

LoRA Config

[ ] r=64, alpha=16 for most cases
[ ] Target all linear layers for best quality
[ ] Dropout 0.05-0.1 for regularization
[ ] Don't quantize LoRA weights (keep in bf16/fp16)

Deployment

[ ] Merge weights for inference speed
[ ] Quantize merged model to GGUF Q4_K_M
[ ] Test inference with same prompt format
[ ] Benchmark latency before production

Complete Example: Domain Adaptation

# Complete workflow for adapting LLaMA-3.2-3B to medical QA
# 1. Collect 2k high-quality medical QA pairs
# 2. Format as Alpaca
# 3. Train with QLoRA r=64, 3 epochs
# 4. Evaluate on held-out medical questions
# 5. Merge, quantize to GGUF, deploy via Ollama

# Expected: 90%+ accuracy on medical specialty vs 60% base model

Resources

Fine-tuning is no longer reserved for big tech. With QLoRA, a $300 GPU can customize a 7B model to beat GPT-3.5 on your specific task. Start small, iterate fast, deploy locally.

Fine-Tuning vs RAG vs Prompt Engineering: When to Use Which

Phil IPS — Mon, 12 Oct 2026 16:50:29 +0000

Fine-Tuning vs RAG vs Prompt Engineering: When to Use Which

A practical decision framework for choosing the right LLM customization approach.

Introduction

Three main ways to adapt LLMs to your needs. Each has different cost, complexity, and effectiveness. This guide helps you choose.

Quick Decision Matrix

Factor	Prompt Engineering	RAG	Fine-Tuning
Setup time	Minutes	Hours	Days-Weeks
Cost	$0 (API only)	$0-50/mo	$50-5000+
Data needed	None	Documents	Labeled examples
Updates	Instant	Re-index	Retrain
Hallucination fix	Partial	✅ Strong	✅ Strong
Domain knowledge	Limited	✅ Full	✅ Full
Style/format control	✅ Good	Partial	✅ Excellent
Latency added	None	~100-500ms	None
Expertise needed	Low	Medium	High
Reversibility	Trivial	Easy	Hard

Approach 1: Prompt Engineering

What It Is

Crafting input instructions to guide model behavior without changing the model.

When to Use

Quick experiments
General knowledge tasks
Style/format control
One-off or varied tasks
No proprietary data

Techniques

# Zero-shot
"Write a SQL query for..."

# Few-shot
"Examples:\nInput: ...\nOutput: ...\n\nNow: ..."

# Chain-of-thought
"Think step by step: ..."

# Structured output
"Return ONLY valid JSON: {...}"

# Role + Context
"You are a senior engineer. Context: ... Task: ..."

Pros/Cons

✅ Pros	❌ Cons
Instant iteration	Limited by context window
No training cost	Can't learn new facts
Works on any model	Inconsistent on complex tasks
Version controllable	Prompt injection risk

Cost

API calls only: ~$0.001-0.01 per request
No infrastructure

Approach 2: RAG (Retrieval-Augmented Generation)

What It Is

Ground model responses in your documents by retrieving relevant chunks at query time.

Architecture

Query → Embed → Vector Search → Top-K Chunks → LLM + Context → Answer

When to Use

Large document collections (100+ docs)
Frequently changing data
Need citations/sources
Regulatory compliance (audit trail)
Multiple domains in one system

Implementation Levels

Level 1: Basic (Chroma + OpenAI)

# 50 lines, runs locally
import chromadb
from chromadb.utils import embedding_functions

ef = embedding_functions.SentenceTransformerEmbeddingFunction("all-MiniLM-L6-v2")
client = chromadb.PersistentClient("./db")
col = client.create_collection("docs", embedding_function=ef)
col.add(documents=texts, metadatas=metas, ids=ids)
results = col.query(query_texts=[question], n_results=5)

Level 2: Production (Qdrant + Reranker)

# Hybrid search + cross-encoder reranking
from qdrant_client import QdrantClient
from sentence_transformers import CrossEncoder

client = QdrantClient(url="...", api_key="...")
reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")

# Retrieve 20, rerank to 5
results = client.query_points(collection, query, limit=20)
pairs = [(q, r.payload["text"]) for r in results]
scores = reranker.predict(pairs)
top5 = sorted(zip(results, scores), key=lambda x: x[1], reverse=True)[:5]

Level 3: Advanced (Agentic RAG)

# Multi-hop reasoning, tool use
# Query → Decompose → Retrieve per subquery → Synthesize

Pros/Cons

✅ Pros	❌ Cons
Grounded in your data	Added latency (retrieval)
Easy updates (re-index)	Chunking strategy matters
Citations built-in	Retrieval can miss context
No training needed	Embedding quality critical
Works with any LLM	Context window limits

Cost

Component	Monthly (1M vectors)
Vector DB (Qdrant Cloud)	$0-150
Embeddings (OpenRouter)	$0-20
LLM calls	$10-100
Total	$10-270

Approach 3: Fine-Tuning

What It Is

Update model weights on your data to bake in knowledge, style, or behavior.

Types

Method	Params Trained	VRAM (7B)	Use Case
Full FT	100%	80 GB+	Research, max quality
LoRA	0.1-1%	16 GB	Most production
QLoRA	0.1-1%	8 GB	Consumer GPU
Prefix/Adapter	<0.1%	6 GB	Style only

When to Use

Consistent style/format requirements
Domain-specific terminology (legal, medical, code)
High-volume, repetitive tasks
Model must work offline
Regulatory: model = artifact

Data Requirements

// Alpaca format (instruction tuning)
{"instruction": "Write a Python function", "input": "calculate fibonacci", "output": "def fib(n):..."}

// Chat format
{"messages": [
  {"role": "system", "content": "You are a Python expert"},
  {"role": "user", "content": "Write a fibonacci function"},
  {"role": "assistant", "content": "def fib(n):..."}
]}

// Preference pairs (DPO)
{"prompt": "...", "chosen": "better response", "rejected": "worse response"}

Minimums:

LoRA: 500-1000 quality examples
Full FT: 10K-100K+
DPO: 1K-5K pairs

Training Pipeline (QLoRA)

# 1. Prepare data
python prepare_data.py --input raw.jsonl --output train.jsonl --val-size 100

# 2. Train (4-bit, 7B model, 1x A100 40GB)
python train.py \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --data train.jsonl \
  --output ./lora-adapter \
  --lora-r 64 --lora-alpha 16 \
  --batch-size 2 --grad-accum 4 \
  --lr 2e-4 --steps 500 \
  --quantization 4bit

# 3. Merge & export
python merge.py --base meta-llama/Llama-3.1-8B-Instruct \
  --adapter ./lora-adapter --output ./merged-model

# 4. Convert to GGUF
python convert-hf-to-gguf.py ./merged-model --outfile model.q4_k_m.gguf

Pros/Cons

✅ Pros	❌ Cons
Best task performance	High upfront cost
No retrieval latency	Hard to update
Works offline	Catastrophic forgetting risk
Bakes in style/format	Needs ML expertise
Single artifact to deploy	GPU required

Cost

Scale	Hardware	Time	Est. Cost
7B QLoRA	1× A100 40GB	2-4 hrs	$10-30
13B QLoRA	1× A100 80GB	4-8 hrs	$40-80
70B QLoRA	4× A100 80GB	8-24 hrs	$300-1000
7B Full FT	8× A100 80GB	12-48 hrs	$1000-5000

Decision Framework

START: What's your primary need?
│
├─→ "Control output format/style" 
│   └─→ **Prompt Engineering** (try first, 15 min)
│
├─→ "Answer from my documents/data"
│   ├─→ Data changes frequently?
│   │   ├─→ Yes → **RAG** (re-index on change)
│   │   └─→ No → Continue
│   ├─→ Need citations?
│   │   └─→ Yes → **RAG**
│   ├─→ < 100 docs?
│   │   └─→ Yes → **RAG** (Chroma local)
│   └─→ > 10K docs, high query volume?
│       └─→ **RAG** (Qdrant/Milvus) or **Fine-tune** if style critical
│
├─→ "Model must know domain deeply"
│   ├─→ Can you get 500+ quality examples?
│   │   ├─→ Yes → **Fine-tune (QLoRA)**
│   │   └─→ No → **RAG + few-shot prompts**
│   ├─→ Need offline/air-gapped?
│   │   └─→ Yes → **Fine-tune** (deploy GGUF)
│   └─→ High volume (>10K req/day)?
│       └─→ **Fine-tune** (cheaper at scale)
│
└─→ "Reduce hallucinations on facts"
    ├─→ Facts in documents?
    │   └─→ Yes → **RAG**
    └─→ Facts = training knowledge?
        └─→ **Fine-tune** or better prompts

Hybrid Approaches (Real World)

1. RAG + Fine-Tuned Embedder

# Fine-tune embedder on your domain, use with RAG
# Better retrieval → better answers

2. RAG + Fine-Tuned Generator

# Fine-tune small model (7B) for style/format
# Use RAG for knowledge
# Best of both worlds

3. Prompt → RAG → Fine-Tune (Progression)

Week 1: Prompt engineering (validate concept)
Week 2: Add RAG (ground in data)
Week 3: Fine-tune if needed (optimize)

4. Router Pattern

# Route queries to best approach
def route(query):
    if is_factual(query): return rag_chain
    elif is_creative(query): return prompt_chain
    elif is_coding(query): return finetuned_coder
    else: return default_chain

Cost Comparison at Scale

Approach	1K req/mo	100K req/mo	1M req/mo
Prompt (API)	$5	$500	$5,000
RAG (managed)	$30	$300	$2,000
Fine-tuned (self-host)	$100*	$100*	$100*
*Fixed GPU cost	(A100)	(A100)	(4×A100)

Break-even: Fine-tuning pays off at ~50K req/mo if you own GPUs.

Common Mistakes

Mistake	Reality
"Fine-tune for facts"	Use RAG; FT memorizes poorly
"RAG solves everything"	Bad retrieval = bad answers
"Prompt engineering is easy"	Requires systematic testing
"Bigger model = better"	7B FT > 70B prompt for specific tasks
"One approach fits all"	Hybrid usually wins

Evaluation Checklist

Before committing:

[ ] Defined success metrics (accuracy, latency, cost)
[ ] Created test set (50+ representative queries)
[ ] Benchmarked current approach
[ ] Estimated total cost (dev + infra + ongoing)
[ ] Defined update strategy (how often, how)
[ ] Considered compliance/privacy needs

My Recommendations by Use Case

Use Case	Primary	Secondary
Customer support bot	RAG	Fine-tune style
Code assistant	Fine-tune (7B coder)	RAG for docs
Legal document analysis	RAG	Fine-tune format
Creative writing	Prompt engineering	Fine-tune voice
SQL generation	Fine-tune (7B)	RAG for schema
Chat with PDFs	RAG	-
API documentation bot	RAG	Prompt for format
Internal knowledge base	RAG	-
Offline mobile app	Fine-tune (GGUF)	-
High-volume classification	Fine-tune (small)	-

Conclusion

Start with prompt engineering — 15 minutes, validates the task.

Add RAG when you have documents and need grounded answers — hours to set up.

Fine-tune only when: you have 500+ examples, need consistent style/format, high volume, or offline deployment.

Most production systems use hybrids: RAG for knowledge + fine-tuned model for behavior.

Part of the LLM Customization series on Dev.to

Fine-Tuning LLMs on Consumer Hardware: LoRA Guide 2024

Phil IPS — Mon, 12 Oct 2026 16:23:44 +0000

Fine-Tuning LLMs on Consumer Hardware: LoRA Guide 2024

Train custom models on your GPU using LoRA/QLoRA — no cloud credits needed.

Introduction

Fine-tuning used to require A100s and thousands of dollars. With LoRA (Low-Rank Adaptation) and QLoRA (Quantized LoRA), you can fine-tune 7B-70B models on a single 24GB GPU (RTX 3090/4090) or even 12GB with quantization.

What is LoRA?

LoRA freezes the base model and trains only small adapter matrices:

Base Model (frozen) + LoRA Adapters (trainable) = Fine-tuned Model

Benefits:

100-1000x fewer trainable parameters
Fast training (minutes to hours)
Small checkpoint files (MBs not GBs)
Mergeable back into base model

Hardware Requirements

Model Size	Method	VRAM Needed	GPU Example
7B	LoRA (fp16)	~16 GB	RTX 3090/4090
7B	QLoRA (4-bit)	~8 GB	RTX 3060 12GB
13B	QLoRA (4-bit)	~12 GB	RTX 3080 12GB
33B	QLoRA (4-bit)	~20 GB	RTX 3090/4090
70B	QLoRA (4-bit)	~40 GB	Dual 3090/4090

Setup Environment

# Create environment
conda create -n lora python=3.10 -y
conda activate lora

# Install dependencies
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121
pip install transformers peft trl accelerate bitsandbytes datasets wandb

Prepare Dataset

// data/train.jsonl
{"text": "### Instruction:\nWrite a Python function to reverse a string\n### Response:\ndef reverse_string(s):\n    return s[::-1]"}
{"text": "### Instruction:\nExplain what a decorator does in Python\n### Response:\nA decorator is a function that takes another function and extends its behavior..."}

Format: Alpaca/ShareGPT style with ### Instruction: and ### Response: markers.

Training Script (QLoRA)

# train_lora.py
import os
import torch
from datasets import load_dataset
from transformers import (
    AutoModelForCausalLM,
    AutoTokenizer,
    BitsAndBytesConfig,
    TrainingArguments
)
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training
from trl import SFTTrainer

# Config
MODEL_ID = "mistralai/Mistral-7B-v0.1"  # or "meta-llama/Llama-2-7b-hf"
OUTPUT_DIR = "./mistral-7b-code-lora"
DATA_PATH = "data/train.jsonl"

# 4-bit quantization config
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True
)

# Load model
model = AutoModelForCausalLM.from_pretrained(
    MODEL_ID,
    quantization_config=bnb_config,
    device_map="auto",
    trust_remote_code=True
)
model = prepare_model_for_kbit_training(model)

# LoRA config
lora_config = LoraConfig(
    r=64,                    # Rank (higher = more capacity)
    lora_alpha=16,           # Scaling factor
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"],
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM"
)

model = get_peft_model(model, lora_config)
model.print_trainable_parameters()
# Trainable params: ~0.5% of total

# Tokenizer
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID, trust_remote_code=True)
tokenizer.pad_token = tokenizer.eos_token
tokenizer.padding_side = "right"

# Dataset
dataset = load_dataset("json", data_files=DATA_PATH, split="train")

# Training arguments
training_args = TrainingArguments(
    output_dir=OUTPUT_DIR,
    per_device_train_batch_size=2,
    gradient_accumulation_steps=4,
    warmup_steps=10,
    max_steps=100,           # Adjust for your dataset
    learning_rate=2e-4,
    fp16=True,
    logging_steps=5,
    save_strategy="steps",
    save_steps=50,
    report_to="wandb",       # or "none"
    optim="paged_adamw_8bit"
)

# Trainer
trainer = SFTTrainer(
    model=model,
    train_dataset=dataset,
    dataset_text_field="text",
    max_seq_length=512,
    tokenizer=tokenizer,
    args=training_args,
    packing=False
)

# Train
trainer.train()

# Save
trainer.model.save_pretrained(OUTPUT_DIR)
tokenizer.save_pretrained(OUTPUT_DIR)
print(f"Model saved to {OUTPUT_DIR}")

Run Training

# Single GPU
python train_lora.py

# Multi-GPU (if available)
accelerate launch --multi_gpu train_lora.py

Merge & Export

# merge_lora.py
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

BASE_MODEL = "mistralai/Mistral-7B-v0.1"
LORA_PATH = "./mistral-7b-code-lora"
OUTPUT_PATH = "./mistral-7b-code-merged"

# Load base
model = AutoModelForCausalLM.from_pretrained(
    BASE_MODEL,
    torch_dtype=torch.float16,
    device_map="auto"
)

# Load & merge LoRA
model = PeftModel.from_pretrained(model, LORA_PATH)
model = model.merge_and_unload()

# Save merged
model.save_pretrained(OUTPUT_PATH)
tokenizer = AutoTokenizer.from_pretrained(BASE_MODEL)
tokenizer.save_pretrained(OUTPUT_PATH)

Convert to GGUF (llama.cpp)

# Install llama.cpp
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp && make

# Convert
python convert-hf-to-gguf.py ../mistral-7b-code-merged \
  --outfile ../mistral-7b-code.Q4_K_M.gguf \
  --outtype q4_k_m

Inference with Ollama

# Create Modelfile
cat > Modelfile << 'EOF'
FROM ./mistral-7b-code.Q4_K_M.gguf
TEMPLATE "{{ .System }}\n\n{{ .Prompt }}"
PARAMETER temperature 0.7
PARAMETER top_p 0.9
EOF

# Create model
ollama create mistral-code -f Modelfile

# Test
ollama run mistral-code "Write a Python decorator for timing functions"

Cost Breakdown (RTX 3090 24GB)

Item	Cost
Electricity (4 hrs @ 350W)	~$0.20
GPU depreciation	~$1.00
Total per run	~$1.20

vs. Cloud (A100 40GB): $1.50-3.00/hr → $6-12 for same training.

Tips for Success

Dataset quality > quantity — 500 clean examples beat 5000 noisy ones
Start small — 100 steps, check loss, then scale
Monitor VRAM — Reduce batch size if OOM
Use gradient checkpointing — model.gradient_checkpointing_enable()
Eval regularly — Keep a validation set

Common Issues

Error	Fix
CUDA OOM	Reduce batch size, increase grad accumulation, use 4-bit
Loss not decreasing	Increase learning rate, check data format
NaN loss	Lower learning rate, add gradient clipping
Slow training	Enable flash attention, use `packing=True`

Conclusion

LoRA/QLoRA democratized LLM fine-tuning. With a consumer GPU, you can create domain-specialized models for coding, writing, analysis — whatever your use case. Start with 7B QLoRA, iterate on data, and deploy locally via Ollama or llama.cpp.

Next: Try DPO/ORPO for preference alignment, or merge multiple LoRAs for multi-task models.

Part of the LLM Fine-Tuning series on Dev.to

What I Learned Trying to Ship a Fine-Tuned Model to the NPU

Jaydeep Shah (JD) — Wed, 07 Oct 2026 04:18:40 +0000

There is a quiet assumption in the on-device AI ecosystem: that the models shipping from Hugging Face and litert-community are good enough. Drop them into your app, write a good system prompt, and you are done.

I believed that too, until I benchmarked it.

This post is about what I learned when a strong prompt and an under-resourced fine-tune went head to head, why task-aligned fine-tuning still matters for serious on-device apps, and why the toolchain makes it hard to ship fine-tuned models to the hardware where they would matter most.

General models are general. Your app is not.

Gemma 4 E2B is a strong general-purpose model. Instruction-tuned, INT4-quantized, 2.59 GB - it runs comfortably on a phone GPU at about 25 tokens per second. For chatbot-style apps, it works out of the box.

But Redacto is not a chatbot. It is a zero-trust PII/PHI redaction engine with 7 specialized document categories: medical records, police reports, financial statements, journalism source notes, field service logs, legal filings, and a general fallback. Each category has its own detection rules. A patient's daughter's name is PHI in a medical record. A suspect's description is explicitly not PII in a police report - it needs to be preserved. These distinctions are domain-specific, and a general model does not know them by default.

Prompt engineering gets you surprisingly far. With carefully crafted system prompts, explicit preserve lists, and the structured [CATEGORY_N] output format, the standard Gemma 4 E2B model scores 80.5% overall on our 85-entry benchmark suite. That is real and usable.

But there are domains where prompt engineering hits a ceiling.

The numbers, and what they do and do not prove

We ran a head-to-head benchmark: standard Gemma 4 E2B versus a QLoRA fine-tuned variant, both on GPU, same device (Galaxy S25 Ultra), same 85-entry dataset across all 5 modes (the benchmark covers 5 of the 7 document categories; legal filings and the general fallback were not part of this run).

The overall scores tell one story - the fine-tuned model scored 70.3% versus the standard model's 80.5%, largely because of an output format mismatch (the fine-tuned model was trained on generic [REDACTED] labels, not Redacto's [CATEGORY_N] format). That is a training data problem, not a fine-tuning problem. In other words, a strong prompt on the stock model beat a quick, under-resourced fine-tune overall.

A caveat before the table: every number here comes from a single directional benchmarking session in May 2026 on one Galaxy S25 Ultra. No raw per-run logs were saved and the device is no longer available, so treat these as directional signals, not a rigorous multi-run study.

The per-mode entity recall tells a different story:

Mode	Standard	Fine-tuned	Delta
TACTICAL	63.7%	76.8%	+13.1%
FIELD_SERVICE	82.1%	95.3%	+13.2%
FINANCIAL	83.8%	85.5%	+1.7%
JOURNALISM	71.1%	61.3%	-9.8%
HIPAA	95.7%	39.9%	-55.8%

Look at TACTICAL. The standard model catches 63.7% of entities - victims, witnesses, minors - that should be redacted. The fine-tuned model catches 76.8%. That is a 13-point improvement.

For a police report, that 13% gap is the difference between protecting a victim's identity and exposing it. An app that misses 1 in 3 names is not a privacy tool. It is a liability.

FIELD_SERVICE shows the same pattern: +13.2%, driven by the fine-tuned model's ability to recognize contextual security information - "the key is under the mat," WiFi passwords embedded in natural language, gate codes in conversational text. These are not standard PII patterns. A general model does not look for them. A fine-tuned model does.

The HIPAA regression (-55.8%) and JOURNALISM regression (-9.8%) are instructive too. They show that fine-tuning on generic PII data actively hurts domains where the standard model's specialized prompt engineering is already strong. The standard model's HIPAA system prompt was specifically engineered for relational PHI - "the patient's daughter Lisa" - which the fine-tuned model's generic training data does not cover. The lesson is not "don't fine-tune." The lesson is "fine-tune on your app's actual task data, with your app's actual output format." If we retrained with Redacto's system prompts and [CATEGORY_N] labels, the gains in TACTICAL and FIELD_SERVICE would likely hold while the regressions would disappear. Both quantity and alignment of the training data matter here, and this run was short on both.

Consistency is the hidden killer

An 80% entity recall score sounds decent in a benchmark table. In production, it means 1 in 5 sensitive identifiers leaks through. Every time.

Users do not experience averages. They experience individual documents. A medical provider using Redacto to redact a discharge summary before faxing it to an insurance company does not care that the model works 80% of the time. They care that this document, right now, is clean.

This is why task-aligned fine-tuning matters for apps where accuracy matters. The jump from 63.7% to 76.8% on TACTICAL is not just a benchmark improvement - it is the difference between an app that is trustworthy and one that is not. And with proper task-specific training data, those numbers should climb further.

The toolchain works - up to a point

Here is the fine-tuning pipeline, step by step, with what actually happens at each stage:

Step 1: QLoRA on Colab. Works. We fine-tuned Gemma 4 E2B using QLoRA (LoRA rank 8, alpha 16, dropout 0.05) on 3,000 examples from the ai4privacy/pii-masking-400k dataset. Trainable parameters: 2,850,816 - just 0.06% of the model. Training time on an NVIDIA RTX PRO 6000: 217 seconds. Final loss: 4.3064. The tooling here is mature. peft, transformers, bitsandbytes - it all works.

It is worth being honest about what this fine-tune was: 3,000 of the roughly 400,000 samples in ai4privacy/pii-masking-400k, a single epoch, only a few minutes of actual training (about 217 seconds, though wrestling the toolchain end to end took the better part of a day), and trained on generic [REDACTED] labels rather than Redacto's [CATEGORY_N] output format. That is an under-resourced and partly misaligned run, not a serious training effort. So the results above are best read as "what a strong prompt did against a quick, under-resourced fine-tune," not as the ceiling of what fine-tuning can do. With the full dataset and task-aligned labels, the overall picture could flip.

Step 2: Export to .litertlm. Works, with a trap. The litert_torch.generative.export_hf tool converts the merged weights to a quantized .litertlm bundle. But there is an undocumented incompatibility: the Gemma 4 chat template from Hugging Face uses map.get() in Jinja, which LiteRT-LM's on-device template parser does not support. The error - Failed to apply template: unknown method: map has no method named get - will stop every developer who tries this pipeline. The fix: swap in Gemma 3's chat template before export. Not obvious. Not documented. But it works.

Step 3: GPU inference. This is where the story gets muddy. In our notes the fine-tuned .litertlm loaded on the Adreno 830 GPU and ran inference at 9.0 tokens per second (slower than the standard model's roughly 25 tok/s, which we attributed to the fine-tuned export being 4.7 GB versus the standard bundle's 2.59 GB - a quantization granularity difference, not a fundamental limitation). The fine-tuned export is 4.7 GB and GPU-only; it is too large and the wrong shape to compile for the NPU.

Step 4: NPU inference. Blocked.

The wall

The standard Gemma 4 E2B model runs on the Hexagon V79 NPU at about 42 tokens per second - roughly 1.7x faster than GPU. Because NPU decode is memory-bandwidth-bound, that rate stays roughly flat rather than scaling with much else, which is exactly what you would expect. Time to first token drops from 366ms to 92ms. For a redaction app processing sensitive documents, that speed difference is the gap between a noticeable pause and an instant response.

But the standard model was compiled for the NPU by Qualcomm's hardware team. The .litertlm file at litert-community contains a QNN-prepared bundle (QNN being Qualcomm AI Engine Direct) with custom DISPATCH_OP operations compiled specifically for the Hexagon V79 DSP on the Snapdragon 8 Elite. This compilation step - converting quantized weights into a QNN graph that the NPU can execute - requires the AIMET (AI Model Efficiency Toolkit) and an ahead-of-time (AoT) QNN toolchain.

That toolchain is not part of the public LiteRT-LM SDK.

Community projects have since demonstrated ahead-of-time QNN compilation for LLMs through their own toolchains, which suggests the gap is specific to the LiteRT-LM export path rather than QNN itself. For developers working inside the LiteRT-LM pipeline, though, the wall is the same.

This is not an engineering shortcut we missed. This is a hardware-team integration boundary. The tools required to compile a model into a QNN-prepared NPU bundle are internal to Qualcomm's AI stack. Developers can download the Qualcomm AI Engine Direct SDK and access QNN APIs for classical ML models, but the specific pipeline for compiling LLM-class models - with their attention mechanisms, KV caches, and multi-head architectures - into Hexagon-optimized graphs is not publicly available for independent use on arbitrary fine-tuned weights.

The net result, at least in our attempt: a fine-tuned Gemma 4 E2B that showed a double-digit TACTICAL entity-recall gain in a directional benchmark, but that we could not compile for the NPU. The gap between "I improved my model" and "my users benefit from it on the fastest hardware" is hard to bridge today without vendor support.

What this means for every developer

This is not a Redacto-specific problem. Every developer who needs domain-specific accuracy is likely to hit this wall in the same sequence:

Ship the standard model. It works.
Discover domain-specific accuracy gaps. They exist.
Fine-tune. It can help, if the data is aligned.
Try to deploy the fine-tuned model to NPU. Blocked.

The result: fine-tuning becomes a GPU-only optimization. The penalty is not just speed, either. Our fine-tuned model drew roughly 3x the current of the standard model on GPU (301 mA vs 101 mA) - a system-wide reading, so directional rather than precise, but the direction matters for battery life too. And the entire value proposition of the NPU - purpose-built silicon that runs inference about 1.7x faster than GPU (and several times faster than CPU), and by widely reported figures at lower power - stays tied to the standard models that were pre-compiled for it.

This is less a villain than two reasonable design choices colliding. Ahead-of-time NPU compilation lives inside the vendor's AI stack, which was not built with third-party fine-tunes in mind; and the open export path targets generic INT4, not a QNN graph. The practical effect is still awkward: the developers who invest in accuracy by fine-tuning end up on the slower runtime, while those who ship the stock model get the faster one. That is an outcome to fix, not a conspiracy to blame.

What the ecosystem needs

The fix is conceptually simple: a public, documented pipeline from fine-tuned Hugging Face weights to NPU-compiled .litertlm.

In concrete terms:

A public QNN AoT compilation path for LLMs. Today, Qualcomm's QNN SDK supports classical model compilation. Extending this to LLM architectures - or providing a hosted compilation service - would let developers compile their fine-tuned weights for Hexagon without needing internal toolchain access.

AIMET integration with the LiteRT-LM export pipeline. AIMET handles quantization-aware training and post-training quantization for Qualcomm targets. If litert_torch.generative.export_hf could target QNN as a backend (not just generic INT4), the export step would produce NPU-ready bundles directly.

A compilation service, if open-sourcing the toolchain is not feasible. Upload merged weights, receive a QNN-compiled .litertlm. This is how some cloud TPU compilation workflows operate - the developer never touches the compiler directly, but they get hardware-optimized artifacts back.

Documentation of the compilation requirements. Even knowing what the pipeline needs - AIMET version, QNN SDK version, target SoC specification, supported quantization schemes, memory layout constraints - would let the community start building bridges.

The bigger picture

On-device AI is at an inflection point. The models are good enough. The hardware is fast enough. The runtimes are stable enough. The missing piece is the development loop: train, evaluate, compile, deploy, measure, iterate.

Right now, that loop is broken at the compile step for anyone who fine-tunes. And task-aligned fine-tuning is often what closes the last domain-specific accuracy gap. Which means the loop is broken for exactly the developers who are building the most demanding applications.

The standard Gemma 4 E2B on NPU at about 42 tokens per second with 92ms time-to-first-token is genuinely impressive hardware. Getting fine-tuned models onto that hardware - with the kind of domain-specific gains we saw in the benchmark - is the difference between on-device AI as a demo and on-device AI as a product.

The models are ready. The silicon is ready. The remaining gap is a public path through the toolchain.

Related in this series of "Edge AI from the Trenches"

Prompt Engineering Beat My Fine-Tuned Model. Here Is Why. - the decision framework for when fine-tuning is the right lever to pull
My Fine-Tuned Gemma 4 Loaded Fine, Then Broke on the First Message - the undocumented export failure that blocks fine-tuned models before they even reach the NPU wall
I Opened a .litertlm File. Here Is What Is Actually in There. - why the compiled bundle is the end of the pipeline, and what gets sealed at export time

Jaydeep Shah is a developer with roots in embedded systems, Android platform internals, and silicon-level AI optimization. He now explores on-device AI inference - bringing models from the cloud to phones and edge hardware. Along with his team Edge Artists, he builds applications using LiteRT-LM and Gemma models on mobile hardware, and writes about what works, what breaks, and what he learns along the way. This post is part of the Edge AI from the Trenches series.

Last updated: Oct 2026
22nd of 23 posts in the "Edge AI from the Trenches" series

Fine-tuning a 0.8B model to beat hosted LLMs at one job

Abhash Chakraborty — Mon, 05 Oct 2026 16:07:46 +0000

I needed a model for one narrow decision. When a new fact about someone comes in, does it replace something already known? "I moved to Berlin" should retire "I live in Pune". "Watched two episodes of The Bear" should leave "works at Infosys" alone.

It sounds trivial, but any assistant with long-term memory makes this call all the time, and both kinds of mistake hurt. Miss an update and it keeps repeating stale facts. Replace too eagerly and it throws away things that are still true.

This post covers the models I compared, how I fine-tuned the best starting point, how I tested it, and what it took to train. The short version: a 0.8B open model, fine-tuned on 6,000 examples on a single T4, beat every hosted LLM I tested at this one job.

Every model on the same held-out tests. For my definition of an update, not a general ranking.

Why a System-One model

The obvious approach is to prompt an LLM and parse what comes back. That works, but it's slow, every decision is an API call, and the answer is free text that you then have to interpret.

TypeSafe's Jev takes a different approach. It's a "System One" model: instead of writing text, it answers typed questions with calibrated probabilities, like a choice from a list or a yes/no. For this task I send it the new fact and up to eight existing ones, and ask two questions about each existing fact:

How does the new fact relate to it: duplicate, updates, elaborates, related or unrelated?
Is the existing fact now outdated?

The score I use is the average of the two: p_update = (P(updates) + P(outdated)) / 2. At 0.70 or above, the old fact is replaced.

The candidates

Jev (hosted, by TypeSafe). Solid, but tuned for general decisions rather than my definition of an update.
Laya, an open 421M ModernBERT model that answers in Jev's format. Fast and light, but off the shelf it replaced too many everyday events.
Kev, Jared Palmer's open-weight model, compatible with Jev. I tried two sizes. Kev-0.8B was the best of everything as released on stated updates. Kev-4B scored lower on that, and on the T4 it ran out of memory on longer requests and took 5.5 seconds per decision.

For reference I also ran a zero-shot NLI model (DeBERTa-v3-large) and several hosted LLMs, each prompted with the same definition of an update. Kev-0.8B was the clear starting point: the best as released, and the only size that both trains and runs on a 16 GB T4.

The data

Each training example is one request: a new fact, up to eight existing facts, and the typed questions, with a label for every question. I built them from two pools:

6,818 logged Jev decisions. Training on these is distillation: Kev learns to make Jev's calls on real requests.
10,219 decisions with ground truth. I simulated 48 lives of a year each. People move city, change jobs, partners and managers, switch gyms, cars, courses, diets and laptops, reschedule appointments, mention crucial facts and make small talk. Because the simulator knows what actually changed, every pair has the right answer: same attribute with a new value is an update, the same value is a duplicate, anything else is related or unrelated. That's 53,870 fact pairs, 2,298 of them real updates.

The 6,000 training examples took every decision that contains a real update first, then filled up from the shuffled mix. Nothing from the test data went in.

Stage 1: supervised fine-tuning

LoRA on Qwen3.5-0.8B-Base, starting from Kev's released weights: rank 16, alpha 32, dropout 0.05, on every linear and linear-attention projection, plus a small decision head. One epoch over the 6,000 examples at a learning rate of 5e-5, with label smoothing of 0.05.

The GPU was a single NVIDIA T4 with 16 GB. It has no bf16, so everything ran in fp32, with batch 1, gradient accumulation of 8 and gradient checkpointing. The requests are short (87 tokens on average, 138 at the 95th percentile), so nothing was truncated. That came to 750 optimizer steps and 5.52M tokens in 6.55 hours, peaking at 5.09 GB of GPU memory.

Stage 2: reward-ranked RL (ReST)

Supervised training teaches the format and the definition, but it doesn't know which mistakes matter more. So I let the stage-1 model make the real decision, replace or keep at the 0.70 threshold, on 2,000 training decisions (10,574 pairs), and scored every pair:

+1 for catching a real update
−1 for missing one
−2 for replacing a fact that was still true
+0.25 for correctly keeping a fact

The mean reward was 0.265. There were 15 wrong replacements and 65 missed updates, spread over 52 decisions with a negative reward. Those 52 were relabelled with the truth (repeated twice when they included a −2) and mixed with replayed stage-1 examples, for 366 records in total. A second, gentler pass at a learning rate of 1e-5 took 46 steps, 0.34M tokens and 23 minutes, peaking at 4.98 GB.

Both stages on one T4. About 12 GPU-hours for the whole project, baseline tests included.

I picked the final checkpoint on six separate validation lives (400 decisions), where it scored 0.9991 AUC against 0.9983 for stage 1. The test sets were never used to choose anything.

How I tested it

Two held-out tests, both kept out of training:

A two-year simulated life with template text: 122 decisions, 428 pairs, 186 updates, 45 of them stated outright.
A live run where an LLM wrote the facts from a three-month simulated life, so they read more like real conversation: 127 decisions, 498 pairs, 104 updates, 36 stated. This is the harder and more realistic one.

A stated update says it directly ("I moved to Berlin"). An implied one doesn't, like a new address in another city with no mention of moving. I report both.

The main metric is ROC AUC. Take one real update and one non-update at random: AUC is the chance the model scores the real update higher. 0.5 is a coin flip and 1.0 is perfect. It isn't accuracy, so 0.954 doesn't mean "right 95% of the time". I also check precision and recall at the 0.70 threshold, because that's where the model actually acts.

Results

The live test. Most of the gain comes from SFT; ReST makes the model more careful about replacing.

On the live test, stated updates went from 0.752 as released to 0.929 after stage 1 and 0.945 after stage 2. For any update, including implied ones, it went from 0.708 to 0.873 and 0.872. Jev scores 0.781 and 0.570 on the same test. On the two-year life the stated score went from 0.952 to 0.962, and for any update from 0.583 to 0.840.

At the 0.70 threshold on the live test, precision was 1.00: it never replaced a fact that was still true. The cost is a recall of 0.14, so it only replaces when it's very sure. Kev and Laya as released never crossed the threshold at all. For this job, a cautious model is the right default, and a lower threshold chosen on validation data recovers more of the misses.

Averaged over both tests on stated updates, with every hosted LLM given the same definition:

my fine-tuned Kev (kev-mem): 0.954
Gemini 3.7 Flash: 0.892
Gemma 4 31B: 0.887
GPT-5.4 mini: 0.873
Kev as released: 0.852
Jev: 0.834
DeBERTa-v3-large NLI, zero-shot: 0.815
Laya: 0.711

Size and cost

The model has about 885M parameters: 873.4M in the base, 10.8M in the LoRA adapter and 0.5M in the decision head, so only 11.3M were trained. The adapter is 43 MB. It runs in about 4 GB of GPU memory and takes 0.81 seconds per decision on the T4, without any of the optimised kernels. Counting the baseline tests and a second stage-1 run, the whole project used about 12 GPU-hours.

Caveats

This is my use case and my definition of an update. Jev and Kev are general-purpose models and the hosted LLMs were only prompted, so a lot of the gain comes from specialising. That's the point, but it isn't a general ranking.
These are single training runs, with no confidence intervals yet.
Jev answered 234 of the 249 test decisions; the rest failed after retries.
The test lives are simulated, apart from the LLM-written text of the live run.

Try it

kev-mem-0.8b on Hugging Face, served through Kev's own server, so any Jev client works with it
kev-mem on Kaggle and the benchmark notebook
The memory-update detection benchmark for hosted LLMs

Thanks to TypeSafe for Jev, Jared Palmer for Kev, the Laya authors for open-sourcing their work (Apache-2.0), and the Qwen team for Qwen3.5.

Originally published at abhashchakraborty.tech.

QLoRA fine-tuning: a complete guide for students

PRANJUL RATHOUR — Sun, 04 Oct 2026 04:32:21 +0000

Full fine-tuning updates every parameter in a model, which for anything past a few billion parameters needs GPU memory most students don't have access to. QLoRA sidesteps this by quantizing the base model to 4-bit precision and training small, low-rank adapter matrices on top of it instead.

Why the memory savings are so large

The frozen base model in 4-bit takes a fraction of the memory of a full-precision copy, and the trainable adapters are tiny by comparison. That's how a 1.7B model fine-tunes at around 3.2 GB of peak VRAM instead of needing tens of gigabytes — the exact setup behind FineTune Studio.

The steps, in order

Validate your dataset first. Malformed examples fail loudly hours into training if you don't catch them at upload.
Pick a small base model appropriate to your task and hardware — bigger isn't automatically better for a narrow use case.
Set a learning rate and rank conservatively for a first run; QLoRA is forgiving, but not infinitely so.
Watch the loss curve live, not after the run finishes.
Evaluate base versus tuned on the same prompts. This step is the one people skip and shouldn't.

The honest tradeoff

QLoRA won't match full fine-tuning on every metric. For a narrow, well-defined task with a good dataset, the gap rarely matters — and it's the difference between a project you can actually run and one you can only read about.

Pranjul Rathour · GenAI engineer, 3x hackathon winner, campus mentor. Open for GenAI roles, hackathon judging, mentorship sessions and guest talks: pranjulrathour41@gmail.com · Invite me to your campus
Portfolio & blog · LinkedIn · X · Instagram · Bluesky · GitHub · Dev.to

How to evaluate a fine-tuned model honestly (base vs tuned)

PRANJUL RATHOUR — Fri, 02 Oct 2026 04:32:30 +0000

The most common fine-tuning mistake I see isn't a bad hyperparameter — it's skipping evaluation entirely and assuming the loss curve going down means the model got better at the task you actually care about.

Run the same prompts through both models

Before touching the fine-tuned model, save a fixed set of representative prompts and the base model's answers to them. After training, run the identical prompts through the tuned model. Side by side, not from memory — memory of "how the base model used to answer" is unreliable after a training run.

Score what actually matters for your task

A general benchmark score tells you very little about whether the model got better at your specific narrow task. Write five to ten task-specific questions with known correct answers, and check both models against those.

What a real regression looks like

Sometimes the tuned model is worse on general knowledge and better on your task — that's expected and fine. Sometimes it's worse on both, which means something in the dataset or hyperparameters needs fixing before you ship it. You only find out which one happened if you actually run the comparison.

From benchmarking to fine-tuning: what I learned about small decision models

Gokula Krishna — Wed, 30 Sep 2026 05:47:25 +0000

I tested two small decision models. First I tested TypeSafe AI's hosted model, Jev. Then I tested Laya, an independent open-source alternative. I started out benchmarking and ended up fine-tuning.

Here is what I found, in the order I found it.

Part 1: Benchmarking Jev

TypeSafe AI launched Jev on September 15, 2026. It does not write text. You give it a message and a list of possible answers. It tells you how confident it is in each one.

I skipped the launch demos and ran my own evals on public Kaggle datasets:

31,369 test samples
12 classification tasks
Three areas: anti-money laundering, customer support intent, and malicious network traffic
0.3 s median response time, calling from Singapore to the US

The main lesson: accuracy depends on the task, and even more on how you set the task up.

Describe what the data shows, not what the criminal wants

My first anti-money laundering pattern descriptions said what the criminal was trying to achieve. I rewrote them to describe what the transactions look like.

Accuracy went from 65% to 77%. The model was the same. Only the descriptions changed.

The model sees transactions, not motives. So describe the labels using what's in the input.

Remove hidden bias from the setup

In account-level laundering detection, I switched to using the same time window for every account. That removed a hidden bias in the setup. Accuracy went from 63% to 75%.

Some tasks have no signal

Not everything worked. Classifying a single transaction as laundering or not gave 54%, which is about chance. One transaction on its own doesn't carry enough signal. This is a limit of the task, not of the model.

Confidence you can trust

For customer support intent, Jev picked the right answer 92.3% of the time. On banking intents it was 79.8%.

The confidence score is honest. When Jev said it was 90% sure, it was right about 90% of the time.

That makes thresholds useful. If you accept only predictions above 90% confidence, accuracy rises to 97.6%, and that still covers 82% of cases. The other 18% go to a fallback: a bigger model, a person, or a request for more information.

Ask the question directly

To catch off-topic messages, I first used a low top score as the signal. That gave 91.7%.

Then I added "is this off-topic?" as a separate yes/no question. That gave 97.7%. It costs no extra time, because everything is answered in one pass.

If you care about a decision, ask for it directly. Don't infer it from another score.

The dataset can be wrong too

On malicious network traffic, Jev scored 78% against the dataset labels as given.

One capture had about 400 windows labeled "benign". They were unanswered scans of thousands of hosts. Jev flagged them as malicious with 80% to 90% confidence. Without that capture, the score was 98%.

I report both numbers. The 98% is a filtered result. But the lesson holds: when the model disagrees with the ground truth, look at the data before you count it as a model error.

Part 2: Laya, the open-source alternative

Less than a week later, an open-source alternative appeared. Laya is a separate project, not a release of Jev's weights. It claims performance equal to Jev.

I ran it on 11 of the same tasks, using the same harness I built for Jev.

Accuracy

Task	Jev	Laya (untuned)
Help-desk requests	92.3%	49.7%
Banking questions	79.8%	39.4%
Money-laundering patterns	77.3%	13.5% (no better than guessing)

Across 11 tasks and 3 setups, Laya never beat Jev. On binary (yes/no) decisions it came much closer. On choosing between many answers it fell well behind.

Confidence was the real problem

The accuracy gap was not the biggest issue. The confidence scores were.

Help-desk task: 4,173 of 4,500 answers were at 99% confidence, and about half of them were right.
Banking task: Laya claimed about 97% confidence on almost everything, and got more than half wrong.

Calibration error measures how far confidence is from actual accuracy, where 0 is perfect. Jev scored 0.013. Laya scored 0.486.

That breaks confidence-based routing. With Jev, keeping only answers at 90%+ confidence raised accuracy to 97.6%. With Laya, the same filter dropped 7% of questions and gained only 2 points, to 51.7%.

It also breaks refusals. Of the off-topic questions Laya should have refused, it still answered 88% with 90%+ confidence. Jev did that on 14%.

My conclusion then: I wouldn't use untuned Laya to route between many options.

Part 3: Fine-tuning changed my mind

The next day I fine-tuned Laya on the banking task. It took about 140 minutes on my laptop.

Accuracy went from 39.4% to 78.3%, just 1.5 points below Jev's 79.8%. The task has 77 possible answers, so random guessing would get 1.3%.

The confidence fix mattered more. After fine-tuning, when Laya said it was 97% confident, it was right 96% of the time.

That is one confidence group, not a full calibration curve. But it's a big change from "99% sure, right half the time."

	Jev	Laya (fine-tuned)
Banking accuracy	79.8%	78.3%
Latency	~309 ms (hosted)	~90 ms (local)
Inference cost	~$0.07 per 1,000 requests	$0 (runs on my laptop)
Setup	none	fine-tune per task

The catch is that you need to fine-tune for each task, and that takes task-specific data. I have not yet tested off-topic detection on the fine-tuned model.

Part 4: Laya experts

So I built Laya experts: Laya models fine-tuned for specific tasks. Each one is lightweight, runs on a single machine, and makes a decision in about 80 ms.

One expert detects personally identifiable information (PII). On entity F1, the untuned Laya scored 0.156, Jev 0.736, and the Laya PII expert 0.967. A check like this could run at each stage of a data pipeline to support compliance.

Laya experts is published here: Laya experts.

What I'd tell someone trying these models

These models look like a real "System 1" decision layer. They handle fast classification, routing and other bounded decisions, so an LLM or agent doesn't have to call a generative model for everything.

To get good results, spend less time on the model and more on the setup around it:

Descriptions. Describe labels using what's visible in the input.
Ask directly. Make important decisions their own question.
Check confidence. Test whether "90% sure" really means right 90% of the time. If it doesn't, thresholds won't work.
Set thresholds and a fallback. Decide what happens to the cases the model isn't sure about.
Inspect disagreements. Sometimes the label is wrong, not the model.
Fine-tune when needed. An untuned open-source model can be far off. A short fine-tune on task data can close most of the gap.

Benchmark note: all figures are my own results on public benchmarks, with thresholds tuned per task. They are not vendor claims. Jev 1.13.0 (hosted). Laya 421M via laya-mlx, run locally on an Apple M3. The untuned Laya evaluation covered 23 runs and 65,078 requests.

Originally published at gokulakrishna.co.

Unwrapping the wrappers: who actually trains their own models, and why

Tyler Edwards — Tue, 29 Sep 2026 10:54:39 +0000

Cursor, Intercom, Replit and Perplexity all train specialist models on their own production data. Each picked one narrow, high-volume task, trained on traffic no competitor can access, and shipped a model that matches or beats frontier APIs on that job at lower latency and cost. None of them tried to out-build OpenAI.

Originally published at overmindlab.ai.

Application layer companies, often labelled as 'ChatGPT wrappers', are starting to train their own models. Specialist models, post-trained or fine-tuned on a dataset their competitors cannot touch: their own production traffic.

Four recent examples:

Cursor trained Composer, the model inside its coding agent, with online RL on production agent traces. Delivering frontier-level coding at a fraction of the latency and cost.
Intercom trained Fin Apex, a customer-service model post-trained on billions of support interactions. It bought them control of the resolution-rate metric at scale.
Replit trained a 7B code-repair model, a fine-tune of DeepSeek-Coder-Instruct-v1.5. It bought them a fast, narrow specialist trained on IDE data nobody else has.
Perplexity trained Sonar, its search-answer model, a fine-tune of Llama 3.3 70B served on Cerebras hardware. It bought them sub-second, citation-heavy answers at search scale.

Which companies train their own models?

The same four, with the base model, the method and the measured result in one view.

Company	Model	Base model	How they trained it	Measured result	Source
Cursor	Composer	Not disclosed, a mixture-of-experts architecture	Online RL on production agent traces, inside a harness mirroring Cursor's real tool space; new checkpoints ship several times a day	Generates tokens several times faster than comparable frontier models, landing close to the top tier on internal coding benchmarks	Cursor, Composer 2 report
Intercom	Fin Apex 1.0	Not disclosed	Post-trained on the company's own support interaction data, billions of interactions	73.1% benchmark resolution rate against 71.1% for GPT-5.4 and Claude Opus 4.5; one enterprise customer went from 68% to 75% overnight	Intercom, VentureBeat
Replit	7B code-repair model	DeepSeek-Coder-Instruct-v1.5 (7B)	Fine-tuned on IDE telemetry, scoped to roughly 20 diagnostic categories	Matched or beat GPT-4 and Claude 3 on real-world fixes inside that use case	Replit
Perplexity	Sonar	Llama 3.3 70B	Fine-tuned for factual grounding, citation quality and readability, served on Cerebras wafer-scale hardware	Roughly 1,200 tokens per second, and matched or beat larger frontier models on user satisfaction in Perplexity's own A/B tests	Perplexity, Cerebras

Where we sit, for the record. Overmind is the model training platform for AI teams, and the pattern running through these four case studies is the one it is built for. The case studies themselves are other people's work, cited at source.

Cursor: reinforcement learning on live agent traces

Cursor didn't fine-tune once and ship. The Composer family of models run on real-time RL: model checkpoints go to production, real user interactions become the reward signal, and an improved checkpoint ships several times a day. The model is a mixture-of-experts architecture trained inside a harness built to mirror Cursor's actual tool space, including file edits, semantic search, and terminal commands. The reward signal reflects how developers really use the product rather than a synthetic benchmark, according to Cursor's technical post on Composer.

Cursor reports Composer generates tokens several times faster than comparable frontier models while landing close to the top tier on internal coding benchmarks, a tradeoff explored in more depth by the Composer 2 technical report. Nobody else has Cursor's trace data, which means nobody else can train Cursor's model, even with an identical base checkpoint and an identical training recipe.

What it bought them: coding performance competitive with larger models, at preferential latency and cost, plus a moat provided by their own proprietary data.

Intercom: post-training on the metric the product is sold on

Fin launched in 2023 on top of GPT-4. Fin 2 moved to Claude in late 2024. Then Intercom's roughly 60-person AI research group, led by Fergal Reid, built Fin Apex 1.0, a model post-trained on the company's own support interaction data, as described in the Fin Apex launch post.

One large enterprise customer saw its resolution rate jump from 68% to 75% overnight after the switch to Apex, a 22% cut in unresolved conversations according to Intercom's own numbers. Independent coverage from VentureBeat puts Apex's benchmark resolution rate at 73.1%, against 71.1% for both GPT-5.4 and Claude Opus 4.5.

Model	Benchmark resolution rate
Fin Apex 1.0	73.1%
GPT-5.4	71.1%
Claude Opus 4.5	71.1%

Intercom's own model documentation breaks the pipeline into separate retrieval, reranking, and generation models rather than one monolithic system, detailed on the Fin CX models page.

Worth noting: Intercom hasn't abandoned rented models. Apex is layered on top of retrieval and routing components, and the company still uses third-party models elsewhere. This architecture is common as companies balance performance, latency and cost.

Replit: a narrow 7B specialist trained on IDE telemetry

Replit's bet was the opposite of "bigger is better." The team fine-tuned a 7B open-weights model, DeepSeek-Coder-Instruct-v1.5, chosen after testing several code-focused base models and documented in its code-repair write-up.

The model targets a specific use case, roughly the 20 diagnostic categories responsible for the majority of everyday coding errors, and the fine-tuned 7B checkpoint matched or beat GPT-4 and Claude 3 on real-world fixes in that use case. Small open source models can beat much larger general-purpose models at many tasks.

Perplexity: search-optimised inference at 1,200 tokens per second

Perplexity fine-tuned Meta's Llama 3.3 70B into Sonar, tuned specifically for factual grounding, citation quality, and readability in its default search mode, as the company explains in its Sonar launch post. Sonar runs on Cerebras wafer-scale inference hardware, which Perplexity and Cerebras jointly announced pushes throughput to roughly 1,200 tokens per second, an order of magnitude faster than comparable models on conventional GPU inference.

In Perplexity's own A/B testing, Sonar matched or beat larger frontier models on user satisfaction inside search, while running far cheaper and faster than calling those models directly.

Why build a specialist model instead of prompting a frontier one?

Calling a frontier model for every request is sometimes the equivalent of running a data centre to toast bread. It works, but it's the wrong tool for a narrow, high-volume, well-defined job. Across these four companies, three conditions kept showing up:

A narrow task, not a general one. Small specialised models are excellent at one narrow task. Don't ask them to be broadly intelligent.
Proprietary production data no competitor can access. Cursor has Cursor's traces. Intercom has Intercom's support conversations. Your production traffic is the one dataset nobody can buy.
Enough volume that per-token latency and cost actually move the P&L. Fine-tuning a model can be expensive. It only pays off once inference volume is high enough that shaving cost per call compounds.

None of these companies want to out-build OpenAI or Anthropic on general intelligence. Each trained a model that wins one specific task, using the one dataset it uniquely owned.

What does the traces-to-training loop look like?

Stage	What it produces
Production runs	Traces of what the agent actually did
Labelled dataset	Those traces, curated and labelled
Fine-tuned model	A specialist trained on that dataset
Redeploy	The specialist back in production, generating the next round of traces

Capture production traces, turn them into a labelled dataset, train a model on it, and ship that model back into production. That's where the application layer is going. We break the middle two stages down in how do you turn traces into a training dataset and how to train your agent.

FAQ

Do companies that train their own models still use frontier APIs?

Usually yes, alongside. Intercom hasn't abandoned rented models. Fin Apex sits on top of retrieval and routing components, and the company still uses third-party models elsewhere. A mix of owned and rented models is the common architecture, chosen to balance performance, latency and cost.

Do you need a large model to beat a frontier one?

Not on a narrow task. Replit's 7B fine-tune of DeepSeek-Coder-Instruct-v1.5 matched or beat GPT-4 and Claude 3 on real-world fixes inside its target diagnostic categories. Perplexity's Sonar, a 70B fine-tune, matched or beat larger frontier models on user satisfaction inside search.

What data do you need to train a specialist model?

Your own production traffic. Cursor used live agent traces, Intercom used its support interactions, Replit used IDE telemetry, and Perplexity tuned against its own search behaviour. The common ingredient is a dataset the product generates itself, which is the one thing a competitor cannot buy.

When does training your own model not pay off?

When any of the three conditions is missing. A broad task rather than a narrow one, no proprietary data sitting behind it, or volume too low for per-call savings to compound. Fine-tuning costs money, so it only pays back once inference volume is high enough.

Overmind is the model training platform for AI teams. It turns your production traces into specialised models you own. Get started.

How to Run Unsloth on AMD GPUs (ROCm) [2026 Guide]

Kunal — Sun, 27 Sep 2026 12:43:38 +0000

Originally published at kunalganglani.com — read it there for inline code, hero image, and live links.

You don’t need another “it installed!” ROCm post. You need Unsloth fine-tuning on an AMD GPU to actually train, with a loss curve that goes down for the right reasons.

By the end of this guide, you’ll have Unsloth fine-tuning running on an AMD GPU with ROCm, plus a quick sanity run that proves it’s converging (not just “it launched”). If your setup is close to supported, you can get from zero to a minimal LoRA run in ~45–90 minutes. If you’ve been stuck in “HIP error / no kernel image available” hell, plan for an afternoon.

This is an Unsloth AMD GPU ROCm fine tuning guide written for late‑2026 reality: ROCm support is way better than it was, but it’s still version-fragile, kernel-fragile, and “works” is not the same as “trains fast and converges”.

I’m going to be blunt. Most AMD fine-tuning failures aren’t because Unsloth is “broken”. They’re because your stack is a Jenga tower: kernel + driver + ROCm userland + PyTorch ROCm wheels + Triton/attention kernels + mixed precision. One mismatch and you get either a hard crash or, worse, silent slow fallbacks.

(Visual break: matrix screenshot)

What is Unsloth?

Unsloth is a training and inference optimization toolkit that patches Hugging Face Transformers/TRL flows to make fine-tuning large language models faster and more memory-efficient, typically using LoRA and related parameter-efficient methods.

Where Unsloth earns its keep is (a) smart model patching and (b) optional high-performance kernels. On NVIDIA/CUDA, those kernels are usually just… there. On AMD/ROCm, kernel availability depends on your exact GPU architecture and software versions. That’s the whole game.

If you’re still deciding whether you need fine-tuning at all, read my decision framework on fine-tuning vs RAG (retrieval-augmented generation) and prompt work.

Unsloth AMD GPU ROCm fine tuning guide: the late‑2026 compatibility matrix

Here’s the matrix I wish existed the first time I tried to help someone debug Unsloth on a Radeon card. It’s not “official”. It’s the set of combinations that tend to behave in the real world, plus the landmines that waste your time.

Two anchors before we start:

Treat the PyTorch ROCm wheel selector as the source of truth. PyTorch’s install page currently shows Stable PyTorch 2.7.0 and ROCm wheel availability for ROCm 6.3 (that compute platform option exists right in the selector). If your installed ROCm runtime is 6.1 and you pip-installed a 6.3 wheel, you’re basically asking for missing hip libraries and runtime weirdness. See the official selector on PyTorch.
ROCm itself ships a published compatibility matrix and install guidance. Start from AMD’s docs, not random Reddit incantations. AMD ROCm documentation has an explicit “Compatibility matrix” section and per-component pages.

Known-good stacks (pin these first)

“Known-good” means one thing: you can run a small SFT, loss decreases smoothly, no NaNs, and throughput isn’t obviously crippled. Anything else is a science project.

GPU class	Example GPUs	OS baseline	ROCm baseline	PyTorch ROCm baseline	Attention kernel mode	Quantization path	Notes
Consumer RDNA3 (24GB)	RX 7900 XTX (24GB)	Ubuntu 24.04 / kernel 6.x	ROCm 6.2–6.3	torch 2.6–2.7 ROCm build	SDPA first, try FlashAttn later	FP16 LoRA	Most reliable Radeon path. SDPA is usually “good enough”.
Consumer RDNA3 (16GB)	RX 7800 XT (16GB)	Ubuntu 24.04	ROCm 6.2–6.3	torch 2.6–2.7 ROCm build	SDPA	FP16 LoRA + grad ckpt	VRAM is the constraint. Keep seq len sane (see VRAM section).
Consumer RDNA4	RX 8800/8900 class	Ubuntu 24.04+	ROCm 6.3+	torch 2.7 ROCm build	SDPA	FP16 LoRA	Newer arch usually means better kernel coverage, but also sharper version coupling.
Datacenter CDNA (MI series)	MI250/MI300/MI350	ROCm-certified distros	ROCm 6.2–6.3+	torch matched to ROCm	FlashAttn more likely	FP16/BF16 LoRA	MI parts tend to have fewer “invalid device function” surprises.

A few sharp edges I’ll save you from:

If you’re on Windows: ROCm exists there, but Unsloth fine-tuning is still dramatically less predictable than on Linux. If your goal is “fine-tune this weekend”, use Linux.
If you’re on WSL2: it’s great for a lot of dev. For ROCm fine-tuning, you’re stacking two “almost works” layers. If you want my honest recommendation: don’t.

Based on the benchmark data I maintain at kunalganglani.com/llm-benchmarks, the pattern I keep seeing across non-NVIDIA platforms is: getting the model to load is easy; getting stable throughput is the real work. ROCm fine-tuning is the same story.

(Visual break: “known good” command snippet screenshot)

Install ROCm and verify HIP works (rocminfo/hipcc)

This section is intentionally boring. Boring is the point. If you skip verification, you’ll spend the next 6 hours blaming Unsloth for a driver/runtime problem.

1) Pick a supported OS and kernel first

Start with a mainstream distro ROCm expects. In 2026, Ubuntu LTS is the pragmatic choice.

Ubuntu 24.04 LTS is a solid baseline.
Linux kernel 6.x is common.

The moment you go off-road (custom kernels, odd GCC/libstdc++ combos), you’re volunteering to debug C++ ABI errors instead of training models.

2) Install ROCm from AMD’s docs

Use AMD’s official install steps and their compatibility matrix: AMD ROCm documentation.

Practical advice I stick to:

Keep your ROCm install single-source. Don’t mix distro packages + random tarballs + Docker bind mounts unless you know exactly why.
If you’re using containers (often best for Radeon), make sure the host driver stack matches what the container expects. ROCm is not magic.

3) Verify `rocminfo` and `hipcc`

After install:

# should print GPU + agent info
/opt/rocm/bin/rocminfo | head -n 50

# should print ROCm/HIP compiler info
/opt/rocm/bin/hipcc --version

If rocminfo can’t see your GPU, stop. Don’t “try pip installs until it works”. Fix the driver/runtime first.

Common failure modes:

rocminfo: HSA runtime not found → ROCm userland isn’t installed correctly or not on PATH.
Permission issues accessing /dev/kfd → user not in the right group (often video/render).

Install PyTorch for ROCm (matching wheel to ROCm)

This is the #1 root cause of “it imports, but training explodes”. The PyTorch ROCm wheel bakes assumptions about the ROCm runtime.

The only rule that matters

Your installed ROCm runtime version and your PyTorch ROCm build must match.

PyTorch makes this easy if you actually use their selector. On the official page, the compute platform dropdown includes ROCm 6.3 and generates the matching --index-url command. That selector is maintained by the people shipping the wheels. Use it: PyTorch.

Example (ROCm wheel install style):

python3 -m venv .venv
source .venv/bin/activate

# example only. generate your exact command from pytorch.org
pip install --upgrade pip
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/rocm6.3

Verify PyTorch sees the AMD GPU

python - <<'PY'
import torch
print('torch', torch.__version__)
print('hip available', torch.cuda.is_available())
print('device', torch.cuda.get_device_name(0) if torch.cuda.is_available() else None)
PY

Yes, it still prints cuda in APIs. No, you’re not secretly using NVIDIA.

If is_available() is false:

You installed a CPU wheel.
Or you installed a ROCm wheel that doesn’t match your ROCm runtime.

If you see linker errors about missing libamdhip64.so or friends, you’re mismatched.

Set up Transformers + TRL + Unsloth

Unsloth rides on top of Hugging Face Transformers and TRL.

Unsloth repo: Unsloth AI
TRL docs for trainer knobs (packing, clipping, etc.): TRL

My opinionated dependency strategy for AMD

On CUDA I’m okay with “latest-ish” because the ecosystem is brutally well-tested.

On ROCm I pin. Always.

Pin your layers like you mean it:

Pin torch to the ROCm wheel you installed.
Pin transformers, trl, accelerate, peft to a tested set.
Upgrade only one layer at a time.

Example:

pip install "transformers==4.56.*" "trl==0.12.*" "accelerate==1.0.*" "peft==0.13.*"

# install unsloth
pip install "unsloth[rocm]" || pip install unsloth

If the extra [rocm] doesn’t exist in the current Unsloth packaging when you read this, that’s fine. The point is: install Unsloth, then verify what optional kernels it pulled in.

Attention kernels: assume SDPA first

One of the most common traps is people assuming FlashAttention is mandatory because every NVIDIA guide reads like it’s oxygen.

On AMD, start with SDPA (PyTorch scaled dot-product attention). Chase FlashAttention only after you have a converging baseline.

Why I’m so stubborn about this:

FlashAttention availability on ROCm is architecture- and version-dependent.
The failure mode is often not “missing package”. It’s “Triton compiled something, ran it, and now you get NaNs at step 200.” That’s the worst kind of failure.

Run a minimal LoRA/SFT sanity check (prove it converges)

If you only do one thing from this post, do this. Don’t launch your full dataset run first. That’s how you burn an evening.

The 5-step checklist (minimal moving parts)

Confirm rocminfo sees the GPU.
Confirm PyTorch ROCm sees the GPU (torch.cuda.is_available() is true).
Run a single-batch forward+backward in fp16.
Run 50–200 training steps on a tiny dataset.
Confirm loss decreases smoothly and no NaNs appear.

That’s your “stack is sane” certificate.

Minimal TRL SFTTrainer script (ROCm-friendly defaults)

This uses conservative knobs: fp16, gradient checkpointing, modest learning rate, clipping.

# sft_rocm_sanity.py
import os
os.environ.setdefault("TOKENIZERS_PARALLELISM", "false")

import torch
from datasets import Dataset
from transformers import AutoTokenizer
from trl import SFTTrainer, SFTConfig
from peft import LoraConfig

from unsloth import FastLanguageModel

model_name = "meta-llama/Meta-Llama-3-8B-Instruct"  # 8B class is a sane starting point

# Tiny dataset on purpose
data = Dataset.from_list([
  {"text": "Write a haiku about GPUs."},
  {"text": "Explain what ROCm is in one paragraph."},
  {"text": "Summarize the difference between LoRA and full fine-tuning."},
] * 200)

max_seq_length = 512

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name=model_name,
    max_seq_length=max_seq_length,
    dtype=torch.float16,
    load_in_4bit=False,   # start without QLoRA on AMD
)

lora_config = LoraConfig(
    r=16,
    lora_alpha=32,
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM",
    target_modules=["q_proj","k_proj","v_proj","o_proj","gate_proj","up_proj","down_proj"],
)

trainer = SFTTrainer(
    model=model,
    tokenizer=tokenizer,
    train_dataset=data,
    peft_config=lora_config,
    args=SFTConfig(
        output_dir="./out_rocm_sanity",
        per_device_train_batch_size=1,
        gradient_accumulation_steps=8,
        learning_rate=2e-4,
        warmup_ratio=0.03,
        max_steps=200,
        logging_steps=10,
        fp16=True,
        bf16=False,
        gradient_checkpointing=True,
        max_grad_norm=1.0,
        packing=False,
        report_to=[],
    ),
)

trainer.train()
print("done")

Run it:

python sft_rocm_sanity.py

What “good” looks like:

Loss starts high and trends down over the first 50–200 steps.
No nan loss. No sudden spikes to absurd values.
GPU utilization is non-zero and stable.

If you get NaNs, don’t immediately blame ROCm. Do the boring trainer triage first:

Lower LR from 2e-4 → 1e-4.
Ensure max_grad_norm=1.0 is set.
Turn off packing.

TRL documents these knobs and what they change: TRL.

Troubleshoot common ROCm errors (HIP runtime, missing libs, kernel image issues)

Most guides turn this section into an error-message graveyard. That’s useless. You want a decision tree.

1) “HIP error / invalid device function / no kernel image available”

Translation: a compiled kernel doesn’t match your GPU architecture, or you installed a build that simply wasn’t compiled for your card.

Fix path:

Confirm your GPU is actually supported by your ROCm version in AMD’s matrix: AMD ROCm documentation.
Confirm your PyTorch wheel matches your ROCm runtime. Reinstall PyTorch using the official selector: PyTorch.
If this happens when enabling FlashAttention/Triton kernels: force SDPA and disable fused attention paths.

Useful env knobs to try (these move around across versions, so treat this as a debugging tactic, not a forever setting):

export TORCH_LOGS="+dynamo,+inductor"
export TORCHDYNAMO_VERBOSE=1

If you see Triton compilation errors referencing HIP, that’s your sign to stop chasing fancy kernels and get a clean SDPA baseline.

2) Missing `hipblaslt` / `rccl` / shared library errors

This is almost always packaging mismatch:

ROCm userland not fully installed.
Container image expects a different ROCm stack than the host.

My stance: containers are usually more reliable on Radeon, because you can pin userland and Python deps together. But your host driver still matters.

If you want the broader ROCm vs CUDA story for local AI, I wrote that comparison already. This post is narrower: making Unsloth fine-tuning stop failing.

3) bitsandbytes fails on ROCm (4-bit/8-bit)

This question shows up constantly: “Why does bitsandbytes 4-bit/8-bit quantization fail on ROCm?”

Because bitsandbytes has historically been CUDA-first. ROCm support has improved, but it’s still fragile, and it lags the newest training stacks.

Practical alternatives that don’t waste your weekend:

FP16 LoRA (the baseline I recommend in this post).
8-bit optimizers only if your ROCm build supports the exact kernel path you need.
If your goal is cheap inference, skip training quantization entirely. Use GGUF for inference-only. I’ve written about quantization tradeoffs in LLM quantization levels and GGUF vs GPTQ vs EXL2.

If you’re on a 16GB Radeon card and trying to QLoRA a 13B model because TikTok told you it’s “easy”, stop. Use a smaller base model or add VRAM.

4) “It runs” but throughput is awful (kernel fallbacks)

This is the sneaky failure. No crash. No obvious error. Just performance that feels like you’re training through a straw.

Symptoms:

GPU utilization is low.
Tokens/sec is way lower than expected.
You see unexpected CPU usage spikes.

How to diagnose:

Turn on PyTorch compile logs (TORCH_LOGS above) to see if Inductor/Triton is bailing.
Start from SDPA. If FlashAttention “works” but is slower, you’re probably hitting a fallback path.

Also: measure something real.

In my benchmark work for this site, throughput is the metric people hand-wave the most. That’s why I keep pushing tok/s and TTFT in posts like local LLM benchmark methodology.

5) NaNs, divergence, and fake convergence

On AMD you’ll sometimes get “training” that doesn’t crash but is numerically unstable.

Things I check, in this order:

Loss: do you ever see nan? If yes, don’t continue.
Gradient clipping: set max_grad_norm to 1.0 (or even 0.5).
Precision: prefer fp16 first. bf16 can be great, but bf16 stability varies by GPU + stack.
Sequence packing: turn it off until you’re stable.

Rule of thumb VRAM budgets (practical, not theoretical):

7B/8B LoRA: 16GB can work with small batch, seq len ~512–1024, grad checkpointing.
13B LoRA: 24GB is the sane starting point for non-suffering.
Anything beyond: either multi-GPU (more pain on Radeon) or change strategy.

If you want the bigger hardware sizing story for local LLM work, I have a dedicated guide. This post stays focused on Unsloth + ROCm.

Here’s an official baseline walkthrough from AMD itself. It’s worth watching once to calibrate your environment expectations:

[YOUTUBE:GNF32XuRAN4|How to Fine-Tune a Model on AMD GPUs using LoRA]

Container vs bare-metal for Radeon: my default recommendation

If you’re on consumer Radeon, I generally recommend containers for fine-tuning.

Reasons:

You can pin Python deps and ROCm userland together.
You can share a known-good image with your team.
You reduce “my system Python upgraded libstdc++ and now hip runtime crashes” incidents.

Bare-metal is fine too if you’re disciplined. Most people aren’t.

If you do bare-metal:

Keep a dedicated venv per ROCm/PyTorch stack.
Don’t upgrade blindly.

This mindset is similar to what I preach for reproducible dev envs in general, like in my Python dev setup and Linux migration plan posts.

My prediction for 2027: “ROCm works” won’t be enough

By 2027, “can I run it?” will be boring. It’ll be solved for AMD consumer GPUs.

The dividing line will be: can you update your stack without accidentally losing fast kernels, falling back to slow paths, or reintroducing numerical weirdness? NVIDIA has earned that kind of stability. AMD is getting closer, but Unsloth-style optimization stacks are exactly where the cracks still show.

If you’re building a serious fine-tuning workflow on Radeon today, don’t treat this as a one-off setup. Turn it into a pinned, scripted environment you can recreate on a fresh machine. That’s how you stop ROCm from becoming a recurring calendar event.

Originally published on kunalganglani.com

Enterprise Fine-Tuning Services: Scale Smarter AI Solutions

Michael Keller — Thu, 24 Sep 2026 05:58:17 +0000

Enterprise organizations are adopting artificial intelligence to improve operational efficiency, support employees, automate repetitive tasks, and deliver more personalized customer experiences. However, general-purpose AI models may not always perform effectively when applied to specialized business processes. Enterprise Fine Tuning Services help organizations adapt AI models to specific operational requirements, domain terminology, response formats, and workflow expectations.

Large organizations often manage complex systems, multiple departments, diverse datasets, and strict security requirements. A model that performs well in a general demonstration may produce inconsistent results when introduced into a real enterprise environment. Fine-tuning can help address certain behavioral and task-specific limitations, particularly when a business has reliable training data and a clearly defined objective.

Enterprise fine-tuning is not simply about training a model with more information. It requires careful planning around model selection, data preparation, infrastructure, security, evaluation, deployment, and ongoing maintenance. Business leaders must understand when customization creates value, when other approaches are more suitable, and how to scale AI responsibly across the organization.

What Are Enterprise Fine-Tuning Services?

Enterprise fine-tuning services involve adapting a pretrained AI model to meet the requirements of a particular organization or business function. The process uses curated examples to improve how a model performs a specific task or follows a defined response pattern.

Depending on the use case, fine-tuning may help a model learn:

Industry-specific terminology
Organization-specific communication patterns
Structured classification categories
Document processing formats
Specialized customer service responses
Internal workflow instructions
Domain-specific language patterns
Consistent output structures

Fine-tuning does not automatically provide continuous access to current enterprise information. A model trained on historical examples may still require retrieval systems, application integrations, or external tools to access updated policies, records, and business data.

The objective is to adapt model behavior for a particular task, not to assume that fine-tuning alone can solve every enterprise AI requirement.

The 2027 Outlook for Enterprise AI Fine-Tuning

As enterprise AI adoption develops, organizations are likely to combine multiple customization methods based on their workflows, data, and performance requirements. Fine-tuning may become one component of a broader AI architecture that includes retrieval, automation, governance, and evaluation.

Expected Direction in 2027	Potential Development	Enterprise Business Implication
Specialized model deployment	Organizations may customize models for individual departments and business functions	AI systems can be aligned more closely with operational needs
Efficient training methods	Businesses may explore parameter-efficient fine-tuning and smaller specialized models	Some use cases may require fewer computational resources
Integrated evaluation	Fine-tuned models may be tested through structured business-specific benchmarks	Enterprises can make more informed deployment decisions
Hybrid AI architectures	Fine-tuning may be combined with retrieval, tools, and workflow automation	Businesses can use different methods for different requirements

These are strategic expectations rather than confirmed outcomes. Their success will depend on model capabilities, dataset quality, infrastructure, governance, security, and organizational readiness.

Why Enterprises Consider Model Fine-Tuning

General-purpose AI models are designed to handle many types of tasks. This flexibility makes them useful for broad applications, but it can create limitations when organizations need highly consistent performance in specialized workflows.

For example, an enterprise may require an AI system to classify thousands of service requests according to internal categories. A general-purpose model may understand the overall meaning of a request but struggle to apply the company’s exact classification rules consistently.

Fine-tuning may be considered when:

The task is repeated frequently
The organization has high-quality examples
The desired behavior is difficult to achieve through prompting alone
Consistent formatting is important
The workflow requires specialized terminology
The business needs predictable task-specific performance
The organization can support testing and ongoing evaluation

Fine-tuning should be assessed against alternatives. If a workflow mainly requires access to current documents, retrieval-augmented generation may be more appropriate. If the issue is unclear instructions, prompt engineering may be sufficient. If the task follows fixed rules, traditional software logic may offer more predictable results.

Enterprise Fine-Tuning Compared With Other AI Customization Methods

A strategic AI implementation begins with identifying the actual problem.

Prompt Engineering

Prompt engineering improves how instructions are written for an existing model. It can define roles, context, task requirements, output formats, and restrictions.

This approach may be suitable when:

The model already understands the subject
The task changes frequently
The organization needs rapid iteration
Training data is limited
The desired improvement can be achieved through clearer instructions

Retrieval-Augmented Generation

Retrieval-augmented generation connects an AI model with external sources such as internal documentation, knowledge bases, and business databases.

RAG is often useful when the system needs access to information that changes regularly. It can provide relevant context without requiring the model to be retrained whenever a document or policy is updated.

However, retrieval quality, source permissions, document structure, and citation behavior must be evaluated.

Fine-Tuning

Fine-tuning adapts model behavior through additional training on task-specific examples. It may be appropriate for classification, formatting, response style, and other repeatable behaviors.

Fine-tuning and RAG can be used together. For example, a model may be fine-tuned to follow a specific output format while retrieval supplies current enterprise information.

Common Enterprise Applications

Fine-tuning can support multiple enterprise use cases, but the suitability of each application depends on data quality, risk level, and performance requirements.

Enterprise Function	Potential Fine-Tuning Use Case	Possible Business Benefit
Customer support	Classify customer requests and follow approved response patterns	More consistent handling of recurring inquiries
Document processing	Extract or categorize information from specialized documents	Reduced manual organization and processing effort
Internal knowledge operations	Generate structured summaries using defined formats	More consistent internal documentation
Financial operations	Categorize selected records or identify predefined data patterns	Improved support for repetitive information processing
Sales enablement	Organize account information and classify customer intent	More standardized preparation for sales activities

These use cases require appropriate validation. Fine-tuned models should not be treated as autonomous decision-makers in sensitive processes without suitable controls and human oversight.

The Enterprise Fine-Tuning Workflow

A scalable fine-tuning project should follow a structured development process.

Business Objective → Data Assessment → Model Selection → Training → Evaluation → Controlled Deployment → Continuous Monitoring

Each stage should have clear ownership, success criteria, and documentation.

1. Define the Business Objective

The project should begin with a specific operational or strategic goal. Instead of pursuing model customization simply because it is technically possible, enterprises should identify the business problem they want to address.

Possible objectives include:

Improving document classification
Reducing repetitive manual processing
Standardizing customer response drafts
Supporting specialized information extraction
Improving the consistency of internal summaries
Reducing the number of corrections required in a workflow

The objective should be connected to measurable outcomes. For example, a business may evaluate whether fine-tuning reduces classification errors or improves the consistency of structured outputs.

2. Assess the Data

Training data is a major factor in the success of fine-tuning. Enterprises should review the availability, quality, ownership, and permitted use of the data before beginning development.

Important considerations include:

Is the data relevant to the task?
Does it represent real business scenarios?
Are the labels consistent?
Does it contain duplicate or conflicting examples?
Is sensitive information handled appropriately?
Does the dataset include difficult and unusual cases?
Is the data sufficiently current for the intended use?

A large dataset is not automatically a high-quality dataset. Poor examples can teach the model incorrect patterns and create additional evaluation challenges.

3. Select an Appropriate Model

Model selection should consider the intended task and deployment environment. Relevant factors may include:

Model capabilities
Fine-tuning support
Context requirements
Inference cost
Response latency
Hosting options
Security requirements
Integration compatibility
Maintenance complexity

Enterprises should evaluate whether a large model, smaller specialized model, or existing open-source model is appropriate for the workflow. The most advanced model is not always the most practical option for a narrow and repetitive task.

Training Data and Enterprise Governance

Enterprise data often includes confidential information, customer records, internal policies, and commercially sensitive material. Fine-tuning projects must therefore incorporate governance from the beginning.

Data Privacy

Organizations should determine whether data can be used for training and whether sensitive information must be removed, anonymized, or protected through other methods.

Access Management

Only authorized teams should be able to access training datasets, model checkpoints, evaluation results, and deployment environments.

Data Lineage

Teams should document where training examples originated, how they were processed, and which version of the dataset was used. This information supports troubleshooting and audit requirements.

Policy Alignment

The fine-tuning workflow should align with applicable organizational policies, contractual obligations, and relevant regulatory requirements. Specific controls will vary depending on the industry and use case.

Dataset Maintenance

Enterprise data changes over time. New products, customer behaviors, policies, and workflows may create patterns that are not represented in the original dataset. The organization should define when data needs to be reviewed or updated.

Evaluating Fine-Tuned Enterprise Models

Evaluation should determine whether the fine-tuned model performs better for the intended task than an appropriate baseline.

A baseline might be:

A general-purpose model
A model using improved prompts
A retrieval-based workflow
An existing manual process
A traditional machine learning system

Evaluation criteria should reflect the actual use case.

For a classification workflow, useful measures may include accuracy, precision, recall, and error distribution. For structured content generation, the evaluation may focus on completeness, formatting, relevance, and adherence to business requirements.

Testing should include:

Common inputs
Unusual examples
Incomplete information
Ambiguous requests
Conflicting information
Out-of-scope questions
Inputs that require escalation

A model should not be approved based only on examples it handled successfully. Enterprises need to understand where the system fails and how those failures will be managed.

Executive Questions Before Scaling Fine-Tuning

What problem will fine-tuning solve?

Leaders should define the specific limitation that requires model customization. If the problem can be solved through better prompts, improved retrieval, or workflow rules, fine-tuning may introduce unnecessary complexity.

Do we have reliable training examples?

The organization should evaluate whether the available data accurately represents the intended task. Missing categories, inconsistent labels, and low-quality examples can reduce the value of the training process.

How will the model be monitored after deployment?

Enterprise AI systems require ongoing monitoring because data patterns, user expectations, and business requirements can change. Teams should define how performance issues will be detected and addressed.

What are the consequences of incorrect outputs?

The risk associated with an incorrect classification or response varies by workflow. Customer communications, financial processes, and sensitive internal operations may require additional approval and validation.

Can the solution scale economically?

The total cost may include data preparation, training, infrastructure, inference, evaluation, security, monitoring, and maintenance. Leaders should assess the complete operating model rather than only the initial development expense.

Who owns the model after deployment?

Ownership should cover performance monitoring, dataset updates, version control, incident response, and decisions about retraining or rollback.

Challenges in Enterprise Fine-Tuning

Data Inconsistency

Enterprise datasets may come from different systems and departments. Differences in terminology, formatting, and labeling can make training more difficult.

Model Overfitting

A model may memorize or over-adapt to training examples instead of learning patterns that generalize to new inputs. Separate evaluation data and carefully designed testing can help identify this issue.

Model Drift

Performance may change as business processes, customer behavior, and data patterns evolve. Ongoing monitoring is necessary to determine whether the model remains suitable.

Infrastructure Requirements

Training and hosting may require specialized infrastructure, technical expertise, and cost management. Enterprises should assess whether the chosen approach fits their operational environment.

Security Exposure

Sensitive training data, model artifacts, and deployment interfaces may introduce security risks. Access control, monitoring, encryption, and appropriate data handling procedures should be considered.

Maintenance Complexity

Fine-tuned models require ongoing management. Organizations may need to update datasets, evaluate new versions, monitor quality, and maintain compatibility with other enterprise systems.

Excessive Reliance on Model Output

A fine-tuned model may produce consistent but incorrect responses. Enterprises should establish review procedures for high-impact tasks and avoid treating model confidence or fluency as proof of accuracy.

Building a Scalable Enterprise Fine-Tuning Strategy

A sustainable strategy should consider both technical performance and organizational adoption.

Establish a Pilot Project

Begin with a well-defined workflow that has measurable outcomes and manageable risk. A pilot allows the team to evaluate the training process, deployment requirements, and user feedback before expanding the solution.

Create Reusable Standards

Develop common standards for data preparation, testing, documentation, security, and deployment. These standards can reduce duplicated effort across departments.

Separate Development and Production

Testing environments should be separated from production systems where appropriate. This helps teams evaluate changes without affecting active business workflows.

Introduce Version Management

Track model versions, training datasets, prompts, configuration settings, and evaluation results. Version management supports troubleshooting and controlled releases.

Plan for Rollback

If a new model version performs poorly, the organization should have a process for reverting to a previously approved version or switching to a fallback workflow.

Train Employees

Employees should understand the model’s purpose, limitations, appropriate usage, and escalation procedures. Technical deployment alone does not guarantee successful adoption.

Measuring Enterprise Business Value

Fine-tuning projects should be evaluated through business and technical measures. Potential indicators include:

Task-specific accuracy
Reduction in manual corrections
Processing time
Output consistency
Cost per request
Response latency
Employee adoption
Escalation frequency
Error severity
Workflow completion rates

Measurements should be collected before and after implementation when possible. Organizations should also examine whether improvements in one area create new costs or risks elsewhere.

For example, a model may reduce processing time but produce more errors that require additional review. A meaningful assessment should consider both efficiency and output quality.

The Future of Enterprise AI Customization

Enterprise AI architecture is likely to include multiple models and customization methods. Organizations may use smaller fine-tuned models for focused tasks, larger models for complex requests, and retrieval systems for access to current information.

Model routing may allow businesses to select a system based on task complexity, cost, latency, security, and accuracy requirements. This approach can reduce the pressure to use a single model for every workflow.

Fine-tuning may also become more closely integrated with AI agents and enterprise automation. However, greater integration increases the importance of permissions, monitoring, evaluation, and human oversight.

Businesses should view fine-tuning as one component of a broader AI strategy. Its value will depend on how well it supports actual business processes and whether the organization can maintain the solution over time.

Conclusion

Enterprise Fine-Tuning Services can help organizations adapt AI models to specialized workflows, business terminology, response formats, and operational requirements. When supported by high-quality data, appropriate model selection, and structured evaluation, fine-tuning may improve consistency and task-specific performance.

However, fine-tuning is not a universal solution. Enterprises should compare it with prompt engineering, retrieval-augmented generation, traditional software, and other customization approaches before making an investment.

For business leaders, successful enterprise fine-tuning requires more than model training. It involves data governance, security, infrastructure planning, evaluation, monitoring, employee adoption, and long-term ownership. A structured approach can help organizations scale AI solutions while maintaining control over performance, cost, and risk.

Frequently Asked Questions

1. What are enterprise fine-tuning services?

Enterprise fine-tuning services adapt pretrained AI models to specific organizational tasks, domains, workflows, or response requirements using curated training data.

2. Why do enterprises need fine-tuned AI models?

Enterprises may need fine-tuned models when general-purpose systems do not consistently meet specialized task requirements and when prompting or retrieval alone is insufficient.

3. Is fine-tuning suitable for every enterprise AI project?

No. Fine-tuning may not be necessary when the main requirement involves current information, simple instructions, or fixed business rules. Organizations should evaluate alternative approaches before selecting fine-tuning.

4. What data is needed for enterprise fine-tuning?

The data should be relevant, accurate, representative, and consistently formatted. It should reflect the intended task and include important variations, edge cases, and examples requiring escalation.

5. How can enterprises evaluate a fine-tuned model?

Organizations can compare the model with an appropriate baseline using task-specific metrics such as accuracy, consistency, relevance, latency, cost, and error frequency.

6. What are the risks of enterprise fine-tuning?

Potential risks include poor training data, overfitting, model drift, security exposure, maintenance costs, and inaccurate outputs. Governance, testing, and monitoring help manage these risks.

7. Can fine-tuning work with RAG?

Yes. Fine-tuning can adapt model behavior, while RAG can provide access to relevant and updated external information. The combination should be tested according to the workflow’s requirements.