Fine-Tuning Explained (And When You Actually Need It Instead of RAG)

After I wrote about RAG, a bunch of comments asked some version of the same question. "Okay but what if I want the model to actually behave differently, not just answer using my documents." Good question, and the honest answer is that's a different problem with a different tool. That tool is fine-tuning.

I put off learning this properly for a long time because it sounded expensive and complicated, something only big companies with server farms do. Turns out a basic understanding of it is way more approachable than I expected, and knowing when NOT to use it is honestly the more valuable lesson. So let's get into it properly, real example first, then actual code.

A real life example to set the stage

Think about training a new employee at a customer support job.

If you hand them a manual and tell them "look things up in here whenever you're unsure," that's RAG. They still talk and think like themselves, they just consult reference material when they need specific facts. Their personality, tone, and general way of communicating hasn't changed at all, they've just got a good reference book next to them.

Now imagine instead you actually trained that person for months, had them shadow senior staff, corrected their tone repeatedly, drilled them on your company's specific way of phrasing things until it became second nature. After that training, they don't need to look anything up for common situations, it's just baked into how they naturally respond now. That's fine-tuning. You're not handing the model a reference book, you're actually reshaping how it responds by showing it enough examples that the new behavior becomes part of it.

Both approaches can get you a good employee. But they solve different problems. If your issue is "the person doesn't know our specific product details," a manual fixes that instantly and cheaply. If your issue is "the person's tone and style just isn't matching what we need no matter how much reference material we give them," training is what actually fixes that.

What fine-tuning actually is, technically

An LLM, underneath everything, is a giant pile of numbers called parameters, sometimes billions of them, that got adjusted during training to make the model good at predicting text. Fine-tuning means taking a model that's already been trained once, and training it a bit further on a smaller, specific set of examples so its parameters shift slightly toward behaving the way you want in that specific area.

You're not starting from scratch. You're not building a new model from nothing, that's what the original training already did and it costs an amount of money most of us will never personally spend. Fine-tuning is more like giving an already capable model some focused extra practice on exactly the kind of task or tone you care about.

When fine-tuning is genuinely the right call

I used to think fine-tuning was just "the more advanced version of RAG," like a level up. That's wrong and it cost me some wasted time. They solve different kinds of problems.

Fine-tuning makes sense when you need consistent style or tone across every single response, like a brand voice that has to be exactly right every time. It also helps with teaching the model a specific format or structure it doesn't naturally produce, or teaching it a narrow skill, like classifying support tickets into your company's exact categories, or writing code in a very specific internal style your team uses. It's also useful when you need the model to reliably follow a particular behavior pattern that's hard to fully explain in a prompt every single time.

Fine-tuning does not make sense as the fix when your actual problem is that the model doesn't know specific facts or up to date information. That's what RAG solves, and it solves it far more cheaply and it updates instantly when your source data changes. A lot of people reach for fine-tuning to "teach the model our documentation" and end up frustrated, expensive, and still not getting current information reliably, when what they actually needed was retrieval the whole time.

A decent rule of thumb I settled on, if the problem is about knowledge, reach for RAG first. If the problem is about behavior, tone, or format, fine-tuning is worth considering.

The data problem nobody warns you about

Here's the part that actually matters more than the code, and I learned this the hard way. Fine-tuning is entirely dependent on the quality of your example data. Feed it fifty inconsistent, messy examples and you'll get a model that's inconsistently, messily different from before, not better.

The examples need to actually represent the behavior you want, consistently, and there needs to be enough of them to establish a real pattern rather than a fluke. I've seen people try to fine-tune on twenty examples and wonder why nothing changed. The model needs enough repetition of the pattern to actually shift its behavior, not just see the idea once.

Let's actually do this with code

I'll walk through two versions. First, using a hosted fine-tuning API, which is the easier route and what most people should start with. Then a quick look at what's happening if you were to do it yourself locally, so you understand what the hosted version is doing for you behind the scenes.

Version one, using a hosted fine-tuning API

Most major providers offer fine-tuning as an API service where you upload your examples and they handle the actual training infrastructure for you. The general shape looks like this, using OpenAI's fine-tuning setup as the example since it's widely used and well documented.

First, you prepare your training data in a specific format, usually a JSONL file, which just means one JSON object per line, each one representing a single example conversation.

import json

training_examples = [
    {
        "messages": [
            {"role": "system", "content": "You are a support assistant for a software company. Always respond in a friendly, concise tone and end with an offer to help further."},
            {"role": "user", "content": "My app keeps crashing on startup."},
            {"role": "assistant", "content": "Sorry about that! Try clearing the app cache first, that fixes it most of the time. Let me know if it's still crashing after that and we'll dig deeper together."}
        ]
    },
    {
        "messages": [
            {"role": "system", "content": "You are a support assistant for a software company. Always respond in a friendly, concise tone and end with an offer to help further."},
            {"role": "user", "content": "How do I export my data?"},
            {"role": "assistant", "content": "Easy, head to Settings then Export Data, pick CSV or JSON, and it'll download right away. Want a hand finding where to import it afterward?"}
        ]
    }
    # in a real project, you'd want at least fifty to a few hundred
    # examples like this, all consistently following the tone and
    # format you actually want
]

# write it out as JSONL, one example per line
with open("training_data.jsonl", "w") as f:
    for example in training_examples:
        f.write(json.dumps(example) + "\n")

Once you have that file, you upload it and kick off the fine-tuning job.

from openai import OpenAI

client = OpenAI(api_key="your-api-key-here")

# upload the training file
uploaded_file = client.files.create(
    file=open("training_data.jsonl", "rb"),
    purpose="fine-tune"
)

# start the fine-tuning job using that file
job = client.fine_tuning.jobs.create(
    training_file=uploaded_file.id,
    model="gpt-4o-mini-2024-07-18"
)

print(job.id)
# this runs in the background on their servers, could take
# anywhere from minutes to a couple hours depending on data size

You can check on it while it trains.

status = client.fine_tuning.jobs.retrieve(job.id)
print(status.status)
# will show something like "running" then eventually "succeeded"

Once it's done, you get back a custom model name, and from then on you just call it exactly like you'd call any other model.

response = client.chat.completions.create(
    model=status.fine_tuned_model,
    messages=[
        {"role": "user", "content": "The app won't let me log in."}
    ]
)

print(response.choices[0].message.content)

And that's genuinely the whole hosted workflow. Prepare consistent examples, upload them, start a job, wait, then use the resulting model name like normal.

Version two, a peek at what's happening underneath, using LoRA locally

If you're curious what the hosted service is actually doing for you, or you want to fine-tune an open source model yourself instead of a closed one, this is roughly what it looks like using a technique called LoRA, which stands for Low Rank Adaptation. Instead of updating all of a model's billions of parameters, which would need a serious amount of computing power, LoRA only trains a small set of additional parameters that get layered on top of the original model. It's dramatically cheaper and faster while still getting you most of the benefit.

# this example uses the Hugging Face ecosystem, a common
# open source setup for this kind of thing
# pip install transformers peft datasets torch

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import LoraConfig, get_peft_model
from datasets import Dataset

model_name = "meta-llama/Llama-3.2-1B"  # a small open model as an example

tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name)

# LoRA config, this defines how much extra "adjustable" capacity
# we're adding on top of the frozen original model
lora_config = LoraConfig(
    r=8,                 # rank, basically how much extra capacity to add
    lora_alpha=16,
    target_modules=["q_proj", "v_proj"],  # which parts of the model to adapt
    lora_dropout=0.05,
    task_type="CAUSAL_LM"
)

model = get_peft_model(model, lora_config)

# your training examples, formatted as plain text
training_texts = [
    "User: My app keeps crashing.\nAssistant: Sorry about that! Try clearing the app cache first, that fixes it most of the time. Let me know if it's still crashing after that and we'll dig deeper together.",
    "User: How do I export my data?\nAssistant: Easy, head to Settings then Export Data, pick CSV or JSON, and it'll download right away. Want a hand finding where to import it afterward?"
]

dataset = Dataset.from_dict({"text": training_texts})

def tokenize(example):
    return tokenizer(example["text"], truncation=True, padding="max_length", max_length=256)

tokenized_dataset = dataset.map(tokenize)

# from here you'd hand this off to a Trainer object and call .train()
# the important conceptual point is just this, the model's original
# weights stay frozen, only these small LoRA layers get updated,
# which is why this is so much cheaper than training everything

I won't pretend that second example is copy paste ready for a beginner, actually running local fine-tuning involves GPU setup, dependency management, and enough moving parts to fill its own article. The point of showing it is just so the hosted API from version one stops feeling like a black box. Underneath, it's the same basic idea, feed it consistent examples, adjust a manageable number of parameters, get a model that's shifted toward your examples.

Mistakes I made so you don't have to

I fine-tuned on way too few examples the first time and got confused why nothing meaningfully changed. Consistency and volume both matter, a handful of examples isn't enough to establish a real pattern.

I tried to fine-tune my way out of a knowledge problem, feeding the model our internal documentation as training examples hoping it would "learn our product." It kind of half worked and was way more expensive and slower to update than just building a small RAG setup would have been. Lesson learned the hard way, painfully.

I didn't keep a solid evaluation set aside. I trained, tried a few examples, thought it looked good, and moved on, only to find out later it had gotten noticeably worse at some other unrelated task I hadn't tested. Always hold out some real test cases you didn't train on, and check the model against those before assuming a fine-tune actually helped.

The honest summary

If you remember one thing from this, remember the employee example. A manual next to someone's desk fixes a knowledge gap instantly and cheaply, and that's RAG. Months of training reshapes how someone naturally behaves, and that's fine-tuning. Most problems people bring to fine-tuning are actually knowledge problems in disguise, and RAG would've solved them faster and cheaper. But for the real behavior and consistency problems, fine-tuning genuinely is the right tool, and now you've actually seen what's happening inside it instead of just hearing the word thrown around.

---

If you try fine-tuning your own small model on something, I'd genuinely like to hear what you built it for, drop it in the comments.