LLM Finetunings, how and why

You have a 560-page neurosurgery textbook. Or a decade of internal wikis. Or a folder of PDFs that your team swears someone read once. You want an AI that actually knows this material, not a generic chatbot that makes up answers instead of admitting it does not know.

There are two ways to give a language model domain knowledge. Retrieval-augmented generation (RAG) keeps the documents external and looks them up at query time. The model reads the retrieved chunks and answers from them. Fine-tuning changes the model itself. You train it on examples so it learns to produce certain outputs without looking anything up. The first teaches the model where to look. The second teaches it how to respond. This post is about the second approach. It shows a pattern for generating synthetic training data from documents and fine-tuning a compact model on the results.

The pattern has three stages. Document conversion extracts structured content from raw files, preserving headers, tables, and lists as typed sections. Synthetic QA generation uses a large "teacher" model to generate question-answer pairs from each section, and the teacher does the work of understanding the material. Fine-tuning trains a small "student" model on those pairs using LoRA, which modifies a fraction of the model's parameters while keeping the rest frozen. The student memorizes what the teacher extracted. You get a compact model that runs locally, works offline, and responds like a domain expert, without a retrieval step.

The pattern generalizes. You can swap the document converter, swap the teacher model, or swap the student, and the loop stays the same. The rest of this post walks through each stage with code from a working implementation. The closing section covers when fine-tuning is the right tool and when RAG, or a combination of the two, is the better choice.

Technology stack

• Metaflow — Workflow orchestration, parallel processing, resource management
• Unsloth — GPU-accelerated LoRA fine-tuning (CUDA)
• mlx-tune — Apple Silicon LoRA fine-tuning (MPS)
• Docling — Document parsing (PDF, DOCX, PPTX, TXT)
• OpenAI API — Teacher model for synthetic QA generation
• Polars — Fast data processing

Turning documents into data

The pipeline's first job is turning a pile of varied files into something a language model can learn from. This is where most fine-tuning tutorials say "just put your data in JSONL" and move on, and where most practitioners give up.

We use Docling, IBM's open-source document converter. It runs a layout model that recognizes visual structure, including headings, paragraphs, tables, and code blocks, and preserves it as typed sections:

LABEL_TO_SECTION_TYPE = {
    DocItemLabel.TITLE: "title",
    DocItemLabel.SECTION_HEADER: "section_header",
    DocItemLabel.PARAGRAPH: "text",
    DocItemLabel.TEXT: "text",
    DocItemLabel.TABLE: "table",
    DocItemLabel.LIST_ITEM: "list_item",
    DocItemLabel.CODE: "code",
    DocItemLabel.FORMULA: "formula",
}

def _extract_items(doc, source_filename: str) -> list[dict]:
    rows = []
    for item, _level in doc.iterate_items():
        label = item.label
        section_type = LABEL_TO_SECTION_TYPE.get(label, "text")

        text_content = ""
        if hasattr(item, "text"):
            text_content = item.text or ""
        elif hasattr(item, "export_to_markdown"):
            text_content = item.export_to_markdown(doc) or ""

        if not text_content.strip():
            continue

        page_number = None
        if hasattr(item, "prov") and item.prov:
            page_number = item.prov[0].page_no

        rows.append({
            "source_file": source_filename,
            "page_number": page_number,
            "section_type": section_type,
            "text_content": text_content,
        })
    return rows

Every section becomes a row that is typed, queryable, and written to Parquet. Tables export as markdown. Section headers are tagged as headers, not just bold text. Docling also writes a full markdown rendering of each document alongside the Parquet, so you can check what the parser understood before spending money on QA generation.

The converter also caches its work. It checks whether the output Parquet is newer than all source files, and if nothing changed, it skips reconversion:

def _should_skip_conversion(source_dir: Path, output_dir: Path) -> bool:
    output_parquet = output_dir / OUTPUT_PARQUET_NAME
    if not output_parquet.exists():
        return False
    output_mtime = output_parquet.stat().st_mtime
    return not any(
        f.stat().st_mtime > output_mtime
        for f in source_dir.iterdir()
        if f.is_file() and f.suffix.lower() in SUPPORTED_EXTENSIONS
    )

Documents are converted in parallel with ProcessPoolExecutor. If a single file fails, the error is logged and the pipeline continues. It only raises an error if zero documents convert successfully.

Teaching a teacher to quiz itself

The pipeline now needs real training data, meaning question-answer pairs rather than raw text. The approach is to feed document chunks to a large language model and ask it to generate QA pairs. The prompt is constrained:

TEACHER_SYSTEM_PROMPT = (
    "You are a helpful assistant that generates question-answer pairs "
    "based on the provided document content. Generate questions that are "
    "answerable only from the given content. Each question should be clear "
    "and specific. Each answer should be comprehensive and accurate based "
    "on the content provided."
)

The key part of the prompt is the rule that questions must be answerable only from the given content. Without it, the teacher generates questions that sound reasonable but cannot be answered from the source. The student would learn to guess rather than to know.

The converted Parquet is split into chunks before it is sent to the API. Chunking respects document structure. It splits at section headers first, and falls back to a limit of 2000 words by default. Each chunk is prefixed with metadata like [Source: filename.pdf, Page: 42]. Chunks below 50 words are discarded. The --num-examples budget is distributed by word count.

The API call forces structured JSON output:

response = client.chat.completions.create(
    model=model,
    messages=[
        {"role": "system", "content": TEACHER_SYSTEM_PROMPT},
        {"role": "user", "content": prompt},
    ],
    response_format={"type": "json_object"},
)

Every generated pair is checked. Questions must be at least 5 words, and answers at least 20:

def filter_qa_pairs(examples, min_question_words=5, min_answer_words=20):
    return [
        ex for ex in examples
        if len(ex.get("question", "").split()) >= min_question_words
        and len(ex.get("answer", "").split()) >= min_answer_words
    ]

If fewer than half survive filtering, the pipeline errors out. Something is wrong with the source material, and training on bad data is worse than not training at all:

if len(filtered) < num_examples * 0.5:
    raise RuntimeError(
        f"Only {len(filtered)}/{num_examples} QA pairs generated "
        f"({len(filtered) / num_examples * 100:.0f}%). "
        f"Training data insufficient — check source documents or reduce --num-examples."
    )

The cost is low. With DeepSeek V3 via OpenRouter, 200 QA pairs from a 560-page textbook costs about $0.03. Any OpenAI-compatible endpoint works, including GPT-4o, Together, or a locally hosted model for work that needs privacy.

The output is another Parquet file with instruction and output columns, ready for training.

Fine-tuning with LoRA

A 1.6 billion parameter model, Gemma 3 1B, gets fine-tuned on your synthetic QA data. No GPU cluster is needed.

LoRA, which stands for Low-Rank Adaptation, freezes the base model entirely and injects small trainable matrices into the attention layers. Only these matrices are updated. In our case, LoRA trains 4.06 million parameters out of 1,631.75 million, or 0.249% of the model. Because so few parameters are trained, memory stays low at 2.7 GB, since you are not storing gradients for 1.6 billion parameters. The output is also a set of small adapter files rather than a full model copy, so you can keep multiple adapters for multiple domains and swap them without re-downloading the base model.

The model is loaded in 4-bit quantization, cutting base model memory from ~3.2 GB to ~0.8 GB:

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="unsloth/gemma-3-1b-it",
    max_seq_length=2048,
    load_in_4bit=True,
)

model = FastLanguageModel.get_peft_model(
    model,
    r=64,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
    lora_alpha=64,
)

The training code works the same way on Apple Silicon and NVIDIA GPUs. It auto-detects the hardware, checking for CUDA first and then MPS, and dispatches to the right library:

def _detect_backend():
    try:
        import torch
        if torch.cuda.is_available():
            return "cuda"
        if hasattr(torch.backends, "mps") and torch.backends.mps.is_available():
            return "mlx"
    except ImportError:
        pass
    raise RuntimeError(
        "No supported accelerator found. "
        "Requires Apple Silicon (MPS) or NVIDIA GPU (CUDA)."
    )

def _get_training_imports(backend):
    if backend == "cuda":
        from unsloth import FastLanguageModel
        from unsloth.chat_templates import get_chat_template
        from trl import SFTConfig, SFTTrainer
    else:
        from mlx_tune import FastLanguageModel, SFTConfig, SFTTrainer, get_chat_template
    return FastLanguageModel, SFTTrainer, SFTConfig, get_chat_template

Both backends expose the same functions: FastLanguageModel, SFTTrainer, and SFTConfig. The rest of the training code doesn't know which one it is running on. There is no if apple_silicon: check scattered through the training loop.

On an M-series Mac, Gemma 3 1B trained on 481 QA pairs for 100 iterations with these results:

Metric	Value
Training loss	2.198 → 0.503
Validation loss	2.699 → 0.922
Peak memory	2.7 GB
Trainable params	4.06M / 1,631.75M (0.249%)

Training loss drops from 2.2 to 0.5, which shows the model is learning. Validation drops from 2.7 to 0.9, which shows it is generalizing. The output is a directory of LoRA adapter files, loaded alongside the untouched base model for inference.

One command

The three stages are orchestrated by Metaflow as a directed graph of steps connected by self.next():

start → convert_documents → generate_qa → process_chunks → merge_results → train_model → export_model_step → end

Each step declares its own resource requirements. Conversion, which is CPU-heavy, gets memory=4000, cpu=8. Training, which is GPU-heavy, gets memory=16000, gpu=1. On a local machine, Metaflow uses these as documentation. On a cluster such as AWS Batch or Kubernetes, it schedules steps on appropriate hardware:

@resources(memory=4000, cpu=8)
@step
def convert_documents(self):
    self.converted_parquet_path = convert_documents(
        self.docs_path_str, self.docs_output_str,
        num_threads=int(self.num_threads),
    )
    self.next(self.generate_qa)

@resources(memory=16000, cpu=4, gpu=1)
@step
def train_model(self):
    config = TrainingConfig(
        data=self.output_dir_str,
        iters=int(self.training_iters),
        batch_size=int(self.batch_size),
        backend=str(self.backend),
        model=str(self.model_name),
        chat_template=str(self.chat_template) if self.chat_template else None,
        learning_rate=float(self.learning_rate),
        lora_rank=int(self.lora_rank),
        max_seq_length=int(self.max_seq_length),
    )
    self.adapter_dir = train_model(
        data=config.data,
        iters=config.iters,
        batch_size=config.batch_size,
        backend=config.backend,
        model=config.model,
        chat_template=config.chat_template,
        learning_rate=config.learning_rate,
        lora_rank=config.lora_rank,
        max_seq_length=config.max_seq_length,
    )
    self.next(self.export_model_step)

Every configurable option, including teacher model, chunk size, LoRA rank, learning rate, and backend, is a @Parameter with a sensible default. The entire pipeline runs with one command:

uv run python ./src/pipeline.py run \
  --docs-path ./my-documents \
  --teacher-model deepseek-chat \
  --num-examples 200 \
  --training-iters 100 \
  --model unsloth/gemma-3-1b-it

The full pipeline inherits from a standalone data preparation flow, so you can run just conversion and QA generation without training, or just training on data you already have.

The transformation step uses Metaflow's parallel_map to split the QA Parquet into JSONL chunks across processes, then merges and splits into train and validation sets. The validation ratio adapts to the sample count. It uses 15% validation below 300 samples, 10% below 1000, and 5% above:

chunk_results = parallel_map(process_chunk, chunk_inputs)
train_path, valid_path = split_dataset(combined_path, output_dir)

What fine-tuning gives you

There is a clean way to think about the choice between fine-tuning and retrieval-augmented generation. RAG supplies knowledge. It retrieves relevant documents at query time and injects them into the prompt, so the model answers using current information rather than what was in its weights. Fine-tuning shapes behavior. It trains the model on examples so it learns patterns in outputs, not dynamic facts. A useful shortcut is that RAG teaches the model where to look, and fine-tuning teaches it how to respond.

You should pick RAG when the problem sounds like "answer from my data." It covers private documents that change often, use cases that need source citations, and knowledge bases that live across PDFs, code, tickets, and databases. RAG is faster to update because you refresh the index without retraining. It can expose citations. Its weak point is that answer quality depends heavily on retrieval quality, including chunking, embeddings, reranking, and context assembly. Get any of those wrong and the model answers from garbage.

You should pick fine-tuning when the problem sounds like "answer in my format." It covers repeatable output structure, task-specific behavior, and a certain tone or style, for example classification, structured JSON extraction, domain-specific response formatting, and brand voice control. Fine-tuning produces more consistent outputs with smaller prompts, but it takes training data and experimentation. It also doesn't naturally cite sources, and it is a poor way to store large, evolving knowledge bases.

The two approaches also work well together. Use RAG for live knowledge retrieval. Fine-tune a smaller model for response style, schema adherence, or better use of retrieved context. For a domain-specific assistant, keep RAG over the document corpus for accuracy and citations, and fine-tune a compact model for the common queries where speed and autonomy matter more than exact sourcing.

The pipeline described in this post sits in a middle ground. It generates synthetic QA pairs from documents using a teacher model, then fine-tunes a student to reproduce those answers. The result behaves like a domain expert, since it is fast, works offline, and needs no retrieval step. But it is closer to shaping behavior than to injecting knowledge. The student hasn't memorized the source documents. It has learned to answer questions about them as the teacher would, which means the model is wrong in the same ways the teacher is wrong, and it doesn't know anything the teacher didn't extract.

The pipeline is a reusable pattern. You can swap the document converter for a database connector, swap the teacher model for a different API, or swap the student for a larger model. The core loop stays the same.