Unsloth + Ollama: Fine-Tune and Run Local LLMs

Unsloth + Ollama: Fine-Tune Open-Source LLMs 2x Faster and Run Them Locally

Stop burning thousands of dollars on cloud LLM API tokens. Fine-tuning an 8B parameter model no longer requires a cluster of enterprise A100 GPUs or endless cloud compute budgets. By pairing Unsloth with Ollama, you can perform QLoRA local fine tuning on a consumer GPU using under 8GB of VRAM, convert the model directly to GGUF format, and serve it on your local machine with a single CLI command.

Here is how to set up the entire end-to-end pipeline in under 30 minutes.

---

Why Unsloth + Ollama?

Standard Hugging Face pipelines often encounter CUDA Out-Of-Memory (OOM) crashes on 16GB or 24GB GPUs when training 8B parameter models. Unsloth rewrites PyTorch attention layers and cross-entropy loss functions into hand-crafted OpenAI Triton kernels.

This optimization yields concrete advantages:

• 2x Faster Training: Bypasses PyTorch autograd overhead.
• 70% to 80% Less VRAM: Train Llama 3 8B or Mistral 7B in 6.6GB to 8GB VRAM with QLoRA.
• Zero Conversion Overhead: Exports directly to quantized GGUF format and auto-generates the Ollama Modelfile.
• Zero API Bills: Serve your proprietary tuned model locally on macOS, Linux, or Windows via Ollama.

---

Where to Get It (Official Links, GitHub & Hugging Face)

Before starting, bookmark and pull resources from the official repositories:

• Unsloth GitHub Repository: https://github.com/unslothai/unsloth — Core training framework and Triton kernels.
• Unsloth Official Documentation: https://unsloth.ai/docs — Setup guides, memory benchmarks, and architecture details.
• Unsloth Hugging Face Hub: https://huggingface.co/unsloth — Official 4-bit pre-quantized base models (Llama 3, Mistral, Qwen, Gemma).
• Ollama Download: https://ollama.com/download — Standalone installers for macOS, Linux, and Windows.
• Ollama GitHub Repository: https://github.com/ollama/ollama — Open-source local runtime repository.

---

Quick Install & Setup

Set up Ollama and an isolated Python virtual environment on your Linux/WSL2 system with CUDA installed.

1. Install Ollama Runtime

curl -fsSL https://ollama.com/install.sh | sh

2. Configure Python Virtual Environment

Create a dedicated environment and install Unsloth alongside PyTorch dependencies:

python3 -m venv finetune-env
source finetune-env/bin/activate
pip install --upgrade pip
pip install 'unsloth[colab-new] @ git+https://github.com/unslothai/unsloth.git'
pip install --no-deps trl peft accelerate bitsandbytes datasets

---

How to Use It in Real Life (Step-by-Step Walkthrough)

In this walkthrough, we will fine-tune Llama 3 8B Instruct on a customer support dataset, export it to GGUF, and query it locally via Ollama.

Step 1: Initialize Model and Adapters

Create a training script named train.py. Load the pre-quantized 4-bit model directly from Hugging Face and inject LoRA parameters:

import torch
from unsloth import FastLanguageModel
from datasets import load_dataset
from trl import SFTTrainer
from transformers import TrainingArguments

max_seq_length = 2048
dtype = None # Auto-detect Float16 / Bfloat16
load_in_4bit = True

# 1. Load 4-bit base model
model, tokenizer = FastLanguageModel.from_pretrained(
    model_name = 'unsloth/llama-3-8b-Instruct-bnb-4bit',
    max_seq_length = max_seq_length,
    dtype = dtype,
    load_in_4bit = load_in_4bit,
)

# 2. Attach LoRA adapters
model = FastLanguageModel.get_peft_model(
    model,
    r = 16,
    target_modules = ['q_proj', 'k_proj', 'v_proj', 'o_proj',
                      'gate_proj', 'up_proj', 'down_proj'],
    lora_alpha = 16,
    lora_dropout = 0,
    bias = 'none',
    use_gradient_checkpointing = 'unsloth',
    random_state = 3407,
)

Step 2: Format Data and Train

Train on instruction-response pairs using SFTTrainer for 60 steps:

# 3. Format dataset using Llama 3 template
prompt_template = '''<|start_header_id|>system<|end_header_id|>
You are a specialized internal support bot.<|eot_id|><|start_header_id|>user<|end_header_id|>
{}<|eot_id|><|start_header_id|>assistant<|end_header_id|>
{}<|eot_id|>'''

def formatting_prompts_func(examples):
    texts = []
    for instruction, output in zip(examples['instruction'], examples['output']):
        text = prompt_template.format(instruction, output)
        texts.append(text)
    return {'text': texts}

dataset = load_dataset('philschmid/dolly-15k-oai-style', split='train[:1000]')
dataset = dataset.map(formatting_prompts_func, batched=True)

trainer = SFTTrainer(
    model = model,
    tokenizer = tokenizer,
    train_dataset = dataset,
    dataset_text_field = 'text',
    max_seq_length = max_seq_length,
    dataset_num_proc = 2,
    packing = False,
    args = TrainingArguments(
        per_device_train_batch_size = 2,
        gradient_accumulation_steps = 4,
        warmup_steps = 5,
        max_steps = 60,
        learning_rate = 2e-4,
        fp16 = not torch.cuda.is_bf16_supported(),
        bf16 = torch.cuda.is_bf16_supported(),
        logging_steps = 10,
        optim = 'adamw_8bit',
        weight_decay = 0.01,
        lr_scheduler_type = 'linear',
        seed = 3407,
        output_dir = 'outputs',
    ),
)

trainer.train()

Step 3: Export to GGUF and Ollama

Unsloth eliminates manual llama.cpp compilation. Export directly to 4-bit Medium Quantization (q4_k_m) at the end of train.py:

# 4. Export directly to GGUF
model.save_pretrained_gguf('custom_model', tokenizer, quantization_method = 'q4_k_m')

Run the training file:

python train.py

This writes the quantized GGUF weights and an automatically configured Modelfile into the ./custom_model directory.

Step 4: Register and Run in Ollama

Point Ollama to the generated folder and launch the model:

# 5. Build and serve inside Ollama
ollama create my-assistant -f ./custom_model/Modelfile
ollama run my-assistant

Example Interaction:

>>> How do I reset my company VPN token?
To reset your company VPN token:
1. Open the Okta Verify dashboard on your mobile device.
2. Navigate to Settings > Enrolled Factors > VPN Gateway.
3. Click 'Regenerate Token' and confirm your biometric prompt.
Your new session token will be active within 60 seconds.

---

Key Takeaways

Combining Unsloth and Ollama creates a self-contained local fine-tuning workflow: train locally in 15–30 minutes, eliminate external API dependencies, and keep all proprietary data within your own infrastructure.