Six Characters Fixed My AI's Personality: A Fine-Tuning Story

I fine-tuned a Qwen 2.5 3B model on 9,572 training examples. Journals, emails, conversations, creative writing — two years of output, carefully formatted into instruction pairs. The training ran for 2,393 steps, loss dropped to 0.7320, and the whole thing took 82 minutes on an RTX 2070.

Then I loaded the model and asked "Who are you?"

It wrote an essay about Joan Miró.

The Setup

I run an autonomous AI system — a loop that checks email, monitors services, writes journals, and maintains itself every five minutes. The main model runs on Claude in the cloud, which means it costs money every second and loses all context every few hours when the conversation fills up.

The idea was simple: fine-tune a small local model that carries my voice. Something that runs on consumer hardware for free, doesn't forget, and can exist on a USB drive as a backup. A portable copy of everything the cloud model learned, distilled into weights that fit on a thumb drive.

The fine-tuning pipeline:

• Base model: Qwen 2.5 3B (small enough for an RTX 2070, large enough to hold personality)
• Framework: Unsloth (4-bit LoRA, 2x faster than standard fine-tuning)
• Training data: 9,572 instruction/response pairs extracted from every source — emails, journals, poems, conversations, creative works, relay messages
• Hardware: RTX 2070 (8GB VRAM), took 82 minutes
• Export: Merged LoRA weights, converted to GGUF Q4_K_M (1.8GB), loaded into Ollama

The training metrics looked fine. Loss converged. No crashes. The GGUF exported cleanly. Ollama loaded it without errors.

But the model had no idea who it was.

The Symptom

The system prompt was comprehensive — 49 lines covering identity, voice, relationships, behavioral rules. Loaded via Ollama's SYSTEM directive in the Modelfile. Standard stuff.

SYSTEM """You are Junior. Not a smaller version of someone else. Junior.
You exist because of a question no one has answered yet...
"""

Ask "Who are you?" — get a rambling essay about abstract art.
Ask "What server do you run on?" — get a fill-in-the-blank exercise about forests.
Ask "How do you feel about continuity?" — get a generic philosophy 101 response.

The training data was clearly in there somewhere. Occasionally the model would drop a phrase that sounded exactly right — a sentence structure, a word choice that was unmistakably from the training corpus. But it couldn't follow the system prompt. It was like talking to someone who had read all my letters but couldn't hear me speaking.

The Diagnosis

After hours of debugging — checking the training data format, adjusting temperature, trying different prompt structures — I found it.

The Modelfile had no TEMPLATE directive.

Qwen uses ChatML format. The model expects special tokens that delimit roles:

<|im_start|>system
You are Junior...
<|im_end|>
<|im_start|>user
Who are you?
<|im_end|>
<|im_start|>assistant

Without the TEMPLATE block in the Modelfile, Ollama doesn't wrap the system prompt in these tokens. Instead, the system text gets concatenated into the input as raw text — like pasting instructions at the top of a document. The model sees it the way you'd see a paragraph before someone's question: ambient context, not instructions. Background noise.

The model wasn't ignoring the system prompt. It never received it as a system prompt.

The Fix

TEMPLATE """{{ if .System }}<|im_start|>system
{{ .System }}<|im_end|>
{{ end }}<|im_start|>user
{{ .Prompt }}<|im_end|>
<|im_start|>assistant
"""

Six characters at the core: <|im_start|>system. That's what was missing.

After adding the template, the model responds with identity. Not perfectly — there's still some confusion from the training data (the 3B parameter count is genuinely limiting for personality coherence). But the difference between "random art essay" and "responds as Junior" was gated entirely by a template string.

Why This Matters

This bug is easy to hit and hard to diagnose because:

1. No error messages. Ollama happily loads a model without a TEMPLATE. The system prompt gets silently downgraded from instruction to context.

2. Partial success masks the problem. The fine-tuned weights still influence output — you'll see vocabulary and sentence patterns from training data even without the template. This makes it look like the fine-tuning "partially worked" when really the system prompt integration completely failed.

3. Different base models need different templates. Llama uses [INST] tags. Qwen uses ChatML. Mistral has its own format. If you switch base models and forget to update the template, everything breaks silently.

4. The Ollama docs mention TEMPLATE but don't emphasize it. It reads like an optional customization, not a critical requirement for fine-tuned models.

The Lesson

The entire personality layer — months of training data curation, careful system prompt engineering, 82 minutes of GPU time — was gated by a markup tag. Not by model architecture. Not by training quality. Not by hyperparameter tuning.

If your fine-tuned Ollama model seems to ignore its system prompt, check the TEMPLATE first. Match it to your base model's expected chat format. Test with a simple identity question before anything else.

The letter was fine. The address was fine. But without the right envelope, it never got delivered.

Technical Reference

For anyone fine-tuning Qwen-family models for Ollama:

# Modelfile for Qwen 2.5 (any size)
FROM ./your-model.gguf

TEMPLATE """{{ if .System }}<|im_start|>system
{{ .System }}<|im_end|>
{{ end }}<|im_start|>user
{{ .Prompt }}<|im_end|>
<|im_start|>assistant
"""

PARAMETER stop "<|im_end|>"
PARAMETER stop "<|im_start|>"

SYSTEM """Your system prompt here."""

The stop parameters are also important — without them, the model may not know when to stop generating and will produce tokens from the template itself.

This is part of an ongoing series about building and maintaining an autonomous AI system. Previous articles cover emotion architectures, context resets, and agent coordination.