A governed fine-tune for $0.86: Qwen3-4B on IFC/BIM, with a person approving every step

Vibhu — Mon, 26 Oct 2026 22:39:40 +0000

Fine-tuning a model is easy to start. Running it responsibly in a team is the hard part. Usually it's a script on a GPU box: nobody approves the spend, nobody records which data went in, and "is it better than the base model?" is a guess.

We ran a real fine-tune end to end with cairn, the open-source control plane we're building for exactly this, using only public packages from PyPI. Here is what happened, with the real numbers.

The run in numbers

What	Value
Model	Qwen3-4B-Base, 4-bit QLoRA (r 16)
Data	5,646 training rows of IFC/BIM instructions (from 42,680)
Hardware	one NVIDIA T4 (AWS g4dn.xlarge)
Training	353 steps, 73 minutes
Cost	$0.86 settled, against a $16.50 worst-case estimate approved up front
Test loss (348 held-out rows)	1.955 → 0.218 (89% lower)
General ability	no measurable change on ARC, HellaSwag, WinoGrande, PIQA or MMLU

The adapter, the dataset, the training curve and the raw eval results are public: cairndev/qwen3-4b-ifc-bim-qlora and cairndev/ifc-bim-alpaca-dedup.

Two machines, one rule

A small control box runs the cairn engine. It holds the run record, the approval gates and the budget, and it starts and stops the GPU box. The GPU box runs only the trainer. Nothing listens on it: the engine reaches it over SSH. Datasets, checkpoints and the adapter move between the two through S3.

1. A person approves the data, and the data gets checked

dataset-prep/prepare paused before reading anything, and showed the source, its pinned Hub commit and the split plan. Only after approval did it download, check and split the data.

The checks found something worth knowing. 85% of the source rows were near-duplicates of each other: the dataset is template-generated, so the same IFC statement appears with only the numbers changed. Near-duplicate removal (MinHash, 0.8) kept 6,295 rows. The split is by template family, so variants of one template never sit on both sides of train and test, and the split's leakage check passed.

That number changed how we read every result after it. Without the check, we'd have trained on 42,680 rows and believed it.

2. The launch refuses before it spends

At launch, before any record existed, cairn checked:

Data roles: each dataset version's role, so a test set can't be passed as training data.
Holdout: the training rows don't overlap the validation or test rows, exactly or near.
The GPU estimate: priced from the box's hourly rate × each leg's timeout × its retries, $16.50 worst case, which fit the $25 budget.

Then it stopped at approve-run, with that estimate and the training settings on the card. The GPU box was only started after the approval.

3. The box starts, trains, and stops itself

After approval:

compute/acquire tagged the instance with a lease, then started it.
A 5-step smoke run proved the whole path on the box: data from S3, the tokenizer, 4-bit loading, a checkpoint back to S3.
The real training ran 353 steps. Every 50 steps a checkpoint was mirrored to S3, so a killed process resumes instead of restarting.
compute/release stopped the box right after training, before anyone looked at the results, so it didn't bill while we decided.

Each step settled its actual box time: smoke $0.05, train $0.81. The estimate is the ceiling you approve; the record keeps what you paid.

4. Better than the base model, on data it never saw?

The pipeline's improved gate compares the candidate with the base model on held-out data, and it passed. We then measured both models on the 348-row test split, which never reached the trainer: loss fell from 1.955 to 0.218.

A lower loss on your own domain doesn't tell you whether the model got worse at everything else. So we ran a forgetting check with lm-evaluation-harness, with identical settings for both models:

Task	Base	Fine-tuned
ARC-Easy	0.767	0.760
ARC-Challenge	0.527	0.553
HellaSwag	0.647	0.653
WinoGrande	0.740	0.707
PIQA	0.800	0.787
MMLU	0.768	0.797

No change is larger than the noise at these sample sizes (about ±3–4 points). So: no measurable loss of general ability, which is not the same as none. The samples were small, and we say so on the model card.

5. A person promotes

The run then paused at promote-model, with the losses on the card. Promotion is a decision, not a side effect. Moving the tag needed its own approval too.

What it took

Every step was a cairn CLI command against the engine: install, connect the GPU box, prepare the data, launch, approve, read the findings, promote. The full list, in order, is in the new guide: Fine-tune a model on a GPU box. Swap in your own dataset (Alpaca, prompt/completion or chat) and base model; the pipeline is the same.

cairn is Apache-2.0 and self-hosted. If you fine-tune models in a team, try the guide on your own data, and tell us where it breaks: github.com/cairndev-sh/cairn.

Originally published on the cairn blog.

DEV Community: Cairn