Fine-tuning a model is easy to start. Running it responsibly in a team is the hard part. Usually it's a script on a GPU box: nobody approves the spend, nobody records which data went in, and "is it better than the base model?" is a guess.
We ran a real fine-tune end to end with cairn, the open-source control plane we're building for exactly this, using only public packages from PyPI. Here is what happened, with the real numbers.
The run in numbers
| What | Value |
|---|---|
| Model | Qwen3-4B-Base, 4-bit QLoRA (r 16) |
| Data | 5,646 training rows of IFC/BIM instructions (from 42,680) |
| Hardware | one NVIDIA T4 (AWS g4dn.xlarge) |
| Training | 353 steps, 73 minutes |
| Cost | $0.86 settled, against a $16.50 worst-case estimate approved up front |
| Test loss (348 held-out rows) | 1.955 → 0.218 (89% lower) |
| General ability | no measurable change on ARC, HellaSwag, WinoGrande, PIQA or MMLU |
The adapter, the dataset, the training curve and the raw eval results are public: cairndev/qwen3-4b-ifc-bim-qlora and cairndev/ifc-bim-alpaca-dedup.
Two machines, one rule
A small control box runs the cairn engine. It holds the run record, the approval gates and the budget, and it starts and stops the GPU box. The GPU box runs only the trainer. Nothing listens on it: the engine reaches it over SSH. Datasets, checkpoints and the adapter move between the two through S3.
1. A person approves the data, and the data gets checked
dataset-prep/prepare paused before reading anything, and showed the source, its pinned Hub commit and the split plan. Only after approval did it download, check and split the data.
The checks found something worth knowing. 85% of the source rows were near-duplicates of each other: the dataset is template-generated, so the same IFC statement appears with only the numbers changed. Near-duplicate removal (MinHash, 0.8) kept 6,295 rows. The split is by template family, so variants of one template never sit on both sides of train and test, and the split's leakage check passed.
That number changed how we read every result after it. Without the check, we'd have trained on 42,680 rows and believed it.
2. The launch refuses before it spends
At launch, before any record existed, cairn checked:
- Data roles: each dataset version's role, so a test set can't be passed as training data.
- Holdout: the training rows don't overlap the validation or test rows, exactly or near.
- The GPU estimate: priced from the box's hourly rate × each leg's timeout × its retries, $16.50 worst case, which fit the $25 budget.
Then it stopped at approve-run, with that estimate and the training settings on the card. The GPU box was only started after the approval.
3. The box starts, trains, and stops itself
After approval:
-
compute/acquiretagged the instance with a lease, then started it. - A 5-step smoke run proved the whole path on the box: data from S3, the tokenizer, 4-bit loading, a checkpoint back to S3.
- The real training ran 353 steps. Every 50 steps a checkpoint was mirrored to S3, so a killed process resumes instead of restarting.
-
compute/releasestopped the box right after training, before anyone looked at the results, so it didn't bill while we decided.
Each step settled its actual box time: smoke $0.05, train $0.81. The estimate is the ceiling you approve; the record keeps what you paid.
4. Better than the base model, on data it never saw?
The pipeline's improved gate compares the candidate with the base model on held-out data, and it passed. We then measured both models on the 348-row test split, which never reached the trainer: loss fell from 1.955 to 0.218.
A lower loss on your own domain doesn't tell you whether the model got worse at everything else. So we ran a forgetting check with lm-evaluation-harness, with identical settings for both models:
| Task | Base | Fine-tuned |
|---|---|---|
| ARC-Easy | 0.767 | 0.760 |
| ARC-Challenge | 0.527 | 0.553 |
| HellaSwag | 0.647 | 0.653 |
| WinoGrande | 0.740 | 0.707 |
| PIQA | 0.800 | 0.787 |
| MMLU | 0.768 | 0.797 |
No change is larger than the noise at these sample sizes (about ±3–4 points). So: no measurable loss of general ability, which is not the same as none. The samples were small, and we say so on the model card.
5. A person promotes
The run then paused at promote-model, with the losses on the card. Promotion is a decision, not a side effect. Moving the tag needed its own approval too.
What it took
Every step was a cairn CLI command against the engine: install, connect the GPU box, prepare the data, launch, approve, read the findings, promote. The full list, in order, is in the new guide: Fine-tune a model on a GPU box. Swap in your own dataset (Alpaca, prompt/completion or chat) and base model; the pipeline is the same.
cairn is Apache-2.0 and self-hosted. If you fine-tune models in a team, try the guide on your own data, and tell us where it breaks: github.com/cairndev-sh/cairn.
Originally published on the cairn blog.
Top comments (2)
The "person approves every step" model is honest about blast radius, but the real bottleneck is usually approval latency, not cost. Two things make governance sustainable:
1) An automated budget gate that rejects any run over X tokens before it starts — the human reviews the plan, not the bill.
2) Cheap open-weight models: a full fine-tune here runs $0.86 instead of $200, so you can afford to experiment with governance on.
The $0.86 is the tell — at that price the human-approval overhead can exceed the compute, so the gate has to be cheap and automatic while human sign-off stays reserved for the high-blast-radius calls. Do you gate on total token budget, or just the steps that need a human?
Human approval is most useful when the approver sees the proposed change, its evidence, and its blast radius in one place. Recording the approval decision alongside the dataset and configuration versions would make the governance claim auditable later.