Most fine-tuning guides start at "install these 14 packages". I wanted to start one step earlier: will it even fit?
So I built two free things that answer that before you download 19 GB of weights.
1. A calculator in the browser: huggingface.co/spaces/byjameson/will-it-fit — type a model id, pick your VRAM and sequence length. It reads config.json and does the arithmetic.
2. A CLI:
pip install streamft
streamft fit --model Qwen/Qwen2.5-14B-Instruct --gpu 6 --seq 512
13.99B params, 48 layers
NF4 blocks ~7.5 GB + bf16 head ~1.6 GB + reserve 0.9 GB (embedding stays on CPU)
-> whole model needs ~9.9 GB on 6 GB
verdict: needs streaming
measured on RTX 3050 6 GB: 6 block(s), 3.86 GB peak, 6.726 s/step at seq 512
The rule I set for the tool
Memory is arithmetic. Speed is only printed where it was measured.
The memory number comes from config.json: NF4 weight bytes, the bf16 LM head, the embedding pinned to CPU, and a 0.9 GB reserve. That is deterministic.
The speed number exists only for configurations I actually ran on the reference machine (RTX 3050 Laptop, 6 GB). For anything else it says not measured and tells you to start with one block and let the trainer's autotune measure the real MB/block. Planners that guess throughput are worse than no planner.
What the engine does
streamft fine-tunes models that do not fit by never holding them: the NF4 weights are mmap'd from disk, one transformer block is materialised on the GPU at a time, and the backward recomputes through it. LoRA is trainable, the base stays frozen.
Measured on a 6 GB laptop card: a 32B model at 1024 tokens/step, 36.7 tok/s, 4.61 GB peak. Thirty SFT steps moved held-out loss from 1.8755 to 1.0008, with 64/64 layers receiving gradients.
The comparison (same data, masking, metric, LoRA)
| model | streamft | PEFT/transformers | streamft peak | PEFT peak |
|---|---|---|---|---|
| 3B | 0.94 s/step | 0.58 s | 3.85 GB | 4.90 GB |
| 7B | 1.70 s | 8.96 s | 5.51 GB | 10.05 GB |
| 14B | 6.73 s | 18.56 s | 3.86 GB | 16.48 GB |
| 32B | 12.5 s | fails before step 1 | 3.90 GB | — |
That third row is the honest one: on a 3B, PEFT is faster than this engine and that's fine. On Windows, PEFT does not OOM for 7B/14B — it silently spills into shared system memory (10 GB and 16 GB reserved on a 6 GB card) and pays 5.3x and 2.8x the step time. At 32B it dies in prepare_model_for_kbit_training before training starts.
Why I trust the numbers
Two checks are in the repo and runnable:
-
scripts/verify_gradients.py— one resident block (with recompute) vs all blocks resident, same adapter weights: loss identical to 10 decimals, adapter gradients bit-identical (288 tensors, relative error 0.0). Against PyTorch autograd: 1.33% global — and the same 1.33% with all blocks resident, so it's bf16 dequantisation, not the streaming path. -
scripts/compare_peft.py— the matched runs above. It also caught a bug in my first comparison: PEFT's eval averaged per row while mine averaged per token, which made the two look 0.7 apart. Fixed, both start at the same number (1.42087).
What it does not do
- 2k context does not fit on 6 GB (~0.6 GB short, bf16
lm_head);--head nf4is broken in the bitsandbytes version I tested. - 3B/7B fine-tuning is faster with PEFT. This is not a speed tool.
- One GPU, Windows/WDDM, AC power, warm medians.
Links
- Calculator: https://huggingface.co/spaces/byjameson/will-it-fit
- Code, guide, verification: https://github.com/byjameson/stream-finetune
- The measurement write-up: https://dev.to/mycute_97b39c4e34027570f9/fine-tuning-a-32b-model-on-a-6-gb-laptop-gpu-and-the-three-measurements-that-nearly-fooled-me-2ojm
If you try it on a different card, the interesting thing is the disagreement between the planner and reality — that is exactly what the autotune log shows, and I would like to hear where it is wrong.
Correction (0.2.1): the masked SFT loss had a per-chunk normalization bug (gradients off by 112–427% on uneven masks). Fixed and every number re-run; the 32B held-out loss above is the corrected 1.8755 → 1.0008.
Since publication (streamft 0.3.0): fp16 compute for pre-Ampere GPUs (GTX 10/16-series have no bf16), a CPU training path (0.5B, ~1.7 s/step on a laptop with no GPU), a local data generator (self-instruct, 43% keep rate with 1.5B), local JSONL datasets for SFT, and an inference budget mode in fit (weights + KV cache). The 32B 50-step run was also re-measured on the fixed masked gradient: 1.8755 → 0.9659.
Since publication (streamft 0.4.1): the adapter can now be merged into the base model (streamft merge), exported to GGUF with llama.cpp, and run in a browser tab with no server and no upload. Measured on the same laptop with a 0.5B Q4_K_M (379 MB): 1.3 tok/s with a plain static server, 6.0 tok/s behind 12 threads — the difference is the COOP/COEP headers that enable SharedArrayBuffer (multi-threaded WASM). The demo is wasm/index.html in the repository.
Still true: the 0.5B is a small model, and the browser path is CPU/WASM with no WebGPU offload wired up. It is a serving path, not a trainer.
Top comments (2)
"Memory is arithmetic. Speed is only printed where it was measured" is a rule more tools should follow. Streaming one block at a time to fine-tune a 14B on 6GB is clever; the trade-off is time per step, so it would be useful to show where that crosses over with just renting a bigger card for the same job.
Fair question — and there are three different crossovers inside it: whether the model fits, money, and wall-clock. They land in different places.
1) Does it fit — this one is measured. Same 6 GB card, same data, same LoRA, PEFT vs streamft:
On fixed hardware the crossover is exactly the VRAM boundary: below it PEFT is ~1.5x faster, above it PEFT spills into shared system memory and pays 3–6x the step time. That is the only crossover I have real numbers for.
2) Money. 14B is 5.86 s/step, so 1,000 steps ≈ 1.6 h. This laptop pulls maybe 90 W at the wall (estimated, not metered); at $0.20/kWh that is ~3 cents per 1,000 steps. A rented GPU is $0.5–2/h. Renting only wins on money if it is tens of times faster per step — 30x at $0.5/h, 110x at $2/h — and no card is. Renting never pays for itself in dollars; it buys time.
3) Wall-clock — this is where it actually crosses. One epoch of Dolly-15k at 14B, batch 1 = 15,000 steps ≈ 24 h here. At a 5–10x speedup on a 24 GB card that is 2.5–5 h, so $1–10. The question becomes: is 20 hours of waiting worth ~$5? For a 50-step sanity run, no. For a full epoch, almost always yes.
Caveat: the 5–10x is an estimate. I have no rented-card rows — every number above comes from one RTX 3050 laptop with 6 GB. If you rent a 4090 or A100 and run the same config, the template in
docs/MEASURED.mdtakes the row and it gets added as measured, not estimated.One framing note: streamft is not competing with renting. If you rent 24 GB, use PEFT — it is faster when the model fits, and the README says so. streamft is for the card you already have, and for the models that do not fit on it.