A fit calculator answers one question: will these weights load in your memory. A leaderboard answers a different one: is this model good at tests in general. The question that decides what you actually run is neither. It's "will Q4_K_M hurt my invoice extraction accuracy on my machine," and the only reliable way to answer it is to run your extraction on Q4_K_M and measure.
QuantProof automates that. Point it at a folder of real examples from your task. It sweeps every model you already have in Ollama (or whatever a Rapid-MLX server is serving), scores every output with deterministic scorers, measures time to first token, tokens per second, and peak memory, then recommends the smallest quant whose quality stays within 2% of the best measured result.
npm install -g quantproof
quantproof ingest my-tasks.md # a local model drafts the task pack from your notes
# or: quantproof init my-task # scaffold it yourself
quantproof run --pack my-tasks
quantproof report --markdown
Why measure instead of estimate
Three task packs ran on an M5 Max: ticket classification, invoice extraction, config generation. Three models, 540 scored generations. No model won all three.
An 8B at Q8_0 took classification at 0.950, beating a 30B at 45% of the memory. The 30B was the only model to score 1.000 on extraction. They tied at 1.000 on config generation, where the 8B needs about 17 GiB less.
The 8B's zero on extraction was a token-budget truncation issue, not a quality failure. On 57 of 60 generations it spent its entire 512-token budget on reasoning and never emitted content. Because every raw output is journaled, the report flags the zero trunc! and names the fix (raise max_tokens) instead of letting it read as "this model can't extract."
Which model is "best" changed with every task. That is the leaderboard problem in one paragraph: the ranking is a property of the model and the workload together, and a leaderboard only knows the model.
Determinism is a backend version property, not a model property
Every example runs 3 times at temperature 0 with a fixed seed, and repetitions are compared byte for byte. On Apple Metal, every candidate repeated exactly, across 240+ scored units. Then the same check ran on an RTX 5070. Same pack, same seed, same driver, two Ollama versions:
| candidate | ollama 0.15.2 | ollama 0.32.1 |
|---|---|---|
| llama3.1:latest (Q4_K_M) | nondet | deterministic |
| llama3.1:8b-instruct-q4_0 | nondet | deterministic |
| gemma3:4b | nondet | nondet |
| gemma3:1b | deterministic | nondet |
Both 8B quants became deterministic after the upgrade. The 1B flipped the other way.
Not even the direction of the effect survived a backend upgrade, which is why the check runs on every sweep instead of being trusted once. One example answered "As needed" on repetition 1 and "Weekly" on repetitions 2 and 3, at temperature 0 with a seed: identical conditions, different outputs.
Predictions print next to measurements
The fit prediction is deliberately simple: weights on disk, plus f16 KV cache for the context, plus a fixed 1 GiB compute allowance. On the 5070, three of four candidates measured within 15% of prediction; the 1B ran over 20% under its intentionally conservative estimate. Every report shows both. Use predictions for planning. Measured peak is your actual constraint.
Peak memory comes from the most honest source each machine has:
- nvidia-smi on NVIDIA GPUs
- Backend process RSS on Apple Silicon (validated against Ollama's own accounting)
- Metal accounting on Rapid-MLX
- /proc/meminfo-based RSS on everything else
A machine with no working source reports "not measured." Nothing is ever estimated.
Anyone can recheck the scores
quantproof report --bundle exports a zip with every raw output, all scores, model digests, sampler params, and the backend version. Re-scoring the 5070 sweep from bundle contents alone reproduced 264 of 264 scores exactly, on both Ollama versions. A number you can't recheck is marketing.
The fine print (read before citing any of these numbers)
quantproof ingest measures agreement with its drafting model until human review. One run: drafter got 1 of 22 expected values wrong. The provenance label exists because of that.
Run it on your task
Node 22+, Ollama or Rapid-MLX. Out-of-memory failures are recorded as results, not crashes, and the same packs can also sweep Claude models over the Anthropic API when no GPU is around.
moonrunnerkc
/
quantproof
Run your real task on quantized models and get measured quality, latency, and peak VRAM, plus the smallest quant that holds quality.
Why measure · How it works · Case study · Run it on your task · API runs · Documentation
QuantProof
Point QuantProof at a folder of real examples from your task, and it tells you which quantized model performs best on your hardware.
It measures, not estimates:
- Quality
- Latency
- Peak VRAM usage
Then it recommends the smallest quantized model whose quality is within 2% of the best-performing model.
Why measure
Fit calculators only tell you whether a model's weights will fit in memory Benchmark leaderboards rank models on general tests, not your specific workload.
If your real question is:
"Will Q4_K_M hurt my invoice extraction accuracy?"
there's only one reliable way to answer it: run your invoice extraction on Q4_K_M and measure the results.
How it works
QuantProof automates that process. It:
- Tests candidate models one at a time
- Scores every output deterministically
- Measures latency and peak VRAM
- Reports…
Top comments (1)
The determinism table is the most interesting thing here and I suspect it'll get less attention than the model comparison. Two Ollama versions, same seed, same pack, and determinism flips in both directions depending on the model — that quietly invalidates a lot of "we pinned temperature 0 so our evals are reproducible" assumptions. Temperature 0 buys you a decoding rule, not a byte-identical backend.
The truncation catch is the other thing worth calling out. An 8B scoring 0.000 on extraction reads as "this model can't do the task" in any report that only shows the score, and the real answer was that it burned its whole budget on reasoning tokens and never emitted content. We've been bitten by exactly that shape — a scorer faithfully reporting a number that's measuring the harness, not the model. Journaling raw outputs so the report can distinguish the two is the fix, and almost nobody does it until they've been burned once.
Does QuantProof re-run the determinism check per backend version automatically, or is that opt-in per sweep? Given the flip you found, defaulting it on seems right.