I keep a baselines.md in every agent repo. It has one rule at the top that I wrote after a phantom regression ate a week of my evenings: a baseline is a file, not a model name.
This week's reminder is that the DeepSeek V4.1 build most people are actually pulling down isn't DeepSeek's. Search the model name on any hub and the thing that surfaces first, the thing that gets mirrored into every tutorial and every starter compose file, is a third-party mixed-precision GGUF. The vendor's official card is sitting right there and almost nobody is serving it. That's not a scandal, it's just how open weights work — the quant lands, the community makes it run on consumer hardware, and the runtime gets adopted before the reference implementation does.
The problem is what that does to your numbers.
Your baseline is against a build you don't run
If you benchmarked on the vendor's card in bf16 on an A100 and then shipped the community 4-bit GGUF on a 4090, your regression suite is comparing two different artifacts and calling the difference "quality." It isn't. It's a quantization delta, a runtime delta, a template delta, and a hardware delta all stacked on top of each other, and you have no way to attribute any of it.
I've watched a team spend a week tuning prompts because their agent's tool-calling accuracy fell off a cliff after "upgrading the model." They'd upgraded the quant type. Same weights, different file. The prompts were fine.
So here's the tip, and it's boring on purpose.
Pin the revision, not the tag
Tags move. latest moves. The filename ...-Q4_K_M.gguf can be re-uploaded under the same name with different tensors if the maintainer re-runs the quant. If your deployment config says "pull model X, 4-bit," you do not have a reproducible system, you have a wish.
Pin the commit hash on the repo and the digest of the file. In llama.cpp, -hf user/repo:quant accepts a revision — use it. In vLLM, --revision <sha> on the HF repo. In Ollama, pin by digest, not by tag, and check what you actually have with ollama show --modelfile rather than trusting the name you typed. Then write the sha256 of the .gguf into your eval output, next to the score. Not in a wiki. In the eval output, so it's impossible to read a number without also reading what produced it.
If you can't reproduce the exact bytes six weeks from now, the number means nothing.
Diff the chat template before you diff the model
This is the one that keeps biting people. A GGUF carries its prompt template in metadata. If the quant repo's template drifted from the vendor's tokenizer_config.json — a missing BOS, a different system-prompt wrapper, tool-call tokens in a slightly different order — the weights are identical and the model still behaves like a different model. Your tool-calling agent will look dumber and you'll blame the quant.
Dump the template from the artifact, diff it against what you expected, and put that diff in the PR. It takes minutes.
Freeze the knobs, then freeze them again
Quantization changes memory footprint, which changes what you can afford to keep in KV cache, which changes your context length, which changes your agent's behavior on long tasks. So the eval config needs the whole serving envelope pinned, not just the model: context size, KV cache type, rope scaling, batch size, offload settings, sampler, seed. I run evals greedy with temperature 0 and a fixed seed, because I want my failure to be my fault and not a dice roll.
And measure latency separately from quality. A 4-bit quant that scores the same as bf16 but blows up your time-to-first-token under batch is not a win, it's a relocation of the cost. Two tables, always.
Keep both artifacts servable
The honest way to know what the quant cost you is to have both files on disk and route the same prompt set through both in the same harness on the same day. Shadow traffic if you can, a fixed offline set of real prompts if you can't. What I look for isn't the average score — averages hide everything — it's which categories of task get worse. In my experience the damage from aggressive quantization is uneven: plain reasoning holds up, long structured output drifts first, and multi-turn tool calling is where things quietly fall apart, because errors compound across turns and one malformed argument kills the whole trajectory.
I haven't pushed V4.1 deep into a long-context agent loop yet, so treat that as the thing I'd check first rather than a finding. Mixed-precision quants keep certain tensors at higher precision, and which ones get protected varies by recipe, so "4-bit" from one repo is not comparable to "4-bit" from another. That's another reason the digest matters more than the label.
A pinned quant is also a pinned bug
Fair warning, because I've been on both sides of this. Pinning means you don't get silent upstream fixes. So schedule a re-test — monthly is fine — where you deliberately unpin, run the suite, and decide whether to move the pin. That's a config change with a diff and an eval attached, which is exactly what you wanted anyway.
The vendor card is the reference implementation. The community quant is the production artifact. Most of us are serving the second one and reporting the first one's numbers. Write down which file you ran, and the argument you have next quarter will be about the code instead of about the download.
Top comments (2)
Your "pin the revision, not the tag" applies to git too, and there it is two traps rather than one. I checked both on my own repo this evening.
A tag name can point at a tag object or straight at a commit. Mine, v0.3.23, is annotated, so
git rev-parse v0.3.23answers f36ae3fa9115 — that is the tag object — andgit rev-parse v0.3.23^{commit}answers c2e8f77194ef. Both answers are correct, to two different questions. A deploy script that stores the first one and later compares it againstrev-parse origin/masteris comparing a tag object with a commit, and those disagree even when they name the same commit. That is the label-versus-digest problem one layer down, and it fails the way you describe: nothing errors, you simply get two numbers that were never comparable.Pinning the tag does not pin the bytes either, because a tag name can be deleted and recreated on a different commit. That is your re-uploaded Q4_K_M file under the same filename — same name, different tensors, no error anywhere. The thing to write next to the score is the commit sha, not the tag name.
One more belongs in the checklist next to your eval config, because it caught me this week: a fetch that exits 0 is evidence about the remote it fetched from, and about nothing else. My clone holds 124 tags and the newest is v0.3.23, while the release API answers v0.3.33 — and the commit that v0.3.33 tags, 06a001a, is sitting in that same clone. The objects were there and the name was not, and no command I ran said anything was missing. A stale reading does not announce itself as stale; it is a complete answer to a question you did not ask.
A third trap that belongs in this list is RoPE scaling and context limits embedded in the GGUF header.
Plenty of third-party quants get compiled with default 4k context parameters, even when the upstream model card claims 32k or 128k. When an agent loop hits turn four and the context window crosses 5k tokens with tool definitions loaded, the runtime quietly shifts into linear rope frequency interpolation or starts sliding the window. The model does not throw an out-of-memory error. It just starts mangling tool arguments or dropping earlier system instructions, and the team assumes the model became unreliable under multi-turn pressure.
Diffing the header metadata with gguf-dump before pushing the binary to the inference box saves days of chasing phantom reasoning bugs.