TL;DR — Downloading open weights feels like portability, but the real deployable unit is a five-part tuple: weights, quantization recipe, runtime kernel, chat template, and tokenizer config. Most local AI setups only pin the first one, which is why the 'same model' produces different outputs on different machines. Treat the whole tuple as a versioned artifact, or your local-AI stack will quietly drift under you.
Everyone treats open-weight models as the portable alternative to closed APIs. Download the file, point a runtime at it, and you own your inference stack forever. That story is half true. The weights are portable. The thing that actually produces your output — the full runtime contract around those weights — is not, and almost nobody versions it as a unit.
This is the uncomfortable discovery teams make six months into running their own models: two engineers load "the same model" on two machines and get subtly different completions. Not because the weights changed, but because everything around the weights did.
The artifact is a tuple, not a file
A deployed local model is not one thing. It's five things, bound together at inference time:
The weights themselves
The quantization recipe applied to them (format, calibration data, group size)
The runtime kernel that executes the forward pass
The chat template that turns a conversation into a token sequence
The tokenizer and its special-token configuration
Most teams version exactly one of these — the weights, usually by a model name and a quantization suffix. The other four live wherever the runtime defaults put them, inherited silently from whatever version of the serving tool happens to be installed that week. Nobody writes down the Jinja template revision. Nobody pins the tokenizer's end-of-turn token mapping. Nobody diffs kernel implementations across releases of their inference engine.
Open weights solved the licensing and portability problem at the model layer. They did nothing to solve it at the stack layer, and the stack layer is where correctness actually lives.
Chat templates are the hidden API contract
The chat template is the least glamorous and most consequential piece of this tuple. It's the function that converts a list of role-tagged messages into the literal string of tokens the model sees. Change the whitespace around a role tag, drop a system-prompt wrapper, or update which end-of-sequence token signals a turn boundary, and the model's behavior shifts — not because it got dumber, but because you changed its input distribution relative to what it was tuned on.
This is functionally identical to shipping a breaking change to an internal API, except there's no contract test catching it. Runtime projects update default templates between releases, sometimes to fix a real bug, sometimes just to track upstream changes from the model publisher. If your deployment pulls "latest" on the runtime without pinning the template, you can update one dependency and watch your eval scores move for reasons that have nothing to do with the model.
Teams that have been burned by this eventually start treating the chat template the way they'd treat a serialization schema: checked into version control, diffed in code review, and tested against a fixed set of conversations with expected token-level output. That's not overkill. It's the bare minimum for a component that directly determines what the model reads.
Quantization and kernels are a second, separate lock-in
The quantization format you pick — and more importantly, the kernel that executes it — is where local AI reinvents the exact vendor lock-in it was supposed to escape. A model quantized for one runtime's kernel implementation doesn't just run slower on another; it can produce measurably different outputs, because different kernels make different rounding, fusion, and numerical-precision tradeoffs in the matrix operations. "GGUF" or "AWQ" is not a universal format in the way a JPEG is universal. It's a contract with a specific execution engine's interpretation of that format.
This means switching local-AI runtimes isn't like swapping an HTTP client. It's closer to swapping a database engine: the data format might be nominally compatible, but the actual computed results, performance characteristics, and failure modes are runtime-specific. Teams that pick a runtime for convenience early on often find themselves unable to leave without re-quantizing and re-validating everything, which is a lock-in cost nobody accounted for when they chose "open" over "hosted" to avoid lock-in in the first place.
What reproducibility actually requires
If you're running open-weight models in production, the unit you need to version is the tuple, not the checkpoint. Concretely, that means a few unglamorous practices:
Pin the runtime version and its kernel build, not just the model file hash.
Check the chat template into your own repository rather than trusting the runtime's bundled default, and diff it on every dependency bump.
Keep a small, fixed golden set of prompts with expected outputs (or expected output distributions, for non-deterministic sampling) and run it as a regression test whenever any piece of the tuple changes — weights, quantization, runtime, template, or tokenizer.
Treat quantization as a model change, not an infrastructure change, and re-run your evaluation suite against it the same way you would after a fine-tune.
Record the full tuple alongside every model version in whatever registry you use, so "which model is serving traffic" actually answers the question instead of half-answering it.
This is more process than most teams want to run for something that feels like it should be as simple as copying a file. But the complexity was always there — closed-API providers were absorbing it for you, silently, behind a stable endpoint. Open weights hand you the capability and the responsibility in the same package. You can't take the first without the second.
The real cost of "just self-host it"
None of this is an argument against open-weight models or local inference. It's an argument against treating the weights file as the whole story. The pitch for local AI is control and reproducibility — you're not subject to a provider silently swapping the model behind an API name. But you only get that benefit if you actually exercise the control, which means owning and testing the full tuple instead of letting four-fifths of it default to whatever your runtime shipped with this week.
The teams that get burned aren't the ones running obscure models with sketchy licenses. They're the ones running a well-known open model, correctly, who upgraded their inference server for an unrelated bug fix and never noticed the chat template moved underneath them. Open weights made the model portable. They didn't make your stack a model. That part is still your job.
Top comments (0)