TL;DR— Running open-weight models locally shifts the real cost from API bills to evaluation debt. Quantization, engine choice, and version churn all silently change model behavior, and most teams never re-test after making these changes. The hard part of local AI isn't installing weights— it's building the continuous eval pipeline that vendors used to run for you.
Every pitch for open-weight models ends the same way: download the weights, point an inference engine at them, and you're free. No rate limits, no vendor lock-in, no surprise deprecations. What that pitch leaves out is that the vendor was doing something for you besides serving tokens. They were running eval suites against every change before it shipped. The moment you self-host, that job doesn't disappear. It becomes yours, and almost nobody budgets for it.
Quantization Is Compression, and Compression Is Lossy
The whole reason local AI is viable on normal hardware is quantization. A dense model at full precision needs GPU memory most teams don't have; a 4-bit or 8-bit quantized version of the same weights fits on a single card. Formats like GGUF, AWQ, and GPTQ made this almost mechanical— pick a scheme, run a script, get a smaller model that mostly behaves the same.
"Mostly" is the word that should worry you. Quantization is lossy compression applied to a function approximator, and the loss isn't evenly distributed. It tends to concentrate in exactly the places that matter for hard tasks: multi-step reasoning, numeric precision, rare tokens, long-context retrieval. A quantized model can pass a casual smoke test— write me an email, summarize this paragraph— and still have quietly degraded on the narrow slice of behavior your product actually depends on.
The vendors who trained the original weights ran their eval harness against the full-precision checkpoint. They rarely publish results for every quantization level a community will eventually produce, because they didn't make that artifact— someone downstream did, often with a script and a prayer. If you're running that artifact in production, you are the only party in the pipeline who could know whether it still meets bar. Most teams never check.
The Engine Is Not a Neutral Pipe
Quantization gets attention because it's a deliberate, visible choice. Inference engine selection gets almost none, and it shouldn't. Serving stacks like vLLM, TGI, llama.cpp, and SGLang differ in more than throughput. Continuous batching strategies, KV cache eviction policies, prefix caching, sampling implementations, and default stopping behavior all vary between engines— and all of them can shift output distributions in ways that show up as behavioral differences, not just latency differences.
Switch your serving engine to chase better tokens-per-second and you may have also changed how the model handles long conversations, how it truncates context under memory pressure, or how deterministic its outputs are at temperature zero. None of this is in the model card. It's implementation detail three layers below the weights, and it's exactly the kind of thing that turns into a support ticket three weeks after a migration that everyone thought was a pure infra change.
Vendors Run Continuous Eval. You Probably Don't
Hosted model providers treat evaluation as a standing pipeline: every checkpoint, every serving change, every quantization or distillation gets tested against a battery of benchmarks and internal regression suites before it reaches a customer-facing endpoint. That process is invisible to API users, which is precisely why it feels like it doesn't exist. It exists. You're just not paying for it directly— it's baked into the price of the token.
Self-host the same model family and that pipeline vanishes unless you build it. Most local-AI stacks get evaluated exactly once: at adoption time, when someone runs a handful of prompts, likes what they see, and ships it. After that, every subsequent change— a new quantization, a newer point release of the same model, a swapped inference engine, a bumped context window— goes untested. The stack keeps running. Nobody notices the regression because nothing crashes. The model just gets a little worse at the thing it used to be good at, and that erosion is invisible until a user complains or a downstream metric quietly slides.
What Actually Owning the Stack Looks Like
Real ownership of a local-AI deployment isn't infra control. It's a promotion gate. Before any change— a new quantization level, a new engine version, a new checkpoint from the same model family— goes to production, it should run against a fixed, versioned eval set that reflects your actual task distribution, not a generic public benchmark. Generic benchmarks tell you if the model can still do arithmetic. They tell you nothing about whether it can still extract line items from your specific invoice format.
That eval set doesn't need to be enormous. It needs to be stable, scored consistently, and treated as a regression suite the same way you'd treat unit tests for a service. Store the outputs from your current production build as a baseline. Diff new candidates against that baseline before swapping them in, not after. Canary the new build against a slice of traffic and compare, rather than trusting a spot-check from someone's laptop.
This is more work than pulling a new GGUF file off a model hub and pointing your server at it. It's also the only way "local" actually buys you the control it promises. Without it, you haven't taken ownership of the model— you've taken ownership of its failure modes, minus the visibility into when they change.
The Trade You're Actually Making
Open-weight models are a genuine architectural win: no per-token billing, no opaque model swaps you didn't ask for, full control over data residency and latency. None of that is in question. What's underpriced in most local-AI adoption plans is the operational half of the trade. You're not removing a dependency on someone else's eval pipeline. You're inheriting the job of building one, running it continuously, and treating every quantization, engine upgrade, and checkpoint bump as a change that needs to earn its way into production.
Teams that skip this step get the upside of local AI— cost, control, independence— right up until a quiet regression ships to every user simultaneously, with no vendor status page to tell them what changed. The fix isn't more caution about adopting open weights. It's budgeting for the eval harness at the same time you budget for the GPU.
Top comments (1)
Self-hosting the model without self-hosting evaluation creates a blind spot. You save on inference control, but then have no local answer to "is this model still good enough for our actual workflow?"