I run a hosted chat and coding agent on open-weight models. This is the single finding that cost me the most time in the last few months, and almost nobody talks about it.
The number on the model card is not the number you get
The context window advertised on a model card is a property of the weights. The context window you actually receive is a property of whoever is serving them.
Across the hosted endpoints I have tested, most serve around 32K regardless of what the card claims. A handful reach 256K. I have not yet found one actually serving the 1M figures that appear in release announcements.
This is entirely reasonable from the operator's side. Max context length is a KV-cache budget traded off against concurrency. Serving a million tokens per request to everyone would be ruinous. The problem is not that operators do this. The problem is that it is almost never documented and it fails silently.
Nothing errors
That is the part that makes it expensive. You do not get a 413. You do not get a warning. The request is served against whichever ceiling is lowest and the front of your context is simply gone.
For chat, this is close to invisible. The conversation gets slightly dumber over time and you assume that is just how it goes.
For anything agentic it is the binding constraint, and it fails in a way that does not look like a context failure:
- a long build run hits the ceiling
- compaction fires
- the goal statement gets summarised into vagueness
- the model reads back its own compacted notes, no longer knows precisely what it was doing, and quietly restarts the plan
From the outside that reads as "this model is bad at long tasks." It is the serving config.
How to find your real number
There is no clean way to do this, which is itself the story. Two methods that work:
Send a deliberately oversized prompt and read the error. Providers usually leak the true maximum in the error text even when the docs do not mention it. Push 500K tokens at an endpoint claiming 1M and see what comes back.
Watch where truncation starts biting. In a long run, find the point at which the earliest content stops influencing output, and count backwards. Uglier, but it works when the error text is unhelpful.
Provider documentation has not been reliable for this in my experience. Neither has the model card.
Three consequences worth internalising
1. A benchmark number is a score for a model at one served context length, not a score for a model. The same weights at 32K and at 200K are not the same agent. If you are comparing providers on cost per token without pinning the served ceiling, you are comparing two different things and calling the difference price.
2. If you use a gateway with fallbacks, the ceiling can change mid-session. A run that starts on a 200K provider and fails over to a 32K one does not error. It truncates. That is a correctness problem, not a performance one, and nothing in the response tells the caller it happened.
3. In RAG, this silently invalidates your retrieval tuning. Chunk size, top-k and reranking are all tuned against an assumed budget taken from the card. If you tuned for 128K and you are served 32K, you are over-retrieving, your reranked chunks get truncated, and the answer comes back confident and wrong. Then you go tune the embedding model, which was never the problem. Reranking in particular buys you nothing if the top results do not survive into the prompt.
What I would like to exist
A discoverable way to read the effective served context per request. Model metadata, a response header, anything. Right now every client reverse-engineers it from error strings, which is absurd for a number that determines whether your application works.
If you know of a provider that publishes this honestly, I would genuinely like to hear about it.
Founder disclosure: I build Grunz, a chat and coding agent running open-weight models, which is how I ended up learning all of this the expensive way.
Top comments (0)