DEV Community: Eryk Kubiak

Local Coding Agent: Weak Model? No, Bad Plumbing

Eryk Kubiak — Thu, 08 Oct 2026 16:02:58 +0000

In this video:

0:00 Why agents fail around turn ten
0:19 It runs but is useless
1:54 The base-URL swap
3:26 Where compatibility breaks
5:28 From tokens to a tool call
8:31 Context, speed and turn count
12:25 Local vs hosted trade-offs
14:11 Harness, server and model pairings

A local coding agent that falls apart by turn ten is usually blamed on a weak model. Usually it's plumbing. Under 24 GiB of VRAM, Ollama defaults to a 4k context and silently drops history. Three budgets decide whether the agent survives.

An agent that runs but is useless

A fully local coding agent has four parts. The first is an inference server that exposes the API the harness speaks: chat completions, Responses, or Anthropic Messages. The second is a model that emits tool calls. The third is a harness pointed at that endpoint. The fourth is three tools: read a file, write a file, run a command.

Wire those together and the agent starts, accepts a prompt, and answers. By every status check, it works. Then you give it a real task.

A coding task is not one request. It is tens of turns: read the file, edit it, run the tests, read the error, edit again. Any small defect per turn gets multiplied by the turn count. A tool call may carry a malformed argument. A file may silently fall out of the context window.

Take a two percent chance of a bad call on each turn. Over thirty turns, that is about a forty-five percent chance the run hits at least one.

Speed compounds the same way. On a cache miss, the server re-reads about fifteen thousand tokens at roughly five hundred a second. That is thirty seconds, and across forty turns, twenty minutes of staring at a spinner. That is a worst case, and it varies by hardware.

Three budgets that land on you

A hosted model mostly hides these problems behind per-token pricing. The provider handles serving, parsing and throughput. Run the same loop locally and all three land on you: tool-calling reliability, context length, and throughput.

These are the constraints that separate a workable agent from one that only technically works. When one of them fails, nothing crashes. The agent just stops being useful. Very often the cause is the plumbing between the model and the harness, such as a small context default or the wrong tool-call parser, before it is the model's raw ability.

The obvious setup: swap the base URL

The first move is almost always the same, and it is a sensible one. Start an inference server: llama-server, Ollama, vLLM or LM Studio, whichever is already on the machine. Pull a well-regarded chat model, one that ranks high and feels sharp in conversation. Then change a single value in the harness settings, the base URL, so it points at localhost instead of a hosted provider.

This looks like it should work because every one of those servers advertises an OpenAI-compatible endpoint, and the label reads like a single contract. The request shape is the same and so is the response shape. So any model that chats well should be able to drive the loop, and the harness should not care who answers.

The check seems to agree. You send a message, get a fluent reply, maybe a clear explanation of a function. Everything is green.

But look at what that check exercised: one short prompt and one plain-text answer. It involved no structured output, no long prompt and no repeated turns. Those are exactly the three things the loop demands, and a chat check touches none of them.

What the harness can and cannot repair

The harness is the loop around the model: a system prompt, a set of tool definitions, and some context management. That is all. It sends text in and reads text back.

Suppose the server fails to parse the model's tool call. Or it quietly trims the prompt. Or the model was never trained on the call format. In each case the harness receives the broken result and has nothing to repair it with. So the swap succeeds, and proves almost nothing.

Where the swap actually breaks

The label OpenAI-compatible used to mean one thing: the chat completions endpoint. It no longer does. Codex CLI removed the chat completions wire API for custom providers. A local server therefore has to implement the responses endpoint, and a config still set to chat refuses to start.

Claude Code speaks the Anthropic Messages format instead. Ollama added that in version 0.14.0, in January 2026, and llama-server serves it at /v1/messages. The label is the same, but there are three different wire formats. A mismatch here is the loud failure, and the easy one. The quiet ones are worse.

Tool templates and parsers

llama-server does tool use only with the jinja flag on. Without it, the model's tool template is never applied, and what should be a call comes back as ordinary text. Some models need a template file override on top of that.

vLLM needs enable auto tool choice, plus the tool call parser that matches the model family. Pick the wrong parser and the model can emit a perfectly good call that the server cannot read. The harness sees prose where a tool call should be, and the verdict is: this model can't call tools. The weights were fine.

Context defaults

Ollama sets its default context by VRAM: 4k under 24 GiB, 32k from 24 to 48, and 256k at 48 or more. Its own docs say agents need at least 64000.

A system prompt, a set of tool definitions and two source files can fill four thousand tokens quickly. Aider's docs warn that whatever overflows is silently discarded. There is no error and no warning. The agent simply forgets the file it read three turns earlier, and edits it from a guess. That looks like a model with a bad memory.

No grammar behind auto tool choice

Even with the right flags, vLLM's auto tool choice has no grammar behind it. Calls are extracted from raw text. The docs say arguments may occasionally be malformed or violate the schema, and that malformed markup can leak into the response. Most clients never set strict, so nothing stops it.

"Occasionally, over thirty turns" is the arithmetic from the opening. Every one of these failures reads from the outside as bad reasoning. None of them is.

From tokens to a tool call

The model never emits an API object. It emits text. Depending on the model family, that text is JSON inside special tokens, or XML-style tags, or Harmony markup with channels and recipients.

The tool_calls field your harness reads is built afterwards, by the server, in two steps. First, the chat template renders your tool definitions into the prompt in the format the model was trained on. Then a parser reads the generated text and tries to find a call in it.

In llama.cpp, a native handler covers templates it recognises. Anything else falls to a generic fallback, which is a best guess at the format. In vLLM, you choose the parser yourself. Either way, the call is only as real as the parser's ability to find it.

How a missed parse ends the loop

When the parser misses, nothing dramatic happens. The text is not a call, so it becomes the assistant's final answer, and the loop ends.

One LangChain forum user reported exactly this with gpt-oss-120b and the Harmony format. After about five tool calls, the model put a raw call on the analysis channel instead of the commentary channel. The parser ignored it, the harness treated it as the answer, and the run stopped. Earlier calls in the same run had parsed fine. That is a user report, not an official finding. But it shows the shape of the failure: a text-level slip, with no error anywhere.

The fix is to stop relying on the model to get the markup right. Constrained decoding forces the output through a grammar, so the markup must be valid. vLLM supports this with strict tools and an operator flag called tool-strict-level. The catch is that most clients never set strict on their tools. So the server operator has to raise the floor for every request. If you run the server, that is your job.

Quantization: parsed, but with worse values

The common story is that four-bit breaks JSON. The evidence is more nuanced. One small BFCL test of tiny Qwen3 models found schema validity almost unchanged at Q4_K_M: 0.877 at full precision, 0.873 quantized. But argument correctness fell, from 0.605 to 0.575. The call still parsed. It just carried a worse value.

That is one author, with very small models, so hold it loosely. Community long-context reports point the same way. Quantized weights or KV cache can silently shift which tool gets picked, or what arguments it gets, while the prose stays fluent. Fluency tells you nothing here.

Per-call error rates compound

This is arithmetic, not a measurement. Say each call has a two percent chance of going wrong. 0.98 to the twentieth power is about 0.67. Over twenty calls, the run survives only about two-thirds of the time. Small per-call rates are not small per run.

The parser is part of that rate. Unsloth's log for Qwen3-Coder-Next shows a llama.cpp bug fix on February 4 for looping and output problems. It shows another on February 19, after which tool calling should be even better once parsing was fixed. The weights were the same, but a different runtime gave different reliability.

A tool call that parses and carries the right arguments is one budget. The second fails with even less noise: whether the model still has the file it read three turns ago.

The loop budget: context grows every turn

A model does not remember anything between turns. Every turn, the harness sends the entire history again: the system prompt, the tool definitions, the files being edited, every tool output, every earlier reply. So the prompt only grows. By turn twenty it holds everything the agent has read and done, and the KV cache for that context grows with it.

That cache lives in memory next to the weights. On a long session it often hits the memory limit before the weights ever do. A 256K window on a model card is a ceiling, not a promise.

Prefill, decode and the prefix cache

The time cost has two separate parts. Prefill is the model reading the prompt. Decode is the model writing the answer. They run at very different speeds.

One measurement of a Radeon AI PRO R9700 running Qwen3.6-35B-A3B at Q4 under llama.cpp, at 32k tokens of depth, gave 2,113 tokens per second of prefill and 111 of decode. That depth matters. The headline pp512 figure, a short prompt, flatters every machine, because both numbers slow as context grows.

Prefix caching is what keeps the loop affordable. The server keeps the computed state for the unchanged start of the prompt. A normal turn then only prefills the new tokens: the last tool output and the model's last reply.

A cache miss breaks that, and the server must re-prefill everything. One first-hand report from a Mac Studio, a blog post and not a controlled benchmark, measured about 500 tokens per second of prefill. At that rate a 15,000-token context is a 30-second pause before the first new token. Anything that changes the early part of the prompt can cause it.

Turn-time arithmetic, and what thinking does to it

The number that decides usability is turns multiplied by prefill plus decode time. Here is an illustration, and it is arithmetic, not a measurement.

Take the R9700 figures and forty turns. Those rates were taken at 32k depth, so they are pessimistic for the early, short-context turns. Each turn prefills 2,000 new tokens, about one second, and decodes 500 tokens, about four and a half. That is under four minutes for the whole task.

Then let the model think. Some models overthink by default: Simon Willison wrote in August 2026 that Qwen3.8-27B is excellent but defaults to heavy overthinking. Suppose the same Qwen3.6-35B-A3B setup did that, adding 3,000 thinking tokens per turn. Decode alone is about thirty-one seconds. Forty turns then take roughly twenty-two minutes. The hardware, the model and the task are all the same.

A dense model like that 27B would likely decode slower on this card, so its real figure would be worse. Decode is the term that thinking inflates, and it is the one you multiply the most.

How model shape changes the budgets

Hardware and model shape change these terms in different ways. Mixture-of-experts models decode fast because only the active parameters are read for each token. Qwen3-Coder-Next has 80 billion parameters in total but only 3 billion active.

The catch is memory. Every expert still has to be loaded, so the whole 80 billion must fit, and Unsloth lists about 46 gigabytes for a 4-bit build. Fast decode does not make the model small.

The other lever is attention. Only 12 of that model's 48 layers use full attention, each with 2 KV heads. The rest use linear attention. That keeps the KV cache unusually small for a 256K window. The context budget therefore stretches much further than the parameter count suggests.

Put the pieces together. Context sets how much the agent remembers and how much memory the cache eats. Prefill sets the cost of a long prompt, and the cache decides whether you pay it once or every turn. Decode sets the cost of everything the model says. All three get multiplied by the turn count.

These numbers come from specific machines, and a loop only has to fit on one of them.

What local costs you

With a hosted model, the main constraint you manage is cost per token. Context limits, serving speed and most tool-call parsing are handled by the provider, though hosted models have limits and failures of their own.

Going local swaps that bill for three others. The first is hardware, with memory sized for the weights plus the KV cache, which grows with every turn. The second is tuning time: the flags, the parser, the context setting, all the plumbing described above. The third is slower turns, because your machine prefills and decodes at its own pace, not a data center's.

When to use a hosted model instead

Local is the wrong choice when the work is long-horizon. Vendors and StorageReview both say that long-running agentic work lags frontier cloud models. A job that needs fifty turns across a large repository stacks every budget at once.

Context is the clearest case. Raising it raises memory use, and it slows both prefill and decode. Cline's docs say to keep tasks focused and to start a new task when context grows. That is advice for a local setup, and it also describes the work local suits: focused, short-horizon tasks.

Local also wins where hosted cannot go. That includes privacy-bound code that must not leave the machine, and offline use. There, the slower turns are the price of being allowed to do the task at all.

Leaderboards don't answer the question

The model landscape moves monthly. As of October 2026, Qwen3.8-27B, open weights released on August 14, 2026, is the newest Qwen open-weight release. Qwen 4 27B was only named at Apsara on September 22, with no specs and no date.

A leaderboard position cannot tell you whether a model's tool calls parse on your server, or whether its cache fits beside its weights. The three budgets can.

Real setups: harness, server and model that fit together

A working local setup starts with the harness, because the harness decides which wire format the server must speak. Claude Code is pointed at Ollama or llama-server through its Anthropic base URL setting, and both serve the messages endpoint. Codex CLI needs a backend that implements the responses endpoint. GitHub Copilot CLI has run against Ollama, vLLM and Foundry Local since April 2026, and its offline mode needs a model with a 128K context window.

Harness design also changes how things fail. Cline offers a compact prompt built for local models, which spends less of a small context on instructions. Aider goes further. It parses diff or whole-file edits out of plain text, so there is no JSON tool call to mangle. That sidesteps JSON tool-call parsing, but swaps it for edit-format failures, and Aider's own docs say quantized local models are more prone to those.

Then pick the model by memory. VDF's September 2026 shortlist names Devstral Small 2 or Qwen3.8-27B for a 32 GB GPU, and Qwen3-Coder-Next or gpt-oss-120b for 128 GB. OpenHands' docs, in a May 2026 note, recommend Qwen3.6-35B-A3B as the first local model to try.

A four-step check before the first task

Before the first real task, verify in a fixed order. One: the API surface matches what the harness speaks. Two: the jinja flag or the parser flag is set. Three: context is at least 32k to 64k. Four: measure turn time at realistic depth, not on a short prompt.

If all four hold and tool calls still fail constantly, OpenHands says the model may be the problem, not your setup.

That order is the thesis in practice. Tool calls, context and throughput are three budgets, and most of the time the leak is in the plumbing, so check the plumbing before you blame the model.

Sources & further reading — Local Coding Agent: Weak Model? No, Bad Plumbing

Tool Calling - vLLM — https://docs.vllm.ai/en/latest/features/tool_calling/ vLLM needs auto tool choice plus the parser matching the model family. Auto tool choice has no grammar behind it, so arguments can be malformed and markup can leak into the response. Strict tools and the operator-level strict setting force valid output.
llama.cpp/docs/function-calling.md at master — https://github.com/ggml-org/llama.cpp/blob/master/docs/function-calling.md llama-server does tool use only with the jinja flag. Native handlers cover recognised templates and other templates fall back to a generic format. Some models need a template override.
Context length - Ollama — https://docs.ollama.com/context-length Ollama's default context depends on VRAM (4k under 24 GiB, 32k for 24 to 48, 256k at 48 or more), and its docs say agents need at least 64000.
Deprecating chat/completions support in Codex · openai/codex · Discussion #7782 (2025) — https://github.com/openai/codex/discussions/7782 Codex CLI dropped the chat completions wire API for custom providers, so a local server must implement the Responses endpoint.
Ollama (aider docs) — https://aider.chat/docs/llms/ollama.html A small Ollama context window silently discards whatever overflows, with no error, so the agent forgets files it already read.
Qwen3-Coder-Next: Pushing Small Hybrid Models (2026) — https://qwen.ai/blog?id=qwen3-coder-next Qwen3-Coder-Next has 80B total and 3B active parameters, and only 12 of its 48 layers use full attention. This keeps the KV cache small for a 256K window.
Qwen3-Coder-Next: How to Run Locally (2026) — https://unsloth.ai/docs/models/qwen3-coder-next A 4-bit build needs about 46 GB of memory. The llama.cpp fixes of February 4 and February 19 for looping and tool-call parsing show that the same weights behave differently on different runtimes.
Run Local LLMs with OpenHands - OpenHands Docs (2026) — https://docs.openhands.dev/openhands/usage/llms/local-llms OpenHands recommends Qwen3.6-35B-A3B as the first local model to try, and says that if tool calls still fail constantly the model may be the problem.

Disclosure: this article is the written companion to the video above. Its script was drafted with AI assistance and the narration is an AI voice.

Prompt Tricks Died. Context Engineering Didn't

Eryk Kubiak — Sat, 03 Oct 2026 20:04:24 +0000

In this video:

0:00 Do 1M windows end choosing? No
0:19 Prompt engineering is half right
1:25 Two reflexes: polish or dump
2:34 Personas, stuffing and the experiment
5:15 Selection, order and length
8:15 Tools, schemas and examples
10:58 Costs, and when stuffing is fine
12:37 Prompt, context and harness layers

Most developers assume a million-token window means you can stop choosing what goes in. That assumption is wrong. Chroma tested eighteen models in 2025, and reliability fell as inputs grew, even on simple retrieval. The phrasing tricks faded, but deciding what enters the window is still the job.

Rewording the prompt changed nothing

Take a summarisation feature on a current frontier model and rewrite the instruction ten different ways: shorter, longer, politer, with a persona, with a step-by-step nudge. Practitioners often report that the output moves far less than expected.

So the claim that prompt engineering is over is half right. Persona and magic-phrase tricks don't reliably improve accuracy. Tone and behaviour still live in prose, though, and vendor guidance stresses precise instructions plus the logic and data the task needs.

Now leave the wording alone and change the material around it. Swap which documents were retrieved, reorder them, add a tool, trim the history. The same feature gives a different answer. That failure is real, and it was never in the wording.

The claim worth defending is that one technique has been confused with the whole job. Phrasing was the technique. The job is controlling what the model sees. Anthropic's engineering team says as much: it calls context engineering the natural progression of prompt engineering. The discipline was extended, not killed. If your feature is still wrong, rewording was never going to fix it.

Two reflexes: polish the words, or dump everything in

When the output is bad, a competent developer reaches for one of two fixes, and both feel like engineering.

The first is to polish the words. You are a world-class expert. Take a deep breath and think step by step. Then come the bribes, the threats, the elaborate politeness. This is not superstition from nowhere. On earlier models, a line like that could nudge the output toward a register: more formal, more careful, closer to what you wanted. You watched it work, so you keep reaching for it.

The second reflex is to stop selecting at all. Claude Sonnet 5.5, released in late September 2026, has a one-million-token window. Gemini 3.8 Flash accepts 1,048,576 input tokens. GPT-6.1 Sol is reported at 1,050,000. Choosing documents is work, and work has bugs. So why choose? If the window holds that much, include everything and let the model sort it out. It looks free.

The two reflexes share something. Both treat the model as the variable to tune and the input as fixed. One coaxes the model with better words. The other hands it more material and trusts it to cope. Neither asks whether the input itself was the problem. And there is a measurable reason to doubt that dumping is free.

What the persona and reasoning evidence says

Start with the persona line, because it has been tested. A report on a December 2025 Wharton study, covering six frontier models on PhD-level questions, found that telling a model it is an expert did not reliably improve factual accuracy. A report on an EMNLP 2024 study, with 162 personas across four model families, found no persona strategy that consistently beat picking one at random.

Both reached me through secondary write-ups, so read the papers before you quote them. The claim is also narrow. Personas can still shape tone, and some writers still recommend them. The finding is that they are unreliable for accuracy, not that they are useless.

Reasoning has moved too. Claude's docs list the old manual extended-thinking setting as unsupported on the Claude 5 models, and adaptive thinking is the control that replaced it. On Sonnet 5.5 it is on by default, at high effort. Thinking is now a setting you configure, not a sentence you write.

A test: wording versus selection

Here is the test, on Claude Sonnet 5.5. Thirty retrieved chunks sit in a fixed pool. In the first run, three chunks stay fixed and the instruction, summarise this, is reworded fifteen ways: shorter, longer, polite, with a persona, with a step-by-step nudge. In the second run, the wording stays fixed and only the choice changes, meaning which three of the thirty go into the window. Every output is scored the same way.

The expected result is that the spread across wordings comes out small next to the spread across chunk selections. But that is an expectation, not a finding. One caveat matters here: I found no primary source that has measured this contrast, so this is one run on one model, not a law.

Where the dump-everything reflex breaks

Chroma's Context Rot study tested eighteen models and found that reliability falls as input grows, even on simple tasks like retrieval and text replication. Lost in the Middle found position effects: models did best when the relevant passage sat at the start or end of the input, and worse in the middle.

Both studies used older models. Chroma's were mid-2025 systems, and I found no replication on the million-token models shipping now. Treat them as a reason to test your own pipeline, not as a verdict. Thirty chunks is also nowhere near a million tokens, so my experiment tests selection, not rot.

Then there is the bill. GPT-6.1 Sol is reported to have a window of 1,050,000 tokens, but prompts above 272,000 input tokens bill at double the input rate for the entire request. That figure comes from a secondary source, so check the pricing page. Even so, the lesson holds: the size of the window is not the size of the range where use is reliable or cheap. A window is capacity. It does not tell you what belongs in it.

The window is an attention budget

Every token you add to the window is paid for in attention. Anthropic's engineering guidance, as summarised in a knowledge-base write-up, calls context an attention budget. The reasoning is that a transformer relates every token to every other one, so n tokens means n squared pairwise relationships. Add tokens and each one gets a thinner share. Context is not storage. It is a budget, and every passage you include competes with the passage that actually answers the question.

Chroma's study, run on 2025-era models, points the same way. Distractors hurt performance, unevenly, and how similar the needle was to the question changed the results. That hints the worst filler is not random noise but the near miss: the chunk that looks relevant and isn't. Treat that as a hypothesis for current models.

Four levers: selection, order, length, timing

The first lever is selection. Choosing the top three of thirty chunks is a ranking problem and a retrieval-precision decision. If your ranker puts a plausible wrong chunk in the top three, no instruction will rescue the answer. That is the bet worth testing: the choice of chunks is the variable to tune, not the wording. No published measurement of that contrast turned up, so run it on your own task and model. In practice that means measuring retrieval precision as a first-class number, and cutting at a hard count instead of a loose similarity threshold.

The second lever is order. In Lost in the Middle, models used evidence at the start or end of the input more reliably than evidence buried in the middle. That was measured on older models, so the rule is a hypothesis for yours. Put the question-critical material at an edge and the question itself near the end. Then test on your model by moving the key passage through the positions and watching the score.

The third lever is length, and the tool for it is compression. Anthropic's multi-agent pattern, as reported in a summary, lets a subagent burn tens of thousands of tokens on a search and hand back a condensed summary of roughly one to two thousand. The parent agent never sees the forty pages of dead ends. It sees the conclusion. The work is isolated, and only its result enters the window you care about.

The fourth lever is when the data enters. Anthropic describes a just-in-time strategy. Instead of loading every document up front, the agent holds lightweight identifiers: file paths, stored queries, links. It pulls the real data in at runtime, through tools. Claude Code does this with large databases, and it looks at the head and the tail of an object rather than loading the whole thing. Anthropic also notes that Claude Code caps tool responses at twenty-five thousand tokens by default.

The agent learns what exists, then opens only what it needs, then opens the next thing. That is progressive disclosure. The window stays small because the reference is cheap and the load is deliberate.

Selection, order, compression and deferral all decide what earns a place. None of them asks how big the window is. A million tokens only raises the ceiling on how much you can get wrong.

Tool descriptions are context too

Documents are only part of what the model sees. Before your first retrieved chunk arrives, the model has already read your tool definitions. Names, descriptions, parameter docs: all of it is loaded into the window, and all of it steers the agent.

Anthropic's engineering team calls prompt-engineering your tool descriptions one of the most effective ways to improve tools, and its advice is plain. Write the description as if you were onboarding a new hire. Make the implicit explicit: the query format you assume, the niche terminology your team uses, the meaning of an ID.

A tool described as "searches tickets" leaves the model guessing. Searches by what? Does it return titles or full threads? Is the date a string or a range? Claude's own docs draw this line. A good description says what the tool does, when to use it, what data it returns, and what each parameter means. The poor one is too brief, and the gaps are exactly those four points. The model fills them with guesses, and you see the guesses as wrong tool calls.

The other direction matters as well. A tool's response is a selection problem too. If a tool returns every matching record, you have rebuilt the dump-everything reflex inside a function call. Anthropic suggests combining pagination, range selection, filtering and truncation, with sensible defaults, so the first response is small and the agent asks for more on purpose. The Claude Code cap mentioned above is the same idea, enforced by the harness.

Output schemas and examples

An output schema constrains the space the model can write into. Fields, types, allowed values: each one removes a whole family of wrong answers before generation starts.

Examples do similar work, with one warning. Anthropic's guidance favours a few diverse, canonical examples over a pile of edge cases. Three examples that cover the typical shapes pin the format. Twenty edge cases teach the model that the edge cases are the norm, and they cost attention every time.

The ground moves, and prose still has a job

A caution about how fast this ground moves: on Claude Sonnet 5.5, forced tool use now returns an error. The switch many of us used to make a model call a particular tool is gone on that model. Steering tool choice now leans on what you wrote in the descriptions and what else is in context. Your prompt didn't change. The harness under it did.

Prose still has a real job in this window, just a different one. OpenAI's prompting guidance for GPT-6 Astra tells the model to infer intent from context and show a bias towards action. A phrase like "can you" or "help me" is to be treated as a request to act, not an invitation to ask a follow-up. That is a behaviour spec. It is not a persona and not a magic phrase. It states a policy the model cannot read off the documents.

So the window is more than documents. It is tools, schema, examples and a few lines of policy, all competing for the same attention budget. Once all of that is in place and the model is updated underneath you, how do you know it still works?

What this costs, and when to just paste it in

Selection is code, and code has bugs. An aggressive retriever or summariser can drop the one fact the answer needs, and the final answer will still read fluently. That is why the selection step needs its own evals, separate from the output. Did the needed chunk survive into the window? Scoring only the final answer hides exactly this failure.

Price is a selection variable too. Anthropic lists Claude Sonnet 5.5 at two dollars per million input tokens, against twenty cents for cache reads. GPT-6.1 Sol's cached input is reported at ten cents per million, though that comes from a secondary source, so check the pricing page.

Either way, the layout has a cost. Use a stable prefix with a volatile tail. Tools, schema and examples go first, where the cache can reuse them. The retrieved chunks that change on every request go last.

The savings come from tokens used, not wording. Anthropic credits Sonnet 5.5 with needing far fewer tokens to do the same work, for up to thirty percent lower cost per task than Sonnet 5.

So when is plain stuffing fine? When the corpus is small. When the analysis is a one-off. When the whole input sits far below the degradation and billing cliffs, and the reported 272,000-token cliff on GPT-6.1 Sol is one of them. In those cases, curation is the wrong investment. Paste it in and move on.

One more limit, on me as much as anyone: the evidence is thin. The long-context degradation studies predate the 2026 models, so no rule of thumb here has been confirmed on the models you are actually running. Measure your own model and your own task. The practical question is what that measurement looks like, and how you keep it running when the model changes underneath you.

Prompt, context and harness: three nested layers

Practitioners now split this work into three layers. The prompt is the wording. The context is everything the model sees. The harness is the tools, memory, constraints and feedback loops around the model. These layers are nested, not competing: the prompt sits inside the context, and the harness assembles the context.

The term context engineering was popularised in June 2025 by Andrej Karpathy and Tobi Lutke. It now appears in Anthropic's official engineering guidance, as the natural progression of prompt engineering.

Look at what agents in production actually run on. Retrieval decides which documents arrive. Tool design shapes what the agent can do. Long history gets compacted. Subagents hand back a summary of a thousand or two thousand tokens instead of forty pages of dead ends. The wording of the top-level instruction is the smallest part of that. The skill used to be writing the question. Now it is assembling everything around it.

Treat that assembly as code. Version it, so you can say which retriever, which tool descriptions and which schema produced which result. Log exactly what reached the window on each request, not what you meant to send. And evaluate it: the selection step and the final answer, on your model, and again whenever the model changes.

Persona and magic-phrase tricks don't reliably improve accuracy, according to secondhand reports of persona studies. Behaviour is still specified in prose, as OpenAI's GPT-6 Astra guide shows. The window still does not fill itself. At a million tokens, deciding what goes in is harder, not easier. And that decision is still your job.

Sources & further reading — Prompt Tricks Died. Context Engineering Didn't

Claude Sonnet 5.5 - Claude Platform Docs (Anthropic, 2026) — https://platform.claude.com/docs/en/models/sonnet-5-5/overview Sonnet 5.5's one-million-token window, adaptive thinking on by default replacing manual extended thinking, forced tool use returning an error, and the $2 per million input and $0.20 cache-read pricing.
Introducing Claude Sonnet 5.5 (Anthropic, 2026) — https://www.anthropic.com/claude-sonnet-5-5 Sonnet 5.5's release timing and the claim that it needs fewer tokens for the same work, cutting cost per task by up to 30 percent versus Sonnet 5.
Gemini 3.8 Flash (Google AI for Developers, 2026) — https://ai.google.dev/gemini-api/docs/models/gemini-3.8-flash Gemini 3.8 Flash accepting 1,048,576 input tokens.
Effective context engineering for AI agents (Anthropic Engineering) — https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents Context engineering as the natural progression of prompt engineering, context as an attention budget, subagents returning condensed summaries, just-in-time retrieval via lightweight identifiers, and Claude Code's 25,000-token tool-response cap.
Writing effective tools for AI agents—using AI agents (Anthropic Engineering, 2025) — https://www.anthropic.com/engineering/writing-tools-for-agents Prompt-engineering tool descriptions like onboarding a new hire, making implicit conventions explicit, and returning small responses via pagination, filtering, range selection and truncation.
Define tools - Claude Platform Docs (Anthropic) — https://platform.claude.com/docs/en/agents-and-tools/tool-use/define-tools A good tool description says what the tool does, when to use it, what it returns and what each parameter means; too-brief descriptions leave gaps the model guesses at.
OpenAI shares prompting tips for GPT-6 Astra including a blocklist of slop words (The Decoder, 2026) — https://the-decoder.com/openai-shares-prompting-tips-for-gpt-6-astra-including-a-blocklist-of-slop-words/ OpenAI's GPT-6 Astra prompting guidance (secondary report): infer intent from context and bias toward action, as an example of prose acting as a behaviour spec.
GPT-6.1 Sol: Features, Benchmarks, Pricing, and Access (DataCamp, 2026) — https://www.datacamp.com/blog/gpt-6-1-sol Reported GPT-6.1 Sol figures (secondary source, to be checked against the pricing page): roughly 1,050,000-token window, double input rate above 272,000 tokens, and cached input pricing.

Disclosure: this article is the written companion to the video above. Its script was drafted with AI assistance and the narration is an AI voice.

Why Does Your Local Model Crash at 32k Tokens?

Eryk Kubiak — Fri, 25 Sep 2026 21:08:00 +0000

In this video:

0:00 The Crash Nobody Can Explain
0:18 It Loads, It Answers... Then Dies
1:36 Just Match Weights to VRAM
2:44 OOM at 32k Tokens Anyway
4:32 Weights vs KV Cache, the Real Math
9:00 The 4-Bit Quality Cliff
11:15 Why Hosted APIs Never See This

You checked the model size against your VRAM, confirmed it fit, and watched it run fine. So why did it crash forty minutes into a long conversation? The weights never grew. Something else did.

Most guides budget one number: the model's weight size. They ignore a second, independent one. That second number is the KV cache, and it grows with every token you generate.

It Loads, It Answers, Then It Dies

You load a 7B model, check the GPU memory, and it fits with room to spare. You ask it a quick question, it answers in a second, and everything looks fine. So you keep going.

Twenty minutes and a few dozen turns later, on a straightforward stack that allocates memory as the conversation grows, the same model, on the same GPU, with the same settings, throws an out-of-memory error and dies. Nothing about the model changed. Nothing about the prompt content changed either — you didn't switch tasks, didn't load a bigger file, didn't touch a config. The only thing that changed was length: how long the conversation had gotten, how many tokens had piled up in the context window.

That's what makes it feel random. If a 7B model was going to run out of memory, you'd expect it to fail immediately, on the first prompt, not four thousand tokens into a conversation that started just fine.

Most developers budget memory once, at load time: weights plus a bit of overhead, done. On a straightforward local setup, that number is fixed the moment the model loads and it never moves again. But there's a second number growing quietly the entire time you're talking to the model, turn after turn, token after token — and it isn't fixed at all. It gets bigger with every single word the model reads or writes. Nobody sized for it because nobody named it.

The crash isn't a bug in the model, and it isn't bad luck. It's the arithmetic of a number that was never on the budget in the first place, finally catching up to the hardware.

Just Check the Model Size Against Your VRAM

The standard advice sounds reasonable, and it isn't wrong, as far as it goes. Weights in fp16 cost roughly two bytes per parameter, so a 7B model comes out to about fourteen gigabytes. Quantize down to 4-bit and each parameter shrinks to roughly half a byte, so that same model now costs three and a half to four gigabytes, depending on overhead — small enough to sit comfortably on a consumer GPU with plenty of room left over.

Plenty of guides stop right there, and the checklist feels complete. Pick a model, pick a quantization level, check that the resulting file is smaller than your VRAM, and load it. One number, one comparison, done. That's why the crash from the last section feels so unfair: the file fit, the math checked out, everything on the label was correct.

The problem is that this whole exercise treats VRAM budgeting as a single question — does the weight file fit on the card — as if the weights were the only thing that ever occupies memory during a conversation. They're not. They're just the only thing that gets budgeted, because they're the only number that's fixed and printed on the model card before you ever run it.

What's missing isn't a rounding error. It's an entire second consumer of memory that doesn't show up in a file size, doesn't get quantized the same way, and doesn't stay still once the conversation starts.

It Still Runs Out of Memory at 32k Tokens

So you follow the checklist. You quantize the 7B model down to 4-bit, watch it settle into about four gigabytes, and load it onto a GPU with twelve or sixteen to spare. It runs. It answers fast. Everything the guides promised checks out.

Then you keep the conversation going, push past ten thousand tokens, then twenty, then somewhere in the thirty thousand range — and the same model, on the same card, with room to spare at load time, dies with an out-of-memory error anyway.

That's the part that doesn't fit the story. The weights were four gigabytes. They're still four gigabytes. Loading a model onto a GPU is a one-time allocation: the file gets read, the parameters get placed in memory, and that number is locked in the moment the model finishes loading. It does not creep upward while you chat. It does not know or care whether you've sent ten messages or ten thousand.

So if the crash is real, and it is, the weights cannot be what's causing it. They're the one part of this system that was never going to move. Which means something else in that twelve or sixteen gigabytes of VRAM is filling up while you talk, and it's filling up in a way that quantizing the weights never touched.

Whatever this is, it gets allocated per token, not per file on disk, and what each token costs in memory is set by the model's architecture, not by how small you squeezed the weights. It grows with conversation length, turn after turn, the same way a log file grows with every line appended to it. A short exchange barely touches it. A long one, the kind that stretches into tens of thousands of tokens, apparently grows it large enough to exhaust memory that the weights themselves left untouched.

The model card never mentions it. The quantization guide never budgets for it. And yet it's sitting in the same VRAM, competing for the same bytes, quietly getting bigger with every single token you send and receive.

Two Numbers: Weight Memory and the KV Cache

There are two numbers here, not one, and they scale in completely different ways.

The first is weight memory, and it's simple: bytes per parameter times parameter count. In fp16, every weight is stored as sixteen bits, two bytes, so a seven billion parameter model costs seven billion times two bytes, fourteen gigabytes. Four-bit quantization doesn't just round that number down for fun — it maps each weight onto a much smaller range of possible values, roughly sixteen levels instead of sixty-five thousand, and accepts the rounding error that comes with representing a weight that coarsely. The payoff is that each parameter now costs about half a byte instead of two, so the same seven billion parameters shrink to three and a half to four gigabytes, depending on overhead. That's the whole mechanism. It's a fixed multiplication, decided once, at load time, and it never moves again for the rest of the session.

The second number is the KV cache, and it doesn't work like a file size at all — it works like a running total. Every time the model processes a token, each attention layer computes a key vector and a value vector for that token, for each of its key-value heads. Those vectors get stored, not because the model might need them again in some vague sense, but because every future token has to attend back to every past token, and recomputing the entire sequence from scratch at each new step would be absurd. So the key and value for token one thousand get computed exactly once, cached, and then reused unchanged for token one thousand one, one thousand two, and every token after that, for as long as the conversation continues. That reuse is what makes generation fast. It's also what makes the cache grow forever.

The size of that cache is a straightforward product: number of layers, times the number of key-value heads, which in many recent models is only a fraction of the attention heads thanks to grouped-query attention, times the dimension of each head, times how many tokens are currently in context, times the bytes used per stored value, times two, because you're storing both a key and a value for each token. Every term in that product except context length is fixed by the model's architecture the moment it's loaded. Context length is the one term that isn't fixed — it's whatever the conversation has reached so far.

Which means the KV cache scales linearly with how many tokens you've fed the model and how many it's generated, layer by layer, head by head, one pair of vectors per token, forever accumulating until the conversation ends or the memory runs out first. That is the standard case for full-attention models; some current models cap the cache with a sliding window instead, which bounds the growth but does not remove it.

At short context lengths this is easy to miss, because the numbers are small next to the weights. A few hundred tokens into a chat, the cache might be a few hundred megabytes, a rounding error against four gigabytes of weights or fourteen. But push that same model to thirty-two thousand tokens of context, and the arithmetic turns on the architecture: an older seven billion parameter model with full multi-head attention carries around seventeen gigabytes of cache at that length, several times its quantized weights, while a newer one using grouped-query attention holds it to a few gigabytes, still growing, just on a gentler slope. Same model, same GPU, same quantization setting — but a completely different memory profile, because one of the two numbers held still and the other one didn't.

And here's the detail that breaks the checklist from before: quantizing the weights does absolutely nothing to this second number, unless the KV cache is separately, explicitly quantized in its own right. Dropping the weights to four bits changes the multiplication in the first budget. It has no term in the second one. Keys and values are typically still stored in fp16 or something close to it regardless of what precision the weights were loaded at, because the cache is a runtime data structure, not a file on disk — nobody quantized it, because nobody budgeted for it in the first place.

So a model that was shrunk down to a tidy four gigabytes to make it fit on a modest GPU can still exhaust that GPU's memory purely through context growth, with the weight file sitting there completely unchanged, exactly as small as it was the moment it loaded. The crash was never about the weights being too big. It was about a second number that was linear, unquantized, and growing the entire time, layer after layer, token after token, while every guide kept its eyes on the first one.

The Quality Curve Has a Cliff, Not a Slope

Quantization does not just save memory on the weights: it also shrinks the bytes-per-value term in the key and value cache, if you let it, which means the same lever that fixed the loading problem also helps with the growing one.

That raises an obvious question. If dropping precision buys back memory on both fronts, why not just keep dropping it? The answer is that precision and correctness are not on the same curve, and the curve has a cliff in it, not a slope.

Take a fourteen billion parameter model and quantize it down to four bits, and in most published comparisons you get something that still beats a seven billion parameter model running at full sixteen-bit precision, with how decisively depending on the model family and on how carefully the quantization was done, all while the four-bit version takes up less memory than the fp16 model it's outperforming. Bigger and compressed usually wins over smaller and pristine, because a model's capacity comes from how many parameters it has and what they learned, not from how many bits are used to store each one.

Four bits per weight is not the sign of a broken model. Down around that level, errors introduced by rounding each weight into one of sixteen possible values are still small enough, and spread out enough across billions of parameters, that the output degrades gracefully: slightly less precise word choices, an occasional weaker reasoning step, nothing that reads as damage.

But that graceful decline does not continue indefinitely. Push below roughly four bits per weight — down to three, or two — and something different starts happening. It is not incrementally worse text. It is text that repeats the same phrase in a loop, or drifts into words that do not form real sentences, or answers a question with something that is grammatically present but semantically empty. That is not a shallower slope. It is a cliff, and once you are over the edge, adding more compute or a longer prompt does not pull you back.

Where exactly the edge sits depends on how the quantization was done: naive round-to-nearest falls off it early, calibrated methods push it lower, but the same cliff shows up eventually. So the memory budget you have been building, weights plus a growing cache, has a floor built into it: a point past which shrinking the bit-width further stops trading a little accuracy for headroom and starts trading the model's ability to produce language at all.

Why Hosted APIs Don't Have This Problem

None of this shows up when you're calling a hosted API, and that's worth explaining rather than waving away.

When you send a prompt to a hosted model, someone else's GPU is holding the weights, someone else's GPU is holding the growing key and value cache, and someone else is deciding how many other requests get packed onto that same card at the same time. The provider is running thousands of sessions across a fleet of machines, batching them, evicting them, moving them around, all so that no single conversation's cache growth becomes your problem to manage.

You never see a VRAM number because you were never allocated a fixed slice of VRAM in the first place. What you're actually charged for is tokens: so many cents per thousand tokens in, so many cents per thousand tokens out. That's most of the meter, give or take discounts for cached prompts and batched requests. Memory pressure on the provider's side gets absorbed into that price somehow, folded into margins and capacity planning you don't have visibility into, but from where you sit it never appears as an out-of-memory error. It appears as a bill.

Which is why the two worlds optimize for completely different things. Local deployment revolves around bit-width and context budgeting, because memory is the wall you hit. Hosted deployment revolves around prompt length and pricing tiers, because dollars are the wall you hit — a long conversation doesn't crash a hosted session, it just costs more, quietly, request after request. There is still a hard edge: every hosted model caps how many tokens one conversation can carry, but crossing it gets you an explicit error telling you to trim the prompt yourself, not a dead process.

Neither constraint is fake, and neither one is more sophisticated than the other. They're just different currencies for the same underlying growth: every token you push through a model has to be stored somewhere, by someone, at some cost. Running a model yourself means that cost lands on your GPU as memory, term by term, layer by layer. Renting one means that cost lands on your invoice as price.

Choosing local versus hosted was never really a choice about which model to run. It's a choice about which constraint you'd rather be responsible for tracking.

Sources & further reading — Why Does Your Local Model Crash at 32k Tokens?

Attention Is All You Need (Vaswani et al., 2017) — https://arxiv.org/abs/1706.03762 Explains the attention mechanism and why key/value vectors for past tokens must be stored and reused for every future token (the basis of the KV cache).
GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints (Ainslie et al., 2023) — https://arxiv.org/abs/2305.13245 Grounds the claim that grouped-query attention reduces the number of key-value heads and therefore slows KV cache growth compared to full multi-head attention.
The case for 4-bit precision: k-bit inference scaling laws (Tim Dettmers, Luke Zettlemoyer, 2023) — https://arxiv.org/abs/2212.09720 Supports the claim that 4-bit quantized larger models can outperform smaller fp16 models, and that quality degrades gracefully down to ~4 bits before a sharper drop at lower bit-widths.
Quantization overview — Hugging Face Transformers documentation — https://huggingface.co/docs/transformers/main/en/quantization/overview Explains how weight quantization (e.g., 4-bit) reduces per-parameter storage and how it differs from quantizing runtime structures like the KV cache.
llama.cpp (Georgi Gerganov) — https://github.com/ggerganov/llama.cpp Canonical reference implementation for local model loading, quantized weight formats, and KV cache handling described throughout the video.
Optimizing your LLM in production (Hugging Face) — https://huggingface.co/blog/optimize-llm Explains the KV cache memory formula (layers × heads × head dimension × sequence length × precision) and why it grows linearly with context length.
OpenAI API Pricing — https://openai.com/api/pricing Backs the claim that hosted APIs charge per token rather than exposing VRAM limits, shifting the constraint from memory to cost.
Attention is all you need (Wikipedia overview of Transformer architecture) — https://en.wikipedia.org/wiki/Transformer_(deep_learning_architecture) General background on transformer architecture terms (layers, heads, head dimension) referenced when describing the KV cache formula.

Disclosure: this article is the written companion to the video above. Its script was drafted with AI assistance and the narration is an AI voice.