DEV Community: schultzbehrnt9-jpg

Your agent did not forget your instructions. It compressed them out.

schultzbehrnt9-jpg — Sat, 26 Sep 2026 01:30:07 +0000

Long agent runs rarely fail at a tool call. They fail at a compaction boundary, and almost nobody instruments it.

The failure

When the context window fills, something has to decide what to throw away. Every agent harness does this. Most do it by summarising.

If the goal statement is not preserved verbatim through that step, it gets summarised into vagueness. The model then reads back its own compacted notes, no longer knows precisely what it was doing, and quietly restarts the plan - frequently re-deriving work it already finished.

From outside this looks like "the model forgot my instructions." But they were not buried in the middle of a long context, which is the failure everyone knows about. They were compressed out.

Nothing errors. So it does not surface as a failure. It surfaces as a run that took three times as long and ended somewhere adjacent to what you asked for.

Why it is hard to see

Teams log tool calls, messages and decisions. Almost nobody logs the compaction event: the moment the window filled and something chose what to discard.

When you read the trace afterwards, every individual action looks locally reasonable. The agent made a series of defensible decisions. Nothing threw. You can explain every action and still not explain the run - which is precisely the thing traceability was supposed to buy you.

Three things that helped

Pin the goal verbatim. Not summarised, not paraphrased, not "the user wants X". The literal original text of the goal and the acceptance criteria, re-injected after every compaction. Single highest-leverage change I made.

Log the compaction event with what was dropped and what was kept. Then you can correlate a bad run with the exact point it lost the thread. Cheap to add, and it turns an unexplainable failure into an obvious one.

Denominate retry budgets in remaining context, not in attempts. Every failed retry consumes context, which pulls the next compaction forward. Two retries on a task that began at 60% context is a completely different risk from two retries on one that began at 5%. A loop that retries across a compaction boundary works from a worse specification each time - attempt 3 is dumber than attempt 1, which is the opposite of what a retry is for.

Multi-agent makes this worse, not better

The usual pitch for multi-agent is that decomposition solves context pressure: each agent holds less, so nobody fills their window. In my experience it relocates the problem rather than removing it, for two reasons.

Every handoff is a lossy compaction with extra steps. When agent A summarises its state for agent B, that is the same operation as a compaction - something decides what survives - except it now crosses an agent boundary, so B has no access to what A dropped and no way to know it is missing. A compaction at least leaves the raw history recoverable in principle. A handoff usually does not.

The boundaries do not line up. An agent's context can fill mid-subtask. So it compacts, loses fidelity on its own assignment, and then hands off a summary of a plan it had already partially forgotten. The orchestrator sees a completed handoff and a plausible summary. Nothing looks wrong at the supervision layer.

The net effect is that the group can be confidently, collectively wrong in a way a single agent with a full trace usually is not - because every participant's view is locally coherent.

If you take one thing

Compaction quality beats orchestration. Before adding another agent to the graph, check whether your existing one survives its own context boundary with the goal intact. Most do not, and that is a cheaper thing to fix.

Founder disclosure: I build Grunz, a coding agent on open-weight models, which is where all of this came from. Open models make it sharper because most hosted endpoints serve around 32K regardless of the advertised window, so compaction fires every few minutes instead of every few hours.

Abliterated models lose obedience before they lose knowledge

schultzbehrnt9-jpg — Sat, 26 Sep 2026 01:22:24 +0000

There are thousands of abliterated models on Hugging Face now. If you are evaluating one, the thing that degrades is probably not what you are testing for.

What abliteration does

Briefly: you identify a refusal direction in the model's activation space and project it out of the weights. No gradient steps, no training data. It is a weight edit, not a finetune.

The assumption most people carry into evaluation is that this is a capability tradeoff - you get compliance, you lose some general intelligence, and you test for that by checking whether the model still knows things and still writes well.

That is not where the damage shows up.

The first thing to go is obedience, not knowledge

Across variants and across model families, what degrades first is instruction-following and output-format adherence.

The model still knows the material. What gets measurably worse:

respecting the instruct template and prefill
honouring stop sequences
staying inside a structured-output contract (JSON schema, tool-call syntax)
holding a system-prompt constraint across a long context

So the model that scores within noise of its base on MMLU will fail structured output at a materially higher rate. It knows what to do. It has become worse at doing what it was told.

Why your evaluation misses it

The standard way people assess an abliterated variant is: chat with it, check it does not refuse, check the prose is good. That test cannot detect this. The model looks fine. Often better than fine, because the refusal behaviour that used to interrupt you is gone.

Then you wire it into something that parses its output and the failure rate climbs.

Perplexity does not help either. On a generic corpus it barely moves while format compliance falls off a cliff. The metric says the model is fine and the application says it is not.

What to measure instead

Test format compliance separately from quality, and score it independently of whether the answer is correct.

The shape that works:

issue N requests that specify an exact output contract
score binary compliance with the contract, not correctness of the content
report a compliance rate you can compare across variants and across quants

The key property is independence from correctness. A model returning a well-formed wrong answer scores 1.0. A model returning a correct answer wrapped in prose that breaks the schema scores 0. That is the number you actually care about when something downstream is parsing.

This will separate two variants that look identical in a chat window.

The quantization interaction

A hunch I have not measured properly, offered as a hunch: the damage appears to compound with quantization damage rather than sit alongside it. An abliterated model seems to degrade faster down a quant ladder than its base does, measured on format adherence.

If that is right, the mechanism would be that abliteration has already flattened some weight structure that quantization then rounds away. Which would mean staying at higher quants is worth more on an ablated model than on a clean one - I have had better luck at q6_K and above on smaller models, and a q8_0 on a 4B is only around 4.3GB, so the saving from going lower is small next to what it seems to cost.

If you want to test it properly: quantize a base model and its abliterated counterpart to the same bpw with the same calibration set, then measure compliance rate rather than perplexity.

Practical implications if you are building on these

Your tool-call parser needs to be more forgiving than it does against a frontier model. A strict parser turns a recoverable formatting wobble into a failed run.

Grammar-constrained decoding is doing more load-bearing work here than it does on clean models, because it forecloses malformed paths at the token level rather than relying on the model to want to follow the schema.

Track compliance rate as a first-class metric, not something you infer from task success. When a run fails you want to know whether the model was wrong or merely malformed. Those have different fixes.

Founder disclosure: I build Grunz, which serves these models, so I learned this from production rather than from a paper. If anyone has measured the quant interaction properly I would rather cite a real result than keep repeating a hunch.

Your LLM provider is probably serving you 32K context no matter what the model card says

schultzbehrnt9-jpg — Sat, 26 Sep 2026 01:20:13 +0000

I run a hosted chat and coding agent on open-weight models. This is the single finding that cost me the most time in the last few months, and almost nobody talks about it.

The number on the model card is not the number you get

The context window advertised on a model card is a property of the weights. The context window you actually receive is a property of whoever is serving them.

Across the hosted endpoints I have tested, most serve around 32K regardless of what the card claims. A handful reach 256K. I have not yet found one actually serving the 1M figures that appear in release announcements.

This is entirely reasonable from the operator's side. Max context length is a KV-cache budget traded off against concurrency. Serving a million tokens per request to everyone would be ruinous. The problem is not that operators do this. The problem is that it is almost never documented and it fails silently.

Nothing errors

That is the part that makes it expensive. You do not get a 413. You do not get a warning. The request is served against whichever ceiling is lowest and the front of your context is simply gone.

For chat, this is close to invisible. The conversation gets slightly dumber over time and you assume that is just how it goes.

For anything agentic it is the binding constraint, and it fails in a way that does not look like a context failure:

a long build run hits the ceiling
compaction fires
the goal statement gets summarised into vagueness
the model reads back its own compacted notes, no longer knows precisely what it was doing, and quietly restarts the plan

From the outside that reads as "this model is bad at long tasks." It is the serving config.

How to find your real number

There is no clean way to do this, which is itself the story. Two methods that work:

Send a deliberately oversized prompt and read the error. Providers usually leak the true maximum in the error text even when the docs do not mention it. Push 500K tokens at an endpoint claiming 1M and see what comes back.

Watch where truncation starts biting. In a long run, find the point at which the earliest content stops influencing output, and count backwards. Uglier, but it works when the error text is unhelpful.

Provider documentation has not been reliable for this in my experience. Neither has the model card.

Three consequences worth internalising

1. A benchmark number is a score for a model at one served context length, not a score for a model. The same weights at 32K and at 200K are not the same agent. If you are comparing providers on cost per token without pinning the served ceiling, you are comparing two different things and calling the difference price.

2. If you use a gateway with fallbacks, the ceiling can change mid-session. A run that starts on a 200K provider and fails over to a 32K one does not error. It truncates. That is a correctness problem, not a performance one, and nothing in the response tells the caller it happened.

3. In RAG, this silently invalidates your retrieval tuning. Chunk size, top-k and reranking are all tuned against an assumed budget taken from the card. If you tuned for 128K and you are served 32K, you are over-retrieving, your reranked chunks get truncated, and the answer comes back confident and wrong. Then you go tune the embedding model, which was never the problem. Reranking in particular buys you nothing if the top results do not survive into the prompt.

What I would like to exist

A discoverable way to read the effective served context per request. Model metadata, a response header, anything. Right now every client reverse-engineers it from error strings, which is absurd for a number that determines whether your application works.

If you know of a provider that publishes this honestly, I would genuinely like to hear about it.

Founder disclosure: I build Grunz, a chat and coding agent running open-weight models, which is how I ended up learning all of this the expensive way.

i got tired of AI subscriptions, so i built a coding agent that runs obliterated open models

schultzbehrnt9-jpg — Fri, 25 Sep 2026 14:21:46 +0000

every ai coding tool i used had the same two problems: a monthly bill that never stops, and a usage cap or a refusal right when i actually needed the thing. so i built grunz. no subscription, no account, running on open weight models with the refusal training stripped out.

what it is

grunz is a native app for mac, windows and linux that works like claude code. you give it a goal and it plans, writes and edits files across the whole repo, runs commands, watches the output, fixes itself, and hands you the finished thing. it also does cited web research and makes images. one app instead of five tabs. there's a chat only version in the browser at grunzai.com/app if you just want to talk to the models.

the part that's different

no subscription, no seats, no account. a new device gets a small free balance so you can try it. because the models are open weight and abliterated, there are no weekly caps and no "i can't help with that."

under the hood

a tool loop with ~50 tools: read/write/edit files, shell, dev servers, a headless chrome browser, image generation, mcp servers, subagents
a plan mode that proposes before it touches anything
a permissions dial: ask before every change, auto accept edits, or full auto
a model picker

privacy

the wallet is anonymous and lives on your device. messages, prompts and files aren't stored.

if you've been looking for a claude code style agent that isn't a subscription, go look: https://grunzai.com

i'm the founder. i genuinely want feedback on the agent loop. roast it.