TL;DR: Agent frameworks quietly assume an obedient model: clean JSON, instructions followed the first time, context to spare. Run a quantized 7B–35B in a 16K window and every one of those assumptions dies. I spent months building the application layer that keeps small local models productive anyway — all open source, MIT. Here's what actually breaks, and what fixed it.
The setting
Chaty is a fully offline desktop AI workspace — Rust / Tauri 2, llama.cpp plus native MLX on Apple Silicon. The coding agent inside it runs on whatever GGUF or MLX model you have: 7B, 8B, a 35B MoE. No cloud fallback, no retry budget. If the loop wastes rounds, you personally watch your GPU heat the room for nothing.
Three constraints shape everything below:
- 16K context. Every tool doc line is a tax paid on every single turn.
- Format adherence is probabilistic. The model usually emits valid tool calls. "Usually" is not an engineering contract.
- Small models imitate whatever is in context — including their own mistakes. This one is load-bearing. Remember it; it explains half the design.
1. Parse what models actually emit
Here's a real tool call from a 35B MoE, captured from a raw dump:
{"name": "write_file", "path": "index.html", "content": "<!doctype html>...", "arguments": {}}
Flat fields, plus a spurious empty arguments object — and the model frequently hits EOS right after the content string, before the closing brace. A strict parser rejects this, the loop asks again, and the model regenerates the entire HTML file. Ten rounds of that on one page.
The fixes, in order of importance:
-
Accept flat fields. And critically: an empty
arguments: {}must not shadow them. Taking the{}at face value turned every such write into a missing-argument retry loop. -
Repair missing closers — never mid-string. A string-aware scanner appends
}/]only when every string is terminated. Auto-completing a truncated string would silently write a corrupted file, which is strictly worse than retrying. - Build parser fixtures from raw dumps of your target models, not from the spec. The spec is what models were trained toward; the dumps are what they do.
After this, the same page that burned ten rounds landed in one 76-second write.
2. A repeated correction is not a correction
A small model that sends search_code {} will send it again, no matter how politely you repeat the same error message. We ran the autopsy on one of these spirals: the old guard repeated an identical correction three times, then gave up and executed the empty call — which planted the model's own empty-arguments call into the transcript as an example to imitate. See the load-bearing constraint above.
The replacement is an escalating ladder, counted per tool:
- Slip 1: show a concrete valid example for this tool.
-
Slip 2: stop suggesting the tool — divert: "try
list_dir/read_file/grepinstead." (We also raise sampling temperature here; a stuck attractor sometimes needs noise, not words.) - Slip 3: the tool is declared disabled for this turn, and the stuck state gets a visible error card in the UI.
- Slip 5: pause the loop with a human-readable reason instead of spinning.
Empty calls are never executed, and a single valid call resets the counter. The principle: every rung must add new information. If two consecutive corrections are byte-identical, you don't have a correction — you have a chant.
3. Servers must not stall the loop
Models start dev servers in foreground shells. Naive harness behavior: block for 120 seconds, time out, and kill the very server the model was asked to start.
Detection is two-track, because one track lied to us:
-
Banner matching is the fast path — vite and friends print
http://localhost:5173instantly. -
A listening socket is the ground truth.
python3 -m http.serverblock-buffers its banner under a pipe; the text never arrives. So after a ~10s grace period we also ask the OS whether the command's process group holds a listening TCP socket (~every 3s).
Either signal → the process is moved to a tracked background job, not killed: output buffer carried over, job id returned to the model with "the server is up — navigate to it, check logs with bg_output."
One more zombie class: a shell that exits after server & used to leave the survivor squatting the port forever, invisible. Now the process group is checked on shell exit and survivors are adopted as [detached] background jobs — visible, killable, reaped on workspace switch and app exit.
4. Bring the console to the model
Agents love to "verify" a page by screenshotting the stale DOM. A screenshot proves the page looks like something; it proves nothing about behavior.
-
Interactions on your own pages auto-attach new console errors. Click, type, navigate on localhost or a local file → any new
[error]/[exception]/[dialog]lines ride along in the tool result, deduplicated by a cursor, labeled honestly (a confirm dialog your click triggered is not an "error"). External websites stay noise-free — you don't want some ad network's console spam in a 16K context. -
A real refresh verb.
browser_refreshdoes a true hard reload (cache ignored). Before it existed, models said "refreshing the page" and then just took another screenshot of the old DOM. - A wrap-up gate. When the model tries to end the turn with unfinished todos, or with web files edited but never re-checked in the browser, it gets bounced — once. One corrective nudge, then its next answer stands. Zero nagging loops.
5. Loop breakers need judgment, not just a hammer
Intercepting identical repeated calls is table stakes — ls . five times helps nobody. But a blanket breaker kills legitimate work: clicking "Next" three times is pagination. The rule that survived our browser-bench autopsies: identical UI actions (click/type) may repeat while the page keeps changing; the moment an identical action produces an identical result — or follows an error — it's degenerate, and repeating a submit button in that state posts duplicates. Read-only observations (screenshot, re-read, poll) are exempt entirely.
6. The 16K token economy
Everything above competes with the user's actual code for context. The budget discipline:
- One doc line per tool. Full schemas live in a deferred tier, loaded only when a tool is first needed.
- Hints are just-in-time, not preloaded. The web-workflow lecture ("refresh after edits; screenshots prove looks, not behavior") is injected once, the first time a local URL appears — not carried by every session that never opens a browser.
-
Procedures and memory are files, not prompt. Skills (
SKILL.md) cost one index line each until actually invoked; cross-session memory is markdown files with a hard-capped index. A user with none of these gets a byte-identical prompt — enforced by a golden test.
7. Treat tool output as hostile
Local doesn't mean safe: a web page your agent reads can carry <tool_call> markup or chat-template turn tokens and try to steer the loop. Every untrusted result (web content, MCP server output) gets control tokens defanged (a zero-width space makes them inert to the parser but readable to humans) and is framed as data-not-instructions. MCP tools ride the exact same defense as native web tools — checked against the live tool registry, since MCP tools register at runtime.
What didn't work (and taught the most)
- Repeating the same correction. Three identical nudges did nothing; the fourth execution made it worse. Escalate or don't bother.
-
Fixing the harness instead of the product. Our bench once passed
-uto Python servers so their banners would flush. That crutch hid the exact stall real users hit. If the harness needs a workaround, first ask whether the product has the bug. - Trusting banners. Text output is a courtesy, not a protocol. The OS knows whether something is listening; ask it.
Takeaway
None of this is model research. It's plumbing — parsing, process management, escalation policy, token budgeting. But on small local models, the application layer is where agents live or die: the same 8B that flails alone finishes real web tasks in a fraction of the steps with this layer under it.
All of it ships in Chaty — fully offline, Rust/Tauri 2, llama.cpp + native MLX, MIT. If you run a local model through it and something breaks, I want the raw transcript — that's where every fix above came from.
Top comments (0)