I've been building an open-source tool that tests whether self-hosted models can actually drive agents. Running it against my own setup produced a result I didn't expect, and I think it generalizes.
Disclosure up front: I build QuantaMind (Apache 2.0, runs offline, no telemetry). The data below came out of it. This post is about the finding, not the tool.
Setup
- Qwen3.5-9B, Q4_K_M
- llama.cpp, native function calling
- 16GB M-series Mac
- 5 multi-step agentic tasks, k=4 runs each
k=4 matters. Scoring an agent on a single successful run tells you almost nothing — what you care about is whether it succeeds every time, so a task only counts as passed if all four runs pass.
Aggregate: 80% pass rate (16/20 runs), avg 4 steps per task, ~2,000 tokens per task.
The context cliff
I padded prompts with semantically unrelated prose, inserted before the tool definitions, and re-measured tool-call accuracy at increasing depth.
| Prompt depth | Accuracy |
|---|---|
| 704 tok | 100% (5/5) |
| 2,999 tok | 93.3% (14/15) |
| 6,045 tok | 93.3% (14/15) |
| 8,845 tok | 73.3% (11/15) |
Flat through roughly 6k. Then a 20-point drop by 8.8k.
Why this isn't a memory problem
This is the part I want to argue for, because it contradicts how most people size local agents.
- Model weights: 5.3GB
- GPU-addressable memory: ~11.8GB of 16GB, under the macOS Metal cap
- Context capacity at f16 KV cache: ~53,000 tokens (q8_0 ~107k, q4_0 ~214k)
- Peak context actually used during the agent runs: 1,890 tokens — 12% of the 16,384 window I launched with
So memory could hold roughly five times more context than the model can reason over reliably. The VRAM calculator told me I had room. The model fell over well before I got near it.
The practical implication: "how much context fits" is not the number that determines how much context your agent can use. Those are two different ceilings, and the behavioral one is much lower and much harder to see. It doesn't announce itself — accuracy just quietly degrades.
The failure that scared me more
Four of five tasks passed all four runs. One failed all four.
The failing task was an incident-rollback chain: get_incident → get_feature_flag → flag_off → rollback_release → schedule_fix. Every run failed identically — the model emitted a completion signal partway through the chain and stopped.
No crash. No malformed JSON; schema was clean across the entire batch. No refusal. It announced it was finished and it wasn't.
At k=1 you'd log that as a flaky miss and move on. At k=4 it's obviously structural — the model isn't unlucky, it has a consistent wrong belief about when the task is complete.
This is the failure mode I'd worry about most in production, precisely because nothing errors. Your orchestrator sees a successful completion and advances to the next stage with half the state missing. Every monitoring layer you have will report green.
Most eval tooling scores pass/fail. Pass/fail can't distinguish "the model crashed" from "the model lied about finishing," and those need completely different fixes.
Latency, briefly
The full 5-task batch took 39m 10s wall. The worst single task was 16m 16s across 4 runs — of which 14m 51s was decode and 1m 19s was prefill.
If you're optimizing local agent loops, you're optimizing decode. Prompt caching and prefill tricks barely move the number.
What I can't answer
I have one machine and one model family. Open questions:
- Is the ~9k cliff specific to this quant, or general to the 8–10B tier?
- Does more VRAM headroom move it? My guess is no — if the cliff is attention, not memory, a 24GB card shouldn't change anything. I can't test that.
- Does the cliff move with padding type? I used unrelated prose. Structured JSON padding might behave differently.
If you run something similar, I'd like the number.
Reproducing
GitHub: github.com/QuantaMinds/QuantaMind — Apache 2.0, fully local, no account, no telemetry.
qm cliff --backend llama_cpp --model <model> \
--collection medium-coding-v2 --max-tokens 12288 \
--steps 5 --source corporate_policy --mode native
Scoring is deterministic — correct tool plus correct arguments, checked against a fixed answer key. No LLM judge, because using a model to grade a model's tool calls introduces exactly the kind of error this test is trying to measure.
If the methodology is wrong, I'd rather hear it than not.
Links:
Website: https://www.quantamind.co/
Discord: https://discord.com/invite/6CjSJyZTfG
X: https://x.com/QuantaMind_2025
Top comments (0)