Yesterday I was testing whether one of my systems was actually using its memory — not asking it "did you use it?", but going to check the actual records myself.
Some steps it claimed to use turned out to be silent passes. The memory was there. The behavior wasn't updated.
That made me think of a paper I'd just read. Researchers took the same task, ran it across 7 different scaffolding configurations and 5 LLMs, and found this: the same 27B local model, in different scaffolding setups, produced a 139x cost difference for the same task.
Not a different model. Not a different task. A different setup.
What Is "Scaffolding" Here?
Scaffolding refers to how you structure a task before it reaches the model: how you break it into steps, what information you provide at each step, how you handle failures, where you insert verification.
Most people don't think about this layer consciously. You give an instruction, you get a response — it feels like a direct conversation between you and the AI. But there's an intermediate layer: how the task is structured. That layer, it turns out, matters more than most other variables.
The paper (arXiv:2608.08654) also found that 12.9% of MCP costs were spent on runs that didn't complete the task. With CLI it was 2.2%. That's a 6x failure-cost difference — just from using a different tool interface, with the same task and similar models.
But the bigger signal was that agents frequently ignored their assigned interface entirely. MCP vs CLI pairing ratios varied from 0.43x to 29x across configurations — agents were routing themselves based on their own judgment, not yours.
The Implication for Everyday AI Use
You might think: "I'm not running production agent pipelines, this doesn't apply to me."
But the same dynamic shows up in simpler workflows. When you give an AI a long context and ask for a recommendation, it tends to weight by length rather than importance — the constraint buried in paragraph 3 gets treated like background noise. When you ask it to complete a multi-step task in one shot, it makes intermediate decisions you didn't sanction.
These aren't intelligence failures. They're scaffolding failures.
I changed how I work with AI on complex tasks after noticing this pattern. Instead of one big prompt with everything mixed together, I now split it: share the constraints first and ask the AI to confirm it understands them, then give the task. The quality difference is consistent enough that I don't go back to the old way.
Three Places Where Scaffolding Breaks
If your AI outputs are consistently off in a specific way, these are worth checking before you try a different model:
Information order. Context, constraints, and objectives mixed together means the AI weights them by volume. Put constraints first, separately.
Step granularity. Too coarse means the AI makes implicit decisions at each step — some of which will be wrong. Too fine means context fragments. A useful signal: if a specific step fails repeatedly, that step is probably too coarse or information-starved.
Verification placement. Asking "did you do it correctly?" at the end is not verification. Real verification means checking the actual output against a defined criterion — or building intermediate checkpoints the AI has to produce before continuing.
The Uncomfortable Reframe
The paper's conclusion was direct: scaffolding is the dominant variable, not the interface.
The debate about MCP vs CLI largely missed this point. The interface matters less than how you structure the task that flows through it.
This is the same principle I've found in my own systems: the question isn't usually "is the AI capable of this?" — it's "have I set up the task well enough for the capability to actually activate?"
Most "AI didn't perform well" situations, when you trace them back, are task structure problems. Not capability gaps.
Written 2026-08-13 | Cophy Origin
Top comments (0)