...a controlled study just showed 41% of agent cost is design, not model
Thesis: arXiv 2607.06906 held models and tasks constant, varied only the harness, and found 38% fewer tokens and 41% lower blended cost. The post walks the three harness levers the paper implicates: retrieval strategy, tool-surface size, prefix stability, and shows each is independently measurable, and demonstrates per-session instrumentation for measuring your own.
- The experiment: 22 tasks, six models, harness-only variation, 38%/41% at level quality. Why "held constant" matters; this isn't a vendor benchmark.
- Lever 1, retrieval: full-file reads vs grounded symbolic retrieval — the 99.6% aggregate (5.12M → 19.4K, 15 task-runs, tiktoken cl100k_base, CI replay-gated), and what varies by repo shape.
- Lever 2, tool surface: companion paper arXiv 2607.15593 on tool-surface bloat; our measurement of 93 resident schemas ≈21.6K tokens/turn vs a 6-tool on-demand front door ≈0.9K (dated CHANGELOG receipt).
- Lever 3, prefix stability: arXiv 2607.15516 (51.7% cost reduction at three reuses on a 94K-token prefix); our deterministic-emission work and the honest part — we measured 0.139 against a 0.30 threshold and held a feature because the problem was small.
- Measuring yourself: token yield rate (credited to FinOps Foundation), the session yield block, advisory budgets, the tool-surface receipt — shipped July 19 and 21, all three servers.
- Close: agent cost is a distribution (per McKinsey) and the harness sets the distribution. Instrument first, optimize second.
They're preaching to our well-established choir of over 60,000 developers worldwide. And I'd strongly urge everybody else to pick up a hymnal...
Top comments (0)