Truong Phung

Posted on Jul 25

🤖 Optimizing AI Agents: Token Economics 💰, the Harness & Context Engineering ⚙️

#ai #programming #webdev #tutorial

🤖 Optimizing AI Agents: Token Economics 💰, the Harness & Context Engineering ⚙️

A practical, no-fluff field guide to making agents cheaper, faster, and better at the same time — the way teams actually do it in 2026.

The big shift: the gains are no longer mostly in the model. They're in the harness (the orchestration layer around the model) and in context engineering (what tokens you let into the window). Get those two right and every model you run — present and future — gets cheaper.

Grounded in recent work: Writer — The Harness Effect: How Orchestration Design Sets the Token Economics of Enterprise Agentic AI (arXiv:2607.06906, Jul 2026), Anthropic — Effective context engineering for AI agents & Writing effective tools for agents, Chroma — Context Rot, Manus — Context Engineering Lessons, Epoch AI — LLM inference price trends, and the classic ReAct / Reflexion / MemGPT / SWE-agent line.

📋 Table of Contents

🧭 The one mental model
💸 Token maxing: the disease
🧮 The token bill, decomposed
🎛️ The harness: the price-setter
🧱 The six mechanisms that rewrite the bill
🧠 Context engineering: the demand side
🧰 Tool design for token efficiency
⚖️ Harness leverage & the capability floor
🚦 Routing, fleets & compounding savings
📊 Change the KPI: measure CPM, not just quality
🛠️ The practical playbook
🧰 The tooling landscape: what actually implements this
🎯 One-page cheat sheet

🧭 The one mental model

An agentic task is not one model call. A single request — "reconcile these two contracts and draft the redline" — unfolds into a dozen or more turns: system prompt, tool schemas, retrieval payloads, intermediate reasoning, tool outputs, and (in naive setups) a full replay of everything above on every subsequent turn.

The bill for the task is the sum over that loop — and the loop is governed not by the model but by the software around it: the harness.

🔑 The core claim (Writer, 2026): hold the tasks and the models constant and swap only the orchestration layer, and you cut cost per task −41%, latency −44%, and tokens per task −38% — with quality at parity. On that workload, the harness moved the bill more than switching from the most expensive model to the cheapest did.

⚠️ One caveat up front: these exact magnitudes come from a **single controlled study (n = 22 tasks, vendor-authored). Treat the **direction* as robust — it's independently corroborated by Anthropic, Manus, and Chroma below — and the precise percentages as indicative, not universal constants.*

Two levers, two disciplines:

Lever	Question it answers	Owned by
The harness	How many tokens get submitted, and at what price?	Your orchestration code
Context engineering	Which tokens are worth submitting at all?	Your curation strategy

Everything below is these two, in detail.

💸 Token maxing: the disease

Token maxing is the dominant (bad) pattern in agent development: buying capability with tokens — longer reasoning traces, more turns, wider tool payloads, bigger replayed contexts — so that tokens per task grow faster than task value.

Formally, a development trajectory exhibits token maxing when token intensity τ keeps rising while marginal quality per token falls:

τ(t+1) > τ(t)   while   [ Q(t+1) − Q(t) ] / [ τ(t+1) − τ(t) ]  <  Q(t) / τ(t)

i.e. each release buys quality at a worse token exchange rate than the system's running average.

Why it persists:

📉 Falling prices hide it. Per-token prices keep dropping — which finances the habit. Teams treat tokens as ~free at the margin and scale consumption to match. Per-task cost falls while total spend rises anyway. This is textbook Jevons paradox (efficiency in a resource lowers its price and raises total consumption), restated for tokens.
🏆 Benchmarks reward it. Token maxing is invisible in benchmark tables (which report quality) and painfully visible in cloud invoices (which report tokens). A team judged on quality alone will token-max, because tokens are someone else's line item.

🚨 The escape is not cheaper tokens — it's doing the same work with fewer tokens (a higher completions-per-million-tokens rate). And most of those tokens are set by code, not the model.

🧮 The token bill, decomposed

Let a task run as a k-turn loop. Turn i submits T_in(i) input tokens and emits T_out(i) output tokens. The cost is:

C = Σ_{i=1..k} ( p_in · T_in(i)  +  p_out · T_out(i) )

The input side is where the money is, and it decomposes into terms the harness constructs:

T_in(i) = S_i + H_i + G_i + R_i + U_i

  where   S_i = system prompt    G_i = tool schemas    U_i = user turn
          H_i = history          R_i = retrieval

Two facts make this brutal — and fixable:

1. Naive history replay is quadratic

A naive harness replays the full transcript every turn, so total input tokens grow as O(k²) in turn count:

Σ_{i=1..k} T_in(i)  ≈  k·S  +  [k(k−1)/2]·m̄  +  k·Ḡ  +  Σ_i R_i

A harness that compacts history, caches the invariant prefix, offloads bulky tool outputs, and trims retrieval to the minimum converts that quadratic term to (roughly) linear. Two different levers are doing two different jobs here, and it's worth keeping them straight: compaction is what bends the token count from O(k²) toward O(k) — it bounds how much history each turn carries. Caching doesn't change the count at all; it changes the price of the tokens that remain (next section). You want both. Nothing about the model changes; the bill does.

The shaded gap between those two curves is spend that buys no quality.

2. Agent workloads are input-dominated — so caching is king

Because the transcript is re-submitted every turn, production agents report input:output token ratios near 100:1. The p_in term is by far the dominant one — though, once you cache aggressively, not the entire bill (see Tier 2b).

And the price of an input token isn't one number. Tokens that repeat a previously-seen prefix are served from cache at ~0.1× the base rate (that's Anthropic's read multiplier; OpenAI and Google sit nearer 0.25×). If a fraction h of input tokens are cache reads at multiplier κ:

p_in(eff) = p_in · ( 1 − h·(1 − κ) ),     with κ ≈ 0.1

Hold h near 1 and you pay roughly a tenth of list price for the dominant term. Crucially, h is not a model property or a provider favor — it's a function of prompt byte-stability across turns, which is set entirely by how your orchestration layer assembles context.

⚠️ Caching isn't free — mind the write. Reads are ~0.1× base, but the first time a prefix is cached you pay a write premium: ~1.25× base for the default 5-minute cache, ~2× for the 1-hour cache. So caching only pays off if the prefix is reused before it expires: on the 5-min cache you break even on the 2nd request (1.25× write + 0.1× read = 1.35×, vs. 2× for two uncached calls); on the 1-hour cache, the 3rd. A prefix cached once and never reused costs more than not caching at all. This is the whole reason byte-stability matters — every byte you hold stable is a write you don't re-pay.

Verify it's actually working — cache_read_input_tokens is the single number that predicts your bill:

resp = client.messages.create(...)
resp.usage.cache_read_input_tokens      # served at ~0.1× — you want this HIGH
resp.usage.cache_creation_input_tokens  # written at ~1.25× — the premium you pay once
resp.usage.input_tokens                 # full price — you want this LOW

If cache_read_input_tokens stays zero across repeated identical-prefix calls, a silent invalidator is breaking the prefix — a datetime.now() in the system prompt, a per-request UUID, or unsorted JSON (json.dumps without sort_keys=True).

🔑 The harness controls both factors of the bill: how many tokens are submitted and the price at which the dominant ones are billed. Cache hit rate is the single highest-leverage cost variable an agent has.

🎛️ The harness: the price-setter

The harness is the runtime between your application and any foundation model. It owns:

Context assembly — system prompt, conversation state, retrieval payloads
The tool layer — native tools + external connectors (MCP), schema exposure, call mediation
Workflow execution — multi-step playbooks run end to end
Delegation — spawning scoped sub-agents and merging their results
Observability — a trace shim recording prompt tokens, completion tokens, tool events, and wall-clock for every turn

That last one matters more than it looks: the layer that meters tokens is also the audit trail; the layer that saves tokens is also the governance surface. Efficiency and control are properties of one component — which is why the harness sits at the core of everything.

The default loop vs. an engineered harness

Here's what the industry-default "conventional loop" looks like — each element a token-economics decision made by omission:

Conventional loop (the anti-pattern)	Engineered harness (the fix)
Monolithic ~49 KB system prompt replayed every turn	Byte-stable, cached prefix
Tool calls parsed by regex from an XML text stream	Native tool calling only
Destructive middle-truncation on overflow	Non-destructive structured compaction
Per-model prompt tuning	One execution path for any model
Waits implemented as polling	Zero-token durable suspension
No delegation	Scoped sub-agents with context firewalls

💡 A common procurement instinct is to compare $/Mtok across vendors. But the bill is p × τ, and τ belongs to the harness. An org that rents its orchestration layer has outsourced the one variable it controls most.

🧱 The six mechanisms that rewrite the bill

This is the heart of it. The design goal in one sentence: maximize the fraction of tokens that are (a) cached, (b) decision-relevant, and (c) spent inside committed, recoverable work — and enforce all three with structure, not model behavior.

1. 🧊 Cache-shape discipline: the two-zone prompt

Give every prompt a deliberate physical shape: a byte-stable prefix (full tool-schema catalog + stable system prompt + append-only transcript) followed by a volatile tail rebuilt each turn (clock, file listings, plan state, one-shot reminders).

Enforce it as a correctness rule, not an optimization: anything that changes per turn is structurally banned from the prefix, and the cache-marker logic refuses to place a breakpoint at or after the first volatile message.

📈 Measured on an identical-prefix call: 7,876 of 7,886 prompt tokens (99.9%) served as cache reads → the dominant input term priced at ≈0.1× list. This is the biggest single discount on an input-dominated workload.

Four mechanics that silently break caching if you ignore them (Anthropic's specifics; the shape holds elsewhere):

Gotcha	What actually happens	The fix
Minimum cacheable prefix	Below a model-specific floor, nothing caches — no error, just `cache_creation_input_tokens: 0`. Floors: Opus 4.8 / 4.7 / 4.6 & Haiku 4.5 → 4096 tokens; Fable 5 & Sonnet 4.6 → 2048; Sonnet 4.5 & older → 1024. A 3K-token prompt caches on Sonnet 4.5 and silently won't on Opus 4.8.	Keep the stable prefix above the floor for your model, or accept it won't cache.
TTL expiry	The default cache lives 5 minutes. A run that waits on a human for 10 minutes (§4) returns to a cold cache — so "zero-token waiting" is not zero-cost waiting; the resume re-pays a full write.	Use the 1-hour cache for slow human-in-the-loop steps, or pre-warm on resume.
20-block lookback	Each cache breakpoint searches back at most 20 content blocks for a prior entry. One agent turn with many tool_use/tool_result pairs blows past 20, and the next turn silently misses.	Drop an intermediate breakpoint every ~15 blocks in long turns.
Concurrent writes	A cache entry is readable only after the first response starts streaming. Fire N identical requests at once and all N pay full price — none can read what the others are still writing.	Send one, await its first token, then fan out the rest (relevant to sub-agents in §3).

💡 Not every change nukes the whole cache. Only tool-definition or model changes do. Swapping tool_choice, toggling thinking, or adding an image invalidates only the message tier — vary those per request for free. And you can pre-warm a cold prefix at startup with a max_tokens: 0 request: it runs prefill, writes the cache, and returns immediately with zero output tokens billed.

2. 🗜️ Structured, cache-aware compaction

At ~80% of the input budget, fold older history into a typed checkpoint — not a destructive truncation. Keep four artifacts: durable memory (decisions, constraints, rejected approaches), an execution summary written for resumability (current state, files touched, errors, next steps), preserved verbatim user requirements, and skill references. A live tail of the 4–12 most recent messages always survives verbatim.

Key co-design point: compaction and caching fight each other if you're careless. A summarizer that rewrites history every turn destroys the very prefix stability that caching prices. So checkpoints become durable rows, and the rebuilt prompt becomes the new cacheable prefix. Run the summarizer on a cheaper helper model, off the paying loop.

🧰 You may not have to build this yourself anymore. The Claude API now ships two of these natively: server-side compaction (context_management: {edits: [{type: "compact_20260112"}]}) auto-summarizes history as it nears a token threshold, and context editing (clear_tool_uses_20250919) strips old tool results in place — precisely the "lightest-touch compaction" of the next section, as one config line. The sharper 2026 recommendation: buy compaction and context-editing from the API, and spend your own engineering on the failure governance and durability (§4–5) that the API can't do for you. And run any custom summarizer as a Batch API job (50% off) — it's off the paying loop anyway.

3. 📤 Context offload: tokens the model never pays for

Keep information available without keeping it in context. The filesystem is the unbounded memory; the context holds pointers.

Technique	What it does
Sub-agents as context firewalls	A child agent reads/searches in its own context and returns a summary capped at ~8 KB; citations ride a metadata sidecar the parent never reads.
Skills via progressive disclosure	Prompt carries only a name-and-description table; the full skill doc is read from the sandbox only when invoked.
Bulky tool outputs spill to files	Shell output beyond ~20K chars is head-and-tail previewed; the full output is written to a workspace file (with a banner forbidding "infer success from the preview").
Event-sourced plan/state	Projected once per turn as a compact rendering; plan-tool results replaced by one-line acks so state is never duplicated. Doubles as objective recitation that counters long-horizon goal drift.
Bounded media	At most ~4 images / 2 MB in context; older ones evicted with a reload stub.

🧰 Two of these are now native too. Progressive disclosure of tool schemas is the tool-search tool (defer_loading: true on tools + a search tool): the model loads only the schemas it needs, and — the part that matters for §1 — they're appended, not swapped, so the cached prefix survives. And on Opus 4.8 you can inject a mid-run operator instruction as a {"role": "system", ...} message appended to messages[] instead of editing the prefix — the clean, prompt-injection-safe way to keep volatile content out of the cached zone.

4. ⏸️ Zero-token waiting; durability as economics

Waiting is a continuation, not a loop. When a run needs a human answer, an approval, or a long background job, it suspends durably at zero token cost and resumes on an event — no polling turns burning tokens.

The same durability layer bounds catastrophic spend: journal every event to a write-ahead log before streaming it, resume crashed runs under generation fencing at the next sequence number, persist tool results before showing them. A crash that loses a 40-turn run means re-buying 40 turns of tokens — unless you can resume from durable state.

5. 🛡️ Failure-spend governance

Retries, dead ends, and doom loops are the multiplier on the whole bill that no per-token discount fixes. Bound the multiplier:

Classify every failure into a typed class (rate limit, stall, timeout, malformed stream, provider outage, permanent) before deciding; only whitelisted classes fall through to the next provider.
Discard mid-stream failures cleanly — clear the partial draft; no side effects can originate from a discarded attempt. (Generic library fallbacks famously omit this — streaming failover often only works before the first chunk.)
Circuit-break a model that re-issues a byte-identical failing tool call 3× (cause-aware: change the args vs. back off).
Terminate loudly on truncated / length-capped outputs — never silently. Cap the loop (e.g. 50 iterations) and tool parallelism (e.g. 4).

6. 🔌 A model-agnostic floor

Which model runs, over which providers, in what fallback order, is a typed route plan supplied as data — the loop never branches on a model name. Every provider stream is normalized into one chunk contract. Native tool calling is the only invocation path, backed by schema hygiene for weaker models: inline $refs, recover double-encoded JSON args, scrub framework internals from validation errors, split overloaded schemas.

🎯 The through-line of all six: token economy and output quality are one lever pulled once. Long, distractor-dense contexts measurably degrade every frontier model — so a mechanism that removes stale/bulky tokens is simultaneously cutting the bill and cleaning the model's working set.

🧠 Context engineering: the demand side

The harness controls supply (how tokens are assembled and priced). Context engineering controls demand — which tokens deserve to be there at all. Anthropic's framing: find the smallest set of high-signal tokens that maximize the likelihood of the desired outcome.

Why it's non-negotiable: context rot. As tokens grow, recall and reasoning precision decline — attention is an n² pairwise budget, and models saw far more short sequences in training than long ones. It's a gradient, not a cliff, but it's real across all models. Treat context as a finite resource with diminishing returns.

The anatomy of good context

System prompt at the right altitude. The Goldilocks zone between brittle hardcoded if-else logic and vague hand-waving. Specific enough to guide, flexible enough to generalize. Organize with clear sections (<background>, <instructions>, ## Tools, ## Output). Minimal ≠ short — but start minimal with the best model and grow only from observed failures.
Few canonical examples, not a laundry list of edge cases. For an LLM, a good example is worth a thousand rules.
Tools that are token-efficient by contract (next section).

Techniques for long-horizon tasks

Technique	Best for	The mechanic
Compaction	Conversational flow, long back-and-forth	Summarize a near-full window into a compact brief (decisions, open bugs, key files) and continue in a fresh window. Maximize recall first, then trim for precision.
Structured note-taking	Iterative dev with clear milestones	Agent writes progress/decisions to external memory (`NOTES.md`) and re-reads on demand. Persistent memory outside the window.
Sub-agent isolation	Complex research / parallel exploration	A sub-agent burns tens of thousands of tokens exploring, returns only a 1–2k-token distilled summary. Detail stays isolated; the lead agent synthesizes.
Just-in-time retrieval	Large/dynamic corpora, codebases	Keep lightweight identifiers (file paths, queries, links); load content at runtime via tools. Sidesteps stale indexes; metadata (names, timestamps, folder) is itself signal.
Tool-result clearing	Any long tool-heavy run	Once a tool result deep in history has served its purpose, strip the raw payload — keep the conclusion.

💡 The safest, lightest-touch compaction is tool-result clearing: once a tool has been called deep in history, why keep re-sending the raw result? Many teams also find a hybrid works best — load a few stable references into context up front (Claude Code pulls CLAUDE.md in eagerly at startup), then let glob/grep fetch the rest just-in-time.

🧰 Tool design for token efficiency

Tools are the contract between a non-deterministic agent and its action space. Bad tools are a quiet token drain and a loud accuracy problem.

Choose the right tools (fewer, higher-level)

More tools ≠ better. The most common failure is wrapping an existing API endpoint 1:1 — which forces the agent to do in context what software should do in memory.

❌ list_contacts → agent reads every contact token-by-token to find one.
✅ search_contacts / message_contact → the tool does the search; the agent gets only the hit.

Consolidate frequently-chained operations into one tool:

Instead of…	Build…
`list_users` + `list_events` + `create_event`	`schedule_event` (finds availability and books)
`read_logs` (dumps everything)	`search_logs` (only relevant lines + surrounding context)
`get_customer_by_id` + `list_transactions` + `list_notes`	`get_customer_context` (compiles recent + relevant at once)

🔑 Litmus test: if a human engineer can't say which tool to use in a situation, the agent can't either. Bloated, overlapping tool sets don't just risk wrong calls — they distract the agent from efficient strategies.

Return only high-signal context

Prefer semantic names over cryptic identifiers. name, image_url, file_type inform downstream actions; uuid, 256px_image_url, mime_type waste context. Resolving UUIDs to meaningful language (or a 0-indexed scheme) measurably improves precision and cuts hallucinations.
Expose a response_format enum (concise vs detailed) — let the agent choose verbosity. In one example, concise used ~⅓ the tokens.
Paginate / filter / truncate by default. Claude Code caps tool responses at ~25K tokens. When you truncate, steer: tell the agent to make many small targeted searches instead of one broad dump.
Make errors instructive. A helpful validation error ("use user_id not user; example: …") is cheaper than a retry loop against an opaque traceback.

Namespace to reduce confusion

Group related tools under prefixes (asana_search, jira_search) so boundaries are legible. This reduces both the tool count in context and the agent's error rate.

🧪 The meta-lesson: teams often spend more effort optimizing tools than prompts. Build an eval of realistic multi-tool tasks, watch where the agent fumbles, and fix the tool (not just the prompt). Track total tool calls, token consumption, and tool-error rate — redundant calls signal missing pagination; frequent param errors signal weak descriptions.

⚖️ Harness leverage & the capability floor

Here's the subtle, important result: efficiency and quality respond to the harness differently.

Efficiency gains are unconditional. Every model tested got cheaper — 33% to 61% — across five vendors and three weight classes, with no exceptions. That uniformity is the signature of a layer-level effect: if the savings came from model-specific behavior, the spread would show it.
Quality gains are earned by capability. The improvement a model extracts from a richer harness tracks its baseline strength almost perfectly (r = 0.99). Strong models convert orchestration structure into quality; weak models can be overwhelmed by it.

This is harness leverage: the rate at which a model converts orchestration structure into quality.

The capability floor

Advanced orchestration features carry a floor below which exposing them produces failures, not function. In the study, delegated sub-agents crossed a usable reliability threshold (~0.85) only on the two strongest models; on the fast tier they sat at 0.42–0.45. Every quality regression in the whole experiment landed on the three smallest models, concentrated in the most orchestration-heavy capabilities (MCP tool use, multi-step playbooks).

🎯 Design consequence: harness features should degrade gracefully by model tier — scope down tool catalogs, disable delegation below the floor — rather than presenting one interface to every model. The harness fixes the floor; the model sets the ceiling.

🚦 Routing, fleets & compounding savings

Route by feature demand, not just difficulty

Classic routing (FrugalGPT, RouteLLM) sends easy queries to cheap models. The capability-floor finding sharpens this: route on the orchestration features a request will exercise, not just how hard its text looks.

A request that will spawn sub-agents belongs on a strong model — regardless of how simple its prompt reads.
A grounded Q&A request can take the 61%-cheaper fast tier with no quality penalty (grounding improved on every model tested).

Why harness savings compound (and model wins don't)

A model-side optimization improves one model's cost. A harness improvement multiplies every model's cost by (1 − s_m) simultaneously — and keeps multiplying when you swap models, because it lives above the model API.

Monthly spend  =  Σ_{m∈M} w_m · N · C_m

   ──(apply harness savings s_m)──▶   Σ_{m∈M} w_m · N · C_m · (1 − s_m)

Three properties make this the unusual asset in the stack:

🔁 Model-portable — implemented above the API, it applies to models that don't exist yet.
📈 Volume-linear — it grows with exactly the quantity (agent task volume) that's growing fastest.
🧲 It stacks — per-token price declines, routing, and prompt compression all multiply against it, not substitute for it.

💰 At the blended rates measured, 1M agent tasks/month = $210k under the baseline loop vs. $120k under the harness — ~$1.08M/year from an orchestration change alone, widening linearly with volume. And 1.8× faster per task is also 1.8× the throughput per unit of infrastructure.

📊 Change the KPI: measure CPM, not just quality

The managerial fix is a measurement fix. Teams that report quality alone will token-max, because tokens are someone else's line item. Teams that report efficiency can't.

Two numbers belong next to quality in every release gate:

η = Q / C                (quality per dollar)
CPM = (Q · 10⁶) / τ      (task-completions per million tokens)

In the controlled swap, both moved against the industry trajectory while quality held:

Metric	Baseline	Harness	Δ
Quality (task-completion)	0.78	0.81	+0.03 (parity at n=22)
Cost / task	$0.21	$0.12	−41%
Wall-clock / task (median)	48 s	27 s	−44%
Tokens / task	14.2k	8.8k	−38%
Quality per dollar (η)	3.71	6.75	+82%
Completions per Mtok (CPM)	54.9	92.0	+68%

🔑 CPM belongs next to quality for the same reason performance-per-watt sits next to performance in chip design: it's the number that predicts the bill. And be honest in reporting — the temptation is to headline "+0.03 quality"; the defensible headline is "−38% tokens at parity."

A note on measurement integrity

Without per-task token accounting built into the orchestration layer, token maxing is unobservable — and what's unobservable is unmanaged. Most widely-used frameworks (LangGraph, CrewAI, AutoGen/AG2) leave prompt-cache policy, compaction, and failure governance to the application and meter none of it per task. Put the meter in the same layer that spends the tokens.

⚠️ Watch the multi-agent multiplier. Shared-transcript multi-agent frameworks are a token multiplier by construction — each agent re-reads the growing conversation and carries its own preamble. Anthropic's own measurement: agents ≈ 4× chat token consumption, multi-agent systems ≈ 15×, with token volume explaining ~80% of performance variance. Worth paying only for high-value, parallelizable work — and only if you meter it.

🛠️ The practical playbook

A prioritized checklist, roughly in order of leverage. Start at the top.

Tier 1 — Highest leverage (do these first)

[ ] Shape prompts into a byte-stable prefix + volatile tail. Ban anything per-turn (clocks, listings) from the prefix. Target a >90% cache-read rate on steady-state turns.
[ ] Cache the tool-schema catalog and system prompt. Schemas broadcast on every call, uncached, are pure waste on an input-dominated workload.
[ ] Replace destructive truncation with structured compaction. Keep decisions/constraints/next-steps; run the summarizer on a cheaper model, off the paying loop.
[ ] Cap the loop and classify failures. Iteration cap, tool-parallelism cap, circuit-breaker on identical repeated failing calls, no side effects from discarded attempts.

Tier 2 — Context & tools

[ ] Offload bulky outputs to files; keep pointers in context. Head/tail previews with a "don't infer success from the preview" banner.
[ ] Adopt just-in-time retrieval (paths/queries/links loaded at runtime) over dumping a knowledge base up front.
[ ] Rerank before you inject. On any RAG step, rerank retrieved candidates and inject only the top 2–3 chunks — retrieval precision is the knob that sets your R_i term. More chunks past that mostly buy context rot, not recall. (Exception: genuinely **recall-sensitive* work — compliance sweeps, exhaustive extraction — needs more; "top 2–3" is the rule for precision-oriented lookups.)*
[ ] Prune the tool set to a few non-overlapping, high-level tools; namespace them; return concise by default with an opt-in detailed.
[ ] Make tool errors instructive, not opaque.
[ ] Use sub-agents as context firewalls for exploration — cap the returned summary, keep citations on a sidecar.

Tier 2b — Output-side controls (bigger than they look)

A correction to the folk wisdom that "input is basically the whole bill." At a 100:1 token ratio, with output priced at 5× the input token (the exact ratio across today's Claude line — Opus 4.8 \$5/\$25, Sonnet 5 \$3/\$15, Haiku 4.5 \$1/\$5), the input side is ~95% of the bill only when nothing is cached. But Tier 1 is caching — and the more you cache the input, the more the output side dominates what's left:

Cache-read rate `h` on the input	Input share of the total bill
0% (uncached)	~95%
90%	~79%
~100%	~67%

So once you've done Tier 1, output is 20–33% of the bill — not ~1%. It also drives latency and turn count. These are one-line settings with an outsized payoff:

[ ] Set max_tokens on every call. A hard ceiling on generation caps both worst-case cost and worst-case latency, and turns a runaway into a clean, classifiable length-cap failure (which §Failure-spend already terminates loudly).
[ ] Use stop sequences. Halt generation at the first ], \n\n, or sentinel token instead of letting the model ramble to its own EOS.
[ ] Constrain the answer shape, not just tool calls. You already force native tool-call schemas; do the same for the model's own reply — JSON-schema / Pydantic / grammar-constrained decoding kills conversational filler ("Sure, here's your info:") and, more importantly, cuts the reparse-and-retry loop that malformed output triggers.
[ ] Instruct for brevity in reasoning, not just answers. "Be concise, no preamble" on the answer, and terse-reasoning styles (Chain of Draft) on the thinking — reasoning traces are output tokens too, and they compound every turn.

Tier 3 — Fleet & governance

[ ] Suspend on waits at zero token cost; journal to a WAL so crashes resume instead of re-buying turns.
[ ] Add a semantic/exact response cache in front of the model (e.g. GPTCache — see §The tooling landscape). Distinct from the prefix cache (which makes tokens ~10× cheaper): an exact-match or vector-similarity cache of prior query → answer pairs skips the LLM call entirely on repeat/near-duplicate requests. Gate on a similarity threshold + TTL; best for stable, high-repeat lookups, not for state-dependent agent turns — and remember a loose threshold serves confidently wrong answers.
[ ] Route by feature demand + difficulty; disable above-floor features (delegation) on sub-floor models.
[ ] Put a per-task token/cost meter in the orchestration layer. Add CPM and quality-per-dollar (η) to your release gate.
[ ] Degrade harness features gracefully by model tier rather than one-interface-for-all.

Guardrails that also save tokens

[ ] Objective checkpoints (tests pass, schema validates, build succeeds) as loop gates — ground every step in environment feedback so errors don't compound.
[ ] Human approval on irreversible/high-blast-radius actions — and remember the lethal trifecta: don't combine untrusted input + private data + external comms in one autonomous session.

🧰 The tooling landscape: what actually implements this

This guide is about principles, not products — but readers reasonably ask "what can I actually install?" Three open-source projects map cleanly onto the framework above, and lining them up surfaces the single most important lesson about buying compression off the shelf.

Tool	What it is	Where it sits	Mechanisms it implements	Cache-safe?	Maturity / license
LLMLingua (Microsoft)	A prompt-compression algorithm — a small LM scores token importance and drops the low-value ones	Context engineering (demand side); shrinks `R_i`, `H_i`	Lossy retrieval/history compaction	❌ breaks it	Mature, peer-reviewed (EMNLP/ACL), MIT
Headroom	A compression layer (library / proxy / MCP) with content-type routing — JSON, AST-aware code, prose	Harness (supply side)	§1 cache-shape, §3 offload (reversible retrieval), Tier 2b output	✅ built for it	Very new (2026), viral (62k★), Apache-2.0
token-optimizer	An external hook-based tool for Claude Code: measure, compress, survive compaction	The CPM meter + governance	§2 compaction-survival, §3 offload, the per-task meter, routing	✅ (freezes prefix)	New (2026), niche, noncommercial license
RTK	A deterministic CLI-output compressor — rewrites 100+ dev commands (`git`, `cargo test`, `grep`…) to emit filtered/deduped output, no model in the loop	Harness — tool-output shaping	§3 offload, the shell-output slice — structure-aware, so no correctness risk from the compression itself	✅ (deterministic → stable, smaller bytes)	Very new (2026), viral (73k★), Apache-2.0, zero-dep Rust

They're complementary, not competing — different layers of the same stack, not rival products (a prompt-compression algorithm, two harness-layer output compressors, and a measurement wrapper).

Two adjacent categories: skip the call, and remember across sessions

The three above all compress what you send. Two more widely-used tools attack the bill from angles the framework names but that aren't compression at all — one skips the model call entirely, the other changes what's worth sending across sessions.

Tool	What it is	Where it sits	Maps to	The catch
GPTCache (Zilliz)	A semantic response cache — embed the query, similarity-search prior `query → answer` pairs, and on a hit return the stored answer without calling the LLM	Before the model	Tier 3 "semantic/exact response cache" — the call priced at zero, not 0.1×	A false-positive hit serves a confidently wrong answer; staleness needs TTL + invalidation; repo is ~a year stale (unstable APIs by its own README) — vendor it, don't depend on it
mem0 (YC S24)	A memory layer — an LLM extracts facts from conversations into a vector/graph store, retrieved by semantic + keyword + temporal signals	Assembling context, across sessions	§Context: structured note-taking + just-in-time retrieval — distilled recall instead of full-history replay	Extraction is itself an LLM call (write-time cost you pay to save reads later); ADD-only accumulation piles up stale/contradictory facts; injected memories must live in the volatile tail or they break prefix caching

Put all five tools on one request's timeline and they resolve into layers applied in order, not rivals:

Read it as four questions in sequence: can I skip the call? (GPTCache) → what's worth putting in, and how small? (mem0, compression) → how cheap are the tokens I do send? (prefix cache, native) → what did it cost? (the meter). A semantic-cache hit short-circuits everything downstream — which is why it's the highest-ROI layer when traffic is repetitive and read-only, and a correctness landmine when it isn't.

🥇 The one lesson: compression that breaks caching can raise your bill

The sharpest line between these tools is whether they respect the prefix cache — a direct corollary of §1's "cache hit rate is the #1 cost variable."

⚠️ Do the math before you trust a compression ratio. Cached input is priced at ~0.1×. Say you send 10k input tokens at a 90% cache-read rate → effective cost ≈ 10,000 × (0.9·0.1 + 0.1·1.0) = 1,900 price-units. Now a per-query compressor cuts it 40% to 6k tokens but rewrites the prefix, dropping cache hits to ~0 → effective cost = 6,000 × 1.0 = 6,000 units. The 40% "saving" made it ~3× more expensive (before counting the compressor's own compute). Token count fell; the bill rose.

This is why LLMLingua — a brilliant, pre-caching-era design (2023) — is a conditional win: use it for large, redundant prose/RAG contexts on a model or provider without good caching, where exact fidelity isn't critical. Avoid it for code, structured data, tool schemas, or any cache-friendly agent loop. The two 2026 tools were built after caching became the dominant lever and treat prefix stability as sacred (Headroom's CacheAligner; token-optimizer's freeze-and-checkpoint).

🥈 The other lesson: their own numbers concede that routing beats compression

token-optimizer honestly reports two figures — ~$313/mo actually metered from compression, versus a ~$1,877/mo "big picture" — and admits the big number is mostly model routing (shifting Opus 95%→60%), not compression. That's this guide's thesis restated by a compression tool: the harness levers (routing, caching discipline) move the bill more than token-squeezing does. Read every vendor benchmark this way — Headroom's honest number is "20% for coding"; the "60–95%" is JSON and logs, where redundancy is extreme and any compressor wins.

RTK models the honesty you want. Its headline "60–90% reduction" is stated, in its own README, as being of the bash output alone — one slice of one term (R_i) in the bill decomposition (§The token bill, decomposed). Fold in the replayed prefix (H), system prompt (S), and output tokens and the bill impact is smaller — and RTK says so plainly. A tool that tells you which term its percentage applies to is doing exactly what §Change the KPI asks of you; treat every tool that doesn't as quoting the flattering number.

How to choose

If you…	Reach for	But first
Have big redundant prose/RAG, weak or no caching, fidelity not critical	LLMLingua	Confirm you're not on a cache-friendly path — it backfires if you are
Run JSON / log / tool-output-heavy agents and want a cache-safe drop-in	Headroom	Measure your real cache-hit rate before/after — not token count
Are on Claude Code and want visibility + compaction-survival	token-optimizer	Check the noncommercial license fits; treat it as a CPM dashboard first
Serve repetitive, read-only front-door queries (FAQ, docs Q&A)	GPTCache	Tune the similarity threshold hard and add a TTL — a loose match returns wrong answers; vendor it (it's stale)
Need cross-session memory / personalization for long-lived agents	mem0	Budget the extraction call; keep retrieved memories in the volatile tail; verify its benchmarks on your data
Want a deterministic, safe reducer for shell / dev-command output	RTK	It touches only command output — one slice of the bill; pair it with caching + routing for real impact

🎯 None of these does the highest-leverage work in §5 — failure-spend governance (retry / doom-loop caps) or zero-token durable suspension. Compression is Tier 2; the meter is the KPI; the biggest wins still live in a harness you own. Buy these for the layers you don't want to build; don't let a star count talk you out of owning the parts that set your bill.

🎯 One-page cheat sheet

Area	The single highest-leverage move
🧊 Caching	Byte-stable prefix + volatile tail; cache schemas & system prompt → dominant term at ~0.1× list
🗜️ History	Structured, cache-aware compaction — never destructive truncation; convert `O(k²)` → `O(k)`
📦 Offload	Filesystem = memory, context = pointers; sub-agents as capped-summary firewalls
⏸️ Waiting	Zero-token durable suspension; WAL so crashes resume, not re-buy
🛡️ Failures	Typed classification, circuit-breakers, no side effects from discarded attempts, loop caps
🔌 Portability	Route plan as data; native tool calling only; schema hygiene for weak models
🧠 Context	Smallest set of high-signal tokens; compaction / notes / JIT retrieval / sub-agents
🔎 Retrieval	Rerank, inject only top 2–3 chunks; precision sets the `R_i` term
🧰 Tools	Few, high-level, non-overlapping; semantic IDs; `concise` default; instructive errors
📤 Output	`max_tokens` + stop sequences + constrained answer shape; terse reasoning — 20–33% of a cached bill, not ~1%
🗄️ Semantic cache	Vector/exact query→answer cache skips the LLM call entirely — distinct from prefix caching
⚖️ Leverage	Efficiency is unconditional; quality is earned — respect the capability floor
🚦 Routing	Route by feature demand, not just difficulty; harness savings compound & stack
📊 KPI	Add CPM + quality-per-dollar to the release gate — headline "−38% at parity", not "+0.03"

🧩 The five habits that prevent token maxing

The harness is the P&L, not the plumbing. It sets the price of work — optimize it before you shop for cheaper models.
Cache hit rate is your #1 cost metric. On input-dominated workloads, prompt byte-stability is the bill.
Treat context as a scarce budget. Every stale or bulky token both costs money and degrades the model's working set.
Bound the multiplier. Retries, dead ends, and doom loops — not the model's verbosity — are where runaway spend hides.
Measure what you don't want to happen. Put a per-task meter in the orchestration layer and gate releases on CPM. What's unobservable is unmanaged.

The models keep getting better — but the durable engineering wins are in the harness and the context. Do the same work with fewer, better-placed, cheaper-priced tokens, and every model you run — present and future — gets cheaper the moment you do.

🗺️ Companion Reads

This guide prices out the harness and context levers. These companion pieces from the same series go deeper on the layers this one only costs out — the loop being metered, the reliability discipline behind it, the tools that fill the window, and the failure modes that blow up the bill.

Document	Why it pairs with this guide
🏗️ Harness Engineering: The Emerging Discipline of Making AI Agents Reliable 🤖	The discipline this guide puts a price on. Where §The harness argues the orchestration layer sets the bill, this makes the reliability case for owning it — the same six mechanisms viewed as engineering practice, not economics.
🛠️ Harness Engineering — Quick Actionable Guide 🤖	The pocket version. A concise checklist of harness patterns and guardrails — read it next to §The practical playbook when you want the moves without the derivations.
🤖 The Agentic Loop 🔄 Loop Engineering: A Practical Field Guide 📘	The k-turn loop this guide sums a cost over. Explains the observe–reason–act mechanics behind §The token bill, decomposed — read it first if the loop model in §The one mental model is new to you.
🏗️ Building High-Quality AI Agents 🤖 — A Comprehensive, Actionable Field Guide 📚	The how to build reliably counterpart. Tool ergonomics and ACI design that make §Tool design's "fewer, higher-level tools" concrete, plus the quality bar behind the capability floor in §Harness leverage.
⚠️ Common Issues with LLMs & AI Agents — and How to Fix Them 🛠️	The failure catalogue behind §Failure-spend governance. Retries, dead ends, doom loops, and truncation are where runaway spend hides — this is the field guide to diagnosing and bounding them.
🔮 Hermes Agent 🤖 — Deep Dive & Build-Your-Own Guide 📘	§1 and §3 shown in code — cache-stable two-zone prompts, progressive-disclosure memory, and a self-improving loop that keeps the byte-stable prefix intact instead of rewriting it every turn.
🤖 SWE-agent — Deep Dive & Build-Your-Own Guide 📘	The Agent-Computer Interface that inspired modern coding-agent tool design — concrete grounding for §Tool design's "return only high-signal context" and instructive-error rules.
🦊 GoClaw Deep Dive 🤖 — A Builder's Guide to a Multi-Tenant AI Agent Platform 📘	Provider resilience and routing that implement §A model-agnostic floor and §Routing, fleets — typed route plans as data, normalized streams, and graceful degradation by model tier.
🏗️ Building Production-Grade Fullstack Products with AI Coding Agents 🤖 — A Practical Playbook 📘	Where the meter meets the pipeline. End-to-end delivery discipline — evals, PR gates, monitoring — that operationalizes §Change the KPI's "put CPM in the release gate."

Suggested reading path:

This guide (the token economics of the harness + context)

→ 🤖 The Agentic Loop 🔄 Loop Engineering: A Practical Field Guide 📘 (the loop being metered)

→ 🏗️ Harness Engineering: The Emerging Discipline of Making AI Agents Reliable 🤖 (why you own the layer)

→ 🏗️ Building High-Quality AI Agents 🤖 — A Comprehensive, Actionable Field Guide 📚 (tool + ACI design)

→ ⚠️ Common Issues with LLMs & AI Agents — and How to Fix Them 🛠️ (bound the failure multiplier)

→ 🏗️ Building Production-Grade Fullstack Products with AI Coding Agents 🤖 — A Practical Playbook 📘 (ship it and meter it)

📚 Sources & further reading

Writer, Inc. — The Harness Effect: How Orchestration Design Sets the Token Economics of Enterprise Agentic AI (arXiv:2607.06906, Jul 2026) — the controlled harness-swap study behind most numbers here.
Anthropic — Effective context engineering for AI agents (2025) · Writing effective tools for agents (2025) · How we built our multi-agent research system (2025) · Prompt caching docs
Chroma Research — Context Rot: How increasing input tokens impacts LLM performance
Manus — Context Engineering for AI Agents: Lessons from Building Manus (KV-cache hit rate as the first metric)
Epoch AI — LLM inference prices have fallen rapidly but unequally (the Jevons backdrop)
Foundational agent scaffolding: ReAct · Reflexion · MemGPT · SWE-agent · Lost in the Middle
Efficiency techniques that stack with the harness: LLMLingua (prompt compression) · Chain of Draft (terse reasoning) · FrugalGPT / RouteLLM (routing) · Speculative decoding · PagedAttention/vLLM
Open-source tools that implement these mechanisms (see §The tooling landscape): Headroom (cache-aware compression proxy) · LLMLingua (prompt-compression algorithm) · token-optimizer (per-task metering + compaction-survival for Claude Code) · GPTCache (semantic response cache — skip the call) · mem0 (cross-session memory layer) · RTK (deterministic CLI-output compressor) — treat vendor benchmarks as marketing; measure the impact on your own workload

If you found this helpful, let me know by leaving a 👍 or a comment!, or if you think this post could help someone, feel free to share it! Thank you very much! 😃