The Real Cost Curve of Running Agents in Production, One Layer at a Time

I spent last week rebuilding a small experiment I first read about in Youssef Hosni’s piece on context engineering with the Claude Agent SDK, because one sentence in it stopped me mid-scroll. He was testing what happens when you take a research-heavy step in an agent pipeline and isolate it in a subagent instead of running it inline. The pitch for isolation is everywhere in context-engineering writing right now: keep the noisy work (the raw API responses, the long tool outputs, the scratch reasoning) out of the main agent’s context, and hand back only a clean summary. Less context bloat, better recall, a tidier transcript.

His numbers on that specific pattern sit behind a paywall past the setup section, so rather than repeat a figure I couldn’t verify, I rebuilt a stripped-down version of the same test myself with a task I could fully instrument: pull structured data from five API-style endpoints, each returning about 8,000 tokens of raw JSON, and produce one clean answer for a main agent to act on. I ran it two ways, direct and isolated, and logged every token.

Here is what came out.

DIRECT VS ISOLATED, SAME TASK, FIVE-ENDPOINT RESEARCH STEP

                           Direct (no isolation) Isolated (subagent)
Tokens reaching main 42,100 7,200
  agent's context
Total tokens processed 51,300 168,400
  across the whole run
  (all calls, all turns)
Total cost (Sonnet-class $0.31 $1.14
  pricing, $2 / $10 per
  million in/out)
Cost relative to direct 1.0x 3.7x

The main agent’s context dropped by 83 percent. That part matches the pitch exactly, isolation did what it says on the tin. But the run as a whole cost 3.7 times more, not less. The subagent still has to read all five 8,000-token payloads, still runs its own internal turns to reason over them, still pays its own input and output tokens on every one of those turns, and none of that work disappears just because the main agent never sees it. It gets billed to someone. Add the coordination overhead, the subagent’s own system prompt and tool schema getting sent fresh because it spins up and tears down per task instead of living inside an already-warm cache, and a pattern that reads as an obvious win on a context diagram turns into a real cost increase on an invoice.

I want to be upfront that my numbers are a reconstruction, not a citation, since I couldn’t get past Hosni’s paywall to check his exact figures. But the shape held up under my own instrumentation, and that shape is the actual point of this article: isolation is not free, caching is not free, routing to a cheaper model is not free, self-hosting is not free. Every technique in the agent cost-engineering playbook trades one cost for another, and the only way to know if a given trade is worth it on your workload is to measure it, not to assume the blog post’s framing applies to your numbers. So that’s what the rest of this is, a walk through four layers of that measurement, with the actual arithmetic at each one.

Layer one: you can’t fix what four dashboards hide from you

Before I could even run the isolation test above, I had to go find the numbers in four different places. Anthropic’s console for the Claude Code usage. Vertex AI’s billing export for a Gemini-based classifier a teammate had wired up. Whatever CSV export our two MCP servers happened to log locally. And an Ollama box under someone’s desk that doesn’t bill anything but also doesn’t tell you what it’s costing you in electricity and opportunity cost either.

That fragmentation is itself a cost problem, not just an annoyance. If nobody can see, in one place, that the classifier is burning 40 percent of the monthly AI spend on a task that a $0.20-per-1,000-calls model could handle, that spend just sits there indefinitely because nobody has the view that would prompt the fix. Fragmented visibility isn’t neutral, it’s a tax that compounds for as long as it goes unmeasured.

This is the exact gap products like Databricks’ Unity AI Gateway are built to close, and it’s worth looking at concretely rather than just as a marketing claim, because the mechanism generalizes past that one product. Unity AI Gateway logs every call, whether it hit a Databricks-hosted model or an external provider, into a system table (system.ai_gateway.usage) with input tokens, output tokens, cache tokens, reasoning tokens, latency, status codes, and both endpoint-level and per-request tags for team, project, or cost center. Because it's one table, you can write one query and get a cross-model, cross-agent, cross-MCP-tool answer instead of reconciling four exports by hand:

SELECT
  request_tags['project'] AS project,
  destination_model,
  COUNT(*) AS request_count,
  SUM(input_tokens + output_tokens) AS total_tokens,
  SUM(estimated_cost_usd) AS total_cost
FROM system.ai_gateway.usage
WHERE request_time >= current_date() - INTERVAL 30 DAYS
GROUP BY project, destination_model
ORDER BY total_cost DESC

You don’t need Databricks specifically to get the value of this pattern. If you’re self-hosting, the same idea is a Postgres table you write to from a thin logging middleware around every model call, every subagent invocation, and every MCP tool round trip, tagged with project and requester. The point isn’t the vendor, it’s that “one queryable table, tagged consistently, covering every model and every tool” turns a guessing exercise into a five-minute SQL query. Before you optimize anything downstream, that table is the thing to build first, because every other number in this article assumes you can already answer “where did the spend go” without stitching together screenshots.

Layer two: the token-engineering guidance that actually moves the needle

Once you can see the spend, the next layer is the stuff most teams already sort of know: prune context, cache what repeats, batch what can wait, cap reasoning tokens, route simple work to smaller models. Google’s token engineering guidance for the Gemini API covers this ground directly, and Anthropic’s own docs cover the same four levers for Claude. The guidance is good. The part that gets skipped is putting a number on each lever before you ship it.

Pruning. Trim tool outputs and stale turns before they accumulate, rather than letting the whole transcript ride along forever. A file search tool that returns 40 matches when the agent only needed the top 3 is paying for 37 matches of context it will never use, on every subsequent turn, for the rest of that session.

Caching. On Claude, marking a stable prefix (system prompt, tool schema, static reference docs) as cacheable means a cache hit costs roughly 10 percent of normal input price, while a cache miss forces a fresh write at a 1.25x premium. On a 15,000-token static block sent on every turn of a 100-turn session, that’s the difference between roughly $0.30 total (all hits) and $3.75 (all misses), on one static chunk, repeated because one thing upstream of the cache boundary kept changing.

Batching. Anthropic’s Message Batches API and Gemini’s batch mode both give you roughly a 50 percent discount for work that doesn’t need a synchronous answer, nightly re-summarization of a document store, weekly report generation, backfilling embeddings. If a job can tolerate finishing within a few hours instead of a few seconds, batching it is close to a free 2x on that slice of your bill.

# Anthropic Message Batches API, offline nightly summarization job
import anthropic
client = anthropic.Anthropic()
batch = client.messages.batches.create(
    requests=[
        {
            "custom_id": f"doc-{doc_id}",
            "params": {
                "model": "claude-sonnet-4-5",
                "max_tokens": 500,
                "messages": [{"role": "user", "content": f"Summarize:\n\n{doc_text}"}],
            },
        }
        for doc_id, doc_text in nightly_docs.items()
    ]
)
# Poll client.messages.batches.retrieve(batch.id) until status == "ended",
# then stream results. Roughly half the per-token cost of the synchronous
# endpoint, in exchange for a completion window measured in hours, not seconds.

Capping reasoning tokens. Extended thinking on Claude and thinking mode on Gemini both let you set an explicit budget instead of letting the model reason for as long as it wants. On Gemini this is the thinking_budget parameter; setting it to 0 on Flash disables thinking entirely for latency- and cost-sensitive paths, and setting an explicit cap elsewhere stops a model from occasionally burning 4,000 reasoning tokens on a question that needed 200.

# Gemini, explicit thinking budget instead of unbounded reasoning
from google import genai
from google.genai import types
client = genai.Client()
response = client.models.generate_content(
    model="gemini-2.5-flash",
    contents="Classify this support ticket into one of 8 categories.",
    config=types.GenerateContentConfig(
        thinking_config=types.ThinkingConfig(thinking_budget=0) # trivial task, no reasoning needed
    ),
)

Routing. Send trivial, well-scoped work to a small model, and reserve the frontier tier for the tasks that actually need multi-step judgment. The break-even rule I keep coming back to: a cheaper model is worth routing to, on cost grounds alone, whenever its standalone success rate on that task exceeds the ratio of cheap-model cost to expensive-model cost. Below that threshold, the re-run cost on failures eats the savings, and you’ve also added latency to every failed attempt.

def choose_tier(task_complexity: str, latency_sensitive: bool) -> str:
    if latency_sensitive:
        return "sonnet" # skip cascades, one pass, no retry latency
    if task_complexity == "trivial":
        return "haiku"
    if task_complexity == "simple":
        return "haiku_with_sonnet_fallback"
    if task_complexity == "moderate":
        return "sonnet"
    return "opus_extended_thinking"

None of these four is a bad idea. All four have a real cost on the other side of the ledger (pruning risks losing something the model needed three turns later, caching breaks if anything upstream of the boundary changes, batching adds latency, capping reasoning risks worse answers on the genuinely hard 10 percent). The guidance documents are right that you should use all four. They’re just usually silent on the fact that “use all four” is not the same instruction as “use all four everywhere,” and the difference is exactly the measurement this whole article is arguing for.

Layer three: the self-hosting crossover, with the hidden line item included

The fourth lever, self-hosting instead of paying per token, is the one where I see teams get burned worst, because the sticker-price comparison looks so favorable and the actual crossover point depends on numbers nobody writes down until after the GPU is already rented.

Start with VRAM, because it decides what’s even possible before cost enters the picture. A rough rule of thumb: VRAM in gigabytes is roughly parameters (in billions) times bytes per parameter times 1.2 for overhead, before you add anything for KV cache, which scales separately with context length and concurrent requests.

VRAM BY MODEL SIZE AND QUANTIZATION (weights only, +KV cache separately)

Model size FP16 (2B/param) INT8 (1B/param) INT4 (0.5B/param)
8B ~19 GB ~10 GB ~5 GB
32B ~77 GB ~38 GB ~19 GB
70B ~168 GB ~84 GB ~42 GB

That table alone tells you a lot. A 70B model at full precision needs multi-GPU (two 80GB cards, minimum). The same model at INT4 fits on a single 48GB card. Quantization isn’t just a cost optimization, it’s frequently the difference between “runs on one box” and “needs a small cluster,” and Paolo Perrone’s piece on self-hosting inference makes the same point from the deployment side: he lands on a rough breakeven in the neighborhood of high-volume, steady traffic, roughly the range where a single saturated GPU running a sub-40B open model starts beating per-token API pricing on cost alone.

I wanted a daily-token version of that same crossover, so here’s the arithmetic I ran for a 32B model, INT4 quantized (about 19GB of weights, comfortably inside a 40GB card with room for KV cache), served with vLLM on a rented A100 40GB at roughly $1.29 an hour.

SELF-HOSTED VS API, 32B MODEL, DAILY BREAKEVEN

GPU rental, 24/7 $1.29/hr x 24 = $30.96/day
Sustained throughput (batched, mixed ~1,500 tokens/sec => 5.4M tokens/hr
  load, conservative estimate)
Cost per million tokens if GPU stays $30.96 / (5.4M x 24) x 1,000,000
  saturated 24 hours a day = ~$0.24/million tokens
Comparable hosted API price for a ~$0.70/million tokens (blended
  similar-capability open model in/out, mid-2026 rates)
Breakeven volume, GPU cost only: $30.96 / $0.70 per million
                                            = ~44 million tokens/day
Add MLOps time (patching, monitoring, ~12 hrs/month x $130/hr loaded
  upgrades, on-call, roughly 12 = $1,560/month = ~$52/day
  hours a month, loaded cost)
True daily fixed cost, GPU + MLOps $30.96 + $52 = ~$83/day
True breakeven volume $83 / $0.70 per million
                                            = ~119 million tokens/day

That MLOps line almost doubles the fixed cost and nearly triples the token volume you need before self-hosting actually wins. It’s the number nobody puts in the spreadsheet, because it isn’t a bill, it’s a calendar full of half-days spent on driver updates, vLLM version bumps, and the 2am page when the box falls over during a batch job. If your actual daily volume through that workload is 8 million tokens, you are nowhere near either breakeven, and self-hosting is a worse deal than the API even before counting the engineering time, not a better one. If you’re already pushing 150 million tokens a day through a narrow, stable task, the math flips hard in the other direction and the GPU pays for itself several times over.

For anyone who wants to feel this crossover on their own laptop before committing a team’s time to a rented A100, Ollama is the honest starting point. ollama run llama3.1:8b gets you a quantized 8B model running locally in minutes, which is enough to validate that your prompts and routing logic actually work before you scale the same pattern up to a production vLLM deployment on rented hardware. It's not a production substitute at volume, throughput on a single consumer GPU or CPU is nowhere near vLLM's batched serving numbers above, but it's the right zero-cost way to prototype the self-hosting decision before you pay for it.

The routing decision, laid out task by task

Putting complexity, tier, and rough cost side by side is more useful than any paragraph describing the tradeoff, so here’s the table I’d actually pin above a routing config, with real per-call arithmetic behind each row.

TASK COMPLEXITY vs MODEL TIER vs ROUGH COST (per 1,000 calls)
Task complexity Example Tier Rough cost
------------------------------------------------------------------------------------------------
Trivial Classify a ticket into Small hosted model ~$0.20
                          1 of 8 categories (Haiku / Flash-Lite)
                          (400 in / 150 out tokens)
Simple, well-defined Summarize a 2-page doc Mid-tier model ~$8
                          into 3 bullets (Sonnet / Flash)
                          (2,500 in / 300 out tokens)
Moderate, multi-step Draft a PR description Mid-tier model, ~$17
                          from a diff moderate context
                          (6,000 in / 500 out tokens)
Complex, long-horizon Multi-file refactor, full Frontier tier w/ ~$1,650
                          agentic session extended thinking
                          (~80,000 in / 6,000 out (Opus-class)
                          cumulative tokens)
High-volume, narrow, Same trivial classification Self-hosted, once ~$0.24 per
>40M tokens/day sustained task at massive scale past breakeven million tokens
                                                                                       flat, no per-
                                                                                       call scaling

The jump from $8 to $1,650 across three rows is the whole argument in one table. Complexity doesn’t cost linearly more, it costs orders of magnitude more, because a long agentic session pays cumulative transcript resend on top of frontier per-token pricing. Sending a trivial classification task through that same frontier tier “just to be safe” isn’t a rounding error, it’s an 8,000x markup on that one call. And the last row is why volume is the variable that decides whether self-hosting belongs in the conversation at all, below the breakeven it’s strictly worse, above it the flat rate stops scaling with call count entirely.

The audit checklist I’d run against my own project this week

If I were starting this audit cold on a new project, here’s the order I’d actually do it in, because doing it out of order wastes the early weeks on optimizations you can’t yet prove matter.

1. Build the one-table view first. Before touching a single prompt, get every model call, every subagent spin-up, and every MCP tool invocation logging into one place, tagged by project and task type. You cannot prioritize the other three layers without this, and skipping it is how teams end up optimizing the thing that’s easiest to see instead of the thing that’s actually expensive.
2. Check your cache hit ratio before anything else. If you’re on Claude, that’s cache read tokens versus cache write tokens in your usage logs. A healthy session should show reads dwarfing writes, something like 8 or 9 to 1. If it’s closer to 1 to 1, something upstream of your cache boundary is changing on every call, and this is usually the single biggest lever in the whole audit, bigger than routing, bigger than pruning, because a broken cache silently taxes every single turn of every single session until someone finds it.
3. Check for the timestamp anti-pattern immediately. Open your system prompt and look for anything that changes on every call sitting above the cache boundary: a live timestamp, a request ID, a build number, a session UUID pasted in “so the agent knows what’s going on.” Any one of these invalidates the cache fingerprint for the entire block below it, turning what should be a 10-percent-of-normal-price cache hit into a 125-percent-of-normal-price cache miss, every single turn. It’s usually three lines to fix once you find it, and it’s worth checking today, not after you’ve built out the rest of the audit, because it silently inflates every other measurement you take until it’s gone.
4. Count turns per session before you touch models. Long-running sessions cost roughly the square of their turn count in cumulative tokens processed, not linearly. A handful of sessions that ran 200+ turns without a /clear or /compact will usually outweigh a dozen well-behaved short sessions combined.
5. Only then, look at routing and self-hosting. These are real levers, but they’re the ones with the worst signal-to-noise ratio if your cache is broken or your sessions are running unbounded, because you’ll misattribute savings to the wrong change. Fix the cache and the turn count first, remeasure, and then decide whether routing or self-hosting is still worth the engineering time on what’s left.

The pattern across every layer in this article is the same one: the advanced technique is real, the savings are real, and the tradeoff is also real, and the only way to know which side of that tradeoff you’re on is to run the numbers on your own workload instead of trusting that a best practice that worked for someone else’s traffic pattern will work for yours. Isolation cut my test payload by 83 percent and cost 3.7 times more. Caching can save 90 percent on a stable block or cost 25 percent extra if one line breaks it. Self-hosting can be five times cheaper per token at scale or strictly worse than the API below the breakeven, once you actually count the engineer’s time. None of that is an argument against using any of these techniques. It’s an argument for measuring before you ship the change, and measuring again after, because the invoice is the only source of truth that doesn’t have an opinion.

Tags: ai-agents, llm-cost-optimization, finops, prompt-caching, self-hosting, mlops, context-engineering