Your Agent Passed Every Eval and Still Cost $4,000 a Day

There's a specific kind of production incident that no dashboard warns you about, because every signal you're watching is green.

Your agent runs. It returns output. The output parses, contains the right fields, reads like a competent answer. Your eval suite goes green. Your CI gate passes. And your inference bill is quietly climbing — four figures a day at real volume — because that same agent has been silently retrying, looping, timing out mid-task and getting retried by the orchestrator, or running away for 40 tool-calls when the job needed 4, all while producing a final message that looks finished.

I run a fleet of autonomous agents in production. They write blog posts, harden repos, sweep stale state, back up memory — unattended, on cron, every few hours. This failure mode is the one that cost me real money before I built the tooling to catch it. Here's the anatomy of it, and the specific reason your evals can't see it.

Why "it passed the evals" and "it burned $4,000" are both true

The mistake is treating an eval as a question about output. It isn't. In production it's a question about the run.

Consider a run that:

1. Started the task correctly.
2. Made 6 tool calls.
3. Hit a provider timeout on call 7.
4. Got silently retried by the orchestrator — twice.
5. Eventually emitted a final message that reads: "I've reviewed the file and applied the fixes."

Every output-level check passes. The message is non-empty. It contains "fixes." It matches your rubric for "sounds like a completed audit." An LLM-as-judge scoring the deliverable gives it a 7/10 — because the deliverable, in isolation, is fine.

But the run abandoned. It burned several times the tokens the task needed. Nothing was actually applied. And the thing you were using to catch problems — a model judging the final text — is precisely the thing that gets fooled, because a polished-but-false final message is designed (by the model that wrote it) to read as success.

This is the gap: your evals grade the story the agent tells about itself. Your bill reflects what actually happened.

The fix is independence, not a smarter judge

The reflex when evals miss something is to reach for a better judge — a bigger model, a finer rubric. That's the wrong axis.

The tool I built for this, agent-eval, is built on one principle: rank your evidence by how hard it is for the agent to forge, not by how expensive it is to compute. Three tiers, on an independence axis:

• Tier 1 — externally observable. Did it crash? Did it stall past a timeout? Does its self-reported "done" match the orchestrator's recorded exit status? The agent cannot forge a JSON parse failure or a real timeout. This is proof.
• Tier 2 — statistically observable. Compared against a baseline the agent didn't author: did the diff actually change anything? Is the output repeating itself? Is it on-topic vs. the task embedding?
• Tier 3 — model-as-judge. A shared-substrate opinion. Useful for the ~20% subjective-quality tail. A signal, never a verdict — and it never belongs in the real-time gate.

That abandoned run fails at Tier 1, instantly, for free, before a judge ever runs. Not because a model decided the output was bad — because the recorded execution says the run abandoned, and no amount of confident final prose can overwrite a timeout the harness actually logged.

Here's what that check looks like against a live run:

import {
  runTiered, tier1, tier2,
  toBeNonEmpty, toNotBeAbandoned, toNotBeStale, toHaveMeaningfulDiff,
} from 'agent-eval';

// `result` came from actually running the agent (via AgentProvider),
// so alongside the final text it carries the real run record:
// tool calls, timing, and exit status.
const tiered = await runTiered(
  result.output,                 // what the agent said it did
  [
    tier1(
      toBeNonEmpty(),            // did it produce anything?
      toNotBeAbandoned(),        // did it FINISH, or die mid-task and get retried?
      toHaveMeaningfulDiff(),    // did the "fix" actually change a file?
    ),
    tier2(
      toNotBeStale(),            // fresh work, or a stalled re-emit?
    ),
  ],
  { prompt: task, metadata: { run: result } },  // the run record the checks judge against
);

// Tiers run in order and SHORT-CIRCUIT: if Tier 1 fails,
// you never pay for Tier 2 or a judge. The gate is deterministic,
// offline, and ~$0 — so it can actually block a run in the hot path.
if (!tiered.passed) {
  throw new Error(`Run failed independent checks: ${tiered.summary}`);
}

toNotBeAbandoned() is the one that saves the money. It doesn't ask a model whether the run looks done. It reads the run's actual outcome.

Finding the money you've already burned

Prevention is one half. The other is triage: across a fleet of past runs, which ones failed expensively, worst first?

agent-eval ships a fleet-triage pass that ranks broken runs by burned tokens and projected dollar cost, straight off raw session logs — abandoned, timed-out, and runaway runs surfaced by how much they cost you, not by when they happened. It's a plain function call, deterministic and free (no judge involved):

import { triageSessions, renderTriageTable } from 'agent-eval';

// Walk a directory of raw agent session logs, classify every run that
// didn't end cleanly (abandoned | timeout | runaway | stalled | errored),
// and project the dollar cost of the tokens each one burned.
const report = triageSessions('./logs', {
  dollarsPerMillionTokens: 9,     // your blended input+output rate
  costlyTokenThreshold: 200_000,  // ignore trivial instant-errors
});

// Worst-first table: kind, tokens burned, projected $, one-line reason.
console.log(renderTriageTable(report, 10));

Each row carries a projectedCostUsd and a costly flag, so you sort the wreckage by money instead of by timestamp. The failures that cost the most are rarely the ones you'd guess — they're the quiet runs that read as completed in the dashboard but stalled or got retried underneath. Green everywhere. Bleeding money.

The trace is what makes the signal trustworthy

There's a reason the checks above can be trusted, and it's the second half of the workflow. AgentLens captures the trace of how the agent got to its answer — every model call and tool step, resolved inputs, raw outputs, tokens, timing, cost. Two things fall out of that:

1. When a Tier-1 check fails, you can replay the run and see exactly where it abandoned — you're debugging a timeline, not guessing from a final message.
2. The tier checks have unforgeable, agent-didn't-author data to score against. The harness-recorded execution record (status, duration, abandon markers) is passed as objective evidence and deliberately kept out of the deliverable — so when you do reach for a Tier-3 judge on the ~20% tail, its execution_integrity criterion means a recorded timeout cannot score pass, no matter how clean the output reads.

agent-eval scores and gates the output. AgentLens captures the trace. You need both, because a gate you can't debug gets ignored, and a trace you don't gate on is just expensive logging.

The takeaway

If your agents run unattended and touch a metered API, stop asking "was the output good?" and start asking "did the run actually do the thing, and what did it cost?" Those are different questions, and only the second one shows up on your bill.

Concretely:

• Gate on independent evidence (crash / stall / abandon / real diff) before you ever pay a judge.
• Triage your fleet by burned dollars, not by recency — the expensive failures are the quiet green ones.
• Keep a trace so every failed gate is debuggable and every judge has ground truth it can't forge.

Both tools are open source (agent-eval, TypeScript; agentlens, Python + browser dashboard) and I run them against my own production fleet — continuous trace capture with AgentLens, a weekly calibrated eval gate with agent-eval.

---

I do a small number of eval-pipeline teardowns each month — if you're running agents in production and suspect green dashboards are hiding burned spend or silent abandons, reply or DM me and I'll take a look at yours.

Comments

[Max Quimby]

"Operational metrics are evals, not monitoring afterthoughts" — I want to put that on a wall. The dimension I'd add to your three is tool-call count, tracked separately from tokens. The nastiest regressions we've seen come from a one-line prompt nudge — "be thorough," "double-check your work" — that leaves correctness flat and the model-judge score identical, but quietly turns a 2-step task into a 9-step tool-calling spiral. Tokens go up as a symptom, but the count is the cleaner signal because it's where the latency and the failure surface both live. The thing I keep wrestling with is thresholds: absolute ceilings ("< 3000 tokens") drift the moment your golden set grows, so we ended up gating on regression-vs-baseline instead. How are you setting the line in agent-eval — a fixed budget per case, or a delta against the previous release's trace?