Your multi-agent run just returned a perfect answer. Clean summary, right resources, no errors. Your APM dashboard (the application performance mon...
For further actions, you may consider blocking this person and/or reporting abuse
200 OK with a silent 1.4× bill is the cost twin of “exit 0 with an empty payload.”
If the eval contract only watches success shape (right answer, green APM), multi-agent systems will optimize for looking done while burning nested calls you never intended. Per-agent cost isn’t vanity observability — it’s part of the quality contract.
The check I’d pin next to task success: cost-per-successful-outcome by agent role, with a hard fail when the run is “correct” but outside the agreed spend envelope.
exactly this is the direction i had in mind as well
a run being technically successful doesnt mean it was efficient or healthy cost per successful outcome by agent role would give a much better signal especially when you start running these workflows at scale i also like the idea of treating the spend envelope as part of the eval itself rather than checking cost separately
Agreed — and the interesting edge case for me is when one agent is green on cost-per-success while another is the silent 1.4× branch. Role-level envelopes catch that; a single crew-total budget can still hide a noisy specialist.
Branch-level pause (as floated elsewhere in the thread) is the runtime twin of that eval cut: stop the noisy role without killing the whole run.
This is really helpful. the way you put hands on video along with detailed article is really helpful.
Thank you so much for your kind words 💯
Is this open source tool?
Yes its open source here is github url - github.com/traccia-ai/traccia-py
The whole article hangs on "billed ~1.4x," but that's the worst of your three cases (1.03x / 1.2x / 1.4x), and the absolute delta is sub-penny ($0.0083 → $0.0119). A skeptical reader does that math and the drama deflates.
Fix the lede to match the honesty of the body: lead with the range, not the ceiling "1.03x to 1.4x, and the cheapest-looking overspend is the one you'll never catch." That pivots tension to the 1.03x case (the genuinely novel insight) and keeps your biggest asset intact: credibility.
Cost visibility is the missing control plane for multi agent systems. I like the focus on silent waste because successful runs can still hide duplicated retrieval and runaway context growth. A useful next step would be a budget alert that pauses only the noisy branch while the rest of the workflow continues.
Exactly a branch level budget guard would be much more useful than stopping the entire workflow detect the noisy agent pause or limit that branch and let the other agents continue.
im curious how youd implement that control at the agent runtime level or through the observability layer?
Love the detailed and hands-on exploration!
Thank You So Much Aditya 😇
I work on agent-inspect, so the distinction between a correct answer and an inefficient trajectory really resonates. The
llm.modelsilent-zero gotcha is exactly the sort of instrumentation failure a baseline should catch. Since your supervisor routes cost-only requests to fewer specialists, would you key the spend baseline by task type and delegation set (plus model/pricing version)? Otherwise a legitimate full-account sweep could look like context bloat relative to a narrower run.Tracking cost per agent rather than globally across the workflow is honestly the only way to catch silent token inflation early. In our multi-agent pipelines, we noticed intermediate routing and evaluation agents often eat 60%+ of the total token budget during retry or handoff loops without producing direct user-facing value. Attaching the trace/span ID down through each subagent invocation makes pinpointing which specific agent drifted way faster.
The “same answer, different bill” part really caught my attention.
Traditional monitoring makes it very easy to think that a successful request is a healthy request. If it returns 200, the latency looks fine, and there are no errors, everything appears green. With multi-agent systems, that can hide a completely different problem: the system may have taken extra reasoning cycles, repeated tool calls, or carried unnecessary context through the workflow.
I also like the decision to track cost at the agent level instead of treating the whole run as one number. Once you know which agent caused the increase, cost stops being just a finance metric and becomes a debugging signal.
The part about changing the instrumentation and accidentally breaking the detection logic was probably my favorite detail. That's exactly the kind of problem that tends to show up in a real system and never makes it into the clean demo.
A correct answer tells you what the system produced. The trace tells you what it took to produce it. That distinction is becoming pretty important for agentic systems.
The 1.4x overbilling problem is real and matches what I have seen in multi-agent setups. The MAST paper finding that failures do not crash but complete while burning money is the key insight. I started tagging every trace with a cost-per-task metric, and it immediately showed me which agent was the bottleneck. How do you handle the case where one agent's "successful" output is actually expensive garbage that the next agent has to clean up?
That case is the cost twin of “success-shaped empty”: the upstream agent’s span looks green while the cleanup cost lands on whoever has to repair it.
I’d attribute the downstream repair tokens back to the producer role in the eval, and fail the run when producer cost-per-usable-handoff blows the envelope — even if its own task-success bit flipped true.
Otherwise per-agent cost still hides the specialist that manufactured expensive garbage.
tracing after the fact still means the money is already spent by the time the pattern shows up across runs. the cheaper fix is upstream: route the boring steps (tool selection, retries, formatting) to a small cheap model and save the expensive one for the actual reasoning step, so the waste never accumulates in the first place instead of getting caught later in a trace.
The context bloat example is interesting. People usually look at output tokens when trying to optimize LLM costs, but the prompt side can quietly grow too.
Nice catch on llm.model. Silent failures like this are probably worse than an obvious error because everything looks like it is working.