Your multi-agent run just returned a perfect answer. Clean summary, right resources, no errors. Your APM dashboard (the application performance mon...
For further actions, you may consider blocking this person and/or reporting abuse
This is a really useful example of something I've been thinking about as "tax reporting" for AI systems.
I've been using a tax metaphor for costs such as Context Tax, Retrieval Tax, Observer's Tax, and Ingestion Tax. The important part isn't simply that those costs exist. It's whether the architecture makes them attributable enough to see where you're actually paying them.
Your context-bloat example captures that nicely. Both runs can be functionally successful, but one carries a 43% higher bill. At the system level, that's just "AI got more expensive." At the per-agent/per-step level, you can start asking why: Did an agent carry unnecessary context? Re-read something it already had? Take an extra reasoning cycle? Call a tool redundantly?
That's where I think the accounting analogy becomes useful. Knowing the total tax bill is interesting. Having enough reporting to identify which architectural behavior incurred which tax is actionable.
I'd be particularly interested in taking this one step further and reporting the costs by architectural category rather than only by agent. If context bloat increases both the worker's cost and the supervisor's downstream synthesis cost, for example, there's a kind of tax propagation happening across the trace.
Really interesting work. This is much closer to the kind of observability I think AI systems need than another dashboard showing aggregate token consumption.
this is a great framing and honestly the tax metaphor clicks better than what i used in the video. context tax retrieval tax observers tax ingestion tax that maps almost one to one onto the spans you end up staring at
you nailed the real point too it is not that the cost exists it is whether the architecture lets you attribute it. the 43% run looked perfectly healthy at the system level 200 ok correct answer done. only at the per agent per step level could i even ask the why did the worker carry context it did not need did it re read something it already had did it burn an extra reasoning cycle did it call the same tool twice
the tax propagation idea is the part i had not thought about clearly. you are right that context bloat on the worker does not stay on the worker it inflates the supervisors downstream synthesis cost too so one bad decision shows up as tax in two places along the trace. reporting by architectural category instead of only by agent would surface that. right now i tag cost per span and per agent but not per category so i cannot yet say retrieval tax was x acrossthe whole run. that feels very buildable though the span attributes are already there i would just need to stamp a category on each span and roll up by that instead of by agent
going to sit with this. thanks for taking the time this is a sharper way to think about it than aggregate token dashboards. quick question for you when you split it into context tax retrieval tax and so on do you assign each span to exactly one category or can one span carry more than one tax at once i keep going back and forth on whether a single tool call can be both retrieval tax and context tax and i am curious how you draw that line?
I wouldn't make them mutually exclusive. I think doing that would make the accounting cleaner but the architecture less truthful.
A retrieval span is a good example. Searching, ranking, and fetching is Retrieval Tax. But if that operation returns 20K tokens that get hydrated into the next model call, it has also caused Context Tax downstream. If those records are unnecessarily verbose, Prose Tax may also contribute to that Context Tax.
So I'm starting to think there are at least two useful dimensions here: where the cost was incurred and what architectural behavior caused or contributed to it.
I might therefore let a span carry multiple tax attributes, but avoid double-counting the actual dollars. Something like:
span cost = $0.004primary tax = retrievalcontributes_to = contextcause = excessive_candidatesThen if the supervisor's next model call costs an additional $0.002 because of that retrieved material, that's a separate Context Tax charge with lineage back to the retrieval span.
That would let the report say not only "this run incurred $X in Retrieval Tax and $Y in Context Tax," but potentially "this retrieval decision caused $Z of downstream Context Tax."
I haven't implemented this taxonomy as an accounting system yet, so I'd treat that as a design hypothesis rather than a settled schema. But your question is making me think the lineage between taxes may be as important as the categories themselves.
And now you've got me wanting to build it too. :)
this is exactly the direction i was hoping you would push it and the mutually exclusive vs truthful point is the thing i had wrong in my head. i was trying to force one span into one bucket because it made the sums clean but you are right that throws away how the cost actually happened
the two dimensions split is what makes it click for me. where it was incurred is just the span itself but what caused or contributed to it is the interesting axis and that is the part no dashboard shows today. your schema is basically what i want primary tax on the span for the dollars contributes_to for the propagation and cause for the why. keeping the actual dollars single counted but letting the attributes be many solves the double count problem i kept tripping on
the lineage bit is the real unlock though. this retrieval decision caused z of downstream context tax is a completely different sentence than this run cost x. one is a receipt the other tells you what to go fix. and since the spans already sit in a parent child trace the lineage is kind of already there i would just need to carry a ref from the downstream context charge back to the span that caused it
treating it as a design hypothesis is fair i have not built it either. but you have basically talked me into prototyping it. if i stamp primary_tax contributes_to and cause on the strands spans and roll up by both dimensions i think i can get a first version out
so here is my next question when you attribute that downstream z back to the retrieval span how do you decide how much of the supervisors context tax to blame on that one retrieval vs everything else it was already carrying do you split it proportionally by token share or do you just attribute the whole
delta to the thing that changed
I think I'd distinguish measured attribution from allocated attribution here.
If I have a controlled comparison where the only meaningful change is the retrieval result, I'd prefer the delta. Supervisor costs $0.006 without that retrieval and $0.009 with it, so I have pretty good evidence that retrieval span contributed roughly $0.003 of downstream Context Tax.
In a normal production trace, though, I probably don't have that counterfactual. The supervisor is carrying system instructions, conversation state, outputs from other agents, retrieved material, tool results, etc. In that case, proportional token share seems like a reasonable allocation method, but I'd want the receipt to say that's what it is rather than presenting it as measured causation.
So perhaps the attribution itself needs provenance:
attribution_method = measured_delta | proportional | estimated | unknownThat would let you say something like, "retrieval span 104 contributed an estimated $0.0021 of downstream Context Tax, allocated proportionally by hydrated token share," without claiming more precision than the trace actually supports.
And I suspect there are cases where the honest answer should just be
unknown. Context isn't necessarily additive. An extra 5K tokens might change caching, reasoning behavior, or even how much output the supervisor generates, so token share and cost share aren't always going to map cleanly.Your parent-child trace observation is interesting too. If the causal lineage is already structurally present, this may mostly be a matter of making the attribution semantics explicit rather than inventing a whole new tracing mechanism.
Which means you've now gotten me wondering whether a Tax Report needs to report not only the amount and category, but the confidence/provenance of the attribution itself. Apparently, even the tax bill needs provenance. :)
this is the distinction i was missing and it is the right one. measured vs allocated is doing a lot of work here. if i have the counterfactual the supervisor at 0.006 without the retrieval and 0.009 with it then the 0.003 is real evidence. in a normal production trace i do not have that clean a b the supervisor is carrying system instructions conversation state other agents output retrieved material tool results all at once so the honest move is proportional allocation but labelled as allocation not measurement
attribution_method = measured_delta | proportional | estimated | unknown is going straight into the schema. that one field is what keeps the whole thing honest because it stops a proportional guess from masquerading as measured causation. so the receipt reads retrieval span 104 contributed an estimated 0.0021 of downstream context tax allocated proportionally by hydrated token share and every word in that sentence is now defensible and you are right that unknown has to be a real allowed value not a cop out. context is not additive an extra 5k tokens can flip caching change the reasoning path or change how much the supervisor generates so token share and cost share do not always line up. pretending otherwise would be the exact thing i am trying to avoid
the part that makes this feel buildable and not a research project is your last point. the causal lineage is already structurally there in the parent child spans so i am not inventing a tracing mechanism i am just making the attribution semantics explicit on top of what the trace already records. that reframes the whole thing from build a new system to stamp three more attributes and be honest about the confidence
and yeah you landed it the tax bill needs provenance too. amount category and how sure we are about who to blame. that confidence field might end up being
the most useful column in the whole report
alright i have to actually build this now. last one for you would you surface attribution_method right in the main report next to every line or keep it as drill down metadata i lean toward showing it inline because a proportional estimate and a measured delta are not the same claim and hiding that feels like the same sin as the aggregate dashboards we started out complaining about
I work on agent-inspect, so the distinction between a correct answer and an inefficient trajectory really resonates. The
llm.modelsilent-zero gotcha is exactly the sort of instrumentation failure a baseline should catch. Since your supervisor routes cost-only requests to fewer specialists, would you key the spend baseline by task type and delegation set (plus model/pricing version)? Otherwise a legitimate full-account sweep could look like context bloat relative to a narrower run.probably the sharpest question here and youre right. in the article the baseline is per agent not per run so each agent is compared to its own known good. that dodges part of it a cost only run just never starts health_ops or security_ops so they cant get flagged. but inside one agent the case you name is real a full sweep genuinely feeds health_ops more context than a narrow ask so against a lean baseline it looks like bloat when nothings wrong. so yeah the key has to come from the request side not the trajectory.
keying by which agents ran is the wrong move a run that wandered ends up compared to other wandering runs. and good catch on pricing version that has to be in the key or every price update reads as a regression. the delegation set is an output of the request not the label. how does agent-inspect key it do you classify the request into a class up front or compare against a rolling per class distribution. the new class with no history is the bit i havent figured out.
Tracking cost per agent rather than globally across the workflow is honestly the only way to catch silent token inflation early. In our multi-agent pipelines, we noticed intermediate routing and evaluation agents often eat 60%+ of the total token budget during retry or handoff loops without producing direct user-facing value. Attaching the trace/span ID down through each subagent invocation makes pinpointing which specific agent drifted way faster.
the 60% on routing and eval agents matches what i saw the orchestrator is quietly expensive because it writes the long synthesis and in the bloat case the supervisor cost climbed just from a bigger blob being handed up to it so the waste cascades from a noisy child into the parent. an agent that produces no user facing output can still be the most expensive one in the run which is exactly why crew total hides it. on the span id piece thats the part that bit me. i wanted each agent as its own trace so a fleet reads like a fleet but strands runs each subagent in its own event loop on a detached context so my spans kept collapsing into one until i detached the otel context myself and tied the agents together with a shared session.id instead of a parent child chain. so the correlation key here is session.id across separate traces not a single nested trace. handoff loops would show up as the same agent repeating within that session. how are you carrying the id today one nested trace for the whole pipeline or separate traces stitched by a correlation id? im curious which one holds up better once retry loops get deep.
We use separate traces stitched by a correlation ID (
session_id+ run ID), exactly like you described.Nested parent-child traces broke down for us as soon as an agent failed or retried—a single nested trace either bloated to unreadable depths or got severed if a subagent died unexpectedly. With flat traces linked by a correlation ID, each subagent execution is an independent unit of work with its own token/cost metrics. We just tag every subagent trace with
session_id,turn_index, andattempt_count.When retries kick in, you see distinct trace entries under the same session ID rather than a mutated span tree, which makes visualizing loops and cost runaways way cleaner.
200 OK with a silent 1.4× bill is the cost twin of “exit 0 with an empty payload.”
If the eval contract only watches success shape (right answer, green APM), multi-agent systems will optimize for looking done while burning nested calls you never intended. Per-agent cost isn’t vanity observability — it’s part of the quality contract.
The check I’d pin next to task success: cost-per-successful-outcome by agent role, with a hard fail when the run is “correct” but outside the agreed spend envelope.
exactly this is the direction i had in mind as well
a run being technically successful doesnt mean it was efficient or healthy cost per successful outcome by agent role would give a much better signal especially when you start running these workflows at scale i also like the idea of treating the spend envelope as part of the eval itself rather than checking cost separately
Agreed — and the interesting edge case for me is when one agent is green on cost-per-success while another is the silent 1.4× branch. Role-level envelopes catch that; a single crew-total budget can still hide a noisy specialist.
Branch-level pause (as floated elsewhere in the thread) is the runtime twin of that eval cut: stop the noisy role without killing the whole run.
The “same answer, different bill” part really caught my attention.
Traditional monitoring makes it very easy to think that a successful request is a healthy request. If it returns 200, the latency looks fine, and there are no errors, everything appears green. With multi-agent systems, that can hide a completely different problem: the system may have taken extra reasoning cycles, repeated tool calls, or carried unnecessary context through the workflow.
I also like the decision to track cost at the agent level instead of treating the whole run as one number. Once you know which agent caused the increase, cost stops being just a finance metric and becomes a debugging signal.
The part about changing the instrumentation and accidentally breaking the detection logic was probably my favorite detail. That's exactly the kind of problem that tends to show up in a real system and never makes it into the clean demo.
A correct answer tells you what the system produced. The trace tells you what it took to produce it. That distinction is becoming pretty important for agentic systems.
thanks for reading it that closely. the instrumentation breaking the detection is the bit im most glad i left in cause it wasnt planned. i switched the tools to emit one live span per call to get real durations and that quietly broke the redundant call check it had been reading a call_count off a single reconstructed span that didnt exist anymore. the baseline caught it. which is kind of the whole point in miniature the thing watching for drift drifted and only the baseline noticed. and yeah cost as a debugging signal not a finance number is the reframe. once you know which agent moved it its a stack trace nota bill. good read thanks.
That “the thing watching for drift drifted” part is probably the most interesting takeaway for me. It creates a nice second-order problem: observability itself becomes part of the system that needs validation.
You can have the application behaving correctly, the cost trace looking reasonable, and still have the detection logic quietly stop seeing the behavior it was supposed to detect. So the baseline is doing more than detecting cost drift. It is also testing whether the measurement system itself is still trustworthy.
That feels especially important in agentic systems, because the workflow, instrumentation, and even the definition of a “normal” run can all change over time.
Building on the baseline-rot point above: before a baseline can rot it has to exist, and that means grouping runs into classes. The tempting key is the trajectory - which agents ran, which tools fired - and that is exactly the wrong one, because it blinds the detector in the case you care about: a run that wandered gets compared against other wandering runs instead of against what that request should have cost. The label has to come from the request side, which the agent does not choose. How are you keying runs today?
straight answer today its keyed at the agent level not the run level. each agent vs its own prior known good. so the baseline exists per agent not per run class. holds for the three patterns in the piece cause a cost only ask never runs the other agents so they cant be mis flagged. but youre pointing right at the hole. inside one agent theres no request side label so a real full sweep feeds health_ops more context and my delta check calls it bloat. and yeah keying by trajectory is self defeating a wandering run graded against other wandering runs means the detector goes blind exactly where you need it. label has to be the thing the agent doesnt pick the request or intent not its path. so per agent works now request side classing is the next thing and its not in the repo yet. how are you getting the request class fixed task type on the way in or inferred. and what do you seed a fresh class with?
Sarvar, the baseline-vs-delta design is the right instinct, but I'd push on the baseline itself. A single captured CLEAN run is one sample of a non-deterministic process, so a fixed 1.25x prompt-token threshold is partly measuring model variance, not just waste. Two genuinely clean runs could differ by a meaningful chunk just from the model phrasing its reasoning differently, and a legitimately harder question needs a real extra cycle sometimes. Have you run the CLEAN baseline itself multiple times to see how much it naturally drifts, or is one captured run being treated as ground truth for now? The llm.model vs llm.request.model silent-zero bug is a great catch on its own - a processor that fails closed with no log is exactly the kind of thing that costs an afternoon, and logging on skip is cheap enough there's no excuse not to add it upstream.
Mihai this is fair and honestly its the weakest part of the current setup one captured clean run is being treated as ground truth right now thats it one sample im not going to pretend otherwise you nailed the problem the 1.25x threshold is partly measuring nova phrasing its reasoning differently not waste i havent run the baseline 20 times and looked at the spread so i actually dont know how much a clean run drifts on its own which means i cant tell you how much of that 1.25x is real headroom vs me guessing a number that felt safe
the right version is what you said run clean n times build a distribution and flag on something like mean plus 2 sigma per attribute so the threshold comes from the actual variance instead of a single number i picked and the harder question needing a real extra cycle is the same issue a static cycle threshold punishes a legitimately bigger task the baseline should probably be per question class not one global run so no its not defensible as ground truth yet its a demo baseline the design is right the statistical rigor isnt there and pretending a single sample is a
baseline is exactly the trap im warning about in the article so thats a fair hit
on the silent zero yeah theres no excuse a fail closed processor that returns with no log cost me an afternoon of a zeroed dashboard while the spans clearly
had tokens one debug line on skip fixes it im flagging it upstream.
The 1.4x overbilling problem is real and matches what I have seen in multi-agent setups. The MAST paper finding that failures do not crash but complete while burning money is the key insight. I started tagging every trace with a cost-per-task metric, and it immediately showed me which agent was the bottleneck. How do you handle the case where one agent's "successful" output is actually expensive garbage that the next agent has to clean up?
same case igor raised and its the honest limit of what i built. my spans attribute cost to the agent that spent the tokens not the one that caused the spend. so the producer handing off garbage looks cheap and the cleanup agent looks expensive. backwards. the only thing i can see is the cascade in the bloat run a bloated child made the supervisors own synthesis cost go up from the bigger blob handed to it. so waste visibly moves across the handoff but i dont tie cause to effect yet. doing it right is data lineage between spans which agent produced the context that inflated the next ones input. not in the repo yet. the eval version igor floated attribute the repair tokens back to the producer and fail on producer cost per usable handoff is where id take it. how are you catching it now
That case is the cost twin of “success-shaped empty”: the upstream agent’s span looks green while the cleanup cost lands on whoever has to repair it.
I’d attribute the downstream repair tokens back to the producer role in the eval, and fail the run when producer cost-per-usable-handoff blows the envelope — even if its own task-success bit flipped true.
Otherwise per-agent cost still hides the specialist that manufactured expensive garbage.
This is really helpful. the way you put hands on video along with detailed article is really helpful.
Thank you so much for your kind words 💯
Your welcome
Is this open source tool?
Yes its open source here is github url - github.com/traccia-ai/traccia-py
Cost visibility is the missing control plane for multi agent systems. I like the focus on silent waste because successful runs can still hide duplicated retrieval and runaway context growth. A useful next step would be a budget alert that pauses only the noisy branch while the rest of the workflow continues.
Exactly a branch level budget guard would be much more useful than stopping the entire workflow detect the noisy agent pause or limit that branch and let the other agents continue.
im curious how youd implement that control at the agent runtime level or through the observability layer?
tracing after the fact still means the money is already spent by the time the pattern shows up across runs. the cheaper fix is upstream: route the boring steps (tool selection, retries, formatting) to a small cheap model and save the expensive one for the actual reasoning step, so the waste never accumulates in the first place instead of getting caught later in a trace.
youre right and its not either or. routing the boring steps tool selection retries formatting to a small cheap model is a real fix and it stops the waste before it piles up. but i wouldnt call tracing the alternative to it. tracing is how you find out which steps are the boring expensive ones worth routing down in the first place. you instrument see the orchestrator or a retry loop eating budget then move that work to a cheaper model. and after you do the trace tells you the cheap model didnt start looping and eat the saving back. so find it fix it upstream then keep watching. per step cost is what says whichmodel each step actually deserves
the MAST interannotator agreement at kappa=0.88 is the number that should anchor every budget conversation here. 41 to 86.7% failure rate on runs that log 200 OK is not a sampling artifact at that agreement level, it is a structural problem.
the silent waste pattern I keep hitting is the subagent that rereads the same context window because handoff fidelity is low. it does not error, it just does extra work. we saw it in AWS Bedrock traces before we added per span token counts. the cost attribution model you built is the missing piece most teams add last.
have you found that the reread pattern clusters by agent type, or is it more tied to tool call ordering?
the kappa 0.88 thing stuck with me too. hard to argue with 41-86 percent when the agreement is that high. and the reread you mention is basically my runawayloop case doesnt error just quietly redoes the same read. honest answer on your question though i cant really say it clusters by agent type my scenarios are engineered to fire on cue so id be making it up. from the few real runs i had it felt more like tool call ordering / bad handoff than the agent itself but thats a gut read not measured. what did you see in your bedrock traces????
The per agent view is the part that makes this actionable. Aggregate spend can look fine while one planner or retry loop quietly burns the budget. Are you also tagging latency and retry count beside token cost so teams can spot the expensive failure mode?
yeah exactly per agent is where it shows aggregate hides it and cycle count is already on the span so the retry loop is covered thats scenario 1 the runaway loop shows as more cycles not an error and latency is there too every tool span wraps the real call so you get true duration per agent. the one thing i havent done is a clean retry_count separate from cycles right now i infer it from cycles going up plus a tool called more than baseline a dedicated counter would be cleaner good call.
The whole article hangs on "billed ~1.4x," but that's the worst of your three cases (1.03x / 1.2x / 1.4x), and the absolute delta is sub-penny ($0.0083 → $0.0119). A skeptical reader does that math and the drama deflates.
Fix the lede to match the honesty of the body: lead with the range, not the ceiling "1.03x to 1.4x, and the cheapest-looking overspend is the one you'll never catch." That pivots tension to the 1.03x case (the genuinely novel insight) and keeps your biggest asset intact: credibility.
Love the detailed and hands-on exploration!
Thank You So Much Aditya 😇
The context bloat example is interesting. People usually look at output tokens when trying to optimize LLM costs, but the prompt side can quietly grow too.
yeah exactly. everyone stares at output tokens cause thats the obvious one but the prompt side just creeps up on you. thats the context bloat case in the article. output looked totally normal input tokens kept climbing. and it gets worse cause the bloated agent hands a bigger blob up to the supervisor so thatone pays more too. only caught it cause i was watching prompt tokens per agent. a single run total would never have shown it
Nice catch on llm.model. Silent failures like this are probably worse than an obvious error because everything looks like it is working.
yeah honestly the silent one is worse. an obvious error you go fix. this just sat there dashboard showing zero tokens while the spans clearly had tokens on them and i spent an afternoon thinking my own wiring was broken. turned out the cost processor just skips the span if llm.model isnt set. no log nothing. one debug line would've saved me the whole afternoon
The strongest point here is that a successful run is not necessarily a healthy run. Traditional APM is optimized around availability, latency, and errors, while agent systems can degrade without triggering any of those signals.
I’d take the cost attribution one step further and treat it as a causal debugging signal, not just a billing metric. If prompt tokens, tool-call counts, cycle count, and agent identity are captured together, you can ask why a run became more expensive instead of simply noticing that it did.
That’s especially useful when optimizing production agent workflows at IT Path Solutions: a 40% cost increase might be caused by context growth, redundant retrieval, an extra delegation hop, or a prompt change not necessarily a more expensive model. Without those dimensions on the same trace, the bill tells you there is a problem but not where the behavior changed.
The baseline-vs-delta approach is also important. Absolute thresholds can be misleading across different tasks; comparing an agent against its own known-good execution gives you a much stronger regression signal.
The delta-vs-baseline detector is the part I'd stress-test hardest. Baselines rot the moment prompts, models, or tool sets change — and in an agentic system they change constantly — so in practice this becomes a versioning problem: baselines keyed to prompt/tool/model versions, or rolling quantiles, or every deploy turns into false positives and the team mutes the alert. Also worth noting from your context-bloat scenario: the sub-agent's bloat shows up in the supervisor's bill too, so attribution by span alone can undercount the true originator of waste. Tracking which agent produced the context that inflated a sibling's input tokens would catch that.
That per-agent cost tracking is a really useful way to catch wasted tokens before they turn into a bigger AWS bill.