Sarvar Nadaf for AWS Community Builders

Posted on Sep 23 Edited on Sep 27 AI-assisted

Per-Agent Cost Tracking for Multi-Agent AI on AWS

#aws #observability #ai #discuss

Catches silent financial waste in successful runs

Your multi-agent run just returned a perfect answer. Clean summary, right resources, no errors. Your APM dashboard (the application performance monitoring you already run: uptime, latency, error rate) says 200 OK, latency fine, everything green.

And you were silently billed about 1.4x what you should have been.

That is the part nobody shows you. Nested traces and per-agent cost are becoming common; the primitives are easy to find now. What stays rare is a data model that lets you act on them: catch the run that looks completely successful while it burns money in the middle. The paper "Why Do Multi-Agent LLM Systems Fail?" (MAST, arXiv:2503.13657) hand-annotated 150 traces across 7 state-of-the-art multi-agent systems, hit an inter-annotator agreement of kappa=0.88, and measured failure rates from 41% to 86.7%. The uncomfortable finding: many of those failures do not crash. They complete. They look fine.

In this article I build a small read-only "AWS Account Investigator" crew, wire real cost into every trace span, and then reproduce three silent-waste patterns with real Amazon Nova Pro dollars. You can run the whole thing for $0 locally. Nothing gets created, modified, or deleted in your AWS account.

If you only have two minutes, jump straight to the unique part: catching silent waste. The build up to it matters, but that section is the payoff.

I spent about a week on this against a real AWS account: a few days probing the SDK's behavior before I trusted it, then several more building the crew, watching the trace design break twice, and reading the SDK source when the docs ran out. What follows is written from that, not from a quickstart. The scars are in here on purpose, because they are the part that saves you the week.

This is for people already building AI agents who have never put a real observability layer under them. You know agents, tools, and crews. Where the tracing vocabulary (spans, traces, OpenTelemetry) is new, I define it the first time it shows up.

Contents

Why agent observability is a different problem
Picking the instrumentation
Prerequisites
Adding Traccia to your code
The stack: AWS native and read only
The cost bridge and one gotcha
Modeling a multi-agent crew in traces
The unique part: catching silent waste
Watching it happen: the live control panel
Build your own, at zero cost and read only
An honest take on Traccia
Honest caveats
FAQ

Why agent observability is a different problem

Traditional application monitoring answers three questions: is it up, is it fast, is it erroring. For a CRUD service that is enough, because the work is deterministic and the failure modes are loud. An AI agent breaks all three assumptions. It decides its own control flow at runtime, it calls tools in an order you did not hardcode, and it pays per token for every reasoning step. A run can be up, fast, and error-free while doing the wrong amount of work: re-reading the same data, dragging bloated context from step to step, looping an extra cycle before it settles. None of that shows up as a 500 or a slow span. It shows up on the bill, and by then it is a trend, not an event.

So agent observability has to record things classic APM never needed: how many reasoning cycles an agent took, which tools it called versus which it was allowed to call, the token count and dollar cost of each step, and which agent in a multi-agent crew did what. Those attributes are what make an invisible regression visible.

This is not a fringe opinion. AWS's own Well-Architected Agentic AI Lens frames the baseline state (its "Level 1") as exactly this problem: agent costs are visible only at the account level, Cost Explorer cannot separate agents or workflows, and "teams react to billing surprises after the fact because per-agent and per-reasoning-phase attribution is missing." The whole point of what follows is to move off Level 1: to make spending "attributable at the reasoning-cycle, agent, workflow, and tenant level rather than only at the account level," which is AWS's own words for the target.

A quick vocabulary anchor, since the rest of the article leans on it. A span is one timed step with attributes attached (one LLM call, one tool call, one AWS read). A trace is the tree of spans for one unit of work. Classic APM records spans too, but only the loud attributes (status, latency). Agent observability is the same trace structure carrying agent-specific attributes: cycle count, tokens, cost, and which agent owned the step. That is the whole idea; everything below is just putting the right attributes on the right spans.

At one run, a 1.4x overspend is a rounding error. At enterprise scale it is a budget line and a governance problem, and it shows up in four concrete ways:

Cost control. A 1.03x-to-1.4x silent overspend per run (the real range I measured across three waste patterns), multiplied across thousands of daily runs and dozens of agents, is real money leaking with no alarm attached. Per-agent, per-tool cost on the trace is the only way to attribute and cap it.
Accountability. When a crew misbehaves, "which agent, owned by which team, cost what" needs to be answerable. Trace-level ownership metadata turns a vague incident into a routed ticket.
Regression detection. Agents change when prompts, models, or tools change. A known-good baseline plus per-run deltas catches the day a prompt tweak silently doubled token usage, before finance does.
Auditability. In regulated environments you need a record of what the agent read, what it decided, and what it cost. A trace is that record.

The theme throughout: a correct-looking answer is not evidence of a healthy run. The evidence lives in the trace, on attributes you put there on purpose. Here is the before and after in one line. Before, a typical demo gives you one lump token count for the whole run, and an inefficient run looks identical to an efficient one. After, every reasoning step and every AWS read is a span carrying real cost, tokens, cycle count, and the owning agent's identity, so two runs that both return the correct answer and both show 200 OK are no longer indistinguishable when one of them costs 43% more.

Picking the instrumentation

Once you know you need per-agent cost, cycle counts, and tool-call attributes on every span, the next question is what to write them with. You could do a lot of this with raw OpenTelemetry, and I nearly did. The reason I did not is that agents need a vocabulary plain OTel does not ship: token counts turned into dollars, a span-level agent identity so one process can render as a real fleet, ownership metadata, and a way to view per-agent cost grouped by session. You end up building all of that yourself, or you find an SDK that already speaks it.

There are options here: LangSmith, Langfuse, and Arize Phoenix all do LLM tracing, and each is worth a look depending on your stack. I went looking for one I would trust in a codebase, which for me means two hard requirements: I can read the source, and I am not locked in. Traccia cleared both cleanly. The SDK is open source, Apache-2.0 licensed, and built on OpenTelemetry (OTel, the vendor-neutral open standard for traces and metrics, the reason you are not locked into any one backend). The spans it produces are standard OTel, the file exporter works with no account and no network, and I could read exactly what it does to my data before committing to it (I did, and the source-grounded critique later in this article is the result). It runs at $0 locally; the hosted dashboard at app.traccia.ai is optional and only comes in when you want the visualization. An open, inspectable SDK with an optional commercial backend is a split I am comfortable adopting, because the instrumentation does not trap me.

That is the real reason it is in this build: agent-native plumbing I did not want to hand-roll, source I could audit, and a real $0 offline path. It also has sharp edges, and I hit several of them; those are documented in full near the end rather than glossed over.

Why not Amazon Bedrock AgentCore Observability or Langfuse, the two obvious AWS-native alternatives? Both are good, and for many teams either is the right call. AgentCore Observability exports traces to CloudWatch and is the natural fit if your agents run on the AgentCore runtime, but AWS's own Well-Architected lens is blunt about the cost gap: "cost reporting stops at the AWS account level, so teams can't separate supervisor overhead from worker execution." Per-agent dollars are something you still assemble. Langfuse is the strong open-source incumbent and I would happily use it; it just was not the tool I was asked to put through its paces here. The point of this build is not "Traccia beats them." It is that whichever tracer you pick, the per-agent cost attribute and the baseline-delta detection are things you wire on purpose, and this article shows exactly how.

Prerequisites

Nothing exotic. Three things to run this yourself:

Python 3.10+ and the SDKs (strands-agents, strands-agents-tools, traccia, boto3). The repo pins the exact tested versions in requirements.txt.
AWS credentials with read-only permissions for the services the crew reads (Cost Explorer, EC2, CloudWatch, S3, Lambda, IAM, GuardDuty), plus bedrock:InvokeModel so the agent can actually call the model. The repo ships a ready-to-use policy at iam/read-only-policy.json; AWS's managed SecurityAudit + ViewOnlyAccess cover the reads, but you still add bedrock:InvokeModel on top of them.
Amazon Nova Pro, which is two separate steps: (a) enable model access once in the Bedrock console (us-east-1, amazon.nova-pro-v1:0) under Model access, and (b) allow bedrock:InvokeModel in your IAM policy. The console grant is not an IAM permission, so you need both.

No Traccia account is required. With no API key it writes traces to a local file, which is the $0 path used throughout this article.

Getting it running is four commands:

git clone https://github.com/simplynadaf/ai-agent-observability-aws.git
cd ai-agent-observability-aws
python3 -m venv .venv && . .venv/bin/activate
pip install -r requirements.txt

Then create the least-privilege policy once (with your own admin credentials) and attach it to whoever runs the crew:

aws iam create-policy \
  --policy-name AgentObservabilityReadOnly \
  --policy-document file://iam/read-only-policy.json

Adding Traccia to your code

Before the crew, here is the smallest version of what "wire cost into a span" actually means, because that is the one non-obvious step. Traccia auto-instruments LangChain, CrewAI, and the OpenAI/Anthropic/Gemini clients, so on those stacks you get most of this for free. It does not yet hook Strands or raw Bedrock, so you stamp the cost yourself. It is a short function, and one attribute name will bite you (more on that below).

def stamp_llm_cost(span, result, model_id="amazon.nova-pro-v1:0"):
    usage = result.metrics.accumulated_usage
    in_tok, out_tok = usage["inputTokens"], usage["outputTokens"]
    cost = (in_tok / 1000 * 0.0008) + (out_tok / 1000 * 0.0032)  # Nova Pro, us-east-1

    span.set_attribute("llm.model", model_id)   # REQUIRED. wrong key = silently zero
    span.set_attribute("span.type", "LLM")
    span.set_attribute("llm.usage.prompt_tokens", in_tok)
    span.set_attribute("llm.usage.completion_tokens", out_tok)
    span.set_attribute("llm.cost.usd", round(cost, 6))

That is the whole idea: read the token usage the SDK already gives you, turn it into dollars against real pricing, and attach it to the span. Everything else in this article is applying this same move across a multi-agent crew and then reading the numbers back. The full wiring (init, the per-agent identity, the tool spans) is in src/crew.py in the repo.

The stack: AWS native and read only

The crew runs on Amazon Nova Pro (amazon.nova-pro-v1:0) through AWS Strands Agents using the agents-as-tools pattern. A supervisor named investigation_run delegates to three specialist sub-agents. Each specialist is a real separation of concerns, owns several read-only tools, and every one of those tools opens its own live span, so in the dashboard you see the agent, then each AWS read nested under it with a real duration. Here is the full fleet and exactly what each agent does.

1. AWS Account Investigator (supervisor, `investigation_run`)

The orchestrator. It does not touch AWS directly; it reads the user's question, decides which specialists are in scope, delegates to them, and synthesizes one report. On its trace span it records agent.delegated_to (which specialists it called this run), and it stamps the shared session.id that ties the whole investigation together.

The delegation is intent-routed, not fan-out-everything. The supervisor's instructions are strict: call ONLY the specialist whose domain the user actually asked about. Ask only about cost and it delegates to the Cost Analyst alone, while Health & Ops and the Security Auditor never run. Ask only about security and only the Security Auditor fires. Only a whole-account question ("what's running, any risks, and where is my spend going?") lights up all three. This matters for the trace and the bill: agent.delegated_to shows exactly which specialists ran, and a scoped question costs a fraction of a full sweep because the agents you did not need never spent a token. You can watch this live in the control panel: a cost-only prompt lights up one agent and leaves the other two idle.

Task	Tool	AWS read-only API
Plan + delegate	`cost_analyst`, `health_ops`, `security_ops`	none directly (delegates)

The supervisor's trace in Traccia: agent.delegated_to records which specialists ran this run, and the shared session.id links the four agents into one investigation.

2. Cost Analyst (`cost_analyst`)

A read-only FinOps specialist. It builds a full spend picture with three tools, and all three show up as separate tool spans in its trace.

Task	Tool	AWS read-only API
Month-to-date total, month-end forecast, top 5 services	`cost_forecast`	`ce:GetCostAndUsage`, `ce:GetCostForecast`
Last full month's total for month-over-month change	`last_month_cost`	`ce:GetCostAndUsage`
Daily cost series to catch a spike	`daily_cost_trend`	`ce:GetCostAndUsage`

What it reports on a real run: actual MTD spend, the account's forecasted month-end total, the top services by spend, the month-over-month direction and rough percentage, and the single most expensive day compared against the daily average (a possible spike).

The Cost Analyst trace: three Cost Explorer tool spans nested under the agent, each with its real read duration and the agent's own per-agent cost. This per-agent cost figure is exactly what turns a "successful" run into a caught overspend later.

3. Health & Ops (`health_ops`)

A read-only SRE specialist. It inventories the account and reads health signals with five tools, so it is usually the heaviest agent on input tokens (it chains the most reads).

Task	Tool	AWS read-only API
List running EC2 instances	`running_instances`	`ec2:DescribeInstances`
Read CPU utilization per instance	`cpu_utilization`	`cloudwatch:GetMetricStatistics`
Find unattached (idle) EBS volumes	`list_volumes`	`ec2:DescribeVolumes`
Inventory Lambda functions	`list_functions`	`lambda:ListFunctions`
Inventory S3 buckets	`list_buckets`	`s3:ListAllMyBuckets`

What it reports: running instances with their CPU, an inventory of volumes, functions, and buckets, and any notable health finding such as an unattached EBS volume.

The Health & Ops trace: five read-only tool spans and, on this run, the highest token count of the fleet (4,772) because it chains the most reads.

4. Security Auditor (`security_ops`)

A read-only security specialist. It runs four independent checks, each its own tool span.

Task	Tool	AWS read-only API
Security groups open to the internet (0.0.0.0/0)	`open_security_groups`	`ec2:DescribeSecurityGroups`
MFA gaps on the root account and IAM users	`mfa_findings`	`iam:GetAccountSummary`, `iam:ListUsers`, `iam:ListMFADevices`
S3 buckets missing a public-access block	`public_s3_buckets`	`s3:ListAllMyBuckets`, `s3:GetPublicAccessBlock`
Whether GuardDuty is enabled	`guardduty_enabled`	`guardduty:ListDetectors`

What it reports: each finding stated plainly with its risk, and it explicitly says so when a check comes back clean.

The Security Auditor trace: four independent read-only checks, each its own tool span with a real duration.

Each specialist stamps its own identity onto its trace span, so from a single crew run the dashboard shows four distinct agents with their own token and cost profiles, not one agent logged four times. Each agent runs as its own top-level trace, tied to the others by a shared session.id, and carries production ownership (type, owner, team) from a catalog file. Every call is a describe or get. There is no create, no modify, no delete. The worst thing this agent can do is read a bit too much, which, as you will see, is exactly the waste we want to catch.

Observability comes from Traccia, an OpenTelemetry-native agent-observability SDK.

pip install traccia

Runs $0 by default using a local file exporter. If you set TRACCIA_API_KEY, it pushes spans to app.traccia.ai. No key, no network, no cost. (The repo pins the exact tested version in requirements.txt; the prose stays unpinned so it does not age.)

Nova Pro pricing, pulled live from the AWS Price List API (effective 2026-08-01, us-east-1):

Token type	Price per 1K
Input	$0.0008
Output	$0.0032

Every dollar figure below is computed from real token counts against these two numbers.

The cost bridge and one gotcha

Here is the part most tutorials skip. Traccia auto-instruments several stacks out of the box (LangChain including BedrockChat, CrewAI, OpenAI Agents, and raw OpenAI/Anthropic/Gemini), and it ships a cost engine with a bundled pricing snapshot that covers Nova and Claude. But there is no Strands integration yet, and it does not hook raw Bedrock converse calls. So for this specific stack, Strands agents-as-tools calling Bedrock directly, you wire the cost in yourself. That is a fair amount of the value proposition for supported frameworks arriving for free, and real manual work for an unsupported one.

Strands hands you the token usage after a run. You read it, compute the cost, and stamp it onto the span, which is exactly the stamp_llm_cost function from earlier. About 40 lines once you handle all four agents and the tool spans; the full version is in src/crew.py.

The gotcha cost me a confused afternoon, and reading the SDK source explained exactly why. Traccia's cost-annotating processor only computes cost for a span when three things are all true: span.type is LLM (or unset), an llm.model attribute is present, and both token counts are set. Miss any one and the processor simply returns, with no error and no warning. I first set llm.request.model (which felt more semantically correct) instead of llm.model, so the processor silently skipped every span, and the "LLM Calls" and "Total Tokens" tiles read zero while my spans clearly had tokens on them. Set llm.model, and the tiles light up. The forgiving fail is reasonable; the fact that it is invisible is the trap. A one-line debug log ("skipping cost: no llm.model") would have saved the afternoon.

No double-counting across agents

The other question that always comes up: if the supervisor calls two sub-agents, and I sum everyone's tokens, am I counting the sub-agent tokens twice?

I wrote probes/probe_doublecount.py to check instead of guessing. Strands runs each sub-agent in its own event loop with its own metrics object. A supervisor's accumulated_usage is exclusive of its sub-agents' tokens. So the arithmetic is clean:

crew total = supervisor + sum(sub-agents)

No subtraction, no overlap, no double-count. Verified, not assumed.

Modeling a multi-agent crew in traces

Once the bridge is in, you get per-agent cost. But here is a design decision worth being explicit about, because most demos hand-wave it: how do you model a supervisor and its specialists in a trace?

You have two reasonable options. You can nest everything under one trace (supervisor is the root, sub-agents are child spans). Or you can give each agent its own top-level trace and tie them together with a shared session.id. I went with the second, because it is what a real production fleet looks like: the Cost Analyst, Health & Ops, and Security Auditor are independently owned, independently operated services. On the Traces page they show up as their own executions, each with its own cost, tokens, and duration; "Group by session" folds them back into one investigation when you want the whole picture.

session 4f4c1ade...  (one investigation, four independent traces)

  investigation_run   AWS Account Investigator   $0.006   delegated -> 3
  cost_analyst        Cost Analyst               $0.004
  health_ops          Health & Ops               $0.005   (highest total tokens: 4,772)
  security_ops        Security Auditor           $0.005
                                          crew total   $0.021

(These are the real per-agent figures from the exported run shown in the trace screenshots above, rounded to the dashboard's own cost tiles; they shift run to run with token usage. The crew total is the investigation_workflow roll-up span, which equals the supervisor's own synthesis plus the three sub-agents, no double-count. Health & Ops carries the highest token count because it chains the most reads, while the supervisor costs about the same because it writes the long final synthesis. The CLEAN and CONTEXT BLOAT numbers later in the article come from separate, labeled runs, so do not expect them to tie back to this one.)

Each agent's own trace still nests its tools underneath it (agent -> tool:running_instances -> the real boto3 call), so you keep the drill-down without pretending four separate services are one call stack. On the Traces page, "Group by session" folds all four agents from one run back into a single investigation, so you can move between the fleet view and the per-agent view without losing either.

(This is a different run from the CLEAN baseline used later; token counts and therefore dollars shift run to run. The point is the per-agent breakdown, not the absolute number.)

Good. Useful. Per-agent cost on its own is becoming common. The reason it matters here is not the number itself but the data model underneath it: once every step carries cost, tokens, cycle count, and an owning agent, you can build the thing that is still rare, which is catching a run that overspends while looking perfectly healthy. That is what the primitives let you build next.

A few things here are easy to get subtly wrong, and I hit them in roughly this order over a couple of days before the trace design held. First, all four agents come from a single crew run in a single process. Traccia bakes the agent identity into the OpenTelemetry resource at init, which is process-level, so my first version labeled every trace with one agent name: three identical "AWS Account Investigator" rows in the dashboard. Reading the SDK's enrichment processor showed that a span-level agent.id / agent.name attribute takes precedence over the process default, so stamping each agent's span with its own identity makes it show up as its own agent. Static ownership (type, owner, team, org) comes from an agent_config.json catalog the SDK auto-discovers, so the dashboard shows a real fleet with owners and teams, not four anonymous rows. No extra processes, no fake agents.

Second, separate traces need a real correlation key or they look disconnected. Every agent stamps the run's session.id, and the supervisor additionally records agent.delegated_to (which specialists it called this run). That is the explicit link that makes four independent traces read as one orchestrated investigation.

Third, the "each agent is its own trace" bit did not happen by wishing, and this one cost me a rebuild. Traccia's span_scope(parent=None) still inherits the current span if one is active, so my agents silently collapsed back into one trace until I detached the OpenTelemetry context before starting each agent's span. One small helper, verified by counting distinct trace IDs in the exported spans.

Fourth, the first time I looked at the timeline every tool span was 0ms, because I was reconstructing tool spans after the fact from the metrics object. The fix was to wrap the real boto3 call in a live span while it runs, so the timeline shows each AWS read's true duration. A 0ms bar is the kind of thing that makes a viewer distrust the whole trace, and it is worth chasing down. Then a subtler follow-on bit me: those live tool spans inherited the process-level default identity, so every tool bucketed under the supervisor and the specialists looked trace-thin. I had to stamp each tool span with its calling agent's identity too. Nothing about that was in the docs; I found it by parsing the exported traces.jsonl and noticing the agent.id was wrong.

One more touch that reads as production, not demo: each agent records both agent.tools_available (the full toolset it was granted) and agent.tools_called (what the model used this run). On this run, Cost Analyst had two tools available (month_to_date_cost and cost_forecast) and used one; Health & Ops had five and used all five. That gap is not a bug to hide, it is real information. "Has two, used one" is exactly the kind of thing you want visible when you are deciding whether an agent is over-provisioned.

The unique part: catching silent waste

Here is my disclaimer up front. I saw versions of all three of these in real runs while building the crew, then engineered them in src/waste_demo.py to trigger reliably so you can watch them on demand instead of waiting for a bad run. That reproduction is on purpose. LLM output is non-deterministic, so in production the same patterns show up on their own, just not on a schedule you can demo. And critically: detection here is delta-vs-baseline, not magic absolute thresholds. That is how real regression detection works. You capture a known-good run, then flag runs that deviate. Every number below is real Nova Pro token usage from a representative run. Your numbers will vary; the ratios are what hold.

First, the clean baseline. This is the "known good" I compare everything against.

CLEAN baseline (one exported run): crew total ~ $0.0083
  health_ops: 2,274 input / 199 output / 3 cycles

Scenario 1: The runaway loop

The agent gets stuck re-reasoning and re-reading the same things.

RUNAWAY LOOP: ~1.2x baseline ($0.0100 vs $0.0083)
  health_ops ran 4 cycles (baseline: 3)
  re-read cpu_utilization twice (baseline: once)
  health_ops input tokens ~1.6x (3,557 vs 2,274)

Same final answer. The APM span is 200 OK. What catches it: agent.cycle_count and tool.call_count. The agent looped more than its baseline and called the same tool repeatedly. No single number is "wrong." The delta is wrong.

Scenario 2: Redundant tool calls

Milder, sneakier. The agent calls a tool it already has the answer for.

REDUNDANT TOOL CALLS: ~1.03x baseline ($0.0085 vs $0.0083)
  running_instances called 3x (baseline: 1x)

A few percent on one run is the kind of thing you never notice. Multiply it across thousands of daily runs and it is a line item. The signal: tool.call_count for running_instances jumped from 1 to 3. Only visible per-tool, per-agent, and easy to miss precisely because the dollar delta is so small on a single run.

Scenario 3: Context bloat (the expensive one)

The agent drags too much context into its prompts. Every extra token in gets paid for, and it cascades.

CONTEXT BLOAT: ~1.4x baseline ($0.0119 vs $0.0083)
  health_ops input tokens elevated, output nearly 4x (803 vs 199)
  supervisor synthesis cost rises too, bloat cascades

This is the meanest one because it compounds. The sub-agent's bloat feeds a bigger blob to the supervisor, whose own synthesis cost then rises too (on this run the supervisor jumped from $0.0032 to $0.0044). The signal: llm.usage.prompt_tokens and llm.cost.usd per agent. You watch prompt tokens creep up where the work did not.

Same answer, different bill

src/compare.py puts CLEAN next to BLOAT side by side.

	CLEAN	CONTEXT BLOAT
Final answer	Correct	Correct
Crew total cost	$0.0083	$0.0119
Delta	-	+43% (~1.4x)
APM status	200 OK	200 OK

Two runs. Both return the right answer. Both are green in any latency-and-errors dashboard. The only place the extra 43% shows up is in the trace, on the per-agent cost attribute you stamped yourself. In the Traccia dashboard this is the moment the tool earns its place: two runs sit side by side, both "successful," and the per-agent cost column is where the bloated one gives itself away. That is the whole argument for agent-native observability in one table.

The detection logic is not clever. It is a delta check: for each agent, compare this run against the baseline and flag three things. More cycles than baseline means a possible runaway loop. Prompt tokens more than 1.25x baseline means possible context bloat. Any tool called more times than baseline means a possible redundant call. That is the whole detector, about fifteen lines in src/compare.py.

The intelligence is in having the baseline and the per-agent attributes to compare against. The trace is what makes those attributes exist.

This detector broke once during the build, and it is a good example of how instrumentation and detection are coupled. When I switched tools to emit one live span per call (the 0ms fix above), the redundant-call check stopped working, because it had been reading a call_count attribute off a single reconstructed span that no longer existed. I had to change it to count span occurrences per tool name instead. The lesson that stuck: change how you record, and you can silently break how you detect. The baseline caught it, which is the whole point.

Watching it happen: the live control panel

This is the panel you saw in the video above. Traces are the source of truth, but a wall of span JSON is not how you show a crew to a teammate. So the repo ships a small live control panel: a single-page UI that runs the real crew and animates the investigation as it happens.

You type a prompt into a command console, hit Investigate, and the view scrolls down to a graph of the crew. The supervisor sits at the top and the three specialists fan out below it, connected by wires. As the run streams, each agent lights up like a traffic signal: idle, then running (with a live activity line, "Reading Cost Explorer", "Scanning security groups"), then done, and the report reveals at the bottom. Every agent card shows the AWS services it touches as small chips, so a viewer can see at a glance that Cost Analyst reads Cost Explorer and the forecast, Health & Ops reads EC2/EBS/Lambda/S3/CloudWatch, and Security Auditor checks security groups, IAM, S3, and GuardDuty.

The panel has two modes. Live runs the real crew: real Nova Pro calls, real read-only AWS reads, real dollars on the trace, about thirteen seconds. Replay animates a saved run from a committed trace file, deterministically and for free, so you can rehearse the visual as many times as you want without spending a token. Both drive the exact same UI from the same event stream; the only difference is whether the events come from a fresh Bedrock run or a recorded one.

The backend is a small FastAPI app that streams the crew's lifecycle as Server-Sent Events. The important part is that the UI is a thin viewer over the same telemetry the trace records; it is not a second, hand-maintained source of truth. What the graph shows is what the crew did.

Build your own, at zero cost and read only

You do not need a paid plan or a live AWS bill to try this. The whole thing runs locally with the file exporter and read-only AWS credentials.

The permission surface is deliberately small: every action is a Get, List, or Describe, across Cost Explorer, EC2, CloudWatch, S3, Lambda, IAM, and GuardDuty. There is no create, no modify, no delete anywhere in the toolset. The full policy JSON is in the repo README; if you would rather not hand-roll it, AWS's managed SecurityAudit and ViewOnlyAccess policies cover the same set. Attach it, invoke Nova Pro through Strands, and you have a crew that can look but never touch. To send traces to the hosted dashboard, set TRACCIA_API_KEY; leave it unset and everything writes to a local file. Same spans either way.

The read-only shape is the same for every tool: wrap the real boto3 describe/get/list call in a live span so its duration in the trace is the true AWS read time, return the fields you need, touch nothing. src/tools.py in the repo has all seven; they are all this shape.

An honest take on Traccia

I shipped a real crew against this SDK and read its source to understand the behavior, so here is the assessment grounded in that, not in the marketing page.

What is genuinely good:

It is OpenTelemetry-native. Spans, processors, and resource attributes are standard OTel underneath, so the data model is not proprietary and you are not locked in.
It runs at $0 and offline by default. With no API key it writes to a local file exporter; set TRACCIA_API_KEY and the same spans push to the hosted dashboard. Same spans either way, which made local development and CI painless.
It ships more than a tracer. There is a real cost engine with a bundled pricing snapshot (covering Nova and Claude, among others), a staleness warning when that snapshot ages, and auto-instrumentation for LangChain, CrewAI, OpenAI Agents, and the raw OpenAI/Anthropic/Gemini clients. If you are on one of those stacks, a lot of what I wired by hand would have come for free.
The span-level agent identity model is the best part. A span-level agent.id and agent.name override the process default, which is precisely what let a single-process crew render as a four-agent fleet with real per-agent cost. That is a thoughtful design decision, not an accident.

Where it made me work, and where it could be better:

No Strands integration yet, and it does not hook raw Bedrock. For this stack the cost bridge was manual. That is fine and it gives you control, but a Strands integration would remove the single biggest chunk of setup for AWS-native builders.
The cost processor fails silently (at the time of writing). It skips a span with no error if llm.model is missing or span.type is not LLM. That forgiving behavior is defensible, but the silence cost me an afternoon of a zeroed dashboard. A debug log on skip would fix it outright, and it is the kind of small papercut an early-stage tool usually closes fast.
A couple of sharp edges are only discoverable in the source (at the time of writing). span_scope(parent=None) still inherits the current context (so separate agent traces silently merge unless you detach first), and span_scope is not a context manager (you call .end() yourself). Neither is obvious from the docs today.
Documentation is the real gap. I learned the identity precedence, the llm.model requirement, and the context-detach behavior by reading the SDK, not the docs. For a bootstrapped, early-version product that is understandable, and the SDK itself is readable enough that this was possible. But better docs would turn a day of spelunking into an hour.

Net: for supported frameworks you get a lot for free, and even off the beaten path the OTel foundation and the cost/identity model are solid. The capability is there; the polish that is missing is mostly documentation and a few developer-experience papercuts, which is exactly what you would expect from a product at this stage.

Honest caveats

I want to be straight about the limits, because that is the whole point of this article.

I saw versions of the three waste scenarios in real runs first; the src/waste_demo.py versions just make them fire on cue. LLM output is non-deterministic. In real life these patterns appear on their own, just not on a schedule you can demo.
Detection is delta-vs-baseline, not fixed magic thresholds. You need a known-good run to compare against, same as any regression system.
Every dollar is real Nova Pro token usage against verified us-east-1 pricing, but the exact numbers shift run to run. Do not treat any single figure as a constant. Treat the relationship (roughly 1.4x on the worst pattern I measured) as the lesson, not the exact decimals.
Traccia does not auto-instrument Strands or Bedrock. The cost bridge is about 40 lines you write and own. That is a feature: you control exactly what goes on the span.
I model each agent as its own trace, grouped by session.id. That is a deliberate choice to match how a real fleet is owned and operated. If you prefer one nested trace per run, keep the supervisor as the parent instead of detaching the context. Both are valid; pick the one that matches how your team reasons about the system.
The agent is read-only by IAM policy, not by hope.

The takeaway is not "buy an observability tool." It is that a correct-looking answer tells you nothing about whether the run was efficient, and the only place the truth lives is in the trace, on attributes you have to put there on purpose.

FAQ

What is AI agent observability, and how is it different from LLM monitoring?
LLM monitoring usually watches one model call: latency, errors, maybe token count. Agent observability watches a whole reasoning session: how many cycles an agent took, which tools it called, the cost of each step, and, in a multi-agent crew, which agent did what. Agent failures show up across a multi-step chain, not on a single call, so you need the full trace to see them.

How do I track per-agent cost on Amazon Bedrock?
Bedrock returns token usage after each call. You multiply input and output tokens by the model's per-1K price (for Nova Pro in us-east-1, $0.0008 in and $0.0032 out) and attach that dollar figure to the trace span for the agent that made the call. That is the stamp_llm_cost function in this article. AWS's own tag-based cost allocation in Cost Explorer works at the account and tag level; per-agent, per-reasoning-cycle attribution is what the trace adds on top.

Can AWS Cost Explorer show per-agent cost by itself?
Not on its own. Per AWS's Well-Architected Agentic AI Lens, the default state is that costs are visible only at the account level and Cost Explorer cannot separate agents or workflows. Tag-based allocation plus AgentCore Observability improves this, but per-agent and per-reasoning-phase attribution comes from instrumenting the trace, which is what this build does.

Why does my multi-agent app cost more than I expected even when it works?
Because a correct answer is not a cheap answer. Agents can loop an extra reasoning cycle, re-call a tool they already have the answer for, or drag bloated context from step to step. None of that returns an error; it just adds tokens. The overspend shows up on the bill, not in a latency-and-errors dashboard, which is the "silent waste" this article is about.

Do I need a paid tool or an AWS account to try this?
No. The whole build runs at $0 locally: the Traccia SDK writes traces to a local file with no API key, and the AWS reads use read-only credentials (or the committed replay run, which needs no AWS access at all). You only need a Bedrock model grant if you want to run the live crew against your own account.

Is Traccia open source?
The SDK (traccia-py) is open source under Apache-2.0 and built on OpenTelemetry, so the spans are standard OTel and you are not locked in. The hosted dashboard at traccia.ai is the optional commercial part; you only reach for it when you want the visualization.

Does Traccia support AWS Strands Agents out of the box?
Not at the time of writing. It auto-instruments LangChain, CrewAI, and the OpenAI/Anthropic/Gemini clients, but not Strands or raw Bedrock converse, so on this stack you stamp cost onto the span yourself (about 40 lines). On a supported framework, most of that is automatic.

Try it

The full code (crew, tools, waste demos, the live control panel, the compare view, and the double-count probe) is on GitHub: ai-agent-observability-aws. There is also a live replay of a run you can click through in the browser: https://simplynadaf.github.io/ai-agent-observability-aws/. Clone it, run python -m src.waste_demo with local export, and watch a perfect answer cost you ~1.4x. Then go instrument your own agents before your bill does the teaching for you.

To put Traccia under your own agents, the on-ramp is deliberately short and free:

Install the SDK (open source, Apache-2.0): pip install traccia. With no API key it writes traces to a local file, so you can see spans at $0 before you sign up for anything. Source and docs: github.com/traccia-ai/traccia-py.
Stamp cost onto your spans using the ~40-line stamp_llm_cost pattern above (or get it for free if you are on LangChain, CrewAI, or the OpenAI/Anthropic/Gemini clients, which Traccia auto-instruments).
See it in the dashboard when you want the visual per-agent cost and the side-by-side run compare: set TRACCIA_API_KEY and the same spans push to traccia.ai. Same spans either way, so nothing about your instrumentation changes.

If you build something with it, tell me what silent waste you found. That is the interesting part.

Top comments (52)

Ken W Alger • Sep 24

This is a really useful example of something I've been thinking about as "tax reporting" for AI systems.

I've been using a tax metaphor for costs such as Context Tax, Retrieval Tax, Observer's Tax, and Ingestion Tax. The important part isn't simply that those costs exist. It's whether the architecture makes them attributable enough to see where you're actually paying them.

Your context-bloat example captures that nicely. Both runs can be functionally successful, but one carries a 43% higher bill. At the system level, that's just "AI got more expensive." At the per-agent/per-step level, you can start asking why: Did an agent carry unnecessary context? Re-read something it already had? Take an extra reasoning cycle? Call a tool redundantly?

That's where I think the accounting analogy becomes useful. Knowing the total tax bill is interesting. Having enough reporting to identify which architectural behavior incurred which tax is actionable.

I'd be particularly interested in taking this one step further and reporting the costs by architectural category rather than only by agent. If context bloat increases both the worker's cost and the supervisor's downstream synthesis cost, for example, there's a kind of tax propagation happening across the trace.

Really interesting work. This is much closer to the kind of observability I think AI systems need than another dashboard showing aggregate token consumption.

Sarvar Nadaf AWS Community Builders • Sep 24

this is a great framing and honestly the tax metaphor clicks better than what i used in the video. context tax retrieval tax observers tax ingestion tax that maps almost one to one onto the spans you end up staring at

you nailed the real point too it is not that the cost exists it is whether the architecture lets you attribute it. the 43% run looked perfectly healthy at the system level 200 ok correct answer done. only at the per agent per step level could i even ask the why did the worker carry context it did not need did it re read something it already had did it burn an extra reasoning cycle did it call the same tool twice

the tax propagation idea is the part i had not thought about clearly. you are right that context bloat on the worker does not stay on the worker it inflates the supervisors downstream synthesis cost too so one bad decision shows up as tax in two places along the trace. reporting by architectural category instead of only by agent would surface that. right now i tag cost per span and per agent but not per category so i cannot yet say retrieval tax was x acrossthe whole run. that feels very buildable though the span attributes are already there i would just need to stamp a category on each span and roll up by that instead of by agent

going to sit with this. thanks for taking the time this is a sharper way to think about it than aggregate token dashboards. quick question for you when you split it into context tax retrieval tax and so on do you assign each span to exactly one category or can one span carry more than one tax at once i keep going back and forth on whether a single tool call can be both retrieval tax and context tax and i am curious how you draw that line?

Ken W Alger • Sep 24

I wouldn't make them mutually exclusive. I think doing that would make the accounting cleaner but the architecture less truthful.

A retrieval span is a good example. Searching, ranking, and fetching is Retrieval Tax. But if that operation returns 20K tokens that get hydrated into the next model call, it has also caused Context Tax downstream. If those records are unnecessarily verbose, Prose Tax may also contribute to that Context Tax.

So I'm starting to think there are at least two useful dimensions here: where the cost was incurred and what architectural behavior caused or contributed to it.

I might therefore let a span carry multiple tax attributes, but avoid double-counting the actual dollars. Something like:

span cost = $0.004

primary tax = retrieval

contributes_to = context

cause = excessive_candidates

Then if the supervisor's next model call costs an additional $0.002 because of that retrieved material, that's a separate Context Tax charge with lineage back to the retrieval span.

That would let the report say not only "this run incurred $X in Retrieval Tax and $Y in Context Tax," but potentially "this retrieval decision caused $Z of downstream Context Tax."

I haven't implemented this taxonomy as an accounting system yet, so I'd treat that as a design hypothesis rather than a settled schema. But your question is making me think the lineage between taxes may be as important as the categories themselves.

And now you've got me wanting to build it too. :)

Sarvar Nadaf AWS Community Builders • Sep 24

this is exactly the direction i was hoping you would push it and the mutually exclusive vs truthful point is the thing i had wrong in my head. i was trying to force one span into one bucket because it made the sums clean but you are right that throws away how the cost actually happened

the two dimensions split is what makes it click for me. where it was incurred is just the span itself but what caused or contributed to it is the interesting axis and that is the part no dashboard shows today. your schema is basically what i want primary tax on the span for the dollars contributes_to for the propagation and cause for the why. keeping the actual dollars single counted but letting the attributes be many solves the double count problem i kept tripping on

the lineage bit is the real unlock though. this retrieval decision caused z of downstream context tax is a completely different sentence than this run cost x. one is a receipt the other tells you what to go fix. and since the spans already sit in a parent child trace the lineage is kind of already there i would just need to carry a ref from the downstream context charge back to the span that caused it

treating it as a design hypothesis is fair i have not built it either. but you have basically talked me into prototyping it. if i stamp primary_tax contributes_to and cause on the strands spans and roll up by both dimensions i think i can get a first version out

so here is my next question when you attribute that downstream z back to the retrieval span how do you decide how much of the supervisors context tax to blame on that one retrieval vs everything else it was already carrying do you split it proportionally by token share or do you just attribute the whole
delta to the thing that changed

Ken W Alger • Sep 24

I think I'd distinguish measured attribution from allocated attribution here.

If I have a controlled comparison where the only meaningful change is the retrieval result, I'd prefer the delta. Supervisor costs $0.006 without that retrieval and $0.009 with it, so I have pretty good evidence that retrieval span contributed roughly $0.003 of downstream Context Tax.

In a normal production trace, though, I probably don't have that counterfactual. The supervisor is carrying system instructions, conversation state, outputs from other agents, retrieved material, tool results, etc. In that case, proportional token share seems like a reasonable allocation method, but I'd want the receipt to say that's what it is rather than presenting it as measured causation.

So perhaps the attribution itself needs provenance:

attribution_method = measured_delta | proportional | estimated | unknown

That would let you say something like, "retrieval span 104 contributed an estimated $0.0021 of downstream Context Tax, allocated proportionally by hydrated token share," without claiming more precision than the trace actually supports.

And I suspect there are cases where the honest answer should just be unknown. Context isn't necessarily additive. An extra 5K tokens might change caching, reasoning behavior, or even how much output the supervisor generates, so token share and cost share aren't always going to map cleanly.

Your parent-child trace observation is interesting too. If the causal lineage is already structurally present, this may mostly be a matter of making the attribution semantics explicit rather than inventing a whole new tracing mechanism.

Which means you've now gotten me wondering whether a Tax Report needs to report not only the amount and category, but the confidence/provenance of the attribution itself. Apparently, even the tax bill needs provenance. :)

Sarvar Nadaf AWS Community Builders • Sep 24

this is the distinction i was missing and it is the right one. measured vs allocated is doing a lot of work here. if i have the counterfactual the supervisor at 0.006 without the retrieval and 0.009 with it then the 0.003 is real evidence. in a normal production trace i do not have that clean a b the supervisor is carrying system instructions conversation state other agents output retrieved material tool results all at once so the honest move is proportional allocation but labelled as allocation not measurement

attribution_method = measured_delta | proportional | estimated | unknown is going straight into the schema. that one field is what keeps the whole thing honest because it stops a proportional guess from masquerading as measured causation. so the receipt reads retrieval span 104 contributed an estimated 0.0021 of downstream context tax allocated proportionally by hydrated token share and every word in that sentence is now defensible and you are right that unknown has to be a real allowed value not a cop out. context is not additive an extra 5k tokens can flip caching change the reasoning path or change how much the supervisor generates so token share and cost share do not always line up. pretending otherwise would be the exact thing i am trying to avoid

the part that makes this feel buildable and not a research project is your last point. the causal lineage is already structurally there in the parent child spans so i am not inventing a tracing mechanism i am just making the attribution semantics explicit on top of what the trace already records. that reframes the whole thing from build a new system to stamp three more attributes and be honest about the confidence

and yeah you landed it the tax bill needs provenance too. amount category and how sure we are about who to blame. that confidence field might end up being
the most useful column in the whole report

alright i have to actually build this now. last one for you would you surface attribution_method right in the main report next to every line or keep it as drill down metadata i lean toward showing it inline because a proportional estimate and a measured delta are not the same claim and hiding that feels like the same sin as the aggregate dashboards we started out complaining about

Ken W Alger • Sep 24

I lean inline too, with one distinction: I think attribution_method belongs next to every attributed cost, but probably doesn't need to clutter costs that were directly observed on that span.

If the trace says:

retrieval-104 | RETRIEVAL | $0.0040

that's a measured direct cost. There's not much epistemic ambiguity there.

But the moment the report says:

retrieval-104 → downstream CONTEXT | $0.0021 | proportional

I think proportional has to stay visible. Otherwise $0.0021 visually acquires a precision the evidence doesn't support.

I'd probably even resist collapsing that into a generic confidence score too quickly. measured_delta, proportional, estimated, and unknown tell me why I'm entitled to believe the number, whereas something like confidence: 0.82 gives me another number whose provenance I now have to investigate. :)

So maybe the main report has something like:

TAX	DIRECT	DOWNSTREAM	METHOD
Retrieval	$0.0040	$0.0021	proportional
Context	$0.0090	—	observed

Then drill-down gives me the evidence behind that method: hydrated token counts, comparison runs, parent/child span refs, assumptions, etc.

That preserves the distinction at the decision surface without dumping the entire receipt into the table.

And I think your point about hiding it being the same mistake as aggregate dashboards is exactly right. If two numbers make materially different claims, the UI shouldn't make them look epistemically equivalent.

Which is a sentence I did not expect to write today about AI cost accounting. :)

Also, I suspect unknown rows could become some of the most operationally useful ones. A high downstream cost with unknown attribution isn't just incomplete accounting. It's telling you where your observability boundary prevents you from explaining system behavior.

At that point, the report isn't merely saying where you're paying taxes. It's showing you which parts of the tax return you couldn't substantiate.

And apparently we've invented AI tax audits now.

Sarvar Nadaf AWS Community Builders • Sep 24

yeah inline it is you talked me all the way there and the distinction you just drew is the one i would have gotten wrong

showing attribution_method only next to attributed cost not on directly observed spans is right retrieval-104 RETRIEVAL 0.0040 has no ambiguity it was measured on the span putting a method tag on it just adds noise but the moment it becomes retrieval-104 downstream context 0.0021 the proportional has to ride along or that number steals a precision the evidence never gave it. exactly the sin we started out complaining about two numbers making different claims dressed to look equivalent

and you are right to resist collapsing it into confidence 0.82 too fast i almost suggested that and would have regretted it. measured_delta proportional estimated unknown each tells me why im entitled to believe the number a bare 0.82 is just another number whose provenance i now have to go chase. the method is the provenance the score would hide it and that table is basically the schema. direct and downstream as separate columns method on the row drill down holds the receipt hydrated token counts comparison runs parent child refs assumptions. decision surface stays clean the evidence is one click away not dumped in the table

the unknown rows being the most useful is the part that actually reframes it for me. a high downstream cost with unknown attribution is not a hole in the accounting it is the tool pointing at where my own observability boundary stops letting me explain the system. that is a genuinely different output than a dashboard it tells you where to go instrument next not just what you spent.

so yeah we invented ai tax audits. im going to build the first version stamp primary_tax contributes_to cause and attribution_method on the strands spans roll up by tax category and direct vs downstream and let unknown be a real row. ill send you what the first tax report looks like once its real. this whole thread was worth more than the article thanks ken much appricate!

Raju Dandigam • Sep 23

I work on agent-inspect, so the distinction between a correct answer and an inefficient trajectory really resonates. The llm.model silent-zero gotcha is exactly the sort of instrumentation failure a baseline should catch. Since your supervisor routes cost-only requests to fewer specialists, would you key the spend baseline by task type and delegation set (plus model/pricing version)? Otherwise a legitimate full-account sweep could look like context bloat relative to a narrower run.

Sarvar Nadaf AWS Community Builders • Sep 25

probably the sharpest question here and youre right. in the article the baseline is per agent not per run so each agent is compared to its own known good. that dodges part of it a cost only run just never starts health_ops or security_ops so they cant get flagged. but inside one agent the case you name is real a full sweep genuinely feeds health_ops more context than a narrow ask so against a lean baseline it looks like bloat when nothings wrong. so yeah the key has to come from the request side not the trajectory.

keying by which agents ran is the wrong move a run that wandered ends up compared to other wandering runs. and good catch on pricing version that has to be in the key or every price update reads as a regression. the delegation set is an output of the request not the label. how does agent-inspect key it do you classify the request into a class up front or compare against a rolling per class distribution. the new class with no history is the bit i havent figured out.

anassBld • Sep 23

Tracking cost per agent rather than globally across the workflow is honestly the only way to catch silent token inflation early. In our multi-agent pipelines, we noticed intermediate routing and evaluation agents often eat 60%+ of the total token budget during retry or handoff loops without producing direct user-facing value. Attaching the trace/span ID down through each subagent invocation makes pinpointing which specific agent drifted way faster.

Sarvar Nadaf AWS Community Builders • Sep 24

the 60% on routing and eval agents matches what i saw the orchestrator is quietly expensive because it writes the long synthesis and in the bloat case the supervisor cost climbed just from a bigger blob being handed up to it so the waste cascades from a noisy child into the parent. an agent that produces no user facing output can still be the most expensive one in the run which is exactly why crew total hides it. on the span id piece thats the part that bit me. i wanted each agent as its own trace so a fleet reads like a fleet but strands runs each subagent in its own event loop on a detached context so my spans kept collapsing into one until i detached the otel context myself and tied the agents together with a shared session.id instead of a parent child chain. so the correlation key here is session.id across separate traces not a single nested trace. handoff loops would show up as the same agent repeating within that session. how are you carrying the id today one nested trace for the whole pipeline or separate traces stitched by a correlation id? im curious which one holds up better once retry loops get deep.

anassBld • Sep 25

We use separate traces stitched by a correlation ID (session_id + run ID), exactly like you described.

Nested parent-child traces broke down for us as soon as an agent failed or retried—a single nested trace either bloated to unreadable depths or got severed if a subagent died unexpectedly. With flat traces linked by a correlation ID, each subagent execution is an independent unit of work with its own token/cost metrics. We just tag every subagent trace with session_id, turn_index, and attempt_count.

When retries kick in, you see distinct trace entries under the same session ID rather than a mutated span tree, which makes visualizing loops and cost runaways way cleaner.

Igor Eduardo • Sep 23

200 OK with a silent 1.4× bill is the cost twin of “exit 0 with an empty payload.”

If the eval contract only watches success shape (right answer, green APM), multi-agent systems will optimize for looking done while burning nested calls you never intended. Per-agent cost isn’t vanity observability — it’s part of the quality contract.

The check I’d pin next to task success: cost-per-successful-outcome by agent role, with a hard fail when the run is “correct” but outside the agreed spend envelope.

Sarvar Nadaf AWS Community Builders • Sep 23

exactly this is the direction i had in mind as well

a run being technically successful doesnt mean it was efficient or healthy cost per successful outcome by agent role would give a much better signal especially when you start running these workflows at scale i also like the idea of treating the spend envelope as part of the eval itself rather than checking cost separately

Igor Eduardo • Sep 23

Agreed — and the interesting edge case for me is when one agent is green on cost-per-success while another is the silent 1.4× branch. Role-level envelopes catch that; a single crew-total budget can still hide a noisy specialist.

Branch-level pause (as floated elsewhere in the thread) is the runtime twin of that eval cut: stop the noisy role without killing the whole run.

Sina Rezaei • Sep 23

The “same answer, different bill” part really caught my attention.

Traditional monitoring makes it very easy to think that a successful request is a healthy request. If it returns 200, the latency looks fine, and there are no errors, everything appears green. With multi-agent systems, that can hide a completely different problem: the system may have taken extra reasoning cycles, repeated tool calls, or carried unnecessary context through the workflow.

I also like the decision to track cost at the agent level instead of treating the whole run as one number. Once you know which agent caused the increase, cost stops being just a finance metric and becomes a debugging signal.

The part about changing the instrumentation and accidentally breaking the detection logic was probably my favorite detail. That's exactly the kind of problem that tends to show up in a real system and never makes it into the clean demo.

A correct answer tells you what the system produced. The trace tells you what it took to produce it. That distinction is becoming pretty important for agentic systems.

Sarvar Nadaf AWS Community Builders • Sep 25

thanks for reading it that closely. the instrumentation breaking the detection is the bit im most glad i left in cause it wasnt planned. i switched the tools to emit one live span per call to get real durations and that quietly broke the redundant call check it had been reading a call_count off a single reconstructed span that didnt exist anymore. the baseline caught it. which is kind of the whole point in miniature the thing watching for drift drifted and only the baseline noticed. and yeah cost as a debugging signal not a finance number is the reframe. once you know which agent moved it its a stack trace nota bill. good read thanks.

Sina Rezaei • Sep 25

That “the thing watching for drift drifted” part is probably the most interesting takeaway for me. It creates a nice second-order problem: observability itself becomes part of the system that needs validation.

You can have the application behaving correctly, the cost trace looking reasonable, and still have the detection logic quietly stop seeing the behavior it was supposed to detect. So the baseline is doing more than detecting cost drift. It is also testing whether the measurement system itself is still trustworthy.

That feels especially important in agentic systems, because the workflow, instrumentation, and even the definition of a “normal” run can all change over time.

build996 • Sep 24

Building on the baseline-rot point above: before a baseline can rot it has to exist, and that means grouping runs into classes. The tempting key is the trajectory - which agents ran, which tools fired - and that is exactly the wrong one, because it blinds the detector in the case you care about: a run that wandered gets compared against other wandering runs instead of against what that request should have cost. The label has to come from the request side, which the agent does not choose. How are you keying runs today?

Sarvar Nadaf • Sep 26

straight answer today its keyed at the agent level not the run level. each agent vs its own prior known good. so the baseline exists per agent not per run class. holds for the three patterns in the piece cause a cost only ask never runs the other agents so they cant be mis flagged. but youre pointing right at the hole. inside one agent theres no request side label so a real full sweep feeds health_ops more context and my delta check calls it bloat. and yeah keying by trajectory is self defeating a wandering run graded against other wandering runs means the detector goes blind exactly where you need it. label has to be the thing the agent doesnt pick the request or intent not its path. so per agent works now request side classing is the next thing and its not in the repo yet. how are you getting the request class fixed task type on the way in or inferred. and what do you seed a fresh class with?

Mihai Perdum • Sep 24

Sarvar, the baseline-vs-delta design is the right instinct, but I'd push on the baseline itself. A single captured CLEAN run is one sample of a non-deterministic process, so a fixed 1.25x prompt-token threshold is partly measuring model variance, not just waste. Two genuinely clean runs could differ by a meaningful chunk just from the model phrasing its reasoning differently, and a legitimately harder question needs a real extra cycle sometimes. Have you run the CLEAN baseline itself multiple times to see how much it naturally drifts, or is one captured run being treated as ground truth for now? The llm.model vs llm.request.model silent-zero bug is a great catch on its own - a processor that fails closed with no log is exactly the kind of thing that costs an afternoon, and logging on skip is cheap enough there's no excuse not to add it upstream.

Sarvar Nadaf AWS Community Builders • Sep 24

Mihai this is fair and honestly its the weakest part of the current setup one captured clean run is being treated as ground truth right now thats it one sample im not going to pretend otherwise you nailed the problem the 1.25x threshold is partly measuring nova phrasing its reasoning differently not waste i havent run the baseline 20 times and looked at the spread so i actually dont know how much a clean run drifts on its own which means i cant tell you how much of that 1.25x is real headroom vs me guessing a number that felt safe

the right version is what you said run clean n times build a distribution and flag on something like mean plus 2 sigma per attribute so the threshold comes from the actual variance instead of a single number i picked and the harder question needing a real extra cycle is the same issue a static cycle threshold punishes a legitimately bigger task the baseline should probably be per question class not one global run so no its not defensible as ground truth yet its a demo baseline the design is right the statistical rigor isnt there and pretending a single sample is a
baseline is exactly the trap im warning about in the article so thats a fair hit

on the silent zero yeah theres no excuse a fail closed processor that returns with no log cost me an afternoon of a zeroed dashboard while the spans clearly
had tokens one debug line on skip fixes it im flagging it upstream.

Kartik N V J K • Sep 23

The 1.4x overbilling problem is real and matches what I have seen in multi-agent setups. The MAST paper finding that failures do not crash but complete while burning money is the key insight. I started tagging every trace with a cost-per-task metric, and it immediately showed me which agent was the bottleneck. How do you handle the case where one agent's "successful" output is actually expensive garbage that the next agent has to clean up?

Sarvar Nadaf • Sep 25

same case igor raised and its the honest limit of what i built. my spans attribute cost to the agent that spent the tokens not the one that caused the spend. so the producer handing off garbage looks cheap and the cleanup agent looks expensive. backwards. the only thing i can see is the cascade in the bloat run a bloated child made the supervisors own synthesis cost go up from the bigger blob handed to it. so waste visibly moves across the handoff but i dont tie cause to effect yet. doing it right is data lineage between spans which agent produced the context that inflated the next ones input. not in the repo yet. the eval version igor floated attribute the repair tokens back to the producer and fail on producer cost per usable handoff is where id take it. how are you catching it now

Igor Eduardo • Sep 23

That case is the cost twin of “success-shaped empty”: the upstream agent’s span looks green while the cleanup cost lands on whoever has to repair it.

I’d attribute the downstream repair tokens back to the producer role in the eval, and fail the run when producer cost-per-usable-handoff blows the envelope — even if its own task-success bit flipped true.

Otherwise per-agent cost still hides the specialist that manufactured expensive garbage.