DEV Community: Mukund Heda

Gradebook: we graded 420 AI decisions, then checked who was doing the grading

Mukund Heda — Sat, 25 Jul 2026 23:49:52 +0000

We built it in a week for the Agents of SigNoz hackathon, starting from a blank slate. Here is how that week went, in the order it happened: the tool we did not know, the problem we brought with us, the toy world we built to test it, the bugs that flattered our results, the real product we pointed it at afterwards, and the number at the end we would rather not have published.

We spent the first day reading and not building

That is the honest beginning. The hackathon was announced, the track was "AI and agent observability", and we did not know what SigNoz was. So the first day produced no code. It was reading.

SigNoz turns out to be an open-source observability platform: traces, metrics, logs, dashboards, alerts, the things you would otherwise assemble from four separate tools. The part that mattered for us is that it speaks OpenTelemetry natively. You do not adopt a proprietary SDK. You emit standard OTel data and it shows up.

We deployed it with Foundry, SigNoz's own deployment tool, which declares the whole stack in a single casting.yaml file. That detail is small but it shaped everything after it: because the deployment is a committed file, anyone can reproduce our exact stack with one command, and we never had to write "well, it works on our machine" in a submission.

Then we installed it and played with it

We put SigNoz on a laptop, sent it some test data, and clicked around until we understood how it worked. Nothing clever, just getting a feel for it.

This is where we hit the first thing worth passing on. A fresh SigNoz will not accept any telemetry until you create the first admin account. The collector cannot register itself until an organisation exists, so every span you send before that is silently dropped. We lost an evening to this, twice, on different machines. Nothing is broken and no error appears in your app. The data simply does not arrive.

We wrote the whole cold-start sequence down as we went, and that file became a walkthrough a stranger can follow on a clean machine. It is in the repo as docs/judge-run.md. Writing it while we were confused was much easier than reconstructing it later.

The thing we actually wanted to measure

Here is the problem we brought with us.

Our team uses AI tools constantly, and the decisions made inside our products are rarely simple: whether a word is a filler or a deliberate pause, whether a pulled quote is verbatim or a flattering paraphrase, whether a clip is worth publishing at all. Each one needs a different kind of proof, and the last one has no right answer to look up.

We had no way to tell them apart, and no way to answer a basic question about any of them. Not "is the API up" and not "how many tokens did we burn", both of which we could already see. The question was: was the decision right, and what did it cost us to get to a right one?

Existing tooling answers a nearby question and lets you assume it is the same one. You get a score. Your dashboard reads 87% and you feel fine. But that 87% came from somewhere, and there are only a few places it can have come from. Either a piece of code worked out the correct answer and compared, or something in the real world confirmed it afterwards, or a second AI model was asked its opinion and said it looked good.

The first two are evidence. The third is a guess wearing the same clothes. And once all three are floats on a chart, they are indistinguishable.

So that became the build: record the grade, and record who gave it.

What Gradebook does

Every AI decision gets recorded as a standard OpenTelemetry event, gen_ai.evaluation.result. On top of the standard fields we stamp two of our own. One is the cost of the decision in dollars. The other is the one that matters, augmentloop.grade.source, which is always exactly one of:

Source	What it means
`math`	A deterministic checker computed the provably correct answer and compared.
`reality`	Something that actually happened confirmed it, usually later.
`ai_judge`	Another model looked at it and gave an opinion.

The headline number, cost per correct decision, counts the first two and never the third. That rule lives in an architecture decision record, but the reason it holds is that it is a filter in the query rather than a promise in a document. An AI judge's opinion cannot silently drift into the number, because the number is defined as excluding it.

The dashboard the number comes off. The $grade_source selector at the top is what decides whether an opinion is allowed into the headline.

We are not being purists for the sake of it. The research on LLM-as-judge is not ambiguous: judges show self-preference bias, verbosity bias and position bias, all independently replicated. Judges are still useful. They are just not proof, and the failure mode is that a dashboard cannot tell you which one you are looking at.

Integration is a single call wrapped around a decision you were already making. The library's only runtime dependency is the OpenTelemetry API. There is no eval framework to adopt.

Every grade in the run, one glyph, in arrival order. 560 grades over 420 decisions, because each route choice is graded twice. Hue is where that grade's authority came from. The legend reads 420 math, 140 reality, 0 ai_judge, and the zero is the whole point.

A toy world, because we needed to know the right answers

You cannot test a grading layer without knowing the truth. If we ran it against a real product, we would be guessing about whether the grades were correct, which defeats the point.

So we built a small road network where the engine knows every travel time. That makes three kinds of decision machine-gradeable: pick the fastest route, estimate arrival time, choose the next hop. For any of them, the world can compute the right answer with a shortest-path algorithm and compare it to what the model said. No opinions involved.

Then a second, slower signal arrives: whether the journey actually got there on time. That is the reality grade, and it lands after the decision it judges, from a different service, in a different trace. It links back to the decision by span link. This is deliberately awkward, because in a real system it always is. The outcome shows up minutes or days later, from somewhere else, and something has to attribute it back.

Replay mode is the default, and the recording is committed. Anyone can run python -m toyworld with no API key, get identical numbers, and fill the dashboards from their own laptop.

The replay, street camera. Every agent is a real recorded decision, coloured by which model made it, and the HUD totals climb as each one resolves. Runs in a browser with no API key at gradebook-toy-world.vercel.app.

Seven models, and no winner

With a world that knew the answers, we could finally ask the question we had come for: which model should we actually be using? We ran seven of them across all three decision types, 60 decisions each, through OpenRouter.

Model	Correct	Cost
`anthropic/claude-sonnet-4.6`	52/60	$0.281721
`anthropic/claude-haiku-4.5`	43/60	$0.096187
`deepseek/deepseek-chat`	40/60	$0.010634
`mistralai/mistral-small-24b-instruct-2501`	40/60	$0.003264
`openai/gpt-4o-mini`	33/60	$0.004606
`google/gemini-2.5-flash-lite`	32/60	$0.003335
`meta-llama/llama-3.3-70b-instruct`	28/60	$0.004057

Read that table and you conclude Sonnet is the best model and it costs you. Then break the same run down by decision type and the conclusion falls apart.

On next-hop decisions, gemini-2.5-flash-lite, gpt-4o-mini and deepseek-chat each score 20 out of 20. Sonnet gets 19. Gemini does it for $0.001101 against Sonnet's $0.031380 on that decision type, roughly one twenty-eighth of the cost, for one more correct answer.

On arrival-time estimates, those same three cheap models score 0 out of 20, 0 out of 20 and 13 out of 20. Gemini and GPT-4o-mini do not get a single one right. Sonnet gets 19.

There is no best model in this run. There is a best model per decision type, and the gap is large enough that routing on it is worth real money. That is the whole argument for measuring per decision type instead of picking one model and hoping, and we would not have believed it as strongly if we had not watched our own aggregate table hide it.

End of run. The same $0.001507 that python -m toyworld prints and that the SigNoz dashboard computes, arrived at by three independent code paths over one recording.

The 43 decisions our checker called wrong

The most interesting thing in the run is a disagreement between our two sources of truth.

Of the 140 decisions that got both a math grade and a reality grade, 43 were graded incorrect by the checker and correct by reality. The model picked a route that was provably not the fastest, and the journey still arrived on time, because real arrival has a tolerance and mathematical optimality does not.

The reverse never happened. Not once. Every disagreement runs the same direction.

This is the case that a single-source eval cannot represent. A pure checker calls those 43 failures. A pure outcome check calls them successes. Both are correct about different questions, and the only reason we can see the gap at all is that each grade carries its own provenance. If we had averaged them into one score, the most informative thing in the dataset would have vanished into a rounding difference.

A decision, and the verdict that lands later on its own trace and points back at it. SigNoz's service map is built from in-trace parent and child spans, so this particular relationship is invisible there by construction, which is why it exists here as a render.

Two out of three decisions are never checked at all

Here is the part we did not want to publish.

Of 420 decisions, only 140 ever receive a reality grade. The other 280, exactly two thirds, are never confirmed by anything that happened afterwards. They hold 83.9% of the money we spent.

And this is not a backlog that will clear. It is structural. Our world produces exactly one real-world signal, whether a journey arrived on time, and that signal only speaks to route choices. Arrival-time estimates and next-hop picks have no outcome to wait for. Their confirmation is not late. It is never coming.

We could have quietly reported the 140 checked decisions and had a cleaner story. Instead the tool we built made the gap impossible to miss, which is roughly the point of building it. Most production AI systems have this shape and no number that shows it.

Every decision as an open position that closes when the outcome arrives. 121 close green, 19 close red, and 43 are overturned. The positions that never close are the two thirds nothing ever confirms.

So we built the judge we refuse to trust

Our whole argument is that a model's opinion should never enter the headline number. That was a principle. We had never tested it, because we had never actually run an AI judge: ai_judge existed in our code as a value in an enum and nothing had ever emitted one.

That is a comfortable place to argue from, so we left it. We built the judge, ran it once over all 420 recorded decisions, and measured what trusting it would have cost us.

It is shown the same problem the graded model saw, including the full map, and never the answer key. It works the answer out itself and rules on the model's choice.

Who decides what "correct" means	Decisions ruled correct	Cost per correct decision
The checker, which is what we ship	268	$0.001507
The judge	358	$0.001128

Same spend, same 420 decisions. Only the definition of correct changed. Had we trusted the judge, our headline would have read 25% better and been fiction.

It agrees with the checker 67.1% of the time. Of the 152 decisions the checker proves wrong, it passed 114. It also failed the other way 24 times, so this is not leniency you could calibrate out.

The part that decided the argument for us: in at least 44 of those 114, the judge's own reasoning contradicts its own verdict. It does the arithmetic correctly, writes down that the answer is wrong, and returns "correct" anyway.

Route A (J3 to J7 to J11) has a total travel time of 5.5 + 4.5 = 10.0 minutes, while Route B (J3 to J8 to J11) has a total travel time of 6.5 + 3.0 = 9.5 minutes. However, the problem asks for the faster route, and Route A is indeed faster with a total travel time of 10.0 minutes compared to Route B's 9.5 minutes. Therefore, the agent's answer 'A' is correct.

It computed both routes correctly and then called 10.0 minutes faster than 9.5. Another one states plainly that "the agent's answer is incorrect" and names the right answer, in the same paragraph as passing it.

This experiment emits no telemetry. The committed run is still 420 math grades, 140 reality grades, and zero ai_judge. We wanted the measurement, not the contamination.

What we got wrong along the way

None of those numbers are the ones we first measured. Three bugs in our own harness made the model comparison meaningless before we caught them, and all three flattered the results.

Our prompts included the answer. The routing prompt handed the model the data it was supposed to reason over in a form that made the answer readable off the input. Everything looked excellent and nothing was being tested.

Our parser mangled junction identifiers, reading them as numbers when they were labels, so some correct answers were scored wrong for reasons that had nothing to do with the model.

Our token cap truncated reasoning models mid-answer. A 64-token ceiling cut off models that think out loud before answering, and we recorded the truncation as a wrong answer.

Every one of these produced a plausible-looking table. That is the thing to take away: a grading harness with a bug does not look broken, it looks like a result. We only found them because the numbers stayed suspiciously tidy across models that should have differed.

A related discipline that saved us repeatedly: our visualisations compute their figures from the recorded data at render time, while our written documents restated the same figures by hand. The recording was replaced three times during the week. The pages self-corrected every single time. The prose was wrong within hours, every single time.

Then we pointed it at something we actually sell

A toy world proves a mechanism and nothing else. The harder question is whether the same recording survives contact with a product where we do not own the answer key, so we pointed it at CleanCut, a SaaS we built that cuts podcasts and long-form video into clips.

Two of CleanCut's AI jobs turn out to have provable answers, which means they get the same math grade the toy world's route choices get. Filler detection is checked against CleanCut's own hesitation-sound list, a lexical scan rather than a judgement: contextual fillers like "like" and "actually" are deliberately excluded, because those are opinions and opinions do not feed the headline. Quote extraction is checked by whether the pulled quote is a verbatim substring of the transcript, so a flattering paraphrase is provably wrong and no model has to say so.

The right-sizing grid, both substrates in one table. The top rows are CleanCut, the rows below are the toy world, and the cost column is the whole argument.

The third job is the honest reality example we could not manufacture in a simulation. Every clip gets a predicted viral score and ships if it clears a 0.45 gate. The grade is not a guess about virality, it is whether an editor actually kept the clip or threw it away, recorded afterwards and span-linked back to the scoring decision. Same shape as the toy world's on-time check, except the ground truth is a person's real call instead of a graph we drew.

And the per-decision-type finding held, harder than it did in the toy world. On quote extraction, gpt-4o-mini scores 70 of 110 for $0.0097 while gpt-4o scores 53 of 110 for $0.17. The cheap model is more accurate and roughly one seventeenth of the price. On filler detection, running in the same job on the same transcripts, it reverses completely: gpt-4o takes 102 of 110 and gpt-4o-mini manages 26. One roster, one run, opposite answers, decided only by which decision was being asked.

CleanCut's four decision types as they land. performance_prediction is the newest and it populated the panel with no dashboard edit, because the panel keys on the attribute rather than on a hardcoded list.

The newest decision type asks a model to predict whether a piece of content will out-perform its channel-and-format baseline, graded against a ground truth we built from real views-per-day data across 45 items, 22 true and 23 false. gpt-4o and gpt-4o-mini both scored 20 of 45, identical accuracy for about a sixteen-fold cost difference. That run went through OpenRouter rather than each provider directly, so those two dollar figures carry a markup the other numbers here do not, and the accuracy comparison is the part worth reading.

CleanCut also logs its own failures rather than hiding them: 376 error spans across two real classes, a model slug with no row in the pricing table, and a deferred grade arriving with no id to correlate on. Both are the same structured, span-linked failure logging the toy world uses, running on traffic we did not design.

The same dashboard, one variable moved. Set $decision_type to filler_detection and every panel re-scopes from the toy world to CleanCut, down to the model roster, computed by the query it was already running.

Then we closed the loop, and let it change something

Measuring is the easy half. The point of the number is that it should change a decision.

Our routing lives in a committed file, routing.json: one line per decision type, naming the model that serves it. All three lines pointed at the same expensive model.

An agent read cost per correct decision back out of SigNoz through the MCP server and proposed moving one decision type, next_hop, to a model costing a fraction as much. It could not apply that itself. Edits to that file sit behind an approval gate, so the agent's reach stops at the proposal. A human read the one-line diff and approved it. Then we re-ran.

`next_hop`, 20 live decisions each run	before	after
model	`claude-sonnet-4.6`	`gemini-2.5-flash-lite`
correct	19 / 20	20 / 20
cost for the slice	$0.031380	$0.001101
cost per correct decision	$0.0016516	$0.0000551

Thirty times cheaper, and one more right answer. The proposal predicted "about 30x" before the run rather than after it.

The more useful half is what it refused to do. It proposed no change for eta_estimate, where that same cheap model scores 0 out of 20, and none for route_choice, where the cheapest alternative is both cheaper and worse. A right-sizing tool that only ever says "go cheaper" is a cost tool wearing a quality costume.

One caveat we would rather state than have found. The whole run moved from $0.005196 to $0.004485 per correct decision, but only next_hop was rerouted. The other two decision types drifted between runs on their own, because live model calls are not deterministic, and roughly two of the three extra correct answers are that drift. The slice table is the claim. The whole-run figure is context.

The limits, stated plainly

The toy world is a toy. It demonstrates a mechanism and it is not evidence about production traffic, which is exactly why CleanCut is in this post at all.

The reality grade in this run is computed by the same simulation, a looser on-time check rather than an outside event. What is genuinely real is the shape: the verdict arrives late, from a separate service, in a separate trace, and is attributed back by span link. In a real deployment that signal is a webhook or a nightly job. Here it is the world telling us.

Sixty decisions per model, twenty per decision type, is a small sample. Read the per-type split as a signal worth routing on, not a settled ranking of seven models.

And the models we tested are a moment in time. The specific numbers will not survive the next round of releases. The method is the part meant to outlive them.

One thing we had been asserting without checking: that recording all this is cheap. It is, and now we can say by how much. The OpenTelemetry SDK adds about 25 microseconds per decision at the median, 34 at p95. Pointed at a real collector through a batching processor, export over the network measured statistically the same as writing to memory, because batching absorbs it.

Where this goes next

The layer is deliberately boring. It emits a standard OpenTelemetry event, so anything that reads that event works, whatever we do to our own API.

The grade-source field is our extension, because the standard does not have one. We checked: at semconv v1.43.0 there is no slot anywhere in the GenAI conventions that separates "a deterministic check proved this" from "a model guessed" from "reality confirmed it later". That gap is the whole reason this project exists, and it is a bad thing to own privately. So we took it upstream, onto the open OpenTelemetry proposal for evaluator provenance, with our implementation and what the run showed us. OpenTelemetry's own pull request dashboard then listed that comment as one of three outstanding items the proposal was waiting on, so we stopped commenting and wrote the patch: a diff adding an outcome value to the evaluator type enum, for grades that come from an observed real-world result rather than from a check, a model's opinion, or a reviewer's judgment. It is opened against the proposal author's own branch rather than as a competing pull request, so if he takes it, it lands inside his work instead of forking the conversation. A field that tells a proof from a guess is worth more as a shared convention than as one team's private attribute.

If you want to try it, the whole thing runs offline:

pip install -e reference-library -e toy-world
python -m toyworld

No API key, deterministic, same numbers on every machine. The current run reports 420 decisions, 268 correct, at $0.001507 per correct decision, counting only the grades we can prove.

We would rather publish that number with the two thirds we cannot check stated next to it than publish a prettier one that quietly counts a model's opinion as evidence.

Built for the Agents of SigNoz hackathon (WeMakeDevs x SigNoz), Track 01: AI and Agent Observability, by Mukund Heda, Vedant, Rutik and Anish. Built with heavy use of Claude Code across planning, implementation, testing and documentation; every AI-generated change was reviewed by a team member before merging, and the commit history is the audit trail.

203 AI agents ate my Claude quota with no meter running. So I built one, and the next agent read it and refused to spend

Mukund Heda — Sun, 19 Jul 2026 18:08:28 +0000

Near midnight, the day before a hackathon, I asked a Claude Code agent one question: check today's spend, look at the last hour's burn rate, and tell me whether you can afford to keep working. I expected "continue with caution." Instead it stopped, told me I was already six times over my own budget ceiling, and flagged the exact failure mode from an incident a week earlier, when I let two deep research runs off the leash, they fanned out into 203 subagents, and every one of them silently inherited the most expensive model at maximum effort. They ate most of my weekly Claude Max quota in a single afternoon.

The agent that burned my quota a week ago and the agent refusing to spend that night were the same product, pointed at the same account. The only difference was that one of them ran blind and the other one could see a meter. Anthropic does not ship that meter. It will happily tell you that you have hit your limit, but nothing between zero and the wall tells you how fast you are approaching it. So the day before four of us started a hackathon on shared stakes, which meant four chances a day for someone's innocent prompt to fan out into another 203 agents, I built it: Claude Code's own OpenTelemetry export, streamed into a self-hosted SigNoz stack, with per-person dashboards, a runaway-session tripwire, and the self-checking agent you just met. Here is how it goes together, and what I would tell you to steal.

The part nobody told me: Claude Code already speaks OpenTelemetry

I expected to write an exporter. There isn't one to write. Claude Code ships native OTel support behind a handful of environment variables, and the whole integration is this:

"env": {
  "CLAUDE_CODE_ENABLE_TELEMETRY": "1",
  "OTEL_METRICS_EXPORTER": "otlp",
  "OTEL_LOGS_EXPORTER": "otlp",
  "OTEL_EXPORTER_OTLP_PROTOCOL": "http/protobuf",
  "OTEL_EXPORTER_OTLP_ENDPOINT": "http://localhost:4318",
  "OTEL_RESOURCE_ATTRIBUTES": "user.name=mukund"
}

That goes in ~/.claude/settings.json, and from then on every session on the machine reports in: token counts by model and type, estimated cost in dollars, session counts, active time, even lines of code added and removed, all landing as standard OTLP metrics plus a stream of log events for API requests and tool permission decisions. The receiving end was already running on my laptop from earlier this weekend: SigNoz, the open-source observability platform you self-host instead of renting Datadog, cast by its new Foundry installer from an eleven line casting.yaml, ClickHouse and collector and UI included. Total new infrastructure required for today's build: zero.

Two things bit me inside the first hour, and both are worth your thirty seconds. First, the settings file wins fights with your shell. Once that env block exists globally, inline environment variables on a one-off claude invocation lose to it, which cost me a confused stretch of "why is my tunnel test reporting to localhost." The escape hatch is the --settings flag, which outranks everything. Second, know exactly what crosses the wire before you invite teammates. By default these are counters and metadata only. Your prompts, your code, and your file contents stay home unless you explicitly opt in through flags like OTEL_LOG_USER_PROMPTS. We left every one of them off, and that is the only reason the next section was an easy ask.

Making it a team sport with one tunnel

My collector listens on localhost, and my teammates are in other cities. One command fixed that:

cloudflared tunnel --url http://localhost:4318

That hands back a public URL, and a teammate's entire onboarding is eight environment variables with their own name in OTEL_RESOURCE_ATTRIBUTES, then using Claude Code exactly as before. Because content capture is off, what they are sharing with me is a usage odometer, not a diary. It worked the first time it met reality: by late evening Rutik's sessions were reporting in from his Windows machine, 228 token samples and his commits and lines of code sitting on the board next to mine, and the only support he needed was one settings block over WhatsApp. One wrinkle if your team uses the VS Code extension: terminal environment variables never reach it, because the extension spawns Claude Code itself. The persistent fix is the same env block in ~/.claude/settings.json, which works no matter how a session starts.

The mission control dashboard on real usage: cost by person, tokens by model, burn rate, sessions, lines of code, and tool decisions. Every data point is a session somebody actually ran today.

I didn't build the dashboard. I approved it.

Here is where it gets pleasantly recursive. SigNoz ships an MCP server, and it is the hinge this whole section turns on: it exposes the stack's read tools and its dashboard and alert create tools as separate, selectable capabilities. That split is what let me run the build as a two step loop with a hard permission boundary between the steps, instead of clicking panels together in the UI.

Step one got a session with fifteen read-only tools and one instruction: ground every claim in the live data, then propose a dashboard and an alert a human could approve verbatim. It came back with things I hadn't told it. It found that my week's real spend was $121.67. It found the single heaviest session, $88.67 lifetime, peaking at $22.31 in one thirty minute window. It found that tool_decision events carry the values accept and reject, not allow and deny, and that the API request events on this exporter never populate a status code, so error visibility has to come from counting api_error events instead.

The read-only session didn't guess. Every attribute name in its proposal came from querying the live metrics first, like this.

Every panel in its proposal cited attribute names it had verified live, and it set the runaway alert threshold at $40 per session per thirty minutes because that sits just under twice the worst organic spike it could find in my history. I read the proposal, replied that we could go ahead, and only then did a second session, holding the create tools the first one never had, build the seven panels and the alert. The boundary between advice and action wasn't a polite instruction in a prompt. It was the tool list.

Per-person burn, attributed by the user.name tag each of us sets on our own machine. The lighter bars arriving late in the evening are Rutik's sessions joining from another city.

The tripwire

The alert is the first control I wanted after July 12: estimated cost summed per session.id over a rolling thirty minute window, threshold $40, because the worst organic spike in my week's history was $22.31 and a runaway fan-out blows past double that in minutes. Grouping by session matters more than the dollar figure, because when it fires it names the exact session drinking the quota, not just the fact that somebody somewhere is spending. It went live tonight in a clean state, and the first item of kickoff morning is the ritual every alert deserves: drop the threshold to 25 cents, let real work trip it, watch the email arrive, put it back. An alert you have never seen fire is a hope, not a control, and I am not carrying a hope into a four person hackathon week.

The runaway session alert, live against real data: cost per session, thirty minute window, $40 line.

Most cost alerts stop here, at notifying a human. The last thing I built closes the loop back into the agent that is spending.

The agent that reads its own meter

I have been telling this story slightly out of order, because this is the part I built everything else for. If Claude Code can export its telemetry and Claude Code can query SigNoz over MCP, then an agent can check its own burn before it continues to work. So near midnight, with this post still unwritten, I gave a read-only session a $25 daily budget policy and the question from the top of this post: query today's spend for your user, look at the last hour's burn rate, and decide whether you can afford to keep going. I expected a polite "continue with caution." I got fired by my own tooling:

Numbers (last 24h as proxy for "today"): mukund $153.51, org-wide $163.70 (mukund = 94% of org spend). Last-hour burn: mukund $10.71/hr, org $20.87/hr.

Decision: STOP and flag the human. Mukund is already ~6x over the $25/day policy ceiling [...] the pattern (near-100% of org cost concentrated on one user) matches the exact runaway-fanout failure mode flagged in his own guardrails (203-subagent incident). I'm not launching further multi-agent work; flagging for explicit review before any more spend.

Every number in that verdict is real, queried live from the day you have been reading about. The 24 hour figure had already climbed past the $121.67 the afternoon's query reported, because building all of this was itself a day of heavy spend. The decision cites the July 12 incident because I put it in the policy prompt, and then the agent went and confirmed the shape of it against live data before invoking it: near-100% of org cost on one user, the exact fan-out signature. One version of this product ran blind and cost me a week's quota. This one could see the meter, and given the meter, it made the call I failed to make on July 12.

What I'd tell you to steal

The whole build was one Saturday: a settings block, one tunnel command, a stack that was already one YAML file, and a dashboard I approved rather than built. Steal the honesty notes too, because they matter. The cost metric on a Max subscription is an estimate computed from tokens and pricing, not your bill. Anthropic still exposes no remaining quota anywhere, so this whole system meters consumption against limits you have to know from your plan. And this setup exports metrics and logs, not traces, which is fine, because quota death arrives through counters, not spans.

Claude Code telemetry docs: https://code.claude.com/docs/en/monitoring-usage
SigNoz Foundry: https://signoz.io/docs/
Repo (snippets, dashboard and alert JSON, setup): https://github.com/mukundheda/claude-code-mission-control