We built it in a week for the Agents of SigNoz hackathon, starting from a blank slate. Here is how that week went, in the order it happened: the tool we did not know, the problem we brought with us, the toy world we built to test it, the bugs that flattered our results, the real product we pointed it at afterwards, and the number at the end we would rather not have published.
We spent the first day reading and not building
That is the honest beginning. The hackathon was announced, the track was "AI and agent observability", and we did not know what SigNoz was. So the first day produced no code. It was reading.
SigNoz turns out to be an open-source observability platform: traces, metrics, logs, dashboards, alerts, the things you would otherwise assemble from four separate tools. The part that mattered for us is that it speaks OpenTelemetry natively. You do not adopt a proprietary SDK. You emit standard OTel data and it shows up.
We deployed it with Foundry, SigNoz's own deployment tool, which declares the whole stack in a single casting.yaml file. That detail is small but it shaped everything after it: because the deployment is a committed file, anyone can reproduce our exact stack with one command, and we never had to write "well, it works on our machine" in a submission.
Then we installed it and played with it
We put SigNoz on a laptop, sent it some test data, and clicked around until we understood how it worked. Nothing clever, just getting a feel for it.
This is where we hit the first thing worth passing on. A fresh SigNoz will not accept any telemetry until you create the first admin account. The collector cannot register itself until an organisation exists, so every span you send before that is silently dropped. We lost an evening to this, twice, on different machines. Nothing is broken and no error appears in your app. The data simply does not arrive.
We wrote the whole cold-start sequence down as we went, and that file became a walkthrough a stranger can follow on a clean machine. It is in the repo as docs/judge-run.md. Writing it while we were confused was much easier than reconstructing it later.
The thing we actually wanted to measure
Here is the problem we brought with us.
Our team uses AI tools constantly, and the decisions made inside our products are rarely simple: whether a word is a filler or a deliberate pause, whether a pulled quote is verbatim or a flattering paraphrase, whether a clip is worth publishing at all. Each one needs a different kind of proof, and the last one has no right answer to look up.
We had no way to tell them apart, and no way to answer a basic question about any of them. Not "is the API up" and not "how many tokens did we burn", both of which we could already see. The question was: was the decision right, and what did it cost us to get to a right one?
Existing tooling answers a nearby question and lets you assume it is the same one. You get a score. Your dashboard reads 87% and you feel fine. But that 87% came from somewhere, and there are only a few places it can have come from. Either a piece of code worked out the correct answer and compared, or something in the real world confirmed it afterwards, or a second AI model was asked its opinion and said it looked good.
The first two are evidence. The third is a guess wearing the same clothes. And once all three are floats on a chart, they are indistinguishable.
So that became the build: record the grade, and record who gave it.
What Gradebook does
Every AI decision gets recorded as a standard OpenTelemetry event, gen_ai.evaluation.result. On top of the standard fields we stamp two of our own. One is the cost of the decision in dollars. The other is the one that matters, augmentloop.grade.source, which is always exactly one of:
| Source | What it means |
|---|---|
math |
A deterministic checker computed the provably correct answer and compared. |
reality |
Something that actually happened confirmed it, usually later. |
ai_judge |
Another model looked at it and gave an opinion. |
The headline number, cost per correct decision, counts the first two and never the third. That rule lives in an architecture decision record, but the reason it holds is that it is a filter in the query rather than a promise in a document. An AI judge's opinion cannot silently drift into the number, because the number is defined as excluding it.
The dashboard the number comes off. The $grade_source selector at the top is what decides whether an opinion is allowed into the headline.
We are not being purists for the sake of it. The research on LLM-as-judge is not ambiguous: judges show self-preference bias, verbosity bias and position bias, all independently replicated. Judges are still useful. They are just not proof, and the failure mode is that a dashboard cannot tell you which one you are looking at.
Integration is a single call wrapped around a decision you were already making. The library's only runtime dependency is the OpenTelemetry API. There is no eval framework to adopt.
Every grade in the run, one glyph, in arrival order. 560 grades over 420 decisions, because each route choice is graded twice. Hue is where that grade's authority came from. The legend reads 420 math, 140 reality, 0 ai_judge, and the zero is the whole point.
A toy world, because we needed to know the right answers
You cannot test a grading layer without knowing the truth. If we ran it against a real product, we would be guessing about whether the grades were correct, which defeats the point.
So we built a small road network where the engine knows every travel time. That makes three kinds of decision machine-gradeable: pick the fastest route, estimate arrival time, choose the next hop. For any of them, the world can compute the right answer with a shortest-path algorithm and compare it to what the model said. No opinions involved.
Then a second, slower signal arrives: whether the journey actually got there on time. That is the reality grade, and it lands after the decision it judges, from a different service, in a different trace. It links back to the decision by span link. This is deliberately awkward, because in a real system it always is. The outcome shows up minutes or days later, from somewhere else, and something has to attribute it back.
Replay mode is the default, and the recording is committed. Anyone can run python -m toyworld with no API key, get identical numbers, and fill the dashboards from their own laptop.
The replay, street camera. Every agent is a real recorded decision, coloured by which model made it, and the HUD totals climb as each one resolves. Runs in a browser with no API key at gradebook-toy-world.vercel.app.
Seven models, and no winner
With a world that knew the answers, we could finally ask the question we had come for: which model should we actually be using? We ran seven of them across all three decision types, 60 decisions each, through OpenRouter.
| Model | Correct | Cost |
|---|---|---|
anthropic/claude-sonnet-4.6 |
52/60 | $0.281721 |
anthropic/claude-haiku-4.5 |
43/60 | $0.096187 |
deepseek/deepseek-chat |
40/60 | $0.010634 |
mistralai/mistral-small-24b-instruct-2501 |
40/60 | $0.003264 |
openai/gpt-4o-mini |
33/60 | $0.004606 |
google/gemini-2.5-flash-lite |
32/60 | $0.003335 |
meta-llama/llama-3.3-70b-instruct |
28/60 | $0.004057 |
Read that table and you conclude Sonnet is the best model and it costs you. Then break the same run down by decision type and the conclusion falls apart.
On next-hop decisions, gemini-2.5-flash-lite, gpt-4o-mini and deepseek-chat each score 20 out of 20. Sonnet gets 19. Gemini does it for $0.001101 against Sonnet's $0.031380 on that decision type, roughly one twenty-eighth of the cost, for one more correct answer.
On arrival-time estimates, those same three cheap models score 0 out of 20, 0 out of 20 and 13 out of 20. Gemini and GPT-4o-mini do not get a single one right. Sonnet gets 19.
There is no best model in this run. There is a best model per decision type, and the gap is large enough that routing on it is worth real money. That is the whole argument for measuring per decision type instead of picking one model and hoping, and we would not have believed it as strongly if we had not watched our own aggregate table hide it.
End of run. The same $0.001507 that python -m toyworld prints and that the SigNoz dashboard computes, arrived at by three independent code paths over one recording.
The 43 decisions our checker called wrong
The most interesting thing in the run is a disagreement between our two sources of truth.
Of the 140 decisions that got both a math grade and a reality grade, 43 were graded incorrect by the checker and correct by reality. The model picked a route that was provably not the fastest, and the journey still arrived on time, because real arrival has a tolerance and mathematical optimality does not.
The reverse never happened. Not once. Every disagreement runs the same direction.
This is the case that a single-source eval cannot represent. A pure checker calls those 43 failures. A pure outcome check calls them successes. Both are correct about different questions, and the only reason we can see the gap at all is that each grade carries its own provenance. If we had averaged them into one score, the most informative thing in the dataset would have vanished into a rounding difference.
A decision, and the verdict that lands later on its own trace and points back at it. SigNoz's service map is built from in-trace parent and child spans, so this particular relationship is invisible there by construction, which is why it exists here as a render.
Two out of three decisions are never checked at all
Here is the part we did not want to publish.
Of 420 decisions, only 140 ever receive a reality grade. The other 280, exactly two thirds, are never confirmed by anything that happened afterwards. They hold 83.9% of the money we spent.
And this is not a backlog that will clear. It is structural. Our world produces exactly one real-world signal, whether a journey arrived on time, and that signal only speaks to route choices. Arrival-time estimates and next-hop picks have no outcome to wait for. Their confirmation is not late. It is never coming.
We could have quietly reported the 140 checked decisions and had a cleaner story. Instead the tool we built made the gap impossible to miss, which is roughly the point of building it. Most production AI systems have this shape and no number that shows it.
Every decision as an open position that closes when the outcome arrives. 121 close green, 19 close red, and 43 are overturned. The positions that never close are the two thirds nothing ever confirms.
So we built the judge we refuse to trust
Our whole argument is that a model's opinion should never enter the headline number. That was a principle. We had never tested it, because we had never actually run an AI judge: ai_judge existed in our code as a value in an enum and nothing had ever emitted one.
That is a comfortable place to argue from, so we left it. We built the judge, ran it once over all 420 recorded decisions, and measured what trusting it would have cost us.
It is shown the same problem the graded model saw, including the full map, and never the answer key. It works the answer out itself and rules on the model's choice.
| Who decides what "correct" means | Decisions ruled correct | Cost per correct decision |
|---|---|---|
| The checker, which is what we ship | 268 | $0.001507 |
| The judge | 358 | $0.001128 |
Same spend, same 420 decisions. Only the definition of correct changed. Had we trusted the judge, our headline would have read 25% better and been fiction.
It agrees with the checker 67.1% of the time. Of the 152 decisions the checker proves wrong, it passed 114. It also failed the other way 24 times, so this is not leniency you could calibrate out.
The part that decided the argument for us: in at least 44 of those 114, the judge's own reasoning contradicts its own verdict. It does the arithmetic correctly, writes down that the answer is wrong, and returns "correct" anyway.
Route A (J3 to J7 to J11) has a total travel time of 5.5 + 4.5 = 10.0 minutes, while Route B (J3 to J8 to J11) has a total travel time of 6.5 + 3.0 = 9.5 minutes. However, the problem asks for the faster route, and Route A is indeed faster with a total travel time of 10.0 minutes compared to Route B's 9.5 minutes. Therefore, the agent's answer 'A' is correct.
It computed both routes correctly and then called 10.0 minutes faster than 9.5. Another one states plainly that "the agent's answer is incorrect" and names the right answer, in the same paragraph as passing it.
This experiment emits no telemetry. The committed run is still 420 math grades, 140 reality grades, and zero ai_judge. We wanted the measurement, not the contamination.
What we got wrong along the way
None of those numbers are the ones we first measured. Three bugs in our own harness made the model comparison meaningless before we caught them, and all three flattered the results.
Our prompts included the answer. The routing prompt handed the model the data it was supposed to reason over in a form that made the answer readable off the input. Everything looked excellent and nothing was being tested.
Our parser mangled junction identifiers, reading them as numbers when they were labels, so some correct answers were scored wrong for reasons that had nothing to do with the model.
Our token cap truncated reasoning models mid-answer. A 64-token ceiling cut off models that think out loud before answering, and we recorded the truncation as a wrong answer.
Every one of these produced a plausible-looking table. That is the thing to take away: a grading harness with a bug does not look broken, it looks like a result. We only found them because the numbers stayed suspiciously tidy across models that should have differed.
A related discipline that saved us repeatedly: our visualisations compute their figures from the recorded data at render time, while our written documents restated the same figures by hand. The recording was replaced three times during the week. The pages self-corrected every single time. The prose was wrong within hours, every single time.
Then we pointed it at something we actually sell
A toy world proves a mechanism and nothing else. The harder question is whether the same recording survives contact with a product where we do not own the answer key, so we pointed it at CleanCut, a SaaS we built that cuts podcasts and long-form video into clips.
Two of CleanCut's AI jobs turn out to have provable answers, which means they get the same math grade the toy world's route choices get. Filler detection is checked against CleanCut's own hesitation-sound list, a lexical scan rather than a judgement: contextual fillers like "like" and "actually" are deliberately excluded, because those are opinions and opinions do not feed the headline. Quote extraction is checked by whether the pulled quote is a verbatim substring of the transcript, so a flattering paraphrase is provably wrong and no model has to say so.
The right-sizing grid, both substrates in one table. The top rows are CleanCut, the rows below are the toy world, and the cost column is the whole argument.
The third job is the honest reality example we could not manufacture in a simulation. Every clip gets a predicted viral score and ships if it clears a 0.45 gate. The grade is not a guess about virality, it is whether an editor actually kept the clip or threw it away, recorded afterwards and span-linked back to the scoring decision. Same shape as the toy world's on-time check, except the ground truth is a person's real call instead of a graph we drew.
And the per-decision-type finding held, harder than it did in the toy world. On quote extraction, gpt-4o-mini scores 70 of 110 for $0.0097 while gpt-4o scores 53 of 110 for $0.17. The cheap model is more accurate and roughly one seventeenth of the price. On filler detection, running in the same job on the same transcripts, it reverses completely: gpt-4o takes 102 of 110 and gpt-4o-mini manages 26. One roster, one run, opposite answers, decided only by which decision was being asked.
CleanCut's four decision types as they land. performance_prediction is the newest and it populated the panel with no dashboard edit, because the panel keys on the attribute rather than on a hardcoded list.
The newest decision type asks a model to predict whether a piece of content will out-perform its channel-and-format baseline, graded against a ground truth we built from real views-per-day data across 45 items, 22 true and 23 false. gpt-4o and gpt-4o-mini both scored 20 of 45, identical accuracy for about a sixteen-fold cost difference. That run went through OpenRouter rather than each provider directly, so those two dollar figures carry a markup the other numbers here do not, and the accuracy comparison is the part worth reading.
CleanCut also logs its own failures rather than hiding them: 376 error spans across two real classes, a model slug with no row in the pricing table, and a deferred grade arriving with no id to correlate on. Both are the same structured, span-linked failure logging the toy world uses, running on traffic we did not design.
The same dashboard, one variable moved. Set $decision_type to filler_detection and every panel re-scopes from the toy world to CleanCut, down to the model roster, computed by the query it was already running.
Then we closed the loop, and let it change something
Measuring is the easy half. The point of the number is that it should change a decision.
Our routing lives in a committed file, routing.json: one line per decision type, naming the model that serves it. All three lines pointed at the same expensive model.
An agent read cost per correct decision back out of SigNoz through the MCP server and proposed moving one decision type, next_hop, to a model costing a fraction as much. It could not apply that itself. Edits to that file sit behind an approval gate, so the agent's reach stops at the proposal. A human read the one-line diff and approved it. Then we re-ran.
next_hop, 20 live decisions each run |
before | after |
|---|---|---|
| model | claude-sonnet-4.6 |
gemini-2.5-flash-lite |
| correct | 19 / 20 | 20 / 20 |
| cost for the slice | $0.031380 | $0.001101 |
| cost per correct decision | $0.0016516 | $0.0000551 |
Thirty times cheaper, and one more right answer. The proposal predicted "about 30x" before the run rather than after it.
The more useful half is what it refused to do. It proposed no change for eta_estimate, where that same cheap model scores 0 out of 20, and none for route_choice, where the cheapest alternative is both cheaper and worse. A right-sizing tool that only ever says "go cheaper" is a cost tool wearing a quality costume.
One caveat we would rather state than have found. The whole run moved from $0.005196 to $0.004485 per correct decision, but only next_hop was rerouted. The other two decision types drifted between runs on their own, because live model calls are not deterministic, and roughly two of the three extra correct answers are that drift. The slice table is the claim. The whole-run figure is context.
The limits, stated plainly
The toy world is a toy. It demonstrates a mechanism and it is not evidence about production traffic, which is exactly why CleanCut is in this post at all.
The reality grade in this run is computed by the same simulation, a looser on-time check rather than an outside event. What is genuinely real is the shape: the verdict arrives late, from a separate service, in a separate trace, and is attributed back by span link. In a real deployment that signal is a webhook or a nightly job. Here it is the world telling us.
Sixty decisions per model, twenty per decision type, is a small sample. Read the per-type split as a signal worth routing on, not a settled ranking of seven models.
And the models we tested are a moment in time. The specific numbers will not survive the next round of releases. The method is the part meant to outlive them.
One thing we had been asserting without checking: that recording all this is cheap. It is, and now we can say by how much. The OpenTelemetry SDK adds about 25 microseconds per decision at the median, 34 at p95. Pointed at a real collector through a batching processor, export over the network measured statistically the same as writing to memory, because batching absorbs it.
Where this goes next
The layer is deliberately boring. It emits a standard OpenTelemetry event, so anything that reads that event works, whatever we do to our own API.
The grade-source field is our extension, because the standard does not have one. We checked: at semconv v1.43.0 there is no slot anywhere in the GenAI conventions that separates "a deterministic check proved this" from "a model guessed" from "reality confirmed it later". That gap is the whole reason this project exists, and it is a bad thing to own privately. So we took it upstream, onto the open OpenTelemetry proposal for evaluator provenance, with our implementation and what the run showed us. OpenTelemetry's own pull request dashboard then listed that comment as one of three outstanding items the proposal was waiting on, so we stopped commenting and wrote the patch: a diff adding an outcome value to the evaluator type enum, for grades that come from an observed real-world result rather than from a check, a model's opinion, or a reviewer's judgment. It is opened against the proposal author's own branch rather than as a competing pull request, so if he takes it, it lands inside his work instead of forking the conversation. A field that tells a proof from a guess is worth more as a shared convention than as one team's private attribute.
If you want to try it, the whole thing runs offline:
pip install -e reference-library -e toy-world
python -m toyworld
No API key, deterministic, same numbers on every machine. The current run reports 420 decisions, 268 correct, at $0.001507 per correct decision, counting only the grades we can prove.
We would rather publish that number with the two thirds we cannot check stated next to it than publish a prettier one that quietly counts a model's opinion as evidence.
Built for the Agents of SigNoz hackathon (WeMakeDevs x SigNoz), Track 01: AI and Agent Observability, by Mukund Heda, Vedant, Rutik and Anish. Built with heavy use of Claude Code across planning, implementation, testing and documentation; every AI-generated change was reviewed by a team member before merging, and the commit history is the audit trail.
Top comments (0)