Every AI eval tool will tell you how good your agent is. Almost none of them will tell you who decided.
That sounds like a small distinction until you look at how the number is produced. You run your agent, you run a second model over the output, the second model says 87% and that 87% goes on a dashboard, into a board deck, into a decision about which model to ship. Nobody writes down that the 87% is a model's opinion about another model. The number arrives wearing the same clothes as a number that was actually proven.
We built a thing called Gradebook for the Agents of SigNoz hackathon, and the one rule underneath all of it is that every grade carries where its authority came from, and the headline number counts only the grades that were proven.
Three ways to know an answer was right
A grade in Gradebook has exactly one source, and it is stamped on the telemetry as augmentloop.grade.source:
math means a checker computed the provably correct answer and compared. In our toy world the model picks a route between junctions and a Dijkstra shortest path computation says what the right route was. There is no interpretation involved. It is a fact.
reality means a second, independent check proved it, against a different question than math asks: not whether the choice was the provably shortest path, but whether the journey actually arrived inside tolerance. Also a fact, and an independent one, which turns out to matter more than we expected.
ai_judge means another model scored it. That is an opinion. The schema has a place for it, and it never touches the headline number.
The filter lives in the query rather than in a paragraph of documentation, which is the part that makes it real. Anyone can write "we take evaluation seriously" in a README. augmentloop.grade.source = 'math' in the dashboard query is a thing you can check.
Why the judge does not get a vote
Excluding AI judge grades started as taste and we were honest with ourselves that it was taste. Then we went and read the literature, and it stopped being taste.
LLM judges show self preference bias, meaning models systematically score their own outputs higher (arXiv 2410.21819). They show verbosity bias, rewarding longer answers independent of whether the longer answer is better. They show position bias, where the order you present two candidates in changes which one wins. These are independently replicated findings, not one contrarian paper (arXiv 2506.02592).
So if you are shipping a model and using a judge to decide whether the new one is better, and the judge is a sibling of one of the candidates, your headline metric has a thumb on the scale and no field anywhere in your telemetry records that fact. The problem is not that judges are useless, because they are genuinely useful for things no checker can express. The problem is that the output of a judge and the output of a checker look identical once they are both a float on a dashboard.
The check nobody's schema has a slot for
The reality source is the piece we think is actually new, and it is also the piece that does not fit anywhere in the current standards.
In our toy world a reality grade is computed synchronously, from the same graph as the math grade, not after real time has passed. What makes it a second authority is the question, not a delay: math asks whether the choice was the provably shortest path, and reality asks whether the journey actually arrived inside tolerance regardless of optimality. We still emit reality as its own evaluation event, span linked back to the original decision span, with the same gen_ai.response.id as a correlation fallback, because on a real product surface this is exactly the kind of check that resolves minutes or hours after the decision closes, and the telemetry shape has to support that even where, as here, it does not.
The two questions disagree, and always in the same direction. Of the 140 decisions carrying a reality grade in the current run, 43 were marked wrong by the shortest path checker and still arrived inside tolerance. Zero went the other way: nothing math called correct ever missed tolerance in the world. 121 of the 140 landed on time overall. Neither side is wrong. They compute different things, and a route can lose on one and win on the other.
While building this we found an open OpenTelemetry proposal to add evaluator provenance attributes to gen_ai.evaluation.*, with an enum of llm_judge, deterministic, human and custom. Our math maps cleanly onto deterministic and our ai_judge onto llm_judge, but reality also looks deterministic by that same test, since neither math nor reality involves an opinion. The enum has no way to say that two deterministic grades answer different questions, or that one of them is built to arrive after the fact on a real surface. We posted our implementation into that thread, since we had already hit the shape of the problem in running code.
What the honesty cost us
Here is the part that makes this more than a design essay.
Our headline metric had two defects, and both of them flattered us. The numerator summed only what the correct decisions cost, so every dollar spent on wrong answers was invisible, which is backwards for a project whose entire frame is the cost of being wrong. And it counted grades rather than decisions, so a decision that emitted both a math grade and a reality grade landed twice in the correct count while its cost landed once.
We fixed both. The numerator became all cost. The headline scoped to a single grade source so one decision can produce at most one grade in it, which kills the double count structurally instead of patching around it. Reality grades moved to their own adjacent panel where they are visible but not blended in.
Here is what that did to the number, on the run committed in the repo:
The flattering version divided $0.341049, the cost of only the correct decisions, by 389 grades, which was 268 math grades plus 121 reality grades that had already been counted once. It read $0.000877 per correct decision.
The honest version divides $0.403804, all the cost, by 268 correct decisions, each counted exactly once. It reads $0.001507 per correct decision.
The dashboard the number comes off. The $grade_source variable at the top is what decides whether an opinion is allowed into the headline.
We shipped a headline number 1.72 times worse than the one we could have shipped. Both are recomputable by hand from the recording in the repo, which is the only reason either is worth anything.
Our own grader was lying about the best model
The point of grading every decision is to find the cheapest model that is still good enough at each kind of decision. Our toy world runs three: pick the faster of two routes, estimate a journey time, choose the next junction. Seven models, across six providers, answer all sixty questions each.
For a while every model scored the same on everything, and we nearly wrote that up as a finding. It was not a finding. It was three bugs in our own harness, and we only went looking because a clean sweep is exactly the shape a broken test has.
The prompts were handing over the answers. The route question printed each candidate's total time next to it, so the task was to pick the smaller of two numbers already on the screen. The estimate question had the opposite defect and shipped no map at all, so the model was asked about a graph it had never seen and its zero score was blind guessing rather than any limit of the model.
Then, with real questions in place, the strongest model scored zero on the hardest type. That was also us. Our parser took the first number in the reply, and a model that shows its working opens with "starting from J1", so we graded it on the 1 inside a junction name and marked it wrong on answers it had got right. Underneath that sat a 64 token output cap, set back when every answer was a single token, which cut the model off mid calculation before it ever stated a result. We had built a grader that punished the only model doing the work.
With all three fixed:
| model | estimate a journey time | choose the next junction | pick the faster route | total cost |
|---|---|---|---|---|
| mistral-small-24b | 14/20 | 15/20 | 11/20 | $0.003264 |
| gemini-2.5-flash-lite | 0/20 | 20/20 | 12/20 | $0.003335 |
| llama-3.3-70b | 0/20 | 16/20 | 12/20 | $0.004057 |
| gpt-4o-mini | 0/20 | 20/20 | 13/20 | $0.004606 |
| deepseek-chat | 13/20 | 20/20 | 7/20 | $0.010634 |
| claude-haiku-4.5 | 18/20 | 16/20 | 9/20 | $0.096187 |
| claude-sonnet-4.6 | 19/20 | 19/20 | 14/20 | $0.281721 |
Same table, live in SigNoz. next_hop at the top: three models at 20/20, and the cost column is the whole argument.
That is a routing table rather than a leaderboard. On choosing the next junction, three of the seven models score a perfect 20/20: gemini-2.5-flash-lite, gpt-4o-mini, and deepseek-chat. claude-sonnet-4.6 scores 19/20 on the same task, at 28.5, 20.6, and 15.6 times the cost of the three that beat it, respectively. On estimating a journey time, two of those three collapse to zero: gemini and gpt-4o-mini. deepseek-chat does not join them there, scoring 13/20 instead. Sonnet reaches 19/20 at 188.8 times the cost of gemini, the cheapest of the two that scored zero. Gemini and gpt-4o-mini beat sonnet on one decision type and score zero against it on another, in the same run, decided only by which decision they were asked to make. The best score anyone posts on picking the faster route is 14 out of 20, and that is sonnet's own number, so this column stays a weak spot for the whole field rather than a place to rank anyone. No single ranking of these seven models survives all three columns, which is the whole argument for grading per decision type instead of publishing one number per model.
Worth saying plainly: two of those three bugs were ours before this work started, and both were invisible while every question could be answered in one token. Making the questions real is what exposed them.
The limits, stated out loud
We own the oracle in the toy world. We write the junctions, we compute the shortest path, we grade against our own answer key. That is a real limitation and it is worth saying before somebody says it for us.
It is also exactly what makes the grade machine checkable. The reason we built a toy world at all is that a provable answer key is the only way to get a grade with no opinion anywhere in it, and the moment you move to a real product surface you lose that and fall back to reality grades or to judges. So what the toy world proves is narrow. It proves the mechanism records provenance correctly, and the reality grade path is what has to carry that onto a real substrate.
The other limit is that the metric pipeline carries no per decision id, deliberately, because a response id as a metric label would explode cardinality. That means deduplication inside the metrics pipeline is impossible and any true count each decision once fix has to move to a trace query.
Run it
No API key, no model calls, same numbers on every machine:
pip install -e reference-library -e toy-world
python -m toyworld
It replays a committed recording of a real run, recomputes every grade from the answer key rather than reading a canned score out of the file, and fills the SigNoz dashboards from your own laptop. The recording holds what the models actually said. The grades are recomputed at replay time, which means we could not have hand written the answers even if we had wanted to. Every number in this post comes out of that command, including the two headline values and the table above.
Everything is at github.com/mukundheda/augmentloop-signoz-hackathon. The conventions doc is the part worth reading if you only read one file, because the recording contract is the actual contribution and the dashboards are downstream of it.
Built with heavy use of Claude Code across planning, implementation and documentation, reviewed by a human before merge, with the commit history as the audit trail.
Top comments (0)