We’ve built a powerful Forensic Team. They can find books, analyze metadata, and spot discrepancies using MCP.
But in the enterprise, 'it seems to work' isn't a metric. If an agent misidentifies a $50,000 first edition, the liability is real.
Today, we move from Subjective Trust to Quantitative Reliability. We are building The Judge—a high-reasoning evaluator that audits our Forensic Team against a 'Golden Dataset' of ground-truth facts.
Before you Begin
Prerequisites: You should have an existing agentic workflow (see my MCP Forensic Series) and a high-reasoning model (Claude 3.5 Opus/GPT-4o) to act as the Judge.
1. The "Golden Dataset"
Before we can grade the agents, we need an Answer Key. We’re creating tests/golden_dataset.json. This file contains the "Ground Truth"—scenarios where we know there are errors.
Example Entry:
{
"test_id": "TC-001",
"input": "The Great Gatsby, 1925",
"expected_finding": "Page count mismatch: Observed 218, Standard 210",
"severity": "high"
}
Director's Note: In an enterprise setting, "Reliability" is the precursor to "Permission". You will not get the budget to scale agents until you can prove they won't hallucinate $50k errors. This framework provides the data you need for that internal sell.
2. The Judge's Rubric
A good Judge needs a rubric. We aren't just looking for "Yes/No." We want to grade on:
- Precision: Did it find only the real errors?
- Recall: Did it find all the real errors?
- Reasoning: Did it explain why it flagged the record?
3. Refactoring for Resilience
Before building the Judge, we had to address a common "Senior-level" trap: hardcoding agent logic. Based on architectural reviews, we moved our system prompts from the Python client into a dedicated config/prompts.yaml.
This isn't just about clean code; it’s about Observability. By decoupling the "Instructions" from the "Execution," we can now A/B test different prompt versions against the Judge to see which one yields the highest accuracy for specific models.
4. The Implementation: The Evaluation Loop
We’ve added evaluator.py to the repo. It doesn't just run the agents; it monitors their "vital signs."
- Error Transparency: We replaced "swallowed" exceptions with structured logging. If a provider fails, the system logs the incident for diagnosis instead of failing silently.
- The Handshake: The loop runs the Forensic Team, collects their logs, and submits the whole package to a high-reasoning Judge Agent.
The Evaluator-Optimizer Blueprint
This diagram represents our move from "Does the code run?" to Does the intelligence meet the quality bar?" This closed-loop system is required before we can start the fiscal optimization of choosing smaller models to handle simpler tasks.
Director-Level Insight: The "Accuracy vs. Cost" Curve
As a Director, I don't just care about "cost per token." I care about Defensibility. If a forensic audit is challenged, I need to show a historical accuracy rating. By implementing this Evaluator, we move from "Vibe-checking" to a Quantitative Reliability Score. This allows us to set a "Minimum Quality Bar" for deployment. If a model update or a prompt change drops our accuracy by 2%, the Judge blocks the deployment.
The Production-Grade AI Series
- Post 1: The Judge Agent — You are here
- Post 2: The Accountant (Cognitive Budgeting & Model Routing)
- Post 3: The Guardian (Human-in-the-Loop Handshakes)
Looking for the foundation? Check out my previous series: The Zero-Glue AI Mesh with MCP.
Top comments (14)
ken, you're calling me out with the vibe-check comment — i usually just ship and pray. i’m still figuring it out in cursor, but keeping the context straight is getting harder as my apps grow. a judge agent feels like the right way to stop the guessing game when the vibes aren't enough. thanks for showing how to build the bridge between shipping fast and actually knowing it’s right.
That 'ship and pray' era was fun, but as the stakes get higher, the 'praying' part gets a lot more stressful. I think we're seeing a shift where the best developers won't be the ones who can code the fastest, but the ones who can build the best governance loops around what they ship.
Once the app grows, the 'context' isn't just a technical limit, it’s the integrity of the whole system. Glad the 'Judge' concept resonated with you. It’s definitely saved me from a few 'vibe-only' hallucinations.
Thanks for sharing! You are hitting on a massive blind spot in the current agent space.
Relying on a probabilistic LLM to summarize its own context is an architectural trap—you are essentially paying a Prose Tax to generate a summary that might introduce drift or hallucinate historical state.
Pushing context compression down to a deterministic layer before it ever hits the model is exactly how we keep local-first systems reliable and cost-effective on consumer silicon.
This is a great move from “it works” to “we can prove it works,” which is exactly what enterprise AI needs.
I like the focus on a Golden Dataset with known failures, most teams skip that and only test happy paths. The precision/recall + reasoning rubric is also spot on, especially since weak explanations can become a real issue during audits.
Decoupling prompts into config is a nice touch too it makes proper A/B testing and iteration actually possible.
One thing I’m curious about: how do you handle partial matches or differently worded outputs when scoring? That part can get tricky if the Judge isn’t tightly constrained.
Overall, this feels like a solid foundation for building measurable and defensible agent reliability.
That is a great question, and you’ve highlighted the exact reason why traditional string-matching fails in enterprise AI. To handle the 'semantic overlap' without the Judge hallucinating its own criteria, I generally use a three-layered approach:
Semantic Embedding Checks (The Pre-Filter)
Before the Judge even sees the text, I often run a quick cosine similarity check between the agent’s output and the Golden Dataset. If the score is high (e.g., >0.92) but the words don't match, it tells the system we have a 'differently worded but likely correct' candidate. This helps prioritize which outputs need the most 'reasoning' from the Judge.
The 'Facts vs. Style' Rubric
In the Judge's prompt, I explicitly decouple Factuality from Fluency. I instruct the Judge to extract the 'claims' from the agent's response and verify them individually against the Golden Dataset.
• Example: If the agent says 'The book was printed in 1884' and the Golden Data says 'Date of Publication: 1884,' the Judge scores that as a 1/1 match for precision, regardless of the surrounding prose.
Few-Shot 'Edge Case' Examples
The best way to tighten a Judge is to provide it with 3–5 examples of 'Partial Matches' in the prompt itself. I show it an example of a 'technically correct but incomplete' answer and an example of a 'differently worded but perfect' answer. This 'sets the bar' for what constitutes a win.
Forced Reasoning (CoT)
As you noted, weak explanations are a liability. I force the Judge to write the 'Evidence' for its score before it outputs the numerical grade. If the Judge can't cite a specific discrepancy in its reasoning, the score is flagged for human review.
It’s definitely a balancing act—too loose and you get false positives; too tight and you’re back to hardcoded regex. I’ve found that focusing the Judge on claim extraction is the most defensible path for an audit trail!
That makes a lot of sense, especially the claim extraction piece. Framing it as “facts vs. style” feels like the right abstraction, it keeps the Judge focused on what actually matters instead of getting distracted by wording.
I also like the idea of using embeddings as a pre-filter rather than the final decision-maker. It’s a nice way to reduce unnecessary load on the Judge without over-trusting similarity scores.
The few shot edge cases point is interesting too, it’s easy to underestimate how much those examples actually shape the Judge’s behavior, especially around partial correctness.
Out of curiosity, have you run into situations where the claim extraction itself becomes inconsistent? I could see that being another layer where things drift a bit depending on how the model interprets “claims.”
You’ve touched on the 'meta-problem' of LLM-as-a-Judge: who audits the claim extractor? You’re absolutely right—if the model interprets 'claims' differently every time, your precision/recall metrics become noise.
In my experience, 'claim drift' usually happens when the prompt is too open-ended (e.g., 'List all claims'). To stop that drift, I’ve moved toward a Constrained Extraction Pipeline:
• Bad Claim: 'The book was a 1884 first edition in good condition.' (Too many variables).
• Atomic Claims: 1. 'Pub Date is 1884.' 2. 'Edition is First.' 3. 'Condition is Good.'
This makes the 'Fact vs. Style' comparison a 1:1 binary match, which is much easier for a downstream Judge to score consistently.
Schema Enforcement (Pydantic/Zod)
I never let the extractor return raw text. I force it into a strict JSON schema where each claim must be categorized (e.g., Date, Identifier, Physical_Trait). By forcing the model to 'pigeonhole' its thoughts, you drastically reduce the chance of it getting 'creative' with how it phrases a claim.
The 'Decomposition' Sanity Check
For high-stakes audits, I actually run the extraction twice (sometimes with a smaller, faster model like Haiku or GPT-4o-mini) and have a simple logic gate check if the number of claims matches. If Model A finds 4 claims and Model B finds 7, the system flags it for 'Extraction Variance' before the Judge even gets involved.
Reference-Anchored Extraction
Instead of saying 'Extract claims,' I say 'Extract claims that relate to these specific keys in the Golden Dataset.' This anchors the model's 'attention' to only what matters for the audit, preventing it from wasting reasoning tokens on stylistic filler.
Ultimately, the goal is to move the extraction from a 'creative summary' to a 'data parsing' task. It’s definitely an extra layer of engineering, but for defensible agent reliability, that extra layer is the only thing that keeps the metrics honest!
I've been running a similar evaluation loop in production for 14 months. 340+ test scenarios, 85 models, 101 different harness configurations. Independent judge model completely decoupled from any provider being evaluated, so nobody grades their own homework.
Two things I learned the hard way:
The harness matters as much as the judge. Same model, same test, same judge, different harness configuration: scores swing 36 points. If you're not controlling for how the agent accesses tools and manages memory during evaluation, your precision/recall numbers are measuring the wrapper, not the agent.
The judge itself needs adversarial testing. I run 160 adversarial resistance checks against my own evaluation pipeline every deploy. Calibration agents with known-good and known-bad behaviors confirm the judge isn't drifting. Without that, you're trusting the auditor the same way everyone else trusts the agent: on vibes.
Decomposing claims into atomic verifiable facts is what separates measurement from narrative. Most benchmarks in the wild skip that step and report a single composite score that hides everything interesting.
Have you seen the harness effect in your forensic evaluations? Different prompt templates, different tool access patterns, same underlying model. The delta is usually larger than people expect.
A 36-point swing based purely on the harness configuration is a massive callout, and it tracks perfectly with what I’ve observed.
The "Harness Effect" is the industry’s open secret right now. If we aren't locking down tool-access patterns, system prompts, and memory compaction strategies during the evaluation loop, we aren't benchmarking the model—we're benchmarking the engineering choices of the orchestration framework. It’s why I argue that forensic evaluation must decouple the agent's environment entirely from the evaluation environment.
I really like your approach of running adversarial resistance checks against the judge on every deploy. Running calibration agents with intentional failure modes is exactly how we move away from "vibes-based" testing. If the auditor can't reliably spot an intentional hallucination, the auditor shouldn't have a job.
Most evaluation frameworks run the agent and the judge through the same system prompt template. Any prompt-level gaming strategy that inflates the agent's score also inflates the judge's perception of it. Decoupling the environment only works if you also decouple the instruction context. Otherwise you just moved the shared vulnerability one layer up.
The part I haven't solved cleanly is longitudinal drift. A judge that's calibrated today can silently degrade after a provider-side model update that nobody announces. I'm catching it reactively with a drift detector that compares calibration runs against a baseline, but the detection window is still after the fact. By the time the drift shows up in my data, it's already been scoring production runs with a compromised judge for however long the gap was.
Have you found a way to make judge drift detection predictive rather than reactive? That's the unsolved edge I keep running in to.
Spot on. If the agent and the judge share an underlying prompting lineage or are evaluated within the same ecosystem context, you aren't auditing—you're just reflecting an architectural echo chamber.
On your question about longitudinal drift: predicting provider-side model updates before they happen is the holy grail, and because it's a closed-source black box, we can't do it deterministically. However, we can shift the timeline from reactive post-facto detection to what I call a Proactive Canary Loop.
For example, instead of analyzing drift from production logs after the evaluation has been compromised, we run a continuous, high-frequency background worker that pushes your calibration scenarios through the judge every single hour—treating it like a synthetic transaction check. If a provider silently slips an optimization or weights adjustment into their API at 2:00 AM, your canary breaches its SLO and alerts you by 3:00 AM, giving you a window to isolate the judge before the bulk of your production traffic experiences quality degradation. It’s still technically a lagging indicator, but you're compressing the detection window down to a narrow operational boundary.
I run daily automated calibration plus a deploy-triggered pass, so the window right now is 24 hours worst case. Compressing that to an hour is achievable but the cost equation changes. My judge model is GLM-5 Turbo via OpenRouter. Running 135 calibration checks hourly instead of daily is a 24x increase in judge API calls with zero revenue attached. At scale that's manageable. At 26 users (on tabverified.ai and one of them is me) it's a line item that needs to earn its keep.
The threshold sensitivity is the part that gets interesting. A provider-side update that moves a score from 95 to 93 is noise. One that moves it from 95 to 71 is drift. The canary has to distinguish between those without generating false alerts on normal variance. Right now my expected ranges have a 20-point critical drift threshold. Tightening that for an hourly loop means either accepting more false positives or building a moving baseline that adapts to organic scorer variance. Neither is free.
What's your approach to setting the alert boundary on the canary? Fixed threshold or adaptive?
You’ve hit the nail on the head regarding the economics of observability. At 26 users, scaling your GLM-5 Turbo API calls by 24x for an hourly calibration loop is paying a massive infrastructure tax with no immediate ROI. Cost constraints are architecture constraints, period.
To answer your question on boundaries: I lean away from fully adaptive baselines because they introduce a meta-problem—now you have to debug the statistical drift of your drift detector, which is never free.
Instead, the cleanest way to close this loop without ballooning your budget or chasing false positives is a Multi-Layered Fixed Boundary:
Keep the Daily Loop: Keep your comprehensive 135-point pass on a daily/deploy schedule to capture macro drift.
Implement a Micro-Canary: For the hourly loop, drop the 135 checks down to a bare-minimum "critical 5" unit test set.
Set a Hard Floor: Treat a minor 95-to-93 variance as acceptable model noise, but set a hard floor at your 20-point critical threshold. If the micro-canary drops past that floor once, don't page yet. If it drops past it for two consecutive hours, page.
This keeps your OpenRouter line-item tiny while filtering out transient variance. Ultimately, balancing that cost-to-signal ratio is what separates sustainable production monitoring from lab experiments. Thanks for grinding through the practicalities of this with me. This is exactly how we move the industry past vibe-based evaluation!
Some comments may only be visible to logged-in visitors. Sign in to view all comments.