I build AI-native infrastructure, mostly in Rust. Drop-in accelerators for the Python AI stack, local LLM inference, agent tooling, and a safer C. I ship in the open and write about how it works.
The monotonic false-positive drop paired with rising false-rejection is a calibration story, not a capability one: the stronger model sets a higher internal bar for 'acceptable' and starts rejecting valid-but-imperfect work. Using it as a hard gate forces you onto the worst point of the precision/recall curve.
What has worked better for me is treating the judge as a scorer with an explicit abstention band and only escalating the uncertain middle to a stronger model or a human, so you do not pay the 75% false-rejection tax on the easy cases. One more: you smoothed vote instability with majority-of-3, but that instability is signal. A scenario the judge cannot vote consistently on is exactly the one to escalate, not average away.
Dipankar, thank you – your calibration framing is exactly what I needed to read before I published the second post. And it turns out I tested exactly those suggestions in the third post, which just went live:
Let me go through each of your points through the lens of what the harness actually revealed.
Calibration, not capability
You're spot on. The strong model doesn't "understand" better in a way that solves the problem – it just sets a higher internal bar for "acceptable." In the third post, I found that the same model that caught all garbage also rejected 75% of valid work. This isn't a bug; it's calibration. The model's internal distribution of "good" is narrower than the task rubric, so it over‑rejects. I documented this as flaw #3: the harness had no explicit rubric calibration, so the judge defaulted to its own latent standard.
Scorer with abstention band + escalate the uncertain middle
This is exactly what I tried in the harness. I gave the judge a third option: "uncertain", to route to a stronger model or human. It failed – catastrophically. The strong model used "uncertain" as a backdoor, abstaining on ~40% of valid cases, which overloaded the fallback and defeated the purpose.
I realised (flaw #5 in the third post) that "uncertain" must be constrained – the model can only abstain when it can quote the exact missing evidence. Without that constraint, it's a permission to punt. I'm now re‑designing the abstention as a structured output where the model must cite the specific rubric criterion it cannot verify.
Vote instability as signal, not noise
This is the insight I ignored in the second post and rediscovered in the third. I smoothed with majority‑of‑3 and called it a day. But in the harness logs, I saw that the cases where the judge's three votes diverged were exactly the cases that later turned out to be ambiguous or mis‑classified by the human reviewer. I called this out as flaw #1 in the third post: the harness averaged away the instability instead of using it as an escalation trigger.
Your suggestion – "a scenario the judge cannot vote consistently on is exactly the one to escalate" – is now a core rule in my harness v2 design. Instead of taking the majority, I'll compute the vote entropy and route any scenario with entropy above a threshold directly to human review, bypassing the automated decision entirely.
What I'm taking forward
Your comment has reshaped the design:
Explicit calibration: the rubric must be concrete and included in the judge's prompt, so the model's internal bar is replaced with a task‑specific bar.
Constrained abstention: "uncertain" only allowed with cited missing evidence.
Vote entropy as first‑class signal: instability triggers escalation, not averaging.
I'm coding this into harness v2 this week. If you have a specific rubric‑calibration method that's worked for you, I'd love to hear it – your calibration framing has been one of the most useful mental models in this whole series.
Thanks again for pushing past the surface numbers.
We're a place where coders share, stay up-to-date and grow their careers.
The monotonic false-positive drop paired with rising false-rejection is a calibration story, not a capability one: the stronger model sets a higher internal bar for 'acceptable' and starts rejecting valid-but-imperfect work. Using it as a hard gate forces you onto the worst point of the precision/recall curve.
What has worked better for me is treating the judge as a scorer with an explicit abstention band and only escalating the uncertain middle to a stronger model or a human, so you do not pay the 75% false-rejection tax on the easy cases. One more: you smoothed vote instability with majority-of-3, but that instability is signal. A scenario the judge cannot vote consistently on is exactly the one to escalate, not average away.
Dipankar, thank you – your calibration framing is exactly what I needed to read before I published the second post. And it turns out I tested exactly those suggestions in the third post, which just went live:
👉 [dev.to/zxpmail/i-designed-a-harnes...]
Let me go through each of your points through the lens of what the harness actually revealed.
Calibration, not capability
You're spot on. The strong model doesn't "understand" better in a way that solves the problem – it just sets a higher internal bar for "acceptable." In the third post, I found that the same model that caught all garbage also rejected 75% of valid work. This isn't a bug; it's calibration. The model's internal distribution of "good" is narrower than the task rubric, so it over‑rejects. I documented this as flaw #3: the harness had no explicit rubric calibration, so the judge defaulted to its own latent standard.
Scorer with abstention band + escalate the uncertain middle
This is exactly what I tried in the harness. I gave the judge a third option: "uncertain", to route to a stronger model or human. It failed – catastrophically. The strong model used "uncertain" as a backdoor, abstaining on ~40% of valid cases, which overloaded the fallback and defeated the purpose.
I realised (flaw #5 in the third post) that "uncertain" must be constrained – the model can only abstain when it can quote the exact missing evidence. Without that constraint, it's a permission to punt. I'm now re‑designing the abstention as a structured output where the model must cite the specific rubric criterion it cannot verify.
Vote instability as signal, not noise
This is the insight I ignored in the second post and rediscovered in the third. I smoothed with majority‑of‑3 and called it a day. But in the harness logs, I saw that the cases where the judge's three votes diverged were exactly the cases that later turned out to be ambiguous or mis‑classified by the human reviewer. I called this out as flaw #1 in the third post: the harness averaged away the instability instead of using it as an escalation trigger.
Your suggestion – "a scenario the judge cannot vote consistently on is exactly the one to escalate" – is now a core rule in my harness v2 design. Instead of taking the majority, I'll compute the vote entropy and route any scenario with entropy above a threshold directly to human review, bypassing the automated decision entirely.
What I'm taking forward
Your comment has reshaped the design:
Explicit calibration: the rubric must be concrete and included in the judge's prompt, so the model's internal bar is replaced with a task‑specific bar.
Constrained abstention: "uncertain" only allowed with cited missing evidence.
Vote entropy as first‑class signal: instability triggers escalation, not averaging.
I'm coding this into harness v2 this week. If you have a specific rubric‑calibration method that's worked for you, I'd love to hear it – your calibration framing has been one of the most useful mental models in this whole series.
Thanks again for pushing past the surface numbers.