I build AI-native infrastructure, mostly in Rust. Drop-in accelerators for the Python AI stack, local LLM inference, agent tooling, and a safer C. I ship in the open and write about how it works.
The monotonic false-positive drop paired with rising false-rejection is a calibration story, not a capability one: the stronger model sets a higher internal bar for 'acceptable' and starts rejecting valid-but-imperfect work. Using it as a hard gate forces you onto the worst point of the precision/recall curve.
What has worked better for me is treating the judge as a scorer with an explicit abstention band and only escalating the uncertain middle to a stronger model or a human, so you do not pay the 75% false-rejection tax on the easy cases. One more: you smoothed vote instability with majority-of-3, but that instability is signal. A scenario the judge cannot vote consistently on is exactly the one to escalate, not average away.
Dipankar, thank you – your calibration framing is exactly what I needed to read before I published the second post. And it turns out I tested exactly those suggestions in the third post, which just went live:
Let me go through each of your points through the lens of what the harness actually revealed.
Calibration, not capability
You're spot on. The strong model doesn't "understand" better in a way that solves the problem – it just sets a higher internal bar for "acceptable." In the third post, I found that the same model that caught all garbage also rejected 75% of valid work. This isn't a bug; it's calibration. The model's internal distribution of "good" is narrower than the task rubric, so it over‑rejects. I documented this as flaw #3: the harness had no explicit rubric calibration, so the judge defaulted to its own latent standard.
Scorer with abstention band + escalate the uncertain middle
This is exactly what I tried in the harness. I gave the judge a third option: "uncertain", to route to a stronger model or human. It failed – catastrophically. The strong model used "uncertain" as a backdoor, abstaining on ~40% of valid cases, which overloaded the fallback and defeated the purpose.
I realised (flaw #5 in the third post) that "uncertain" must be constrained – the model can only abstain when it can quote the exact missing evidence. Without that constraint, it's a permission to punt. I'm now re‑designing the abstention as a structured output where the model must cite the specific rubric criterion it cannot verify.
Vote instability as signal, not noise
This is the insight I ignored in the second post and rediscovered in the third. I smoothed with majority‑of‑3 and called it a day. But in the harness logs, I saw that the cases where the judge's three votes diverged were exactly the cases that later turned out to be ambiguous or mis‑classified by the human reviewer. I called this out as flaw #1 in the third post: the harness averaged away the instability instead of using it as an escalation trigger.
Your suggestion – "a scenario the judge cannot vote consistently on is exactly the one to escalate" – is now a core rule in my harness v2 design. Instead of taking the majority, I'll compute the vote entropy and route any scenario with entropy above a threshold directly to human review, bypassing the automated decision entirely.
What I'm taking forward
Your comment has reshaped the design:
Explicit calibration: the rubric must be concrete and included in the judge's prompt, so the model's internal bar is replaced with a task‑specific bar.
Constrained abstention: "uncertain" only allowed with cited missing evidence.
Vote entropy as first‑class signal: instability triggers escalation, not averaging.
I'm coding this into harness v2 this week. If you have a specific rubric‑calibration method that's worked for you, I'd love to hear it – your calibration framing has been one of the most useful mental models in this whole series.
Thanks again for pushing past the surface numbers.
For further actions, you may consider blocking this person and/or reporting abuse
We're a place where coders share, stay up-to-date and grow their careers.
The monotonic false-positive drop paired with rising false-rejection is a calibration story, not a capability one: the stronger model sets a higher internal bar for 'acceptable' and starts rejecting valid-but-imperfect work. Using it as a hard gate forces you onto the worst point of the precision/recall curve.
What has worked better for me is treating the judge as a scorer with an explicit abstention band and only escalating the uncertain middle to a stronger model or a human, so you do not pay the 75% false-rejection tax on the easy cases. One more: you smoothed vote instability with majority-of-3, but that instability is signal. A scenario the judge cannot vote consistently on is exactly the one to escalate, not average away.
Dipankar, thank you – your calibration framing is exactly what I needed to read before I published the second post. And it turns out I tested exactly those suggestions in the third post, which just went live:
👉 [dev.to/zxpmail/i-designed-a-harnes...]
Let me go through each of your points through the lens of what the harness actually revealed.
Calibration, not capability
You're spot on. The strong model doesn't "understand" better in a way that solves the problem – it just sets a higher internal bar for "acceptable." In the third post, I found that the same model that caught all garbage also rejected 75% of valid work. This isn't a bug; it's calibration. The model's internal distribution of "good" is narrower than the task rubric, so it over‑rejects. I documented this as flaw #3: the harness had no explicit rubric calibration, so the judge defaulted to its own latent standard.
Scorer with abstention band + escalate the uncertain middle
This is exactly what I tried in the harness. I gave the judge a third option: "uncertain", to route to a stronger model or human. It failed – catastrophically. The strong model used "uncertain" as a backdoor, abstaining on ~40% of valid cases, which overloaded the fallback and defeated the purpose.
I realised (flaw #5 in the third post) that "uncertain" must be constrained – the model can only abstain when it can quote the exact missing evidence. Without that constraint, it's a permission to punt. I'm now re‑designing the abstention as a structured output where the model must cite the specific rubric criterion it cannot verify.
Vote instability as signal, not noise
This is the insight I ignored in the second post and rediscovered in the third. I smoothed with majority‑of‑3 and called it a day. But in the harness logs, I saw that the cases where the judge's three votes diverged were exactly the cases that later turned out to be ambiguous or mis‑classified by the human reviewer. I called this out as flaw #1 in the third post: the harness averaged away the instability instead of using it as an escalation trigger.
Your suggestion – "a scenario the judge cannot vote consistently on is exactly the one to escalate" – is now a core rule in my harness v2 design. Instead of taking the majority, I'll compute the vote entropy and route any scenario with entropy above a threshold directly to human review, bypassing the automated decision entirely.
What I'm taking forward
Your comment has reshaped the design:
Explicit calibration: the rubric must be concrete and included in the judge's prompt, so the model's internal bar is replaced with a task‑specific bar.
Constrained abstention: "uncertain" only allowed with cited missing evidence.
Vote entropy as first‑class signal: instability triggers escalation, not averaging.
I'm coding this into harness v2 this week. If you have a specific rubric‑calibration method that's worked for you, I'd love to hear it – your calibration framing has been one of the most useful mental models in this whole series.
Thanks again for pushing past the surface numbers.