DEV Community: dys5315

Your model can't grade its own homework

dys5315 — Wed, 29 Jul 2026 03:27:50 +0000

Every team I've watched ship a broken measurement system broke it the same way. Not with bad math — with an org chart problem that happened to live in code.

The entity making the claim ended up being the entity that decided whether the claim was right.

Once you have the shape in your head you start seeing it everywhere.

Three roles, not two

Most engineers think about measurement as two roles: the thing that acts, and the thing that grades it. That's one role short. There are three:

Player — makes the claim. Your model, your service, your PR.
Scorer — applies the rubric. Your eval harness, your test suite, your metrics dashboard.
Settler — determines what actually happened. Production outcomes. Reality.

The scorer is a proxy. The settler is the thing the proxy is trying to approximate.

The rule: be the scorer, never the settler. When the player captures the settler, the loop closes on itself and the system can no longer be wrong — which sounds like success and is actually the failure.

What it looks like in code

Tuning on the test set. You check test accuracy, adjust hyperparameters, check again. Twenty iterations later the test set is training data with extra steps. The player is now selecting its own settler. That's what overfitting is, structurally — not a math failure, a role-collapse failure.

LLM-as-judge from the same family. Your generator is GPT-flavored and your judge is GPT-flavored. They share pretraining data, failure modes, and blind spots. The judge doesn't rate quality — it rates similarity to what it would have produced. Correlated error is invisible to averaging; running it 1,000 times makes you more confident of the same wrong answer.

Benchmark contamination. The model scores 94% on the benchmark that's in its training data. Nobody lied. The settler just quietly moved inside the player.

Self-reported health. A service that returns its own health check is a claimant ruling on its own claim. If the process is wedged, the check is wedged too, and your dashboard is green through the outage.

Backtests without decision-time reconstruction. You score a historical decision using a feature computed after the outcome was known. The past gets graded with information from the future. Your Sharpe ratio is beautiful and entirely fictional.

Same failure, five costumes: the claimant captured the settler.

The fix is structural, not moral

Nobody in the above list was dishonest. That's the point — this doesn't get solved with care or seniority, because it isn't a discipline problem. Independence has to be engineered or it decays under ordinary pressure.

What that means concretely:

Make the settler expensive to reach. A holdout you can query freely is not a holdout. Rate-limit it, log every access, treat a query as a spend. If the number of times you've looked at the test set isn't recorded anywhere, you've already lost track of how contaminated it is.

Pick judges from a different failure family. If a human can't review it, at minimum use a different model lineage. You want errors that are uncorrelated, not errors that agree with each other.

Let production be the settler. Ship the prediction, capture the realized outcome, compare. Anything upstream of reality is a proxy, and every proxy is a scorer wearing a settler's coat.

Reconstruct decision-time state. Before scoring a historical decision, rebuild exactly what was knowable at that moment. If that's hard to do, that difficulty is the finding — it means your current backtest is leaking and you didn't know.

Watch who owns the metric. If the team that ships the feature also defines success and reports the number, you don't have a measurement — you have a press release with error bars.

The check that catches it

One question, applied to any number your system produces:

If this claim were wrong, what would tell us — and is that thing under our control?

If nothing would tell you, the claim isn't false, it's unfalsifiable, which is worse. And if the thing that would tell you is something you own, tune, or can silence, you're the settler and the loop is closed.

A closed loop doesn't fail loudly. It drifts, confidently, until reality collects — and reality runs on its own clock, not your sprint schedule.

Why I keep thinking about it

I build software that tells medical practices which claims are worth appealing. The settler is the payer's actual remittance. Not our model's confidence, not our internal rubric, not what our dashboard says. If we ever let those become the settler, we'd have a product that looks excellent and is quietly lying to people who make payroll decisions with it.

I ended up writing this up properly — the same structure turns out to show up in market bubbles, in overfitting, and in a few places I didn't expect: The Uncapturable Judge.

Where's the closed loop in your stack? Mine was the health check, and it took an outage to find it.

How many outcomes before you can quote a percentage?

dys5315 — Wed, 29 Jul 2026 03:26:50 +0000

Your model outputs 0.87. Someone puts it on a dashboard labeled "87% likely to succeed." A customer makes a decision with it.

How many resolved outcomes did you actually need before that number meant anything?

There's a closed-form answer, it's older than all of us, and most teams never run it.

The number

For a proportion you want to quote within ±ε at 95% confidence:

n ≥ p̂(1 − p̂) · (z / ε)²

z = 1.96 for 95%. That's it. Run it:

from math import ceil

def min_n(p_hat: float, eps: float, z: float = 1.96) -> int:
    """Resolved outcomes needed to quote p_hat within ±eps."""
    return ceil(p_hat * (1 - p_hat) * (z / eps) ** 2)

for p in (0.5, 0.8, 0.9):
    print(p, min_n(p, 0.05))
# 0.5 385
# 0.8 246
# 0.9 139

385 resolved outcomes to say "50%, give or take 5 points." Not 385 rows in your training set — 385 cases where you found out what actually happened.

The part that hurts

Precision is quadratic. Tightening ε by 5× costs 25× the data:

for eps in (0.10, 0.05, 0.02, 0.01):
    print(eps, min_n(0.5, eps))
# 0.1   97
# 0.05  385
# 0.02  2401
# 0.01  9604

Going from "roughly half" to "50% ± 1 point" costs about 9,500 outcomes. If you're in a domain where outcomes take 90 days to resolve, that's not a sprint — that's years.

Read it backwards

The more useful direction is inverting it. Instead of asking how much data you need, ask what your current data licenses you to say:

def widest_claim(p_hat: float, n: int, z: float = 1.96) -> float:
    """The ± band your evidence actually supports."""
    return z * (p_hat * (1 - p_hat) / n) ** 0.5

print(round(widest_claim(0.5, 50), 3))   # 0.139

At n=50, your honest claim is ±14 points. So "50%" is really "somewhere between 36% and 64%." That's not a probability you should be printing next to a dollar amount.

This reframes the whole thing. You don't have a precision problem, you have a vocabulary problem: at n=50 you're licensed to say bands, not percentages. Say the band.

Wilson, not the naive interval

One correction: the formula above is the normal approximation, and it degrades badly at small n or extreme p̂. It'll happily hand you an interval that extends past 1.0. Use Wilson instead:

def wilson(successes: int, n: int, z: float = 1.96) -> tuple[float, float]:
    if n == 0:
        return (0.0, 1.0)
    p = successes / n
    d = 1 + z**2 / n
    centre = (p + z**2 / (2 * n)) / d
    half = z * ((p * (1 - p) / n + z**2 / (4 * n**2)) ** 0.5) / d
    return (centre - half, centre + half)

print(wilson(9, 10))   # (0.5958, 0.9821)

Nine out of ten successes is not "90% reliable." It's "somewhere between 60% and 98%," which is a very different sentence to put in front of a customer.

Making it a constraint, not a guideline

Here's the part that matters in production: a rule nobody enforces is a rule that erodes under deadline pressure. Documentation won't save you. Put the gate in the code path.

MIN_N = 385

def render_score(p_hat: float, n_resolved: int) -> str:
    if n_resolved < MIN_N:
        lo, hi = wilson(round(p_hat * n_resolved), n_resolved)
        return f"band: {lo:.0%}–{hi:.0%} (n={n_resolved})"
    return f"{p_hat:.0%} (n={n_resolved})"

Now the honest version is the default path, and overclaiming requires someone to delete code in a reviewed PR. That's a very different organizational fact than a wiki page saying "be careful with probabilities."

One trap: which n?

n is resolved outcomes, and resolution is where the bias sneaks in.

If your wins auto-resolve from downstream data but your losses require somebody to manually mark them, your resolved set is not a random sample of reality. It's skewed toward wins, and every interval you compute on it is confidently wrong. Missing-not-at-random data doesn't announce itself — it just makes your calibration look great.

Before you trust the number, check whether the mechanism that produces outcomes differs by outcome type. That check has caught more bad dashboards for me than any model diagnostic.

Why I care about this

I build billing software. When our system says a claim is likely to be recovered, somebody staffs against that number and a practice makes a payroll decision. The cost of a confidently wrong percentage is somebody's month.

So we ship bands until the volume clears, and the product is architecturally forbidden from saying "probability" before then. It costs us a nicer-looking demo. It has never once cost us a customer.

If you want the longer argument — what it means for a system to be structurally forbidden from overclaiming, and why the entity that makes a claim must never be the one that settles whether it was right — I wrote it up here: The Uncapturable Judge.

What's your team's rule for when a score gets to call itself a probability?