Your Grader Is Drifting Too: A Self-Auditing Prompt Eval Harness

The most fragile component of a prompt evaluation suite is not the prompt under test; it is the grader that decides whether an output is acceptable. Teams assemble golden sets, wire in a model judge, and then trust a green checkmark that nobody ever verifies. This article builds a three-tier grading harness that audits its own judge, stays inside a free token budget, and runs unattended on a free server.

The pattern maps cleanly onto MonkeyCode, an open source project whose free tier includes a substantial model token allowance and a free server option for scheduled workloads. Disclosure: This article was prepared as part of MonkeyCode's product outreach. In this setup the free token quota pays for judge calls while the server owns the cron schedule, so continuous evaluation costs no money and no dedicated hardware. Quotas and endpoints change over time, so verify the current numbers in the project documentation before relying on them.

Recent DEV discussions about trusting LLM output focus on what the model says, but the quieter problem is what the evaluator believes. A judge is a model too, which means it drifts when providers update it, when the rubric prompt is edited, or when golden cases slowly go stale. A minimal harness tells you that an output changed, yet it cannot tell you whether that change matters. That distinction is the difference between a dashboard and an evaluation.

The three-tier design

Tier one runs deterministic checks against structured fields, tier two asks an LLM judge to apply a rubric, and tier three routes ambiguous results to a human reviewer. Most cases never reach tier three because the first two tiers are deliberately conservative, favoring a fail verdict when the evidence is unclear. The harness writes a JSON report with a per-case verdict and a reason, which turns failures into debug artifacts instead of mysterious anecdotes.

# grader.py - three tiers, one report
def tier1_rule_check(case, output):
    missing = [needle for needle in case['expected']
               if needle not in output]
    forbidden = [needle for needle in case['must_not']
                 if needle in output]
    return len(missing) == 0 and len(forbidden) == 0

def tier2_judge(case, output, judge_model):
    rubric = case['rubric']
    prompt = ('Grade this output against the rubric.\n'
              f'RUBRIC:\n{rubric}\nOUTPUT:\n{output}\n'
              'Respond with exactly PASS or FAIL.')
    raw = call_llm(prompt, judge_model)
    return raw.strip().upper().startswith('PASS'), raw

def tier3_human(case, output):
    enqueue_review(case_id=case['id'], output=output)
    return None, 'queued for human review'

The calibration audit

Once a week the harness runs a calibration set containing outputs that are known to pass and outputs that are known to fail. The discrimination score is the fraction of those items that the judge labels correctly, and a score below ninety-five percent marks the entire report as provisional. This audit is the step most eval suites skip, and it is the cheapest protection against a grader that silently changes its opinion. Store the weekly scores in a small JSON or SQLite file so a downward trend is visible before it crosses the threshold.

CALIBRATION = [
    {'case_id': 'sum', 'output': GOOD_SUMMARY, 'label': 'PASS'},
    {'case_id': 'sum', 'output': 'The user is angry.', 'label': 'FAIL'},
    {'case_id': 'sum', 'output': TRUNCATED_SUMMARY, 'label': 'FAIL'},
]

def judge_verdict(item, judge_model):
    passed, _ = tier2_judge(item, item['output'], judge_model)
    return 'PASS' if passed else 'FAIL'

def discrimination_score(judge_model):
    correct = sum(judge_verdict(i, judge_model) == i['label']
                  for i in CALIBRATION)
    return correct / len(CALIBRATION)

The threshold favors false positives over false negatives deliberately, because a failed evaluation costs tokens while a blessed regression costs users. A judge that fails golden cases burns a little budget, yet a judge that passes broken output silently erodes trust in the entire suite. Bias the grader toward failing, then let the human tier settle the borderline cases.

Budget guardrails

Continuous evaluation only makes sense when the cost of catching a regression is smaller than the cost of shipping it. Before every run, the harness estimates tokens from the case count and the number of judge rounds, and it refuses to start when the estimate exceeds a threshold. On a ten million token allowance, a nightly suite of fifty cases with two judge rounds per case consumes roughly sixty thousand tokens, which is well under one percent of the quota. That arithmetic is what makes a nightly eval feasible on a free tier instead of a luxury reserved for enterprise budgets.

MAX_TOKENS_PER_RUN = 50_000

def estimate_tokens(n_cases, judge_rounds=2, tokens_per_round=600):
    return n_cases * judge_rounds * tokens_per_round

def guardrail(n_cases):
    estimate = estimate_tokens(n_cases)
    if estimate > MAX_TOKENS_PER_RUN:
        raise SystemExit(f'estimate {estimate} exceeds budget')
    return estimate

Running unattended on a free server

The free server option converts evaluation from a manual ritual into a scheduled background task. Two cron lines run the nightly suite and the weekly calibration, appending results to log files that the morning review can diff against the previous day. No GPU, no container orchestration, and no paid tier are required for this workload, because the bottleneck is token cost rather than compute.

0 3 * * *   cd /srv/prompt-eval && /usr/bin/python3 runner.py >> eval.log 2>&1
0 5 * * 1   cd /srv/prompt-eval && /usr/bin/python3 runner.py --calibrate-only >> calibration.log 2>&1

When to trust each tier

Tier	Method	Marginal cost	Catches	Trust when
1	Rule checks on expected and must_not fields	~zero	Missing fields, forbidden wording	Output contract is structured
2	LLM judge with a rubric	tokens per verdict	Semantic quality, tone regressions	Calibration score >= 0.95
3	Human review queue	engineer time	Anything the judge rationalizes	A release is about to ship

Tier two earns trust through repeated calibration, not through a single impressive demo. If the discrimination score drops, treat every tier-two verdict as suspect and route the entire run to human review. The table is the decision policy; the harness just executes it.

Limitations

The approach assumes golden cases stay current, because a stale expected field produces confident verdicts about the wrong contract. An LLM judge also inherits the blind spots of the model family behind it, so calibration on synthetic failures does not guarantee performance on unseen failure categories. Teams with regulatory or security requirements should keep the human tier mandatory for every release rather than reserving it for ambiguous verdicts. If your evaluation runs twice a month and a person reviews every sample anyway, this harness adds machinery without adding signal.

The practical next step is to copy the harness, load your own golden cases, and watch the calibration score for one week before trusting any tier-two verdict. Point the call_llm function at MonkeyCode's free model endpoint, place the cron jobs on the free server, and let the budget guardrail keep the whole experiment free. What you gain is not a prettier dashboard; it is a measurable answer to whether your judge is still honest.