I kept seeing agent and eval demos where the honesty — held-out discipline, "no metric gaming" — lives in a prompt, or in a paper's methodology section. So I tried to build the opposite: a search loop where the anti-overfitting rules are enforced in executable code, then ran it against a real external grader (MLE-bench) to see whether that discipline actually costs you anything.
The result is heldout — MIT, zero runtime deps, works as a Claude Code skill: https://github.com/jeet-dhandha/heldout
What the engine enforces — in code, not prompts
- every node is bound to a real measured score — a shell command that must print
SCORE=<n>; it refuses to bind a metric it can't execute - a dev-slice win cannot be kept unless an untouched held-out slice confirms it by a pre-set margin — dev-up / held-out-flat is a hard kill
- the baseline only ratchets up on a fresh re-measure — no keeping by vibes
- every killed branch must leave a lesson, injected into future code-gen prompts, so the loop never re-proposes a dead direction
- no LLM ever grades its own work — scores come from a grader the generated code can't read or influence
- the dollar budget is a wall inside the tool, not a guideline
Harness, not a Kaggle agent
AIDE / AutoKaggle / AIRA bundle the search policy (what to try next) and the ML operator (write + run the code) into one system, and enforce eval honesty by convention. heldout splits them: the engine owns only the search policy and its invariants, domain-agnostically; the operator (LLM draft → sandboxed run → parse score) is a replaceable module. MLE-bench is the proof surface, not the product.
The honest results — including where it did NOT win
All numbers are from the official MLE-bench grader on nomad2018-predict-transparent-conductors (private test, n=240). Never self-reported.
- Phase 1: held-out RMSLE 0.06117 → silver medal, for $3.13 (9 LLM calls, each in a network-less container).
- Phase 2 + 2b: I pre-registered an equal-dollar A/B against a dumb linear-retry loop (same model, same budget) to isolate whether the search policy itself earns its cost. Two nulls — the held-out gap stayed under my pre-set 0.005 decision bar both times (0.00479, then 0.00216). So "search beats retries" is NOT established — and I report that, because the whole point is a harness that can tell you when it didn't work.
The sharpest moment
In Phase 2b, a Random Forest node scored 0.06227 on held-out — silver-tier, and better than what the retry loop shipped. The gate killed it: its edge over the baseline was 0.00288, below the noise-calibrated 0.0036 margin, and its dev→held-out gap (0.0055) flagged overfit. A naive "ship the best held-out score among all nodes" rule would have declared a win — by 0.00004 over the bar. The honest last-kept rule refused it → null.
That refusal is the product.
Honest limitations
n=1 run per arm, one task — a controlled A/B, not a statistical claim; the Phase 2b direction flip is inside single-draw noise (the retry arm drew weak, 6 of 10 attempts empty-drafted). This is a harness plus a methodology, not a SOTA result.
Repo, pre-registrations, full trees, and grader ledgers: https://github.com/jeet-dhandha/heldout
I'd love to be told the comparison is still too generous to the search arm, or where the enforcement has a hole.
Top comments (0)