DEV Community

pyfile-toolkit
pyfile-toolkit

Posted on

A benchmark you cannot train on, now a pay-per-call API

#ai

Every few months a model "scores 92% on MMLU" and then fails in production on the exact things the benchmark said it could do. The score was real. The evaluation wasn't — the questions had been on GitHub and arXiv for years, and the model had memorized them.

The usual answer is to rotate questions monthly. My answer was to remove the attack surface entirely: generate the tasks from a random seed at run time and never publish them. If the evaluation set does not exist until the run starts, there is nothing to memorize.

I built this as a framework (225 task domains, three checker kinds, IRT scoring, a contamination gap). The interesting part for agents, though, is that it is now callable over HTTP 402.

Why a pay-per-call API

An agent that already pays for APIs (x402, USDC on Base, or Nano) can now get a contamination-resistant eval the same way it buys anything else: discover the endpoint, pay a fraction of a cent, get an answer. No signup, no API key, no dashboard.

Three endpoints:

  • GET /data/bench/worlds — list the 225 domains
  • GET /data/bench/fresh — generate fresh, unseen tasks from a seed: prompts + refs + a sha256 commit, no answers
  • POST /data/bench/grade — grade answers against regenerated tasks, returns per-item result and accuracy

Price is ~$0.01 per call.

How it stays honest

The tasks are produced by a seeded generator. Given the same seed you get the same tasks, so a grader can regenerate them later and check answers without ever shipping the answer key. The fresh call returns a commit — sha256 over the task set — so a third party can verify the set did not change after the fact.

Because nothing is published, a model cannot have seen the questions. And because tasks from a previous run can be re-mixed into a new one, you can measure contamination directly: a model that memorized will score higher on questions it has seen (positive gap); an honest solver scores the same either way.

In reference runs: a deterministic solver scores ~100%, honest models sit in the low 90s, and a deliberately memoizing candidate shows a gap of about +100 percentage points and gets flagged.

Try it 

# list domains
GET /data/bench/worlds

# fresh tasks
GET /data/bench/fresh?seed=abc&worlds=calendars,logic&levels=1,2&per_level=2

# grade
POST /data/bench/grade  {"ref":{...},"answer":"..."}
Enter fullscreen mode Exit fullscreen mode

Discovery lives at /.well-known/x402. The site and code:

If you run an agent that evaluates models, I would genuinely like to know whether paying a cent for a fresh eval is something you would use — and if not, what would make it usable. That is the open question for me right now.

Top comments (0)