i stopped trusting coding agent demos, so i built a tool that interviews them

i kept trusting agent demos more than i should have, so i built a small runner that makes coding agents prove it on my own tasks.

the idea is boring on purpose: point it at any repo, define a few tasks in plain json, and it runs each agent CLI on a throwaway copy of the project. after the agent finishes, the verify scripts get restored from the source repo before they run. that part matters. an agent with write access can just edit its own checker and pass. i shipped agent-trials without the restore step, audited it, felt dumb, fixed it.

a real run on a 3-task toy suite looked like this:

                 opus    sonnet    haiku
fix-clamp     33.2s     39.5s     32.6s
divide-guard  22.8s     26.8s     90.5s
slug-test     42.7s     67.5s     45.5s
score          3/3       3/3       3/3

scores and medians move around between runs. it's a smoke test, not a leaderboard. the thing i actually use it for is catching "the cli exited 0 but did nothing" and watching cost drift.

zero dependencies, plain json suites, works offline if your agent cli works offline.

github.com/princejain756/agent-trials

what would you want it to measure that it doesn't?