Originally published on AI Tech Connect.
What you need to know A leaderboard and a procurement decision are different questions. Public benchmarks rank agents on somebody else's distribution. You are buying against yours. Your git history is already a benchmark. Every merged bug-fix pull request that changed source and touched tests is a candidate task with a built-in grader. Validation is the whole game. If the new tests do not fail at the parent commit and pass at the merge commit, it is not a task. Discard it. Grade by execution, never by resemblance. Fail-to-pass tests prove the fix; pass-to-pass tests prove nothing else broke. Both must hold. An undisclosed harness makes a number meaningless. Record model, scaffold, tool set, turn cap, timeout and container digest as a run manifest. One run per task is noise. Sample each…
Top comments (0)