Supabase just open-sourced supabase/evals, a benchmark that runs Claude Code, Codex, and OpenCode against real-world Supabase tasks using containerized environments, MCP servers, and live CLI tools.
The scores are interesting.
The engineering decisions behind the benchmark are even more valuable.
Here are three choices that stood out—and why they matter if you're building AI agents in production.
1. They capped retries
Supabase allows one retry before grading a task.
The reason is simple: more retries reduce false negatives, but they also make evaluations more expensive. At some point, the benchmark stops being practical.
Production agents usually don't have this discipline.
Most frameworks let you cap total iterations, but that's different from limiting retries of the same failed action. An agent can fail several times, eventually succeed, and still burn far more tokens than expected.
Success doesn't always mean efficiency.
If you're running agents at scale, retry budgets deserve to be treated as first-class limits—not just an implementation detail.
2. Tool usage matters as much as accuracy
One benchmark result caught my attention:
- Claude Code averaged roughly 2 documentation reads per scenario.
- Codex averaged roughly 8.
Both may solve the task.
One simply consumes much more context along the way.
Accuracy leaderboards rarely show this, but production bills certainly do.
When comparing models, measuring tool calls, token usage, and cost per completed task is often more useful than looking at completion rate alone.
3. Better descriptions changed agent behavior dramatically
Supabase noticed one internal skill was only activating about 10% of the time.
They rewrote its description.
Activation jumped to around 60%.
Nothing about the underlying capability changed—only how the agent understood when to use it.
That's an important reminder that prompts, skill descriptions, and routing logic are part of your system design. If you only measure task success, you'll miss whether your agents are actually using the components you built.
The bigger lesson
None of these decisions were made because they're academically interesting.
They were made because Supabase runs these evaluations every day, and daily benchmarks need predictable cost and reliability.
Production agents face the same constraints.
The difference is that benchmark designers build those limits in from day one, while production systems often discover them only after expensive failures.
If I had to copy one idea immediately, it would be the retry cap.
Top comments (0)