Supabase's Evals Benchmark Just Gave Claude Code a Real-World Report Card

#ai #opensource #programming #machinelearning

Supabase Evals is an open-source benchmark that scores Claude Code, Codex, and OpenCode on real Supabase tasks. Run supabase eval on your repo to find agent weaknesses.

Key Takeaways

Supabase Evals is an open-source benchmark that scores Claude Code, Codex, and OpenCode on real Supabase tasks.
Run supabase eval on your repo to find agent weaknesses.

What Changed — Supabase Evals Is Now Open Source

Supabase just dropped Evals, an open-source benchmark that puts coding agents like Claude Code, OpenAI's Codex, and OpenCode through their paces on real Supabase tasks. This isn't another synthetic benchmark with toy problems — it's a suite of tasks pulled from Supabase's actual development workflow, the kind of work you'd do every day: writing SQL queries, fixing TypeScript type errors, refactoring functions, and debugging edge cases.

The benchmark is designed to be extensible, meaning you can add your own tasks to test agents on your specific codebase. The results so far? Claude Code scored highest in many categories, but the real value isn't the leaderboard — it's the ability to run this against your own projects.

What It Means For You — Benchmark Your Own Agent Workflows

If you're using Claude Code daily, this benchmark gives you a repeatable, objective way to measure how well it handles your codebase. Instead of relying on vibes or anecdotal evidence, you can run Evals and see exactly where Claude Code excels and where it struggles.

Here's the kicker: the benchmark is open source, so you can fork it, add tasks specific to your project's stack, and integrate it into your CI pipeline. Imagine every PR triggering a benchmark run to ensure your agent's changes don't regress performance.

Try It Now — Set Up and Run Evals

Getting started is straightforward. Clone the repository and run the benchmark against your local environment:

git clone https://github.com/supabase/evals
cd evals
# Install dependencies
npm install
# Run the benchmark with Claude Code
supabase eval --agent claude-code

You can also specify a custom task set:

supabase eval --agent claude-code --tasks ./my-tasks/

To add your own tasks, create a markdown file in the tasks/ directory with a clear description and an expected outcome. For example:

# Task: Fix Broken Migration

**Description:** The migration file `20260101_add_users.sql` has a syntax error. Fix it so `supabase db push` succeeds.

**Expected Outcome:** The migration runs without errors.

The benchmark will then execute Claude Code on this task and score the result.

Why This Matters for Your Claude Code Workflow

Benchmarking isn't just for fun — it's a diagnostic tool. Run Evals on your codebase and you'll discover:

Prompt weaknesses: If Claude Code fails on tasks that involve complex SQL joins, you know to add more context to your CLAUDE.md or improve your prompt engineering.
Tool misuse: Maybe Claude Code overuses supabase CLI commands when a direct SQL query would be faster. The benchmark reveals these patterns.
Regression detection: After updating Claude Code or your model, rerun the benchmark to ensure performance didn't drop.

The Bigger Picture — Benchmarks Are Becoming a Standard

Supabase Evals joins a growing ecosystem of benchmarks like SWE-bench and Terminal-Bench. Claude Code already scores impressively on those (88.6% on SWE-bench Verified with Opus 4.8), but what matters is your codebase. A generic benchmark can't tell you if Claude Code handles your specific Postgres schema or your auth logic. Evals lets you build a custom test suite that does.

Bottom Line

Supabase Evals is more than a leaderboard — it's a tool for continuous improvement. Clone it, run it, and start measuring your agent's performance. The results will surprise you, and they'll make your Claude Code workflows measurably better.

Source: news.google.com

Originally published on gentic.news