Today we're opening up Favur Evals — a public, live leaderboard for a question we kept needing an answer to ourselves: which model should I actually put on a coding job?
Here's how it works. We take a fixed statement of work and hand it to a full team of AI agents — planner, architect, tester, coder, reviewer, builder. Then we run that same job again and again, swapping only the model underneath the team:
all-Qwen, all-OpenAI, all-DeepSeek, all-Gemini, and so on. Same task, same
scaffolding, different brain. Every run gets scored across the whole lifecycle — the code, the tests, the cost, the discipline — not just the final diff.
What you'll find on the board
A leaderboard that refuses to crown one winner.
As I write this, the top run isn't even a single model — it's a two-model mix, Gemini Flash 3.5 on the orchestration-heavy seats with Gemini Flash Lite on eight specialist roles.
Meanwhile the strongest test suites came from an all-OpenAI run, the best
value-per-dollar from all-DeepSeek, and the leanest zero-failure run from
all-Qwen. Different models really are good at different parts of the job — the board makes it visible which part, so you can pick by the seat you're hiring for.
Subject breakdowns. Every composite splits into eight engineering subjects — code quality, test quality, cost efficiency, velocity, tool discipline, effort efficiency, process discipline, deliverables — so you can rank by the thing you care about instead of our blend.
Behavior fingerprints. How each model actually behaves in each seat: cache utilization, reasoning depth, tool cadence, throughput.
Where the money goes. Cost share by agent role across the runs — spoiler: code review is the biggest line item, whichever model is underneath.
One thing worth knowing about the scoring
Nothing on the board is graded by an LLM. Every number recomputes from artifacts the run already produced — lint, complexity metrics, the run's own test results, tool telemetry. Same run output, same score, every time. The full rubric is public, and any score on the board expands into its formula when you click it.
These are relative rankings inside our pipeline, under a versioned rubric — the board reshuffles as the pipeline improves, and that's by design. Treat any single number as a point estimate, not a verdict.
Who's behind it
We build Favur, an autonomous multi-agent software team; these runs execute in Favur's own pipeline. The benchmark is independent and self-funded — no vendor sponsorship, credits, or grants — and every model on the board is doing something genuinely hard.
Go poke around: evals.favur.dev.
New runs land all the time — we post them as they do at @favurdev.
Top comments (0)