Most AI coding benchmarks ask whether a model can solve a bounded programming problem. That is useful, but it misses much of what happens when a real person asks an agent to create an app.
A real request may arrive with screenshots, videos, PDFs, a half-working repository, conflicting follow-ups, or a design the user already approved. The agent has to decide whether to discuss, plan, prototype, or build. It then has to preserve earlier decisions, produce a runnable result, test it, and recover when the user changes direction.
Eazo recently published a benchmark built around that full creative chain rather than a collection of isolated code-generation prompts.
The benchmark shape
The benchmark contains 103 real-user creative tasks across seven capability sections:
- Inputs and references
- Intent routing and design depth
- Asset sourcing, presentation, and integration
- Development and runtime
- Disruption and recovery
- Delivery and publishing
- Cross-cutting capabilities
The tasks include multi-turn work involving images, video, CAD/STEP files, documents, existing projects, design feedback, rollback requests, browser QA, exports, and publishing.
The user's words are sent verbatim. When the agent asks a question, the simulated user answers only with facts available in the task; it does not reveal the scoring criteria.
That detail matters. A benchmark becomes much easier if the evaluator quietly helps the agent toward the expected answer.
Separate quality from completion
Every task is judged in two ways.
First, a weighted quality score across eight dimensions:
- Correct stage routing
- Clear deliverable per step
- Continuity of prior state
- Asset, design, and code reuse
- Core feature completeness
- Interaction and failure states
- Responsiveness, performance, and accessibility
- Self-testing and delivery quality
Second, item-by-item acceptance. Each task has explicit deliverable and process requirements. An item only passes when the frozen evidence supports it.
This distinction prevents a common failure mode: a polished final message claiming success when the app, files, or runtime do not actually support the claim.
The browser—not the agent's narration—is the evidence
At the end of a run, the conversation, files, screenshots, and runtime state are frozen. Reviewers judge that evidence without asking the agent follow-up questions or modifying the result.
For runnable apps, a separate tester opens the product in a real browser, exercises the interactions, and captures desktop and mobile evidence. Reviewers then decide whether each requirement passed.
A useful rule emerges:
“The agent said it works” is not evidence that it works.
This sounds obvious, but it changes benchmark design. A build command exiting successfully does not prove that the main interaction works. A screenshot does not prove that state changes correctly. A deployed URL does not prove that the requested version was deployed.
Five lessons from the 103 tasks
1. Routing errors happen before coding errors
An agent can write competent code and still fail the user's request by going too far.
If the user asked to compare options, building an app is not a bonus. If they said “read the existing project and wait,” editing files is a failure. If they asked for one lightweight prototype, producing a full product can make iteration harder.
The benchmark treats the correct stage—discussion, plan, artifact, prototype, or runnable app—as a dedicated quality dimension.
2. Continuity is a first-class capability
Many real failures are not syntax errors. They are quiet reversals of earlier decisions:
- A selected design disappears during implementation.
- A supplied asset is replaced with a generic substitute.
- A local-only requirement turns into a public deployment.
- A rollback restores the UI but not the underlying behavior.
Single-turn benchmarks rarely expose this. Multi-turn tasks do.
3. Multimodal input is only valuable when it changes the result
“Supports images” can mean anything from accepting an upload to actually using local details from the image in the product.
The same applies to PDFs, CAD assemblies, videos, and existing repositories. A meaningful evaluation checks whether the agent parsed the material, understood the relevant parts, and carried those facts into the deliverable—not whether it merely acknowledged the attachment.
4. Recovery deserves dedicated tests
Users change their minds. Tools fail. A generated asset may be unusable. A design direction may be rejected after implementation begins.
Rollback, mid-task pivots, asset locking, and failure recovery need explicit tasks. Otherwise a benchmark over-rewards agents that work only when every earlier step goes perfectly.
5. Aggregate scores hide important failures
Across the benchmark, Eazo's average composite quality score was 88.5, compared with 84.2 for Codex. Average task time was 11.1 minutes versus 11.7 minutes, and total token usage was 158M versus 179M.
But the deliverable pass rate was 60% for both systems. The process pass rate was 52% for Eazo and 47% for Codex.
Those numbers are a useful reminder: a higher average quality score does not mean every required item passed. The detailed task evidence is more informative than a single leaderboard number.
The methodology, task-level cases, scoring dimensions, and current aggregate results are available on the full Eazo benchmark.
A reusable checklist for agent benchmarks
When evaluating an app-building agent, check whether the benchmark includes:
- Raw user language rather than evaluator-cleaned prompts
- Multi-turn changes and corrections
- Existing files and projects
- Images, documents, video, or structured data
- A distinction between discussion, planning, prototyping, and building
- Frozen evidence after execution
- Independent browser testing
- Mobile and desktop checks
- Loading, empty, and error states
- Rollback and recovery tasks
- Item-level evidence citations
- Separate deliverable and process requirements
- A way to record review corrections rather than overwrite them
The goal is not to make the benchmark complicated for its own sake. It is to make it difficult for a system to receive credit for work a user could not actually use.
Keep the limitations visible
Any benchmark reflects its task selection, scoring weights, product versions, and review process. A comparison between two systems is not a universal ranking of every coding agent, and aggregate scores can obscure individual failures.
The most useful benchmark is therefore not just a leaderboard. It is an inspectable collection of tasks, artifacts, runtime evidence, and review decisions that other people can question.
Top comments (0)