What We Learned Benchmarking AI App Builders on 103 Real Creative Tasks

Jack — Sat, 17 Oct 2026 14:47:33 +0000

Most AI coding benchmarks ask whether a model can solve a bounded programming problem. That is useful, but it misses much of what happens when a real person asks an agent to create an app.

A real request may arrive with screenshots, videos, PDFs, a half-working repository, conflicting follow-ups, or a design the user already approved. The agent has to decide whether to discuss, plan, prototype, or build. It then has to preserve earlier decisions, produce a runnable result, test it, and recover when the user changes direction.

Eazo recently published a benchmark built around that full creative chain rather than a collection of isolated code-generation prompts.

The benchmark shape

The benchmark contains 103 real-user creative tasks across seven capability sections:

Inputs and references
Intent routing and design depth
Asset sourcing, presentation, and integration
Development and runtime
Disruption and recovery
Delivery and publishing
Cross-cutting capabilities

The tasks include multi-turn work involving images, video, CAD/STEP files, documents, existing projects, design feedback, rollback requests, browser QA, exports, and publishing.

The user's words are sent verbatim. When the agent asks a question, the simulated user answers only with facts available in the task; it does not reveal the scoring criteria.

That detail matters. A benchmark becomes much easier if the evaluator quietly helps the agent toward the expected answer.

Separate quality from completion

Every task is judged in two ways.

First, a weighted quality score across eight dimensions:

Correct stage routing
Clear deliverable per step
Continuity of prior state
Asset, design, and code reuse
Core feature completeness
Interaction and failure states
Responsiveness, performance, and accessibility
Self-testing and delivery quality

Second, item-by-item acceptance. Each task has explicit deliverable and process requirements. An item only passes when the frozen evidence supports it.

This distinction prevents a common failure mode: a polished final message claiming success when the app, files, or runtime do not actually support the claim.

The browser—not the agent's narration—is the evidence

At the end of a run, the conversation, files, screenshots, and runtime state are frozen. Reviewers judge that evidence without asking the agent follow-up questions or modifying the result.

For runnable apps, a separate tester opens the product in a real browser, exercises the interactions, and captures desktop and mobile evidence. Reviewers then decide whether each requirement passed.

A useful rule emerges:

“The agent said it works” is not evidence that it works.

This sounds obvious, but it changes benchmark design. A build command exiting successfully does not prove that the main interaction works. A screenshot does not prove that state changes correctly. A deployed URL does not prove that the requested version was deployed.

Five lessons from the 103 tasks

1. Routing errors happen before coding errors

An agent can write competent code and still fail the user's request by going too far.

If the user asked to compare options, building an app is not a bonus. If they said “read the existing project and wait,” editing files is a failure. If they asked for one lightweight prototype, producing a full product can make iteration harder.

The benchmark treats the correct stage—discussion, plan, artifact, prototype, or runnable app—as a dedicated quality dimension.

2. Continuity is a first-class capability

Many real failures are not syntax errors. They are quiet reversals of earlier decisions:

A selected design disappears during implementation.
A supplied asset is replaced with a generic substitute.
A local-only requirement turns into a public deployment.
A rollback restores the UI but not the underlying behavior.

Single-turn benchmarks rarely expose this. Multi-turn tasks do.

3. Multimodal input is only valuable when it changes the result

“Supports images” can mean anything from accepting an upload to actually using local details from the image in the product.

The same applies to PDFs, CAD assemblies, videos, and existing repositories. A meaningful evaluation checks whether the agent parsed the material, understood the relevant parts, and carried those facts into the deliverable—not whether it merely acknowledged the attachment.

4. Recovery deserves dedicated tests

Users change their minds. Tools fail. A generated asset may be unusable. A design direction may be rejected after implementation begins.

Rollback, mid-task pivots, asset locking, and failure recovery need explicit tasks. Otherwise a benchmark over-rewards agents that work only when every earlier step goes perfectly.

5. Aggregate scores hide important failures

Across the benchmark, Eazo's average composite quality score was 88.5, compared with 84.2 for Codex. Average task time was 11.1 minutes versus 11.7 minutes, and total token usage was 158M versus 179M.

But the deliverable pass rate was 60% for both systems. The process pass rate was 52% for Eazo and 47% for Codex.

Those numbers are a useful reminder: a higher average quality score does not mean every required item passed. The detailed task evidence is more informative than a single leaderboard number.

The methodology, task-level cases, scoring dimensions, and current aggregate results are available on the full Eazo benchmark.

A reusable checklist for agent benchmarks

When evaluating an app-building agent, check whether the benchmark includes:

Raw user language rather than evaluator-cleaned prompts
Multi-turn changes and corrections
Existing files and projects
Images, documents, video, or structured data
A distinction between discussion, planning, prototyping, and building
Frozen evidence after execution
Independent browser testing
Mobile and desktop checks
Loading, empty, and error states
Rollback and recovery tasks
Item-level evidence citations
Separate deliverable and process requirements
A way to record review corrections rather than overwrite them

The goal is not to make the benchmark complicated for its own sake. It is to make it difficult for a system to receive credit for work a user could not actually use.

Keep the limitations visible

Any benchmark reflects its task selection, scoring weights, product versions, and review process. A comparison between two systems is not a universal ranking of every coding agent, and aggregate scores can obscure individual failures.

The most useful benchmark is therefore not just a leaderboard. It is an inspectable collection of tasks, artifacts, runtime evidence, and review decisions that other people can question.

DEV Community: Jack