A familiar argument is making the rounds in web development: if an AI agent writes most of the code, developer experience matters less, so teams should choose the framework that works best for agents. It sounds plausible. It also skips the hard question: what does “works best” mean, and how would you know?
Nolan Lawson tried to answer that with a small benchmark. He asked Claude Opus 5.5 to build the same Pokémon catalog app in 12 JavaScript stacks, tested each result against a shared conformance suite, and measured cost, time, code size, and selected Lighthouse metrics. He ran five iterations per stack, except Angular, which had three measurable results for some metrics. The benchmark write-up and its open-source harness make the experiment inspectable.
The headline is not that one framework won. The useful finding is that “agent-friendly” is not a single score. In this test, Hono with htmx was among the least expensive and shortest-code options, while Astro sent no JavaScript on the tested page. Angular cost more and generated more code, but this is one content-oriented application, one model, one benchmark specification, and a small number of runs. The evidence is enough to question vague claims. It is not enough to declare a universal winner.
That distinction matters because framework decisions are long-lived. If the agent saves a few minutes today but creates more code to inspect and maintain tomorrow, the initial speedup may be a poor trade. The better response to AI-assisted development is not to pick a framework based on intuition about what models prefer. It is to extend the measurement window from “how fast did the agent finish?” to “what does this choice cost across the life of the application?”
What the experiment actually measured
A shared task, not a framework tutorial. Lawson specified a small but non-trivial Pokémon app using local data from PokéAPI. Agents had to implement the same user-visible features and pass the same conformance suite. They were allowed to use official framework MCP tools or skills where available, and the harness recorded whether the agent actually used them. The benchmark repository says each run used a fresh Podman container and that the generated app was tested separately from the agent environment. The repository README describes the controls and reproduction setup.
More than token count. Lawson reported API-equivalent cost, wall-clock time, tokens, lines of code, and Lighthouse measurements for performance and accessibility. That is a useful choice: an agent can spend more tokens to produce code that is smaller, more accessible, or faster for users. Optimizing only for the first completion time would miss those trade-offs.
The median reported cost ranged from $2.67 for Hono plus htmx to $4.97 for Angular. These are not the experimenter's cash charges. Lawson says he used a Claude Max plan and reported the API prices quoted by Claude Code. The cost range is therefore a way to compare the reported usage, not a billing receipt or a forecast of what a team will pay.
The time and token figures also varied. Astro's median run took 8.2 minutes, while Angular's took 17.2 minutes. Median token use ranged from 3.85 million for Nuxt to 9.76 million for Angular. But those figures came from five runs per framework, with Angular's reduced sample in some measurements. The author's own write-up notes stochastic results and overlapping ranges. A median is a helpful summary, not proof that the same ordering will hold on your task.
The counterintuitive result is the important one
Raw JavaScript was not automatically the cheapest route for the agent. The benchmark's “Vanilla” condition used JavaScript or TypeScript with Vite and no UI framework. It was not simply “write the smallest possible plain JavaScript app.” The agent made architectural choices inside the allowed setup, and the result used a median of 5.00 million tokens and took 10.1 minutes. Several framework conditions used fewer tokens or completed sooner.
That does not prove frameworks are better for agents. It demonstrates that removing a framework does not automatically remove complexity. A framework can supply conventions, routing, data loading, and a structure an agent can follow. Conversely, it can introduce conventions the model does not know well. Which effect dominates depends on the framework, task, tool setup, and model version.
The web payload and the agent's build cost are different budgets. Astro's tested page transferred a median of 0 KiB of JavaScript, while Next.js transferred 142 KiB. Hono plus htmx used 18 KiB, and Angular 93 KiB. On that metric, the result separates framework bundles more clearly than the agent's cost rankings do. But a zero-JavaScript page is not a complete performance verdict: the application still has server work, network requests, content, and user interactions. The benchmark also reports different results for blocking time and largest contentful paint, with caveats around variance and the Lighthouse setup.
If a team collapses those measurements into one “best” framework score, it will hide the very trade-offs that make the exercise useful. Agent usage cost is an engineering input. Browser payload and responsiveness are user-facing outputs. Maintainability is a future cost. They should not be treated as interchangeable just because they came from the same run.
Code size is a proxy, not maintainability itself. Hono plus htmx generated a median of 1,370 non-blank source lines; Angular generated 2,009. That is a measurable difference, and smaller implementations may mean fewer tokens for later agent edits. Lawson explicitly connects code volume with the cost of future changes. Yet line count cannot tell you whether code is understandable, well-factored, secure, or easy for a human team to change. A short, opaque implementation is not automatically maintainable.
Why “better for agents” is a weak architecture rule
Framework advocates can point to model familiarity, official agent skills, MCP servers, and established examples. Those can matter. But the benchmark found that agents did not consistently use the available tooling. For example, the report says some frameworks' tools were used in only a subset of runs, and Nuxt's MCP was not used in the observed attempts. Installing an integration is not the same as proving it changed the result.
There is also a training-data trap. Lawson suggests that SvelteKit's higher measured cost may partly reflect its recent release: the agent repeatedly read its migration guide, possibly because the latest version was not represented well in the model's training data. That explanation is plausible, but it is not isolated as a causal factor. A follow-up would need to vary model version, tool access, and framework documentation exposure independently to determine whether unfamiliarity caused the extra work.
The result is a moving target. Framework versions change, models change, tools change, and the same agent can make different choices across runs. A one-time benchmark can help choose what to investigate. It cannot permanently rank a technology stack. That is especially true when official AI tooling is unevenly available or the agent ignores it.
How to use this kind of benchmark without overfitting
Start with your application shape. A content catalog is not a collaborative editor, admin dashboard, offline-first mobile web app, or real-time interface. Lawson says the sample is closer to a content site than a highly interactive product. If your application is dominated by complex client-side state, this result is a poor proxy. Pick a representative slice of your own product before treating any framework ranking as relevant.
Define acceptance criteria before the run. Specify user-visible behavior, accessibility expectations, performance thresholds, supported browsers, and operational constraints. Keep the same tests across candidates. If one framework gets a different prompt or extra hints, the result measures prompt quality as much as framework choice.
Measure failed attempts, not just successful output. In the benchmark, Angular failed to create a measurable app in two of five runs. Lawson retained the three usable iterations for some metrics and explained the difference. That transparency is important. In your own evaluation, record failures, retries, timeouts, test results, and cleanup effort. Do not quietly discard the runs that make a preferred tool look worse.
Separate repeatability from generalization. Run enough iterations to see whether differences exceed run-to-run noise. Then repeat with a second task and, if relevant, a second model. The original author notes variance in both agent output and Lighthouse. Re-running one task reduces some uncertainty; it does not make that task representative of every product.
Track the whole cost surface. At minimum, keep separate measures for:
- Agent spend or token use per accepted change.
- Human review and correction time.
- Test pass rate and severity of defects.
- Browser performance and accessibility.
- Code size and complexity signals.
- The effort required to add a second feature after the first implementation.
That last measure is easy to skip and potentially more revealing than initial generation time. A framework that lets an agent produce a first screen quickly may not help when the next request changes data flow, permissions, or navigation. Add a follow-up task that tests whether the generated structure supports change without a rewrite.
A decision checklist for framework evaluation
Before you switch stacks because a model appears to prefer one, ask:
- Is the benchmark task representative of our actual product?
- Did each candidate receive the same specification, model, resource limits, and acceptance tests?
- Were official agent tools installed, and did the agent use them?
- Are the reported differences larger than the run-to-run spread?
- Did we count failed runs and human repair work?
- Are user outcomes measured independently from agent speed and cost?
- Does the generated code remain workable after a second, realistic change?
- Would we make the same choice if the current model or tool integration disappeared next month?
The last question is a useful stress test. A framework should not be selected solely because one model has recently seen more examples of it. Model familiarity can improve today's result, but application needs, ecosystem support, and maintainability still matter after the next model update.
The takeaway: benchmark claims, not vibes
This experiment is valuable because it exposes its scope, shares its harness, reports multiple measures, and discusses limitations. Its results are not a framework league table. They are a reminder that the cost of a stack now includes both the work an agent spends producing code and the work people spend checking, operating, and changing that code.
For a team choosing a framework, the practical move is modest: use public benchmarks to form hypotheses, then test those hypotheses on a small, representative feature with your own requirements. Keep the agent's cost, user-facing quality, reliability, and maintenance effort separate. If the decision changes when you add a second task or another model, you have learned something important about how fragile the original ranking was.
I write about developer tools, infrastructure, and AI. Subscribe if you want evidence-based engineering analysis without treating a benchmark as a universal answer.
Have you measured how your framework changes an AI agent's cost or the quality of the code it produces? What would you include in a fair test?
Top comments (0)