Nolan Lawson built the same Pokédex app 12 times, once per JavaScript framework, 5 iterations each, one agent per run, and published the whole thing with costs and Lighthouse scores. The cost table got the attention. The table I keep scrolling back to is the one that lists the AI tooling each framework's docs recommend for coding agents, next to whether the agent actually used it.
Eight of the twelve frameworks had something to install. Across those eight, invocation ranged from 0 of 5 runs to 5 of 5. Two of them landed at zero on the headline mechanism their docs point you at.
What the harness actually does
The setup matters for reading the rest, and it's documented well enough to copy. Each agent gets the same SPEC.md, the same conformance suite, the same tooling, and the same one-line prompt: implement the app in this framework, use the latest stable version, work autonomously until npm test passes. One agent at a time in a fresh Podman container, 4 CPUs and 8 GB, Opus 5.5 at high effort, Claude Code pinned to 2.1.310.
When the agent stops, only its app/ directory gets copied into a clean container to run the tests. Everything else it touched (tests, fixtures, node_modules, whatever it left running) stays behind, so it can't edit its way to a pass. Measurement happens in a third container behind Caddy, Lighthouse mobile only, 6 URLs, 5 runs each, median kept. The benchmark repo is forkable and the raw results are published separately.
Costs land between $2.67 and $4.97 per run as medians of 5. Hono+htmx came in cheapest, Angular most expensive, and the spread between adjacent ranks is inside the noise. More on that at the end.
Installed tooling and invoked tooling are two different numbers
This is the part I'd steal. From the post's own table:
| Framework | Tooling installed | Used by the agents |
|---|---|---|
| Hono+htmx | Hono skills | skill tool in 5/5 runs |
| React Router | React Router skill | skill tool 5/5, skill files 5/5 |
| SvelteKit | Svelte Claude Code plugin | MCP in 5/5 runs |
| Angular | Angular MCP + agent skills | MCP 3/5, skill tool 4/5, skill files 2/5 |
| Astro | Astro MCP server | MCP in 2/5 runs |
| TanStack Start | none official | skill files in 2/5 runs |
| Next.js | next-devtools MCP + skill | skill files 1/5, AGENTS.md 2/5 |
| Nuxt | Nuxt MCP | not used |
| Qwik, SolidStart, Gracile+Lit, Vanilla | none | n/a |
Nuxt's hosted MCP server was available in every run and invoked in none of them. Next.js got a stdio MCP server pinned through npx plus a skill, and the usage column only ever mentions the skill files and the AGENTS.md file. For SvelteKit the agents hit the MCP in all five runs.
The harness installs this stuff the way the framework's own docs tell you to, and you can read the exact install path for each framework in setup-tooling.sh. Three shapes show up there. Skills copied into .claude/skills by the skills CLI (Hono, React Router, Angular). Hosted HTTP MCP servers registered in the MCP config (Nuxt, Astro). And a Claude Code plugin installed from a marketplace that starts its own MCP server through npx (SvelteKit).
The three frameworks where the tooling got used in every run are the two skills installs and the plugin install. The two hosted HTTP servers are the bottom of the column. With 5 runs per framework, one model, one effort level, I'm not going to pretend that's a law about MCP transports. But if you maintain a framework's agent integration, "we shipped a server, the docs mention it" is not the same as "agents call it," and this is what the gap looks like when somebody finally counts. Lawson's own read is that the current crop of MCPs and skills may be doing more harm than good.
Counting invocations is the cheap part of an eval
The only reason that table exists is that the harness runs claude -p with --output-format stream-json and keeps the agent's repo. Every tool call is in the transcript. If you throw the transcript away, "with MCP" against "without MCP" becomes a comparison of two environments you can't check afterwards, and the result is unfalsifiable in the boring way: you can't tell a tool that helped from a tool that was never called.
I ran into the same shape testing rules files. A reviewer's config file can be present, correct, and simply not loaded on the run you measured, which is why I put a canary word in the rules and checked whether it came back instead of reading the file and calling it configured. Installed, loaded, and invoked are three separate states and only the last one can change a result.
So when you build an arm in an agent eval, log tool names per run and report the invocation rate next to the score. If the tool fired in 2 of 5 runs, label the arm as partially applied, because the other 3 runs are a control you didn't ask for and can't separate. Same question applies when you're the buyer rather than the benchmarker: comparing code review agents like Kodus, CodeRabbit or Greptile, the useful thing to ask is which checks ran on which pull requests, and whether the tool's report separates "no issues found" from "the review didn't run."
What the harness records instead of pinning
Frameworks and their AI tooling are deliberately left unpinned, since a real user installs whatever is current that week. The harness compensates by recording provenance instead. npx-launched MCP servers get resolved to an exact version at setup time and pinned to it. Hosted MCP servers can't be pinned, so mcp-probe.sh initializes a session, fetches the tool list, and stores a SHA-256 of it. Skill and plugin repos get their commit SHA. Alongside those, the run records the container image IDs, the Ubuntu base digest, Node, npm, Playwright, Claude Code, the model and effort level, and the app's own installed dependency versions.
That's the right shape for a benchmark that cannot be frozen. The model, the frameworks, the docs and the search results all move, so reproducibility has to mean "here is the dated snapshot and everything that produced it" rather than "re-run this and get the same number." If your eval claims a number is reproducible while silently depending on whatever latest resolved to, you're describing the weather on the day you measured.
The variance is printed next to the medians
Cost per run for React Router: $3.81, range $2.82 to $3.97. Hono+htmx: $2.67, range $2.23 to $3.19. The cheapest React Router run beat the most expensive Hono+htmx run. That's a single-iteration comparison that would have flipped the other way on a different Tuesday, which is why the README says one iteration is an anecdote and why the results repo ships the per-run data instead of a leaderboard.
The README also lists what the benchmark can't see. The app is content-heavy, which favors server-rendered MPAs on first-load metrics. Only page loads are measured, so client-side navigation and interactions where SPAs win back ground aren't. The harness's text helpers read innerText, which skips shadow-root content. And the data has to be fetched at request time, so a framework can't win by prerendering everything.
Those caveats are the reason I trust the numbers that do survive them. Hono+htmx produced the same app in roughly 1,370 lines against Angular's 2,009, and Lawson's point about code size is the one worth carrying into agent work: in a codebase an agent keeps editing, line count is a recurring token cost, not a style preference. Astro shipped 0 KiB of JavaScript and an 810 ms LCP, Next.js 142 KiB and 2,090 ms, and the ranking in between those two is a shrug.
The most useful thing in the whole exercise isn't which framework won. It's that a table of "did the agent call the thing we installed" exists at all, next to the cost numbers, in a post that also tells you how many iterations it took to make each one mean something. The HN thread spent most of its energy arguing that the app type flatters Astro, which is a fair fight to have and a fight the author already conceded in the limitations. I'd rather see the next round of these run the same tooling with and without the MCP servers, and print the invocation counts. Fork it, spend your own tokens.
Top comments (0)