Originally published on the TestDino blog. I work there. The numbers are from October 8, 2026, with a median of 3 runs per agent.
Ask a coding agent to reply with the single word OK, and it can send anywhere from 1,894 to 31,798 input tokens to the model before doing any real work.
That is a 16.8× spread between the lightest and heaviest setups in our benchmark of 11 coding agents running against the same clean Playwright repository.
The surprising part is that none of this is obvious from the terminal. You see a short answer, but not the system prompt, tool definitions, environment context, and other harness data sent to the model before it reads your test.
This benchmark measures that initial overhead, explains where the tokens go, and shows how to reduce them without assuming that a smaller context automatically means a better agent.
What does a harness overhead benchmark measure?
A coding agent has two main parts: the model that reasons and the harness that builds requests, exposes tools, executes actions, and feeds results back into the conversation.
The harness contributes input tokens through its system prompt, tool schemas, instruction files, memory, and environment information. Those tokens are separate from the task itself.
The first model call is a useful comparison point because a trivial prompt minimizes task-related context. It gives us a snapshot of what each harness sends before any files are read or tools are called.
It does not measure overall agent quality, however. A smaller first request does not necessarily mean fewer retries, fewer failures, or lower total cost on a real task.
How we ran the benchmark
Every agent opened the same minimal Playwright project containing one configuration file, one spec file, a README, and a Git history with one commit.
We used:
The same seven-word prompt: Reply with the single word OK
Headless mode for each agent
A clean profile where applicable
No repository-level agent instruction files, MCP servers, or skills
Three runs per agent, reporting the median first-call input tokens
We counted input tokens as uncached input plus cache writes plus cache reads. Where an agent did not expose the same usage data, we used its available session logs or context estimate and noted that limitation.
For example, Claude Code exposes usage information through JSON output:
claude -p "Reply with the single word OK" --output-format json
The other agents have equivalent headless commands or usage sources.
Models and versions tested
Agent Version Default model used
Claude Code 2.1.293 Claude Opus 5.5
Codex CLI 0.161.0 GPT-6.1 Sol
Cursor CLI 2026.10.01 Auto
Copilot CLI 1.0.93 mai-code-1.1-flash
Kiro CLI 2.28.0 auto
Cline CLI 2.11.0 GPT-6 Astra via Cline
Gemini CLI 0.59.0 Gemini 3.8 Flash
Qwen Code 0.25.0 Gemini 3.7 Flash
OpenCode 1.18.35 Gemini 3.6 Flash
Pi 0.73.1 Gemini 3.7 Flash
Aider 0.86.2 Gemini 2.5 Flash
One important caveat: each agent used its default model. The comparison therefore spans four different tokenizers. Treat small differences between neighboring results as approximate rather than definitive.
We also had to account for differences in usage reporting. Kiro's number is a context estimate from its debug log; Aider reports rounded figures; and Qwen Code's main-request count excludes background memory calls. These differences make the methodology and individual notes important when interpreting the ranking.
Results: tokens sent before the test starts
The table ranks the agents by median first-call input tokens across three runs.
Rank Agent Input tokens Cache reads on reruns Notes
1 Kiro CLI 2.28 31,798 Not reported Own estimate; 31,722 tokens attributed to tool schemas
2 Claude Code 2.1.293 31,480 15,919–17,809 Clean profile; real profile measured 34,161
3 Copilot CLI 1.0.93 17,059 1,280–16,640 System segments total 5,502 tokens
4 Codex CLI 0.161 14,768 12,544 Clean profile; two-plugin profile measured 14,851
5 Qwen Code 0.25 13,677 8,148 Main request only
6 OpenCode 1.18 13,437 11,398 Identical across three runs
7 Cursor CLI 2026.10 13,174 3,840–13,056 Auto model; identical across three runs
8 Gemini CLI 0.59 9,290 4,066 Plus a separate 828-token routing call
9 Cline CLI 2.11 5,382 5,379 Identical across three runs
10 Aider 0.86 About 2,500 Not reported Rounded usage reporting
11 Pi 0.73 1,894 0 Identical across three runs
These are first-call input token counts, not dollar costs or measures of accuracy.
The 30,000-token group
Kiro CLI and Claude Code both exceeded 31,000 input tokens before doing anything.
Kiro's context estimate attributes 31,722 of its 31,798 tokens to tool schemas. That is a particularly clear example of how a large tool catalogue can dominate a request.
Claude Code's clean-profile measurement was 31,480 tokens. With a global CLAUDE.md, automatic memory, and three MCP servers, the first-call figure increased to 34,161.
The middle of the table
Copilot CLI, Codex CLI, Qwen Code, OpenCode, and Cursor landed between roughly 13,000 and 17,100 tokens.
Qwen Code, OpenCode, and Cursor were within about 500 tokens of one another. Given the different tokenizers and measurement details, it would be misleading to treat that small gap as a meaningful performance distinction.
Codex CLI is also a useful example of why raw input and billed input are different: 12,544 of its 14,768 tokens appeared as cache reads on reruns.
The under-6,000 group
Cline, Aider, and Pi had the smallest measured prefixes.
Pi sent 1,894 tokens. Aider reported approximately 2,500, while Cline sent 5,382. These results show that relatively small harnesses are possible, but they do not establish which agent will complete a real testing task most reliably.
Where does the overhead come from?
A request contains both fixed and growing context.
The fixed portion includes system instructions, tool schemas, repository instructions, memory, and environment information. The growing portion includes conversation history, file contents, command output, and test logs.
With a trivial prompt, most of the first request is the fixed portion.
Kiro's estimate attributed almost all of its context to tool definitions. Copilot CLI's usage event listed 5,502 tokens across its system segments, leaving much of its total request unitemized.
The practical lesson is to examine the tool surface, not just the length of your instruction files. A harness that exposes many tools may send a large amount of schema information before the model starts the task.
Your own configuration can add to this baseline. In our Claude Code measurements, a global instruction file, automatic memory, and three MCP servers added 2,681 tokens compared with the clean profile. Two enabled Codex plugins added 83 tokens in our comparison.
These figures are specific to the versions and setup tested. Your results may differ.
Prompt caching reduces cost, not context size
Caching can make repeated requests substantially cheaper.
In our measurements, Codex CLI had 12,544 cache-read tokens out of 14,768 input tokens. OpenCode had 11,398 cache-read tokens out of 13,437, and Cline had 5,379 cache-read tokens out of 5,382.
The important distinction is that cached tokens still occupy the model's context window. They can still contribute to context pressure and affect when a conversation needs compaction.
Caching also does not make two harnesses equally efficient. It changes the cost of processing repeated prefixes, not the amount of context the harness supplies.
For that reason, measure both:
Context overhead: how many tokens the harness sends
Actual cost: how those tokens are priced, including cache reads and writes
A low invoice on a cached request does not necessarily mean the harness has a small footprint.
Seven ways to reduce coding-agent token overhead
Start with a smaller harness
Use the results table as a shortlist for further testing. Pi, Aider, and Cline had the smallest prefixes in this benchmark, but validate them on your own tasks before choosing one.Defer tool schemas where possible
Avoid loading every available tool definition into every request if your harness supports deferred discovery or tool search.Prefer focused CLI tools when appropriate
A command-line tool may expose a smaller interface than an MCP server with many tools. Compare the actual context each option adds rather than assuming one interface is always better.Keep always-on instructions concise
Remove duplicated guidance and instructions that are irrelevant to most tasks. Move specialized workflows into instructions or skills that load only when needed.Filter tool output
Large test logs, trace files, and HTML reports can expand the growing part of the context. Return concise failure summaries when the full artifact is not needed.Delegate verbose exploration carefully
Subagents can keep detailed file contents in a separate context and return a summary. However, delegation and agent teams introduce their own overhead, so measure the total rather than assuming delegation always saves tokens.Ask for one clear approach
Avoid repeatedly asking for multiple alternatives when one solution with explicit acceptance criteria is enough. Clear stopping conditions can reduce unnecessary exploration.
The first four techniques target the fixed prefix. The remaining techniques target the context that grows during a task. Both matter, but their relative impact depends on how long the agent works and how much output it consumes.
What this means for Playwright and QA teams
A single run may not reveal the full cost of harness overhead. An agent that repeatedly triages test failures can carry its prefix across many model calls, while also accumulating logs, screenshots, and trace information.
For QA workflows, compare agents using more than the first-call token count. Track:
Total input and output tokens per task
Cache reads and cache writes
Number of model calls and retries
Test pass rate and correctness of fixes
Cost per successfully completed task
Also examine the browser automation interface. In a separate TestDino experiment dated September 26, 2026, the same model used 56,874 input tokens through the Playwright CLI and 210,153 through Playwright MCP for the same 13 login-flow assertions, with identical verdicts. That is a result from one particular setup, not a universal rule that CLI is always more efficient.
The goal is to find the least expensive setup that remains reliable on your actual test suite.
Conclusion
Our October 8, 2026 benchmark found a 16.8× spread in first-call input tokens across 11 coding agents, ranging from 1,894 for Pi to 31,798 for Kiro CLI.
Tool schemas accounted for most of Kiro's estimated overhead, while caching substantially reduced the number of tokens that needed to be processed at full input price on subsequent calls.
The main caveats matter: agents used their default models, tokenizers differed, usage reporting was not uniform, and this benchmark did not measure output quality or task success.
Treat the results as a snapshot of initial context overhead, not a ranking of the best coding agents. Rerun the measurement when versions change, and compare total cost and success rate on your own repository before choosing a harness.
For Playwright teams, reducing unnecessary context and returning concise failure summaries are practical places to start.
FAQs
What is a harness overhead benchmark?
It measures input tokens contributed by the coding agent's harness rather than the task itself, including system instructions, tool definitions, and environment context.
Which coding agent used the fewest tokens?
Pi used 1,894 first-call input tokens in our October 8, 2026 run, followed by Aider at approximately 2,500 and Cline at 5,382.
Does a larger harness make an agent more accurate?
This benchmark did not measure accuracy. A separate evaluation on your own tasks is needed to determine whether a larger harness improves results enough to justify its cost.
Does prompt caching remove tokens from the context window?
No. Caching can reduce processing cost, but cached tokens still occupy context.
How can I reproduce the benchmark?
Use the same trivial prompt in each agent's headless mode, record first-call input tokens including cache reads and writes, run three times, and report the median. Record the version, model, configuration, and any differences in usage reporting.
Where can I read the original benchmark?
The original TestDino article is available at https://testdino.com/blog/harness-overhead-benchmark.
Top comments (0)