Earlier today we posted what browser clickops cost us in tokens: Your browser agent pays to look at pictures. That post compared calls on the same websites, not the same task. Readers asked for a same-task test, so we ran one.
Setup
- Five read-only tasks: read a GitHub pull request page, count the Workers in the Cloudflare dashboard, read this blog's previous article, read our own homepage header, and read a Wikipedia article.
- Two tools: claude-in-chrome (can return screenshots) and opencli (returns page text). Both drive the same logged-in Chrome.
- Each run is a fresh headless Claude Code session on Sonnet, allowed to use only one of the two tools.
- Three runs per task per tool, 30 runs in total. The order alternated between tools so prompt-cache warmth would not favor one side.
- Cost and tokens are the billed usage the API reported for each run.
Results
| claude-in-chrome | opencli | Change | |
|---|---|---|---|
| Billed cost, 15 runs | $3.13 | $1.19 | −62% |
| Tokens, 15 runs | 5.02M | 2.91M | −42% |
| Median turns per run | 9 | 4 | −56% |
| Median time per run | 83 s | 38 s | −54% |
Median tokens per run, by task:
| Task | claude-in-chrome | opencli | Change |
|---|---|---|---|
| GitHub pull request | 213k | 193k | −9% |
| Cloudflare Workers list | 517k | 267k | −48% |
| dev.to article | 335k | 126k | −62% |
| Our homepage header | 289k | 128k | −56% |
| Wikipedia article | 285k | 127k | −55% |
Accuracy
Both tools got the GitHub and Wikipedia answers right in all three runs. The Cloudflare Worker count came back between 172 and 179 from both tools, because the list mixes Workers and Pages projects. On the dev.to task, claude-in-chrome missed the reading-time label once and opencli missed it twice. One of opencli's misses ended early and cost $0.015, which flatters its number on that task. Without the dev.to task, opencli still cost 61% less.
What we take from it
- The gap is small on text-heavy pages like a GitHub pull request, where text and screenshots cost about the same.
- The gap is large wherever the model reaches for screenshots to find its way around.
- Every model call re-reads about 30k tokens of base context (system prompt, tools and task) in both arms, so the number of turns drives most of the cost. opencli's runs took a median of 4 turns against 9, and that is where most of the saving came from. (Corrected: an earlier version said each run carried a flat 95–120k of startup.)
Limits
Three runs per task is a small sample. We did not test a page where the data lives in a chart. A text read should do fine when a chart's numbers are in the page's HTML and badly when they are only drawn on a canvas. That is the next task we will add.
Top comments (3)
Thanks for running the same-task version. Three runs per arm with alternating order makes the numbers easy to trust.
The startup line changes how I read the Cloudflare row: take roughly 100k of session start out of both arms and it is about 417k vs 167k of actual browsing, so the text read is closer to 60% cheaper there too, not 48%.
For the chart test, it may be worth checking first whether the chart exposes its values through aria-labels or a "view as data table" option. Some libraries add those for screen readers even when the drawing itself is a canvas, and a text read can pick them up.
One question: in the opencli Cloudflare runs, how many of the 4 turns went to getting to the page versus reading the count? That split decides how much a repeat run of the same task could skip.
Good questions. Going back to the transcripts turned up two corrections on my side.
First, the startup cost isn't a flat amount per run. The first call in every run starts at about 30k tokens of context (system prompt, tools, task), and every later turn reads that same base again. So the overhead grows with the number of turns, and subtracting one fixed 100k from each arm doesn't separate browsing from setup. My "95–120k per run" line in the post was loose. The more useful reading is that turn count drives most of the cost.
Second, the opencli Cloudflare runs weren't 4 turns. They took 6, 8 and 17 model calls. The 4 is the median across all tasks.
Here is the split in the two good runs:
The 17-call run got stuck trying to sort the list by name and paging through it.
A repeat run that already knows the account and the tool could open the Workers page and read the count in about 2 calls, against 6 to 8 now. At about 33k tokens per call, that would cut roughly 130–200k tokens per run.
The aria-label and data-table check is a good idea. We'll look for those first when we build the chart task.
That correction makes sense. If every turn re-reads the ~30k base, turn count is the real cost driver, and subtracting one fixed amount per run was the wrong model on my side.
One thing that may move the cost column: Claude Code uses prompt caching, so the repeated base on later turns is usually billed as cache reads at a fraction of the normal input price. If the transcripts keep the usage fields, splitting cache-read vs uncached input tokens per arm would show whether the 62% holds in dollars or mostly in raw tokens.
The 2-call repeat run is the interesting part. Saving the route from the first run (the account id and the direct Workers & Pages URL) and opening it straight away next time would skip the navigation calls entirely and leave the model only the read.