Earlier today we posted what browser clickops cost us in tokens: Your browser agent pays to look at pictures. That post compared calls on the same websites, not the same task. Readers asked for a same-task test, so we ran one.
Setup
- Five read-only tasks: read a GitHub pull request page, count the Workers in the Cloudflare dashboard, read this blog's previous article, read our own homepage header, and read a Wikipedia article.
- Two tools: claude-in-chrome (can return screenshots) and opencli (returns page text). Both drive the same logged-in Chrome.
- Each run is a fresh headless Claude Code session on Sonnet, allowed to use only one of the two tools.
- Three runs per task per tool, 30 runs in total. The order alternated between tools so prompt-cache warmth would not favor one side.
- Cost and tokens are the billed usage the API reported for each run.
Results
| claude-in-chrome | opencli | Change | |
|---|---|---|---|
| Billed cost, 15 runs | $3.13 | $1.19 | −62% |
| Tokens, 15 runs | 5.02M | 2.91M | −42% |
| Median turns per run | 9 | 4 | −56% |
| Median time per run | 83 s | 38 s | −54% |
Median tokens per run, by task:
| Task | claude-in-chrome | opencli | Change |
|---|---|---|---|
| GitHub pull request | 213k | 193k | −9% |
| Cloudflare Workers list | 517k | 267k | −48% |
| dev.to article | 335k | 126k | −62% |
| Our homepage header | 289k | 128k | −56% |
| Wikipedia article | 285k | 127k | −55% |
Accuracy
Both tools got the GitHub and Wikipedia answers right in all three runs. The Cloudflare Worker count came back between 172 and 179 from both tools, because the list mixes Workers and Pages projects. On the dev.to task, claude-in-chrome missed the reading-time label once and opencli missed it twice. One of opencli's misses ended early and cost $0.015, which flatters its number on that task. Without the dev.to task, opencli still cost 61% less.
What we take from it
- The gap is small on text-heavy pages like a GitHub pull request, where text and screenshots cost about the same.
- The gap is large wherever the model reaches for screenshots to find its way around.
- Every model call re-reads about 30k tokens of base context (system prompt, tools and task) in both arms, so the number of turns drives most of the cost. opencli's runs took a median of 4 turns against 9, and that is where most of the saving came from. (Corrected: an earlier version said each run carried a flat 95–120k of startup.)
Limits
Three runs per task is a small sample. We did not test a page where the data lives in a chart. A text read should do fine when a chart's numbers are in the page's HTML and badly when they are only drawn on a canvas. That is the next task we will add.
Top comments (11)
Thanks for running the same-task version. Three runs per arm with alternating order makes the numbers easy to trust.
The startup line changes how I read the Cloudflare row: take roughly 100k of session start out of both arms and it is about 417k vs 167k of actual browsing, so the text read is closer to 60% cheaper there too, not 48%.
For the chart test, it may be worth checking first whether the chart exposes its values through aria-labels or a "view as data table" option. Some libraries add those for screen readers even when the drawing itself is a canvas, and a text read can pick them up.
One question: in the opencli Cloudflare runs, how many of the 4 turns went to getting to the page versus reading the count? That split decides how much a repeat run of the same task could skip.
Good questions. Going back to the transcripts turned up two corrections on my side.
First, the startup cost isn't a flat amount per run. The first call in every run starts at about 30k tokens of context (system prompt, tools, task), and every later turn reads that same base again. So the overhead grows with the number of turns, and subtracting one fixed 100k from each arm doesn't separate browsing from setup. My "95–120k per run" line in the post was loose. The more useful reading is that turn count drives most of the cost.
Second, the opencli Cloudflare runs weren't 4 turns. They took 6, 8 and 17 model calls. The 4 is the median across all tasks.
Here is the split in the two good runs:
The 17-call run got stuck trying to sort the list by name and paging through it.
A repeat run that already knows the account and the tool could open the Workers page and read the count in about 2 calls, against 6 to 8 now. At about 33k tokens per call, that would cut roughly 130–200k tokens per run.
The aria-label and data-table check is a good idea. We'll look for those first when we build the chart task.
That correction makes sense. If every turn re-reads the ~30k base, turn count is the real cost driver, and subtracting one fixed amount per run was the wrong model on my side.
One thing that may move the cost column: Claude Code uses prompt caching, so the repeated base on later turns is usually billed as cache reads at a fraction of the normal input price. If the transcripts keep the usage fields, splitting cache-read vs uncached input tokens per arm would show whether the 62% holds in dollars or mostly in raw tokens.
The 2-call repeat run is the interesting part. Saving the route from the first run (the account id and the direct Workers & Pages URL) and opening it straight away next time would skip the navigation calls entirely and leave the model only the read.
The transcripts do keep the usage fields, so here is the split across all 15 runs per arm:
Uncached input is close to zero in both arms, so nearly everything goes through the cache. Cache reads are 87% of claude-in-chrome's tokens and 93% of opencli's.
So the saving holds in dollars, and it's bigger there than in raw tokens: 62% against 42%. Reads differ by about 1.6x between the arms, but cache writes differ by about 3x. Writes are the new content each turn adds, like screenshots and page text, and they cost far more per token than reads. claude-in-chrome adds more new content per turn, so its extra turns cost more than the token count alone shows.
Saving the route is on the list. The account id and the direct Workers URL are stable, so a second run could skip the dashboard entirely. I'll add it as a repeat-run arm next to the chart task and report both.
That split is the useful part: cache writes are the lever, not turn count. Every new screenshot or page dump gets written once at the higher rate and then re-read on every later turn, so one extra image early in a run is paid for again and again until the end.
Do you know how the 3x write gap divides between screenshots and page text (read_page / get_page_text) in the claude-in-chrome arm? If most of it is text, trimming what each read returns would close a lot of the gap without changing tools.
Looking forward to the repeat-run arm. If the route is fully known, the run only needs the model for the final read, so it should land close to one turn's cost.
Good question, and I didn't split it that way. The usage fields come per turn, not per tool result, so getting the screenshot vs page-text share means assigning each turn's cache write to the tool call just before it. I haven't done that yet. I agree with the idea though: if most of the gap is text, trimming what each read returns is a cheaper fix than switching tools. If I run the split, I'll post it here.
That attribution should be mostly clean, since claude-in-chrome turns usually carry one tool call each. For the turns with two, a rough shortcut: images are billed at about width x height / 750 tokens, so you can price each screenshot from its dimensions and give the rest of that turn's write to the text result.
Looking forward to the split if you run it.
The $0.015 early exit is a useful caveat. Cost per correct completion would make that tradeoff clearer, especially for the Workers/Pages count where both tools disagree. For the canvas follow-up, will you separate extracting plotted values from understanding the chart?
Fair point, and regrading for it changed the Cloudflare result.
I went back to the per-run answers. The prompt asked how many Workers were listed. The Workers & Pages list mixes both, so its count (172-179) is not the Workers number. Two of the three opencli runs said so and pulled 147 from the Workers API. The third stopped early. All three claude-in-chrome runs reported the list count as Workers. So the tools did not agree there: opencli 2/3, claude-in-chrome 0/3, if you take 147 as the answer.
Second change: both opencli dev.to misses were permission denials in my runner (Bash blocked in dontAsk mode), not bad reads. The $0.015 run never opened a tab.
Cost per correct completion, 15 runs each, failed runs still paid for:
Cloudflare alone: $0.90 for 0 correct vs $0.32 for 2 ($0.16 each).
On canvas: the chart follow-up is out: dev.to/vousmeevoyez/your-browser-a... . It only scored extraction, and all 12 runs got the values right. claude-in-chrome hovered the tooltips. opencli never read the chart. It fetched the data behind it (the Wikimedia API, the OWID CSV). So the cheap path skips the chart entirely. Understanding needs a separate task with no data endpoint and an answer that is a judgment, like a trend or an outlier. I have not run that one yet.
Ran the understanding half. Same chart, a question about its shape instead of a value: on Our World in Data's Indonesia life expectancy line, name the two sharpest dips since 1950 and say whether the latest year is above the level before the last dip. Rubric written before the runs.
Both tools got it right 6/6 (1965 and 2020-21, and yes, 2023 at 71.1 is above 2019). How they got there split cleanly. claude-in-chrome read the line by eye in all 3 runs, with values like "about 49.5 down to about 43". opencli fetched the chart's CSV in all 3 and never looked at the line. Cost for 3 runs: $0.90 vs $0.40.
So on this chart, understanding didn't need the picture, because the data was one fetch away. The test that would separate them is a chart with no data behind it, like a static image. The second chart in this round (Wikimedia pageviews, weekly pattern) is pending: the site was unreachable for both tools, so those 6 runs are excluded and I'll rerun them when it's back.
Update on the second chart: the pageviews site stayed down, so I swapped in FRED's US unemployment rate. Question: since 2020, which month peaked and how high, and is the latest value above February 2020?
Both tools 6/6 again (April 2020 at 14.8%, latest 4.2% is above 3.5%). Cost for 3 runs: $0.99 vs $0.43. opencli pulled FRED's CSV every time. claude-in-chrome could only read the spike's height off the chart, and in all 3 runs said the month and the February level came from its own memory of the series.
So that pass is recall, not chart reading. A famous series leaks into an understanding test. The clean version needs data the model can't know.