DEV Community: Arthur Katcher

I benchmarked Jev's box tool on ScreenSpot-v2 and ScreenSpot-Pro. Here is how well it reads a screen

Arthur Katcher — Sun, 18 Oct 2026 07:16:19 +0000

If you're building an agent on Jev, you want to know one thing about its new box tool: can it find what your agent needs to click? I tested it on ScreenSpot, the standard benchmark for this.

The short version is that size decides everything. On normal screens it hits 84%. In 4K pro apps, where the typical target is 21 pixels in the image Jev sees, it drops to 19%. Zooming in on its first answer brings it back to 43%. The long version ends with 5 rules for a harness.

The orange bars are Jev on ScreenSpot-Pro. The bottom one is the raw 19%, next to Claude Sonnet 4. The other two are the same model looking twice and three times, with nothing trained. Only the harness changed.

Some context first. Jev is already the action picker in a lot of computer use agents. It is fast and costs almost nothing, which is why people want it to do the clicking too. I build harnesses that let agents use computers, so I wanted the answer myself.

Nobody had checked

Before running anything, I looked for someone who had. TypeSafe itself calls images and box preview features:

// Detect dark theme var iframe = document.getElementById('tweet-2110536685805355064-540'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=2110536685805355064&theme=dark" }

I found no public ScreenSpot run of box, so I made one.

Finding the element an instruction refers to is called GUI grounding, and ScreenSpot is the standard test for it. It comes in two sizes. ScreenSpot-v2 has 1,272 targets on phone, desktop and web screenshots. ScreenSpot-Pro has 1,581 targets in professional apps like Photoshop, AutoCAD and MATLAB, mostly on 2K to 4K screens. Each case is a screenshot, an instruction and an answer box, and a click inside the box counts. Every target is either text or an icon.

Jev does not write text. You send it a state and typed questions, and it answers each one. A box question returns a rectangle on the image and a confidence.

The setup is as plain as I could make it. One request per click: the screenshot, scaled so both sides fit Jev's 280 to 1,568 px limit, and one box question. The click is the centre of the box Jev returns.

state = [
    {"type": "text", "text": "Screenshot of a user interface:"},
    {"type": "image_url", "image_url": {"url": data_url(screenshot)}},
]
questions = {"target": {"type": "box", "instructions":
    f"In the screenshot, the UI element to use for this task: {instruction}"}}

Scoring is each benchmark's official eval code in the leaderboard setting: English instructions, every case, micro accuracy (the share of all cases hit). The wording was fixed before the first call. No prompt tuning on test cases, no thinking model in front, no planner, no tools, no retries on wrong answers.

Before spending anything I ran offline tests, a smoke run checked by eye, and on Pro two independent review rounds of the harness. Every request, response and cost is saved. The end of the Pro run log:

scored 1581/1581, wrong_format 0, infra_error 0, not run 0
latency median 0.888s p90 1.396s, cost of logged responses $0.2523

That pass, 1,581 clicks, cost 25 cents. Even with the extra looks, Jev costs a fraction of anything near it on the board. The other points here are floors I estimated from list prices and each provider's image-token rule, before any thinking tokens, which nobody publishes.

To sort the misses I had AI reviewers look at each screenshot with every answer, then ran a second pass and a spot-check on their calls. The guardrails are the ones from my last article:

// Detect dark theme var iframe = document.getElementById('tweet-2109698761115251038-309'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=2109698761115251038&theme=dark" }

On ordinary screens, Jev can click

First came ScreenSpot-v2, and Jev hit 84.36%. On phones it hit 89%. A v2 task looks like this one, with a target close to the median size:

Most of the misses were close. The median miss lands 24 px from the target in the image Jev saw. Almost half are one pattern: Jev boxes the whole toolbar or row the target sits in, and the centre of that box lands beside it.

A miss that lands next to the target is the kind a closer look should fix.

So I let it look twice

TypeSafe's Diogo describes box as closer to pointing than to detection:

// Detect dark theme var iframe = document.getElementById('tweet-2110703276673147011-213'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=2110703276673147011&theme=dark" }

If it points, it can point again. Foveation is how eyes work: look roughly, then look again, closer, where you pointed. My zoom step does the same. Call 2 crops half the width and half the height of the screenshot, centred on call 1's box, and asks the same question. The crop goes in at the same image size as the full screenshot, so every target in it comes out twice as big. The answer maps back to the full screen.

On v2 this adds 3.5 points, from 84.36% to 87.89%: 95 misses fixed and 50 hits broken. A plain rerun flips only 3.9% of cases, so the gain is real and not noise. The breaks are mostly small slips, plus crops that cut off the label naming the target.

The zoomed answer also comes with a confidence you can use. On v2, zoomed answers at 0.9 confidence or more were right 98.7% of the time.

Then I froze the rule before the first Pro call, so nothing in it is tuned on Pro.

In Photoshop, it can't

On ScreenSpot-Pro, Jev scored 18.66%. That is where Claude Sonnet 4 sits, at the bottom of the board.

Same model, same prompt, same harness. What changed is the size of the things it has to hit. The median Pro target covers 0.035% of the screen, against 0.72% on v2, about 20 times smaller. A 4K screenshot also has to shrink to 0.41x to fit Jev's limit, so a 40 px icon arrives as 16 px. In the image Jev actually gets, the typical Pro target is 21 px. Here is a Photoshop task on a screen the same size as the v2 one:

The second look was already frozen, and it would make that typical target 43 px. I ran it untouched.

The second look was worth 24 points

Pro went from 18.66% to 42.76%, with 436 misses fixed and 55 hits broken. That is 24 points, about seven times the v2 gain.

That is the middle orange bar in the first chart, between Phi-Ground-7B at 43.2% and GUI-Actor at 42.2%, models trained only for this job. All the runs in one place:

Benchmark	Setup	Calls per click	Score	Cost per 1,000 clicks
ScreenSpot-v2	Jev box	1	84.36%	$0.15
ScreenSpot-v2	+ 2x zoom	2	87.89%	$0.30
ScreenSpot-Pro	Jev box	1	18.66%	$0.16
ScreenSpot-Pro	+ 2x zoom	2	42.76%	$0.32
ScreenSpot-Pro	+ 4x zoom, post-hoc	3	60.15%	$0.48

Raw is one call on the whole screenshot. Raw and zoom are the same run, so each zoom row is the same cases seen again. A separate raw pass scored 83.33% and 19.17%, the same within noise.
The 4x row was designed after I saw the 2x result, and its prediction was committed before it ran. It is a labelled extra, and 42.76% stays the clean Pro number.
Every score uses the official eval code and answer key, including the cases where I think the key is wrong.
Costs are what OpenRouter billed. The key balance matched my logs to the cent.
The other scores in the first chart are as published, with different setups: single-shot, zoom, tools or thinking. Grounding models given their own zoom step score higher, for example MAI-UI-8B at 70.9%.

Why seven times more on Pro? Size decides most of it. Below 20 px Jev almost never hits. Above 80 px it hits 85%. Each zoom step doubles the target, so the typical one moves from 21 px to 43 px to 88 px.

Aviz Maeir found the same in his own tests the day after launch. His box misses were tiny targets like a "+" button or a page number.

// Detect dark theme var iframe = document.getElementById('tweet-2110854390999306324-603'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=2110854390999306324&theme=dark" }

Size is not all of it. Pro is still harder at the same apparent size, by 20 to 35 points against v2. Its screens are dense toolbars of icons with no text, and 295 of its cases ask in English about a Chinese interface.

A third look, called before it ran

If a second look doubles the target, a third should double it again, from 43 px to 88 px. I wanted to know whether I understood the effect well enough to call the number before seeing it.

So before any call I committed a prediction to git: 63.2%, with a range of 59.7 to 66.2. The step is a 4x crop around the 2x answer, a third call per click.

It scored 60.15%, inside the range and 3 points under the centre, with 340 misses fixed and 65 hits broken. That is the top orange bar in the first chart, next to UI-TARS-1.5 at 61.6% and POINTS-GUI-G-8B at 59.9%, though those are single-call scores, and Jev here makes three calls.

I designed this step after seeing the 2x result, so it is an extra. 42.76% stays the clean number.

It also shows where zoom stops. Of the clicks that missed after 2x, the 4x step fixed 56% where Jev had clicked close to the target, and 11% where it had picked something else. Looking closer at the wrong spot rarely helps.

Except on phones

That is also why one platform never moved:

ScreenSpot-v2 platform	Cases	Raw	+ 2x zoom	Change
Mobile	501	89.42%	89.02%	-0.40
Desktop	334	85.93%	91.32%	+5.39
Web	437	77.35%	83.98%	+6.64

Mobile is Jev's best platform raw, and zoom does nothing there. Its mobile misses are not near misses: a median 95 px away on text and 228 px on icons, against 10 to 19 px on desktop and web. Jev picked a different element, so a closer look does not help. A half crop of a phone screen also often cuts off the row label that names the target: 9 of the 14 clicks zoom broke that way were mobile settings rows.

Both benchmarks also score their cases in chapters, by platform on v2 and by app type on Pro, each split into text and icon targets. Every chapter shows the same split. On Pro zoom lifts every row, and after zoom every text chapter scores above every icon chapter.

The benchmarks have bugs too

Looking at every miss turned up label problems in both benchmarks. On v2, one answer box was copied from another screenshot and sits on empty space. On Pro, one English instruction says "redo" where the original Chinese says undo.

On Pro the label problems are worth at most 2.3 points, so the misses there really are the model. I reported them upstream:

Benchmark	Misses with a label issue	Broken tasks	Reported
ScreenSpot-v2	43 of 271	4 wrong keys, 1 duplicate case	OS-Atlas #65
ScreenSpot-Pro	65 of 905	3 wrong labels, 1 contradictory pair, 2 near-duplicates	Issue #32, HF #5

The 271 are every v2 case at least one run missed, and the 905 every Pro case the second look still missed. None of this changes my scores. They stay on the official key.

What this does not show

One model version, jev-1.14-20261014, and box is a preview feature. The next version could move every number.
Jev is not deterministic. On v2 the same request twice returns the same box only 30% of the time, though the click moves a median 4.8 px.
I re-ran no other model. Their scores are as published, and their costs are my estimates.
ScreenSpot is one click on a still screenshot. A real agent also has to notice when the target is missing, and box always returns a box.

Nathan LeClaire, who worked at TypeSafe, says the same: treat box as one signal among several.

// Detect dark theme var iframe = document.getElementById('tweet-2110808613644276211-289'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=2110808613644276211&theme=dark" }

My 5 rules

So can Jev find what your agent needs to click? Yes, if you let it look twice. The harness I would ship from these runs:

Send the screenshot with one box question and click the centre of the box.
Ask again on a crop half the width and height around that box, and map the answer back.
Skip the second look on phone screens, where the misses are wrong picks.
Trust zoomed answers at 0.9 confidence or more. Send the rest to a check or a stronger model.
On dense 4K apps, add a third look at a quarter crop.

The harness, every request and response, the run logs and the audits are at github.com/arthurkatcher/jev-click-bench. A full v2 pass costs about $0.19, so you can check any number here for less than a dollar:

python3 screenspot.py run --pass 1 --workers 4

Next I want a crop that keeps whole rows on phone screens, tested on a separate set so the benchmark stays untouched.

For how people wire Jev into agents in the first place, see my earlier article, How to engineer a Jev harness.

I build harnesses that let agents use computers. If you run Jev for clicking and want to compare notes, reply or write to me.

Originally published on arthurkatcher.com.

Why computer use benchmarks are so hard to run

Arthur Katcher — Mon, 12 Oct 2026 18:17:55 +0000

Let's say you built a computer use agent and want to know how good it is. The usual way is OSWorld, a popular benchmark of 108 tasks on a real desktop. It is easy to start and hard to get right. I read about 200 GitHub issues to see what goes wrong. Here are the 10 problems you will face, and how I would handle each.

The first hard part is the machines. A coding benchmark gets by with a container and a shell. A computer use benchmark needs a whole desktop for every attempt, with its own apps and websites, and the agent takes hundreds of steps per task.

The second is that one run may not be enough. Agents are not deterministic, so a single run can come out higher or lower just by luck. That is why it is good to run the benchmark a few times and average the results. Three runs of OSWorld 2.1 are 324 attempts and roughly 400 machine-hours.

All that machine time is a plain reason so few published scores get re-run. My last article found it on OSWorld 2.0:

// Detect dark theme var iframe = document.getElementById('tweet-2107541507032928639-344'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=2107541507032928639&theme=dark" }

Every September flagship number there is self-reported, and the maintainers' own board has not run those models yet.

Most teams run once: in Yutori's tracker of OSWorld-Verified results, 94 of the 111 entries that list a run count are a single run. Some do repeat: Anthropic averaged five runs for Claude Fable 5.1 on OSWorld 2.0, and Yutori reported three runs for its own Navigator n2.

The 400 is my estimate from Simular's numbers. Simular published every run of its Sai agent on OSWorld 2.1. It took 14 runs and reruns to get one clean result on every task, and that clean pass took about 139 machine-hours, with a median task of 46 minutes and the longest at 8.9 hours. Three passes like that come to roughly 400. OSWorld-Verified, with 369 shorter tasks, costs about the same: Simular reports 4 to 5 hours per full run on 30 parallel environments, close to one 2.1 pass in machine time.

Each of the ten can change the score, and they come in order: on the way from one desktop to a daily run, each fix exposes the next one.

The problems come from other people's runs

I did not run a campaign. Every problem below was reported by people who did, in issues and pull requests on OSWorld, SWE-bench and Terminal-Bench. Most are known, and many are already fixed. The "how I would handle it" parts are my design, and some of it comes from agent runtimes I built before.

The ten problems

Problem	What it does to the number	What to check
1. Capacity	The slowest tasks hold the run	Valid results per hour
2. Preparation	Setup time hides inside task time	Each readiness step
3. Rate limits	Throttled tasks end early	Calls per attempt against quota
4. Isolation	Attempts change each other's results	Two attempts at once
5. Runtime failures	Crashes look like agent failures	Timeouts and real health checks
6. Retries	Actions run twice; best-of reads as mean	When the retry happened
7. Grading	Infrastructure errors become zeros	Ran, graded, evidence kept
8. Reproducibility	Same model, different number	The run manifest
9. Observability	Stuck looks like slow	One ID per attempt
10. Cost	Cheap runs that prove nothing	Cost per valid result

What happens in one attempt

An attempt is one task run once, and it has four parts. The benchmark gives the task and the checks. The desktop gives the apps and the starting state. The agent decides what to do. The harness runs the loop between them: screenshot in, action out, until the agent stops. Then a grader checks what is left on the machine.

In OSWorld 2.1 the desktop is a QEMU virtual machine inside a Docker container, with 4 GB of RAM and 4 virtual CPUs by default. The agent sees screenshots, and in the OSWorld 2.0 paper Claude Opus 4.7 averaged 318 tool calls per task.

Once one task works, you know the model can use the desktop. What you need to know next is how long all 324 attempts take and how many of their results you can trust.

1. Capacity: rent the desktops or run them yourself

On one desktop, the 324 attempts run one after another, and 400 machine-hours means 400 hours of waiting. Parallel desktops are the first answer, and they work on the wait, not the work: the total stays 400 machine-hours. On 40 desktops that is 10 hours if everything packs perfectly, and it never does, because the run ends when the slowest task ends. Simular reports 4 to 5 hours per OSWorld run on 30 parallel environments, sometimes 6, while the run waits for its last tasks.

Parallel desktops need real hardware. Forty of them at the default size is 160 GB of guest RAM and 160 virtual CPUs before anything else runs on the host. Tianbao Xie, who leads OSWorld, described what AWS changed for them:

// Detect dark theme var iframe = document.getElementById('tweet-1954939490101010700-206'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1954939490101010700&theme=dark" }

That hardware can move the score as well as the speed. Anthropic found that with strict per-task CPU and memory limits, up to 6% of Terminal-Bench tasks failed on pod errors unrelated to the model, and loosening the limits moved scores by up to 6 points. Their summary:

// Detect dark theme var iframe = document.getElementById('tweet-2019501512200974686-562'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=2019501512200974686&theme=dark" }

So capacity is a question of managing containers, one per desktop, and there are two ways to get them. One is to rent. The OSWorld team moved to AWS, and sandbox providers such as Cua now run the official OSWorld image on their own cloud. Their limits then become yours: one OSWorld user hit their AWS account's vCPU limit at five desktops.

The other is to run them on your own servers, which makes you the orchestrator. You plan per container: 4 GB of RAM and 4 virtual CPUs each, plus headroom, because tight limits are what cost those 6 points. On one server, the OSWorld team ran 8 or 16 desktops at a time.

Either way, I would size one container first, then add more in steps and watch completed, validly graded attempts per hour. If 32 desktops are already waiting on the model API, 64 mostly doubles the bill.

2. Preparation: boot desktops before the attempts need them

Once you have the machines, the next wait is getting each one ready for its task. Ready is four steps: the image is cached on the host, the VM has booted, control works (the screenshot server answers and the apps open), and the task's files and websites are in place. Each step can be slow on its own.

The OSWorld team saw LibreOffice take a minute to open on first launch, because AWS loads disks restored from snapshots lazily. They cut the image from 50 GB to 25 GB, and first launch went to 5 seconds. Coding benchmarks have the same problem with bigger images. Epoch found SWE-bench's task images totaled 684 GiB and rebuilt them down to 67:

// Detect dark theme var iframe = document.getElementById('tweet-1943369812308017277-915'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1943369812308017277&theme=dark" }

The usual answer is a warm pool: a few desktops booted and checked before they are needed. I built one for agent containers in another system, and in the case I measured a deploy went from about 60 seconds to under a second. For a benchmark, warm also has to mean clean, so a desktop goes back to the pool only after a verified reset. Ten clean desktops keep up with attempts as they arrive. Keeping all 324 warm in advance is a separate spending decision.

So I would gate each attempt on every readiness step, pin the image by digest, and cache it near the workers.

3. Rate limits: every service the run touches has a limit

With the desktops ready, the agents start calling the model. At 318 tool calls per task, 324 attempts come to about 100,000 tool calls driven by the model, most of them carrying a screenshot.

Providers cap that traffic. Princeton's HAL team ran 21,730 agent rollouts across nine benchmarks and found that parallel evaluation runs into rate limits at every provider. When they ran, Anthropic capped API spending at $5,000 a month by default, even on its highest usage tier, so larger runs needed special approval. The tiers have changed since, but the standard ones still have a monthly cap, and reaching it pauses the API with the same 429 error a rate limit returns.

The model API is one of several services a run depends on, and each one has its own limit. Nebius hit rate limits on PyPI, Docker Hub and the Ubuntu package archive while building SWE-bench task images, before any model was involved. Benchmarks that touch the live web meet bot detection, which HUD described in a walkthrough on browser evals:

// Detect dark theme var iframe = document.getElementById('tweet-2104666241453769186-468'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=2104666241453769186&theme=dark" }

A provider's content policy can stop a task outright. When Cua tried to run its benchmark on Muse, a KiCad task was flagged as a violation:

// Detect dark theme var iframe = document.getElementById('tweet-2085715336318386657-20'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=2085715336318386657&theme=dark" }

In OSWorld, the grader and the simulated user can call models as well as the agent, and all of them draw on the same quota unless you split it.

So I would do most of the work before the run starts. First, confirm with the provider the limits on the tier you will actually run on, requests and tokens per minute and the monthly spend cap, and ask for more before the campaign, not in the middle of it. Pull every image and package once into your own mirror, so the public registries see one download instead of one per desktop. Give live-web tasks proxies you control, and pin one IP to any task where a site checks for humans; the OSWorld team did both. And run each task once on the provider you plan to use, so a content policy flag shows up before the campaign instead of as a zero inside it.

During the run, I would put one admission controller in front of the shared model budget. It counts the grader and the simulated user against that budget, starts a desktop only when there is capacity for its traffic, and records a rate-limited attempt as an infrastructure failure to rerun, never as a zero.

4. Isolation: give each attempt its own world, then test it

With attempts running side by side, each one needs a clean world, and a fresh VM only cleans the desktop. A task can post a comment, create an account or upload a file, and those writes live on a server. The next attempt inherits them, and an attempt running at the same time can see them.

WebArena's tooling handles this by running tasks in a fixed order, and its maintainers note that dropping the order for more parallelism can cost 1 to 2 percent. OSWorld 2.0 took the other route: it ships 31 self-hosted websites and scopes each attempt's state by cookie, so attempts do not contaminate each other.

A clean world also means one without the answer in it. SWE-bench images once kept the repository's future commits, and some agents found the fix with git log. The maintainers rebuilt the images and measured the effect as small, about 5 in 10,000 trajectories.

So I would give each attempt one identity across desktop, browser and website, and test it directly: run two attempts at once, change each world differently, and check that neither sees the other.

5. Runtime failures: the desktop dies before the agent does

An attempt that runs for an hour or more outlasts some of the parts under it. One of them crashes, expires or hangs, and the attempt fails for a reason that is not the agent's.

A crash can come from the image itself. OSWorld's in-VM server leaked a display connection on every screenshot, so long tasks crashed after about 220 steps. The fix went into the code in May, but the published image is older than the fix. In the logs it looks like a network error at a random step, until someone counts the steps.

Clocks end attempts the same way. OSWorld 2.1's AWS provider can schedule a VM's termination after 300 minutes by default, while H Company's Holo4 notes list a per-task budget of 500 steps and 6 hours, an hour more than that VM lives. Tokens, sandboxes and leases all have lifetimes, and each has to be longer than the longest task.

A hang is harder to see, because nothing fails. An open pull request on OSWorld found that 10 of 14 controller calls into the VM had no timeout, so a desktop that stops answering leaves the call waiting forever. I built a watchdog for the same shape of problem in an agent runtime: the process looks alive, but its event loop has frozen and nothing answers.

So I would put a timeout on every call, check health by asking the desktop to do something real, and set every lifetime above the task budget.

6. Retries: mean of three and best of three are different scores

Once failures happen, the instinct is to retry. Before the agent's first action that is safe, because the desktop has not changed yet. After it, the desktop has state, and a retry can repeat something that already happened.

An open OSWorld pull request shows how that goes unnoticed. The desktop stops any command that runs past 120 seconds and reports an error, and the controller retried every error, including commands the desktop had already partly run. In that runner's logs, 357 of 23,244 actions ran twice. When the repeated command was a ctrl key press, the key stayed down for the rest of the task, and the trajectory looked normal. Harbor's desktop runtime has the rule in a code comment: replaying a click is not idempotent.

Ashton Chew's input daemon for computer use on Modal makes the same call: once a press may have reached the X server, it releases what it pressed, returns the error, and does not retry.

// Detect dark theme var iframe = document.getElementById('tweet-2083302382545580399-177'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=2083302382545580399&theme=dark" }

Retries at different layers multiply. A Harbor issue describes SDK and harness retries stacking until one hung call could take about 90 minutes.

Rerunning a whole task brings back the policy question from my last article, which run counts. Simular's published runs give 0.7674 if you count the first clean run per task and 0.7938 if you count the best one, 2.6 points apart from the same campaign.

So I would retry freely before the first action, start and log a new attempt after it, give every layer one shared deadline, and say in the report whether the score is a mean of three or a best of three.

7. Grading: check that it ran, was graded and kept its evidence

When the run ends, you have a folder of result files, each with a score. A zero in that folder can mean the agent failed. It can also mean the grader crashed, the output went missing, or the evidence never uploaded, and the file looks the same in every case.

Simular hit this while making its OSWorld 2.1 reference runs. A usage limit on the model key stopped tasks after one step, and they were logged as success with a score of 0. The average over the folder came out at 0.40 and, in their words, looked like a measurement. The step count was what gave it away.

Graders can write the same false zero themselves. The OSWorld maintainers have an open fix for 14 of the 108 tasks, whose graders turn a failed call to the judge model into a zero.

The grader also has to be out of the agent's reach. Some graders read the answer by running commands inside the VM the agent controlled. An engineer at H Company suggested loading the grading code in memory, so it never touches a disk the agent can read.

Teams that publish numbers decide before the run how infrastructure failures count. An engineer at poolside described their rule:

// Detect dark theme var iframe = document.getElementById('tweet-2079860031017693631-838'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=2079860031017693631&theme=dark" }

So I would record three things per attempt: did it run, was it graded, and is the evidence there. I would run the reference solutions before each campaign, because a grader that fails the right answer shows up there first. One team logged 240 SWE-bench rollouts at 0% and blamed their agent until the gold patches failed too. And I would report coverage, the share of attempts that were validly graded, next to the score.

8. Reproducibility: pin everything the number depends on

Once a run is reliable, you run it again after the next harness change. If the number moves, you need to know whether your change moved it.

The harness alone can do it. Anthropic re-ran older models after fixing a bug in its zoom tool and raising the token limit per turn, and wrote that it had been underreporting OSWorld across its model families. Claude Opus 4.7 went from 78.0% to 82.8% with no change to the model.

The rest of the setup moves too. Between OSWorld 2.0 and 2.1, 174 pull requests to the task files were merged. Simular ran the same agent on two desktop images and got 0.7766 on one and 0.7414 on the other. HAL notes that OpenRouter can serve different quantizations of a model under the same endpoint name, so the same name does not guarantee the same model.

So every run gets a manifest, with a line for each thing that can move the number:

task and grader revisions
image digest and website builds
the model and the provider route that actually served it
the judge model
harness commit
observation and action settings, screen size
step and time budgets
resources per desktop
retry and repeat policy: mean or best of three

A manifest does not make a model deterministic. It tells you what stayed the same when the number moved.

9. Observability: you need to tell slow from stuck

Hours into a run, you want to know which attempts are working, which are slow and which are stuck. The first thing you reach for is a live view of one desktop. Terminal-Bench had one early:

// Detect dark theme var iframe = document.getElementById('tweet-2099691702533648842-946'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=2099691702533648842&theme=dark" }

A live view shows one screen. It cannot tell you whether twenty other attempts are stuck in preparation or waiting on the model. A Harbor user spent about two days on a stall like that. Trials sat silent until a 30-minute timeout. The cause was two crashed trials whose 4 GB sandboxes kept running and filled the account's 10 GB memory limit, so every new sandbox was refused and retried without a word.

You need the same visibility after the run, to explain a score. Simular found that OSWorld 2.1's rubric graders return only the total, so a partial score like 0.6969 cannot be explained or re-graded later.

So I would carry one attempt ID from allocation through model calls, desktop actions, grading and cleanup, record how long each phase took, and keep the evidence of failed attempts instead of deleting it.

10. Cost: count cost per valid result

Once the run works every day, it has a bill, and the model is only one line of it. The others are desktops, idle warm capacity, retries, failed preparation, storage, and anything left running after a crash.

Simular published its model spend for the OSWorld 2.1 reference campaign: $2,417.63 over 27,662 gateway calls, about $18 per clean attempt. Desktop hours are not in that number.

Leftovers add up quickly. OSWorld's Docker provider once left a 32 GB volume behind for every task, enough to fill a disk within an hour of a full run, until it was fixed.

At scale, cost decides the method. HAL spent about $40,000 on its rollouts and still had to rely on single runs for most evaluations. Some models they did not run at all:

// Detect dark theme var iframe = document.getElementById('tweet-1978790456197579152-979'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1978790456197579152&theme=dark" }

So I would track total cost divided by validly graded attempts, shown next to coverage. A cheap run that failed early has few valid attempts, so its cost per result goes up. A run that skipped the hard tasks shows up as lower coverage.

A daily run has to hold all ten at once

Every fix in this article is ordinary on its own. The trouble is that each one exposes the next problem, so a daily run has to hold all ten at once. Here is what that takes: eight stations every attempt passes through, and one attempt ID from allocation to cleanup.

You can find each station somewhere today. Parallel desktops, warm pools, retries and dashboards exist in OSWorld's own runner, Harbor, HUD, Cua and the sandbox providers. One setup that holds all of them, on the benchmark's exact package, every day, is still rare. That setup is the hardest benchmark.

What to take away

A benchmark score is the output of an operation. Before you trust one, yours or a lab's, ask how it was run.

Save the table and the manifest. They work for SWE-bench and Terminal-Bench too.

I build harnesses that let agents use computers. If you want the full notes with every source, reply and I will send them.

Originally published on arthurkatcher.com.

How to engineer a Jev harness

Arthur Katcher — Mon, 12 Oct 2026 13:30:00 +0000

I went through how people actually wire Jev into their agents. I skipped the demos and looked at the plumbing: where it sits in the loop, what they feed it, and when they stop trusting it.

This is what the working setups have in common, and what happened to the Jev demo with 3.6M views once someone measured it.

What Jev is

Jev is the model TypeSafe AI launched on September 15. It does not generate text. You send it a state and a list of typed questions, and it answers all of them in one parallel pass: pick one of these options, place this on a scale, or how likely is this to be true. Each answer is a set of probabilities your code can branch on. A call takes a few hundred milliseconds and costs a fraction of a cent.

// Detect dark theme var iframe = document.getElementById('tweet-2099925682726002904-404'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=2099925682726002904&theme=dark" }

That is the whole interface. It does not chat, call tools or remember the previous call, so anything beyond a single decision has to live in your code.

The one sentence to keep:

Jev is a fast judge with no memory, no hands and no voice. It looks at the state you give it, answers the questions you wrote, from the options you listed. Everything else is your code.

So a Jev harness is three things: what you put in the state, how you word the questions and build the options, and what your code does with the answers.

Six seats people are giving Jev in a harness

I read the top hundred posts of the wave. Under the demos, the harness uses collapse into six seats.

1. Router

Before the expensive model runs, Jev picks which model runs. @dani_avila7 shipped a Claude Code mod that classifies the subagent model, the effort level and the main model. The detail that matters: the main model is routed "only at session start to avoid breaking the cache". LangChain ships the same idea as ModelRouterMiddleware.

// Detect dark theme var iframe = document.getElementById('tweet-2101176629745561686-442'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=2101176629745561686&theme=dark" }

2. Gate

A safety check on every command before it executes. @rauchg reports the fx auto-mode reviewer running "5-18x faster and more accurate" on Jev than on the small LLM they used before. LangChain's AutoModeMiddleware is the open version.

A gate that answers in a few hundred milliseconds can run on every call. A slow gate gets turned off.

// Detect dark theme var iframe = document.getElementById('tweet-2100307962262872105-726'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=2100307962262872105&theme=dark" }

3. Selector

Computer use and browser use. Jev never sees a screen. The harness observes, builds a list of candidates, Jev picks one ID, the harness executes and looks again.

@trycua has the cleanest statement of the loop: driver observes, perception parses if needed, client builds candidates, Jev selects one ID, driver executes and verifies fresh state. @awlevin did it with OCR boxes as the option list, about $0.003 and 1.5 seconds a step by his numbers, and hands off to a small LLM when something has to be typed.

// Detect dark theme var iframe = document.getElementById('tweet-2100649543079502213-284'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=2100649543079502213&theme=dark" }

4. Verifier

@omarsar0 put Jev behind the /goal feature of his harness: a check on whether the goal is actually complete, cheap enough to run far more often than the reasoning model he used before.

A reply under his post is worth more than most demos: a check at the moment the agent says "done" caught the fake finishes, but a check on every turn "started overruling steps the agent needed to try".

// Detect dark theme var iframe = document.getElementById('tweet-2101443311454036477-141'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=2101443311454036477&theme=dark" }

5. Advisor

@kunchenguid's compact-adviser does not compact anything. It answers one question: are you at a task boundary where compacting is safe.

He labelled checkpoints from 40 real sessions by hand, tuned the prompt against that set, and made the threshold move: favour precision while the window is mostly empty, shift toward recall as it fills, because the cost of a wrong "no" grows.

// Detect dark theme var iframe = document.getElementById('tweet-2101032677940117875-238'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=2101032677940117875&theme=dark" }

6. Triage

@redp314 reviews PRs with one call. 14 typed checks come back as probabilities: hardcoded secret, touches auth, deletes tests, does the description match the diff. Code turns them into BLOCK / security review / nits / merge, and anything between 0.35 and 0.65 on a critical check goes to a human or a bigger model. His cost: $0.00007 a PR.

// Detect dark theme var iframe = document.getElementById('tweet-2100585126652481915-666'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=2100585126652481915&theme=dark" }

In all six, Jev never acts on its own. The code builds the list of options, Jev picks one, and the code carries it out. All six also have a rule for what happens when Jev is unsure.

The seat that got audited

The biggest post of the wave, 3.6M views, was a seventh seat: compaction. Score every tool call, drop the irrelevant ones, no summarization prompt. One repost reported a session going from nearly 1M tokens to 86K in a second.

// Detect dark theme var iframe = document.getElementById('tweet-2100694549362553153-13'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=2100694549362553153&theme=dark" }

Then people measured it.

@Teknium ran it on the public Nous compaction eval. His finding: what the scoring learned was a rule you can write in one line of code, remove the tool calls. And it decays. The first pass frees half the window, the next pass a quarter, then less, because the only thing it can remove is the thing it already removed.

His numbers against a plain summary: 115K tokens retained versus 55K, recall 76 percent versus 79, and at the same budget the Jev ranking tied with plain recency, 77.8 to 77.8.

// Detect dark theme var iframe = document.getElementById('tweet-2101398453578555898-838'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=2101398453578555898&theme=dark" }

An independent tester, @servasyy_ai, reported the same pattern with a Claude Code compaction plugin: it behaved identically when he swapped Jev for a fake model that always answers 0. Several replies to the original pointed at a third problem: every filtered history is a prompt cache miss.

Teknium was clear that this is a problem with that compaction strategy, not with Jev. Nobody had measured the harness before it went viral, and from the outside the demo looks identical whether the scoring inside it works or not.

Nine rules the working harnesses share

1. Code builds the menu, Jev points

Never let it invent options, it cannot. Every selector that works builds the candidate list in code: from the DOM, from OCR boxes, from the accessibility tree, from the retriever, from the tool registry. And they rebuild it after every action, because a click changes what is on the screen. Otherwise, as @0xCodila put it, the model is choosing from yesterday's menu.

The cap is 255 options per question. For bigger lists the pattern is: filter the obvious mismatches in code, score what is left, then choose among the shortlist.

2. Put the exits in the menu

Jev will always pick something. If "none of these" is not an option, you will get a confident wrong pick. @CodingGarden built a tool-calling chatbot with no LLM at all and handles this inside the option list: one option says "no device or room is mentioned at all", another says "more than one tool call can satisfy this request", and either one triggers a follow-up question instead of an action.

// Detect dark theme var iframe = document.getElementById('tweet-2100665210419950031-156'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=2100665210419950031&theme=dark" }

3. Give it the facts it cannot know

It has no memory and no clock. @ctatedev found that Jev does not know today's date, and adding the date to the state moved one of his answers from 0.35 to 0.94.

It is also text only, with roughly 32K tokens of context. That limit is where the compaction and disk-cleanup experiments of @servasyy_ai died: to fit the limit he had to cut the content, and once the content was cut the model was judging blind. If the evidence does not fit in the state, the answer is a guess with a probability attached.

4. Ask everything in one request

Questions in the same request are answered in parallel, so twenty questions take about as long as one, and output tokens are free. Rosson's PR review is 14 checks in a single call. The catch: questions cannot read each other's answers. If a decision depends on a search result, run the search first.

5. Decide what "unsure" means before you ship

Every builder quoted above had to define this:

Rosson sends anything between 0.35 and 0.65 on a critical check to a human or a bigger model.
@DeRonin_: under 0.5 escalate, 0.85 or more before anything irreversible.
Kun Chen moves the threshold as the context window fills.
@servasyy_ai labelled 300 judgments by hand: all 255 answers at 90 percent confidence or higher were correct, and the middle band had too few samples to say anything.

Confidence is not an accuracy percentage. Treat it as a map of where to spend your expensive model.

// Detect dark theme var iframe = document.getElementById('tweet-2100917158922387537-557'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=2100917158922387537&theme=dark" }

6. Keep a small LLM on the bench

Jev cannot write. The moment a task needs text (type a URL, fill a form field, explain a finding) the harness hands off. @awlevin calls a small model for type_text. Browser Use does the same for input fields. Rosson runs a second LLM pass only on the PRs that Jev flags. The result is that the expensive model gets called far less often.

7. Respect the prompt cache

A router that switches the main model mid-session throws away the cached prompt, and the saving goes with it. That is why the Claude Code router only routes the main model at session start, and why the replies under the compaction demo were full of "you busted your cache". Any harness that edits history or swaps models has to price the cache miss in.

8. Check at the moment it matters, not on every turn

Cheap checks tempt you to run them constantly. The verifier reports say don't: a check when the agent claims "done" catches fake finishes, a check on every turn starts vetoing steps the agent needed to try. Put the judge at decision points: before an irreversible tool call, at a claimed finish, at a task boundary.

9. Measure it against a dumb baseline before you believe it

The compaction demo tied with "keep the most recent messages". A plugin behaved the same with a model that always answers 0. The projects that held up were measured first: Kun Chen hand-labelled 40 sessions before tuning a prompt, and LangChain ran every judge 100 times per case before writing a word.

A cheap test: replace Jev with a constant answer or a coin flip and rerun. If the result looks the same, Jev was not contributing.

And check the rest of the loop before blaming the model. In the Browser Use numbers @0xCodila collected, cutting browser protocol calls from 1,092 to 101 moved task time more than the model swap did.

A short note from my own testing

I ran a few of these ideas on a task with a free referee, chess, where an engine can grade every single decision. Two results I have not seen posted elsewhere:

When Jev hesitates between its own top options, its probabilities are close to a coin flip, but asking "is A better than B" in both orders picks the better one about two times out of three. And with the same answers in hand, weighting a yes/no veto much harder than a plain multiply cut the loss almost in half, with zero extra calls. It is a small sample from one domain, so treat it as a lead. It did convince me that what the code does with the answers deserves as much attention as the prompt.

The model costs almost nothing now, so the work has moved into the harness.

Which seat did you put Jev in, and what broke? Reply with it. The failures are teaching more than the demos right now.

Originally published on arthurkatcher.com.

Terminal UIs for Harnesses in 2026: who builds on what, and why

Arthur Katcher — Thu, 08 Oct 2026 13:30:00 +0000

We spent forty years escaping the command line. Then AI agents arrived, and the best-funded interface teams in software went back to a grid of monospaced characters. So what's happened?

The terminal came back beautiful

In 1984 the Macintosh promised we would never look at a blinking cursor again. In 2026, OpenAI, Anthropic, Google, and xAI all ship their flagship coding products as terminal apps. Not grudgingly, either. These are polished, animated, mouse-aware interfaces that happen to be drawn entirely out of text. Something strange happened on the way to the future, and it is worth understanding what.

A screen made of cells

A terminal gives a program almost nothing. There is no canvas, no DOM, no pixels. There is a grid of monospaced character cells, and two streams of bytes: one going out, one coming in. Print text and it simply fills the next cells in the grid.

Everything else is a trick. The trick is called an escape sequence: a run of bytes starting with the escape character that the terminal reads as a command instead of text. \x1b[5;10H moves the cursor to row 5, column 10. \x1b[31m turns the next characters red. Other sequences clear lines, hide the cursor, or make mouse clicks arrive on the input stream as encoded bytes. An app drawing itself is just a program printing instructions; those are the boxed bytes in fig 01.

A full-screen app adds two more moves. It switches the input to raw mode, so each keystroke arrives instantly instead of waiting for Enter; that is why a single j moves the cursor in vim. And it opens the alternate screen buffer, a second blank screen to draw on. Quit vim and your shell history is exactly where you left it. That is the alternate screen closing.

So every TUI framework ever written has the same job. Hold a model of the grid in memory. Turn your interface into cells. Compare that with what is already on screen, and emit the fewest escape sequences that make the two match. It is the same paint, diff, and flush loop as a game engine, or as React against the DOM. The whole craft lives in that loop, and in one cursed detail: a Japanese character is two cells wide, an emoji is anyone's guess, and every miscount shows up as a broken border.

Why we left, and what pulled us back

We left because the terminal of 1985 deserved to be left. Sixteen colors on a good day. No mouse. No images. The GUI won on fidelity, then the web won on reach, and the terminal became the boiler room: always there, never decorated.

What pulled us back was not nostalgia. It was AI agents, and three unsentimental facts about where their work happens.

First, the terminal is the one interface that already exists everywhere an agent works: every server, every SSH session, every CI box. Second, it composes. Pipes, logs, and replayable event journals are native citizens, and an agent's output is exactly that kind of material. Third, it is cheap to render. A status row updating thirty times a second costs nothing, which matters when the interesting part of your product is the model, not the chrome.

A chat web app shows you what an agent says. The terminal is where the agent's actual work already lives. Putting the interface there removes a whole layer of translation.

The hardware quietly got good

The return would have looked terrible on the terminal we abandoned. It looks good now because the substrate improved while nobody was watching.

Color came first. Two shades became sixteen, then 256, then 24-bit truecolor: 16.7 million colors per cell, standard in every serious emulator today.

Then came the protocols that fixed the old embarrassments. Synchronized output lets an app wrap a whole frame in a pair of escape sequences so the terminal paints it atomically. No flicker, no tearing. Support landed in VS Code's terminal and tmux this year, work driven in part by Claude Code's anti-flicker push. The kitty graphics protocol transmits real pixels, and Ghostty, WezTerm, Konsole, iTerm2, and Warp all speak it. Where it is missing, there is a lovely fallback: the half-block character ▀, with the foreground color as the top pixel and the background as the bottom. Two pixels per cell, on any terminal made since the Reagan administration.

Add GPU-rendered emulators like Ghostty, and beauty stopped being impossible. It just needed a reason. Millions of developers staring at agent harnesses all day turned out to be the reason.

What the AI coding tools run on

Look at what the major coding agents actually ship, and the field splits into camps.

Tool	Stack	Notes
Claude Code	TS, React, forked Ink	custom reconciler, 140+ components
Codex CLI	Rust + ratatui	rewrote from TS + Ink in 2025
Gemini CLI	TS, React 19 + forked Ink	open source
Grok Build	Rust + ratatui	open-sourced July 2026
opencode	OpenTUI + SolidJS	was Go + Bubble Tea
Crush	Go + Bubble Tea v2	the original opencode lineage
Aider	prompt_toolkit + Rich	a REPL by choice

Half that chart says Ink, so Ink deserves a proper introduction. Ink is a renderer that points React at the terminal instead of the browser. You write ordinary React components with hooks and state, lay them out with flexbox through Yoga, the same layout engine React Native uses, and Ink turns the tree into styled lines of text, re-printing what changed. Its pull is simple: millions of developers already think in React, and Ink lets them keep thinking in it. Its limits are just as simple: it is pure JavaScript, line-oriented, capped around 30 frames per second, with no mouse support and no images.

Those limits explain the table's real story. The biggest Ink users all ended up forking Ink rather than using it stock; Claude Code's build carries a custom React reconciler and its own layout engine. And OpenAI threw away a working TypeScript app to rewrite Codex in Rust on ratatui. When teams this well resourced keep rebuilding the same layer by hand, the layer is missing from the ecosystem.

OpenTUI is the most direct answer to that gap, and its origin is the point. The team behind opencode (Anomaly, formerly the SST crew) started with a Go interface built on Bubble Tea, but wanted the whole product in TypeScript. Nothing in the JS world could draw fast enough, so they built their own framework and rebuilt opencode on it. OpenTUI keeps React's programming model but replaces Ink's machinery: a native rendering core written in Zig holds the screen as a buffer of cells, diffs each frame, and repaints only what changed. That buys the things Ink cannot do: no frame cap, full mouse support, real images over the kitty graphics protocol with half-block fallback. It is MIT-licensed, documented at opentui.com, runs opencode in production today, and is the closest thing the TypeScript terminal world has to a common foundation forming in real time.

The other camps are healthy too. Charm shipped Bubble Tea v2 in February, hardened inside Crush. Rust has ratatui, and Python has Textual. Nobody is starving; the fight is over which model wins the next decade of terminal software.

The parts everyone rebuilds

Here is the gap all those forks point at. Chat components exist in every ecosystem: message bubbles, spinners, markdown renderers. But a harness is not a chat. A harness needs a status row that proves the agent is alive and tells you what this turn costs. An approval prompt that gates a real action on your real machine, with the safe option first. A home screen that says which model is pointed at which screen. Nobody ships those, so every team on that chart has built them from scratch, and several built them wrong in the same off-by-one-cell ways.

A copy-paste culture is forming around the problem, in the mold of shadcn/ui: registries like termcn, TermUI, and InkUI distribute terminal component source you own and edit. Vercel's @ai-sdk/tui attacks the same space from the SDK side.

I got tired of rebuilding the harness parts too, so I decided to open source the parts I use. **agentparts* is a small library of exactly these components, built on OpenTUI: the turn status row, the approval prompt, the shell layout, the prompt box. All the width arithmetic lives in a renderer-free core that runs on plain Node. Feel free to use for your harnesses!*

>> https://github.com/arthurkatcher/agentparts

So you want to build one

Start with an honest question: does this need to be full-screen at all? Aider and Goose are proof that a well-formatted REPL can carry a serious product. If your interface is a conversation with occasional structure, print good output and stop. Reach for a TUI framework when you need layout: panes, status rows, things that update in place.

If you do need one, the choice is mostly made by the language you already live in:

You write	Reach for	Why
TypeScript	Ink, or OpenTUI	Ink is mature; OpenTUI adds speed, mouse, images
Rust	ratatui + crossterm	the default; start from ratatui/templates
Go	Bubble Tea v2 + Bubbles	best component library, Lip Gloss for styling
Python	Textual	CSS-like styling, rich widget gallery

The beautiful part is not the framework. It is discipline about the medium:

Design a palette, then design its absence. Pick a handful of truecolor values on purpose, and decide what the interface looks like with color stripped away: plainly unstyled, never broken. Honor NO_COLOR, the environment variable users set when they want none. Get width right. Test with CJK text and emoji, and snapshot your layout at 40, 80, and 132 columns, because every TUI bug is an off-by-one-cell bug. Never flicker. Wrap each frame in synchronized output, the escape pair that tells the terminal to paint it in one go, and diff before you draw. Shed, don't squish. When the window gets narrow, drop the least important element entirely instead of truncating everything a little.

And steal the parts that already exist: the spinners, the status rows, the prompts. Bubbles has them in Go, the registries have them in TypeScript. The chrome is the least differentiated thing you will build, which is exactly why you should not build it twice.

Step back far enough and the cycle stops looking like a cycle. We did not return to the terminal we left. We brought everything back with us: React reconcilers running flexbox, CSS-like styling, component registries, GPU rendering, twenty years of interface craft applied to a character grid. The real trade was never terminal versus GUI. It is fidelity versus leverage, how rich an interface can be against how many places it can reach, and right now leverage is winning.

Good design finally reached the terminal, one escape sequence at a time.

Originally published on arthurkatcher.com.

My checklist for reading computer use benchmarks

Arthur Katcher — Wed, 07 Oct 2026 07:39:42 +0000

I read about 40 benchmark repositories and 230 sources to understand computer use scores. I came out with six questions I now ask before I believe any of them.

I started because the September launch posts stopped making sense. Claude Opus 5.5 is at 81.8%. GPT-6 Astra is at 72.6%. Both numbers say OSWorld 2.0, and they come from different task files, different subsets and different harnesses.

// Detect dark theme var iframe = document.getElementById('tweet-2102435517165912464-239'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=2102435517165912464&theme=dark" }

// Detect dark theme var iframe = document.getElementById('tweet-2095595742975197690-59'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=2095595742975197690&theme=dark" }

One model, Claude Opus 5, sits anywhere from 31.4% to 79.2% on the benchmark's own official board. And I caught OpenAI moving Astra's score from 72.6% to 73.5% after launch, with no note.

So nobody can say who is best at computer use right now. What you can do is learn what a number was measured on.

What I did not do

I did not run any of these benchmarks. I never started a VM, an agent or a judge.

Everything here comes from reading the labs' launch posts and system cards, the leaderboards, and the benchmark repositories as they stood on October 4, 2026. Every lab number is the lab's own run unless I say otherwise.

The checklist

Ask this	Why it matters
Which benchmark?	Three different tests are all called computer use. Winning one says little about the others.
Which task release?	The test keeps getting fixed. Newer task files alone add up to 13 points.
Full set or offline subset?	Skipping the 26 internet tasks adds about two points, the size of the gap between leaders.
Strict or partial credit?	Partial credit counts half-done jobs. The same run is 82% one way, 49% the other.
Whose run, with which tools?	Labs pick their own setup and grader. An agent that writes code can skip the screen.
How many runs, and what is the spread?	One model, same settings, sits twice on one board 4.4 points apart. The leads are smaller than that.

Each question marks a place where two numbers with the same name stop measuring the same thing. The rest of this article is one block per question.

1. Which benchmark?

In September every frontier lab put a computer use number in its launch post. They did not put the same numbers there.

The set has narrowed to three names. OSWorld 2.0, or its 2.1 revision, is the headline everyone quotes. Next to it sit Agents' Last Exam and AutomationBench, and sometimes ScreenSpot-Pro. Outside the US, OSWorld-Verified is still the main row. xAI's Grok 4.7 launch page does not use the phrase "computer use" at all.

The names hide very different tests. This is what each one gives the agent and how it checks the result.

Benchmark	Where the agent works	Example task	Graded on
ScreenSpot-Pro	1,581 static screenshots	Point at one element in a pro app	Click lands in the right box
OSWorld-Verified	Ubuntu VM, 369 short tasks	"Change the 2 in H2O to a subscript"	Mostly the output file
OSWorld 2.0	Ubuntu VM and mock sites, 108 tasks	Rebuild a part from a PDF drawing in FreeCAD	About 27 weighted checkpoints
Agents' Last Exam	Linux and Windows VMs	Add a bridge to a 3D site model in Rhino 8	The files the agent leaves behind
AutomationBench	Fake business APIs, no screen	Mark a deal as won and route the notice	Every check must pass
WebArena	Five self-hosted sites from 2023	"Top-1 best-selling product in 2022"	An answer string, URL or page state

OSWorld 2.0 is the closest to what most people picture. The agent gets an Ubuntu machine at 1920x1080, sees only screenshots, and has up to 500 steps. A browser is needed in 62 of the 108 tasks, and most tasks need at least two apps: mock mail and chat apps, Writer, Calc, VS Code, a video editor. A skilled human needs about 1.6 hours for the median task. In the paper's example run, a FreeCAD task took 202 steps and scored 0.35.

The maintainers keep the tasks behind a gated download to slow down leakage. @XLangNLP launched OSWorld 2.0 on June 26 by pointing out that agents already scored 83.5% on the first OSWorld:

// Detect dark theme var iframe = document.getElementById('tweet-2070517498974253269-286'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=2070517498974253269&theme=dark" }

Agents' Last Exam comes from Berkeley. It gives the agent a task description and a real machine, lets it work, and scores the files it leaves behind. The tasks look like work. One asks for a bridge added to an existing site model in Rhino 8, and the grader renders the model from 12 views. Another asks for American option prices in Python, with the answer in a results file. About two thirds of the public tasks run on Linux and the rest on Windows with software like Rhino, KiCad and Blender. By my count only about a quarter of them name a GUI app at all. The entrants on its board are Claude Code and Codex.

AutomationBench, from Zapier, has no screen of any kind. The agent gets three tools, an API search, an API fetch and a base64 encoder, over a pretend world of business apps. A typical prompt reads "We just closed the Meridian Corp Platform Deal! Mark it as won and route the win notice to the right team per our routing policy." A task passes only if every check holds, and about four in ten of those checks make sure the agent did not touch something it should not have.

So the three benchmarks labs print together measure three different skills: driving a desktop, finishing a professional job by any route, and calling business APIs in the right order. All three get filed under computer use.

The older names are mostly gone, and each one left for a reason.

WebVoyager came first. Agents graded themselves at close to 90%. When humans graded the same agents in 2025, Browser Use fell from 89% to 30% and OpenAI's Operator from 87% to 61%. @ysu_nlp, one of the authors of that study, posted the result:

// Detect dark theme var iframe = document.getElementById('tweet-1904592235728896199-794'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1904592235728896199&theme=dark" }

WebArena was next. Four in ten of its tasks are question answering, and in April 2026 a Berkeley audit scored about 100% on all 812 tasks without doing any of them, by reading the task files that held the answers. GPT-5.4 in March 2026 was the last OpenAI, Anthropic or Google release to print a web-only benchmark.

Then the first OSWorld. A replay script that never looks at the screen scored 71.1 against a frontier model's 70.6. Models passed the human baseline of 72.4%, and by June 2026 they sat at 81 to 85%. When OSWorld 2.0 launched on June 26, Gemini 3.6 Flash went from 83.0% on the old test to 33.8% on the new one.

Three months later, lab-run partial numbers on OSWorld 2.0 are already above 80%. The maintainers see it too. When Meta's Muse Spark went from 47.6 to 66.9 in one month, @TianbaoX wrote that they need to cook the next OSWorld:

// Detect dark theme var iframe = document.getElementById('tweet-2095240838171553860-313'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=2095240838171553860&theme=dark" }

So the first thing I check is the name of the benchmark, and then whether the agent in it ever had to look at a screen.

2. Which task release?

OSWorld 2.0 is not one test. It has three task releases: June 24, August 8, and v2.1 from September 10.

The changes between them are not cosmetic. Between June and August, 55 merged pull requests touched 43 of the 108 tasks, with titles like "reward hack", "solution-leak hack" and "evaluator bypass". Between August and v2.1, another 119 touched 68 tasks. By their titles, 23 loosened graders, 13 hardened them against cheating and 7 clarified instructions. Some tasks could not reach full credit at all before v2.1.

On the maintainers' own board, the same model at the same effort moved a lot. Claude Opus 5 went from 31.4% to 44.3% strict and from 68.3% to 77.7% partial, only from the v2.1 revision.

The labs do not use the same release. Anthropic reports on v2.1. OpenAI, Google and Meta report on the August files. Any chart that puts an Anthropic number next to an OpenAI number has this gap inside it.

There is also a version from outside the lab. @shaped audited all 108 tasks of the August snapshot and upheld 43 findings, 18 of them major:

// Detect dark theme var iframe = document.getElementById('tweet-2098561915253612969-949'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=2098561915253612969&theme=dark" }

Numbers also move inside one release, and nobody says why. This is the one I caught myself. GPT-6 Astra was at 72.6% in its launch post on September 3. In the two OpenAI posts that followed it is at 73.5%. The label under both charts is the same, and I found no explanation in the posts.

// Detect dark theme var iframe = document.getElementById('tweet-2106679076635185290-545'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=2106679076635185290&theme=dark" }

It is not only Astra. GPT-5.6 Sol has three values on the same setting across OpenAI's posts: 62.6%, 65.7% and 66.2%. Anthropic's Opus 5 was 70.6% at launch in July, 75.4% on the August files with Anthropic's own fixes, and 74.0% on the September files. Anthropic at least says its results are not comparable with earlier releases or other harnesses, and gives that as its reason for showing no competitor.

The other benchmarks drift the same way. AutomationBench's changelog shows bug fixes moving public scores from 30.3% to 41.0% for Opus 4.8 and from 29.2% to 45.8% for GPT-5.6 Sol, and says the private tasks were made a bit harder to compensate. OSWorld-Verified has no release tags at all, and 114 of its task files changed after the July 2025 refresh. A score there is only reproducible against a commit hash.

So the second thing I check is the date or version of the task files. If the post does not say, I treat the number as unplaced.

3. Full set or offline subset?

OSWorld 2.0 has 108 tasks. Some of them depend on the live internet, so there is also an offline subset of 82. The 26 tasks it leaves out are not documented. Google describes the offline subset as the one with "more robust verifiers", which suggests the live-internet tasks are the fragile ones.

The subset alone is worth about two points. On the maintainers' board, Claude Opus 5 is 68.3% partial on the August full set and 70.2% on the August offline subset. On v2.1 it is 77.7% and 79.2%.

Two points sounds small until you look at the size of the test. With 108 tasks, one task is worth 0.93 points. The top gap in the OSWorld 2.0 launch paper, 20.6% against 18.2%, is two or three tasks.

The labs split here too. OpenAI and Google report the offline subset. Anthropic and Meta report the full set. Google left Anthropic out of its OSWorld row for exactly this reason: Anthropic only reports the online and offline tasks combined.

That leaves one group of numbers you can actually line up: the August files, the offline subset, partial credit.

On that setting Astra leads with 72.6% to 73.5%, then GPT-6.1 Sol at 71.4%, the maintainers' Opus 5 at 70.2% and Gemini 4 Argon at 69.2%. Claude Opus 5.5, the highest number anyone has published, is not in this group at all.

The same split exists on the other two benchmarks. Agents' Last Exam has a full set and a Linux-only split of 105 tasks, which is the one most Chinese labs quote. AutomationBench has a private set that Zapier runs, a public set of 600 that labs run themselves, and a third, partial-credit variant from Artificial Analysis. Meta's 49.6% and DeepSeek's 54.8% are on the public set. Google's 51.3% is on the private one. They are three different scales with one name.

So the third thing I check is which tasks were in the run. If two numbers come from different subsets, they do not go in one chart.

4. Strict or partial credit?

Each OSWorld 2.0 task has about 27 weighted checkpoints. Partial credit is the weighted sum. Strict means the score is exactly 1.0, every checkpoint passed.

The gap between the two is about 30 points. Opus 5.5 is 81.8% partial and 48.7% strict. Meta's Muse Spark 1.3 is 66.9% and 32.0%. Meta is the only US lab that prints both in its scorecard. OpenAI and Google publish partial only.

The partial number is the one in every headline. It tells you the agent got most of the way through most tasks. The strict number tells you how often it finished the job. If I hand an agent a task at work, the strict number is the one I care about.

One of the OSWorld maintainers, @taoyds, said the same thing under a post celebrating Opus 5 at 70.6%. That number is partial credit, and under strict success Opus 5 was still only about 30%:

// Detect dark theme var iframe = document.getElementById('tweet-2080910519566045250-46'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=2080910519566045250&theme=dark" }

The same thing happens on Agents' Last Exam, and here it decides who leads. OpenAI quotes Astra at 59.3%. Google quotes Astra at 34.2% and its own Gemini 4 Argon at 39.5%.

Both are right. The board has two columns. "Score" is the average partial credit and "pass rate" is the share of perfect runs. OpenAI took the first and Google took the second. So "Astra leads" and "Argon leads" are both true, depending on the column. Argon's 39.5% is Google's own run and is not on the board.

AutomationBench is strict by design, which is why its top scores sit between 40% and 51%. The partial-credit variant from Artificial Analysis puts the same models near 70%: Argon 77.5, Sonnet 5.5 71.8, Opus 5.5 69.5.

The aggregator sites mix the two. One lists Anthropic's strict 48.7% for Opus 5.5 in the same column as Astra's partial 72.6%. Another calls Anthropic's 81.8% partial score a binary completion rate.

So the fourth thing I check is the metric. If the post does not say strict or partial, it is almost always partial.

5. Whose run, with which tools?

Every September flagship number on OSWorld 2.0 is self-reported. The maintainers' OSWorld 2.0 board stops at Claude Opus 5 and GPT-5.6 Sol. There is no maintainer-run row for GPT-6 Astra, GPT-6.1 Sol, Opus 5.5, Fable 5.1, Sonnet 5.5, any Gemini or Muse Spark.

That matters less when a lab uses the public release as it is. OpenAI's GPT-5.6 Sol number sits within about two points of the maintainers' run. It matters more when a lab changes the task files or the grading. Anthropic's 75.4% for Opus 5 on the August release was five to seven points above the maintainers' 68.3% to 70.2%.

The run itself is set up differently by each lab. Anthropic swaps the model that grades about a ninth of the score, using Opus 4.8 where the official setup uses Sonnet 4.6.

The harness can move a score as much as a new model. Anthropic's Opus 4.7 went from 78.0% to 82.8% on OSWorld-Verified after a bug fix in its zoom tool and a larger token limit per turn, and Anthropic wrote that it had been underreporting OSWorld across its models. OpenAI's GPT-5.3-Codex went from 64.7% to 74.0% with a new setting that keeps screenshots at full resolution.

Then the tools. I expected these benchmarks to test clicking and typing. Mostly they test getting the work done by any route. The OSWorld 2.0 paper looked at how models solved tasks. GPT-5.5 got 78% of its successes through code, API or file routes and 16% through the interface.

A reader saw it on launch day. @labomen001 opened one of the first runs and watched GPT go the code way and find the source of the local service:

// Detect dark theme var iframe = document.getElementById('tweet-2070649723946668443-133'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=2070649723946668443&theme=dark" }

Version 2.1 ships scaffolds for Claude Code and Codex, including a mode with no screen tool at all. Meta runs with a screen tool only, so its 66.9% measures a different skill than a run where the agent has a shell. ByteDance's Seed 2.1 card shows the effect inside one document: Seed 2.1 Pro gets 78.8% on OSWorld when the agent may run bash commands and scripts, and 72.6% when it may only use the screen.

The last part of the setup is the bill. When the top numbers sit within a few points, the price is the real difference. In OpenAI's own charts, GPT-6.1 Sol gets 71.4% at $1.27 per task. Astra gets 73.5% at $9.07. Claude Opus 5 gets 70.2% at $24.11. That is two points for seven times the price.

// Detect dark theme var iframe = document.getElementById('tweet-2104986129686741046-49'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=2104986129686741046&theme=dark" }

The specialist companies already print dollars under every number. H Company's Holo4 gets 61.7% at $1.22 per task. Yutori's n2 gets 65.2% at $1.46. Simular's Sai gets 73.0% at $15.70. I think cost per task is the column to watch next year.

So the fifth thing I check is who ran it, with which tools, and for how much.

6. How many runs, and what is the spread?

Agent runs are noisy. On the OSWorld-Verified board one model appears twice with identical settings, at 82.6% and 78.2%. That is a 4.4 point spread from nothing but chance. The gap between the leaders in the setting OpenAI and Google share is about two points.

So the run count matters. Anthropic reports the mean of five runs. Google reports the best of three. H Company prints "mean over 2 to 4 runs, single run on OSWorld 2.0" under its table. OpenAI does not say. A best-of-three number and a mean-of-five number are not the same statistic, even on the same files.

The Astra edit reads differently through this lens. 72.6% to 73.5% is less than one point, and at high effort the same model moved down, 71.6% to 70.0%. Both moves sit inside the noise. That is the strongest argument for a note: without one, a reader cannot tell a rerun from a fix.

None of the launch posts publishes per-task results for each run. With those you could compute a confidence interval yourself and see whether a two point lead means anything. Without them a single percentage is a point with no error bar.

Arhan Canli raised this in the comments under this article, and he is right that it belongs on the list.

So the sixth thing I check is how many runs sit behind the number, and whether anyone published the spread.

So who is best?

There are four honest answers, and they name four different models.

"Best" on	Model	Number
Highest published	Claude Opus 5.5	81.8% partial, v2.1 full set, own run
The setting OpenAI and Google share	GPT-6 Astra	72.6% to 73.5% partial, August offline
The maintainers' own board	Claude Opus 5	77.7% partial, v2.1 full set
The one board a neutral party runs	Gemini 4 Argon	51.3% strict, AutomationBench, no screen

Opus 5.5 and Astra have never been run on the same files by anyone. The only benchmark where one outside party runs every flagship is AutomationBench, and it has no screen in it.

The honest caveats

As I said at the top, I read these benchmarks and did not run them. The counts from the repositories are static counts at the commit I read.

Google and Meta publish their numbers inside images, and the Anthropic system card figures come from the PDF text. I checked them, but a misread is possible.

This is a snapshot from October 4, 2026. The maintainers could run the September models next week and settle some of it.

The takeaway

A computer use number is only worth what it was measured on. Before you compare two of them, ask which benchmark, which release, which subset, strict or partial, whose run, and how many runs.

Save the checklist table above. It works on the next launch post too.

I build harnesses that let agents use computers. If you want the full notes with every source, reply and I will send them.

Originally published on arthurkatcher.com.