Let's say you built a computer use agent and want to know how good it is. The usual way is OSWorld, a popular benchmark of 108 tasks on a real desktop. It is easy to start and hard to get right. I read about 200 GitHub issues to see what goes wrong. Here are the 10 problems you will face, and how I would handle each.
The first hard part is the machines. A coding benchmark gets by with a container and a shell. A computer use benchmark needs a whole desktop for every attempt, with its own apps and websites, and the agent takes hundreds of steps per task.
The second is that one run may not be enough. Agents are not deterministic, so a single run can come out higher or lower just by luck. That is why it is good to run the benchmark a few times and average the results. Three runs of OSWorld 2.1 are 324 attempts and roughly 400 machine-hours.
All that machine time is a plain reason so few published scores get re-run. My last article found it on OSWorld 2.0:
Every September flagship number there is self-reported, and the maintainers' own board has not run those models yet.
Most teams run once: in Yutori's tracker of OSWorld-Verified results, 94 of the 111 entries that list a run count are a single run. Some do repeat: Anthropic averaged five runs for Claude Fable 5.1 on OSWorld 2.0, and Yutori reported three runs for its own Navigator n2.
The 400 is my estimate from Simular's numbers. Simular published every run of its Sai agent on OSWorld 2.1. It took 14 runs and reruns to get one clean result on every task, and that clean pass took about 139 machine-hours, with a median task of 46 minutes and the longest at 8.9 hours. Three passes like that come to roughly 400. OSWorld-Verified, with 369 shorter tasks, costs about the same: Simular reports 4 to 5 hours per full run on 30 parallel environments, close to one 2.1 pass in machine time.
Each of the ten can change the score, and they come in order: on the way from one desktop to a daily run, each fix exposes the next one.
The problems come from other people's runs
I did not run a campaign. Every problem below was reported by people who did, in issues and pull requests on OSWorld, SWE-bench and Terminal-Bench. Most are known, and many are already fixed. The "how I would handle it" parts are my design, and some of it comes from agent runtimes I built before.
The ten problems
| Problem | What it does to the number | What to check |
|---|---|---|
| 1. Capacity | The slowest tasks hold the run | Valid results per hour |
| 2. Preparation | Setup time hides inside task time | Each readiness step |
| 3. Rate limits | Throttled tasks end early | Calls per attempt against quota |
| 4. Isolation | Attempts change each other's results | Two attempts at once |
| 5. Runtime failures | Crashes look like agent failures | Timeouts and real health checks |
| 6. Retries | Actions run twice; best-of reads as mean | When the retry happened |
| 7. Grading | Infrastructure errors become zeros | Ran, graded, evidence kept |
| 8. Reproducibility | Same model, different number | The run manifest |
| 9. Observability | Stuck looks like slow | One ID per attempt |
| 10. Cost | Cheap runs that prove nothing | Cost per valid result |
What happens in one attempt
An attempt is one task run once, and it has four parts. The benchmark gives the task and the checks. The desktop gives the apps and the starting state. The agent decides what to do. The harness runs the loop between them: screenshot in, action out, until the agent stops. Then a grader checks what is left on the machine.
In OSWorld 2.1 the desktop is a QEMU virtual machine inside a Docker container, with 4 GB of RAM and 4 virtual CPUs by default. The agent sees screenshots, and in the OSWorld 2.0 paper Claude Opus 4.7 averaged 318 tool calls per task.
Once one task works, you know the model can use the desktop. What you need to know next is how long all 324 attempts take and how many of their results you can trust.
1. Capacity: rent the desktops or run them yourself
On one desktop, the 324 attempts run one after another, and 400 machine-hours means 400 hours of waiting. Parallel desktops are the first answer, and they work on the wait, not the work: the total stays 400 machine-hours. On 40 desktops that is 10 hours if everything packs perfectly, and it never does, because the run ends when the slowest task ends. Simular reports 4 to 5 hours per OSWorld run on 30 parallel environments, sometimes 6, while the run waits for its last tasks.
Parallel desktops need real hardware. Forty of them at the default size is 160 GB of guest RAM and 160 virtual CPUs before anything else runs on the host. Tianbao Xie, who leads OSWorld, described what AWS changed for them:
That hardware can move the score as well as the speed. Anthropic found that with strict per-task CPU and memory limits, up to 6% of Terminal-Bench tasks failed on pod errors unrelated to the model, and loosening the limits moved scores by up to 6 points. Their summary:
So capacity is a question of managing containers, one per desktop, and there are two ways to get them. One is to rent. The OSWorld team moved to AWS, and sandbox providers such as Cua now run the official OSWorld image on their own cloud. Their limits then become yours: one OSWorld user hit their AWS account's vCPU limit at five desktops.
The other is to run them on your own servers, which makes you the orchestrator. You plan per container: 4 GB of RAM and 4 virtual CPUs each, plus headroom, because tight limits are what cost those 6 points. On one server, the OSWorld team ran 8 or 16 desktops at a time.
Either way, I would size one container first, then add more in steps and watch completed, validly graded attempts per hour. If 32 desktops are already waiting on the model API, 64 mostly doubles the bill.
2. Preparation: boot desktops before the attempts need them
Once you have the machines, the next wait is getting each one ready for its task. Ready is four steps: the image is cached on the host, the VM has booted, control works (the screenshot server answers and the apps open), and the task's files and websites are in place. Each step can be slow on its own.
The OSWorld team saw LibreOffice take a minute to open on first launch, because AWS loads disks restored from snapshots lazily. They cut the image from 50 GB to 25 GB, and first launch went to 5 seconds. Coding benchmarks have the same problem with bigger images. Epoch found SWE-bench's task images totaled 684 GiB and rebuilt them down to 67:
The usual answer is a warm pool: a few desktops booted and checked before they are needed. I built one for agent containers in another system, and in the case I measured a deploy went from about 60 seconds to under a second. For a benchmark, warm also has to mean clean, so a desktop goes back to the pool only after a verified reset. Ten clean desktops keep up with attempts as they arrive. Keeping all 324 warm in advance is a separate spending decision.
So I would gate each attempt on every readiness step, pin the image by digest, and cache it near the workers.
3. Rate limits: every service the run touches has a limit
With the desktops ready, the agents start calling the model. At 318 tool calls per task, 324 attempts come to about 100,000 tool calls driven by the model, most of them carrying a screenshot.
Providers cap that traffic. Princeton's HAL team ran 21,730 agent rollouts across nine benchmarks and found that parallel evaluation runs into rate limits at every provider. When they ran, Anthropic capped API spending at $5,000 a month by default, even on its highest usage tier, so larger runs needed special approval. The tiers have changed since, but the standard ones still have a monthly cap, and reaching it pauses the API with the same 429 error a rate limit returns.
The model API is one of several services a run depends on, and each one has its own limit. Nebius hit rate limits on PyPI, Docker Hub and the Ubuntu package archive while building SWE-bench task images, before any model was involved. Benchmarks that touch the live web meet bot detection, which HUD described in a walkthrough on browser evals:
A provider's content policy can stop a task outright. When Cua tried to run its benchmark on Muse, a KiCad task was flagged as a violation:
In OSWorld, the grader and the simulated user can call models as well as the agent, and all of them draw on the same quota unless you split it.
So I would do most of the work before the run starts. First, confirm with the provider the limits on the tier you will actually run on, requests and tokens per minute and the monthly spend cap, and ask for more before the campaign, not in the middle of it. Pull every image and package once into your own mirror, so the public registries see one download instead of one per desktop. Give live-web tasks proxies you control, and pin one IP to any task where a site checks for humans; the OSWorld team did both. And run each task once on the provider you plan to use, so a content policy flag shows up before the campaign instead of as a zero inside it.
During the run, I would put one admission controller in front of the shared model budget. It counts the grader and the simulated user against that budget, starts a desktop only when there is capacity for its traffic, and records a rate-limited attempt as an infrastructure failure to rerun, never as a zero.
4. Isolation: give each attempt its own world, then test it
With attempts running side by side, each one needs a clean world, and a fresh VM only cleans the desktop. A task can post a comment, create an account or upload a file, and those writes live on a server. The next attempt inherits them, and an attempt running at the same time can see them.
WebArena's tooling handles this by running tasks in a fixed order, and its maintainers note that dropping the order for more parallelism can cost 1 to 2 percent. OSWorld 2.0 took the other route: it ships 31 self-hosted websites and scopes each attempt's state by cookie, so attempts do not contaminate each other.
A clean world also means one without the answer in it. SWE-bench images once kept the repository's future commits, and some agents found the fix with git log. The maintainers rebuilt the images and measured the effect as small, about 5 in 10,000 trajectories.
So I would give each attempt one identity across desktop, browser and website, and test it directly: run two attempts at once, change each world differently, and check that neither sees the other.
5. Runtime failures: the desktop dies before the agent does
An attempt that runs for an hour or more outlasts some of the parts under it. One of them crashes, expires or hangs, and the attempt fails for a reason that is not the agent's.
A crash can come from the image itself. OSWorld's in-VM server leaked a display connection on every screenshot, so long tasks crashed after about 220 steps. The fix went into the code in May, but the published image is older than the fix. In the logs it looks like a network error at a random step, until someone counts the steps.
Clocks end attempts the same way. OSWorld 2.1's AWS provider can schedule a VM's termination after 300 minutes by default, while H Company's Holo4 notes list a per-task budget of 500 steps and 6 hours, an hour more than that VM lives. Tokens, sandboxes and leases all have lifetimes, and each has to be longer than the longest task.
A hang is harder to see, because nothing fails. An open pull request on OSWorld found that 10 of 14 controller calls into the VM had no timeout, so a desktop that stops answering leaves the call waiting forever. I built a watchdog for the same shape of problem in an agent runtime: the process looks alive, but its event loop has frozen and nothing answers.
So I would put a timeout on every call, check health by asking the desktop to do something real, and set every lifetime above the task budget.
6. Retries: mean of three and best of three are different scores
Once failures happen, the instinct is to retry. Before the agent's first action that is safe, because the desktop has not changed yet. After it, the desktop has state, and a retry can repeat something that already happened.
An open OSWorld pull request shows how that goes unnoticed. The desktop stops any command that runs past 120 seconds and reports an error, and the controller retried every error, including commands the desktop had already partly run. In that runner's logs, 357 of 23,244 actions ran twice. When the repeated command was a ctrl key press, the key stayed down for the rest of the task, and the trajectory looked normal. Harbor's desktop runtime has the rule in a code comment: replaying a click is not idempotent.
Ashton Chew's input daemon for computer use on Modal makes the same call: once a press may have reached the X server, it releases what it pressed, returns the error, and does not retry.
Retries at different layers multiply. A Harbor issue describes SDK and harness retries stacking until one hung call could take about 90 minutes.
Rerunning a whole task brings back the policy question from my last article, which run counts. Simular's published runs give 0.7674 if you count the first clean run per task and 0.7938 if you count the best one, 2.6 points apart from the same campaign.
So I would retry freely before the first action, start and log a new attempt after it, give every layer one shared deadline, and say in the report whether the score is a mean of three or a best of three.
7. Grading: check that it ran, was graded and kept its evidence
When the run ends, you have a folder of result files, each with a score. A zero in that folder can mean the agent failed. It can also mean the grader crashed, the output went missing, or the evidence never uploaded, and the file looks the same in every case.
Simular hit this while making its OSWorld 2.1 reference runs. A usage limit on the model key stopped tasks after one step, and they were logged as success with a score of 0. The average over the folder came out at 0.40 and, in their words, looked like a measurement. The step count was what gave it away.
Graders can write the same false zero themselves. The OSWorld maintainers have an open fix for 14 of the 108 tasks, whose graders turn a failed call to the judge model into a zero.
The grader also has to be out of the agent's reach. Some graders read the answer by running commands inside the VM the agent controlled. An engineer at H Company suggested loading the grading code in memory, so it never touches a disk the agent can read.
Teams that publish numbers decide before the run how infrastructure failures count. An engineer at poolside described their rule:
So I would record three things per attempt: did it run, was it graded, and is the evidence there. I would run the reference solutions before each campaign, because a grader that fails the right answer shows up there first. One team logged 240 SWE-bench rollouts at 0% and blamed their agent until the gold patches failed too. And I would report coverage, the share of attempts that were validly graded, next to the score.
8. Reproducibility: pin everything the number depends on
Once a run is reliable, you run it again after the next harness change. If the number moves, you need to know whether your change moved it.
The harness alone can do it. Anthropic re-ran older models after fixing a bug in its zoom tool and raising the token limit per turn, and wrote that it had been underreporting OSWorld across its model families. Claude Opus 4.7 went from 78.0% to 82.8% with no change to the model.
The rest of the setup moves too. Between OSWorld 2.0 and 2.1, 174 pull requests to the task files were merged. Simular ran the same agent on two desktop images and got 0.7766 on one and 0.7414 on the other. HAL notes that OpenRouter can serve different quantizations of a model under the same endpoint name, so the same name does not guarantee the same model.
So every run gets a manifest, with a line for each thing that can move the number:
- task and grader revisions
- image digest and website builds
- the model and the provider route that actually served it
- the judge model
- harness commit
- observation and action settings, screen size
- step and time budgets
- resources per desktop
- retry and repeat policy: mean or best of three
A manifest does not make a model deterministic. It tells you what stayed the same when the number moved.
9. Observability: you need to tell slow from stuck
Hours into a run, you want to know which attempts are working, which are slow and which are stuck. The first thing you reach for is a live view of one desktop. Terminal-Bench had one early:
A live view shows one screen. It cannot tell you whether twenty other attempts are stuck in preparation or waiting on the model. A Harbor user spent about two days on a stall like that. Trials sat silent until a 30-minute timeout. The cause was two crashed trials whose 4 GB sandboxes kept running and filled the account's 10 GB memory limit, so every new sandbox was refused and retried without a word.
You need the same visibility after the run, to explain a score. Simular found that OSWorld 2.1's rubric graders return only the total, so a partial score like 0.6969 cannot be explained or re-graded later.
So I would carry one attempt ID from allocation through model calls, desktop actions, grading and cleanup, record how long each phase took, and keep the evidence of failed attempts instead of deleting it.
10. Cost: count cost per valid result
Once the run works every day, it has a bill, and the model is only one line of it. The others are desktops, idle warm capacity, retries, failed preparation, storage, and anything left running after a crash.
Simular published its model spend for the OSWorld 2.1 reference campaign: $2,417.63 over 27,662 gateway calls, about $18 per clean attempt. Desktop hours are not in that number.
Leftovers add up quickly. OSWorld's Docker provider once left a 32 GB volume behind for every task, enough to fill a disk within an hour of a full run, until it was fixed.
At scale, cost decides the method. HAL spent about $40,000 on its rollouts and still had to rely on single runs for most evaluations. Some models they did not run at all:
So I would track total cost divided by validly graded attempts, shown next to coverage. A cheap run that failed early has few valid attempts, so its cost per result goes up. A run that skipped the hard tasks shows up as lower coverage.
A daily run has to hold all ten at once
Every fix in this article is ordinary on its own. The trouble is that each one exposes the next problem, so a daily run has to hold all ten at once. Here is what that takes: eight stations every attempt passes through, and one attempt ID from allocation to cleanup.
You can find each station somewhere today. Parallel desktops, warm pools, retries and dashboards exist in OSWorld's own runner, Harbor, HUD, Cua and the sandbox providers. One setup that holds all of them, on the benchmark's exact package, every day, is still rare. That setup is the hardest benchmark.
What to take away
A benchmark score is the output of an operation. Before you trust one, yours or a lab's, ask how it was run.
Save the table and the manifest. They work for SWE-bench and Terminal-Bench too.
I build harnesses that let agents use computers. If you want the full notes with every source, reply and I will send them.
Originally published on arthurkatcher.com.
Top comments (1)
"One run may not be enough" deserves a number, and your own figures supply it, so here are the two that decide how to read every board in the piece.
The floor on a single run is set by the task list, not by the agent. With 108 tasks and a per-task pass/fail, the 95 % band on one run is about ±9 pp at a 50 % score, ±8.6 pp at 30 %, ±8.1 pp at 76 %. On the 369-task Verified set it is ±5.1 / ±4.7 / ±4.4 pp. That is before any of your ten problems: it is what a perfectly run single pass can still get wrong about the model, by luck of which tasks landed.
What averaging k runs buys, and the part that is easy to get wrong: it shrinks the run-to-run component by √k and the task-list component by nothing, because every run draws the same 108 tasks. Three runs buy 1.7× on one term and 1.0× on the other. That matters for your Anthropic citation: limits that "moved scores by up to 6 points" are a property of the campaign's environment, not of a run — three repeats under the same container limits share that 6-point shift, so the average of three does not touch it. Only varying the thing you are attributing the effect to does, which is the general form of your own point 4.
Every leaderboard comparison is paired, and that is why so few of those 94 single-run entries are comparable. Two entries run the same 108 tasks, so the gap a pair can resolve is about 1.96·√d/n, where d is the number of tasks they disagree on — not the number they got right. At n = 108: disagree on 20 tasks → 8.1 pp, on 30 → 9.9 pp, on 40 → 11.5 pp. Two strong agents on 108 hard desktop tasks do not have near-identical failure sets; tens of tasks differ. So a five-point gap between two single-run entries on OSWorld 2.1 is not a finding, and the same five points on Verified (n = 369) is: at d = 20 there the resolution is 2.4 pp. Your 94-of-111 statistic is really a statement that most published numbers are lower bounds on their own uncertainty, and that it is worse on the 108-task revision than on the 369-task one.
Where I would spend the machine time, since you are right that repeats are the expensive part. For a paired comparison, the tasks both agents score the same contribute exactly zero to the difference — they cancel. Only the discordant set decides the ordering. So the second pass worth buying is not another 400-machine-hour sweep; it is the discordant subset, re-run until the sign of the difference is stable. On a 25-point discordance over 108 tasks that is a fraction of the attempts, it directly measures the run-to-run variance where it matters (inside the disputed set, not averaged over the easy tasks), and it answers the question a leaderboard position is actually claiming. If the re-run flips entries inside that subset, the honest published form is the interval, not the rank.
Assumptions stated so they can be attacked: per-task outcome binary (your grader's "ran, graded, evidence kept" is exactly what makes it binary), tasks independent for the binomial band, and the pairing exact because both entries run the same list. The first is worth checking on the revision you use — if a task's score can be partial, the band narrows and the paired formula changes shape.