Two runs on the same model can end in very different results. What sits around the model decides it: what reaches it in one turn, which tools it can hold, and what it checks before saying done.
I built Aetox harness for Windows and measured it: same model on both sides, tasks and starting files published, judged by hidden tests written before the runs. Here are the numbers, the places it loses, and what it costs in tokens.
What a harness actually changes
Three parts decided this.
The first is what reaches the model in one turn. A harness can either paste everything it knows into the prompt, or map what it knows and send only the points a job touches. I went with the map. A skill's body and a tool's schema are not sent up front; the model sees a shelf of names and pays for a body only when it opens one. Fifty skills then cost about what five cost, and adding an MCP server adds names rather than a block of schemas.
The second is what the agent can hold. It drives a real browser with its own tabs and profile, so it fills a form instead of describing one, and it runs the project's own test command before it says done.
The third is memory. Mine lives in a folder of plain .md files: your profile, this machine, each room's notes, one note per day of work, and the routes it has seen you repeat. You can open them, edit them, or delete them, and nothing is trapped in a format only Aetox reads. A way of working is only kept when it repeats across different days, which is the rule that stopped it from deciding who I am after one conversation.
The numbers
All same-model, three rounds each, judged by hidden tests, with no user settings on any harness.
| Suite | What it measures | Aetox (two OpenAI models) | The four other harnesses |
|---|---|---|---|
CWEval func-sec@1
|
functions that both work and survive the attack tests, of 15 | 93.3% and 97.8% | 68.9% to 77.8% |
| SlopCodeBench, tests passed | six multi-turn tasks, checked so old code does not break | 91.6% and 95.2% | 90.8% to 92.8% |
| SlopCodeBench, checkpoints | checkpoints whose tests all pass, of 14 | 4.0 and 7.0 | 3.7 to 6.7 |
| SWE-bench Multilingual (caddy-6350) | one real bug from a public repo, official grading | 1/3 and 2/3 rounds | best of the four was 3/3 |
| Web-Bench | tasks whose Playwright spec fully passes, after 6 tasks | 5.3/6 on both | 5.3 to 5.7 |
Two things about that table that I would want to know if I were reading it.
The CWEval lead comes from a security skill that ships switched on. With the skill off, Aetox lands in the same range as everyone else. That is a real advantage, and it is also a specific one: it is the skill doing the work, not magic in the harness.
The SWE-bench row is a loss and I left it in. Two of three rounds against three of three for the best of the four. Rounds swing 1.5 to 2.5 points on this kind of suite, so a gap of a few points reads as a tie, but this one is not a tie.
What it costs
Total input on some long coding jobs runs 1.1 to 8.5 times what the other harnesses used. About 91% of that is cached input, which is billed several times cheaper, and the part that actually costs money, uncached input plus output, differs by 1.1 to 2.1 times. Ordinary jobs do not show this: following an instruction, adding an MCP server, or installing a batch of skills is handled by the same mechanism and comes out cheap.
If you want to cut it, cut it by picking a model that fits the job.
Before you try it
Windows only. The screen, the browser and the computer-use layer are all Win32. Free to use, not free to run: you need a key for a provider, or a local model through Ollama or LM Studio. Current version is 1.10.31.
Any model works, and that is deliberate. OpenAI, Anthropic, Gemini, DeepSeek, Qwen, Groq, xAI, Mistral, OpenRouter, GitHub Copilot, plus local servers. Switch mid-chat and the context comes with you.
How to check it yourself
Everything above is published, prompts and tests included, in the reports folder of the repo: https://github.com/Mikedev115/Aetox-Harness/tree/main/docs/reports
The tasks, the starting files and the hidden tests are there for the coding suites, along with every round. Run them on your machine and on your model. If you get a different number, I want to know which suite and which round, because that is the kind of thing I can actually fix.
Top comments (0)