DeepSWE's harness comparison made me curious, so I tried mini-swe-agent myself on a matched set of debugging tasks with GPT-5.6 SOL at High reasoning.
The current numbers surprised me:
- Codex CLI High: 151.91M tokens, 60% pass rate
- plain mini-swe-agent: 70.33M tokens, 67%
- the same mini harness plus macro execution, without changing the prompt: 60.59M tokens, 78%
So plain mini-swe-agent used about 54% fewer tokens than Codex and scored 7 percentage points higher. In the smaller ablation, changing only execution cut another ~14% of tokens and added 11 points.
That does not prove it is universally better. It is one matched benchmark slice, and DeepSWE itself notes that prompt tuning may explain part of the harness gap. Still, for debugging, the very small “bash + linear history” setup looks unusually strong.
Has anyone used it for day-to-day bug fixing or feature work? What breaks first outside a benchmark: repository setup, long-running tests, permissions, or patch quality?
- My runs: https://turaai.net/benchmark
- mini-swe-agent: https://github.com/swe-agent/mini-swe-agent
- DeepSWE's explanation: https://deepswe.datacurve.ai/blog/deepswe
Disclosure: I maintain Tura and ran the benchmark above.
Top comments (1)
The first thing that breaks for me is usually not patch quality. It is the repo boundary around the patch: setup that only exists in someone’s shell history, long tests that need a timeout policy, and permissions that were fine for a human but too wide for a loop that can keep trying.I like mini harnesses for this reason. The transcript is small enough to inspect when it goes wrong. Macro execution sounds useful if it removes command noise without hiding the actual state changes. Did you see the gain mostly from fewer shell turns, or from fewer bad intermediate decisions?