I built Stepswitch, a free, MIT-licensed orchestrator for coding agents. It splits a task into steps and sends each one to the model that suits it, local first. Every change has to pass your tests and a review by a model from a different family, and every number in the README is generated from evidence files that ssw prove re-creates.
To see how it holds up on real issues, I ran it against mini-SWE-agent on SWE-bench Verified Mini: 50 django and sphinx issues, graded by the official harness on tests the agent never sees. Both agents used gpt-oss-120b on one workstation's Ollama, one run each, with the claims registered before the run.
The results
| Stepswitch | mini-SWE-agent | |
|---|---|---|
| Resolved (49 compared) | 24 (49.0%) | 7 (14.3%) |
| Runs ended by a server error | 1 | 24 |
| Resolved where neither run hit a server error (24 instances) | 7 | 6 |
| Tokens per instance, counting cached | 1.61M | 2.37M |
| Uncached tokens per instance | 142k | 46k |
| Minutes per instance | 15.5 | 53.5 |
One instance is left out for both because mini-SWE-agent's run hit the four-hour cap.
The headline gap looks big, but the third row tells the real story: most of it is error handling, not smarts.
The error that decided it
When gpt-oss writes a tool call that Ollama can't parse, Ollama answers with HTTP 500 ("error parsing tool call"). That one error ended 24 of mini-SWE-agent's 49 runs.
Two things I learned about it:
- Retrying the same request doesn't help. It fails the same way.
- Asking for the step again in plain text works. Describe the tools in the prompt and ask for a JSON reply. Stepswitch met the error 23 times and asked again this way; only one of its runs ended on a server error.
If you run any agent on gpt-oss through Ollama, catch this error and re-ask, or expect some runs to end early.
Other things I learned running gpt-oss locally
-
Set
num_ctxon every Ollama request. qwen2.5:32b at the server default of 32k answered with nothing at all; at 16k it was fine. - Avoid extra user turns after tool calls. With gpt-oss's chat template, a new user turn after tool calls drops the earlier reasoning from the prompt, so the prompt cache misses from the first assistant turn. One "please call finish" nudge per task cost 35% of my uncached input.
- Accept `* Begin Patch` patches.** gpt-oss likes to write them whatever your tool schema says.
- Watch cached and uncached tokens separately. Stepswitch used 32% fewer tokens counting cached ones (95% interval 2% to 49%), but about 3x the uncached tokens. mini-SWE-agent's append-only transcript caches almost perfectly.
Small suites vs Claude Code
On my own small suites (42 task runs), I also compared the local setup (gpt-oss:120b for most steps, qwen2.5:32b as the reviewer) with Claude Code:
| Passed | Tokens per task run | Uncached | Hosted tokens | Minutes | |
|---|---|---|---|---|---|
| Stepswitch, local | 41/42 | 24.8k | 7.0k | 0 | 0.7 |
Claude Code via ssw proxy, local |
42/42 | 173k | 48.2k | 0 | 2.8 |
| Claude Code, Sonnet 5.5 | 42/42 | 73.4k | 7.8k | all | 0.3 |
| Claude Code, Opus 5.5 | 42/42 | 93.9k | 9.4k | all | 0.4 |
These suites are small and easy, so treat them as a sanity check, not a benchmark.
What isn't good yet
- In 11 of 25 django patches, Stepswitch also changed the template loader to make a test pass that fails only because the containers run as root. No grade changed, but I wouldn't merge those patches. It's the top open issue.
- One run per setup, on one machine, and the local runs are slower than hosted models.
- Three claims failed their confirmatory runs. They're withdrawn in the README with the reason, not reworded.
Check it yourself
Predictions are committed in the harness's format, so you can grade them yourself, and ssw prove re-runs every claim's test. CI runs those checks on Linux, macOS and Windows.
Repo: https://github.com/sarthak-1111/stepswitch
Feedback welcome, especially from anyone running agents on local models.
Top comments (0)