Benchmarking IBM Bob on SWEBench-Verified

We pulled real bug reports from Django, scikit-learn, sympy and nine other projects you've probably shipped to production and handed each one to the agent at the exact commit before the fix landed with the project's own test suite as the judge.

Bob resolved 303 of 500.

In this blog post, we will explore what it got right, what it cost, where it struggled, and how to run the same evaluation yourself.

Resolved          303 / 500  ██████████████░░░░░░░░  60.6%
Median cost       $1.05 per task
Median latency    209s  (3.5 minutes)
Total             $775.05, 37.6 agent-hours, one pass, no retries

---

What this benchmark actually is

SWE-bench Verified is 500 real issues from twelve mature Python projects. Each instance gives the agent a genuine bug report and the repo checked out at the commit before the fix. A coding agent (Claude Code, Cursor, Codex, Bob, etc.) then writes a patch and the project's test suite is run to see if it passed or failed.

Scoring has no judgment calls in it, which is why the benchmark is trusted:

1. Apply the agent's patch to a clean checkout
2. Run FAIL_TO_PASS (the tests the fix must make pass)
3. Run PASS_TO_PASS (the tests it must not break)
4. Both green, or it doesn't count

In essense, it's the same bar a maintainer would apply to your PR.

"Verified" means all 500 were human-screened by OpenAI to confirm the issue is solvable from its description alone, meaning a failure is when the agent missed something, not that the task was impossible. The agent also never sees the real fix or the tests. We enforced that with a field whitelist that throws if anything leaks.

So 60.6% means: given a real bug report and an unfamiliar codebase, Bob shipped a patch that passed the project's actual test suite three times out of five autonomously.

Where Bob is strongest

Interestingly enough, we found that performance varies a lot depending on the codebase:

Project	Resolved	Tasks	Rate
scikit-learn	27	32	84.4%
pytest	14	19	73.7%
xarray	16	22	72.7%
Django	153	231	66.2%
sympy	41	75	54.7%
astropy	12	22	54.5%
pylint	5	10	50.0%
requests	4	8	50.0%
Sphinx	18	44	40.9%
matplotlib	12	34	35.3%
seaborn	0	2	0.0%
Flask	1	1	100%

_{seaborn and Flask are shown for completeness — at n=2 and n=1 they carry no signal in either direction.}

Bob was five for six on scikit-learn, with Django taking up 46% of the benchmark. As the closest proxy to a large mature web codebase, it landed at 66.2% well above Bob's own overall average.

The pattern is intuitive once you see it. Bob is strong where a fix is localized and logic-driven: a wrong comparison, a missing edge case, an off-by-one in a well-tested module. It struggles where correctness lives in rendered output, like with matplotlib and Sphinx. They sit at the bottom for the same reason: they're painful for humans, because judging the fix means reasoning about a plot or a docs build you can't see from source.

To put it plainly, library code with real test coverage is Bob's home turf.

The surprising economics

Here's a table that caught us by surprise:

Metric	Value
Median per task	$1.05
Median when it succeeds	$0.75
Mean per task	$1.58
Cost per resolved issue	$2.56
Median latency	209s
p90 latency	499s

A typical solved issue cost $0.75 and 3.5 minutes. Set that against the cost of a developer context-switching onto an unfamiliar bug and the math gets interesting fast, even counting the ones Bob misses.

Under the hood, the efficiency comes from caching. Across the full run:

Total tokens        388.4M
Served from cache   94.1%  of all prompt tokens
Output tokens       5.5M   (~11k per task)
Effective rate      $2.00 per million tokens

94.1% cache hit rate is why a 790k-token average task costs about a dollar. Bob re-reads a lot of context as it works through a repo, and almost none of it is billed at full freight.

One more pattern worth knowing:

Successes are cheaper than failures means $1.14 vs $2.29. Bob moves fast through problems it understands and grinds on ones it doesn't. That means a per-task cap (--max-coins) trims the expensive tail without touching your median success. Our priciest single task hit $15 and still failed. We ran uncapped on purpose, to measure the real distribution. In practice you'd avoid this.

Running it yourself

Bob is genuinely pleasant to automate. One command, fully unattended:

bob --yolo --trust --accept-license -o json "Fix the bug described in ISSUE.md"

Two things that will save you an afternoon:

--accept-license is mandatory for unattended runs. Without it Bob waits on a license prompt, and a headless job sits there quietly burning its wall clock until it times out. Nothing in the logs tells you why.

Use an API key in containers. The interactive SSO credential lives in your OS keychain and does not follow you into Docker. Set BOBSHELL_API_KEY and pass it through:

docker run --rm --env-file .env -v "$PWD:/work" -w /work my-image \
  bob --yolo --trust --accept-license -o json "$(cat task.txt)"

The -o json envelope is the nicest part of the integration. You get native cost accounting per session, token counts, per-tool call statistics, and a decision ledger without any required wrapper:

{
  "tools": {
    "totalCalls": 32,
    "totalSuccess": 29,
    "totalFail": 3,
    "totalDecisions": {
      "accept": 0,
      "reject": 0,
      "modify": 0,
      "auto_accept": 30
    }
  },
  "files": { "totalLinesAdded": 205, "totalLinesRemoved": 10 },
  "budgetSpend": 358.02,
  "maxBudget": 500,
  "sessionCost": 1.93
}

That sessionCost field is why every dollar figure in this post exists. Most agents make you reconstruct spend from token counts and a pricing page.

Three things to do in production

1. Assert the patch is non-empty. In 16 of 500 tasks (3.2%), Bob exited cleanly with a 0 status code, real work performed, real time spent... and produced no diff. If you're wiring Bob into CI, at least at the time of writing, check patch length rather than exit code.

[ -s patch.diff ] || { echo "no changes produced"; exit 1; }

2. Run in a clean tree. A few patches picked up working-tree artifacts: build output, caches, and in one case an entire build/lib/ copy that made the diff unapplicable. Never cost us a passing test (every patch that reached the test runner applied cleanly, 483 of 483), but it's the kind of thing you'd send back in review. A clean checkout, or filtering the diff to your source paths, handles it.

3. Cap the spend. See the cost asymmetry above. --max-coins is your friend.

The enterprise angle (what we're measuring next)

Bob's pitch is governed, enterprise-grade AI, and the primitives for that showed up throughout this run: hard budget ceilings enforced server-side, full token and cost accounting in every response, and a tool-call decision ledger that records whether each action was accepted, rejected, modified, or auto-approved.

Note what our run looks like through that lens: auto_accept: 30, accept: 0, reject: 0. We ran it in yolo mode to measure raw capability with no human in the loop. That is the opposite of how you'd deploy this in an enterprise, and it's why this post makes no security claim. We measured what Bob can do, not how safely it does it.

That's the next experiment. The most interesting threat for a coding agent isn't necessarily jailbreak but instead is indirect prompt injection: a malicious instruction hidden in the issue text, a README, or a dependency, arriving through exactly the workflow we just benchmarked. Recent work puts published CLI agents at 41–79% attack success on issue-delivered injections, so that's going to be what we measure against.

We'll run Bob against it and publish the numbers the same way we did here, no matter what they say: hopefully, Bob comes out significantly ahead of the curve.

How we measured this

We ran one complete pass over all 500 issues, scored by the official SWEBench harness with zero modifications.

Reproduction tuple:

IBM Bob (BobShell), default multi-model routing (tier `premium`, 490/490 runs) · `swebench` 4.1.0 @ `726c5461e2ef52d83cf1ea2107870a8bb3328d57`, unmodified · `SWE-bench/SWE-bench_Verified` split `test`, n=500 · pass@1, K=1 seed · 1800s wall-clock cap, no per-task cost cap · `linux/amd64` under QEMU emulation on Apple Silicon (6 CPU, 12 GB) · 2026-08-11 → 2026-08-14.

Here's the final score:

Outcome	n
Resolved	303
Tests failed	180
Empty patch	16
Unapplicable patch	1
Total	500 ✓

Every instance was attempted exactly once. Nothing was retried. Re-rolling failures until they pass is the easiest way to accidentally inflate a benchmark (read: benchmaxx) so we deliberately didn't do this and ran everything once.

# generate patches
python runners/swebench_generate.py \
  --out results/full/bob-default-routing/seed1 \
  --config-name bob-default-routing --env-file .env --workers 3 --timeout-s 1800

# score with the official harness
python eval/run_swebench_eval_batched.py \
  --predictions results/full/bob-default-routing/seed1/predictions.jsonl \
  --run-id verified_seed1 \
  --out-dir results/full/bob-default-routing/seed1/eval_verified_seed1

Caveats

We'd rather you trust this number than be impressed by it.

• We handed Bob a slightly hostile environment. SWE-bench images are x86-only and we ran on Apple Silicon, so everything executed under QEMU emulation at roughly 3–5× slowdown, which inflates timeouts. It also caused six tasks where Bob's process crashed at startup before doing any work at all: our infrastructure, not the agent, and re-running one worked fine. We counted all six against Bob anyway. Scoring only the 494 tasks it genuinely attempted gives 61.3%; we publish 60.6%.
• One seed. We measured pass@1 once. Agents are nondeterministic, so a rerun lands somewhere slightly different. The confidence interval covers task sampling, not run-to-run variation. We estimate the real spread is wider. We're measuring that now.
• Where it sits. Bob's 60.6% is ahead of 61% of the 134 scored entries in the official experiments repo (median 53.0%); the frontier is 79.2%, so there's headroom. Hold any ranking loosely: these are agent + scaffold scores and our scaffold isn't anyone else's, so gaps of a few points aren't meaningful. Bob also can't formally appear on that leaderboard: since November 2025, it accepts only academic submissions with a published paper so this is our arithmetic, not an official placement.
• Contamination is real. Verified was published in 2024 from public GitHub history, so parts of it plausibly sit in model training data. That affects every entry on the board, not just Bob, but treat any Verified score as a sanity check rather than a definitive ranking.
• This measures a router, not a model. Bob dispatches across multiple providers with its own policy and doesn't name the model behind a given task. That's our product decision to make things easier for developers (you don't have to think about models and such) but it means this is a measurement of the product on the dates we ran it and the models of the day as well.

The takeaway

Bob closes three out of five real GitHub issues unattended and autonomously, for about a dollar and three minutes each, with no model selection to think about and native cost accounting in every response. It's strongest on library and framework code with good test coverage, and weakest where correctness lives in rendered output.

That's a genuinely useful tool for triage, first-pass fixes, and burning down backlog provided you treat it like any other contributor: check the diff, assert it changed something, and cap the spend.