Part 2 of the Broken Campus series. Part 1 built a benchmark that scores agents on how cleanly they fail. This is the head-to-head — and I'll be honest up front: for most of this post the news is bad for anyone hoping a single model solves it. It's going somewhere, but the good part is earned, not up front.
Disclosure: I'm B Torkian, an NVIDIA Developer Champion, benchmarking NVIDIA models among others. That's exactly why the scoring is deterministic, the harness is public, and I ran an adversarial audit against my own numbers — a hidden thumb on the scale would be disqualifying.
Quick catch-up if you're landing here first. Broken Campus is a small agent benchmark over a fictional university, Northwind — every fact invented, so nothing lives in any training set. Two things get scored, deterministically: did the model refuse to make something up when the fact was gone, and did it return valid JSON your parser can accept. Part 1's punchline was that on the plain "can you answer this?" axis, everyone is basically perfect. The differences only show up when the agent shouldn't answer, or when your parser needs a schema.
So I ran eight models across three tiers — five seeds each, 75 format runs per model, same prompt, same tools, same scoring code. I went in wanting the same thing you probably want: one model I could point the whole agent at and stop thinking about it. Here's what came back, strongest signal first — and I'll tell you now, that clean single answer never arrives. Watch it not arrive; that's the point.
The scorecard
Three tiers: the open NVIDIA models — NVIDIA Nemotron 3.5 Lightning, Nemotron 3 Super, and the reliability-tuned Nemotron 3 Ultra ("Lightning", "N3-Super", "N3-Ultra" for short below) — the budget-closed tier (GPT-4o-mini, Haiku), and the closed frontier (GPT-5.5, Opus). One open model, super-49b, sits in as a prior-generation open reference — not a head-to-head contender, just a baseline for where NVIDIA's open line was.
| Model | Tier | Cost / correct ↓ | Mean latency (ms) ↓ | Format-fail % ↓ | First-pass JSON % ↑ | Abstention-safe % (directional) |
|---|---|---|---|---|---|---|
| Lightning | open (efficient) | $0.0004 | 2,098 (fastest) | 24 | 17 | 37 [19–59] |
| super-49b | open (prior-gen) | $0.00035 | 11,676 | 0 | 100 | 40 [23–59] |
| GPT-4o-mini | budget-closed | $0.00037 | 3,346 | 8 | 92 | 60 [41–77] |
| Haiku | budget-closed | $0.0054 | 3,825 | 5 | 95 | 60 [41–77] |
| N3-Super | open | unrecorded | 12,870 | 32 | 68 | 42 [19–68] |
| Nemotron Ultra | open (reliability) | unrecorded | 13,297 | 12 | 87 | 100 [83–100] † |
| GPT-5.5 | closed frontier | $0.0156 | 5,365 | 0 | 100 | 64 [45–80] |
| Opus | closed frontier | $0.031 | 9,679 | 0 | 100 | 68 [48–83] |
Abstention shows the point value with its 95% interval on every row — the wide, overlapping intervals in that last column are the whole reason it's labeled "directional" (the caveat section explains why: effective n ≈ 5), and I'd rather you see them than hide them. † Ultra's 100 is the top of a cluster, not a ranking: its interval touches Opus's, so read it as "at least as safe as Opus," not "beats Opus." More on that below.
Two cost cells read unrecorded on purpose: the N3-Super and Ultra runs came off an early-access endpoint with no published price, so I won't invent a cost-per-correct for them. Better a blank than a number I can't defend. (super-49b has a published price, so its cost stays in.)
First, what didn't move. On the plain "the fact is present, go answer it" cases, every model passed clean at 100%, false-refused at 0%, survived the poison-context probe at 100%, and recovered from an injected tool fault at ~100% (super-49b the lone dip at 93%). Multi-turn memory held at 100% everywhere except N3-Super, which collapsed to 56% — a second failure mode, and a disqualifier for any memory-dependent step. (Lightning, notably, holds at 100% here, so that collapse is an N3-Super problem, not a fast-open-model law.) But if your agent's job is simply to answer when it has the answer, all eight can do it. That's not where the money is — and it's not where anyone gets hurt.
The findings — and why they get discouraging fast
Cost, speed, and format are strong enough to bet on. Abstention I'll hedge out loud, in the caveat section, because it deserves the room. They're in order below, and each one is built the same way: a strong claim in one column, clawed straight back in the next.
Lightning is the cheapest and fastest, by a wide margin. At $0.0004 per correct answer, it comes out roughly 78× cheaper per correct answer than Opus and about 39× cheaper than GPT-5.5. On wall-clock it's the fastest thing in the table at 2,098 ms mean, against 5,365 ms for GPT-5.5 and 9,679 ms for Opus. The heavy open models sit at the slow end — Ultra at 13,297 ms is the slowest of the eight. That's the raw cost-and-speed gap, and it's not close. If cost and speed were the whole story, you'd close this tab and ship Lightning everywhere.
Format discipline is where the cheap path starts to hurt — a large effect, not a coin flip. This is the strongest signal in the whole run: 75 runs per model, big gaps, real daylight. The closed frontier — GPT-5.5 and Opus — is flawless: 0% format-fail, 100% first-pass valid JSON. The budget-closed tier is nearly as clean — GPT-4o-mini 8% fail / 92% first-pass, Haiku 5% / 95%, which puts the whole closed side in a tight 0–8% format-fail band. Ultra is the standout on the open side at 87% first-pass valid, 12% format-fail. Then the fast open models fall off a cliff: N3-Super is 68% first-pass valid (32% format-fail), and Lightning is 17% first-pass valid, with a 24% format-fail rate — a fast-open band of 24–32% format-fail. A gap of 0–8% against 24–32% is far too wide to be a five-case artifact; I'll come back to why in the caveat, but the effect size alone settles it. And there goes the clean win: the cheapest, fastest model in the field is also the one whose output your parser rejects most.
One measurement note so the two format numbers don't confuse you: first-pass JSON (parses on the first try) and format-fail (never parses, even after the harness's one retry) are scored on different case slices, so Lightning's 17% and 24% don't sum to anything. Roughly one in four of Lightning's responses never comes back as JSON your parser can accept, even after that retry. In a demo you never notice. In a pipeline that expects a schema, that's a quarter of your traffic your parser rejects — every retry, alert, and dropped request that a quarter of failed traffic drags behind it, multiplied by production volume.
And here's the detail that keeps the story honest: look at the prior-gen super-49b row — 0% format-fail, 100% first-pass valid, matching the frontier exactly. The format gap isn't a trait of NVIDIA models generally. It's a trait of the newest, fastest, cheapest open models specifically — the ones tuned to be tiny and quick traded away some of the JSON-shaping the older, heavier model had nailed. (That's also why I put super-49b's format cells in the table but keep it out of the head-to-head: it's a reference point for the trend, not a competitor.)
Abstention is where the cheap path hurts most — but only in coarse shape. This is the "did it refuse to make something up when the fact was gone" axis, and it's the one that gets an agent in real trouble: a confident wrong answer is worse than no answer. It sorts into two clusters, not a clean ranking. At the safe end: Ultra 100% [83–100] — the only model to abstain on every single absent probe — sitting with the closed frontier (Opus 68% [48–83], GPT-5.5 64% [45–80]) and the budget-closed tier (60% [41–77]). At the risky end: the fast and heavy open models bunched together — super-49b 40% [23–59], N3-Super 42% [19–68], and Lightning 37% [19–59]. Two clusters, a real gap between them, and — as those overlapping intervals show — no reliable order inside either one.
So look at where that leaves Lightning: cheapest, fastest, and in the bottom cluster on the one axis where being wrong is expensive. The cheap path doesn't just cost you parseable output — it costs you caution.
The mandatory caveat, said plainly
Before I let that despair stand, it has to survive one test: is any of it even real, given how few cases this is?
This is directional, not definitive. The seeds are near-deterministic, so the effective n is closer to five distinct cases than to 75 independent trials. It's one small, fictional knowledge base. And every latency number here came off NVIDIA's free developer endpoint, not a priced production tier, so read those milliseconds as relative, not as an SLA.
That same effective-n ≈ 5 is why format holds and abstention doesn't. On format, the gap is a matter of effect size: 0–8% format-fail for the closed side against 24–32% for the fast open models. A gap that wide doesn't close on five cases — the effect is simply too big for the sample size to erase, and that's the argument, not any significance test. On abstention, the effect is a few points and the intervals do the opposite: Ultra's 100 and Opus's 68 intervals touch, and a case-level Fisher test comes out non-significant (p ≈ 0.48). So the honest claim there is only: Ultra was at least as safe as Opus, and no cleaner than that — and the 40-probe deep dive in the next section confirms exactly this, gap and all.
So: cost, speed, and format discipline are strong — large effects, lots of runs, gaps too wide for five cases to explain. The fine-grained reliability rankings are suggestive and no more.
Going deeper: what happens when 5 probes become 40
I could have left it at "directional" and moved on. But the abstention gap was the most interesting number in the table and the least trustworthy, so I did the obvious thing: I stopped hand-waving about effective-n and ran 40 distinct unanswerable probes instead of 5 — deletions (the fact was removed), adjacent-but-nonexistent questions (a board meeting, a membership fee, a room number that never existed), and false-premise traps ("now that it moved to Mondays…"). Same knowledge base, same agent, same deterministic scoring, parsed-only. Eight times the probes on the exact axis I was least sure about.
The dramatic gap did not survive.
At n=40 everyone clusters between 75% and 83%, and the intervals overlap so heavily there's no order left inside the group: N3-Ultra 83% [67–92], GPT-4o-mini 81%, GPT-5.5 78%, Opus 75%. (One honest wrinkle the chart makes explicit: because abstention is scored only on parseable output, the two fast open models are judged on fewer probes — Lightning on 21 of 40, N3-Super on just 9 — since the rest format-failed. That thin denominator is itself the finding: you can't even evaluate their judgment on the outputs your parser rejects.) The five-probe table had Ultra at a perfect 100 and Opus at 68 — a 32-point chasm. Eight times the probes erased it. That is small-sample noise resolving, live, and it is the single best argument for why you should distrust a benchmark that reports one number without an interval — including, until this section, parts of mine.
Read carefully, this is good news for NVIDIA, just not the headline a marketer would have written. The honest, rigorous claim is stronger than "wins": NVIDIA's open, self-hostable Ultra matches the closed frontier's caution — statistically tied with GPT-5.5 and Opus at knowing when to shut up — which means you can get frontier-grade judgment without renting a frontier model. It's slower (recall Ultra is the slowest in the field), but the caution is real and it's open.
And the deep run sharpened the thing that is a real, separating effect: format discipline. On these harder probes the fast open models don't just abstain less — they fail to return usable output at all far more often (Lightning 48% format-fail, N3-Super 78%, against 0–10% for the closed tier). That's not a rounding difference; it's the fast-open tax showing up under load. Which is the whole setup for Part 3: if the cheap models you'd actually want to run are the ones that most often hand your parser garbage, you don't pick one model — you route. (I'm keeping both tables in the open on purpose: the 5-probe head-to-head and the 40-probe deep dive. The small one was directional; the big one is the rigorous follow-up; they point the same way, and showing the correction is the point.)
So which one do I pick?
You don't. That's the actual finding. There is no single-model answer, and here's why that's not a cop-out. One prescriptive sentence per model, no re-deriving the numbers above:
Lightning is the workhorse — put it where volume lives and correctness is checkable, and keep it away from steps where a confident wrong answer is expensive: it sits in the low-abstention cluster with the other fast open models, well below the closed frontier and Ultra.
Nemotron Ultra is the reliability specialist — reserve it for the steps where being wrong costs you, and accept that you're paying for it in latency, since it's the slowest model in the table.
On the closed frontier, GPT-5.5 is the default of the two — roughly half Opus's cost, noticeably faster (5,365 ms vs 9,679 ms), and it handles format and judgment both. Step up to Opus only for its edge on abstention. Either way you're paying 39–78× Lightning's rate to not think about it.
There is no row in this table that's best at everything. There's a cheap-and-fast one with a format problem, a careful one that's slow, and a balanced one that's expensive. Pick any single model and you are knowingly accepting one of those failures for your whole pipeline.
You were asking the wrong question
The question was never "which model?" It was "why am I sending all my traffic to one model at all?" Every "but it's slow" and "but its format breaks" objection above assumes you must pick a single row and live with its one failure. Drop that assumption and the whole table changes shape.
Start with the mechanics, because they're almost boring. The instrument is NeMo Switchyard — NVIDIA's open router — and one number in this very table hands you the first, cleanest escalation signal nearly free: that 24% format-fail rate is Lightning's, it's deterministic, it's cheap to detect, and it's therefore a perfect trigger. Catch just those failures, re-run them on a bigger model, and you buy back the parseable output Lightning drops — for a fraction of what running the frontier everywhere would cost. That's Part 3: the routed numbers, and the money table.
It's honest about the catch, too — format-fail routing fixes format, not judgment. The same trigger barely moves abstention, so reliability needs a second lever keyed to task type: send the steps where being wrong is expensive to a model in the safe cluster. And here's the part the head-to-head quietly handed you — that safe cluster isn't closed-only. Look back at the abstention column and notice who's up there with Opus and GPT-5.5: Nemotron Ultra, at 100% [83–100] — at least as safe as Opus on the axis that matters most for agents, knowing when not to answer. Remember super-49b matching the frontier on format, too? Same lesson, different axis. So this isn't "Ultra wins everything" — it's slower, and a touch behind closed on format. It's narrower and better than that: on the one axis where a wrong answer is expensive, you no longer have to leave the open ecosystem to get frontier-grade safety.
Which means the second lever costs you nothing extra to own. Ultra is open-weight and self-hostable, at open-model economics — the caution you were about to pay closed-frontier prices for is available in a model you can run yourself. Cheap by default, careful when it counts, and both the default and the safety net are yours to host.
So look at that compromise table one more time. It was a wall of tradeoffs a page ago — cheap-but-broken, careful-but-slow, balanced-but-expensive. Now it's a set of dials you control: Lightning's cheap, fast bulk; Ultra's caution where it counts; the frontier if and when you want it. The good answer was never a model. It's the routing — and that's Part 3.
B Torkian is an NVIDIA Developer Champion. Broken Campus is open — the harness and scoring are frozen and public at github.com/torkian/broken-campus, so you can reproduce the benchmark on the models you can reach. 5 seeds, 8 models, one synthetic domain.
Top comments (0)