Broken Campus, Part 3 of 3. This is where the descent pays off. (Disclosure: I'm B Torkian, an NVIDIA Developer Champion; the harness is public and deterministically scored, and the money table below is reproducible from the repo — verify it, don't trust me.) Part 2 closed on one line — the good answer was never a model, it's the routing — and this is where that cashes out. You spent two parts watching every model fail at something an agent can't afford to fail at, with no single one that wins, and this is where that stops being a wall. You stop picking a model and start building a system of them, and you get the thing every team actually wants: fast, cheap, and careful, at the same time.
Here's where the descent left us: the cheapest, fastest model in the field, NVIDIA's Nemotron 3.5 Lightning ("Lightning"), trades away the three things an agent most needs — it returns malformed output 24% of the time, lands first-pass JSON only 17% of the time, and on the questions it should refuse it abstains just 37% of the time. Pick one model and there was no happy answer.
Here's the turn. You were never supposed to pick one. NVIDIA shipped the pieces to combine them — open weights plus an open router — and when you do, the discouraging data from Parts 1 and 2 stops being a wall and becomes the map. Lightning's biggest weakness turns into the cheapest routing signal in your entire stack.
Format-fail is a tripwire you don't have to build
When a model returns malformed output — truncated JSON, prose where you asked for a schema, a missing required key — you don't need a classifier to notice. Your parser already noticed. It threw. That exception is a signal, and it has a property most routing signals would kill for: it is deterministic and free. No second model call to decide "was this good enough?" No confidence threshold to tune. The output either parses against your schema or it doesn't, and your code already knows which.
That reframes Lightning's biggest weakness as an asset. Lightning fails format on 24% of calls — among the highest format-fail rates in the field. For most of a benchmark that's a mark against it. For a router, it's a built-in tripwire: 24% of the time, the cheap model tells you — mechanically, in the shape of a stack trace — "escalate me." The other 76% of the time it returns valid output. And across all of Lightning's correct answers, it costs 1/78th of Opus per correct answer (against gpt-5.5 the gap is narrower, roughly 39×) — the cost-per-correct ratio that made it the workhorse in Part 2.
So the policy writes itself. Run every request on Lightning. If the output parses, keep it. If it doesn't, re-run that single request on a bigger model. You escalate only the fraction that failed — you never pay premium prices for the majority that already worked.
The money table
Here is what that policy does, re-routed from the actual per-case Lightning outputs:
| System | Valid output | Cost / correct | Latency (mean) |
|---|---|---|---|
| Lightning only | 76% | $0.0004 | 2,098ms |
| gpt-5.5 everywhere | ~100% (measured) | $0.0156 | 5,365ms |
| Opus everywhere | ~100% (measured) | $0.031 | 9,679ms |
| Lightning → gpt-5.5 on format-fail | ~95–100% (imputed) | $0.0039 | 3,397ms |
| Lightning → Opus on format-fail | ~95–100% (imputed) | $0.010 | ~4,800ms |
Escalating only the ~24% that failed lifts valid output from 76% to near-100%, and the routed system costs $0.0039 per correct answer — about 4× cheaper than running gpt-5.5 on everything, or $0.010 escalating to Opus, roughly 3× cheaper than running Opus everywhere. You eat the escalation hop on one call in four, so the routed mean latency lands at 3,397ms, about 1.6× faster than gpt-5.5-everywhere (5,365ms). (The Opus-routed row lands near 4,800ms — well under Opus-everywhere's 9,679ms, if you'd rather escalate to Opus.) You summon the expensive model only for the fraction of requests the cheap one visibly botched.
Before you bank that, the table earns its trust from what it doesn't claim — two honesty notes, then the payoff. First, exactly one figure here is projected rather than measured: the escalation leg is an imputation, not a fresh run. I re-routed Lightning's real per-case outputs and, for the calls that failed format, credited the frontier with its own measured average format-pass rate rather than re-running it against Lightning's specific failing cases. Those ~18 failing runs (24% of 75) may be the hardest slice to format, not a random one, so the frontier could do a touch worse on them than on average — which is why the routed rows read "~95–100%," not a hard 100%, while the two baseline rows carry a cleaner "~100% (measured)" (gpt-5.5 and Opus each hit 0.0% format-fail on the full run, a measurement, not a projection). Everything else falls out of the per-case records, not blended headline averages, so you can regenerate it from the frozen scorecard (results/scorecard_multiseed.json) with the routing script in the repo — a "correct" routed answer is a request that ends in valid, parseable output, cost/correct divides total blended cost by that count, and code scores every output, not an LLM judge. Second, the honesty note carried through from Part 1: the 24% escalation fraction is itself a point estimate with a wide interval, so at this sample size the routed system's behavior can hinge on a small number of distinct cases. The shape holds; the exact break-even is yours to measure. (More on the sample-size machinery at the end.)
With that on the table — cheap by default, careful when it counts, and you no longer have to choose. Sit with that for a second, because it's the whole point: the tradeoff you were staring down at the end of Part 2 just dissolved. Not with a bigger model, not with a bigger budget — with a stack trace you were already throwing away.
Fast, cheap, accurate — the one that gets all three
Every team wants the same three things from an agent: fast, cheap, and accurate. The uncomfortable lesson of Parts 1 and 2 is that no single model gives you all three — you get two and pay for the third. Put every option to a plain three-bar test — under 4 seconds, under half a cent per correct answer, at least 95% valid usable output — and only one configuration clears all three:
Read the rows. Lightning alone is fast and cheap but misses accuracy (76% usable). GPT-5.5 and Opus are accurate but fail both speed and cost. The routed system — NVIDIA's Lightning carrying 76% of the traffic, escalation cleaning up the rest — is the only row that passes all three: 3,397 ms, $0.0039 per correct answer, ~100% usable. That is the entire argument for a system of models, in one grid. And the thing that unlocks it is Lightning: without a genuinely fast, genuinely cheap default to carry the bulk, you can't clear the speed and cost bars at all — the two frontier-everywhere rows prove it, stuck at 1/3. ("Accurate" here means valid, usable output — the deterministic axis that breaks pipelines; on the answerable cases every model already lands the correct fact ~100% of the time. Knowing when not to answer is the fourth dimension, and it's the next section.)
The nuance most routing posts skip: format is not judgment
The reason you can trust that win is that I'm about to tell you exactly what it doesn't cover — the good news holds up because it isn't hiding the inconvenient half. Format-fail escalation fixes format. It does not fix judgment.
When Lightning invents a plausible-but-wrong answer to a question it should have refused, that answer is valid JSON, with all the right keys. It parses cleanly. Your tripwire never trips. Format-fail routing catches the fraction of outputs that are broken; it is structurally blind to the outputs that are confidently wrong. A schema check cannot see a well-formed lie — the outputs that put a hallucinated dorm policy or a made-up deadline in front of a user are exactly the ones it waves through.
And this is not a gap you can route your way out of with the parse-check, by construction. On the absent-fact probes — the designed slice where the harness asks for something that genuinely isn't in the knowledge base — Lightning alone abstains only ~37% of the time, and format-fail routing nudges that only from ~37% to ~52%. It can't do more: the tripwire fires on broken shape, and a confident over-answer has perfect shape. That ~15-point move is a side effect — some of Lightning's format-fails happen to land on absent-fact probes, so escalating them lets the target abstain there — not the mechanism doing its job. And at this sample size that ~15 points is roughly one case flipping; it's directional at best, and I would not bank on it. The structural point stands regardless of the exact number: a schema check cannot detect a confident wrong answer, so format routing does not meaningfully move abstention.
And size the fear honestly: Lightning over-answers on roughly 6 in 10 of those absent-fact probes — and format routing barely dents that — but this is on the absent-fact probes specifically, a designed slice, not your traffic mix. How much it costs you in production depends entirely on how much of your traffic is genuinely unanswerable, which is exactly the thing running the harness on your own knowledge base tells you. The residual is real; its blast radius is yours to measure.
So the honest conclusion is two-part:
- Format failures → route on the free, deterministic signal (did it parse?). This is the money table.
- Abstention-critical steps — anywhere a confident wrong answer is expensive — → route on task type, decided before the call. You can't detect these after the fact from output shape, so you send the whole class of request to the careful model up front.
And this is where the second piece of good news lives, because it changed this week: you no longer have to leave the open ecosystem to get frontier-grade caution. On this run the careful tier is Ultra — NVIDIA's bigger open model, and the only model to abstain on every absent probe (100%, CI 83–100). On the axis that matters most for an agent — knowing when not to answer — Ultra is at least as safe as Opus, and it is open-weight and self-hostable, at open-model economics. Say it exactly that carefully: on effective-n≈5 the intervals touch, so treat this as the direction to test, not a settled ranking — Ultra is at best at least as safe as Opus here, not decisively better, and the fast NVIDIA models are the weakest abstainers, not the strongest. But the shape of the relief is real: your escalation target is your call — a bigger NVIDIA open model (Super or Ultra) or, if you'd rather, a closed frontier. You are no longer locked in.
Be clear-eyed about what that second route costs. The parse-check is reactive and nearly free — you already ran Lightning; you only pay the target on the calls that broke. The task-type route is the opposite: it's a proactive bet that spends careful-tier calls on an entire category of traffic, before you know whether any given request needed it. And on this run the careful tier, Ultra, is also the slowest model in the field (13,297ms) with a per-call cost that's unrecorded on the developer endpoint. So the proactive route does not inherit the money table's cheap-and-fast halo — it's a safety decision you pay for in latency, and possibly cost, sized by how much of your traffic is abstention-critical. Quantify that fraction before you commit the budget.
One signal is reactive and nearly free. The other is proactive and worth what it costs when the stakes are real. A real system uses both, and it matters to say plainly which problem each one solves — and which one has a bill you haven't measured yet.
Both signals map onto a real, open router
Those two signals are exactly the two routing modes a production router exposes, which is where NVIDIA's NeMo Switchyard — open, Apache-2.0 — comes in: it sits in front of your providers as an OpenAI-compatible endpoint and decides, per request, which target handles it. The reactive parse-check is a cascade/escalation route; the proactive task-type tag is a classifier/stage route. Both are config blocks instead of hand-written glue — that's the whole pitch. Broken Campus can run through it as a plain OpenAI-compatible proxy, scored by the exact same deterministic code as every model, apples to apples. (The money table above is computed from the per-case records, not read off a live router — Switchyard is how you'd operationalize the shape, not the source of the figures.)
So step back and look at what this router actually buys you — because it's bigger than a cheaper benchmark.
The payoff you can actually deploy: cheaper production and the edge
It makes production dramatically cheaper. Cheap by default, careful only where you've decided it matters: near-100% valid output at roughly a quarter of what running the frontier everywhere costs. That's the answer to Parts 1 and 2's discouragement. You don't sacrifice accuracy to get Lightning's price; you spend the frontier's price only on the slice that earns it.
And it unlocks deployments the frontier simply can't reach. Lightning isn't just cheap in the cloud — it's a 30B-total / 3B-active Mixture-of-Experts model (model card), so only ~3B parameters are active per token and the active footprint is small enough to target a single modern GPU or edge-class hardware rather than a datacenter. That's a class of use case, not a benchmark number I measured, but it's the one that matters here: because the default model in this routed design fits on-device, the exact same architecture serves latency-sensitive, on-device, and robotics/edge workloads — not only the datacenter. A robot on a factory floor, a kiosk with no reliable uplink, an app that can't send every keystroke to a cloud API: they all get the same deal — fast, cheap, local Lightning for the bulk of requests, escalating over the network to a bigger open model or a frontier model only when the parse-check trips or the task type demands care. The cheap-by-default design and the on-device story are the same design. You don't rebuild it to move from cloud to edge.
That's the resolution the series was walking toward. Not "one model finally won" — none did, and I won't pretend otherwise. The win is the system: route, and you get fast, cheap, and careful at the same time, in the cloud and on the edge.
The caveat, stated once and meant
Everything above is directional, not definitive, and I'd rather you know why than trust me. The benchmark's seeds are near-deterministic, so the effective sample is closer to ~5 distinct cases than the 75 runs-per-model headline suggests; the confidence intervals are wide and, if anything, optimistic. That's the machinery behind every hedge in this post — the wide format-fail interval, the imputed escalation leg, the within-noise abstention delta all trace back to it. The knowledge base is one small synthetic domain. The latency figures come from NVIDIA's free developer endpoint, not a priced production tier.
Where I'll stand firmly: cost, speed, and format discipline are large, clean, well-measured effects — the table's shape is real. Where I'll only point a direction: the fine-grained reliability rankings (who abstains a few points better than whom) sit inside overlapping intervals — treat "Lightning < Super < Ultra" as a hypothesis, not a verdict, and note that Ultra is at best at least as safe as Opus, not decisively better. The routing shape — cheap-by-default, escalate-on-a-signal — is the durable takeaway. The exact break-even in your system depends on your traffic mix, and that's precisely what the harness is for.
Run it on your own agent
Get the pieces first — all free/open: try the models hosted on build.nvidia.com (free NVIDIA API key), grab the open weights from huggingface.co/nvidia to self-host, and the router is NeMo Switchyard (Apache-2.0). Then Broken Campus is open and runs on your own keys:
git clone https://github.com/torkian/broken-campus
cd broken-campus
pip install -r requirements.txt
cp .env.example .env # paste your NVIDIA / OpenAI / Anthropic keys
python list_models.py # the exact model IDs your keys can call
python run.py # the full benchmark → results/ + scorecard
You only need keys for the providers you want to test; the rest are skipped. Same prompt, same tools, same scoring code for every model — one minimal harness, apples-to-apples. Swap in your knowledge base and your facts, and you'll get your own over-answer rate, your own format-fail rate, and your own break-even for when escalation beats running the frontier everywhere — with the per-case transcripts so you can see how it failed, not just that it did.
To run the routed variant through Switchyard, add BC_SWITCHYARD=1 python run.py — but that path is the operationalization route, not the source of the headline figures. Those come from the frozen scorecard and the per-case routing script, which run with no proxy at all. Two field notes bit me while wiring the proxy up, so save yourself the afternoon:
-
The classifier target must be a NIM, not an OpenAI model. Switchyard injects a vLLM-specific hint (
chat_template_kwargs={enable_thinking: false}) into every classifier call, and OpenAI endpoints 400 on it. Point the classifier at a fast NIM likemeta/llama-3.1-8b-instruct, which accepts and ignores it. - An "efficient" model is only efficient in fast mode. In one wiring test I left a Nemotron reasoning tier in thinking mode, and it answered through the proxy about 10× slower than Opus did — for a target whose benchmarked latency is ~13s, not the ~23s I was seeing, so this was a config artifact, not a property of the model. I chased it for an afternoon before realizing the "cheap" model was quietly burning a reasoning budget on every call. The lesson holds even though the through-proxy number is anecdotal: routing to a cheap model buys you nothing if fast mode isn't actually on for the target. Confirm it.
Then break it. PRs adding models, domains, and new case categories are the entire point — I'd rather you falsify my numbers than take them on faith.
And this is the note I want to leave you on, because it's the real reason the answer exists at all. Back at the top of Part 1 I asked you to stay with some discouraging data and promised it paid for something — that the destination would land harder as a discovery than a promise. This is the discovery, and here's the part I kept to myself: the pieces that make this system work — the small model that fits on your desk, the bigger open models that reach frontier-grade caution without renting a closed one, and the Apache-2.0 router that ties them together — are all open, and NVIDIA shipped them, open weights and open tooling, in the same week. That's what the ugly data was pointing at the whole time: not a single model that finally wins, but a working developer able to assemble fast-cheap-and-careful out of parts they actually own. It's one of the biggest open-source contributions moving generative AI forward right now, and it's why the honest ending to this series isn't a warning — it's an invitation. Bring your own knowledge base, route it, and ship the thing that used to be a tradeoff.
The number this whole series was built to measure was never your model's leaderboard rank. It was quieter than that: the next time your agent doesn't know the answer, does it say so — or does it hand you clean, well-formed, confident nonsense that sails right past your parser? Now you have a system that can answer that honestly, on your hardware, on your terms.
B Torkian is an NVIDIA Developer Champion. Broken Campus is open and lives at github.com/torkian/broken-campus. All figures are from an 8-model × 5-seed, audit-hardened run with Wilson 95% confidence intervals on a synthetic domain — directional, not definitive; seeds are near-deterministic, so the effective sample is closer to ~5 distinct cases than the 75-run headline.
Top comments (0)