DEV Community: Developer at Fortitude Omnis Group

The inner loop is back, and it's a matmul

Developer at Fortitude Omnis Group — Sun, 30 Aug 2026 15:24:28 +0000

I started on 8-bit machines, writing games where you drew the triangles yourself and counted clock cycles in Intel's V-Tune until the inner loop stopped hurting. The whole job was getting under the abstraction and making the metal do exactly what you asked. Then compilers got good, machines got fast, and for twenty-odd years that skill went quiet.

It came back, because a language model spends almost all its time doing the one operation I used to hand-tune on a 386: multiply a matrix by a matrix, add up the results, move on. So I worked on an inference engine by hand, in WGSL, running a real model in a browser tab, on kernels I can read line by line. Here's the road to it, and why it tells you something about your prompts.

2008: a model is a pile of numbers you can save to disk

The instinct pointed me at neural networks in 2008, on a contract at the British Library. I wrote a 4 layer network for a project, starting it out in LINQPad (as I did with most throw away tests), in C#, to sort academic paper titles into Dewey Decimal subjects: physics, medicine, agriculture etc. Four layers. Turn the title into a bag of words, multiply by a matrix of weights, bend the result through a sigmoid so it can do more than draw straight lines, multiply again, read off the winner with a softmax cross entropy layer.

Two things stuck. The maths is a matmul and a curve, over and over. And the knowledge it learns is nothing but those weight matrices. In my pre-school AI level knowledge, I called them synapses. Once trained, you serialise them to a file and that file is the model, the same idea as the multi-gigabyte files on Hugging Face today, only mine was a few hundred kilobytes. The toy became real at the library: instead of my 62 sample titles I trained it on 1.6 million PDF titles, which took weeks on a 2008 Core i7 CPU . Expensive training makes the weights, a cheap forward pass uses them. That split is the whole LLM game, then and it's the same now.

The HLSL detour that paid off later

Around then I built a small library of WPF pixel-shader effects in HLSL, the shader language for DirectX: colourise, desaturate, a skin-shade for softening portraits. Nothing to do with machine learning, mostly an excuse to get back to assembly-era fiddling, this time on a GPU. It put HLSL in my hands a decade before I actually needed it. Skills you pick up for one thing sit on the shelf and pay off in another. I love keeping the saw sharp.

2019: I wanted to run one on the GPU myself

After the library I kept building nets for the fun of it. MNIST, the handwritten-digits, Hello World of Neural Networks. Then a CNN to spot a dog fouling a footpath, PoopScanner, which I was going to run on a Raspberry Pi and sell to councils with an audible "You're being watched. Clean that up.". The councils were not ready. Finally a network for recognising vehicle panel damage, this was ever so slightly before the frontier multimodal models came into being.

But...they're all the same shape underneath: multiply by a matrix, apply a curve, do it again. And from the shader work I already knew how to drive a GPU, rustily, slowly & badly, but enough to make progress. So in 2019 I put the two together and started an inference engine in HLSL, to watch the kernels do the work rather than call someone else's library (there's a Donald Knuth line at the end that sums up why). I got a chunk working, then life / job / contracting happened and it sat half-finished for nearly a decade. I have a whole bunch of dumpster fire code sat rotting away just like this one, that might be useful to resurrect if Claude can help take the strain ;)

Finishing it in a week, with an agent

A few weeks ago I picked it up again, and since I'm pretty much AI first I'll be straight about how. I sat down with Claude and we finished it together, rewritten from HLSL into WGSL so it runs in a browser tab. The design is mine and decades old, the kernels are hand-written and I can read you almost every line of tghe HLSL, but I won't pretend I typed every character anymore. Claude did a lot of the typing. The English I write to steer a model is now as real a language as the HLSL I wrote for the GPU, and this project wanted both, for the cost of a few hundred million tokens out of my 20x subscription.

The model I used in the demo is Qwen2.5-0.5B, downloaded from Hugging Face: a 265MB file of weights, the same species as my 2008 "synapses". Mine had a few thousand numbers, this has half a billion. Mine knew a few hundred words, this carries 151,936 pieces of text. Open it up and the inner loop is my C# toy again, 24 times over.

Tokens

You type words. The model sees tokens, chunks of text with a number attached. "Paris" might be one token, "unbelievably" three. The tokeniser that splits them is a lookup table of about 151,000 pieces plus a set of merge rules. I had written mine in 2018 in plain C# with the help of a pre-built stemmer - Stemmers.Net. Claude re-wrote it in JavaScript and tested it against the real Qwen one until they matched on every character, because if they disagree by a single token the model produces confident gibberish and you lose a day blaming the maths. So 50% mine I guess, but the important part was that I could write it if I had to.

Each token becomes a list of 896 numbers, its vector embedding. A prompt is a stack of those vectors, and everything after is arithmetic on the stack. The model does one job forever: score all 151,936 possible next tokens, take the highest, append it, run again. That word-by-word streaming in every chatbot isn't an animation. It's the real speed showing that loop! Every word is a full pass through the model.

The inner loop is still a matmul

In our tiny demo case, each pass runs the same block of maths 24 times: attention, then per-token crunching. Every step is dominated by one operation, a matrix multiply. The attention projections, the crunch, the final scoring against a 151,936-wide table, all matmul. So I wrote the matmul kernel by hand, and two old tricks from my assembly days make it fast.

Tiling. Memory is slow and there are millions of multiplies, so you don't fetch from GPU main memory each time. You grab a 16 by 16 block, park it in a scratchpad next to the compute units, and reuse it hard. Same instinct as hoisting a value out of a loop. This was similar to the 3d polygon filling with small textures on the CPU and using V-Tune to make sure that the cache never got thrown out.

Four-bit weights. The weights ship as 16-bit numbers, half a billion of them, about a gigabyte. Moving a gigabyte through the GPU per word is a real bottleneck, the fetching, not the multiplying. So I store each weight in 4 bits: take 32 of them, set a shared scale from the biggest, store the rest as small integers against it. A gigabyte becomes 265MB. You never unpack them back into memory, you unpack each one inside the loop at the moment you multiply. Cheap arithmetic to save expensive memory, the oldest trade there is.

On the LLM scale we're still on the tardigrade level - if you imagine GPT3 is ~175 billion parameters using 16bit floats, that's 2 bytes each amounting to ~350GB. That model is old, we're now entering the age of trillion weight models!

Attention is you, steering. Steering IS your job.

So, here's the part most explanations skip, and the one that connects the maths to what you do all day: writing prompts.

Every layer, each token looks back and works out how much each earlier token matters right now. Each makes a query and a key, you multiply one token's query by another's key, a big number means relevant, softmax turns the scores into percentages, and you blend the earlier tokens in those proportions.

So a prompt is you loading those weights by hand. Write "explain this simply" and the word "simply" sits in the context, and every word the model generates glances back at it and gets pulled towards something short and plain. Swap it for "rigorously" and the same weights write something denser, because the attention lands elsewhere.

You didn't change the model, you changed what it looks at.

That's the whole skill of prompt wording, no magic, just understanding the maths.

And because I wrote the attention kernel, I've got those weights in hand ready to surface in something you can see. Usually the model throws them away after each word. I don't. So the demo can draw them: run a prompt and watch the earlier tokens light up to show what each new word leaned on and see the actual % numbers, change one word and watch the lights move. It'll even knock out each word of your prompt in turn when you check the MEASURE STEERING checkbox and measure which ones genuinely steer the answer, so you can watch a word do work or watch it get ignored.

A 0.5B model with two attention heads is a small window so don't expect it to solve any complex problems beyond writing a haiku, and the big models will attend in subtler ways across many heads. But the mechanism is the mechanism, and seeing it once on something you can hold in your head changes how you read the large ones.

Honest numbers

I raced my version against transformers.js, Hugging Face's tuned browser runtime, same model and machine, both on WebGPU. I know I can't win, but want to know if I'm still employable ;)

I win the first word: first token on screen in about 450ms against their 590, because my prefill is lean with no framework on top. They win sustained speed: warmed up, about 20 tokens a second to my 12, because a large team has spent far longer than a fortnight tuning it. Arguably the most important stat. Ask both "capital of France, one word" and they give the identical token, Paris. Ask something open-ended and they drift apart after a sentence, because my four-bit rounding and theirs aren't bit-for-bit the same. They agree where there's a right answer and drift where there isn't.

That's the four-bit tax, visibly measured. Four bits is just a plaything here because I'm not bothered about showing 'correct' answers, only how we reach them

The point was never to win a benchmark. It was to understand every line, and get close enough to a tuned engine that "close" is the interesting word. I can live with my failure to match a team of 20.

Two traps, because you'll hit them

My output came out as pure garbage, all zeros then noise, and only once the whole thing was assembled, though every kernel passed its own test. The cause was one missing flag on a GPU buffer, the one that says "you may copy out of this". Without it, the copy filling the attention cache silently did nothing, so the model attended over zeros, which becomes divide-by-zero, which becomes NaN, which spreads through the network in a single pass, like a forest fire started with a dumb ass disposable barbecue. No crash, no console error, because WebGPU reports that class of mistake down a side channel you have to go and ask for. One flag, half a day. This never gets old.

The other: I set a starting value one notch past the largest 32-bit float. The shader compiler rejected the whole file for that one constant, silently, and handed me a kernel that ran and did nothing. If you write GPU code, the compiler failing in silence will cost you the most hours. THE MOST HOURS. Check your shaders compiled at all. My fault again. I think I preferred C++ STL compiler errors to this.

Finally, coding isn't fun if you can't open the box

There's a line from Knuth I keep coming back to.

The problem is that coding isn't fun if all you can do is call things out of a library, if you can't write the library yourself.

He wasn't gatekeeping dependencies. He was mourning the job turning into plumbing between black boxes you're never allowed to open, and pointing out that once you can see inside one you can usually improve it. The "don't use what you couldn't write" version is the folk edit. For most libraries I'd pass it anyway, which is how I know it was never the point.

Doing tasks like this is what keeps me sane in a world where 90% of the profession is exactly that, parameter plumbing using other's code. Dull, yes, but it pays the mortgage, so no one complains, and it's not your employer's job to make it interesting.

I could have just called transformers.js in four lines and had a chatbot by lunchtime, and for a product that's 100% the right call. But I wanted to open the box, to see what happens between the prompt and the words at the level of the arithmetic, because you can't tune what you can't see. Half a billion parameters, and at the bottom of the well it's still just a matmul I can read, and it's taken me nearly 20 years to get here.

Anyway, it's live: type a prompt, watch the tokens land, race it against transformers.js, and turn on the attention view to watch your words steer it, at attentionspan.fortitude-omnis.group. It's one of the builds from the Fortitude Omnis R&D lab. Forty years in, and understanding the inner loop is still as important as ever. Who'd have thought? Are we redundant yet?

Any questions, comment & I'll get back to you :)

I put my cost router on a neutral benchmark. It ranked near the bottom, and that's the interesting part

Developer at Fortitude Omnis Group — Wed, 26 Aug 2026 13:37:36 +0000

A few weeks back I repriced three months of my own Claude Code usage. Real traffic, not a demo: 39.5 billion tokens across 139,835 requests. At API rates that's about $30,000, and 91% of it went to Opus because that's what the default reaches for. Route the fraction a smaller model handles as well, and the bill drops by roughly 50 to 58%. Five or six grand a month, on work that was already done and graded.

That's why OmnisRouter exists. It sits in front of your agents, reads each request, and sends it to the cheapest model that can answer it. The full write-up and the method are in the flagship post.

Saving your own money on your own data is one thing. I wanted an outside check on somebody else's turf, with somebody else's scoring. So I took the router to RouterArena.

What RouterArena is

RouterArena is an independent benchmark for LLM routers, out of an ICLR 2026 paper. It's a proper piece of work: 809 queries in the quick split, spread across 39 datasets, everything from MMLU-Pro and code generation to translation, chess and medical multiple-choice. You plug your router in, it picks a model per query, the harness runs that model and grades the answer, and you get an accuracy number, a cost number, and a combined Arena score. There's a hard rule I stuck to: you evaluate on their data, you never tune on it. No gaming the test.

I wired OmnisRouter in with the pool I care about: GPT-5, Opus-5, Claude Haiku, and GPT-5-nano. The models a team running coding agents genuinely uses.

First, two bugs, because honesty starts at home

The first scored run came back at 42% accuracy, which was obviously wrong. So before I say a word about the result, here's what I found when I opened it up.

Opus was scoring 0% on every single query. The scoring path was grabbing the model's extended-thinking block instead of its answer text, so the grader was marking a raw thinking object wrong 40 times out of 40. Second, my query file was missing the per-dataset answer-format instruction the grader keys on, so most models answered correctly in prose and got marked wrong because the grader couldn't find the answer in the shape it expected.

Both fixed. Opus went from 0% to 85%, the overall number went from 42% to 72.7%, and abnormal entries went from 558 to zero. If you run a benchmark and the number looks too bad to be true, it usually is, and it's usually your plumbing.

The honest result

On its own pool, OmnisRouter scores 72.7% accuracy at $3.71 per thousand queries, for an Arena score of 0.669.

That ranks it 16th out of 18. Near the bottom. On the easier split, even.

I could stop there and let you think I'm burying it, so I won't. Here's the top of the cost board next to us:

router              accuracy   cost/1K   Arena
Paix2-router          79.7%    $0.27     0.776
cross-router          78.2%    $0.29     0.762
sqwish-router         76.4%    $0.16     0.754
hybrid-router         71.4%    $0.04     0.721
omnisrouter           72.7%    $3.71     0.669

Look at the cost column. The leaders come in at four cents to twenty-nine cents per thousand. We come in at $3.71. That's the whole gap, and it isn't routing skill.

What the cost board rewards

I went and looked at what the top routers route to. Every one of them runs a cheap open-model pool:

Paix2 sends 78% of its traffic to a single flash model.
cross-router and sqwish run mostly DeepSeek-v4-flash and Gemini-3.1-flash-lite.
hybrid-router puts 95% of everything through one Qwen model.

Not one of the leaders uses GPT-5, Claude, or any frontier model. And that makes sense, because the Arena cost axis rewards spend, so the way to win it is to route to the cheapest models that still clear the accuracy bar on an academic test set. Fair enough as a benchmark design. But it means the board measures how cheap your pool is at least as much as how well you route it, and a router built for the premium models people run their agents on starts the race carrying a piano.

I'm not knocking the benchmark, to be clear. It's careful and the accuracy scoring is sound. I'm saying the cost ranking answers a different question than the one I care about.

The warts, in full

"The board's unfair to premium pools" is a convenient thing for me to say, so I ran the check that could prove me wrong. RouterArena also computes optimality metrics: how close each router gets to the oracle, the best model per query, within its own pool. That strips out the cheap-pool advantage entirely and measures routing quality on its own.

Here's where we land:

router            Opt.Sel   Opt.Acc
Paix2-router       89.7%     100.0%
BARouter           64.4%      93.8%
agentforge         51.1%      98.7%
auto_router        37.7%      86.0%
r2-router          24.5%      99.8%
omnisrouter        23.7%      81.5%

Opt.Sel is how often the router picks the cheapest correct model. Opt.Acc is how much of its pool's achievable accuracy it captures. We're 8th of 13 on selection, and last of 13 on accuracy captured.

That last number is the honest one. 81.5% means OmnisRouter leaves more winnable accuracy on the table than anyone else on the board, and it does that on purpose. It's a cost-first router. Faced with a query a cheaper model can probably handle, it takes the cheaper model and the saving, even when the strongest model in the pool would have nudged the answer over the line. Tune it to chase accuracy and that number climbs, but then it stops being the thing I built. Some of the high selection scores above come from routers that barely route at all, by the way: park 95% of traffic on one cheap model that's usually the cheapest-correct and you "win" selection by barely choosing at all. I'd rather show you my mediocre 23.7% and tell you why than dress that up as skill.

The number that matters

A leaderboard rank isn't the point, and I'm not going to pretend a benchmark built around cheap open models is the scoreboard OmnisRouter should be judged on. The point is the bill.

On real Claude Code traffic, on the frontier models a working team runs every day, routing the cheaper-capable fraction cuts the cost by about half. The bill and the 91% Opus share are measured from three months of real requests. The saving is what comes off that bill once you route the work a smaller model handles, sized from the fraction OmnisBench measures as routable, so it's a reprice rather than a reroute I ran end to end. No academic test set moves it, and no cost board can rank it, because none of them run the models the work is happening on.

If your agent bill has a comma in it and you don't know where the money goes, that's the problem worth solving. The leaderboard was me checking my own work in public. Turns out the honest version is more useful than a good rank would've been.

OmnisRouter and the OmnisBench benchmark behind it are both open, Apache-2.0, and the numbers above are reproducible with RouterArena's own harness. Bring your own keys and check me.

I repriced 40 billion tokens of real AI coding. The bill goes where nobody tells you

Developer at Fortitude Omnis Group — Wed, 26 Aug 2026 08:23:37 +0000

Everyone selling an LLM router will tell you it saves 60% on your AI bill. Nobody shows you the numbers on a real workload. So I pulled my own.

Over the last three months I've run coding agents hard, Claude Code mostly, day in and day out. Every request it makes is logged locally with its token counts, so I parsed the lot: 1,231 sessions, 1.25 GB of logs, 139,835 billed requests, 1 June to 25 August. Here's what a real agent workload costs, and what routing would do to it.

The raw number

39.5 billion tokens. At Anthropic's API rates that's about $29,962, call it $10,000 a month.

Two things jump out.

First, 91% of that cost is Opus. Almost every request went to the biggest, most expensive model, whether it needed it or not. A one-line rename, a test refactor, a "summarise this diff", all of it to the frontier model at frontier prices. That's not a criticism of me. It's the default, and the default is expensive.

Second, and this surprised me, the bill is dominated by context, not output. The single biggest line is cache-read tokens, the whole conversation re-billed on every turn. By the hundredth turn of an agent session you're paying for the entire codebase again to add one function. Output is the small part.

What routing does to it

The pitch for routing is simple. Send the work a cheaper model can handle to a cheaper model, and escalate to the frontier only when you need it. The question is how much of the work is genuinely cheap-model-able.

My companion benchmark, OmnisBench, measures that on fresh coding problems the models can't have memorised. There, ideal routing matches the frontier model's quality at roughly 60% lower cost, because most coding requests don't need the biggest model.

Apply that to my $30k. Route the 60% of requests a cheaper model handles down to a small model, keep the rest on Opus, and the equivalent bill drops to somewhere between $12,500 and $15,600. A 48 to 58% cut. About $5,000 to $6,000 a month, on a $10,000 workload. That lands right on the benchmark's number.

The honest bits

Two things I won't paper over.

I didn't pay $30k. I ran this through a Claude Max subscription, which already saves something like 98% versus the API. So this isn't a bill I paid, it's a realistic workload standing in for a team that does pay API rates. If your team runs agents on your own keys, this is roughly your shape.

And the reprice is modelled, not proven. The 60% comes from the benchmark's aggregate, not from re-running every one of my requests through a cheaper model to confirm it passed. Proving that costs real API money, and it's the honest next step, not a number I'm going to pretend I already have.

Why I trust the direction anyway

Because there's now an independent check. RouterArena, an ICLR 2026 benchmark, ranks routers on accuracy against cost, and it found the loud commercial ones frequently over-pick expensive models. It's the neutral scoreboard the field didn't have. OmnisRouter is built for exactly what it measures: pick the cheapest model that's still correct, escalate when unsure, and show the receipt.

Do it yourself

The whole point is that you don't have to take my word for it. Your Claude Code logs are on your disk, with the token counts, in ~/.claude/projects. Parse them, price them at API rates, and see your own split. Mine was 91% Opus. I'd bet yours is close.

The three pieces, all open:

OmnisBench measures which model each kind of work needs, on fresh problems, re-gradable offline.
OmnisRouter routes each request to the cheapest capable model, with a receipt on every response.
OmnisVigil rolls it up per team, so a lead sees the runaway agent before the invoice does.

Tell me what your split looks like.

Links: github.com/Fortitude-Group/OmnisRouter · omnisbench.fortitude-omnis.group · omnisvigil.fortitude-omnis.group

We built a benchmark, then caught it strangling the models it was grading

Developer at Fortitude Omnis Group — Fri, 21 Aug 2026 23:37:46 +0000

A couple of day ago I posted about OmnisBench, our open benchmark for LLM routing, specifically our LLM Router OmnisRouter , and made a fuss about how you can re-grade every number yourself because we publish the actual model responses. Two commenters, deanlee and jugeni, very politely pointed out that the whole thing might be resting on a fib.

Their point: HumanEval and GSM8K are old. The models have almost certainly read the answers. So when I cheerfully reported that the cheapest model already scores 94.5% and routing only recovers the last few points, that's maybe less "small models are quietly brilliant" and more "small models have seen the exam paper." Fair. Annoyingly fair.

So we did the thing you're supposed to do and actually built the fresh split we said we would.

The fresh split

The idea's simple. Take LiveCodeBench, which stamps every problem with a release date, and keep only the ones published after the models could plausibly have trained on them. Grade the same routing policies on the old, probably-memorised tasks and on the new, definitely-not tasks, side by side. If the routing story only survives on the old set, you deserve to know.

I ran it. Here's where it took a turn.

The first numbers, and a brief crisis of faith

They came back grim. The cheap model fell off a cliff. Even the big expensive frontier model, the one whose entire job is to be good at this, was landing well under where a frontier model has any business landing. My first thought was that we'd accidentally built the most damning benchmark in the history of our own company, and my second was whether it was too late to go back to doing something honest for a living.

Then I remembered the one feature I keep banging on about: we publish every response. So instead of trusting the sad little number, I opened the responses and read them.

Every one of the frontier model's failures on the hard problems was the same thing: an empty answer. Not a wrong answer. Nothing at all. Eleven problems, eleven blank pages, each one exactly 4,096 tokens of the model thinking out loud and then having the microphone cut off before it wrote a single line of code.

Here's what happened. Our request budget was 4,096 output tokens, a number that's perfectly sensible for "write this short Python function" and perfectly ridiculous for "here's a genuinely hard competitive-programming problem, think it through." The reasoning models did exactly what reasoning models do. They thought. At length. And then they hit the ceiling somewhere in the middle of it and got cut off before writing a line of the solution.

So the benchmark wasn't measuring how good the models are at hard problems. It was measuring how good they are at running out of room. We'd built, with some effort, a very reproducible way to gag a model and then mark it absent.

I'd love to tell you we spotted this through rigour and foresight. We spotted it because the responses were sitting right there in the file, empty, quietly judging us. Which is, if you squint, exactly the argument for publishing them. A closed benchmark makes this same mistake and ships the number, and everyone nods, because who's going to check.

The corrected numbers

We made the output budget a config setting, gave the hard suite the room it needed, and ran it again.

Policy	Likely-contaminated (20)	Fresh, 2025+ (15)
cheapest model only (nano)	90.0%	60.0%
ideal routing (oracle)	100.0%	93.3%
always the frontier model	100.0%	86.7%
how often ideal routing reached for the frontier	0%	20%

Three things fall out of that, now the models can actually finish. The frontier model's fine: 86.7% on the fresh problems, not the panicked 60% the truncated run reported. The cheap model, though, drops from 90% on the old benchmarks to 60% on the fresh ones, which is the contamination showing through exactly as predicted. And routing earns far more on the fresh set: on the old tasks it buys ten points over the cheap model, on the fresh tasks it buys thirty-three, while reaching for the expensive model a fifth of the time instead of never. The honest asterisk: the fresh set's competitive-programming, which is both newer and harder than grade-school maths, so difficulty and freshness are tangled together here, and the samples are small. It's a first signal, not a verdict.

What I'll actually stand behind

Two things, and I'll keep them honest because you can check both.

The contamination point stands. Even with the models given all the room they wanted, the cheap one's thirty points worse on problems it can't have memorised. The "small models are quietly brilliant" story was, in part, small models sitting an exam they'd already seen.

The harness point is the one I didn't expect to be writing about, and it's the more useful one. Half of what looks like a model being bad is a benchmark being broken, and you can't tell the difference from a leaderboard. You can only tell by reading the answers. If a routing benchmark, or a routing vendor, won't show you the responses, you've no way of knowing whether their impressive number is a real result or a bug wearing a nice suit.

Cost of finding all this out: about $7.47. Four dollars of that was the run that produced the numbers above; the rest was the earlier runs that produced the wrong ones and taught us the lesson. I'm counting the mistakes, because a benchmark that hides its own costs has no business lecturing anyone about hidden numbers.

Thanks to deanlee and jugeni for the nudge. The suite grows from here, more fresh problems and harder ones, and every number stays re-gradable offline with omnisbench verify.

Results: https://omnisbench.fortitude-omnis.group/
Code: https://github.com/Fortitude-Group/OmnisBench

OmnisBench is built by Fortitude Omnis. We make small, sharp tools, and when we catch ourselves measuring the wrong thing, we write it up rather than quietly fixing the chart.

I built an LLM router that hands you a receipt for every request

Developer at Fortitude Omnis Group — Thu, 20 Aug 2026 11:15:55 +0000

A router that decides where your money goes is asking for a lot of trust. It sits in front of every request, quietly picks a model, and sends you a bill at the end of the month. If it tells you it saved you 60%, you mostly have to take its word for it.

I didn't want to take its word for it. So OmnisRouter attaches a receipt to every response, and the receipt is the whole point.

It's an open source (Apache 2.0), self-hosted proxy that speaks the Anthropic, OpenAI and Gemini formats. You change one base URL, keep your own provider keys, and each request goes to the cheapest model that can actually handle it. The response comes back in your client's own format, with a note explaining what was chosen and what it cost.

The receipt

Every response carries a set of headers:

X-Omnis-Model: gemini/gemini-2.5-flash
X-Omnis-Decision: Routed
X-Omnis-Confidence: 0.26
X-Omnis-Policy: v3-omnisbench-2026-08-20
X-Omnis-Cost-Delta-Vs-Big: -0.0121

And if you want to know what it'd do before spending anything, there's a cost-free endpoint that returns the full decision without calling a provider:

curl -s localhost:8080/v1/route -H "authorization: bearer $TOKEN" \
     -d '{"messages":[{"role":"user","content":"Summarize this thread."}]}'

{ "policy_version": "v3-omnisbench-2026-08-20",
  "decision": "ROUTED", "reason": "cheapest_capable",
  "chosen": { "provider": "gemini", "model_id": "gemini-2.5-flash" },
  "est_cost_delta_vs_big_usd": -0.0121 }

No screenshot, no dashboard you have to trust. The decision's in the response, and you can log it.

How it actually decides

There's no network hop to work out where to send a request. A small pinned ONNX model (bge-small-en-v1.5) embeds the prompt in-process, the embedding maps to the nearest intent cluster, and the cluster's policy table picks the cheapest candidate that clears the quality bar for that kind of work. If it isn't confident, it escalates to a strong model rather than guessing. The whole thing happens in well under the 50ms I gave it as a budget.

Here's a live trace from the shipped model:

Request	Routed to	Decision
Summarize this thread	gemini-2.5-flash	routed (cheap)
Prove sqrt(2) is irrational	gpt-5-nano	routed (cheap)
Write a Python function to merge two lists	gpt-5	routed (strong)
Hello.	claude-opus-5	escalated

The maths row is the interesting one. On grade-school maths, the tiny gpt-5-nano is essentially as accurate as the frontier model, so OmnisRouter sends the work there and keeps the roughly 25x price difference. That's not a guess. It comes from measurement.

Where the numbers come from

The routing model isn't a black box I'm asking you to believe in. The intent clusters and the policy table ship in the repo and rebuild from public data, so the same inputs produce the same model, and every decision is stamped with the policy version that made it.

The coding and maths policy is driven by OmnisBench, a companion benchmark that grades each model per task and publishes every response so you can re-grade it offline. So when the router says "the cheap model is good enough here", that's a number you can go and check, not a line on a slide. Other domains use sensible estimates for now, and the benchmark coverage grows from there.

The honest bit

Cheapest-capable isn't the same as cheapest. The point isn't to slam everything into the smallest model and hope. It's to spend frontier money only where the work needs it, and to be able to show, per request, why it went where it went. When a capability can't be carried faithfully to the chosen provider, it refuses with an explicit error instead of silently dropping it. Surprising the first time, correct every time.

It runs as a single self-hosted process with an embedded database, your keys are encrypted at rest, and prompt content only ever leaves your infrastructure to go to the model you chose.

Have a look

Site and the full routing story: https://omnisrouter.fortitude-omnis.group/
Code: https://github.com/Fortitude-Group/OmnisRouter
The benchmark behind the routing: https://omnisbench.fortitude-omnis.group/

If you can make it route something badly, or you think the receipt should carry more than it does, tell me. That's what the open model and the decision log are for.

OmnisRouter is built by Fortitude Omnis. We make small, sharp tools, and we try not to lie in our own marketing.

Everyone is saving 60% on LLM costs. Nobody will show you the numbers.

Developer at Fortitude Omnis Group — Wed, 19 Aug 2026 13:05:03 +0000

Every few weeks the same post does the rounds. Someone's routing layer cut their LLM bill by 60%, "with no drop in quality", and here's a lovely chart to prove it.

Except it doesn't prove anything. The eval set is private. The grading is private. The actual model responses that produced that score are never published. You're being asked to trust a screenshot, and we've been in I.T. long enough to know roughly what a screenshot is worth.

So we built the thing that was missing. It's called OmnisBench, it's open source (Apache 2.0), and its whole job is to measure how good LLM routing actually is, in a way you can check for yourself.

What routing is, and why it should save money

If you're not knee deep in this yet: an LLM router looks at each request and sends it to the cheapest model that can still do the job, rather than firing everything at one expensive frontier model. Easy prompts go to something small. Hard ones go to the big model. In theory you keep most of the quality and pay a fraction of the bill.

That's the theory. OmnisBench measures the practice.

The number we care about most is what we call the oracle. For every task, the oracle picks the cheapest model that actually got it right. We know which ones did, because we ran them all. That is not a router you can ship. It's the ceiling. It's the best any router could possibly do on a given set of tasks, and it tells you how much money is genuinely on the table before anyone starts making claims.

The first run

We ran it over 364 tasks: HumanEval (164 coding problems, graded by their own unit tests) and GSM8K (200 maths problems, graded on the answer). The model pool was Claude Opus 5, GPT-5, Claude Haiku 4.5 and GPT-5-nano.

Here is what came out.

Policy	What it does	Task success	Cost per 1,000 requests
oracle	cheapest model that actually solved each task	99.7%	$0.62
always-frontier	everything to Claude Opus 5	99.2%	$6.25
random	pick a model at random	96.2%	$4.01
always-cheapest	everything to GPT-5-nano	94.5%	$0.43

So the headline writes itself. Ideal routing hit 99.7% success for roughly 90% less than sending everything to the top model. Very nice.

Now let me talk you out of it a little.

The honest bit

Two things you will not see on most of those 60% charts.

First, the oracle is a ceiling, not a product. It cheats. It knows, after the fact, which model got each task right, and no live router has that luxury. The real score for a router you can actually deploy is how close it gets to that oracle line, and the gap is the interesting part. Anyone quoting an oracle-style number as their shipping product is selling you the ceiling.

Second, and this is the one that surprised us: the cheapest model on its own already scored 94.5%. On this particular set of tasks GPT-5-nano is just quietly good. So routing is not buying you a magic 60%. It's buying you the last few points of quality without paying frontier prices for all of it. That is still very much worth having. It just isn't the fairy tale.

We would rather tell you that up front than have you find out after you have quoted the shiny number to your boss.

The part that actually matters: you can check it

This is the whole reason OmnisBench exists. It does not ask you to trust the table above. It publishes the actual model response for every task, and ships a command that re-grades the lot offline.

pip install -e .
python scripts/prepare_datasets.py                                  # HumanEval + GSM8K
python -m omnisbench.cli run    --config configs/v0.yaml --run runs/mine
python -m omnisbench.cli report --run runs/mine
python -m omnisbench.cli verify --run runs/2026-08-19               # re-grades the published run, no API calls

verify re-runs the graders against the published responses and rebuilds the entire leaderboard, cost included, without calling a single API or needing a key. Tamper with one stored answer and it fails. That is the point. If our numbers are wrong, you can prove it in about a minute, and we would genuinely like you to try.

Have at it

The results, the method and the honest caveats are all on the site, and the code is on GitHub:

Results: https://omnisbench.fortitude-omnis.group/
Code: https://github.com/Fortitude-Group/OmnisBench

We also build an actual open router, OmnisRouter, which this benchmark grades on every release, so it can't quietly regress and hope nobody notices. It is live now at https://omnisrouter.fortitude-omnis.group/.

If you can break the numbers, poke a hole in the method, or you just reckon we've measured the wrong thing, tell us. That is what open and verifiable is for.

OmnisBench is built by Fortitude Omnis. We make small, sharp tools, and we try not to lie in our own marketing.