DEV Community: Developer at Fortitude Omnis Group

Three-quarters of my Claude classification calls can run on a gaming GPU. How to measure yours

Developer at Fortitude Omnis Group — Tue, 29 Sep 2026 12:59:50 +0000

Rob Hill, Fortitude Omnis. Measured on 27 and 28 September 2026.

I sent a thousand Banking77 support messages through a model on my own RTX 3080 Ti first, and only passed the uncertain ones to Claude Opus 5.5. With a fine-tuned Laya decision model doing the local work, 73.0% of decisions stayed on the GPU, blended accuracy was 93.3% against 94.2% for Claude alone, and the estimated bill fell from £3,898 to £1,054 per million decisions. The caveats belong right here. It's one dataset (Banking77), one card, one night. The £ figures are estimates from published list prices, not invoices. And Claude's answers came from an interactive Claude Code session working through batched answer sheets, not from the API.

The twist is the model that did best. A plain fine-tuned MiniLM classifier kept 94.7% local at the same 94.2% blended accuracy as Claude alone. I'll come back to that. I also measured TypeSafe's hosted Jev on the same items, and it lands between the two on Banking77.

This post is the hands-on version. It shows how to run the server, how to calibrate a model's confidence, and how to get the same report for your own decision. The full write-up is on the Fortitude Omnis R&D page.

What is Tau?

Tau is two things, both Apache-2.0.

The Runtime is a .NET server that answers the /v1/systemone decision contract locally. It runs the open Laya and Von decision models through ONNX Runtime on CUDA, DirectML or a CPU. You send it some state and a set of questions, and it sends back an answer with a probability for each option.

The Workbench is a command-line tool called tau. It asks one question of any /v1/systemone endpoint: can I trust this model's confidence enough to gate on it, and what does gating save? It measures a hosted endpoint over the network the same way as a local one. Every stage writes its output to disk, and the last one writes a single HTML report with the misses left in.

How do I run a self-hosted /v1/systemone server?

The repo's README walks through fetching a model, exporting it to ONNX and fetching the ONNX Runtime natives. After that, starting the Runtime on an NVIDIA card is one line:

dotnet run --project src/Tau.Runtime -c Release -- --urls http://localhost:8088 --Tau:Provider=cuda

Leave out --Tau:Provider=cuda and it runs on the CPU. On Windows, --Tau:Provider=directml runs on any DirectX 12 GPU. It refuses to start if the provider you asked for won't load, which I prefer to a silent fallback to the CPU.

Then ask it something:

curl -s http://localhost:8088/v1/systemone \
  -H "Content-Type: application/json" \
  -d '{
    "model": "jev-latest",
    "state": "My card payment failed three times this morning and I need it sorted today.",
    "questions": {
      "category": {
        "type": "choice",
        "instructions": "Which support category fits this message?",
        "criteria": {
          "billing": "Billing or payments issue",
          "technical": "App or website technical issue",
          "account": "Account access or security issue"
        }
      },
      "urgent": { "type": "noul", "instructions": "Does the customer need an answer today?" }
    }
  }'

jev-latest is an alias that lets the Runtime pick a model. English text goes to laya-en. You get back a choice with a probability for each option, plus a noul score for the yes/no question. GET /v1/models lists what's loaded.

From C#, the Tau.Client package wraps the same call and maps the answer onto an enum:

using Tau.Client;

using var client = new SystemOneClient(new Uri("http://localhost:8088/"));

var decision = await client.DecideAsync<Category>(
    state: "My card payment failed three times this morning and I need it sorted today.",
    instructions: "Which support category fits this message?");

Console.WriteLine($"{decision.Value} p={decision.Probabilities[decision.Value]:F2}");

enum Category { Billing, Technical, Account }

How fast is it on a gaming GPU?

On the 3080 Ti with CUDA, laya-en answers one question in 18.59 ms and ten questions about one state in 114.11 ms. On the i9-11900K's CPU, the single question takes 529.21 ms. Those are FP32 medians with no FP16 tricks. For comparison, TypeSafe's hosted Jev had a median of 223 ms, but that's measured over the network from my desk, so it isn't like for like.

How do I calibrate Laya's confidence scores?

This is the bit that matters, and the reason the Workbench exists. A decision model gives you an answer and a probability. You want to keep the answer when the probability is high and escalate when it's low. That only works if 0.9 means right nine times in ten.

Out of the box, it doesn't. On Banking77, laya-en was 37.2% accurate with an expected calibration error (ECE) of 0.502. Its stated confidence sat about 50 points away from its hit rate. A confidence score that looks certain and means nothing.

Calibration fixes the meaning of the number and leaves the model alone. Fitting a small calibrator on a separate 1,000-item split took laya-en's ECE from 0.502 to 0.056, and Von's from 0.185 to 0.021. Accuracy didn't move.

The Workbench does it in stages. You describe the decision in a decision.yaml (the question, the labelled data, the local models, the frontier model and a target error), then run:

tau label     examples/banking77/decision.yaml   # ingest cached frontier answers
tau measure   examples/banking77/decision.yaml   # raw accuracy and ECE, calibration and held-out splits
tau calibrate examples/banking77/decision.yaml   # fit temperature and isotonic calibrators
# restart the Runtime with --Tau:CalibratorsDirectory=examples/banking77/calibrators, then:
tau measure   examples/banking77/decision.yaml --phase calibrated
tau threshold examples/banking77/decision.yaml   # pick the threshold for the target error
tau cascade   examples/banking77/decision.yaml   # simulate local-first and price it
tau report    examples/banking77/decision.yaml   # write report.json and report.html

tau run does the lot in order and skips stages whose outputs are current. It never calls a paid API unless your spec lists a hosted endpoint under external:, and then only under the budget you set there. If frontier answers are missing, tau label exports batches to be answered and exits with code 2.

One trap I nearly fell into. Laya's contract returns a confidence field for choice questions that's derived from the entropy of the whole distribution, so it isn't a calibrated probability. Calibrate and threshold on the probability of the chosen answer instead. That's why the C# example above prints Probabilities[decision.Value].

How do I decide how much can stay local?

A calibrated model is only useful if it's also right often enough. Raw laya-en wasn't: no threshold got its error below 5%. So I fine-tuned it on the 9,003-item training split, on the same card, in 1,518 seconds.

With a threshold of 0.95, picked on the calibration split and judged on the held-out split, the fine-tuned model kept 73.0% of decisions local and sent the rest to Claude, for an estimated £1,054 per million.

The money is arithmetic on list prices. Each Banking77 prompt is about 1,259 input tokens once you list all 77 intents, so Claude Opus 5.5 comes to £3,898 per million decisions, or £1,949 through the Batch API. Tokens are estimated from characters, not counted. I haven't costed the GPU, which I already owned for less serious reasons.

Why did a classic encoder win?

I put a fine-tuned all-MiniLM-L6-v2 through exactly the same measurement, calibration, threshold and cascade code. It scored 91.5% on the held-out set, with a calibrated ECE of 0.024, against 87.3% for the fine-tuned Laya. In the cascade it kept 94.7% local for an estimated £207 per million. It isn't served through Tau, so that figure leaves out local latency and energy.

On a fixed task with thousands of labelled examples, a small classifier is still the thing to beat. Decision models earn their keep on questions you haven't trained for, or many questions against one state. I'd rather the Workbench told me that than hid it.

The second dataset was worse. On synthetic support tickets, Claude agreed with the labels on 23.8% of items against a 41.0% majority baseline, MiniLM "learned" them to 55.6%, and scored against Claude instead, no local decision model kept more than 0.1%. On urgency, nothing local stands in for the frontier call.

How does a hosted model compare?

The Workbench takes hosted /v1/systemone endpoints as well as local ones. You list them under external: in the spec, with the name of the environment variable that holds the key. The key is read at run time and never written anywhere, and a budget guard stops the run before spend passes the limit you set. I used it on TypeSafe's hosted Jev (the responses say jev-1.13.0), on the same items as everything else. It cost an estimated $0.14 for Banking77 and $0.04 for the tickets, at the published $0.042 per million input tokens.

On Banking77 it's a solid generalist. It scored 79.2% held-out out of the box, ahead of laya-en's 37.2% and Von's 77.1%, with an ECE of 0.093 before calibration and 0.029 after. As the first stage of a cascade it kept 51.3% at 93.6% blended, for an estimated £1,952 per million against £3,898, with its own calls priced in. The fine-tuned Laya beat it on accuracy (87.3%) and on share kept local (73.0%), and MiniLM beat both.

The tickets are where it earns its place. Scored against Claude, it agreed on 52.5% of items out of the box, ahead of every local model. At the 20% target it kept 39.8% of decisions, with the served answers agreeing with Claude on 89.3%, for an estimated £614 per million against £997. Nothing local came close.

Two things to know before you gate on it. Jev returns probabilities rounded to 2 decimal places, and a hosted endpoint can't load your calibrator, so its calibrated figures are the Workbench applying one offline. You'd do the same on your side of the call.

What went wrong?

I fitted the calibrators on rounded numbers. The Runtime rounds probabilities to 4 decimal places, the Workbench fitted its calibrators on that, and the Runtime applied them to the unrounded values. With 77 options most probabilities round to zero, so it mattered. My earlier figures (laya-en 0.065, Von 0.039) came from that. Refitted on full precision they're the 0.056 and 0.021 above. I'd also claimed the Workbench and the Runtime agree within 2.4e-4, having only checked inputs that didn't saturate. The decisions log has the fix.
Three FP32 models on one 12 GB card slowed the last one down. The fine-tuned Laya, measured last, had a median of 15,219 ms per request in its raw phase, against 243 ms in a fresh Runtime. VRAM spill is the likely cause, not proven. Accuracy isn't affected, and the published latency comes from the clean phase.
The classic encoder beat every decision model on Banking77, Tau's and Jev, on accuracy and on calibration.
The ticket labels are close to noise. Claude agreed with them less often than a constant guess would.
On urgency, no local model stands in for Claude. No local decision model kept more than 0.1% of decisions.
Calibration barely helped the fine-tuned Laya: ECE went from 0.071 to 0.060. Fine-tuning had already fitted its temperature.
laya-en missed the 50% calibration target on the tickets: ECE went from 0.273 to 0.155. The Workbench picks the calibrator with the lower calibration-split log loss, which chose isotonic even though temperature scaling had a calibration-split ECE of 0.016 against 0.216. I set that rule before seeing any held-out result, so I didn't change it.
Von's tickets calibration moved ECE from 0.045 to 0.042. It was already close.
My first MiniLM run scored 80.8% because I stopped it after five epochs with the loss still falling. That flatters whatever you compare it with, so I retrained it properly.
Claude disagreed with the Banking77 labels on 5.8% of items. Some are Claude's mistakes and some are the dataset's.

How do I reproduce it?

The repo holds both worked examples with their reports, calibrators, cached frontier answers and per-item results. With the models fetched and the CUDA natives in place, one command reruns Banking77 end to end and rewrites its report:

./scripts/examples.ps1 -Example banking77

It checks the data and model packages first and prints the exact command for anything missing. Then read the Banking77 report and the tickets report. They were harder on me than I've been here.

The repo is at https://github.com/Fortitude-Group/tau.

The cheapest tokens you'll ever buy are the ones you already cached

Developer at Fortitude Omnis Group — Tue, 22 Sep 2026 11:47:08 +0000

Most agents throw a chunk of them away every single turn, and never see it happen.

There are two levers on an AI coding bill. The big one is routing: send the easy work to a cheaper model, keep the frontier model for the hard slice, and measure whether the cheap one was actually good enough. That's where the large savings are, and you have to earn them.

This is about the small lever. It's duller and it needs no judgement at all, which is rather the point. If you run an agent, you're almost certainly paying full price for tokens you'd already bought at a discount, over and over, and you can't see it happening.

Here's the mechanism. When you send a request to Claude, the big unchanging part at the front (your system prompt, your tool definitions, the file context an agent reloads every turn) can be cached. The next request that starts with the same prefix reads it back at about a tenth of the normal input price. A cache hit is cheap. That's the whole deal, and it's a good one.

A cache miss is the opposite. You pay the full fresh-read price for a prefix you'd already paid to store. Same tokens in, same answer out, higher price. You get nothing for the difference. It's the purest waste in the whole bill.

And the cache is fragile in a way nobody warns you about. Change one byte near the front and the hit becomes a miss. The usual culprits are dull: a line ending that flipped from LF to CRLF, a trailing space, your tool definitions serialised in a different order this run, a timestamp injected into the system prompt, a volatile header, the order two chunks got concatenated. None of these change what the model sees in any way that matters. All of them bust the cache.

A miss isn't a small, contained thing either. The cache matches your prefix from the front, token by token, and the instant it meets a byte that's changed, everything after it has to be recomputed. Claude Code builds that prefix in a fixed order: system prompt and tool definitions, then your static project memory (your CLAUDE.md and docs), then the conversation, then the new input. Put a volatile line near the top, a git branch, a coverage figure, a "last updated" stamp in the first few lines of a CLAUDE.md, and you don't pay for those few tokens. You pay to recompute the entire cached prefix beneath it, on every request that carries it. One changed byte before a breakpoint takes every cached segment after it down with it.

On a single request the cost is pennies, sometimes less. Easy to wave away. But an agent doesn't send one request. It sends the same fat prefix thousands of times a day, and if a stray character busts the cache every turn, you re-pay every turn. It might not be much each time. It accumulates.

Worth doing the sum once. Say an agent carries a 20k-token cached prefix, turns over 2,000 times a day, and a single stray CRLF busts the cache each turn. On a mid-tier Claude model at roughly $3 per million input tokens, with cached reads about a tenth of that, the gap between hit and miss on that prefix runs to something near $100 a day. From one character, on one agent. That's a deliberately unkind worst case, and your real figure will be smaller and messier, but you can see the shape of it. Nobody notices, because it never shows up as a line item. It's the bill being quietly larger than it needed to be.

So we built the part that makes it visible, and where we safely can, the part that fixes it.

OmnisRouter now labels why each cache miss happened. It watches the traffic going through it, works out the cause of every miss, and tags it. This is content-free: it classifies the reason, it doesn't read or keep your prompts. It hands that up to OmnisVigil as a receipt.

OmnisVigil turns those receipts into a number you can act on, and it's honest about which misses count. A genuine edit to your prompt is a cache miss too, but it isn't waste, you meant to change it. Same for switching model or rewriting the system prompt. OmnisVigil separates those out and puts only the avoidable misses in the headline, attributed to the team, repo and commit that caused them. So the number you're looking at is the part you could actually recover, not a scary total you can't do anything about.

Three of the avoidable causes fix themselves. Line endings, trailing whitespace and tool ordering are safe to normalise without changing a thing the model reads, so you flip them on in OmnisVigil and OmnisRouter cleans up future requests on the way through. The misses turn back into hits. No code change, no tradeoff. The other avoidable causes (an injected timestamp, a volatile header, a concatenation order that drifts) need a one-off change at your end, so OmnisVigil flags those with the evidence rather than touching them silently.

I'm not going to quote you a savings figure. It's your workload, not ours, and measured beats claimed. What I'll say is that the dashboard shows you your own avoidable-waste number, and for most of it the fix is a toggle.

Routing is the big lever. Cache hygiene is the small one, and it's the bit still leaking after you've done the routing and told yourself you're optimised. Both are open. Both are measured. If your agent bill is bigger than you can explain, this is a cheap place to start.

Route: OmnisRouter
Govern: OmnisVigil
Measure: OmnisBench

I built a real-time flood digital twin of a UK river that runs entirely in the browser, GPU and on-device AI included

Developer at Fortitude Omnis Group — Thu, 17 Sep 2026 08:32:59 +0000

I've been chasing GPU cycles since machines had 64K of RAM, and every few years the browser quietly swallows another thing that used to need a workstation. WebGPU was the one that made me want to find the ceiling. So I set myself a question: how much of a genuinely serious flood tool could I put inside a single browser tab, with no server behind it, and still have it feel like a tool rather than a toy?

The answer is Don Watch, a live flood-response digital twin of Sheffield's River Don.

Try it: https://donwatch.fortitude-omnis.group (best on a desktop browser with WebGPU)

It rebuilds the Don valley from open LIDAR elevation, drapes the real river over it, and replays the November 2019 Sheffield flood hour by hour from the Environment Agency's own gauge records. The water you watch rise is the level that was recorded at the time. Scrub the timeline and the flood tracks the readings. Switch to live mode and it runs against the EA's current feed instead.

Then the parts that were interesting to build.

The rendering runs on WebGPU

Terrain, water and post-processing all go through three.js on its WebGPURenderer, shaded in TSL (Three's node shading language) rather than hand-written GLSL strings.

The water isn't a flat plane with a normal map. It's a level-driven flood surface: the sim tracks the gauge height at the current instant, and a breadth-first fill spreads out from the real river vertices through submerged ground only, so isolated dips away from the river never fill. That height field is written to a texture the shader samples for both displacement and depth.

The fiddliest bug was z-fighting at the shoreline. The draped water and the terrain are two different resolutions of the same heightfield, so at the waterline they interpenetrate. A depth-buffer bias can't fix surfaces that actually cross, so the fix was geometric: lift the water a couple of metres so it always sits cleanly above the ground, which is sub-pixel at the camera distance and invisible.

The AI runs on your GPU too

The pitch a lot of "AI X" tools make is a thin front end sat in front of somebody else's cloud. I wanted the reverse. The model, an open model in the 1-to-2-billion range, loads straight into the browser through WebLLM and runs on the visitor's own GPU. No server, no API key, nothing leaving the machine. The inference backend sits behind an interface, so WebLLM is the default and there's a coherent scripted fallback for devices that can't run it.

There's a button that cuts the network, and everything keeps going, because the model is cached in OPFS and the flood data came down with the page.

Agents that show their working, over deterministic maths

Five small agents work the flood while it plays: an orchestrator hands out the job, two ingest agents read gauge and rainfall, a forecast agent projects ahead, a risk agent ranks the sites in the water's path, and a comms agent drafts a plain-language brief. Every step streams into a live reasoning feed and lights up a delegation graph.

The important design decision is what the model is not allowed to do. The hydrology, the projection and the risk thresholds are deterministic TypeScript. The model reads the results and writes them up. It doesn't produce the numbers. That's the honest place for a language model in a safety-adjacent tool, and it's why the whole thing is framed as a decision-support projection rather than a flood warning.

Offline by construction

It ships as a Next.js static export and installs as a progressive web app. A service worker caches the shell, the data and the model, so once it has run it runs with the network cut, on a locked-down laptop or in a field office with no signal.

The data is real and cited

Gauge and rainfall from the Environment Agency, the river, landmarks and place names from OpenStreetMap, terrain from open elevation data, and the official flood-warning areas and live sensors from the EA flood-monitoring API. Nothing is fabricated, and where a series is reconstructed it says so.

A few things I learned

Real sensor networks are sparse. Central Sheffield has a handful of river-level gauges, not a dense grid. You show what's actually there and resist the urge to invent more.
Serving a Next static export under a sub-path means every runtime fetch that basePath doesn't cover needs the prefix, and the service-worker scope moves with it.
WebGPU plus a local LLM in one tab is a lot of GPU contention. Bracketing generation so the render loop can yield turned out to matter.

Have a poke around and let me know what you think: https://donwatch.fortitude-omnis.group

Happy to go deeper on the WebGPU or the on-device model side in the comments.

Hi, I'm Rob building OSPulse at Fortitude Omnis

Developer at Fortitude Omnis Group — Mon, 14 Sep 2026 18:56:55 +0000

Hi everyone! I'm Rob. I build OSPulse at Fortitude Omnis Group Ltd, a small indie company taking on the VC-funded supply-chain names (Snyk, Sonatype, Socket) with no funding and no sales team. The product page is the whole pitch.

What it does: most dependency tools wait for a CVE. I care about the window before one exists, where a package changes hands, an npm account gets hijacked, or a maintainer quietly burns out. event-stream, ua-parser-js, polyfill.io and xz all showed their signs at the source weeks or months before anyone assigned a number. So OSPulse scores project health, maintainer churn, commit-velocity collapse, ownership changes, instead of the length of a vulnerability list. It ingests more than ten feeds, and I'll tell you which detections it misses as readily as the ones it catches.

On the R&D side, the piece I'm proudest of is PoisonBox. Every npm install detonates in its own disposable microVM with hardware-level isolation. It boots, does its worst, and is deleted, so nothing it does touches your machine or your keys. Underneath, an eBPF probe in the kernel logs every process spawned, file opened, secret read and outbound connection, from below the code where the package can't hide. It then draws the causal thread from the instant a package reads your npm token to the connection that carried it off. With malicious packages the hard part is proving exactly what happened and in what order, and that trace is how.

A confession, since honesty is about the only thing separating a security tool from its own marketing. Our checkout showed £249 a month while Stripe was quietly set to take £2,988 in a single hit, a figure that appeared nowhere in the journey. Two systems each holding a correct version of the same number, disagreeing only at the instant of the charge, the one moment the website never sees. It's fixed now (the price is one constant everything derives from, with a test that fails the build if the disclosure ever drops off the page), and it was caught before a single customer paid, which is the only reason I get to tell it as a story instead of a refund.

Great to meet you all.

The fastest trigger produces no CVE

Developer at Fortitude Omnis Group — Mon, 14 Sep 2026 18:49:30 +0000

Every scanner you run is looking backwards.

A CVE is a record of something already found. By the time one exists, a researcher has spotted the flaw, someone has assigned it an identifier, and a database has published it. That chain takes time. The attacks worth worrying about most don't wait for it.

The uncomfortable part is what the industry's exploitation signals are built on. EPSS scores a CVE. CISA's Known Exploited Vulnerabilities catalogue lists CVEs seen being exploited in the wild. Your prioritisation queue sorts CVEs by severity. It's a decent machine for the problem it was built for, and completely blind to the attack that hasn't been given a number yet. That attack is usually the one moving fastest.

The moment that matters has no identifier

Look at how the supply-chain compromises everyone remembers actually unfolded.

event-stream, 2018. A popular npm package, millions of installs a week, maintained by someone who'd stopped caring about it. A stranger offered to take it over and the maintainer said yes, which was a perfectly ordinary thing to do. The new owner shipped a couple of clean releases, then added a dependency carrying code that targeted one specific bitcoin wallet. No CVE existed at the moment of the handover. The handover was the signal.

ua-parser-js, 2021. The maintainer's npm account was hijacked and three poisoned versions went out inside a few hours, carrying a cryptominer and a password stealer. The CVE came afterwards. Anyone who ran an install against a bad version in that window was hit before the advisory was written.

polyfill.io, 2024. No package, no CVE, nowhere for one to live. The domain serving a script to hundreds of thousands of sites was sold, and the new operator began injecting malware into what it returned. Control changed hands and the payload changed with it. No vulnerability database was ever going to tell you, because the whole event happened outside the model a database describes.

xz-utils, 2024. This one did get a CVE, and it's the most instructive of the lot. A contributor spent close to two years building trust, earning commit rights, and leaning on an exhausted maintainer, before staging a backdoor in the release tarballs. CVE-2024-3094 was assigned only because one engineer noticed his SSH logins had got about half a second slower and went looking. The backdoor took months to build, and the human signals were on display the entire time: a new maintainer taking over a critical library, an original author visibly worn down, and a published tarball that no longer matched the git history. None of that is a vulnerability. All of it was a warning, and it all predated the number by the better part of a year.

The shape repeats. Whatever made each package dangerous happened at the source, in the open, well before anyone recorded it. A scanner keyed on CVEs arrives after the interesting part is over.

What we watch instead

So we watch the source, and score the health of the project rather than the length of its vulnerability list.

In practice that means tracking the things that move before an advisory does. Commit velocity that falls off a cliff, which is what abandonment looks like from the outside. Maintainer churn, and the bus factor underneath it, so a critical library resting on one tired person is something you can see rather than discover. Ownership and control changes, the npm handovers and the domain sales, treated as events in their own right. Release artefacts that drift from the source they claim to be built from.

None of those needs a CVE to fire. Most of them precede one, and some of them, like polyfill.io, describe attacks that would never have produced a CVE at all. We're not trying to predict the future. We're trying to stand where the warning actually appears, at the time it appears, instead of waiting by the database for a number that shows up late or never.

What this isn't

We still ingest the CVE feeds, KEV, OSV, and more than ten others. A known, scored, actively exploited vulnerability in something you ship is a real problem, and you should hear about it fast. This doesn't replace that work. It covers the window that work can't reach, the stretch between a package turning dangerous and the world writing it down.

I'd rather be early and occasionally wrong about a project's health than perfectly precise about a CVE that landed a month too late to matter. The fastest trigger produces no CVE. If that's the trigger you care about, the CVE was never going to be where you found it.

Originally published on the OSPulse blog.

The inner loop is back, and it's a matmul

Developer at Fortitude Omnis Group — Sun, 30 Aug 2026 15:24:28 +0000

I started on 8-bit machines, writing games where you drew the triangles yourself and counted clock cycles in Intel's V-Tune until the inner loop stopped hurting. The whole job was getting under the abstraction and making the metal do exactly what you asked. Then compilers got good, machines got fast, and for twenty-odd years that skill went quiet.

It came back, because a language model spends almost all its time doing the one operation I used to hand-tune on a 386: multiply a matrix by a matrix, add up the results, move on. So I worked on an inference engine by hand, in WGSL, running a real model in a browser tab, on kernels I can read line by line. Here's the road to it, and why it tells you something about your prompts.

2008: a model is a pile of numbers you can save to disk

The instinct pointed me at neural networks in 2008, on a contract at the British Library. I wrote a 4 layer network for a project, starting it out in LINQPad (as I did with most throw away tests), in C#, to sort academic paper titles into Dewey Decimal subjects: physics, medicine, agriculture etc. Four layers. Turn the title into a bag of words, multiply by a matrix of weights, bend the result through a sigmoid so it can do more than draw straight lines, multiply again, read off the winner with a softmax cross entropy layer.

Two things stuck. The maths is a matmul and a curve, over and over. And the knowledge it learns is nothing but those weight matrices. In my pre-school AI level knowledge, I called them synapses. Once trained, you serialise them to a file and that file is the model, the same idea as the multi-gigabyte files on Hugging Face today, only mine was a few hundred kilobytes. The toy became real at the library: instead of my 62 sample titles I trained it on 1.6 million PDF titles, which took weeks on a 2008 Core i7 CPU . Expensive training makes the weights, a cheap forward pass uses them. That split is the whole LLM game, then and it's the same now.

The HLSL detour that paid off later

Around then I built a small library of WPF pixel-shader effects in HLSL, the shader language for DirectX: colourise, desaturate, a skin-shade for softening portraits. Nothing to do with machine learning, mostly an excuse to get back to assembly-era fiddling, this time on a GPU. It put HLSL in my hands a decade before I actually needed it. Skills you pick up for one thing sit on the shelf and pay off in another. I love keeping the saw sharp.

2019: I wanted to run one on the GPU myself

After the library I kept building nets for the fun of it. MNIST, the handwritten-digits, Hello World of Neural Networks. Then a CNN to spot a dog fouling a footpath, PoopScanner, which I was going to run on a Raspberry Pi and sell to councils with an audible "You're being watched. Clean that up.". The councils were not ready. Finally a network for recognising vehicle panel damage, this was ever so slightly before the frontier multimodal models came into being.

But...they're all the same shape underneath: multiply by a matrix, apply a curve, do it again. And from the shader work I already knew how to drive a GPU, rustily, slowly & badly, but enough to make progress. So in 2019 I put the two together and started an inference engine in HLSL, to watch the kernels do the work rather than call someone else's library (there's a Donald Knuth line at the end that sums up why). I got a chunk working, then life / job / contracting happened and it sat half-finished for nearly a decade. I have a whole bunch of dumpster fire code sat rotting away just like this one, that might be useful to resurrect if Claude can help take the strain ;)

Finishing it in a week, with an agent

A few weeks ago I picked it up again, and since I'm pretty much AI first I'll be straight about how. I sat down with Claude and we finished it together, rewritten from HLSL into WGSL so it runs in a browser tab. The design is mine and decades old, the kernels are hand-written and I can read you almost every line of tghe HLSL, but I won't pretend I typed every character anymore. Claude did a lot of the typing. The English I write to steer a model is now as real a language as the HLSL I wrote for the GPU, and this project wanted both, for the cost of a few hundred million tokens out of my 20x subscription.

The model I used in the demo is Qwen2.5-0.5B, downloaded from Hugging Face: a 265MB file of weights, the same species as my 2008 "synapses". Mine had a few thousand numbers, this has half a billion. Mine knew a few hundred words, this carries 151,936 pieces of text. Open it up and the inner loop is my C# toy again, 24 times over.

Tokens

You type words. The model sees tokens, chunks of text with a number attached. "Paris" might be one token, "unbelievably" three. The tokeniser that splits them is a lookup table of about 151,000 pieces plus a set of merge rules. I had written mine in 2018 in plain C# with the help of a pre-built stemmer - Stemmers.Net. Claude re-wrote it in JavaScript and tested it against the real Qwen one until they matched on every character, because if they disagree by a single token the model produces confident gibberish and you lose a day blaming the maths. So 50% mine I guess, but the important part was that I could write it if I had to.

Each token becomes a list of 896 numbers, its vector embedding. A prompt is a stack of those vectors, and everything after is arithmetic on the stack. The model does one job forever: score all 151,936 possible next tokens, take the highest, append it, run again. That word-by-word streaming in every chatbot isn't an animation. It's the real speed showing that loop! Every word is a full pass through the model.

The inner loop is still a matmul

In our tiny demo case, each pass runs the same block of maths 24 times: attention, then per-token crunching. Every step is dominated by one operation, a matrix multiply. The attention projections, the crunch, the final scoring against a 151,936-wide table, all matmul. So I wrote the matmul kernel by hand, and two old tricks from my assembly days make it fast.

Tiling. Memory is slow and there are millions of multiplies, so you don't fetch from GPU main memory each time. You grab a 16 by 16 block, park it in a scratchpad next to the compute units, and reuse it hard. Same instinct as hoisting a value out of a loop. This was similar to the 3d polygon filling with small textures on the CPU and using V-Tune to make sure that the cache never got thrown out.

Four-bit weights. The weights ship as 16-bit numbers, half a billion of them, about a gigabyte. Moving a gigabyte through the GPU per word is a real bottleneck, the fetching, not the multiplying. So I store each weight in 4 bits: take 32 of them, set a shared scale from the biggest, store the rest as small integers against it. A gigabyte becomes 265MB. You never unpack them back into memory, you unpack each one inside the loop at the moment you multiply. Cheap arithmetic to save expensive memory, the oldest trade there is.

On the LLM scale we're still on the tardigrade level - if you imagine GPT3 is ~175 billion parameters using 16bit floats, that's 2 bytes each amounting to ~350GB. That model is old, we're now entering the age of trillion weight models!

Attention is you, steering. Steering IS your job.

So, here's the part most explanations skip, and the one that connects the maths to what you do all day: writing prompts.

Every layer, each token looks back and works out how much each earlier token matters right now. Each makes a query and a key, you multiply one token's query by another's key, a big number means relevant, softmax turns the scores into percentages, and you blend the earlier tokens in those proportions.

So a prompt is you loading those weights by hand. Write "explain this simply" and the word "simply" sits in the context, and every word the model generates glances back at it and gets pulled towards something short and plain. Swap it for "rigorously" and the same weights write something denser, because the attention lands elsewhere.

You didn't change the model, you changed what it looks at.

That's the whole skill of prompt wording, no magic, just understanding the maths.

And because I wrote the attention kernel, I've got those weights in hand ready to surface in something you can see. Usually the model throws them away after each word. I don't. So the demo can draw them: run a prompt and watch the earlier tokens light up to show what each new word leaned on and see the actual % numbers, change one word and watch the lights move. It'll even knock out each word of your prompt in turn when you check the MEASURE STEERING checkbox and measure which ones genuinely steer the answer, so you can watch a word do work or watch it get ignored.

A 0.5B model with two attention heads is a small window so don't expect it to solve any complex problems beyond writing a haiku, and the big models will attend in subtler ways across many heads. But the mechanism is the mechanism, and seeing it once on something you can hold in your head changes how you read the large ones.

Honest numbers

I raced my version against transformers.js, Hugging Face's tuned browser runtime, same model and machine, both on WebGPU. I know I can't win, but want to know if I'm still employable ;)

I win the first word: first token on screen in about 450ms against their 590, because my prefill is lean with no framework on top. They win sustained speed: warmed up, about 20 tokens a second to my 12, because a large team has spent far longer than a fortnight tuning it. Arguably the most important stat. Ask both "capital of France, one word" and they give the identical token, Paris. Ask something open-ended and they drift apart after a sentence, because my four-bit rounding and theirs aren't bit-for-bit the same. They agree where there's a right answer and drift where there isn't.

That's the four-bit tax, visibly measured. Four bits is just a plaything here because I'm not bothered about showing 'correct' answers, only how we reach them

The point was never to win a benchmark. It was to understand every line, and get close enough to a tuned engine that "close" is the interesting word. I can live with my failure to match a team of 20.

Two traps, because you'll hit them

My output came out as pure garbage, all zeros then noise, and only once the whole thing was assembled, though every kernel passed its own test. The cause was one missing flag on a GPU buffer, the one that says "you may copy out of this". Without it, the copy filling the attention cache silently did nothing, so the model attended over zeros, which becomes divide-by-zero, which becomes NaN, which spreads through the network in a single pass, like a forest fire started with a dumb ass disposable barbecue. No crash, no console error, because WebGPU reports that class of mistake down a side channel you have to go and ask for. One flag, half a day. This never gets old.

The other: I set a starting value one notch past the largest 32-bit float. The shader compiler rejected the whole file for that one constant, silently, and handed me a kernel that ran and did nothing. If you write GPU code, the compiler failing in silence will cost you the most hours. THE MOST HOURS. Check your shaders compiled at all. My fault again. I think I preferred C++ STL compiler errors to this.

Finally, coding isn't fun if you can't open the box

There's a line from Knuth I keep coming back to.

The problem is that coding isn't fun if all you can do is call things out of a library, if you can't write the library yourself.

He wasn't gatekeeping dependencies. He was mourning the job turning into plumbing between black boxes you're never allowed to open, and pointing out that once you can see inside one you can usually improve it. The "don't use what you couldn't write" version is the folk edit. For most libraries I'd pass it anyway, which is how I know it was never the point.

Doing tasks like this is what keeps me sane in a world where 90% of the profession is exactly that, parameter plumbing using other's code. Dull, yes, but it pays the mortgage, so no one complains, and it's not your employer's job to make it interesting.

I could have just called transformers.js in four lines and had a chatbot by lunchtime, and for a product that's 100% the right call. But I wanted to open the box, to see what happens between the prompt and the words at the level of the arithmetic, because you can't tune what you can't see. Half a billion parameters, and at the bottom of the well it's still just a matmul I can read, and it's taken me nearly 20 years to get here.

Anyway, it's live: type a prompt, watch the tokens land, race it against transformers.js, and turn on the attention view to watch your words steer it, at attentionspan.fortitude-omnis.group. It's one of the builds from the Fortitude Omnis R&D lab. Forty years in, and understanding the inner loop is still as important as ever. Who'd have thought? Are we redundant yet?

Any questions, comment & I'll get back to you :)

I put my cost router on a neutral benchmark. It ranked near the bottom, and that's the interesting part

Developer at Fortitude Omnis Group — Wed, 26 Aug 2026 13:37:36 +0000

A few weeks back I repriced three months of my own Claude Code usage. Real traffic, not a demo: 39.5 billion tokens across 139,835 requests. At API rates that's about $30,000, and 91% of it went to Opus because that's what the default reaches for. Route the fraction a smaller model handles as well, and the bill drops by roughly 50 to 58%. Five or six grand a month, on work that was already done and graded.

That's why OmnisRouter exists. It sits in front of your agents, reads each request, and sends it to the cheapest model that can answer it. The full write-up and the method are in the flagship post.

Saving your own money on your own data is one thing. I wanted an outside check on somebody else's turf, with somebody else's scoring. So I took the router to RouterArena.

What RouterArena is

RouterArena is an independent benchmark for LLM routers, out of an ICLR 2026 paper. It's a proper piece of work: 809 queries in the quick split, spread across 39 datasets, everything from MMLU-Pro and code generation to translation, chess and medical multiple-choice. You plug your router in, it picks a model per query, the harness runs that model and grades the answer, and you get an accuracy number, a cost number, and a combined Arena score. There's a hard rule I stuck to: you evaluate on their data, you never tune on it. No gaming the test.

I wired OmnisRouter in with the pool I care about: GPT-5, Opus-5, Claude Haiku, and GPT-5-nano. The models a team running coding agents genuinely uses.

First, two bugs, because honesty starts at home

The first scored run came back at 42% accuracy, which was obviously wrong. So before I say a word about the result, here's what I found when I opened it up.

Opus was scoring 0% on every single query. The scoring path was grabbing the model's extended-thinking block instead of its answer text, so the grader was marking a raw thinking object wrong 40 times out of 40. Second, my query file was missing the per-dataset answer-format instruction the grader keys on, so most models answered correctly in prose and got marked wrong because the grader couldn't find the answer in the shape it expected.

Both fixed. Opus went from 0% to 85%, the overall number went from 42% to 72.7%, and abnormal entries went from 558 to zero. If you run a benchmark and the number looks too bad to be true, it usually is, and it's usually your plumbing.

The honest result

On its own pool, OmnisRouter scores 72.7% accuracy at $3.71 per thousand queries, for an Arena score of 0.669.

That ranks it 16th out of 18. Near the bottom. On the easier split, even.

I could stop there and let you think I'm burying it, so I won't. Here's the top of the cost board next to us:

router              accuracy   cost/1K   Arena
Paix2-router          79.7%    $0.27     0.776
cross-router          78.2%    $0.29     0.762
sqwish-router         76.4%    $0.16     0.754
hybrid-router         71.4%    $0.04     0.721
omnisrouter           72.7%    $3.71     0.669

Look at the cost column. The leaders come in at four cents to twenty-nine cents per thousand. We come in at $3.71. That's the whole gap, and it isn't routing skill.

What the cost board rewards

I went and looked at what the top routers route to. Every one of them runs a cheap open-model pool:

Paix2 sends 78% of its traffic to a single flash model.
cross-router and sqwish run mostly DeepSeek-v4-flash and Gemini-3.1-flash-lite.
hybrid-router puts 95% of everything through one Qwen model.

Not one of the leaders uses GPT-5, Claude, or any frontier model. And that makes sense, because the Arena cost axis rewards spend, so the way to win it is to route to the cheapest models that still clear the accuracy bar on an academic test set. Fair enough as a benchmark design. But it means the board measures how cheap your pool is at least as much as how well you route it, and a router built for the premium models people run their agents on starts the race carrying a piano.

I'm not knocking the benchmark, to be clear. It's careful and the accuracy scoring is sound. I'm saying the cost ranking answers a different question than the one I care about.

The warts, in full

"The board's unfair to premium pools" is a convenient thing for me to say, so I ran the check that could prove me wrong. RouterArena also computes optimality metrics: how close each router gets to the oracle, the best model per query, within its own pool. That strips out the cheap-pool advantage entirely and measures routing quality on its own.

Here's where we land:

router            Opt.Sel   Opt.Acc
Paix2-router       89.7%     100.0%
BARouter           64.4%      93.8%
agentforge         51.1%      98.7%
auto_router        37.7%      86.0%
r2-router          24.5%      99.8%
omnisrouter        23.7%      81.5%

Opt.Sel is how often the router picks the cheapest correct model. Opt.Acc is how much of its pool's achievable accuracy it captures. We're 8th of 13 on selection, and last of 13 on accuracy captured.

That last number is the honest one. 81.5% means OmnisRouter leaves more winnable accuracy on the table than anyone else on the board, and it does that on purpose. It's a cost-first router. Faced with a query a cheaper model can probably handle, it takes the cheaper model and the saving, even when the strongest model in the pool would have nudged the answer over the line. Tune it to chase accuracy and that number climbs, but then it stops being the thing I built. Some of the high selection scores above come from routers that barely route at all, by the way: park 95% of traffic on one cheap model that's usually the cheapest-correct and you "win" selection by barely choosing at all. I'd rather show you my mediocre 23.7% and tell you why than dress that up as skill.

The number that matters

A leaderboard rank isn't the point, and I'm not going to pretend a benchmark built around cheap open models is the scoreboard OmnisRouter should be judged on. The point is the bill.

On real Claude Code traffic, on the frontier models a working team runs every day, routing the cheaper-capable fraction cuts the cost by about half. The bill and the 91% Opus share are measured from three months of real requests. The saving is what comes off that bill once you route the work a smaller model handles, sized from the fraction OmnisBench measures as routable, so it's a reprice rather than a reroute I ran end to end. No academic test set moves it, and no cost board can rank it, because none of them run the models the work is happening on.

If your agent bill has a comma in it and you don't know where the money goes, that's the problem worth solving. The leaderboard was me checking my own work in public. Turns out the honest version is more useful than a good rank would've been.

OmnisRouter and the OmnisBench benchmark behind it are both open, Apache-2.0, and the numbers above are reproducible with RouterArena's own harness. Bring your own keys and check me.

I repriced 40 billion tokens of real AI coding. The bill goes where nobody tells you

Developer at Fortitude Omnis Group — Wed, 26 Aug 2026 08:23:37 +0000

Everyone selling an LLM router will tell you it saves 60% on your AI bill. Nobody shows you the numbers on a real workload. So I pulled my own.

Over the last three months I've run coding agents hard, Claude Code mostly, day in and day out. Every request it makes is logged locally with its token counts, so I parsed the lot: 1,231 sessions, 1.25 GB of logs, 139,835 billed requests, 1 June to 25 August. Here's what a real agent workload costs, and what routing would do to it.

The raw number

39.5 billion tokens. At Anthropic's API rates that's about $29,962, call it $10,000 a month.

Two things jump out.

First, 91% of that cost is Opus. Almost every request went to the biggest, most expensive model, whether it needed it or not. A one-line rename, a test refactor, a "summarise this diff", all of it to the frontier model at frontier prices. That's not a criticism of me. It's the default, and the default is expensive.

Second, and this surprised me, the bill is dominated by context, not output. The single biggest line is cache-read tokens, the whole conversation re-billed on every turn. By the hundredth turn of an agent session you're paying for the entire codebase again to add one function. Output is the small part.

What routing does to it

The pitch for routing is simple. Send the work a cheaper model can handle to a cheaper model, and escalate to the frontier only when you need it. The question is how much of the work is genuinely cheap-model-able.

My companion benchmark, OmnisBench, measures that on fresh coding problems the models can't have memorised. There, ideal routing matches the frontier model's quality at roughly 60% lower cost, because most coding requests don't need the biggest model.

Apply that to my $30k. Route the 60% of requests a cheaper model handles down to a small model, keep the rest on Opus, and the equivalent bill drops to somewhere between $12,500 and $15,600. A 48 to 58% cut. About $5,000 to $6,000 a month, on a $10,000 workload. That lands right on the benchmark's number.

The honest bits

Two things I won't paper over.

I didn't pay $30k. I ran this through a Claude Max subscription, which already saves something like 98% versus the API. So this isn't a bill I paid, it's a realistic workload standing in for a team that does pay API rates. If your team runs agents on your own keys, this is roughly your shape.

And the reprice is modelled, not proven. The 60% comes from the benchmark's aggregate, not from re-running every one of my requests through a cheaper model to confirm it passed. Proving that costs real API money, and it's the honest next step, not a number I'm going to pretend I already have.

Why I trust the direction anyway

Because there's now an independent check. RouterArena, an ICLR 2026 benchmark, ranks routers on accuracy against cost, and it found the loud commercial ones frequently over-pick expensive models. It's the neutral scoreboard the field didn't have. OmnisRouter is built for exactly what it measures: pick the cheapest model that's still correct, escalate when unsure, and show the receipt.

Do it yourself

The whole point is that you don't have to take my word for it. Your Claude Code logs are on your disk, with the token counts, in ~/.claude/projects. Parse them, price them at API rates, and see your own split. Mine was 91% Opus. I'd bet yours is close.

The three pieces, all open:

OmnisBench measures which model each kind of work needs, on fresh problems, re-gradable offline.
OmnisRouter routes each request to the cheapest capable model, with a receipt on every response.
OmnisVigil rolls it up per team, so a lead sees the runaway agent before the invoice does.

Tell me what your split looks like.

Links: github.com/Fortitude-Group/OmnisRouter · omnisbench.fortitude-omnis.group · omnisvigil.fortitude-omnis.group

We built a benchmark, then caught it strangling the models it was grading

Developer at Fortitude Omnis Group — Fri, 21 Aug 2026 23:37:46 +0000

A couple of day ago I posted about OmnisBench, our open benchmark for LLM routing, specifically our LLM Router OmnisRouter , and made a fuss about how you can re-grade every number yourself because we publish the actual model responses. Two commenters, deanlee and jugeni, very politely pointed out that the whole thing might be resting on a fib.

Their point: HumanEval and GSM8K are old. The models have almost certainly read the answers. So when I cheerfully reported that the cheapest model already scores 94.5% and routing only recovers the last few points, that's maybe less "small models are quietly brilliant" and more "small models have seen the exam paper." Fair. Annoyingly fair.

So we did the thing you're supposed to do and actually built the fresh split we said we would.

The fresh split

The idea's simple. Take LiveCodeBench, which stamps every problem with a release date, and keep only the ones published after the models could plausibly have trained on them. Grade the same routing policies on the old, probably-memorised tasks and on the new, definitely-not tasks, side by side. If the routing story only survives on the old set, you deserve to know.

I ran it. Here's where it took a turn.

The first numbers, and a brief crisis of faith

They came back grim. The cheap model fell off a cliff. Even the big expensive frontier model, the one whose entire job is to be good at this, was landing well under where a frontier model has any business landing. My first thought was that we'd accidentally built the most damning benchmark in the history of our own company, and my second was whether it was too late to go back to doing something honest for a living.

Then I remembered the one feature I keep banging on about: we publish every response. So instead of trusting the sad little number, I opened the responses and read them.

Every one of the frontier model's failures on the hard problems was the same thing: an empty answer. Not a wrong answer. Nothing at all. Eleven problems, eleven blank pages, each one exactly 4,096 tokens of the model thinking out loud and then having the microphone cut off before it wrote a single line of code.

Here's what happened. Our request budget was 4,096 output tokens, a number that's perfectly sensible for "write this short Python function" and perfectly ridiculous for "here's a genuinely hard competitive-programming problem, think it through." The reasoning models did exactly what reasoning models do. They thought. At length. And then they hit the ceiling somewhere in the middle of it and got cut off before writing a line of the solution.

So the benchmark wasn't measuring how good the models are at hard problems. It was measuring how good they are at running out of room. We'd built, with some effort, a very reproducible way to gag a model and then mark it absent.

I'd love to tell you we spotted this through rigour and foresight. We spotted it because the responses were sitting right there in the file, empty, quietly judging us. Which is, if you squint, exactly the argument for publishing them. A closed benchmark makes this same mistake and ships the number, and everyone nods, because who's going to check.

The corrected numbers

We made the output budget a config setting, gave the hard suite the room it needed, and ran it again.

Policy	Likely-contaminated (20)	Fresh, 2025+ (15)
cheapest model only (nano)	90.0%	60.0%
ideal routing (oracle)	100.0%	93.3%
always the frontier model	100.0%	86.7%
how often ideal routing reached for the frontier	0%	20%

Three things fall out of that, now the models can actually finish. The frontier model's fine: 86.7% on the fresh problems, not the panicked 60% the truncated run reported. The cheap model, though, drops from 90% on the old benchmarks to 60% on the fresh ones, which is the contamination showing through exactly as predicted. And routing earns far more on the fresh set: on the old tasks it buys ten points over the cheap model, on the fresh tasks it buys thirty-three, while reaching for the expensive model a fifth of the time instead of never. The honest asterisk: the fresh set's competitive-programming, which is both newer and harder than grade-school maths, so difficulty and freshness are tangled together here, and the samples are small. It's a first signal, not a verdict.

What I'll actually stand behind

Two things, and I'll keep them honest because you can check both.

The contamination point stands. Even with the models given all the room they wanted, the cheap one's thirty points worse on problems it can't have memorised. The "small models are quietly brilliant" story was, in part, small models sitting an exam they'd already seen.

The harness point is the one I didn't expect to be writing about, and it's the more useful one. Half of what looks like a model being bad is a benchmark being broken, and you can't tell the difference from a leaderboard. You can only tell by reading the answers. If a routing benchmark, or a routing vendor, won't show you the responses, you've no way of knowing whether their impressive number is a real result or a bug wearing a nice suit.

Cost of finding all this out: about $7.47. Four dollars of that was the run that produced the numbers above; the rest was the earlier runs that produced the wrong ones and taught us the lesson. I'm counting the mistakes, because a benchmark that hides its own costs has no business lecturing anyone about hidden numbers.

Thanks to deanlee and jugeni for the nudge. The suite grows from here, more fresh problems and harder ones, and every number stays re-gradable offline with omnisbench verify.

Results: https://omnisbench.fortitude-omnis.group/
Code: https://github.com/Fortitude-Group/OmnisBench

OmnisBench is built by Fortitude Omnis. We make small, sharp tools, and when we catch ourselves measuring the wrong thing, we write it up rather than quietly fixing the chart.

I built an LLM router that hands you a receipt for every request

Developer at Fortitude Omnis Group — Thu, 20 Aug 2026 11:15:55 +0000

A router that decides where your money goes is asking for a lot of trust. It sits in front of every request, quietly picks a model, and sends you a bill at the end of the month. If it tells you it saved you 60%, you mostly have to take its word for it.

I didn't want to take its word for it. So OmnisRouter attaches a receipt to every response, and the receipt is the whole point.

It's an open source (Apache 2.0), self-hosted proxy that speaks the Anthropic, OpenAI and Gemini formats. You change one base URL, keep your own provider keys, and each request goes to the cheapest model that can actually handle it. The response comes back in your client's own format, with a note explaining what was chosen and what it cost.

The receipt

Every response carries a set of headers:

X-Omnis-Model: gemini/gemini-2.5-flash
X-Omnis-Decision: Routed
X-Omnis-Confidence: 0.26
X-Omnis-Policy: v3-omnisbench-2026-08-20
X-Omnis-Cost-Delta-Vs-Big: -0.0121

And if you want to know what it'd do before spending anything, there's a cost-free endpoint that returns the full decision without calling a provider:

curl -s localhost:8080/v1/route -H "authorization: bearer $TOKEN" \
     -d '{"messages":[{"role":"user","content":"Summarize this thread."}]}'

{ "policy_version": "v3-omnisbench-2026-08-20",
  "decision": "ROUTED", "reason": "cheapest_capable",
  "chosen": { "provider": "gemini", "model_id": "gemini-2.5-flash" },
  "est_cost_delta_vs_big_usd": -0.0121 }

No screenshot, no dashboard you have to trust. The decision's in the response, and you can log it.

How it actually decides

There's no network hop to work out where to send a request. A small pinned ONNX model (bge-small-en-v1.5) embeds the prompt in-process, the embedding maps to the nearest intent cluster, and the cluster's policy table picks the cheapest candidate that clears the quality bar for that kind of work. If it isn't confident, it escalates to a strong model rather than guessing. The whole thing happens in well under the 50ms I gave it as a budget.

Here's a live trace from the shipped model:

Request	Routed to	Decision
Summarize this thread	gemini-2.5-flash	routed (cheap)
Prove sqrt(2) is irrational	gpt-5-nano	routed (cheap)
Write a Python function to merge two lists	gpt-5	routed (strong)
Hello.	claude-opus-5	escalated

The maths row is the interesting one. On grade-school maths, the tiny gpt-5-nano is essentially as accurate as the frontier model, so OmnisRouter sends the work there and keeps the roughly 25x price difference. That's not a guess. It comes from measurement.

Where the numbers come from

The routing model isn't a black box I'm asking you to believe in. The intent clusters and the policy table ship in the repo and rebuild from public data, so the same inputs produce the same model, and every decision is stamped with the policy version that made it.

The coding and maths policy is driven by OmnisBench, a companion benchmark that grades each model per task and publishes every response so you can re-grade it offline. So when the router says "the cheap model is good enough here", that's a number you can go and check, not a line on a slide. Other domains use sensible estimates for now, and the benchmark coverage grows from there.

The honest bit

Cheapest-capable isn't the same as cheapest. The point isn't to slam everything into the smallest model and hope. It's to spend frontier money only where the work needs it, and to be able to show, per request, why it went where it went. When a capability can't be carried faithfully to the chosen provider, it refuses with an explicit error instead of silently dropping it. Surprising the first time, correct every time.

It runs as a single self-hosted process with an embedded database, your keys are encrypted at rest, and prompt content only ever leaves your infrastructure to go to the model you chose.

Have a look

Site and the full routing story: https://omnisrouter.fortitude-omnis.group/
Code: https://github.com/Fortitude-Group/OmnisRouter
The benchmark behind the routing: https://omnisbench.fortitude-omnis.group/

If you can make it route something badly, or you think the receipt should carry more than it does, tell me. That's what the open model and the decision log are for.

OmnisRouter is built by Fortitude Omnis. We make small, sharp tools, and we try not to lie in our own marketing.

Everyone is saving 60% on LLM costs. Nobody will show you the numbers.

Developer at Fortitude Omnis Group — Wed, 19 Aug 2026 13:05:03 +0000

Every few weeks the same post does the rounds. Someone's routing layer cut their LLM bill by 60%, "with no drop in quality", and here's a lovely chart to prove it.

Except it doesn't prove anything. The eval set is private. The grading is private. The actual model responses that produced that score are never published. You're being asked to trust a screenshot, and we've been in I.T. long enough to know roughly what a screenshot is worth.

So we built the thing that was missing. It's called OmnisBench, it's open source (Apache 2.0), and its whole job is to measure how good LLM routing actually is, in a way you can check for yourself.

What routing is, and why it should save money

If you're not knee deep in this yet: an LLM router looks at each request and sends it to the cheapest model that can still do the job, rather than firing everything at one expensive frontier model. Easy prompts go to something small. Hard ones go to the big model. In theory you keep most of the quality and pay a fraction of the bill.

That's the theory. OmnisBench measures the practice.

The number we care about most is what we call the oracle. For every task, the oracle picks the cheapest model that actually got it right. We know which ones did, because we ran them all. That is not a router you can ship. It's the ceiling. It's the best any router could possibly do on a given set of tasks, and it tells you how much money is genuinely on the table before anyone starts making claims.

The first run

We ran it over 364 tasks: HumanEval (164 coding problems, graded by their own unit tests) and GSM8K (200 maths problems, graded on the answer). The model pool was Claude Opus 5, GPT-5, Claude Haiku 4.5 and GPT-5-nano.

Here is what came out.

Policy	What it does	Task success	Cost per 1,000 requests
oracle	cheapest model that actually solved each task	99.7%	$0.62
always-frontier	everything to Claude Opus 5	99.2%	$6.25
random	pick a model at random	96.2%	$4.01
always-cheapest	everything to GPT-5-nano	94.5%	$0.43

So the headline writes itself. Ideal routing hit 99.7% success for roughly 90% less than sending everything to the top model. Very nice.

Now let me talk you out of it a little.

The honest bit

Two things you will not see on most of those 60% charts.

First, the oracle is a ceiling, not a product. It cheats. It knows, after the fact, which model got each task right, and no live router has that luxury. The real score for a router you can actually deploy is how close it gets to that oracle line, and the gap is the interesting part. Anyone quoting an oracle-style number as their shipping product is selling you the ceiling.

Second, and this is the one that surprised us: the cheapest model on its own already scored 94.5%. On this particular set of tasks GPT-5-nano is just quietly good. So routing is not buying you a magic 60%. It's buying you the last few points of quality without paying frontier prices for all of it. That is still very much worth having. It just isn't the fairy tale.

We would rather tell you that up front than have you find out after you have quoted the shiny number to your boss.

The part that actually matters: you can check it

This is the whole reason OmnisBench exists. It does not ask you to trust the table above. It publishes the actual model response for every task, and ships a command that re-grades the lot offline.

pip install -e .
python scripts/prepare_datasets.py                                  # HumanEval + GSM8K
python -m omnisbench.cli run    --config configs/v0.yaml --run runs/mine
python -m omnisbench.cli report --run runs/mine
python -m omnisbench.cli verify --run runs/2026-08-19               # re-grades the published run, no API calls

verify re-runs the graders against the published responses and rebuilds the entire leaderboard, cost included, without calling a single API or needing a key. Tamper with one stored answer and it fails. That is the point. If our numbers are wrong, you can prove it in about a minute, and we would genuinely like you to try.

Have at it

The results, the method and the honest caveats are all on the site, and the code is on GitHub:

Results: https://omnisbench.fortitude-omnis.group/
Code: https://github.com/Fortitude-Group/OmnisBench

We also build an actual open router, OmnisRouter, which this benchmark grades on every release, so it can't quietly regress and hope nobody notices. It is live now at https://omnisrouter.fortitude-omnis.group/.

If you can break the numbers, poke a hole in the method, or you just reckon we've measured the wrong thing, tell us. That is what open and verifiable is for.

OmnisBench is built by Fortitude Omnis. We make small, sharp tools, and we try not to lie in our own marketing.