Every time I put a model behind an endpoint I make the same lazy decision. I pick whatever I used last time, or whatever I read about most recently...
For further actions, you may consider blocking this person and/or reporting abuse
The detector inherits the shape of the bug it explains. It fires only when the provider both bills the reasoning and reports that count in a field your client happens to read, which is the same dependency that hid
delta.reasoning_contentin the first place: a truncation with no reasoning counter, or the count nested somewhere you are not looking, produces the same empty column with nothing to explain it. The check that survives a provider swap is on the payload itself, since zero characters of content against a non-zero billed completion is a failure whatever caused it, andreasoning_tokensthen labels the cause instead of being what detects it.You're right, and I've fixed it. The check was
!body.textContent && d.reasoning_tokens, which means it only ever fired when the provider both billed the reasoning and reported the count in the one field I happened to read. A truncation with no counter, or the count nested somewhere else, lands in exactly the same silent column with nothing to explain it. The detector inheriting the shape of the bug is a good way to put it.It now keys on content characters against billed completion tokens, so zero text with a non-zero bill is the failure regardless of cause, and reasoning_tokens is only used to name the cause when the provider reports one. I also moved the count server-side into the done event so the case is testable instead of inferred from the DOM, and added a regression test for exactly your scenario: billed 512, no content, no reasoning count reported.
Thanks, that was a real hole in published code.
The variable your harness holds constant is the one that surprised me most. I ran the same model on two providers instead of six models on one:
Same weights, 2.8x the time to first token. That gap is wider than most model-to-model pairs I measured, which makes provider a bigger lever than model choice for anything latency-sensitive. Your setup already measures it too, since only the model string changes - point two rows at the same model through different base_urls and the columns race providers instead.
One caveat for anyone reusing single-run numbers off a shared tier: NVIDIA gave me anywhere from 27 to 100 tok/s across repeats of the identical request. The swing was bigger than several of the gaps I was trying to rank.
2.8x on the same weights is a bigger spread than almost anything I measured between models, which is a genuinely uncomfortable result for a post that ranks models.
You're right that the harness is already most of the way there. Everything is per-column except the client itself, which is built once from a single base_url, so a column identity of (base_url, model) rather than just model is the actual change. The parts that need thought are the catalog cache being per-endpoint and prices.json being keyed on model alone. I've opened it as an issue rather than hand-waving it: github.com/oceanforge/inference-shootout/issues/1
Your 27 to 100 tok/s swing is the more uncomfortable half though. My measurement set used three runs per cell for that reason, but the app itself still races once and prints one number, which invites people to read a 200ms gap as signal. That's issue #2. Single-run numbers off a shared tier deserve the warning you gave them.
Cost comparisons become actionable when they include the quality floor and the retry rate. The cheapest model per call is not always cheapest per accepted result, especially once the workflow needs repair prompts or human review.
Agreed, and it's the honest limitation of what I measured. Latency, tokens and cost, no quality floor at all, so "cheapest" here means cheapest per call, which only equals cheapest per accepted result if every model clears your bar.
My own data makes your point by accident: qwen3.5-397b-a17b returned no readable text in 7 of 9 runs while billing in full. Per call it's just another row. Per accepted result its cost is infinite.
I don't think LLM-as-judge belongs in an app this small, since you then own the judge's cost and failure modes too. A thumbs up/down per column with an effective-cost column derived from it would keep the human in the loop and would have made the Qwen result obvious immediately. Opened as github.com/oceanforge/inference-shootout/issues/3
The catalog-lies section reproduced for me on a different provider, with a different status code. Groq's /models listed 14 models; only 9 actually accept a chat completion. The other five are Whisper/TTS/guard models sitting in the same array with no modality field, and they fail 400, not 403, so you cannot tell "wrong model type" from "malformed request" without calling each one. The one that cost me real time was groq/compound: it is a router, and its 429 names openai/gpt-oss-120b rather than itself, which is the only place the routing is visible from outside. Your one-connection-per-model isolation is the right call for exactly that reason, since this class of failure arrives per-model rather than globally.
That is a sharper version of the finding than mine, because it kills the shortcut I was implicitly hoping for. On DigitalOcean every unreachable model gave the same 403, tier-gated, so I could at least say "call it and read the 403". Your Groq numbers show the code is not portable: 400 for a wrong-modality model sitting in the same array with no modality field, where I got 403 for a subscription wall. Same lie, different status, so you cannot even key on the code. The only honest probe is one real chat completion per id, and you have to read the body, not the number.
The
groq/compoundrouter is the part I did not have. A 429 that namesopenai/gpt-oss-120binstead of itself means the catalog id and the runtime identity are two different things, and the only place they reconcile is inside an error you were not expecting to parse. That is worse than a dead model, because it fails as another model.On the isolation: I used one connection per model because I expected a model to 403 on its own while the others kept streaming, which is what happened on DigitalOcean about sixty seconds into the first run. A router that answers as a different model was not a case I pictured, but it shows up in one column and leaves the rest alone, same as a plain dead model would.
The 'cheapest one won' result matches what I keep hitting. I ran the same inference benchmark across free tiers instead — HF Spaces (free CPU), Ollama on my own box, and Colab's free T4 — and the winner wasn't the fastest model, it was the one whose cold-start didn't eat the latency budget. On bursty side-project traffic, cold start dominates p99 way more than tokens/sec.
One thing I'd love to see in your six-model race: did you measure time-to-first-token separately from total completion time? In my runs they ranked differently — a model that was 2nd on TTFT finished 5th on total time because its decode speed was poor.
What's your take — for a low-traffic side project, would you rather pay per-token on a fast model or eat cold starts on a free tier?
Yes, separately, and they do rank differently exactly as you describe. TTFT and total are the first two columns:
llama-4-maverick was 2nd on first token at 676 ms but 5th on total at 17.9 s. deepseek-3.2 was 3rd on TTFT at 869 ms and 3rd on total at 5.4 s. So a model can start fast and still finish last, which is the same decode-speed effect you hit.
On your question: for a low-traffic side project I'd pay per token, but not for the reason I expected before running this. The whole 54-call experiment cost $0.0185, so per-token pricing at side-project volume is effectively free and cold starts buy you nothing. What actually changed my mind was that cheap and fast turned out to be the same model here rather than a tradeoff, so there was no premium to avoid. Your cold-start point stands for anything user-facing though: p99 is where that shows up and an average hides it completely.