My app generates personalized readings for BaZi — Chinese "Four Pillars" birth charts. Every reading is an LLM call, every call costs money, and the domain is full of trap terminology that models love to botch. So before launch I benchmarked every candidate model on my actual workload, and then built the routing layer around what the benchmark found.
The results generalize to any "LLM in a niche domain" app, so here they are — including the part where the most expensive model lost to one costing 5.8× less.
What "good" means in a niche domain
Generic benchmarks were useless to me. My acceptance criteria were:
- Domain accuracy: 甲 is Yang Wood. A model that renders it "Yin Wood" in English output is not 5% wrong, it's categorically wrong — the way a compiler that flips one bit is wrong.
- No invented jargon: the system has a closed vocabulary (the Ten Gods, fixed star names). A model that confidently introduces terms my engine never computed is a liability.
- Cost per reading, because a free tier exists and every free reading is marketing spend.
Running my own corpus through the candidates produced findings no leaderboard would have surfaced:
-
The flagship preview won't let you turn thinking off. Hybrid reasoning models "think" by default, and the preview snapshot rejects the flag outright (
400 InvalidParameter: The value of the enable_thinking parameter is restricted to True), so you pay for the inner monologue whether you want it or not. On one streamed request that was 286 reasoning events before the first character of the actual answer: 2.1s to the first reasoning token, 10.5s to the first character a user can read. On list price the flagship already costs 5.8× the mid-tier model per call; the reasoning tokens bill on top of that, for prose I could not tell apart. Excluded. - Three mid-tier models flunked domain accuracy — English chart output with elements flipped (Yang Wood → "Yin Wood" class of errors). Excluded regardless of price.
- The "character roleplay" fine-tunes hallucinated worst of all — invented relationships between the Ten Gods that don't exist in the system. The models optimized for persona were the least safe choice for a persona product. Excluded.
- Two well-known open-weight models had quietly been delisted from the provider's international endpoint between planning and testing. A model choice is a dependency with an EOL you don't control.
What survived: a cheap-and-accurate small model for the free tier, and a mid-tier model for paid — with the surprise that the mid-tier's previous generation was equally accurate at lower cost, which is exactly what you want in a fallback.
Price is data: keep it next to the routing
The eval's outputs — which models are allowed, in what order, at what price — live in one file. A Route is a provider (endpoint + key) plus a model plus that model's list price:
const PRICE: Record<string, [number, number]> = {
'small-fast': [0.1, 0.4], // USD per 1M tokens, in/out
'mid-plus': [0.4, 1.6],
'mid-plus-prev':[0.5, 3.0],
'flagship': [2.5, 7.5],
}
const DEFAULT_CHAINS: Record<Tier, string[]> = {
free: ['small-fast', 'legacy-plus'],
paid: ['mid-plus', 'mid-plus-prev', 'flagship'],
}
Each tier gets an ordered fallback chain: the head is the workhorse, the tail is who serves the request when the workhorse can't. If a backup API key is configured, the chain ends with the primary model on the backup account — because when your account balance dies, every model on it dies together, and only a different key helps.
"The model failed" is three different problems
The subtle part of fallback chains isn't trying the next model — it's knowing when the next model helps at all. Every failure gets classified into one of three moves:
function classify(e: unknown): 'retry' | 'next' | 'fatal' {
if (e instanceof LLMHttpError) {
const { status, body } = e
if (status === 401 || status === 403) return 'fatal' // new model won't fix your key
if (status >= 500) return 'retry' // transient, same route
if (status === 429)
return /RateQuota|rate limit/i.test(body) ? 'retry' : 'next'
if (status === 400 || status === 404)
return /model|not.?found|InvalidParameter/i.test(body) ? 'next' : 'fatal'
return 'next'
}
return 'retry' // network-layer: ECONNRESET, DNS, timeout
}
The one that bites people: 429 is two different errors wearing one status code. Rate-limit throttling is transient — back off and retry the same model. Quota/allocation exhaustion is not — retrying the same model just burns time; skip to the next route. You can only tell them apart by sniffing the response body, and the distinction is provider-specific. Learn your provider's error taxonomy; it's load-bearing.
When the whole chain is exhausted, the app returns placeholder text with an ok: false flag — and the flag exists because of a real trap: never persist a fallback stub. A paid user whose reading gets cached as "(placeholder)" sees that placeholder on every revisit, forever, and the system never retries because a cached reading exists. ok gates the database write; failures stay ephemeral and self-heal on the next request.
Cost telemetry that survives fallback
Every business action (one reading = up to 7 parallel calls) emits a usage event, and each call's cost is computed against the model that actually served it, not the one you intended:
const intended = primaryModel(tier)
const fellBack = served.some((m) => m !== intended)
capture('llm_usage', {
kind, tier, model: servedModels, primary_model: intended,
fell_back: fellBack, input_tokens, output_tokens, cost_usd,
})
fell_back: true is the alert condition — it means your workhorse is degraded and your margins quietly changed. With this wiring, real numbers per call (~3.7k in / 0.4k out): $0.0021 on the paid-tier model, $0.0005 on the free-tier one — so a two-call free reading lands near $0.001. Those aren't estimates; they're what the meter read.
Takeaways
- Benchmark on your own corpus. Leaderboards can't see that your domain has a closed vocabulary, and "most capable" models can be your worst performers on it.
-
enable_thinking(or your provider's equivalent) is the biggest single cost lever on hybrid reasoning models — and verify each snapshot actually honors it. A preview build that rejects the flag bills you for reasoning tokens on every call, on top of an already higher list price. - Classify failures before you retry. Same-model retry, next-model failover, and give-up-now are different errors sharing status codes.
- End the chain with a different account, not a different model. Balance exhaustion kills models in bulk.
- Never persist fallback output. Gate the cache write on "this is real content."
-
Emit cost per actual served model with a
fell_backflag. Silent fallback is silent margin change.
The app all this serves is auspiceoracle.com — a bilingual BaZi calculator where a deterministic engine computes the chart and the LLM is only allowed to phrase it. That constraint is its own article (next in the series).
Top comments (0)