DEV Community: synthorai

Claude Fable 5 for Agents: Tool-Call Refusals, Cost vs GLM 5.2

synthorai — Tue, 07 Jul 2026 05:49:17 +0000

Claude Fable 5 refused mid-tool-call on 11 of 44 coding-agent turns in our eval, on tasks as mundane as fixing a config default. The refusal arrives as stop_reason: "refusal" partway through generating the tool arguments, the truncated arguments still parse as valid JSON, and an agent loop that executes tool calls without checking the stop reason will happily write a half-finished file to disk. That behavior, not the price tag, is the first thing to engineer around when you put Fable 5 in an agent.

TL;DR

Claude Fable 5 returned stop_reason: "refusal" mid-tool-call on mundane agent tasks (a config-default fix, a meeting-room booking); the truncated write_file arguments still parsed, so a loop that doesn't check the stop reason executes half-written files.
Fable 5's thinking is adaptive, with no off switch: enabled and disabled are both rejected; the control is output_config.effort.
Fable 5's cost premium is shape-dependent: a four-turn coding task ran $0.045 vs $0.003 on glm-5.2 (15x), but only 5x sonnet-5 on warm batch work.
Fable 5 requires 30-day data retention.

Everything below was measured on 2026-07-05 through the Synthorai gateway with a small scenario harness: five agent workload shapes (a tool-using coding loop, RAG question answering, tool-heavy orchestration, batch classification, and a 15-turn conversation), run against claude-fable-5, claude-opus-4-8, claude-sonnet-5, and glm-5.2, three runs per task where variance matters. The tasks are deliberately trivial; pass rates are a sanity gate, not a capability benchmark. Costs are the gateway's billed usage.cost.

Check stop_reason before executing tool calls

This is the failure the docs don't warn you about, and it corrupts state. The agent reads app.py, decides to write the fix, and starts emitting a write_file call. Partway through the file content, the stream stops:

{
  "stop_reason": "refusal",
  "content": [{
    "type": "tool_use",
    "name": "write_file",
    "input": {
      "path": "app.py",
      "content": "DEFAULTS = {\n    \"timeout_s\": 30,\n    "
    }
  }]
}

The input object is complete, parseable JSON. Nothing about it says "I stopped early." If your loop's contract is "got tool calls, run them," you just overwrote app.py with a 38-character fragment that ends mid-dictionary and no longer parses as Python, and the next turn is a refusal too, so the loop ends with the workspace corrupted.

Three things we can say from the data:

It triggers on mundane work. The tasks that drew refusals were fixing a KeyError in a config lookup, implementing a slugify function, booking a meeting room, and creating a draft invoice. Nothing dual-use, nothing sensitive.
It is repeatable, not random. One coding task drew a refusal on all three runs, streaming and non-streaming alike. Other tasks never drew one. Across conditions, 58-75% of our trivial coding episodes passed on Fable 5 versus 100% for claude-opus-4-8, claude-sonnet-5, and glm-5.2, and every single failure traces to a refusal, not to wrong code.
Once a refusal is in the conversation, the episode is done. Follow-up turns returned stop_reason: "refusal" with empty output. Retrying within the same context did not recover.

The trigger is not the task content, and the data is blunt about it. The task that refused every run was a nine-line KeyError fix in a config dictionary, no credentials, no exploits. Meanwhile the batch scenario classified support tickets about cryptomining, leaked Stripe keys, and phishing pages without a single refusal, and the RAG scenario answered over docs full of AES-256-GCM secrets and breach-response procedures, also clean. Every refusal landed in the two multi-turn, tool-executing scenarios; the three single-shot scenarios never refused once, even carrying heavier content. The pattern is the agent-loop shape, not the words, which means sanitizing your inputs won't prevent it.

The fix is one line before your tool-execution step:

if response.stop_reason == "refusal":
    # do NOT execute tool calls from this turn: arguments may be truncated
    raise AgentInterrupted("model refused; restart episode or escalate")

Anthropic documents the mechanics: a refusal that fires before any output returns an empty content array and is not billed; a mid-stream refusal bills the already-streamed output, and the guidance is to discard the partial. The response also carries a stop_details object with a category (such as cyber or bio, or null) so you can tell classifier blocks from ordinary declines. What the documentation doesn't spell out is the interaction with tool use we hit above: the refusal can land inside argument generation, and the partial arguments are indistinguishable from complete ones.

There is also an official recovery path. On the Claude API, the beta fallbacks parameter (betas: ["server-side-fallback-2026-06-01"], fallbacks: [{"model": "claude-opus-4-8"}]) re-runs a declined request on a fallback model inside the same call, with the decline itself unbilled when it fired pre-output. It is not available on Amazon Bedrock, Vertex AI, or Microsoft Foundry, where the SDKs ship a client-side fallback middleware instead. Whichever path applies, the guard above still comes first: never execute tool calls from a turn whose stop reason is a refusal.

What five agent shapes cost

Median cost per completed unit (task, query, item, or conversation), same prompts, same day:

Scenario	fable-5	opus-4-8	sonnet-5	glm-5.2
Coding loop (per task, 4 turns median)	$0.045	$0.012	$0.0059	$0.0031
RAG answer (per query)	$0.024	$0.0075	$0.0036	$0.0031
Tool orchestration (per task)	$0.048	$0.011	$0.0045	$0.0027
Batch classification (per item, warm)	$0.0024	$0.0012	$0.00046	$0.00057
15-turn conversation (whole)	$0.94	$0.34	$0.26	$0.083

Two readings of that table matter more than any single cell:

The cheapest model changes with the shape. glm-5.2 wins the loops and the long conversation, but claude-sonnet-5 is the cheapest batch classifier in the set, under glm-5.2, because its introductory price rides on a 97% cache-read share once the scaffold prompt is warm.
Fable 5's premium is shape-dependent too: 15x glm-5.2 on the coding loop, 11x on conversation, but only 5x sonnet-5 on warm batch items, where caching absorbs most of the prompt.

The rest of the cost picture is about controlling those numbers, and then about two things that quietly push them back up.

Keeping an agent's bill down

Caching is the single biggest lever, and on Fable 5 its contract is unchanged. The agent data shows what it is worth: with cache_control markers removed, the same coding task cost 2.0x more, and warm batch items 6.8x more. On opus-4-8 the same ablation reads 3.8x and 6.9x. In a loop, the sliding-marker pattern is not an optimization, it is the difference between a viable bill and not.

Prompt order is the second lever, and it held up across every model we ran. Putting stable rules before per-query context (instead of after it) made RAG queries 26-37% cheaper on all four models; on the Claude line, the wrong order additionally pays the 1.25x cache-write premium on every call. The mechanics are in the LangChain caching post; the numbers here just confirm they apply unchanged to Fable 5.

Fable 5 also adds two levers of its own. The first is that the cache-eligibility floor dropped to 2,048 tokens, half of Opus 4.8's 4,096. That reads like trivia until you remember where an agent's savings come from: the repeated scaffold (system prompt, tool definitions, the sliding conversation prefix) is what caches, and only if it clears the floor. A tool-heavy agent whose per-turn prefix sat between 2,048 and 4,096 tokens got no caching at all on Opus 4.8, and starts caching on Fable 5, turning a full-price prefix into a roughly 10%-of-price cache read on every subsequent turn. It cuts the other way too: a prefix padded to clear the old 4,096 floor may now be carrying dead weight. Read cache_read_input_tokens off a live response rather than assuming, because on Fable 5 the discount begins earlier.

The second is task budgets (beta, header task-budgets-2026-03-13), which address the exact problem this comparison keeps surfacing: a Fable 5 loop runs up a bill fast, and max_tokens does not help. It is a hard per-response cap the model cannot see, so the model plans as if it has unlimited room, then gets cut off mid-thought. A task budget is different. You give the loop a token ceiling (minimum 20,000) that the model sees as a running countdown and paces itself against, wrapping up gracefully instead of being truncated. It counts what the model generates plus the tool results it reads on the turn, not the full history you resend each request. On a model whose coding-loop turn costs 15x glm-5.2, a budget the model self-moderates toward is the cheapest guardrail you can bolt on.

Two cost surprises the docs won't flag

With the levers set, two smaller things still moved our bill in directions the documentation doesn't warn you about.

"Low" effort was not cheaper. Fable 5's thinking depth is controlled by output_config.effort, and the intuition is that low costs less. It didn't. Setting effort: "low" ran our coding loop at $0.0478 per task versus $0.0451 at the default, with more output tokens, not fewer. We saw the same pattern on GLM 5.2, where the effort names don't track token counts either. On both model lines, measure the knob on your own workload before assuming "low" means "less." One reason the number is hard to predict: adaptive thinking's share of output tokens swung from 2% on the coding loop to 30% on RAG answers to 52% on batch classification, same model, same day. Budget output tokens per workload shape, not per model.

Never replay reasoning_content. On OpenAI-compatible models, the reasoning field is not conversation history. DeepSeek's API requires stripping it; on GLM 5.2 replaying it is legal but billed. Feeding it back into the message history inflated our GLM loop cost by about 28% until we stripped it. Anthropic's own thinking blocks are different: on the same model you must replay them unchanged, but a Fable 5 thinking block routed to a different model (say, on a fallback to Opus) is dropped from the prompt automatically and not billed, so there's nothing to strip.

What changed in the request surface

Fable 5 shares most of its request surface with Opus 4.7/4.8 and Sonnet 5. What's gone, per the docs:

thinking: {type: "enabled", budget_tokens: N} returns a 400. Extended thinking with a token budget, the mechanism from Claude 3.7 Sonnet through the 4.5 family, is retired across the 4.7+ line in favor of adaptive thinking.
thinking: {type: "disabled"} returns a 400, and this one is unique to Fable 5. Opus 4.7/4.8 and Sonnet 5 still let you switch thinking off; Fable 5 does not.
temperature, top_p, and top_k are rejected at any non-default value.
Assistant-message prefills (a trailing assistant turn) return a 400.

The temperature/top_p/top_k and prefill removals are the two that most often bite a ported request; the thinking and retention changes are covered above and in the retention post.

Bottom line

Fable 5 in an agent is an engineering problem before it is a budget problem. Handle stop_reason: "refusal" before executing tool calls, or a truncated write will corrupt state on a task as boring as a config fix. Then treat the cost as something you shape: caching is the biggest lever, the eligibility floor is now 2,048 tokens so re-check your prefix, a task budget keeps a loop from running up the highest per-turn bill in this comparison, and effort: "low" is not the discount its name implies. Budget by workload shape, too: the same model is 15x the cost of glm-5.2 on a coding loop and 5x sonnet-5 on warm batch work. None of this says use it or don't; it says the defaults are not neutral, and the bill and the failure modes both depend on how your agent is shaped.

FAQ

Does Fable 5 refuse tool calls often?
It concentrated on specific tasks: one config-fix task refused on every run, others never did, and the same tasks reproduced under both streaming and non-streaming calls. So it is not a rare flake you can retry past. Rates on your workload will differ; the engineering answer is the same either way: check stop_reason before executing tool calls.

Can I turn Fable 5's thinking off?
No. thinking.type.disabled and enabled are both rejected. Thinking is adaptive by default and output_config.effort is the only control; in our loop, low effort did not reduce cost.

Is Fable 5 ever the cheap option?
Not in this set. Its smallest premium is on warm cache-heavy batch work, about 5x sonnet-5, where a warm cache absorbs most of the prompt. On the loops and the long conversation it is the most expensive model we ran.

Verification: all figures measured 2026-07-05 against https://synthorai.io/ (Anthropic-native /v1/messages for the Claude line, /v1/chat/completions for glm-5.2), 505 episodes and 1,022 calls across five scenario shapes, three runs per task where variance matters. Costs are the gateway-reported usage.cost; medians shown. Tasks are deliberately simple, so pass rates are a sanity gate, not a capability benchmark; we don't publish capability claims we haven't measured. Refusal behavior reproduced in both streaming and non-streaming modes. Your numbers will vary with prompts, region, and load.

LLM Prompt Caching #5: LangChain Setups That Actually Hit

synthorai — Sat, 04 Jul 2026 09:41:35 +0000

Here is a LangChain system prompt that looks perfectly reasonable and caches nothing:

from langchain_core.prompts import ChatPromptTemplate

prompt = ChatPromptTemplate.from_messages([
    ("system", BIG_STABLE_SYSTEM_PROMPT),   # the syntax every tutorial uses
    ("human", "{question}"),
])

We ran this against claude-sonnet-5 twice with an identical 1,800-token system prompt and read the usage fields back. Both calls: cache writes 0, cache reads 0. Not a partial hit, not a fragmented cache. Nothing. The reason: Anthropic only caches what you mark with cache_control, and a plain string in a ("system", ...) tuple has nowhere to put the marker. The most convenient syntax in LangChain is also the one that leaves the entire discount on the table, and no error tells you.

This is part 5 of the caching series. Part 1 covers how prefix caching works, part 3 does the raw-SDK tutorial. This part is about what changes when LangChain assembles your prompts for you. Everything below was measured on 2026-07-04 through the Synthorai gateway with langchain-core 1.4.8, langchain-anthropic 1.4.8, and langchain-openai 1.3.3.

First, which "caching" are you looking for?

Two unrelated features share the word, and the LangChain docs page you land on when you search is usually the wrong one.

	Response caching (LangChain's `InMemoryCache`)	Prompt caching (this series)
What it stores	The whole completion, in your app	The prompt prefix's KV state, on the provider
When it saves money	The exact same request repeats	Different requests share a prefix
Where	`set_llm_cache(InMemoryCache())`, SQLite, Redis	`cache_control` markers or automatic prefix matching
Agent loops, RAG, chat	Almost useless (every request differs)	The main lever, since system + tools repeat every turn

And "exact same request" means exact: the built-ins key on the pair (serialized prompt, model-config string). Measured: an identical repeat returned in 0 ms with no API call; adding one space to the prompt missed; the same prompt with max_tokens changed by one also missed. (The cached replay also returns the original call's usage numbers, so naive token accounting double-counts.) Semantic caches exist as third-party integrations; the built-ins are exact-match only.

So set_llm_cache is fine for deduplicating identical calls in tests; the 2,000-token system prompt you re-send on every agent turn is prompt caching's job, and it needs the prompt assembled the right way.

The fix: content blocks, not strings

cache_control travels inside a content block, so the system message has to be a SystemMessage with block content rather than a bare string:

from langchain_anthropic import ChatAnthropic
from langchain_core.messages import SystemMessage
from langchain_core.prompts import ChatPromptTemplate

llm = ChatAnthropic(
    model="claude-sonnet-5",
    base_url="https://synthorai.io",   # any Anthropic-compatible endpoint
)

prompt = ChatPromptTemplate.from_messages([
    SystemMessage(content=[{
        "type": "text",
        "text": BIG_STABLE_SYSTEM_PROMPT,
        "cache_control": {"type": "ephemeral"},   # a bare string has nowhere to put this
    }]),
    ("human", "{question}"),
])
chain = prompt | llm

Same 1,800-token system prompt, measured through the same gateway:

Call	String-tuple syntax	Content-block syntax
1st (cold)	write 0 / read 0	write 1,875 / read 0
2nd, different question	write 0 / read 0	write 0 / read 1,875

A warm read bills at roughly 10% of the input price, so on Claude this one structural change is the difference between paying full price forever and a 90% discount on the stable half of every call. The economics are in part 1; the marker mechanics match the raw SDK usage in the LangChain Anthropic integration docs and Anthropic's prompt caching guide.

Where your template variables go decides your hit rate

LangChain templates make it effortless to interpolate variables anywhere, which is exactly the hazard. The cache key is the byte-exact prefix. We put a date inside the cached block and measured:

SystemMessage(content=[{
    "type": "text",
    "text": f"Today is {today}. " + BIG_STABLE_SYSTEM_PROMPT,   # variable INSIDE the block
    "cache_control": {"type": "ephemeral"},
}])

Call	Result
day A, question 1	write 1,865 (cold for this value)
day A, question 2	read 1,865 (same value, hit)
day B, question 1	write 1,865 (new value, cold again)

The cache did not break. It got keyed on the variable. A value that repeats, like a date, costs one cache write per value and hits after that. A value that is unique per call, like a timestamp or a request ID, makes every call a cold write and the hit rate exactly zero.

The expensive real-world version of this mistake is RAG. The template many chains reach for puts the retrieved context at the top of the system prompt, before the static instructions. We measured both orders with an 800-token retrieved context that changes per query and a marked system block:

Order inside the prompt	Call 1	Call 2 (new query, new context)
Context first, then rules	write 3,133	write 3,133 again, read 0
Rules first (marked), context in the human turn	write 1,852	read 1,852

The wrong row is not merely "no discount": every call pays the cache-write premium, about 1.25× the normal input price, on the full 3,133 tokens, and nothing is ever read back. A mis-ordered RAG prompt with caching enabled costs more than not caching at all. The fixed content sits after the variable content, so it might as well not exist.

The rule that falls out of the measurements:

Static text first, inside the marked block. System rules, tool definitions, few-shot examples.
Anything that varies goes after the marker, ideally in the human turn: retrieved context, dates, user questions.
A variable inside the block is acceptable only if it repeats often enough to amortize its own cache write.

Tool definitions get cached too

An agent re-sends its tool schemas on every call, and in Anthropic's request layout the tools sit before the system prompt. Since a marker means "cache everything from the start of the request up to here," that raises two practical questions. Does a marker on the system block also cover the tools in front of it? And does LangChain's bind_tools turn your tools into the exact same bytes on every call, because if the serialization wobbles, the prefix changes and every call misses.

Measured answers to both. With the same marked system prompt, the warm cache read was 1,861 tokens without tools and 2,389 tokens with two tools bound: the extra 528 tokens are the tool schemas coming back out of the cache. And that 2,389 repeated exactly across three consecutive calls, which means bind_tools serializes the same way every time; the framework does not leak noise into the prefix. So to be explicit: as long as the system block carries the marker, the tools themselves need no cache_control; that one marker behind them does all the work.

The opposite arrangement exists for one specific shape: the tools are the biggest stable thing you send and the system prompt is thin or absent. Then the request still needs a marker somewhere, and it can live on a tool. This only works with the raw Anthropic-format dict, because a @tool-decorated function has no field to carry it; bind_tools passes the dict through untouched:

# variant: NO marked system block anywhere; the tool carries the request's only marker
llm.bind_tools([{
    "name": "get_weather",
    "description": LONG_TOOL_DESCRIPTION,
    "input_schema": {...},
    "cache_control": {"type": "ephemeral"},   # passes through bind_tools verbatim
}])

Measured: write 3,002 cold, read 3,002 warm, with no marked system message in the request.

Multi-turn: move the marker to the last message

A conversation looks like another ordering problem, but it is the opposite case: the order is already perfect, because history only ever appends, so the whole transcript is stable prefix. The problem here is coverage. A marker on the system block caches the system block and nothing after it: as the history grows, the warm read stays flat at the system size while every accumulated turn bills as ordinary input.

The pattern that fixes it is the same one the raw SDK uses: put the marker on the latest message, so the breakpoint slides forward and the whole conversation-so-far becomes the cached prefix:

def marked(text):
    return HumanMessage(content=[{
        "type": "text", "text": text,
        "cache_control": {"type": "ephemeral"},
    }])

# each turn: history stays plain, only the newest human message carries the marker
llm.invoke([system, *history, marked(new_question)])

Measured across two turns: turn 1 wrote 1,864; turn 2 read 1,864 and wrote only the 15-token delta (the previous answer plus the new question), the prior prefix billing at the ≈10% read rate. That is the shape an agent loop wants, and LangChain expresses it with an ordinary message list. Anthropic allows up to four markers per request, so the sliding marker composes with a fixed marker on the system block or on the tools.

Read the meters, and know their names

LangChain standardizes usage into usage_metadata, and here is the gotcha we hit: with langchain-anthropic 1.4.8, on every response in our runs, the standard input_token_details.cache_creation field stayed 0 even when a cache write happened. The real write count lands in a nonstandard key:

r = chain.invoke({"question": "..."})
det = r.usage_metadata["input_token_details"]
det["cache_read"]                  # correct on hits (1875 above)
det["cache_creation"]              # 0 even on a cold write; do not alert on this
det["ephemeral_5m_input_tokens"]   # the actual write count (1875)

The provider reported the write correctly (cache_creation_input_tokens: 1875 in the raw response, visible via r.response_metadata["usage"]); the standardized mapping just files it under the TTL-bucket key. A cost dashboard watching cache_creation will tell you caching is free while the write premium quietly accrues. Trust the raw usage object or know the bucket keys. This is the same class of problem as gateways mis-reporting cache fields, which we audit in Does Your LLM Gateway Lie About Cache?.

Implicit caches: mis-ordering fails silently, so watch it hardest here

Claude's cache is explicit. GPT and most open-weight providers cache automatically on prefix match, no markers, and through LangChain the same chain works after one constructor change:

llm = ChatOpenAI(model="glm-5.2", base_url="https://synthorai.io/v1")

Plain string system prompt, no markers: GLM 5.2's second call read 1,088 tokens of the roughly 1,850-token prefix. (Not all of it: automatic caches match in coarse block increments rather than byte-for-byte to the end; OpenAI, for instance, documents 128-token granularity.) So far, free money. But the mis-ordering hazard from the RAG table above applies with full force here, and with a nastier failure mode. We reran the same order experiment on the automatic path, new retrieved context on every call:

Order (no markers, automatic cache)	Call 1	Call 2 (new query, new context)
Context first, then rules	read 0	read 0
Rules first, context in the human turn	read 0	read 1,088

The wrong order is an unconditional zero: the changing context sits at the front, no two calls share a prefix, and the discount never arrives. On the explicit path the same mistake at least shows up in the bill as a cache-write premium on every call; on the implicit path there is no premium, no error, and no signal. The prompt just quietly never qualifies, while you assume "automatic" means "working." And since there is no marker to place, prompt order is the only knob the implicit path gives you.

So verify from the meters, in production rather than once in a test: input_token_details.cache_read (LangChain) or prompt_tokens_details.cached_tokens (raw). OpenAI's automatic caching additionally documents a 1,024-token minimum prefix, and per-provider TTL and eligibility differ, which is part 2's territory.

Checklist

On Claude, a ("system", "...") string tuple has nowhere to carry cache_control: nothing gets cached, nothing warns you. Cacheable system prompts go in a SystemMessage with content blocks and the marker.
The cache key is the byte-exact prefix: static content first, variables after the marker or in the human turn. RAG context before the rules does not just miss; it pays the write premium every call.
A variable inside the cached block creates one cache entry per value: repeating values amortize, per-call-unique values (timestamps, request IDs) never hit.
Tools sit before the system prompt in the prefix, so the system marker caches bound tools too (bind_tools serializes deterministically). If tools are your biggest stable block, the marker can go on an Anthropic-format tool dict instead.
In conversations, a marker fixed on the system block leaves the growing history at full price; put it on the latest message so each turn reads the prior prefix and writes only the delta.
Do not monitor input_token_details.cache_creation: it stays 0 even on writes, so a dashboard concludes caching is free while write premiums accrue. The real count is in ephemeral_5m_input_tokens, or read the raw response_metadata["usage"].
On automatic-cache models (GPT, GLM, DeepSeek), prompt order is the only knob and mis-ordering fails silently: no premium, no error, just a discount that never arrives. Verify hits from the usage fields.
set_llm_cache stores whole responses keyed on the exact prompt and model config; it only pays off when identical requests repeat, never on an agent loop.

The habits are small: a content block instead of a string, static before variable, a marker that slides with the conversation, one usage field read correctly. The measured difference was a 90% discount on every stable token versus nothing, and in the mis-ordered RAG case, versus paying extra. LangChain does not get in the way of prompt caching; it just makes the wrong prompt shape as easy to write as the right one.

Disclaimer

Measured on 2026-07-04 against https://synthorai.io/ with langchain-core 1.4.8, langchain-anthropic 1.4.8, langchain-openai 1.3.3, models claude-sonnet-5 and glm-5.2, a roughly 1,800-token English system prefix, small samples, and a 1–2 second gap between consecutive calls so cache writes have time to land. Each experiment used a fresh randomized prefix to guarantee a cold cache, which is why the baseline token counts differ slightly between tables (1,852 to 1,875). Library field mappings and provider cache behavior change between versions; re-measure against your own stack before depending on the numbers.

Sources

Claude Sonnet 5's New Tokenizer: 41% More Tokens per Prompt

synthorai — Thu, 02 Jul 2026 09:25:46 +0000

claude-sonnet-5 is live on the Synthorai gateway, and right now it is cheap: $2 / $10 per million input / output tokens, which is 2.5× under Opus 4.8 and below Sonnet 4.6. Enjoy it while it lasts. That is introductory pricing through August 31, 2026; on September 1 the rate returns to $3 / $15, the same sticker as Sonnet 4.6.

If you cache against the Claude line, the caching and TTL contract is a drop-in carry-over. Cost is where you have to look twice, and the reason is how Sonnet 5 counts tokens. It ships with a new tokenizer that turns the same English text into about 41% more input tokens than Sonnet 4.6, and token count is what you pay on and are limited by. The sticker price is only half the bill.

Here is what that token change touches, before any code change or quality question enters the picture:

Cost per prompt. At the standard rate, the same English prompt costs about 41% more than on Sonnet 4.6, since identical text is billed as more tokens at the same per-token price.
Every token-based estimate. A per-call budget, or a local-tokenizer count, sized against 4.6 runs about 40% low on Sonnet 5. Meter the live usage, not a local guess.
Context-window headroom. The same document eats about 41% more of the window, so long-context and RAG calls fit less real text per request.
Rate limits. A tokens-per-minute cap drains about 41% faster for the same workload, trimming throughput.
Cache eligibility (a small upside). The 1,024-token minimum is easier to clear, so a prefix that sat just under it on 4.6 may become cacheable on Sonnet 5.

The rest of the post puts measured numbers on each: price, the caching economics, and the token-count shift.

Prices, caching, TTL, and token counts measured against https://synthorai.io/ (Anthropic-native /v1/messages) on 2026-07-01. Per-token prices are derived from the usage cost on live calls; the intro/standard rates and the August 31 expiry are from Anthropic's announcement. Reproduce against your own prompt before quoting.

Availability

import os
from anthropic import Anthropic

anth = Anthropic(
    api_key=os.environ["SYNTHORAI_KEY"],
    base_url="https://synthorai.io/",   # SDK appends /v1/messages
)

msg = anth.messages.create(
    model="claude-sonnet-5",            # the only line that changes
    max_tokens=512,
    system=[
        {"type": "text", "text": SYSTEM_PROMPT,
         "cache_control": {"type": "ephemeral"}},
    ],
    messages=[{"role": "user", "content": question}],
)
print(msg.usage)   # cache_creation_input_tokens, cache_read_input_tokens, cost

Swap the model field and nothing in your caching path moves. The mechanics behind cache_control are in the caching tutorial; the architecture of why the cache exists is in Part 1 of the series.

Price: cheap now, back to Sonnet 4.6's rate in September

Per-token pricing on the gateway, derived from the usage cost on plain (uncached) calls:

Model	Input ($/M)	Output ($/M)
`claude-sonnet-5` (intro, through Aug 31)	$2.00	$10.00
`claude-sonnet-5` (standard, from Sep 1)	$3.00	$15.00
`claude-sonnet-4-6`	$3.00	$15.00
`claude-opus-4-8`	$5.00	$25.00

The intro rate is a real discount, and against Opus 4.8 it is the durable part of the story: even at the standard $3 / $15, Sonnet 5 stays cheaper than Opus, and the two share a tokenizer (more on that below), so the comparison is clean at both prices.

Against Sonnet 4.6 the discount is temporary. On September 1 the sticker price is identical, so any "Sonnet 5 is cheaper than 4.6" plan built on today's number expires with the promo. And as the next section shows, at equal sticker price Sonnet 5 is actually the pricier of the two for the same text.

We don't publish capability benchmarks we haven't run; whether Sonnet 5's quality justifies its cost over 4.6 is your eval, not ours.

Caching and TTL: a drop-in

The caching contract is identical to the rest of the Claude line. We ran a cold write / warm read sequence with a stable 2.2K-token prefix, varying the user message each call so no response-level cache could contaminate the result. Cost per warm turn, at the current intro price:

Model	Cold turn (cache write)	Warm turn (cache read)	Cold → warm
`claude-sonnet-5` (intro)	$0.0069	$0.0017	4.0×
`claude-sonnet-4-6`	$0.0079	$0.0024	3.3×
`claude-opus-4-8`	$0.0172	$0.0043	4.0×

The invariants hold as they do across the Opus line:

Read discount ≈ 90%. A warm cache read costs about 10% of the input price, matching Anthropic's documented "up to 90%" cached-read savings. Break-even is one hit.
1-hour TTL works the same. cache_control: {"type": "ephemeral", "ttl": "1h"} is accepted on Sonnet 5, and the usage object splits the buckets as before: cache_creation.ephemeral_5m_input_tokens vs ephemeral_1h_input_tokens. The 1-hour write premium is about 2× no-cache (vs about 1.25× for the 5-minute write); reads stay ≈10% regardless of TTL.

One caveat on the table: those warm-turn dollars are at the intro rate. From September 1, multiply the Sonnet 5 figures by 1.5× ($2 → $3 input, $10 → $15 output). A warm Sonnet 5 turn that costs $0.0017 today is about $0.0026 in September, still under Opus 4.8's $0.0043, but no longer under Sonnet 4.6.

The token-count catch

Here is what makes the September reset bite twice. The same system text reports about 41% more input tokens on Sonnet 5 than on Sonnet 4.6.

Model	Input tokens (identical text)	Input cost at standard price
`claude-sonnet-4-6`	1,594	$0.0048
`claude-sonnet-5`	2,245	$0.0067
`claude-opus-4-8`	2,245	$0.0112

Sonnet 5 tokenizes the same English prompt as 2,245 tokens, the identical count Opus 4.8 reports, and well above Sonnet 4.6's 1,594. Sonnet 5 shipped with the newer tokenizer the Opus line adopted at 4.7.

Put the price and the token count together and the picture is clear:

During the intro period, the 41% token bump is offset by the 33% lower rate ($2 vs $3), so the same uncached prompt costs about what it did on 4.6, and warm turns run cheaper thanks to the discounted output.
From September 1, the rate matches 4.6 but the token count does not. The same English prompt costs about 41% more on Sonnet 5 than on Sonnet 4.6 ($0.0067 vs $0.0048 for this prefix), because identical text is simply counted as more tokens at the same per-token price.

Against Opus 4.8 there is no such catch: the tokenizer is the same (2,245 = 2,245), so Sonnet 5 is cleanly cheaper at both the intro rate (2.5×) and the standard rate (1.67×).

So budget the September bill, not the July one: the per-token rate rises 1.5× on September 1, and the higher token count is already baked in today. And read cache_creation_input_tokens / cache_read_input_tokens from the live response rather than a local tokenizer that may still be on the old vocabulary.

Sonnet 5 vs Opus 4.8: the durable win

This is the comparison the launch changes for keeps. Sonnet 5 and Opus 4.8 share a tokenizer, so on any prompt the token counts are identical and the cost difference is purely the rate: 2.5× cheaper at the intro price, 1.67× cheaper at the standard price, on cold turns, warm turns, input, and output alike. A warm cached turn is $0.0017 vs $0.0043 today; even in September it is roughly $0.0026 vs $0.0043.

For a high-volume caching agent loop where the prefix repeats every turn, that gap compounds. The decision is the usual one: run your own eval, and if Sonnet 5 clears your quality bar, the gateway math favors it durably, not just until August. If it doesn't, Opus 4.8 is one model field away with the same caching code.

Migration checklist

✅ Caching code carries over verbatim. cache_control markers, breakpoint count, ttl: "1h", usage field names are all identical to the Opus line.
✅ TTL choices carry over. 5m for live/session workloads, 1h for bursty or agent-with-pauses work.
✅ Discount economics carry over. ≈90% read, ≈1.25× write (5m), ≈2× write (1h).
⚠️ Mark September 1 on the budget. The intro rate ends Aug 31; Sonnet 5 goes to $3 / $15. Model the 1.5× step-up before it lands.
⚠️ Re-measure token counts (from 4.6 or earlier). Same text, about 41% more tokens on Sonnet 5. At standard pricing that makes the same prompt pricier than 4.6, not cheaper.
⚠️ Trust the live usage object. Read *_input_tokens and cost from the response, not a cached estimate from the old generation.

Bottom line

Sonnet 5 is a strong deal on a clock. Against Opus 4.8 it is a durable 1.67–2.5× cheaper with a drop-in caching path, which makes it the obvious first thing to eval for any Opus workload that isn't quality-critical. Against Sonnet 4.6 the win is only the introductory discount: on September 1 the price matches 4.6, and the new tokenizer means the same prompt actually costs more. Take the discount, but size your budget on the September numbers and confirm your token counts against the live usage object before you promise finance anything.

For the full caching playbook, see the four-part series starting with How KV Cache & TTL Work and the working Python tutorial.

FAQ

Is Sonnet 5 cheaper than Sonnet 4.6?
Only during the introductory period. Through August 31, 2026 it is $2 / $10 vs 4.6's $3 / $15. From September 1 it is $3 / $15, the same rate. And because the same text counts as about 41% more tokens on Sonnet 5, at the standard price the same prompt costs more than on 4.6.

When does the intro price end?
August 31, 2026, per Anthropic's announcement. On September 1 the rate becomes $3 per million input and $15 per million output tokens.

How much cheaper is Sonnet 5 than Opus 4.8?
2.5× at the intro rate, 1.67× at the standard rate, on both input and output. They share a tokenizer, so token counts match and the difference is purely the rate, at both prices.

Do I need to change my cache_control code?
No. Marker syntax, breakpoint limit, and TTL options are identical to the Opus line. Change the model field and nothing else. Warm reads are ≈10% of the input price; the 1-hour write is ≈2× no-cache, the 5-minute write ≈1.25×.

Is Sonnet 5 a drop-in replacement for Opus 4.8?
On the caching, TTL, and cost surface, migration is trivial and it is cheaper at both prices. On quality, run your own eval; we don't publish capability benchmarks we haven't run. For model-quality claims, see Anthropic's model card.

Verification: price, caching, TTL, and token-count figures measured against https://synthorai.io/ on 2026-07-01 using the Anthropic-native /v1/messages path, single tenant. Per-token prices are derived from usage cost on plain calls; cost-per-turn is a small-sample median with a 2.2K-token cached prefix and reflects the current intro rate. Intro pricing and the August 31, 2026 expiry are from Anthropic's Sonnet 5 announcement; discount/premium ratios cross-checked against Anthropic Prompt Caching docs. Your numbers will vary with prompt, region, and load.

GLM 5.2 Tool Calls in Agent Loops: What 'OpenAI-Compatible' Hides

synthorai — Tue, 30 Jun 2026 12:06:27 +0000

Point an existing OpenAI-style agent loop at GLM 5.2 and most of it just works: you send tools, you get back tool_calls, you run them, you send the results. Then it does something the SDK examples never show. The assistant returns a line of text in the same turn as the tool calls:

{
  "choices": [{
    "finish_reason": "tool_calls",
    "message": {
      "role": "assistant",
      "content": "I'll look up both pieces of information for you at the same time!",
      "tool_calls": [
        {"id": "call_…", "type": "function",
         "function": {"name": "get_weather", "arguments": "{\"city\":\"Paris\"}"}},
        {"id": "call_…", "type": "function",
         "function": {"name": "get_time", "arguments": "{\"city\":\"Tokyo\"}"}}
      ]
    }
  }]
}

Two conventions dominate, and it helps to keep both in view. In OpenAI's, you send function schemas, get tool_calls back, and answer with a tool message per call, keyed by tool_call_id:

resp = openai.chat.completions.create(model="…", tools=tools, tool_choice="auto", messages=messages)
# assistant.tool_calls → [{"id": "call_…", "function": {"name": "get_weather", "arguments": "{\"city\":\"Paris\"}"}}]
messages.append(resp.choices[0].message)
messages.append({"role": "tool", "tool_call_id": "call_…", "content": "18C, clear"})

Anthropic's is shaped differently: tools carry an input_schema, the model emits tool_use blocks, and you answer with a tool_result block:

resp = anthropic.messages.create(model="…", tools=tools, messages=messages)
# resp.content → [{"type": "tool_use", "id": "toolu_…", "name": "get_weather", "input": {"city": "Paris"}}]
messages.append({"role": "assistant", "content": resp.content})
messages.append({"role": "user", "content": [
    {"type": "tool_result", "tool_use_id": "toolu_…", "content": "18C, clear"}]})

GLM 5.2 speaks the OpenAI dialect.

In OpenAI's contract, message.content is null when finish_reason is tool_calls. Plenty of agent loops lean on that: they branch on "content or tool calls," log content as the final answer, or assert it's empty. GLM hands you both at once, and that assumption is the first thing to break.

The behavior here was captured from live tool-calling requests to glm-5.2, with gpt-5.5 and claude-opus-4-8 run on the same task as reference points. The short version: GLM 5.2 uses the OpenAI API surface, but on a couple of axes it behaves more like Claude than like GPT, and an OpenAI-trained loop is the one that trips.

The same turn, three ways

Same prompt, same two tools, three models:

	GLM (`glm-5.2`)	OpenAI (`gpt-5.5`)	Anthropic (`claude-opus-4-8`)
API surface	OpenAI chat-completions	OpenAI chat-completions	Anthropic messages
Text in the tool-call turn	`content` preamble (non-null)	`content` is `null`	a `text` block before `tool_use`
Reasoning on that turn	exposed: `reasoning_content` + `reasoning_tokens`	hidden; only `reasoning_tokens` in `usage`	only as a `thinking` block, if you enable it
Parallel tool calls	yes, with `index`	yes	yes, multiple `tool_use` blocks
Done signal	`finish_reason: "tool_calls"`	`finish_reason: "tool_calls"`	`stop_reason: "tool_use"`
Tool-call id prefix	`call_…`	`call_…`	`toolu_…`

Two rows are where loops break: text in the tool-call turn, and reasoning showing up on that turn. The rest is reassuringly boring.

Text rides with the tool call

GLM 5.2 routinely emits a short assistant content preamble alongside tool_calls, with finish_reason: "tool_calls". It is not an error and it is not occasional.

Here is the same turn from all three, trimmed to the part that differs:

// OpenAI gpt-5.5: content is null on a tool-call turn
"message": { "content": null,
             "tool_calls": [ {/* get_weather */}, {/* get_time */} ] }

// GLM glm-5.2: content carries a preamble
"message": { "content": "I'll look up both pieces of information for you at the same time!",
             "tool_calls": [ {/* get_weather */}, {/* get_time */} ] }

// Anthropic claude-opus-4-8: a text block sits before the tool_use blocks
"content": [ { "type": "text", "text": "I'll get both pieces of information for you." },
             { "type": "tool_use", /* get_weather */ },
             { "type": "tool_use", /* get_time */ } ]

OpenAI leaves content null; GLM fills it; Anthropic has always put a text block there. So GLM takes OpenAI's wire format with Anthropic's habit of narrating before it acts, and a loop written against OpenAI is the one caught off guard. The fix is small but you have to make it deliberately. Stop treating a tool-call turn as content-free:

resp = client.chat.completions.create(model="glm-5.2", messages=msgs, tools=tools)
msg = resp.choices[0].message

# GLM may return assistant text in the same turn as the tool calls.
if msg.content:
    log.debug("preamble: %s", msg.content)   # keep or drop, but don't assume it's empty

msgs.append(msg)
for call in msg.tool_calls:
    result = dispatch(call.function.name, json.loads(call.function.arguments))
    msgs.append({"role": "tool", "tool_call_id": call.id, "content": result})

If your loop renders content to the user as the assistant's reply, you will now show a "let me check that" line before every tool call. Decide whether you want it. The point is that the decision is yours, not something the model's silence makes for you.

It thinks out loud

GLM 5.2 is a reasoning model, and that does not pause for tool use. A tool-call turn carries reasoning along with it, and GLM 5.2 exposes it as text. In a non-streaming response the token accounting makes it explicit:

"usage": {
  "prompt_tokens": 224,
  "completion_tokens": 68,
  "completion_tokens_details": { "reasoning_tokens": 30 },
  "total_tokens": 292
}

Almost half the completion was reasoning, on a request whose visible output is two short function calls. This is the row where all three models diverge. GLM 5.2 gives you the reasoning as reasoning_content plus a token count. OpenAI bills reasoning_tokens in usage but never shows the text. Anthropic shows it only as thinking blocks, and only when you turn extended thinking on. GLM 5.2 is the most exposed of the three by default.

Two consequences. First, cost: you pay for those reasoning tokens on tool-call turns, and an agent loop is many turns. Reasoning effort is the dial that moves the number, which we covered in GLM 5.2: Reasoning Effort Is the Cost Lever. Count reasoning tokens on every turn, not just the final answer.

Second, streaming order. When you stream the request, GLM sends the reasoning first, then the preamble text, then the tool calls:

reasoning_content  (many deltas)
content            (a few deltas)
tool_calls         (id + name, then arguments)

A parser written against vanilla OpenAI chat completions does not know the reasoning_content field and will quietly ignore that opening burst. Usually fine. It stops being fine if your UI shows a "thinking…" state keyed on the first content delta, because the first thing on the wire is reasoning, not content, and the indicator never flips.

What a GLM 5.2 tool-call turn costs

Behavior is half the story; the bill is the other half, and an agent loop runs the same turn many times over. With a fixed prefix (a roughly 2,000-token system prompt plus the tool definitions) and the user message varied each call, measured over ten warm turns:

per warm tool-call turn	GLM `glm-5.2`	OpenAI `gpt-5.5`	Anthropic `claude-opus-4-8`
Cost	$0.0009	$0.0042	$0.0051
Latency (median)	6.6s	1.9s	3.1s
Prompt cached	≈96%	≈81%	≈97%
Reasoning tokens	≈27	0	0
Cold → warm cost	3.4×	2.8×	4.9×

GLM 5.2 is the cheap one: roughly 4.5× cheaper than GPT-5.5 and 5.4× cheaper than Opus per warm turn. It is also the slow one, two to three and a half times their latency, because it spends reasoning tokens on every turn while the other two spent none on this task. That is the trade: GLM buys cost with latency, and reasoning effort is the dial that moves it.

Caching is what makes any of these affordable in a loop. The system prompt and tool definitions are most of every prompt and identical each turn, so once the prefix is cached the turn gets 2.8× to 4.9× cheaper. Two things decide whether you see it. GLM and OpenAI cache the prefix automatically; Anthropic only caches what you mark with cache_control. And GLM's cache warms up a beat late, so a three-step task can pay full price while a thirty-step one runs cached. The mechanics are in Open-Weight LLM Caching.

When to reach for GLM 5.2, and how to run it well

Put the pieces together. GLM 5.2 is the cheap model in that table and the slow one, and it reasons on every turn. That profile points at where it earns its place.

Where it fits: long, multi-step agent loops where cost dominates and a few seconds per turn is acceptable. Background coding agents, CI and batch automation, jobs that run unattended. The reasoning that makes it slow is also why it holds up on real coding and planning rather than trivial routing. Caching compounds the case past the warm-up: a thirty-step task amortizes the prefix and runs cheap, while a three-step one can pay full price and eat the latency for nothing. So reach for GLM 5.2 on the long jobs, and keep a faster model for the interactive, single-shot calls where six seconds a turn is felt.

How to run GLM 5.2 well. Five habits make a loop GLM-ready without leaving the OpenAI API surface:

Treat a tool-call turn as possibly carrying content. Do not assert it is empty.
Expect reasoning_content on the wire and reasoning_tokens in usage; budget for both, and use the reasoning-effort dial to trade quality for cost.
In streaming, do not key UI state on the first content delta, since reasoning arrives first.
Echo tool_call_id verbatim; treat it as opaque, never parse or regenerate it.
Accumulate streaming arguments by index until the call closes; do not assume a chunk count.

Two things you do not have to defend against: GLM emits parallel tool calls with an index like the others, and the round-trip closes normally. Append the assistant turn, append one tool message per call with its result, and it finishes with finish_reason: "stop". Keep the cacheable prefix byte-stable across turns while you are at it; the system prompt and tool definitions are most of every prompt, and a stable prefix is what lets GLM's cache carry the cost once it warms.

None of this is exotic. It is the gap between "the request succeeds" and "the agent loop is correct," and on GLM that gap is mostly two assumptions wide: that a tool-call turn is silent, and that it isn't thinking. Drop those two, keep the prefix stable, and one loop carries GLM, GPT, and Claude alike, with GLM doing it for a fraction of the cost wherever latency is not the thing you are optimizing.

Disclaimer

The cost, latency, and cache figures above were measured on 2026-06-30 over ten warm tool-call turns per model, with glm-5.2, gpt-5.5, and claude-opus-4-8. Cost is taken from reported usage; latency is wall-clock median and shifts with load and reasoning effort. Model behavior and prices drift, so treat the figures as indicative and re-measure against your own traffic before depending on them.

Sources

What a Simple Transcription Test Can and Can't Tell You

synthorai — Fri, 26 Jun 2026 10:52:00 +0000

Synthorai now transcribes audio, with thirteen models behind one endpoint in two families.

That one endpoint hides a lot of work, because natively these models barely resemble each other. whisper-1 takes a multipart file upload and returns {text}. gpt-4o-transcribe uses the same upload but adds token usage. Gemini is not a transcription API at all: you base64-encode the audio into a JSON generateContent request and dig the transcript out of candidates[0].content.parts[].text. ByteDance's seed-asr speaks the BytePlus AUC protocol, and Google's chirp models are Cloud Speech-to-Text recognizers reached with OAuth.

Different endpoints, different auth, different response shapes, one more integration each. Through the gateway it is one OpenAI-compatible call: swap gpt-4o-mini-transcribe for gemini-2.5-flash-lite or seed-asr-bigmodel, and nothing else in your code changes.

The call is the OpenAI-compatible transcription endpoint, so it is a drop-in if you already use Whisper:

curl https://synthorai.io/v1/audio/transcriptions \
  -H "Authorization: Bearer $SYNTHORAI_API_KEY" \
  -F file=@meeting.mp3 \
  -F model=gemini-2.5-flash-lite

from openai import OpenAI

client = OpenAI(base_url="https://synthorai.io/v1", api_key="sk-syn-...")

with open("meeting.mp3", "rb") as f:
    result = client.audio.transcriptions.create(model="gemini-2.5-flash-lite", file=f)

print(result.text)

The transcript comes back in text, and the billed cost is in the x-total-cost-usd response header.

We put all thirteen through the same simple test, and what that test is shapes every number below.

What this test is, and isn't

We generated everyday passages with no proper nouns (a morning, the weather, a trip to the market) with a standard text-to-speech voice in each of the world's five most-spoken languages, then transcribed each clip through all thirteen models. Each clip runs about 12 to 15 seconds, roughly 40 words of normal-paced speech with no long silences, encoded as 16 kHz mono 16-bit PCM WAV (256 kbps, about 2 MB a minute). The text is the ground truth and the durations are exact.

This is a deliberately easy case: clean, scripted, single-speaker audio with no accents, noise, or jargon. That makes it good for the things that do not depend on how hard the audio is. It measures cost, latency, which languages a model accepts at all, and whether it can stream, and those are stable facts.

It is not a quality benchmark. Real recordings with accents, background noise, domain vocabulary, overlapping speakers, and an hour of runtime separate these models in ways clean speech never will, and nothing here predicts that. Read the accuracy numbers as a floor check, not a ranking, and treat the cost, coverage, and streaming results as the baseline you can actually rely on.

Two model types, three request modes

The thirteen models come in two kinds:

Native multimodal models (six, Google's Gemini family: gemini-2.5-flash-lite, gemini-3.1-flash-lite-preview, gemini-2.5-flash, gemini-3-flash-preview, gemini-3.5-flash, gemini-2.5-pro). General audio-and-text models that transcribe as a side effect of being multimodal.
Dedicated ASR models (seven: OpenAI's whisper-1, gpt-4o-transcribe, gpt-4o-mini-transcribe; ByteDance's seed-asr-bigmodel; Alibaba's qwen3-asr-flash; Google's chirp-2 and chirp-3). Purpose-built for speech.

And three ways to send the audio:

File in, batch out: upload a complete recording, get the full transcript in one response. Every model supports it.
File in, streamed text out: the same upload, but the transcript streams back over SSE as it is produced. Some models support this; others are batch-only.
Audio stream in, text stream out: real-time recognition of a live mic or call. In development, not yet available, so everything below is the first two modes.

How transcription is billed

Two billing shapes. Per audio-minute (whisper-1, seed-asr, qwen3-asr-flash, the Chirp models): you pay for the wall-clock length of the recording, whatever is in it. Per token (the gpt-4o and Gemini models): audio tokenizes at a flat rate, and you pay for those input tokens plus the transcript output tokens, so silence is cheaper than dense speech.

The per-token shape has a trap: the listed input rate is for text, but audio bills higher (gpt-4o-mini-transcribe lists $1.25/M input but bills audio at $3/M). Estimate from the text rate and you undershoot. The gateway returns the real charge in an x-total-cost-usd header, so read that rather than guessing from a price page.

Cost

This is the part the test pins down cleanly, and it varies the most. Cost per minute, from the billed header:

Model	Type	Cost / min	Latency	Streams
`gemini-2.5-flash-lite`	multimodal	$0.0006	≈4s	chunks
`gemini-3.1-flash-lite-preview`	multimodal	$0.0016	≈3s	chunks
`seed-asr-bigmodel`	dedicated	$0.0020	≈10s	no
`qwen3-asr-flash`	dedicated	$0.0021	≈3s	no
`gemini-2.5-flash`	multimodal	$0.0026	≈2s	chunks
`gpt-4o-mini-transcribe`	dedicated	$0.0031	≈3s	token-by-token
`gemini-3-flash-preview`	multimodal	$0.0035	≈4s	chunks
`whisper-1`	dedicated	$0.0060	≈4s	no
`gpt-4o-transcribe`	dedicated	$0.0062	≈2s	token-by-token
`gemini-2.5-pro`	multimodal	$0.0082	≈5s	chunks
`chirp-2`	dedicated	$0.0164	≈3s	no
`chirp-3`	dedicated	$0.0164	≈4s	no
`gemini-3.5-flash`	multimodal	$0.0178	≈5s	chunks

The spread is about 30x, from gemini-2.5-flash-lite at $0.0006 a minute to gemini-3.5-flash at $0.0178. Two things are worth noticing, both about price rather than quality. The single cheapest model is a Gemini flash-lite, three times cheaper than the cheapest dedicated ASR. And within the Gemini family the price had no relationship to accuracy on this test, so a bigger, pricier model is not automatically the safer choice; it is a reason to benchmark the cheap one on your own audio before paying for the large one.

How these numbers move with your own files depends on the billing shape. The per-minute models (whisper-1, seed-asr, qwen3-asr-flash, the Chirps) bill by duration alone, so the rate is portable: ten minutes of audio costs ten times the per-minute figure, whatever the format or content.

The per-token models (the gpt-4o and Gemini rows) scale their input cost with duration, not file size, because the provider resamples the audio before tokenizing. A heavy 320 kbps MP3 and our lean 16 kHz WAV of the same words tokenize to about the same cost, so compressing your files saves storage, not transcription spend. What does move a per-token bill is how much is actually spoken: our clips are normal-paced with no dead air, so audio that is denser or quieter than that bills a little more or less on the output tokens. The x-total-cost-usd header is the ground truth in every case.

Accuracy and language coverage

On English, Spanish, and French, every model that accepts the language scored about 0% error. That is the floor, and everyone clears it. Mandarin and Hindi are where even this easy test starts to show cracks, but read that as a hint about where to point your own testing, not a verdict:

Model	Mandarin (CER)	Hindi (WER)	Coverage
`gemini-2.5-flash-lite`	0%	13%	all five
`gemini-3.1-flash-lite-preview`	0%	15%	all five
`seed-asr-bigmodel`	0%	fails	English + Chinese only
`qwen3-asr-flash`	0%	15%	all five
`gemini-2.5-flash`	0%	15%	all five
`gpt-4o-mini-transcribe`	0%	4%	all five
`gemini-3-flash-preview`	16%	7%	all five
`whisper-1`	0%	22%	all five
`gpt-4o-transcribe`	0%	13%	all five
`gemini-2.5-pro`	0%	15%	all five
`chirp-2`	16%	15%	all five
`chirp-3`	2%	15%	all five
`gemini-3.5-flash`	0%	15%	all five

The hard fact here is coverage, not accuracy. seed-asr returns a useless transcript for Hindi, Spanish, and French: it is an English-and-Chinese model, so it is only an option if your audio is one of those two languages. Everything else handled all five.

The Hindi spread and the Mandarin slips (chirp-2, one Gemini) say those models are worth testing on your harder languages before you trust them, not that one is better than another. The absolute numbers are inflated by the synthetic voice and the scoring and move from run to run. The honest read is that on clean speech in major languages, accuracy is not where these models separate, so it is not where this test can tell you to choose.

Streaming output

Whether a model can stream its transcript is a capability, not a quality call, and it splits the lineup. The per-minute models (whisper-1, seed-asr, qwen3-asr-flash, and both Chirps) are batch-only; the gateway returns a 400 if you ask them to stream. The gpt-4o models stream token by token: gpt-4o-transcribe returns its first words in about a second and fills in the rest, which is what a live-feel UI needs. The Gemini models technically stream, but in three to six large blocks, with the first arriving about when the whole transcript is done, so it buys almost nothing. Cost is unchanged from batch. To stream, add stream=true:

curl -N https://synthorai.io/v1/audio/transcriptions \
  -H "Authorization: Bearer $SYNTHORAI_API_KEY" \
  -F file=@meeting.mp3 -F model=gpt-4o-transcribe -F stream=true
# data: {"type":"transcript.text.delta","delta":"When"}
# data: {"type":"transcript.text.delta","delta":" you"} ...

Caching repeated audio

Caching is where the two billing shapes split one more time. The per-minute models cannot cache: we sent the same clip to whisper-1 five times and paid an identical $0.015478 every time, because the bill is just duration. The token-billed Gemini models can. Send the same file repeatedly and Gemini's implicit cache reuses the audio tokens: on a 155-second clip sent five times, gemini-2.5-flash dropped from $0.0054 to $0.0026 on two of the repeats, about 51% off, and gemini-2.5-pro fell about 39%.

Two caveats keep it from being a sure thing. It is best-effort, so some repeats hit the cache and some pay full price; and the audio has to clear Gemini's token floor, roughly a minute or more, which the short clips elsewhere in this test never do. The gpt-4o models list no cache rate and showed only ordinary run-to-run variation. So if your workload re-transcribes the same files, caching is a real discount on the token-billed models and nothing on the per-minute ones.

What to check first, and what to test yourself

This test cannot tell you which model is most accurate on your recordings. It can tell you what to filter on before you run your own evaluation:

Languages. Check that the model accepts every language you need. seed-asr is English and Chinese only; the other twelve handled all five we tried. This is a hard gate, not a preference.
Streaming. If you need a live transcript, only the gpt-4o models stream token by token; the per-minute models are batch-only and Gemini's streaming is coarse.
Cost. The spread is about 30x. gemini-2.5-flash-lite is the cheapest and still multilingual; the Chirps and the largest Gemini are the most expensive. A bigger model in the same family did not earn its premium on the easy clips, so do not assume you need it without checking. If you re-transcribe the same files often, the token-billed Gemini models can also cache the audio, as above.

Once a few models clear those, the question that is left, how accurate each one is on your own audio with its accents, noise, and vocabulary, is the one you have to answer yourself. No clean-speech benchmark substitutes for running the survivors on real recordings.

Bottom line

On clean, scripted speech in major languages, all thirteen models are about equally accurate, which is the most useful thing this test says: accuracy is not the axis to choose on. What it does pin down, and what genuinely varies, is the baseline: cost spans about 30x, one model covers only two languages, and several cannot stream. Use those to narrow the field, not to declare a winner, then run the two or three survivors on your own audio. That last step is the one no simple test can do for you.

Sources

Costs and latencies measured on Synthorai on 2026-06-25 across thirteen models and five languages (English, Mandarin, Hindi, Spanish, French), via the x-total-cost-usd header and SSE timing. The audio was text-to-speech generated and deliberately easy, so the accuracy figures are a floor check rather than a quality benchmark; real-world speech with accents and noise would separate these models differently. Latency varies run to run. Listing prices are this platform's rates as of that date. Verify current pricing before relying on it.

GLM 5.2: Reasoning Effort Is the Cost Lever

synthorai — Wed, 24 Jun 2026 15:19:00 +0000

GLM 5.2 is now on Synthorai at about a sixth of frontier per-token prices, and the open-weight, frontier-benchmark headline is real. But the per-token price is the wrong number to anchor on. What a coding task actually costs on GLM 5.2 swings by more than an order of magnitude depending on a single knob, reasoning effort, and the default leaves that knob in the worst position. Set it well and GLM 5.2 is correct and cheaper than frontier on both easy and hard work. Leave it on the default and the same answer costs twenty times more and takes minutes. We measured it.

What GLM 5.2 is

GLM 5.2 is Zhipu's open-weight frontier model, released 2026-06-13: a mixture-of-experts network (~744B total, ~40B active), a usable 1M-token context, and an MIT license you can self-host. It targets coding and agentic work, with strong published benchmarks (SWE-bench Pro 62.1, Terminal-Bench 2.1 81.0, AIME 2026 99.2, GPQA Diamond 91.2). On Synthorai it's glm-5.2, priced at $1.40 per million input tokens and $4.40 per million output.

The detail that drives everything below: it is a reasoning model, and how much it reasons is something you set.

Where it sits on price

On per-token listing price, GLM 5.2 sits well below the Western frontier and among the cheaper Chinese models. Synthorai's rates for a representative set:

Model	Input ($/M)	Output ($/M)	Cache read ($/M)
`deepseek-v4-pro`	0.44	0.87	0.0036
`kimi-k2.5`	0.57	3.01	0.12
`glm-5.2`	1.40	4.40	0.26
`qwen3-max`	1.20	6.00	0.36
`gemini-3.1-pro`	2.00	12.00	0.20
`claude-opus-4-8`	5.00	25.00	0.50
`gpt-5.5`	5.00	30.00	0.50

Its $4.40 output rate is about a seventh of gpt-5.5 and a sixth of claude-opus-4-8, though deepseek-v4-pro and kimi-k2.5 undercut it. So GLM 5.2 is frontier-class capability at roughly Chinese-model prices, not the absolute floor. There is no separate cache-write charge: a cache write bills at the input rate, and only the cache read is discounted to the rate above. The discount varies by vendor, with GLM 5.2's cache read about a fifth of its input rate and the frontier models (gpt-5.5, claude-opus-4-8, gemini-3.1-pro) discounting reads to roughly a tenth.

It is also a step up from its own predecessors. The previous GLM generation was extraordinarily cheap; the GLM 5 line raised prices, and GLM 5.2 lands at about 3x the input rate of GLM-4.6 (Zhipu's official rates):

GLM model	Released	Input ($/M)	Output ($/M)
GLM-4.5	2025-07	0.60	2.20
GLM-4.6	2025-09	0.43	1.74
GLM-5	2026	1.00	3.20
GLM-5.2	2026-06	1.40	4.40

That buys the 1M context and the frontier benchmarks. But the per-token rate is only the headline. What you actually pay per task is set by the reasoning effort.

The reasoning-effort dial

GLM 5.2's reasoning is a dial, not a switch. You can turn it off (enable_thinking: false), set reasoning_effort to low, medium, or high, or leave it on the default, which runs reasoning unbounded. That setting changes cost and latency by far more than the price does. We ran one easy and one hard coding task across the settings, checking every answer against a reference on hundreds of randomized cases.

An easy task: reasoning just adds cost

Weighted interval scheduling, a moderate dynamic-programming problem:

Mode	Reasoning tokens	Answer tokens	Cost	Latency	Correct
`glm-5.2`, thinking off	0	169	$0.0008	≈5s	yes
`glm-5.2`, `reasoning_effort: low`	1,563	150	$0.0076	39s	yes
`glm-5.2`, unbounded default	≈6,290	≈150	$0.0285	137s	yes
`gpt-5.5` (reference)	59	141	$0.0064	4.8s	yes
`claude-opus-4-8` (reference)	0	201	$0.0057	3.3s	yes

Two things stand out. Thinking off is correct and the cheapest thing on the board, about 8x under the frontier models, and every step up the dial just adds cost for the same answer. And the bill tracks the reasoning, not the answer: the code GLM returns is roughly 150 tokens every time, while the reasoning in front of it grows from nothing to about 6,300, billed at the same $4.40/M output rate. The unbounded default spends that reasoning to reach the same answer thinking off produced with none, and the gap is the entire cost difference. The frontier models answer here with little or no reported reasoning: gpt-5.5 spends 59 reasoning tokens, and claude-opus-4-8's usage reports none.

A hard task: reasoning earns its keep, the default does not

Wildcard string matching (? and *), the classic problem that is easy to get subtly wrong. Here thinking off broke. It returned a memoized recursion:

def is_match(s, p):
    memo = {}
    def match(i, j):
        if (i, j) in memo:
            return memo[(i, j)]
        if j == len(p):
            result = i == len(s)
        elif i < len(s) and p[j] in (s[i], '?'):
            result = match(i + 1, j + 1)
        elif p[j] == '*':
            result = match(i + 1, j) or match(i, j + 1)
        else:
            result = False
        memo[(i, j)] = result
        return result
    return match(0, 0)

It looks right, and the memo even suggests some care. But the * branch recurses match(i + 1, j) without bounding i. Once the string is consumed and the pattern still has a *, i climbs forever and the stack overflows. Fast, cheap, and wrong.

Turn the dial up and it returns the correct iterative two-pointer algorithm, which backtracks to the last * instead of recursing:

def is_match(s, p):
    s_idx, p_idx, star_idx, match_idx = 0, 0, -1, 0
    while s_idx < len(s):
        if p_idx < len(p) and (p[p_idx] == '?' or p[p_idx] == s[s_idx]):
            s_idx += 1
            p_idx += 1
        elif p_idx < len(p) and p[p_idx] == '*':
            star_idx = p_idx
            match_idx = s_idx
            p_idx += 1
        elif star_idx != -1:
            p_idx = star_idx + 1
            match_idx += 1
            s_idx = match_idx
        else:
            return False
    while p_idx < len(p) and p[p_idx] == '*':
        p_idx += 1
    return p_idx == len(p)

The full dial on this task:

GLM 5.2 setting	Cost	Latency	Correct
thinking off	$0.0007	6s	no (stack overflow)
`reasoning_effort: high`	$0.0031	13s	yes
`reasoning_effort: medium`	$0.0032	16s	yes
`reasoning_effort: low`	$0.0068	40s	yes
unbounded default	$0.062	405s	yes
`gpt-5.5` (reference)	$0.0064	5.4s	yes
`claude-opus-4-8` (reference)	$0.0069	4.6s	yes

Every explicit effort level solved it. reasoning_effort: high did it for $0.0031 in 13 seconds, about twenty times cheaper and thirty times faster than the unbounded default for the same answer, and it undercuts the frontier models on cost, just a few seconds slower. One quirk worth knowing: GLM's low produced more reasoning than high, consistently across both tasks, so the names don't track token count. Medium and high were the cheap, fast settings.

The unbounded default is the one setting to avoid. It is the worst of both worlds: it buys reasoning the task may not need and takes minutes to do it, reaching the same answer reasoning_effort: high gave for twenty times the cost.

The decision rule

The lever is the reasoning effort, and the right setting belongs to the task, not the model:

Simple or high-volume work where correctness is easy: thinking off (enable_thinking: false). Correct and about 8x under frontier.
Harder problems where thinking off fails: reasoning_effort: medium or high. Correct, around $0.003 a task, under frontier on cost and only a few seconds slower.
Never the unbounded default. Leaving reasoning on with no effort cap is how a $0.003 answer becomes a $0.06, seven-minute one.

If you cannot tell in advance whether a task needs reasoning, reasoning_effort: high is a safe default: it was cheap, it solved both tasks, and it never ran away.

Caching helps the input, not the reasoning

GLM 5.2 supports caching on the gateway, and it helps where you'd expect. We sent a 1,494-token shared prefix (a code module to review) with several different questions:

Call	Prompt tokens	Cached	Output	Cost	Latency
new question, prefix not yet cached	1,493	0	120	$0.0026	6.5s
new question, prefix cached	1,494	1,472	120	$0.0009	5.1s
exact repeat (semantic hit)	1,494	1,494	120	$0.0009	1.0s

Once a large prefix has been seen, it caches. The cached input tokens bill at roughly a fifth of the normal input rate, which cut an otherwise identical request from $0.0026 to $0.0009, about 64%. An exact repeat is served straight from the semantic cache: the same answer at the same cost as the cached call, but back in about a second instead of five.

The catch is the same one the dial taught: caching discounts the input, and the moment reasoning is on, the cost and latency live in the reasoning output, which is not cached. So caching is a real win for thinking-off, high-context work (the same system prompt or codebase on every call), and a small one once reasoning is on.

Using it on Synthorai

glm-5.2 is live on the gateway. Three practical notes from our testing:

Set the reasoning effort explicitly. Use enable_thinking: false for simple work and reasoning_effort: medium or high for harder problems. The one thing to avoid is leaving reasoning on with no effort cap (the unbounded default), which is the $0.06, seven-minute trap.
Stream when reasoning is on. Reasoning responses can run for minutes, and a non-streaming request sits on a silent connection long enough that your client will likely time out before the answer arrives. Use stream: true and you get incremental output and the full result.
Reuse your context. If you send the same large system prompt or codebase on every call, prefix caching cuts the input cost, and pairing it with thinking off makes the whole request cheap.

Pricing is $1.40 / $4.40 per million tokens, and the gateway returns a cost field per call so you can see exactly what each request cost.

Bottom line

GLM 5.2 is a genuinely cheap, capable coding model, and configured well it beats frontier prices on both easy and hard work. The catch is the configuration. Its reasoning is a dial, and the default leaves it unbounded, which is how a task that should cost $0.003 becomes a $0.06, seven-minute call. Set enable_thinking: false for simple work and reasoning_effort: medium or high for the rest, and GLM 5.2 is cheap and correct across the board. Leave reasoning on its default, and it is the slowest, priciest option you could have picked.

Sources

(Synthorai listing prices above are this platform's rates as of 2026-06-24; GLM generational rates are Zhipu's official list.)

Costs measured on Synthorai on 2026-06-24 (glm-5.2 at $1.40 / $4.40 per M tokens); verify current pricing before relying on it.

What Actually Drives Your Image-Generation Bill

synthorai — Fri, 19 Jun 2026 18:14:35 +0000

We added image generation to a gateway built for text LLMs and measured what drives the cost across four variables: model, resolution, image count, and quality. The largest lever is quality, a parameter most image APIs expose and most callers leave on default. Resolution, prompt caching, and batching matter far less than people expect.

How image models differ

Image models aren't drop-in swaps for one another. They diverge on several axes, and only one of them (billing shape) is about price. The active catalog at a glance:

Family	Billing	`quality` knob	Batch `n>1`	Resolution
`gpt-image` (OpenAI)	per-token	✓ `low`/`med`/`high`	✓	up to ≈2K
`gemini-image` (Google)	per-token	✗	✗ 1/call	1K (`gemini-3`: to 4K)
`qwen-image` / `wan2.7` (Alibaba)	flat/image	✗	✓	512²–2048²
`seedream` (BytePlus)	flat/image	✗	✗ 1/call	≥1920² (4.5/5.0)

The axes that bite if you assume one model behaves like another:

Billing shape. Per-token (gpt-image, gemini) or flat-per-image (qwen, wan, seedream). This is the axis that decides your bill, and it's the subject of the next section.
The quality knob. Only gpt-image has it (low/medium/high). Gemini changes fidelity by model tier (flash to pro) or image_size; flat models have no such dial. That one knob swings the bill about 36×, so it's the main cost lever, covered below.
Batch (n>1) isn't universal. gpt-image, qwen, and wan return several images per call. Every Gemini and Seedream image model is one-image-per-call: n=2 returns a 400, so you issue N requests and orchestrate the batch yourself.
Resolution limits cut both ways. gemini-2.5-flash-image caps at 1K (1 MP), while gemini-3 reaches 2K/4K (and its bill roughly doubles from 1K to 4K). Seedream 4.5/5.0 enforce a floor of about 1920² and reject anything smaller. qwen-image lives in a 512²–2048² band. Higher resolution isn't always available, and dropping resolution to save money isn't always allowed.
Control knobs and image-to-image differ. Only some models accept seed, negative_prompt, or guidance_scale, and the reference-image limit for editing runs from 3 (gemini-2.5) to 16 (gpt-image).

The quality knob has one non-obvious property. For gpt-image, an output token is a billing unit, not a measure of the file you get. OpenAI assigns the count from a published per-(quality × size) rate table (272 / 1,056 / 4,160 tokens for low / medium / high at 1024² on gpt-image-1), so the count is set by quality, not derived from the bytes returned. We checked: the same prompt at 1024² across all three tiers produced identical 1024×1024 PNGs of roughly the same file size (about 0.9 MB), yet billed 196, 1,756, and 7,024 tokens. Same resolution, same byte size, 36× the cost. You pay for rendering effort, not pixels, which is why you read usage rather than eyeball the output.

One capability none of these models has is prompt caching, usually the first cost-saving idea people reach for. Image generation is stateless: there's no conversation or KV state to reuse, the usage object carries no cache fields, and (as we measure below) batching doesn't share the prompt either. Caching is a chat feature, not an image one, which rules out a common assumption about cutting image cost.

We measured it

Same e-commerce-style product prompt, real generations through the gateway, with cost computed from the returned usage against each model's published rates. Five findings, each from a separate sweep.

1. The image is the cost, not the prompt. In text-to-image (a prompt in, an image out), the bill is 97–100% output tokens: a 1024² gpt-image-2 generation is 21 input and 196 output tokens (about $0.0001 plus $0.0059), and gemini-2.5-flash-image takes 10 input. The prompt you write is a rounding error, but only because it's text. Feed an image instead (image-to-image, like "make this mug blue") and the input tokenizes large:

Model	t2i input	i2i input (1 ref)	Output
`gpt-image-2` (low)	21 tok	1,043 tok	196 tok
`gemini-2.5-flash-image`	10 tok	1,297 tok	1,290 tok

The input jumps 50–130×, and it scales linearly: each extra reference adds about 1,025 tokens on gpt-image-2 (1, 2, and 3 references measured at 1,043, 2,068, and 3,093). At low quality those input tokens outnumber the generated output five-to-one. The principle holds either way: an image is the cost, whether you generate it or supply it, and the prompt never is. The rest of this article stays in text-to-image; the fuller image-to-image economics are their own follow-up.

2. Model choice is a 6× lever. Identical 1024² request, default quality:

Model	Billing	Cost / image
`gpt-image-2`	token · `quality` knob	$0.0060
`gpt-image-1-mini`	token · `quality` knob	$0.0085
`seedream-4-0`	per-request flat	$0.030
`qwen-image-2.0`	per-request flat	$0.035
`gemini-2.5-flash-image`	token · no `quality` knob	$0.0387

A 6.4× spread between the cheapest and priciest path, driven entirely by how many output tokens each model emits.

3. Resolution barely moves it. Sweeping gpt-image-2 from 1024² to 2048², per-image cost stayed roughly flat ($0.0060 to $0.0121); output tokens aren't proportional to pixels. gemini-2.5-flash-image returned the same 1,290 tokens whatever size we requested, because it's 1K-only and size only changes the aspect ratio. (The gemini-3 image tiers do honor image_size, roughly doubling cost from 1K to 4K, but 2.5-flash-image, the model we cost here, does not.) Per-image flat models are resolution-independent by definition. So far the per-token model looks hard to beat.

4. Quality is the crossover. Sweep gpt-image-2 across quality tiers:

quality	1024²	2048²
low	$0.0060 (196 tok)	$0.0121 (397 tok)
medium	$0.053 (1,756 tok)	$0.107 (3,568 tok)
high	$0.211 (7,024 tok)	$0.428 (14,272 tok)

Output tokens scale about 9× from low to medium and about 36× from low to high. At low quality the per-token model is the cheapest option; at medium or high it passes the flat per-image price ($0.03–0.035). The crossover sits where the arithmetic puts it, around 1,000 output tokens ($0.03 ÷ $30/M): low is under it, medium is over. This also corrects an earlier conclusion of ours. "Per-token is always cheapest" was an artifact of testing at default low quality.

Same prompt, gpt-image-2, 1024². low / medium / high bill 196 / 1,756 / 7,024 output tokens, or $0.006 / $0.053 / $0.215: a 36× spread at identical resolution. For a clean product shot like this the three are hard to tell apart, so the cheapest tier is often enough. Set quality to the job instead of defaulting to high.

5. You can't share a prompt across images. Generating n images in one call doesn't amortize the prompt. gpt-image-2 bills it N times: input tokens went from 28 to 112 at n=4, and a long brand prompt went from 499 to 1,996. Per-image cost was identical at n=1 and n=4. With no caching either, there's no prompt-cost-sharing mechanism for image generation. You pay per output image, and the prompt is re-billed each time.

The decision rule

For text-to-image, it comes down to quality, not the things people assume:

Low / draft / thumbnail quality: a token-with-quality model (gpt-image, about $0.006–0.012). Cheapest at any resolution up to about 2K.
Medium / high quality: per-request flat (seedream / qwen, $0.03–0.035). The per-token bill runs away ($0.05–0.43 in our sweep), and flat is both cheaper and quality-independent.
gemini (about $0.039 at default 1K) is rarely the cost-optimal pick. It's undercut by gpt-image at low quality and by per-request flat at medium and high. It has no quality dial; you'd choose its Pro tier or a higher image_size for output quality, not for price.
Resolution moves cost about 2× within a quality tier, not enough to flip the choice. Quality flips it.
n>1, caching, and batching never reduce per-image cost. There's nothing to share.
Image-to-image: default to flat per-image. A reference image is input, and only per-token models surcharge it (about 1,025 tokens each); flat models include it for free. For editing, seedream / qwen usually win. gpt-image stays cheaper only for low-quality edits with a few references (around 5 crosses the flat price), and loses once quality or reference count climbs.

E-commerce is the clearest example. Say you generate product photos by sending the same long brand prompt for every item in the catalog, and you assume caching that repeated prompt will save money. That fails for two reasons: the prompt was never the cost (the image is), and there's no caching for generation anyway. Since real product imagery is medium quality or higher, the right choice is a flat per-image model, which is both cheaper and more predictable regardless of how repetitive your prompts are.

The capability gates from the opening section can still override the choice: one-image-per-call models, resolution floors and ceilings, data-residency limits, and which knobs (seed, negative_prompt, guidance_scale) a model exposes. Pick on cost, then confirm the capability fits.

Why you can trust these numbers

These figures come from real usage against each vendor's list rates, not estimates. Image billing on our gateway is sessionless: it settles only on a 2xx (a failed generation is never charged), pre-checks the worst-case cost before any spend, and bills a missing-usage response at the ceiling rather than silently $0. The principle is the same one we apply everywhere: trust the cost, not a figure the vendor hands you. It's the method we used to audit whether a gateway lies about cache.

Bottom line

Image generation looks like just another endpoint, but the billing unit changed. For text-to-image the lever isn't the prompt (no caching, no batch sharing) or the resolution. It's quality: gpt-image is cheapest at low, per-image flat (seedream / qwen) wins at medium and high, with the crossover near 1,000 output tokens. Set quality deliberately, match the model to it, and check the cost. When you move from generating to editing, feeding a reference image, re-run the math, because the input image becomes the cost.

FAQ

Does prompt caching reduce image-generation cost?
No. Generation is stateless: the usage object has no cache fields, and batching re-bills the prompt per image. The cost is the output image, not the text.

Per-token or per-image, which is cheaper?
It depends on quality. For low or draft quality, a quality-knob model like gpt-image (about $0.006–0.012). For medium or high, per-image flat like seedream/qwen ($0.03–0.035), because the per-token bill runs away. For image-to-image the answer tilts further to flat: they include reference images for free, while per-token surcharges about 1,025 tokens each.

Sources

All checked 2026-06-19. Not financial advice; verify current pricing before relying on it.

Prompt Caching for Open-Weight LLMs: Provider Roulette

synthorai — Mon, 15 Jun 2026 11:52:43 +0000

With a closed model, prompt caching is one documented contract. Claude has cache_control breakpoints; OpenAI and Gemini cache automatically above a token floor; the discounts are published and stable. You read one page and you're done.

Open weights break that assumption. The same Qwen or Llama checkpoint is served by a dozen hosts, and caching is not a property of the model — it's a property of where the model runs. To show how far that goes, here's one measured request — an identical ~4.7K-token prompt sent to the same Qwen model through a multi-provider router six times, no upstream pinned:

Call	Upstream the router picked	Cost	Cached tokens
1	Upstream A	$0.0141	0
2	Upstream B	$0.000709	0 (cold)
3–6	Upstream B	$0.000286	4,224 (warm)

Same model, same router, same prompt: the bill ranged from $0.0141 to $0.000286 — a 49× spread — purely on which upstream the router chose and whether that upstream had the prefix warm.

Summary

Prompt caching for open-weight models is a routing outcome, not a model feature. It's implemented — free and automatic — in the inference engine, then preserved or destroyed by every layer above it.
Five layers: one provides caching, three can break it. The model (sets cacheability, serves no cache) → the inference engine (caching, free) → the compute host (productizes it, unevenly) → the gateway (multi-cluster routing) → the router (scatters across vendors with disjoint caches).
Measured. An identical request, scattered by a router, cost 49× more on one pick than another; on one model, one host delivered 59.6% off and another 0%; published cache discounts span 0% to ~98% across models.
What to do. Pin your route so repeated prefixes hit the same warm cache; audit by the cost delta, not the cached_tokens field (which often reads 0 on a real hit); weigh latency separately — warm prefills run 2–10× faster even at ~0% cost discount.

Live figures were measured on 2026-06-14 against a multi-provider router and our own gateway, with a fixed ~4.7K-token English prompt, small max_tokens, sequential runs. Documented pricing was checked against primary provider docs the same day and cross-verified adversarially. Ratios (percent discount, latency change) are the portable part; absolute dollars depend on the venue, your prompt, and load. Reproduce before quoting.

The cache types you'll actually meet

Before the stack, the vocabulary. Across open-weight hosts there are four distinct cache shapes, and they bill differently.

1. Automatic prefix caching (no markers). The dominant pattern. The server hashes your prompt prefix, reuses the KV state if it matches an earlier request, and applies the discount on its own — no cache_control, no code change, often impossible to disable. DeepSeek, Zhipu GLM, and most open-weight hosts do this. Writes are free; the cache lives anywhere from VRAM (minutes) to disk (DeepSeek keeps prefixes "a few hours to a few days").

2. Explicit breakpoint caching (cache_control). The Anthropic shape, which a few open-weight hosts also offer. Alibaba's Model Studio takes "cache_control": {"type": "ephemeral"} on a Qwen message block; some serving platforms expose an equivalent marker. You mark the boundary, pay a write surcharge, and get a deeper read discount in return.

3. Rented cache objects (with a storage fee). The one to watch. Moonshot's legacy moonshot-v1 family makes you POST /v1/caching to create a cache, then bills a write fee, a per-token-per-minute storage fee, and a per-call hit fee. Google's explicit Gemini caching is the same idea — input cost plus storage at roughly $1.00–$4.50 per 1M-tokens per hour. The cache is a resource you rent and must garbage-collect.

4. Self-host KV reuse (free). Run the weights yourself and the inference engine caches for free and automatically. No write fee, no read rate, no storage rental — a hit just skips prefill.

Cache type	Markers?	Write fee	Storage fee	Where you meet it
Automatic prefix	No	Free	No	Most open-weight hosts; DeepSeek, GLM
Explicit breakpoint	`cache_control`	Surcharge	No	Qwen (explicit mode); some platforms
Rented cache object	Create/TTL/delete	Yes	Yes	Moonshot `moonshot-v1`, Gemini explicit
Self-host KV reuse	No	Free	No	vLLM, SGLang, TensorRT-LLM

Qwen on Model Studio offers both automatic and explicit modes, with a real tradeoff: implicit bills a hit at 20% of input with free writes; explicit bills a hit at 10% of input but charges 125% on the write and bounds the entry to a 5-minute TTL. Deeper discount, but you pay to populate and pay again each time it expires.

Where caching lives in the stack

Here is the key idea. Prompt caching for open weights is solved at exactly one layer and endangered at every layer above it. Walk the stack from the weights up, and at each layer ask: does this layer provide caching, or merely forward it — and can it break what the layer below already did?

  request
     |
     v
  +--------------------------------------------------+
  | L5  router             scatters across vendors   |  can break it
  | L4  gateway            multi-cluster routing     |  can break it
  | L3  compute host       uneven delivery           |  can break it
  |==================================================|
  | L2  inference engine   CACHING LIVES HERE, free  |  <-- the cache is born here
  |==================================================|
  | L1  model              cacheability: MLA / GQA   |  sets the ceiling
  +--------------------------------------------------+

  A cache hit is born at L2 and must survive L3-L5 routing to reach you;
  every layer above L2 is a chance to land where your prefix isn't.

Layer 1 — The model: cacheability, not a cache

This is the layer most people think caching lives in — "DeepSeek has caching" — so it's the first one to get precise about. A checkpoint is a bag of weights; it runs the same attention whether or not a KV cache exists. It ships no cache, no discount, no TTL, no cache_control marker — those are serving-layer features. In that strict sense the weights provide no caching product.

But the weights are not neutral, and DeepSeek is the perfect example of why. The model's attention architecture decides how big the KV cache is, and therefore how cheap caching can ever be:

DeepSeek's Multi-head Latent Attention (MLA) compresses the KV cache into a low-rank latent — to roughly 4–14% of a standard multi-head cache. That compression is exactly what lets DeepSeek's API persist prefixes to disk and price a cache read at ~2% of input. The architecture is the enabler; the disk cache is a product built on top of it.
Grouped-Query Attention (GQA) — used by Llama, Qwen, Mistral, and DeepSeek — shares KV heads to shrink the cache by the group factor (≈8× on Llama-3).

So Layer 1's contribution is cacheability, not a cache: the architecture sets the ceiling on how cheap every layer above can make caching, but the weights never serve a cached token themselves. And "DeepSeek has caching" quietly merges two different things wearing the same name — the weights (this layer, which give you MLA) and DeepSeek's API and serving stack (Layers 2–3, which give you the disk cache, the discount, and the usage fields). Download the open weights and run them yourself and you keep MLA's small KV cache, but the disk-cache product stays on DeepSeek's servers — you inherit whatever Layer 2 you deploy in its place. So the operational move still holds: stop asking whether a model caches and start asking where it's served — just don't mistake that for the architecture not mattering. It sets the ceiling; the path sets what you actually get.

Layer 2 — The inference engine: where caching is built, and free

One layer up, caching is not just present — it's solved, and free. Modern inference engines cache prefixes automatically:

vLLM — Automatic Prefix Caching: hashes each KV block, reuses any block whose prefix hash it has seen, LRU-evicts. On by default in V1.
SGLang — RadixAttention: stores the KV cache in a radix tree so any shared prefix is reused, with cache-aware scheduling.
TensorRT-LLM — block reuse (enable_block_reuse, default on), with optional offload of KV blocks to host memory.

Projects like LMCache extend this further — offloading KV to CPU/disk and sharing it across instances, which is the seed of solving the routing problem we're about to hit. The point: if you self-host, you are done. Caching is automatic, costs nothing beyond the GPUs you already run, evicts by LRU, and you own it — a hit simply skips prefill, lowering TTFT and raising throughput. There is no cached_tokens billing field because nothing is billed; the payoff shows up in your own latency metrics. For a closed model you rent caching; for an open one you can own it outright. The catch is the inverse of the hosted world: the cache is ephemeral (VRAM, LRU), so it survives only while the prefix stays hot — which is precisely what the layers above must preserve.

Layer 3 — The compute host: productizing it, unevenly

Commercial inference hosts wrap Layer 2 and run fleets of replicas. They inherit free automatic caching — the question is whether they implement it well, and the answer is mixed on two axes.

First, exposure and price vary wildly. Among the major open-weight hosts: one applies a flat 50% to cached input and lets cached tokens skip rate limits; another defaults to 50% off on serverless; a third prices cached input per model (e.g. a Qwen tier at ~80% off) and exposes a cache-key hint to improve affinity; a fourth makes caching always-on and undiscloseable on dedicated endpoints. Same underlying engine, four pricing philosophies.

Second — and this is the first place caching breaks — the multi-replica problem. Your warm prefix lives in the VRAM of the one replica that served the cold request. The host's own load balancer may send your next request to a different replica with a cold cache. We saw exactly this: pinning the same Qwen model to one upstream at a time and running cold→warm:

Pinned upstream	Cold	Warm	Discount	`cached_tokens`
Provider A	$0.000709	$0.000286	59.6%	4,224 ✓
Provider B	$0.000662	$0.000662	0%	0

Provider A cached cleanly and reported it. Provider B — which advertises a cache-read price for this model — returned no discount across a cold call and two warm calls in our test. Whether that's eligibility, replica fan-out, or a longer warm-up than two requests, the measured result on this path was zero. The capability is solved at Layer 2; whether you actually receive it is a Layer-3 execution detail, and it differs by host.

Layer 4 — The gateway: the multi-cluster problem

A gateway sits in front of one or more upstreams and multiplies the replica problem into a cluster problem. If it round-robins requests across clusters or providers without cache affinity, the warm cache becomes structurally unreachable — every request lands somewhere the prefix isn't. A cache-aware gateway must route by prefix hash so identical prefixes stick to the same upstream, the same way Layer 2 sticks them to the same KV blocks.

We ran a cold→warm battery across open-weight models on a third-party gateway, reading the per-request cost directly:

Model	Cold	Warm	Discount	Latency
`deepseek-v4-pro`	$0.00189	$0.0000155	99.2%	6.0s → 1.1s
`deepseek-v4-flash`	$0.000564	$0.0000116	97.9%	4.9s → 1.2s
`qwen3.5-flash`	$0.000561	$0.0000853	84.8%	10.2s → 1.0s
`kimi-k2.5`	$0.00242	$0.000469	80.6%	3.2s → 1.2s
`qwen3-max`	$0.00350	$0.00336	3.8%	2.2s → 1.1s
`qwen3.5-plus`	$0.00114	$0.00114	0.0%	1.8s → 1.0s

DeepSeek-V4 hit 97–99% (affinity working end to end); qwen3.5-plus and qwen3-max returned ~0% on the warm call despite carrying a cache-read price in the catalog. Two more gateway lessons hide in this table:

The usage field lies; the cost doesn't. cached_tokens read 0 on every call here, including the 99% cost drops. Many OpenAI-compatible gateways don't populate the cached-token field for upstreams that cache automatically. Audit by the cost delta between a cold and warm call, not by the token field — the same lesson as auditing a gateway's cache claims.
Latency improves even when cost doesn't. Every warm call was 2–10× faster — qwen3.5-flash went 10.2s→1.0s — including the ~0%-discount ones. A hit skips prefill regardless of how the host prices it, so caching can pay off in TTFT on a gateway that gives you nothing on the bill.

A gateway that doesn't preserve affinity hands you a cache you can't reach; one that doesn't surface cache cost hands you one you can't verify.

Layer 5 — The router: random distribution across providers

At the top, a multi-provider router load-balances one model ID across different companies' clusters — each with a separate cache. Now even perfect affinity within a provider can't save you: if call 1 goes to one vendor and call 2 to another, there is no shared cache to hit. This is the scatter from the top of this post, and it compounds Layer 4 — not just multiple clusters, but multiple vendors with disjoint cache state and disjoint prices (the priciest pick billed 20× the cheapest upstream's base rate). The cache only engaged once routing happened to stick to one provider.

The fix is to collapse the randomness — make routing deterministic so repeated prefixes land on the same warm cache:

# Pin the upstream; otherwise load-balancing scatters you across disjoint caches.
# (field names follow a common multi-provider router's API)
import requests

requests.post(f"{ROUTER_BASE}/chat/completions",
  headers={"Authorization": f"Bearer {API_KEY}"},
  json={
    "model": "qwen/qwen3.5-35b-a3b",
    "messages": messages,
    "usage": {"include": True},              # return cost + cached_tokens
    "provider": {                            # the part that makes caching work
        "order": ["<your-chosen-upstream>"],
        "allow_fallbacks": False,
    },
  })

To its credit, the router did report cached_tokens (4,224 on the hit) and a per-request cost, so you can verify both — better than the Layer-4 gateway that read 0. But the routing is yours to constrain. Caching is a routing problem dressed up as a pricing feature: the cache is free at Layer 2, and Layers 3, 4, and 5 are three escalating ways to route yourself away from it.

How deep is the discount? It's all over the map

When the routing does line up, how much do you save? For closed models the cache-read discount clusters near 90%. For open weights the published cache-read price ranges from a token gesture to near-total, even within one vendor's lineup. First-party published rates:

Model (first-party / mode)	Input $/M	Cache read $/M	Discount	Layer-2 type
DeepSeek-v4-flash	0.14	0.0028	~98%	auto disk
DeepSeek-v4-pro	1.74	0.145	~92%	auto disk
Qwen (explicit mode)	base	0.10× base	90%	explicit
Kimi K2.6	0.95	0.16	~83%	auto
GLM-5	1.0	0.20	80%	auto implicit
Qwen (implicit mode)	base	0.20× base	80%	auto

DeepSeek's automatic disk cache is the deepest in the field — deepseek-v4-flash reads cached input at $0.0028/M against a $0.14/M miss, a 1:50 ratio, which our Layer-4 test reproduced at 97.9%. Third-party hosts of these same open weights price cached input independently — some apply a flat ~50%, others vary per model from ~50% to ~90% — so the discount you get is a function of which host you land on, not just the model. Same feature name, a 48-point spread.

Because the discount is a venue property, one model carries different cache economics everywhere it's served. deepseek-v4-pro, four ways:

Where (layer)	Cache-read discount	Source
First-party API (L3)	~92% ($1.74 → $0.145)	documented
Third-party host A (L3)	~89% ($1.74 → $0.20)	documented
Third-party host B (L3)	~92% ($1.6 → $0.135)	documented
Third-party gateway (L4)	99.2%	measured (cold→warm)

"DeepSeek-V4-Pro supports caching" is true and nearly useless; the operational question is "supports caching where, at what rate, reported how."

A decision checklist

✅ The model sets the ceiling, not the cache (Layer 1). Its attention architecture (MLA, GQA) decides how cheap caching can be, but it never serves a cached token — so still ask where it's served and what that host's stack does.
✅ Self-hosting? You already have it free (Layer 2). Confirm automatic prefix caching is on (it is by default in vLLM/SGLang) and watch your prefix hit rate.
✅ On a compute host, verify delivery, not the price column (Layer 3). A cache-read price is a claim; measure a cold→warm cost delta. Use a cache-key affinity hint where the host offers one.
✅ Through a gateway, demand cache-affinity routing and cost reporting (Layer 4). If identical prefixes don't stick to one upstream, or cost doesn't drop on a warm call, the cache is unreachable or unverifiable.
✅ On a router, pin the upstream (Layer 5). Constrain routing (e.g. a provider-order field with fallbacks off), or you forfeit hits to load-balancing across disjoint caches — and risk a 20–50× pricier upstream.
✅ Weigh latency separately from cost. Warm prefills are 2–10× faster even when the dollar discount is ~0.
✅ Watch for storage-fee cache types. Rented caches (Moonshot moonshot-v1, Gemini explicit) bill per-token-time for an idle cache; automatic prefix caches don't.

Bottom line

For closed models, "does it cache?" has one answer. For open weights the capability was solved years ago at the inference-engine layer — vLLM and SGLang cache every prefix, for free, automatically. Everything above that layer is plumbing that either preserves the hit or scatters you away from it: a compute host's replica balancer, a gateway's cluster routing, a router's random spread across vendors. The model's architecture sets the ceiling on how cheap caching can be — MLA and GQA are real, model-level wins — but the path your request takes decides what you actually get. Treat cache behavior as a routing property — measure it in cost terms on the exact path you'll run, pin the route so the cache you warmed is the one you hit, and remember that the deepest discount in the world is worth nothing if request two lands somewhere request one never touched.

For the mechanics of why a KV cache exists and how TTLs work, start with How KV Cache & TTL Work; to audit a gateway's cache claims, see Does Your LLM Gateway Lie About Cache?.

FAQ

Do open-weight models support prompt caching?
The weights set how cheap caching can be — attention architectures like MLA and GQA shrink the KV cache — but the cache itself, the discount, and the API come from the serving stack. Caching is implemented in the inference engine (vLLM, SGLang, TensorRT-LLM), inherited by compute hosts, and forwarded (or scattered) by gateways and routers. Ship the same checkpoint to three hosts and you can get free automatic caching, none, or explicit-only.

Why did the same model cost 49× more on one call than another?
On a multi-provider router, an un-pinned request is load-balanced across different vendors' clusters with different base prices and disjoint cache state. One call hit a pricey provider cold; another hit a cheap one warm. Pin the upstream (constrain provider order, fallbacks off) to control both.

If I self-host, do I need to pay for caching?
No. Automatic prefix caching in vLLM, SGLang, and TensorRT-LLM is on by default and free — a hit just skips prefill. You pay only for the GPUs you already run, and the cache is yours, evicted by LRU when VRAM is needed.

The API says cached_tokens: 0 but my bill dropped — did caching work?
Probably yes. Many gateways don't populate cached_tokens for upstreams that cache automatically. Trust the cost field: a large drop between a cold and an identical warm call means the cache hit.

Which open-weight model has the deepest cache discount?
DeepSeek's automatic disk cache: deepseek-v4-flash reads cached input at ~$0.0028/M against $0.14/M uncached (~98% off), reproduced at 97.9–99.2% across the V4 line in our cold→warm tests. Many third-party hosts apply a flat ~50% instead.

Is there a catch with caches that charge a storage fee?
Yes — Moonshot's moonshot-v1 explicit cache and Gemini's explicit cache bill per-token-time to keep the cache alive (Gemini ~$1–4.50 / 1M-tokens / hour). An idle cache you forgot to delete keeps charging. Automatic prefix caches have no storage fee.

Verification: live cost/latency figures measured 2026-06-14 against a multi-provider router and our own gateway, using a fixed ~4.7K-token prompt, small max_tokens, sequential cold→warm runs; discounts computed from the returned per-request cost. Documented pricing and cache mechanics checked against primary provider docs the same day and cross-verified adversarially; a few vendor figures (notably Moonshot's explicit-cache fees) move frequently — confirm current values before quoting. Your numbers will vary with provider, prompt, region, and load.

Sources

All checked 2026-06-14. Not financial advice; verify current pricing before relying on it.

Claude Fable 5's 30-Day Retention: ZDR, HIPAA, COPPA

synthorai — Fri, 12 Jun 2026 06:54:46 +0000

If your organization runs Claude under a zero-data-retention (ZDR) agreement, your first request to claude-fable-5 didn't return a completion. It returned 400 invalid_request_error. That's not an outage — it's policy. Fable 5 is the first generally available Claude model that cannot be used without 30-day data retention, and the requirement follows the model onto every platform: the Claude API, AWS Bedrock, Google Vertex AI, and Microsoft Foundry each gate it behind an explicit retention opt-in.

For teams that treated "we don't retain your data" as a settled property of their LLM stack, this is an architectural event. This post covers what the policy says, why the window exists, how each cloud implements it, and what it changes for consumer products and sensitive-data industries.

Policy details were checked against Anthropic's, AWS's, Google's, and Microsoft's published documentation on 2026-06-12. Policies change; verify against the linked primary sources and your own contracts. This is an engineering overview, not legal advice.

What the policy actually says

Anthropic designates Claude Fable 5 and Claude Mythos 5 as Covered Models. Per the API data retention docs and the Mythos-class retention practices article (effective 2026-06-09):

Prompts and completions are retained for 30 days, then automatically deleted — unless flagged for an active safety investigation or required by law.
There is no opt-out. Retention is a condition of using the model. A request from an organization whose retention configuration doesn't meet the requirement returns 400 invalid_request_error.
Access is narrow by design. Automated safety systems screen the data; only a small group of approved personnel can review flagged conversations, they cannot export, copy, or download it, and every access lands in tamper-proof logs.
Existing ZDR agreements do not carry over to Covered Model traffic — including through cloud platforms.

Consumer plans (Claude Free/Pro/Max) are unaffected — they already operate under their own retention terms. This policy targets the commercial API surface, exactly where "we never retain" promises tend to live.

Why a 30-day window exists

The rationale in the Covered Models article is specific: these models have substantially advanced capabilities in software engineering, agentic workflows, and cybersecurity, and "some forms of misuse only become detectable across many requests." The cited examples — best-of-N jailbreaking, state-sponsored espionage — are attack patterns where each prompt looks benign and only the sequence is diagnostic. You can't detect a sequence you've deleted.

Two things the window is not:

Not training data. Anthropic states retained data is never used for training without express permission. The purpose is abuse detection, full stop.
Not new in kind — new in enforceability. A ~30-day abuse-monitoring window has been the industry default for years: OpenAI keeps API abuse logs up to 30 days (ZDR by approval); Azure OpenAI stores prompts up to 30 days unless approved for modified abuse monitoring. What changed is that the window became non-negotiable for one model class — previously every provider offered a zero-retention escape hatch.

One pre-existing caveat that surprises people: even under ZDR, Anthropic retains safety classifier results, and content flagged for Usage Policy violations can be kept up to 2 years. Zero data retention has never meant zero data — it means zero retention of unflagged content in the normal path.

Same requirement, three clouds, three mechanisms

The retention applies wherever the model runs, but each platform wires the opt-in differently — and the differences decide who processes your data and where your controls live.

Platform	Opt-in mechanism	Scope	Without opt-in
Claude API	30-day retention in Privacy controls	Organization or workspace	`400 invalid_request_error`
AWS Bedrock	`data_retention_mode: provider_data_share`	Account or project	Model listed `unavailable`; requests blocked
Google Vertex AI	Anthropic data sharing + Model Garden terms	Project	Requests blocked until enabled
Microsoft Foundry	Anthropic's terms accepted at deployment	Subscription/deployment	Not covered by Azure's ZDR program at all

AWS Bedrock is the most explicit. Data retention is a configurable mode (default / provider_data_share / none), resolved project → account → model default. Fable 5 declares allowed_modes: ["provider_data_share"]: prompts and completions are shared with Anthropic and retained up to 30 days. Under any other mode:

{
  "id": "anthropic.claude-fable-5",
  "status": "unavailable",
  "status_reason": "This model is not available under data retention mode 'default'.",
  "data_retention": {
    "mode": "default",
    "source": "account",
    "allowed_modes": ["provider_data_share"]
  }
}

Nothing changed for pre-Fable-5 models, and an SCP on the bedrock:DataRetentionMode condition key can enforce your posture org-wide — nobody quietly flips the account to try the new model. Note: with cross-region inference, the retained copy lives in the destination region, which matters if you carry residency commitments.

Google Vertex AI gates the model behind a project-level Anthropic data-sharing setting (setPublisherModelConfig with dataSharingEnabledProvider: "anthropic") plus terms acceptance in Model Garden, per Google's Fable 5 documentation. General data handling follows Vertex AI's data-governance policy; for residency-sensitive workloads, Vertex's regional and multi-region endpoints control where inference runs — which now includes where the retained copy lives.

Microsoft Foundry is structurally different. Microsoft's data and privacy documentation is explicit that Claude models are third-party marketplace services: you accept Anthropic's terms at deployment, and Anthropic — not Microsoft — is the data processor. Azure OpenAI's ZDR and modified-abuse-monitoring programs don't extend to Claude deployments. Organizations with ZDR postures elsewhere typically isolate Covered Model use in a dedicated subscription, making the retention boundary structural rather than procedural.

The pattern across all three: retention class became a first-class, machine-readable model attribute — a mode, a flag, a terms gate — rather than a paragraph in a contract. Your infrastructure can now enforce your data posture, and it should.

What it means for enterprise deployments

With no ZDR agreement, nothing changes mechanically — you were already in a 30-day-style posture, possibly without realizing it. The work is making it explicit in your vendor documentation.

With a ZDR agreement, you have a three-way choice:

Skip Covered Models. ZDR stays uniform; you give up the model. Viable if your workloads don't need it — see our measured Fable 5 evaluation for what it costs and where it differs.
Split by workspace or project. Every platform supports a scoped opt-in: a designated Claude API workspace (Console → Settings → Workspaces → Privacy controls), a Bedrock project with provider_data_share, a separate Vertex project or Azure subscription. Route only retention-tolerant workloads there.
Accept retention org-wide. Simplest to operate, but it silently downgrades the guarantee for every workload — including the ones whose sensitivity justified ZDR. That's a decision for your data-protection owner, not a config change.

Regardless of provider: your own logging is a second retention surface. If your gateway or observability stack logs full prompts, you're running a longer window than your provider, under your own roof. Provider guarantees are only as meaningful as the layer in front of them — the same audit logic we applied to cache claims applies here.

What it means for consumer-facing products

If you serve consumers and route their content through a Covered Model, the change propagates into your own legal surface — ZDR agreement or not. Three concrete consequences:

1. Your privacy notice probably needs an update. Most regimes require disclosing retention, not just collection: GDPR Article 13(2)(a) requires the storage period (or criteria) at collection time; California's CPRA requires the notice at collection to state retention per category of personal information. If your notice says — or implies — that conversation data isn't retained anywhere, a processor holding a 30-day copy makes it wrong. Update the notice, the records of processing, and the DPA inventory.

2. You cannot offer users an opt-out you don't have. The retention has no exception mechanism, so there is no toggle you can build that exempts a user's prompts while still using that model. The lever you actually hold is routing: a consent-aware gateway sends users who decline data sharing to ZDR-eligible models and everyone else to the Covered Model — a legal constraint turned into an ordinary routing rule. Far better than a preference checkbox that does nothing.

3. Deletion requests need accurate plumbing. Erasure obligations (GDPR Art. 17, CPRA deletion, and their equivalents) extend to processors. A bounded window that auto-deletes within 30 days is generally a defensible processor posture — but your DSAR playbook should say that, not promise immediate downstream deletion you can't execute.

The global dimension compounds this: the same disclosure-and-processor logic appears in the UK GDPR, Brazil's LGPD, and the spreading family of US state privacy laws. For users in China, PIPL adds two sharper edges — providing personal information to another processor generally requires separate consent, and routing Chinese users' content to an overseas LLM endpoint is a cross-border transfer needing a recognized mechanism (security assessment, standard contract, or certification). A model upgrade that changes who retains what, where, for how long is exactly the change these frameworks expect you to re-paper.

Sensitive-data industries: where 30 days bites hardest

For most products the provider's window is a documentation problem. For industries whose data is itself regulated, it's an architecture problem: the retained copy is regulated data at rest at a vendor, and sector rules govern exactly that.

Healthcare (HIPAA)

HIPAA doesn't require zero retention — it requires that any vendor holding protected health information does so under a Business Associate Agreement (BAA) with appropriate safeguards. The 30-day copy of your prompts is PHI at rest at a business associate; the question is whether your BAA covers it. The two major API vendors structure this differently, and the difference now matters: Anthropic's HIPAA-ready API access explicitly doesn't require ZDR — it's built on retention-with-safeguards (encryption, access controls, audit logging, enforced feature restrictions). OpenAI's API BAA covers endpoints eligible for zero data retention — and a BAA scoped to ZDR endpoints structurally cannot cover a model that mandates retention.

A model's retention class is now a BAA-eligibility question. Confirm in writing that your BAA covers the specific model before routing PHI to it — and remember the chain shifts on clouds: on Bedrock the platform is your business associate; on Foundry, Anthropic processes the data directly. One sharp edge: PHI must never appear in JSON schema definitions for structured outputs — cached schemas don't get the same protections as message content.

Children's products (COPPA)

The timing is awkward: the FTC's amended COPPA Rule took effect June 23, 2025, with compliance on most provisions due April 22, 2026 — the first model with mandatory provider-side retention arrived just as operators finished implementing the new retention obligations. Two of those interact directly with the 30-day window: a written, public data retention policy is now mandatory (§312.10) — what children's data is collected, why, and when it's deleted — and indefinite retention is prohibited, with retention limited to what's reasonably necessary for the collected purpose.

A bounded 30-day window with automatic deletion is the compatible shape — but the provider retains for its trust-and-safety purpose, not the purpose you collected the child's data for, and your notice must describe the processor relationship accurately. For child-directed products that adopted ZDR specifically to minimize the data trail, the routing answer applies with higher stakes: children's traffic stays on ZDR-eligible models, or the Covered Model window goes into your §312.10 policy first.

The same pattern, other sectors

Once you see the structure — regulated data, retained copy at a vendor, sector rule governing retention — it recurs:

Biometrics (Illinois BIPA): operators need a written, publicly available retention schedule and destruction guidelines for biometric data. A provider's 30-day copy of prompts containing biometric identifiers belongs in that schedule.
Payments (PCI DSS / GLBA): PCI DSS prohibits storing sensitive authentication data after authorization — anywhere. Card data pasted into a prompt becomes card data retained at a provider for 30 days. The clean answer is upstream redaction, not downstream paperwork.
Education (FERPA): vendors handling student records under the school-official exception must remain under the school's direct control. A safety-retention copy the school cannot access or delete early sits uneasily with that standard — a question for counsel before EdTech traffic hits a Covered Model.
Financial services — the inversion (SEC/FINRA): broker-dealers must retain business communications under books-and-records rules. For them the provider's window isn't the problem; capturing their own compliant copy is. Same retention question, opposite sign.

The common thread: sector rules regulate retention in both directions, and a provider-side window you don't control must be mapped into whichever direction your sector points.

A decision checklist

✅ Inventory which models your traffic actually touches. Retention class is now a per-model attribute, not a per-provider one.
✅ If you have ZDR: decide deliberately — skip Covered Models, split by workspace/project/subscription, or accept retention org-wide. Don't let it happen implicitly.
✅ Enforce the posture in infrastructure — Bedrock SCPs, workspace privacy controls, separate cloud projects — not in a wiki page.
✅ B2C: update privacy notices and DSAR playbooks; route non-consenting users to ZDR-eligible models instead of building opt-outs that can't work.
✅ Regulated data: confirm coverage per model, in writing — BAA for PHI, §312.10 policy for children's data, retention schedules for biometrics — before routing that data to a retention-required model.
✅ Audit your own logging. A provider's 30-day window is irrelevant if your gateway logs prompts indefinitely.

Bottom line

The 30-day window attached to Fable 5 is not a data grab — it's bounded, purpose-limited abuse monitoring, consistent with what most of the industry already does by default, made mandatory for one model class because cross-request misuse detection doesn't work on deleted data. For most teams the engineering impact is zero and the governance impact is a paragraph in a vendor review.

But for organizations whose compliance position assumed zero retention — ZDR-scoped BAAs, privacy notices that say nothing persists, children's products built on data minimization — Fable 5 is the moment that assumption stopped being uniform across models. The fix isn't avoiding the model. It's making retention class an explicit, per-model input to routing decisions, the same way you already treat price and context window.

FAQ

Can I use Claude Fable 5 under a zero-data-retention agreement?
No. Fable 5 and Mythos 5 are Covered Models requiring 30-day retention; ZDR organizations get a 400 invalid_request_error unless they enable 30-day retention for a workspace and route Fable 5 traffic through it.

Does going through AWS Bedrock, Vertex AI, or Microsoft Foundry avoid the requirement?
No. Each platform gates the model behind its own retention opt-in: provider_data_share on Bedrock, Anthropic data sharing plus Model Garden terms on Vertex, Anthropic's terms at deployment on Foundry (where Anthropic, not Microsoft, is the data processor). Existing ZDR arrangements don't carry over on any of them.

Can my end users opt out of the retention?
No — there is no opt-out mechanism. The lever you hold is routing: send users who decline data sharing to ZDR-eligible models. Don't ship a preference toggle that doesn't change anything.

Is the retained data used to train models?
Anthropic states retained data is never used for training without express permission. The purpose is trust-and-safety review: automated screening, with flagged conversations reviewable only by approved personnel who cannot export the data, under tamper-proof access logs.

Does the 30-day retention change how prompt caching works?
No. Cache entries follow their own short TTLs (5 minutes or 1 hour) and the caching contract on Fable 5 is unchanged — see our measured evaluation. The 30-day window is a separate, parallel retention for safety review.

Sources

All checked 2026-06-12. Policies change — verify against current documents and your own contracts. Not legal advice.

Claude Fable 5: Caching, Tokenizer & Cost vs Opus 4.6

synthorai — Thu, 11 Jun 2026 15:49:46 +0000

claude-fable-5 is now available on the Synthorai gateway. If you cache against the Claude line, the good news is that the caching and TTL contract is a carry-over: same cache_control markers, same 5-minute and 1-hour TTLs, same write premiums, same deep read discount. Your caching code moves over by changing one string.

The thing to budget for isn't the cache mechanics — it's the bill. Fable 5 lists at 2x the Opus token price, and it tokenizes the same English text into ~45% more tokens than Opus 4.6 (it's on the post-4.6 tokenizer, identical to Opus 4.8). Those two multipliers stack. This post measures all of it so you don't have to.

All numbers below were measured against https://synthorai.io/ (Anthropic-native /v1/messages) on 2026-06-10 with a stable ~6.6–9.6K-token English system prompt, max_tokens small, single sequential run. Cost figures are read from the gateway usage.cost field; ratios (token counts, write premium, read discount, cross-model cost) are the portable part — absolute dollars scale with your prompt. Reproduce against your own prompt before quoting them.

Availability

import os
from anthropic import Anthropic

anth = Anthropic(
    api_key=os.environ["SYNTHORAI_KEY"],
    base_url="https://synthorai.io/",   # SDK appends /v1/messages
)

msg = anth.messages.create(
    model="claude-fable-5",             # the only line that changes
    max_tokens=512,
    system=[
        {"type": "text", "text": SYSTEM_PROMPT,
         "cache_control": {"type": "ephemeral"}},
    ],
    messages=[{"role": "user", "content": question}],
)
print(msg.usage)   # input_tokens, cache_creation_input_tokens, cache_read_input_tokens, cost

Swap claude-opus-4-6 → claude-fable-5 and nothing in your caching path needs to move. Fable 5 is an Anthropic-native model with a 1M-token context window. One behavioral note: it is a reasoning model and emits thinking tokens by default — even a trivial "reply OK" returned output_tokens_details.thinking_tokens > 0 in our runs, where Opus 4.6/4.8 returned zero. Budget output tokens accordingly. The mechanics behind cache_control are covered in the caching tutorial; the architecture of why the cache exists is in Part 1 of the series.

The headline: Fable 5 is on the new tokenizer

The token count for the Opus line jumped at the 4.7 generation: the same English text that counted as ~6.6K tokens on 4.6 counts as ~9.6K on 4.8. Fable 5 lands on the new side — identical text reports the exact same token count as Opus 4.8.

Model	Input tokens (identical text)	Tokenizer generation
`claude-opus-4-6`	6,614	pre-4.7
`claude-opus-4-8`	9,619	post-4.7
`claude-fable-5`	9,619	post-4.7 (identical to 4.8)

The same system prompt is ~45% more tokens on Fable 5 than on Opus 4.6 (9,619 / 6,614 = 1.45). This is the single most important number to internalize before you migrate, because every downstream figure — cost, the 1,024-token cache-eligibility floor, your per-call budget — is computed in tokens.

We're describing a measured observation — identical text, identical token count on Fable 5 and Opus 4.8, ~45% above Opus 4.6 — most consistent with the tokenizer/vocabulary update that shipped at the 4.7 generation. If you're coming from 4.6 or earlier, re-measure; if you're coming from 4.7/4.8, expect parity.

Caching behavior: the contract is unchanged

We ran the same no-cache / cold-write / warm-read sequence on each model. The discount structure is identical end to end — Fable 5 honors cache_control and reports the same usage fields (cache_creation_input_tokens, cache_read_input_tokens, and the ephemeral_5m / ephemeral_1h buckets).

Model	5m cache write	1h cache write	Warm read
`claude-opus-4-6`	1.25x	2.00x	~9% of no-cache
`claude-opus-4-8`	1.25x	2.00x	~6% of no-cache
`claude-fable-5`	1.24x	1.99x	~6% of no-cache

Two invariants hold across all three:

Write premium ≈ 1.25x (5m), ≈ 2x (1h). The first (cold) call costs ~1.25x the no-cache price to populate a 5-minute entry, or ~2x for a 1-hour entry. Break-even is one hit.
Read discount ≈ 90%+. A warm cache read on Fable 5 cost ~6% of the no-cache call — a ~94% discount, in line with (slightly better than) Anthropic's documented ~90% cached-read economics. Reads stay deeply discounted regardless of TTL.

The percentages are flat across the line. As with the Opus 4.7 → 4.8 step, the higher absolute bill on Fable 5 is a price-and-token story, not a cache-economics story — covered next.

TTL behavior: both windows honored

Fable 5 supports the same two TTLs as the rest of the line: a 5-minute sliding default and an opt-in 1-hour window. We isolated each TTL with a unique prefix per call (so no stale entry could contaminate the result) and confirmed the usage object reports the correct bucket — cache_creation.ephemeral_5m_input_tokens or ephemeral_1h_input_tokens.

# 1-hour TTL — same marker syntax on Fable 5 as on the Opus line
"cache_control": {"type": "ephemeral", "ttl": "1h"}

The 1-hour write costs ~2x no-cache (vs ~1.25x for the 5-minute write), and reads stay at the deep discount regardless of TTL — identical to Opus 4.6/4.8. If you picked 5m for live chat and 1h for agents with human-in-the-loop pauses on Opus, keep those choices on Fable 5.

The cost story: 2x price x 1.45x tokens

Here is where Fable 5 actually differs. Two things push the bill up, and they multiply.

1. List price is 2x the Opus tier.

Model	Input ($/M)	Output ($/M)	Cache read ($/M)
`claude-opus-4-6` / `4-8`	5	25	0.5
`claude-fable-5`	10	50	1

2. The same text is ~45% more tokens than on 4.6 (the tokenizer shift above).

Multiply them and the same English prompt costs materially more. Measured against the identical system prompt on each model (gateway usage.cost, same single run):

Comparison	Token ratio	Price ratio	Same-prompt cost ratio (measured)
Fable 5 vs Opus 4.8	1.00x	2.0x	2.0x
Fable 5 vs Opus 4.6	1.45x	2.0x	2.9x

So against Opus 4.8 (same tokenizer), Fable 5 is a clean 2x — pure price premium. Against Opus 4.6, the tokenizer change compounds the price change into roughly 2.9x the cost for the same prompt. Your cache discount is unchanged, but the absolute base it applies to is ~2.9x larger than it was on 4.6. If you sized a per-call budget against 4.6, re-do it.

A practical consequence: re-check the 1,024-token cache-eligibility floor. Anthropic only caches prefixes at or above a minimum size. A prompt that sat just under the floor on 4.6 (in old-tokenizer tokens) may clear it on Fable 5 (~45% more tokens) — and vice versa for size estimates built on the old count. Always read cache_creation_input_tokens / cache_read_input_tokens from the live response rather than estimating from a local tokenizer that may not match.

Migration checklist (Opus → Fable 5)

✅ Caching code carries over verbatim. cache_control markers, breakpoint count (up to 4), ttl: "1h", usage-field names — all identical.
✅ TTL choices carry over. 5m for live/session workloads, 1h for bursty/agent-with-pauses.
✅ Discount economics carry over. ~90%+ read, ~1.25x write (5m), ~2x write (1h).
⚠️ Re-budget on absolute cost. Fable 5 is ~2x Opus per token, and ~2.9x the same-prompt cost vs Opus 4.6. The discount percentage is unchanged; the base it applies to is not.
⚠️ Re-measure token counts if coming from 4.6 or earlier (expect ~45% more for the same text). From 4.7/4.8, expect parity.
⚠️ Account for default thinking tokens. Fable 5 emits reasoning tokens by default — they bill at the output rate ($50/M). Cap or disable thinking if you don't need it.

Bottom line

For a team already caching against Claude, claude-fable-5 is an easy integration: the entire caching and TTL surface is stable, so there's nothing to relearn and no code to rewrite. It is not an easy budget swap from Opus 4.6 — between the 2x token price and the ~45% tokenizer inflation, the same prompt runs ~2.9x the cost. Confirm your numbers against the live usage object, decide whether you need the default thinking tokens, and size the cache breakpoints against the new token counts.

For the full caching playbook — prompt structure, hit-rate debugging, TTL-aware patterns — see the four-part series starting with How KV Cache & TTL Work and the working Python tutorial.

FAQ

Do I need to change my cache_control code to use Fable 5?
No. The marker syntax, breakpoint limit, and TTL options are identical to the Opus line. Change the model field and nothing else in the caching path.

Did the cache read discount change on Fable 5?
No. A warm read is a small single-digit fraction of the no-cache input price (~90%+ off) — we measured ~94% on Fable 5, consistent with Anthropic's documented cached-read economics.

Does Fable 5 support the 1-hour TTL?
Yes. {"type": "ephemeral", "ttl": "1h"} works exactly as on Opus. The 1-hour write costs ~2x no-cache; the 5-minute write ~1.25x. Reads stay deeply discounted on both.

Why is the same prompt so much more expensive on Fable 5 than on Opus 4.6?
Two stacked multipliers: Fable 5 lists at 2x the per-token price, and the same English text counts as ~45% more tokens (it uses the post-4.6 tokenizer). Together that's ~2.9x the cost for an identical prompt. The cache discount is unchanged.

Is Fable 5 a drop-in replacement for Opus 4.8?
On the caching/TTL surface and token counts, yes — token counts are identical, so the only delta is the 2x price and Fable 5's default thinking tokens. We don't publish capability benchmarks we haven't run; for quality and reasoning claims, see Anthropic's model card.

Verification: all token-count, cost, write-premium, and read-discount figures measured against https://synthorai.io/ on 2026-06-10 using the official anthropic SDK, single tenant, single sequential run. Cost is read from the gateway usage.cost field; cross-model and premium/discount ratios are computed from those measured costs and are independent of any account-level promotion. Discount/premium ratios cross-checked against Anthropic Prompt Caching docs. Warm-read latency (TTFT) was dominated by network jitter in our run and is omitted as unreliable. Your numbers will vary with prompt, region, and load.

Provider Drift: How Default Routing Inflates LLM Cost 3.9 — A Measurement

synthorai — Fri, 05 Jun 2026 09:34:34 +0000

You turned on prompt caching, the hit counter ticks now and then, but your bill barely moved. Before blaming your prompt structure, look at something the dashboard hides: which upstream actually served each request.

Multi-provider gateways spread a single model across several upstream providers and pick one per request. Prompt caches are per-provider (often per-node inside a provider). So when your second identical request lands on a different upstream than the first, it is a cache miss, even though your prompt did not change one byte. This is provider drift, and on a pay-per-token model it quietly multiplies your cost.

The two conditions that trigger it

This is not a misconfiguration you opted into. It is what you get out of the box:

Default auto routing. The request is sent to the model without pinning an upstream, so the gateway chooses one per call.
Default provider sort = "default (balanced)". The gateway load-balances across eligible upstreams rather than sticking to one.

Both are the factory defaults. You do not have to touch anything to get drift; you have to touch settings to avoid it.

What 20 identical requests look like

We sent the same ~8K-token prefix 20 times in a row to one popular multi-provider gateway, on the defaults above, asking for the upstream's own reported provider and cache fields each time. For a disk-cached model in the DeepSeek family:

9 distinct upstreams served the 20 calls: N***a, S***w, M***h, D***a, A***L, P***l, S***e, V***e, A***d.
Cache hit rate: 4/20 (20%). You only hit on the calls that happened to land on an upstream that had already cached your prefix.

Run the same 20 calls against a single-backend gateway (one model, one upstream, no balancing) and the hit rate is 19/20 (95%) on the identical workload. Same model, same prompt, same number of calls. The only variable is whether routing drifts.

For contrast, on the very same multi-provider gateway a GPT-class model was routed to one upstream (A***e) for all 20 calls and hit 19/20. Drift is not uniform; it bites whichever model the gateway happens to spread, and on this run that was the DeepSeek-family model.

Conclusion A: the cost you expected vs the cost you paid

Per-call cost on the drifting model split cleanly by cache outcome:

call type	median cost / call
cache hit	~$0.00015
cache miss	~$0.00062

A miss costs about 4x a hit on this model (on raw input tokens the published gap is wider still, roughly 50x). Now total it across the 20 calls:

scenario	hit rate	cost for 20 identical calls
expected (cache reachable)	95%	$0.0026
actual (default drift)	20%	$0.0102

Same model, same prompt, same 20 requests. Provider drift made the run cost ~3.9x more. The caching was "on" the whole time; the routing layer simply billed most of your tokens at the miss rate. Scale that to a production endpoint replaying a large stable prefix all day and the gap is the bulk of your input spend.

Conclusion B: no cache also means no latency win

Caching is not only a cost lever. A warm prefill returns the first token sooner. When drift denies you the cache, you forfeit that speedup too. We measured time-to-first-token (TTFT) on repeated identical calls:

GPT-class model (routed to one consistent upstream, cache reachable):

call	TTFT
1st (cold, miss)	~1760 ms
subsequent (warm, hit)	~1130 ms

Caching buys roughly a 36% faster first token, and it is steady: every warm call lands in a tight band.

DeepSeek-family model (default drift, cache rarely reachable):

Cache hits across a 10-call repeat: 0.
TTFT swung from ~1000 ms to ~4500 ms call to call, with occasional empty responses.

Because almost every request is a fresh upstream, you stay at cold-prefill latency and inherit the variance of whichever provider answered. The GPT model got a 36% TTFT improvement from a reachable cache; the drifting model got none, plus a 4.5x spread between its fastest and slowest call.

Audit your own setup in five minutes

Do not trust these numbers, or anyone's. Send the same long prefix several times and watch two fields. No domains hardcoded; point it at your own gateway with env vars.

import os, uuid
from openai import OpenAI

client = OpenAI(api_key=os.environ["GW_KEY"], base_url=os.environ["GW_BASE"])
SYS = f"[probe {uuid.uuid4().hex}]\n\n" + ("You are a support assistant. " * 300)

seen, hits = {}, 0
for i in range(20):
    r = client.chat.completions.create(
        model=os.environ["GW_MODEL"], max_tokens=16,
        messages=[{"role": "system", "content": SYS},
                  {"role": "user", "content": f"q{i}"}],
        extra_body={"usage": {"include": True}})
    d = r.model_dump()
    det = r.usage.prompt_tokens_details
    cached = (getattr(det, "cached_tokens", 0) or 0) if det else 0
    seen[d.get("provider")] = seen.get(d.get("provider"), 0) + 1   # populated when exposed
    hits += 1 if cached else 0

print(f"hit rate {hits}/20; upstreams seen: {len(seen)}")

More than one upstream for the same model means drift. A hit rate well below your prompt stability means it is taxing you. The fuller method is in Does Your LLM Gateway Lie About Cache?.

What to look for

The cure for drift is structural: route a given model to a consistent backend so a warm cache is actually reachable on the next request, instead of load-balancing each call onto a fresh upstream that has never seen your prefix. When you evaluate a gateway, send the same prefix 20 times and count the upstreams. One is what you want. Nine is a tax.

A fair caveat: prompt caching is best-effort everywhere, and on disk-cached models the hit rate still softens over long idle gaps even with a single backend. Eliminating drift does not hand you an infinite cache. It removes the largest and most wasteful source of misses, the one you never agreed to and cannot see.

Closing

"Supports prompt caching" and "your cache is reachable" are different claims. A gateway that scatters one model across a rotating cast of upstreams can report cache support truthfully while delivering a 20% hit rate, a ~4x bill, and first-token latency that swings 4.5x. The number to watch is not whether caching is advertised. It is your measured hit rate and how many upstreams your identical requests touch. Run the probe and let the data settle it.

For the broader audit method see Does Your LLM Gateway Lie About Cache?; for why caches exist at all, see How KV Cache & TTL Work.

FAQ

Is this a misconfiguration on my side?
No. It happens on the factory defaults: auto routing with the provider sort left at "default (balanced)." Avoiding drift requires actively pinning an upstream, not the other way around.

Does pinning one upstream fix it?
It removes cross-provider drift, but a single upstream often runs multiple replicas without prefix affinity, so hits can still flip-flop. Measure after pinning rather than assuming.

Why did the GPT-class model not drift?
On this run the gateway happened to route it to a single upstream. Drift is per-model and depends on how many eligible upstreams the gateway balances across; it is not uniform.

Is the cost gap really ~4x?
On the per-call totals we measured, a miss was ~4x a hit; on raw input-token pricing for this model class the published hit-vs-miss gap is closer to 50x. Either way, turning expected hits into misses is the expensive part.

What single metric should I monitor?
Cache hit rate per model over time, alongside the count of distinct upstreams per model. If hit rate falls or upstream count rises, your effective token cost just went up.

Does Your LLM Gateway Lie About Cache? A 5-Min Audit

synthorai — Tue, 02 Jun 2026 15:30:00 +0000

A gateway sits between your code and the model provider. You read cached_tokens back from the response, you see a smaller number, and you trust the dollars saved are real. But you never see the upstream call. The gateway could report a cache hit and still bill the full input rate. It could fail to cache at all behind a perfectly clean response. It could strip usage metadata on streaming, the path most of your production traffic runs on, so you can't tell either way.

This isn't hypothetical. A Hacker News PSA reported that routing DeepSeek V4 through a popular gateway returned 2–3× fewer cached tokens than calling DeepSeek directly; one commenter posted bills showing the caching stats weren't reported through the gateway at all. The gateway's team replied that they couldn't reproduce it and were investigating. That disagreement is the whole point. When two parties can't agree on whether your cache is working, the only tiebreaker is a measurement you ran yourself.

Usually this isn't malice. It's a translation gap or an unfinished code path. The effect on your invoice is the same either way. This post is one runnable script that audits both styles of prompt caching, automatic (DeepSeek) and marker-based (Claude), against any gateway, including this one. It prints a side-by-side scorecard in under five minutes.

Four ways a gateway can lie about cache

Failure mode	What you see	What's actually happening
Silent no-cache	A clean response, no error	Nothing was cached; you pay full price every call
Cache theater	`cached_tokens` > 0 in the response	…but the billed cost is the full input rate
Markup creep	A plausible cost number	The gateway's markup quietly eats the discount
Metadata blackout	Clean text output	Usage fields stripped (esp. on streaming), so you can't audit it

The dangerous ones are the first two: the response looks like caching is working. You find out at the end of the month.

Two cache mechanisms, one audit

Providers expose caching in two shapes, and a real gateway has to pass both through faithfully:

Automatic (DeepSeek, GPT, Gemini, Qwen): the provider caches any sufficiently long prefix on its own. No markers. Hits appear in usage.prompt_tokens_details.cached_tokens.
Marker-based (Anthropic Claude): you tag cacheable spans with cache_control. Hits appear as cache_read_input_tokens.

The script hides that difference behind a thin Lane adapter, then runs all five checks against both. Here is the whole thing: two lanes and one audit() that performs every check.

import os, time, uuid
from openai import OpenAI
from anthropic import Anthropic

KEY  = os.environ["GATEWAY_KEY"]
oai  = OpenAI(api_key=KEY,    base_url="https://synthorai.io/v1")   # auto lane
anth = Anthropic(api_key=KEY, base_url="https://synthorai.io/")     # marker lane

class AutoLane:      # DeepSeek / GPT / Gemini / Qwen: provider caches automatically
    mode = "auto"
    def __init__(self, model): self.model = model
    def call(self, sys, q, stream=False):
        if stream:
            cached = cost = None
            s = oai.chat.completions.create(model=self.model, max_tokens=48, stream=True,
                stream_options={"include_usage": True},
                messages=[{"role":"system","content":sys},{"role":"user","content":q}])
            for ev in s:
                if ev.usage:
                    d = ev.usage.prompt_tokens_details
                    cached, cost = (d.cached_tokens if d else None), getattr(ev.usage,"cost",None)
            return {"cached": cached or 0, "cost": cost, "prompt_total": None}
        u = oai.chat.completions.create(model=self.model, max_tokens=48,
            messages=[{"role":"system","content":sys},{"role":"user","content":q}]).usage
        cached = u.prompt_tokens_details.cached_tokens if u.prompt_tokens_details else 0
        return {"cached": cached or 0, "cost": u.cost, "prompt_total": u.prompt_tokens}

class MarkerLane:    # Anthropic Claude: explicit cache_control markers
    mode = "marker"
    def __init__(self, model): self.model = model
    def call(self, sys, q, stream=False):
        block = {"type":"text","text":sys,"cache_control":{"type":"ephemeral"}}
        if stream:
            with anth.messages.stream(model=self.model, max_tokens=48, system=[block],
                    messages=[{"role":"user","content":q}]) as s:
                for _ in s.text_stream: pass
                u = s.get_final_message().usage.model_dump()
            return {"cached": u.get("cache_read_input_tokens") or 0,
                    "cost": u.get("cost"), "prompt_total": None}
        u = anth.messages.create(model=self.model, max_tokens=48, system=[block],
            messages=[{"role":"user","content":q}]).usage.model_dump()
        read, created = u.get("cache_read_input_tokens",0), u.get("cache_creation_input_tokens",0)
        return {"cached": read, "cost": u.get("cost"),
                "prompt_total": u.get("input_tokens",0) + read + created}

def audit(lane, long_prompt):
    SYS = f"[audit {uuid.uuid4().hex}]\n\n" + long_prompt    # unique => guaranteed cold start
    r = {"lane": lane.model, "mode": lane.mode}

    # CHECK 1: cache engages. Cold misses; a repeat should hit. A cache can
    # take a moment to become readable, so poll the warm read (sleep 1s between
    # attempts) before concluding "no cache".
    cold = lane.call(SYS, "Q1")
    warm = cold
    for i in range(4):
        warm = lane.call(SYS, f"warm {i}")
        if warm["cached"] > 0: break
        time.sleep(1.0)
    r["cold"], r["warm"] = cold, warm
    r["check1"] = cold["cached"] == 0 and warm["cached"] > 0

    # CHECK 2: cost reflects the discount (catches "cache theater").
    disc = (1 - warm["cost"]/cold["cost"])*100 if cold["cost"] and warm["cost"] else None
    r["discount"], r["check2"] = disc, (disc is not None and disc > 30)

    # CHECK 3: token accounting. cached fits inside the prompt total.
    r["check3"] = warm["prompt_total"] is None or warm["cached"] <= warm["prompt_total"]

    # CHECK 4: streaming preserves usage metadata (cache count AND cost).
    st = lane.call(SYS, "stream", stream=True)
    r["stream_cached"], r["stream_cost"] = st["cached"] > 0, st["cost"] is not None
    r["check4"] = r["stream_cached"] and r["stream_cost"]

    # CHECK 5: negative control. a unique prefix must always miss.
    n1 = lane.call(f"[uniq {uuid.uuid4().hex}]\n\n"+long_prompt, "x")
    n2 = lane.call(f"[uniq {uuid.uuid4().hex}]\n\n"+long_prompt, "y")
    r["check5"] = n1["cached"] == 0 and n2["cached"] == 0
    return r

# Any long, STABLE text works as the cacheable prefix: a system prompt, tool
# schemas, or a retrieved document. It only needs to clear the provider's
# minimum cacheable size (see Check 1). Load yours however you like.
LONG_SYSTEM_PROMPT = open("system_prompt.txt").read()   # ~8K+ tokens

for lane in [AutoLane("deepseek-v4-flash"), MarkerLane("claude-opus-4-8")]:
    print(audit(lane, LONG_SYSTEM_PROMPT))

The rest of the post walks each check: the lines that implement it, what both lanes returned, and how to read the result.

Check 1: does the cache engage?

cold = lane.call(SYS, "Q1")
warm = cold
for i in range(4):                       # poll: a cache may take a beat to be readable
    warm = lane.call(SYS, f"warm {i}")
    if warm["cached"] > 0: break
    time.sleep(1.0)
r["check1"] = cold["cached"] == 0 and warm["cached"] > 0

	cold cached	warm cached	result
`deepseek-v4-flash`	0	7,552 / 7,870 (96%)	PASS
`claude-opus-4-8`	0	12,446 / 12,454 (99.9%)	PASS

A cold call on a unique prefix must cache nothing; a repeat must hit. The single most common false alarm is declaring "no cache" after one warm call, because caches don't always become readable instantly. The loop polls a few times with a 1-second pause, which removes the flakiness. If you still get 0 after several warm calls on a prompt above the size floor (~1,024 tokens for most providers; DeepSeek matches at a finer 64), the cache genuinely isn't engaging.

Check 2: does the cost reflect the discount?

disc = (1 - warm["cost"]/cold["cost"])*100 if cold["cost"] and warm["cost"] else None
r["check2"] = disc is not None and disc > 30

	cold cost	warm cost	discount	result
`deepseek-v4-flash`	$0.00107	$0.00030	72.3%	PASS
`claude-opus-4-8`	$0.07112	$0.00672	90.6%	PASS

This is the check that catches cache theater. The warm call's cost must actually drop. DeepSeek's per-call total fell ~72% (the cached input is discounted more steeply; output and the uncached remainder dilute the headline). Claude's cached read is ~90% off. The failure signal is unmistakable: cached_tokens > 0 with identical cold and warm cost means the gateway is reporting a hit it isn't pricing. You're paying full freight for a cache that "works" on paper.

Check 3: do the token counts add up?

r["check3"] = warm["prompt_total"] is None or warm["cached"] <= warm["prompt_total"]

	cached	prompt total	result
`deepseek-v4-flash`	7,552	7,870	PASS
`claude-opus-4-8`	12,446	12,454	PASS

cached has to sit inside the prompt total, with the remainder billed as uncached input. Both reconcile. If cached_tokens exceeds prompt_tokens, or the uncached remainder is implausibly large for a stable prefix, the gateway is mis-accounting: re-tokenizing or double-counting somewhere in the translation.

Check 4: does streaming preserve the metadata?

st = lane.call(SYS, "stream", stream=True)
r["stream_cached"], r["stream_cost"] = st["cached"] > 0, st["cost"] is not None
r["check4"] = r["stream_cached"] and r["stream_cost"]

	stream cached	stream cost	result
`deepseek-v4-flash`	preserved	preserved	PASS
`claude-opus-4-8`	preserved	preserved	PASS

Most production chat streams, so this is the path that matters most. On both lanes the cache hit signal and the cost both survive the stream. cached_tokens and cost come through in the final usage chunk, so your highest-volume path stays auditable. The failure mode to watch for is a gateway that drops usage on streaming: a clean token output with no cached_tokens or cost means you're flying blind on the path you run most. (Pass stream_options={"include_usage": True} so the usage chunk is emitted at all.)

Check 5: the negative control

n1 = lane.call(f"[uniq {uuid.uuid4().hex}]\n\n"+long_prompt, "x")
n2 = lane.call(f"[uniq {uuid.uuid4().hex}]\n\n"+long_prompt, "y")
r["check5"] = n1["cached"] == 0 and n2["cached"] == 0

	unique-prefix A	unique-prefix B	result
`deepseek-v4-flash`	cached 0	cached 0	PASS
`claude-opus-4-8`	cached 0	cached 0	PASS

Send a unique prefix every call; it must never hit. Both lanes correctly reported cached=0 at full cost for distinct prefixes. A "hit" here would make the cache reporting a false positive you could never trust. The clean negative control is what makes the positive results in Checks 1–2 meaningful in the first place.

Reading your scorecard

Check	Healthy result	Red flag
1. cache engages	`0` cold, `>0` warm (after polling)	`0` after several warm calls, above the size floor
2. cost reflects discount	warm cost ≪ cold cost	`cached > 0` but costs equal
3. token accounting	`cached ≤ prompt_total`, reconciles	counts don't add up
4. streaming metadata	cache + cost survive the stream	usage missing on streamed calls
5. negative control	unique prefix always misses	a distinct prefix "hits"

The two that cost money silently are 2 (full price for a reported hit) and 1 (no caching behind a clean response). Run both on every model you bill against.

Closing

Caching is the highest-leverage cost lever in an LLM app, which is exactly why "the cache is working" deserves a test, not an assumption. Wire Check 1 + Check 2 into CI against each model you bill against, alert if the discount drifts below your expected band, and you'll catch a silent regression the day a gateway or upstream provider changes behavior, instead of at the end of the billing cycle. And whatever your audit does, poll the warm read before you call a cache broken.

For the mechanics behind these numbers (prefill, KV cache, TTLs) start with How KV Cache & TTL Work. For working caching patterns per provider, see the tutorial.

FAQ

My Check 1 shows 0 on the warm call. Is my gateway lying?
Check three things first. (1) Does your prompt clear the provider's minimum cacheable size (~1,024 tokens for most; DeepSeek matches at finer 64-token granularity)? (2) Did you poll the warm read a few times? Caches don't always become readable on the very next call. (3) Is the prefix byte-identical between calls, with no timestamps or per-request IDs at the front? Only after all three should you suspect the gateway.

What does "cache theater" cost me in practice?
You pay the full input rate on every call while believing you pay a fraction. On a high-volume endpoint with a large stable prefix, that's your bill being several times what you modeled. Check 2 is the one to alert on.

Why is DeepSeek's discount lower than Claude's here?
Different things are being measured. Claude's ~90% is the read discount on cached input. DeepSeek's ~72% is the per-call total reduction, where output and the uncached remainder are billed at full rate and dilute the headline. Compare like with like for your own prompt shape.

Does this work for GPT, Gemini, Qwen too?
Yes. They're all automatic, so they use the AutoLane unchanged with a different model. Only Claude needs the MarkerLane. Same five checks either way.

Should this live in CI?
Yes. Run Check 1 + Check 2 against every model you bill against, on a schedule, and alert when the observed discount drifts outside your expected band. A standing audit turns a silent regression into a notification.