A cache used to be the boring part of the stack. It hit or it missed, and either way the answer was right and the bill was predictable.
Agent workloads broke that. The prompt is the payload now — tens of thousands of tokens, resent every turn — so the cache decides what you pay. And this month, the cache bug reports started arriving as a genre.
EXHIBIT ONE: THE PROMPT NOBODY WROTE
A poster on r/googlecloud reports that Vertex AI appears to insert and remove a hidden prompt mid-session: the prefix changes, the KV cache zeroes, and every turn bills at full price — 64% of their spend, by their figures, $117.36 of $184.37. One report, no vendor reply, no independent replication found tonight. It stays a claim.
Hold it as a claim — and notice it is the third claim of its shape.
EXHIBIT TWO: THE CACHE THAT ANSWERED YESTERDAY'S QUESTION
On Google's own AI developer forum, engineers reported the Vertex API with an explicit context cache leaking prompt state between calls: ask about a cat image, then a dog image, and the second answer describes the cat. They published a reproducer — it failed every time in one region, sometimes in another — and noted the calls on the wire were correct; the leak was behind the API. Google's reply, after running the code: unable to reproduce.
EXHIBIT THREE: THE AUDIO THAT EVICTED THE LIBRARY
On Google's cloud community forum, a team published controlled benchmarks of the implicit cache on the global endpoint: shift an audio block slightly in a long conversation and the platform retroactively evicted 11,000 to 38,000 tokens of untouched earlier history, dropping cached tokens to zero and thrashing across replicas — while text and image shifts preserved the full prefix. Their logs even surface the platform's standing uncached tail of roughly 2,000 tokens, a number that appears in no pricing page.
THE PATTERN UNDER THE THREE
Different mechanisms, one shape: the cache is stateful, opaque, and billed — and its failures present as invoices, not errors. Nothing 500s. The answers stay plausible. The only symptom is a line item, weeks later, in a currency of cached versus uncached tokens most teams never graph.
Meanwhile the honest documentation problem sits underneath: cache behaviour differs by modality, by region, by endpoint flavour — and the deltas live in forum posts, not in the reference docs your cost model was built from.
RUN YOUR CACHE LIKE A SERVICE
Graph the cached-token ratio like an error rate. A cliff in cache reads is an incident, even when every request returns 200.
Pin your prefixes like dependencies. Any change to the stable prefix — yours or the platform's — is a deploy. Diff it, log it, alert on it.
Design so a cold cache is a cost event, never a correctness event. If a cache reset changes an answer, the bug is in your design, not their cache.
Keep one benchmark conversation. A fixed multi-turn script, run daily against each endpoint you pay for. When the platform's behaviour moves, your graph moves first — not your invoice.
Sources
r/googlecloud poster report (single source, unverified — framed as a claim in the text); Google AI Developers Forum, "Critical bug: Vertex API with context cache leaks prompt state between generateContent calls" (November 2025, with reproducer and vendor reply); Google Cloud Community forum, Gemini implicit-cache eviction benchmarks (November 2026). Forum reports are other teams' measurements, presented as such.
Question for you: when did you last look at your cached-token ratio — and if it halved tonight, which dashboard would tell you before the bill does?
Disclaimer
This post describes my personal experience with my own systems and accounts, in my own words. It is a first-person account of what I observed and what I do in my own work — not a statement of any company’s intentions, policies, or internal decisions. Where I do not know something, the post says so.
I am not affiliated with, endorsed by, or speaking for any company, product, or vendor named in this post. All product and company names are trademarks of their respective owners and are mentioned only to describe the experience accurately.
Any error messages, documentation excerpts, or figures quoted here are reproduced from my own records or from the public sources linked in the post, believed accurate at the time of writing. Product behavior changes over time; verify against current official documentation before relying on any detail.
Nothing in this post is professional, financial, legal, or security advice for your specific situation. Examples are illustrative — validate in your own environment before running anything in production.
Corrections are welcome. If any company named here publishes an official statement, or if any detail is wrong, I will update this post and note the correction. The goal is an accurate public record, not a verdict.
© 2026 Honey Kiran. All rights reserved.
Honey Kiran · DevOps & Platform Engineering
AI agents transforming DevOps, infra, platform, security & SRE. The shift nobody works anymore: deep dives, 3AM Files, and OpsBites.
Examples are illustrative — validate in your own environment before running in production.

Top comments (0)