DEV Community

Honey Kiran
Honey Kiran

Posted on Originally published at honeykiran.substack.com on

Check Your Vertex Bill. A Prompt You Never Wrote Is Eating Your Cache.

A poster on r/googlecloud this week did the maths on their Vertex AI bill and found something they never wrote.

Their claim: Google dynamically inserts and removes a hidden prompt mid-session — sitting at the bottom of the system prompt, invisible in the API response. Every flip changes the text above the whole conversation, so the prompt cache stops matching. Cache hits drop to zero. And every miss bills at full price.

Their figures, from their own spend:

  • 64% of their Vertex spend — $117.36 of $184.37 — went to calls where an active cache had dropped to zero mid-session

  • Cached tokens on the affected turns fell from roughly 24,500 to 0 , then recovered

  • It fired with nobody typing : their own heartbeat-driven agent tripped the hidden prompt with its own output

  • A second hidden block reportedly switched on for neutral questions too

Google has not confirmed this account — it is one poster's measurements, shared with a per-model table. What is not in dispute is the shape of the risk, and that part needs no one's permission to check.

The check costs you nothing

Prompt caching is a discount for sending the same prefix twice. Anything that changes the prefix — a vendor insert, your own harness, a tool that rewrites the system prompt — quietly returns you to full price while the dashboard still says "cache enabled."

  • Watch the cached-token count per call , not the feature flag. A drop to zero mid-session is the tell.

  • Log your own prefix. If the text above your first message changes and you didn't change it, ask who did.

  • Price a session both ways before a big run: with cache, without. The gap is your exposure.

My agents work the 3AM shift so no human has to. Somebody still has to read the bill they run up — especially the line items nobody wrote.

When did you last check whether your cache is actually caching?


Disclaimer

  • This post describes my personal experience with my own systems and accounts, in my own words. It is a first-person account of what I observed and what I do in my own work — not a statement of any company’s intentions, policies, or internal decisions. Where I do not know something, the post says so.

  • I am not affiliated with, endorsed by, or speaking for any company, product, or vendor named in this post. All product and company names are trademarks of their respective owners and are mentioned only to describe the experience accurately.

  • Any error messages, documentation excerpts, or figures quoted here are reproduced from my own records or from the public sources linked in the post, believed accurate at the time of writing. Product behavior changes over time; verify against current official documentation before relying on any detail.

  • Nothing in this post is professional, financial, legal, or security advice for your specific situation. Examples are illustrative — validate in your own environment before running anything in production.

  • Corrections are welcome. If any company named here publishes an official statement, or if any detail is wrong, I will update this post and note the correction. The goal is an accurate public record, not a verdict.


© 2026 Honey Kiran. All rights reserved.

Honey Kiran · DevOps & Platform Engineering

AI agents transforming DevOps, infra, platform, security & SRE. The shift nobody works anymore: deep dives, 3AM Files, and OpsBites.

Examples are illustrative — validate in your own environment before running in production.

Top comments (0)