A lot of AI cost conversations start in the wrong place. They start with the model.
- Which model is cheaper?
- Which one gives better output?
- Which one has lower input cost?
- Which one is faster?
Those questions matter, but they do not explain the full bill when you are building agentic workflows.
The more uncomfortable question is this:
How many times are you making the model read the same thing again?
That is where Gemini context caching becomes interesting.
Google’s Gemini API documentation says that in a typical AI workflow, the same input tokens may be passed to the model over and over. Gemini context caching is meant to optimize performance and costs when that happens. For Gemini 2.5 and newer models, implicit caching is enabled by default, and Google says cost savings are automatically passed on when requests hit caches.
That sounds like an infrastructure detail.
It is actually a product architecture detail.
Because if your AI workflow keeps repeating the same context at every step, the cost problem is not only the model. The cost problem is the workflow design.
The repeated context problem
Most AI agents are not one clean prompt and one clean answer.
They work in steps.
A support agent may read the customer message, the support policy, previous tickets, account details, product documentation, and tool results. Then it may call a tool, receive more information, ask the model again, draft a response, check another rule, and maybe ask the model again.
A CRM agent may read customer notes, sales rules, account history, enrichment data, and next-step instructions. Then it may classify the opportunity, update a field, draft a follow-up, and create a task.
A finance workflow may read policy, vendor data, invoice details, approvals, account rules, and past actions. Then it may extract fields, compare conditions, route the case, and prepare a decision.
The same blocks often appear again and again.
The system instructions appear again.
The policy appears again.
The product documentation appears again.
The customer record appears again.
The tool output appears again.
The cost grows because the model is not only answering. It is repeatedly reading the context needed to answer.
Why caching changes the way you think about AI cost
Caching is not just a discount button.
It rewards workflows where stable context is organized clearly.
- If the same large policy text is used across many steps, it should not be treated like fresh context every time.
- If the same product documentation supports repeated customer questions, it should be structured in a way that can be reused.
- If the same instructions are part of every agent run, the workflow should separate stable instructions from changing user input.
This is where AI cost becomes less about “pick a cheaper model” and more about “stop sending the same information badly.”
A cheaper AI workflow usually comes from several small design choices working together:
- keep stable context stable,
- separate changing input from reusable background,
- avoid sending entire documents when only one section is needed,
- shorten tool outputs before the next model call,
- stop retries from rereading the full context,
- and measure the cost of the full workflow, not only one request.
The mistake: counting only the visible answer
The output is the part people notice.
The user sees the reply, the summary, the recommendation, or the generated report.
But the bill includes the work behind that answer.
If the agent needed five model calls, three tool calls, two retries, and a long context block each time, the cost is not represented by the final answer alone.
This is why AI features can feel cheap in a prototype and expensive in production.
In a prototype, one person tests one workflow a few times. The context is manageable. The usage is low. The hidden repetition does not hurt much.
In production, hundreds or thousands of users create repeated workflows every day. Small inefficiencies become normal behavior. A few extra context-heavy calls per workflow start to matter.
That is usually when teams realize the AI bill is not only a model bill.
It is a workflow bill.
A simple way to review an AI workflow cost
Before optimizing the model, look at the path.
Start with one actual customer workflow and trace what happens from the first request to the final action.
Ask what the model reads at each step. Does it need the full policy again? Does it need the whole customer history again? Does it need every tool result again? Does it need the entire conversation, or only the last few useful parts?
Then separate the context into three buckets.
Stable context
This is information that rarely changes during the workflow.
Examples include system instructions, product rules, policy text, feature documentation, tone guidelines, compliance instructions, and workflow boundaries.
Stable context is where caching can help, especially when many requests reuse the same background.
Session context
This is information that belongs to the current user or current workflow.
Examples include the customer message, selected account, recent ticket history, uploaded file, active case details, or tool output from this run.
This needs more care because some of it may be reused during the session, while some of it may become stale or unnecessary after one step.
Step-specific context
This is information needed only for one model call.
Examples include one extracted field, one decision question, one tool result, one short error message, or one approval state.
This should not turn into a long repeated block across the entire workflow.
Once these buckets are clear, the cost conversation becomes much easier.
What should be cached, shortened, or removed?
Not every context block should be cached.
Not every prompt should be shortened.
Not every output should be compressed.
The decision depends on what the workflow needs.
- A stable policy used across many requests is a good candidate for reuse.
- A long document that only matters for one answer may not need to stay in every step.
- A tool result with one useful field should not be passed forward as a full JSON dump.
- A retry should not automatically resend everything if only the final instruction changed.
- A human review step should not force the next model call to reread irrelevant history.
The goal is not to starve the model of context. It is to stop feeding it context that does not help the next step.
Why this matters for SaaS products
SaaS products are full of repeated context.
- Every customer has account rules.
- Every workspace has permissions.
- Every workflow has fixed instructions.
- Every support process has policy.
- Every CRM system has record history.
- Every AI assistant has behavior guidelines.
If these blocks are sent poorly, the AI feature becomes more expensive than it needs to be. If they are structured well, the product can reuse stable context and keep each model call more focused.
That is why AI cost should be discussed during product design, not after the invoice arrives.
The economic question is not only:
Which model should we use?
It is:
What context does this workflow make the model reread, and how often?
Founder takeaway
Gemini context caching is a useful reminder that AI workflow cost is not only about the visible response.
The repeated context matters.
The retries matter.
The tool results matter.
The long instructions matter.
The number of model calls matters.
If an AI agent needs to complete a multi-step workflow, the team should design the context path as carefully as the tool path.
A good AI product does not only choose a model.
It decides what the model should read, what it should not read again, and what the product can reuse safely.
That is how AI cost becomes easier to understand before the bill becomes uncomfortable.
Sources
Google AI for Developers: Gemini context caching
https://ai.google.dev/gemini-api/docs/cachingGoogle AI for Developers: Gemini API pricing
https://ai.google.dev/gemini-api/docs/pricingAnthropic Docs: Prompt caching
https://platform.claude.com/docs/en/build-with-claude/prompt-cachingClaude Platform release notes
https://platform.claude.com/docs/en/release-notes/overview
Top comments (0)