A high cache-hit rate can make an AI agent look efficient while it still burns time and money elsewhere.
The metric I would rather track is cost per successful task:
model price × tokens × cache economics × effort × tool calls × retries
Then put that number beside latency and success rate.
Cache is useful. It is not the finish line.
Prompt caching is one of the best ways to cut repeated-input cost. Anthropic’s guidance is sensible: keep stable instructions and tool definitions before the volatile part of a request, so the reusable prefix stays byte-identical.
But a good cache metric cannot tell you whether the agent made unnecessary tool calls, followed outdated rituals, or spent too long thinking about a task that already had a clear path.
Prompt cleanup is part of performance work
Many prompts carry historical baggage: forced step-by-step procedures, manual scratchpads, duplicate rules, and repeated verification written to compensate for older models.
With stronger models, those instructions can become friction. They may increase tokens, latency, and tool usage without improving the answer.
The practical move is not “always use low effort” or “always use the strongest model.” It is to measure representative tasks, remove prompt anti-patterns, and choose model × effort × tools for the work in front of you.
What to measure
For each task class, ask:
- Did it finish correctly?
- How long did it take?
- How many tool calls and retries did it need?
- What did it cost end-to-end?
- Did higher effort improve the outcome enough to justify the extra time and spend?
Cache belongs in that picture. It should not replace the picture.
Duy /zuey’s source video, LLM Caching is a Lie!, is a useful walkthrough of this trade-off: https://www.youtube.com/watch?v=-ReK4Ag-6hE
Anthropic’s cost-and-performance guidance: https://claude.com/it/blog/reducing-cost-and-improving-performance-with-claude-platform
I write practical notes from building with AI tools at https://im-khang.com.
Top comments (0)