Why a programmer joke from the 1990s suddenly explains prompt caching, KV cache, LLM inference costs, latency, and even a few scary security problems
By Mahan Tavakoli (MahanKenway)
Tehran, Iran
GitHub: MahanKenway
The joke was never really a joke
There is an old line in computer science:
“There are only two hard things in Computer Science: cache invalidation and naming things.”
The quote is commonly attributed to Phil Karlton, a Netscape engineer. The attribution is unusually interesting because there does not appear to be a contemporaneous written source proving exactly when or where he first said it. Karlton's son, David Karlton, has said his father did use the phrase, while Martin Fowler records that Tim Bray had heard it around 1996–97 and later helped popularize it online.
And then somebody improved the joke.
“There are only two hard problems in computer science: cache invalidation, naming things, and off-by-one errors.”
That third item was later associated with Leon Bambrick, not Karlton. Martin Fowler's history of the quote explicitly separates the original two-item joke from the later off-by-one variation.
The reason the line survived for decades is not because programmers love recycled jokes.
It survived because caching creates a very specific kind of lie.
A cache stores an answer that was correct.
Then the world changes.
The cache does not.
And suddenly your system is confidently returning yesterday's truth.
That sounds like a web-development problem.
It isn't anymore.
Because in modern AI systems, we are caching:
- prompt prefixes
- tokenized context
- model attention state
- key/value tensors
- long conversation histories
- documents
- retrieval results
- embeddings
- tool definitions
- generated artifacts
- sometimes entire computational subgraphs
In other words:
AI did not solve cache invalidation.
It made the consequences much larger.
A cache is basically a machine for reusing the past
At the most basic level, caching is simple.
Suppose calculating something expensive costs 100 milliseconds.
You calculate it once.
You save the result.
Next time, instead of spending 100 milliseconds calculating it again, you return the saved value.
That is the entire idea.
WITHOUT CACHE
Request
|
v
Compute expensive thing
|
v
Result
WITH CACHE
Request
|
v
Is result already cached?
| \
YES NO
| |
v v
Return result Compute it
|
v
Store result
|
v
Return result
The performance benefit can be enormous.
That is why caches exist everywhere.
CPU caches.
Disk caches.
Browser caches.
CDNs.
Database caches.
Redis.
Memoization.
Operating-system page caches.
Build caches.
Package-manager caches.
Compiler caches.
And now:
LLM caches.
The problem is hidden inside one innocent question:
When is the cached result no longer valid?
That question is where the comedy ends.
The real problem is not storing a value
Imagine a website shows this:
User: Mahan
Plan: Pro
Tokens: 14,281
The application reads that data from a database.
To make the website faster, it caches the result.
Five minutes later:
Mahan upgrades the account.
The database says:
Plan: Enterprise
The cache says:
Plan: Pro
Now you have two realities.
The database represents the current state.
The cache represents an older state.
Neither machine is malfunctioning.
The CPU is fine.
The network is fine.
The database is fine.
The cache is doing exactly what you instructed it to do.
And that is the terrifying part.
The bug is the architecture.
Redis documentation describes the classic consistency problem the same way: when the source of truth changes but the cached copy does not, the cache becomes inconsistent. Different caching strategies exist precisely because there is no single universal solution.
This is why cache invalidation became famous.
Caching asks:
“Can I reuse this old answer?”
Invalidation asks:
“Can I prove that this old answer is still safe to reuse?”
Those are very different questions.
The first is easy.
The second is where systems engineering starts getting weird.
Why invalidation is fundamentally annoying
Suppose this function exists:
def get_user():
return database.query("SELECT * FROM users")
You cache it:
@cache
def get_user():
return database.query("SELECT * FROM users")
Great.
Now another function updates the user:
def update_user(name):
database.execute(...)
The database changed.
But your cached get_user() result did not.
So the system needs some relationship like:
update_user()
|
+---- invalidates ----> get_user() cache
But real programs do not contain one neat dependency.
They contain graphs.
Database
|
+--> User API
| |
| +--> Dashboard
|
+--> Search index
|
+--> Recommendation engine
|
+--> Analytics
|
+--> Mobile API
|
+--> Background jobs
Now change one piece of data.
How many caches are potentially affected?
That is the actual problem.
Invalidation is dependency tracking disguised as housekeeping.
And the larger your system gets, the less obvious those dependencies become.
Enter the LLM
Now replace our user database with a language model.
Suppose an application repeatedly sends:
You are a senior software engineer.
Project:
[large repository]
Architecture:
[long documentation]
Tool definitions:
[dozens of tools]
Rules:
[system instructions]
Conversation history:
[thousands of tokens]
Now answer this question:
...
Without caching, the system may need to repeatedly process the same large prefix.
With caching, the system can reuse work that has already been done.
This is exactly the kind of workload modern AI providers are optimizing for.
OpenAI introduced automatic prompt caching in October 2024, explaining that repeated input tokens could be reused to reduce latency and lower input-token cost. The initial announcement described a 50% discount for cached input tokens on supported models.
Anthropic similarly describes prompt caching as a mechanism for reusing frequently repeated context, with potential reductions of more than 2× in latency and up to 90% in costs for repetitive long-context workloads.
Google's Gemini API now also supports context caching, including automatic caching on newer Gemini models. Google's current documentation explicitly tells developers to put large, common content near the beginning of prompts and send similar prefixes close together in time to increase cache-hit probability.
Notice what just happened.
A computer-science joke that existed before modern LLMs became mainstream has quietly become part of the economics of AI inference.
Prompt caching is not magic
Let's simplify an LLM request.
You have:
SYSTEM
+ giant documentation
+ tool definitions
+ conversation history
+ user question
The model processes the context.
The beginning of the request is often called the prefill stage.
Then the model generates tokens one by one during decode.
A simplified mental model:
INPUT TOKENS
|
v
PREFILL
|
v
KV CACHE
|
v
DECODE
|
+--> token
+--> token
+--> token
+--> token
The KV cache stores intermediate attention information so that the model does not need to recompute the entire history from scratch for every generated token.
This is one of the foundational optimizations behind efficient transformer inference.
And now our old friend appears again.
We have data that is expensive to compute.
We store it.
We reuse it.
Now we need to decide when that stored data can safely be reused.
That is caching.
And therefore:
cache invalidation.
KV cache is where the joke gets much more interesting
The phrase KV cache sounds almost harmless.
It is not.
During autoregressive inference, the model generates output token by token while maintaining key/value tensors representing prior context.
These tensors consume GPU memory.
And they are dynamic.
And they grow.
And they have to be scheduled across concurrent requests.
And they compete for extremely valuable memory bandwidth.
The original PagedAttention paper behind vLLM described the KV cache as a major memory-management challenge. The authors specifically identified fragmentation and redundant duplication as major reasons existing systems wasted GPU memory, then introduced PagedAttention to manage KV cache blocks similarly to pages in virtual memory. Their evaluation reported 2–4× higher throughput than the compared serving systems at similar latency.
The conceptual connection is beautiful.
Operating systems solved memory management using:
logical memory
|
v
pages
|
v
physical memory
LLM inference engines can use:
logical sequence
|
v
KV blocks
|
v
GPU memory
This is not just a metaphor.
vLLM's documentation describes automatic prefix caching as caching KV-cache blocks and reusing them when a new request shares the same prefix. Its current implementation uses hashed blocks and a cache-management layer to find reusable prefixes.
That means modern AI serving has independently rediscovered a lot of old systems ideas.
Virtual memory.
Paging.
Reference counting.
Hashing.
LRU-style eviction.
Memory pooling.
Cache sharing.
Dependency tracking.
We did not escape old computer science.
We brought it with us.
The first trap: cache hits are not guaranteed
This is where AI marketing language can become misleading.
People hear:
“Prompt caching reduces cost by 90%.”
And mentally convert that into:
“My AI application will now cost 90% less.”
Not necessarily.
A cache only helps when the workload produces useful cache hits.
Consider:
Request 1:
[system][docs][tools][question A]
Request 2:
[system][docs][tools][question B]
Excellent.
Large common prefix.
Potentially very cacheable.
Now:
Request 1:
[system][user context A][question A]
Request 2:
[system][user context B][question B]
Much less reusable.
The prefixes diverge.
The economics change.
The system might still cache the beginning, but the useful reusable portion depends heavily on request structure.
Google's documentation makes this operational reality explicit: developers can improve implicit-cache hit rates by placing common content early and keeping similar prefixes close together in time.
So the performance problem is no longer merely:
“Do we have a cache?”
It becomes:
“Did we architect the workload so that the cache can actually help?”
That is a much harder question.
The second trap: invalidation is different for every layer
Here is where modern AI systems become beautifully messy.
Imagine an AI coding agent.
It has:
System prompt
+
developer rules
+
tool definitions
+
repository files
+
retrieved documentation
+
conversation history
+
current task
Now change one thing.
A single file in the repository is modified.
What becomes invalid?
Maybe the repository context.
Maybe one retrieval result.
Maybe the tool output.
Maybe a cached prefix.
Maybe a generated summary.
Maybe an embedding.
Maybe a derived index.
Maybe nothing.
Or maybe parts of all of them.
That is the nightmare.
The cache does not understand your business semantics.
It understands keys.
So engineers have to create a mapping:
meaning
|
v
cache key
|
v
cached artifact
If the key is wrong, the cache can remain perfectly efficient while being completely incorrect.
A cache key is a tiny piece of architecture
Suppose you cache:
cache["user:123"] = ...
Straightforward.
Now imagine caching a model context:
model
system prompt
tool definitions
conversation
documents
permissions
tenant
locale
version
configuration
Suddenly the key could effectively need to represent:
K =
(model version)
+ (system instructions)
+ (tool configuration)
+ (document version)
+ (conversation prefix)
+ (tenant)
+ (permission state)
+ (runtime configuration)
Forget one important input.
Congratulations.
You just created a stale-cache bug.
And now the bug may not crash the service.
It may simply produce a wrong answer.
That is much worse.
This is where AI turns caching into a semantic problem
Traditional caching usually deals with relatively clear objects.
For example:
URL -> HTTP response
Or:
database query -> rows
LLM systems are different.
The result can depend on a huge amount of context.
For example:
Prompt
+
Model version
+
System instruction
+
Tools
+
Retrieved data
+
Conversation state
+
Sampling/configuration
Now ask:
What exactly determines whether the previous result is still valid?
There is no universal answer.
You have to define one.
That definition is your cache's semantic contract.
And this is why cache invalidation is often less about deletion and more about understanding dependencies.
There are two different “caches” people casually call caching
This distinction matters a lot.
1. Prompt / context caching
The system reuses processing associated with repeated input context.
Conceptually:
same context
|
v
reuse previous work
This is what public APIs increasingly expose as prompt caching or context caching.
OpenAI, Anthropic, and Google all now provide caching mechanisms intended to reduce repeated-input processing cost and/or latency.
2. KV caching during generation
This is the model-runtime side.
Instead of recomputing attention state for previous tokens every time, the serving system retains the relevant K/V tensors.
vLLM, for example, treats KV data as blocks that can be allocated, shared, hashed, evicted, and reused.
They are related.
But they are not identical.
And lumping them together makes discussions about “AI caching” confusing.
The weird part: caching can become a security problem
This is where the story gets properly strange.
We usually think:
cache = performance optimization
But shared caches can create side channels.
Suppose two customers use the same inference infrastructure.
Customer A's private prompt causes certain prefix blocks to be cached.
Customer B sends a request with a guessed prefix.
If B's request gets a noticeably different latency depending on whether the prefix is already cached, the cache itself may leak information.
vLLM's security documentation now explicitly discusses this threat and references CVE-2025-46570, describing how differences in Time To First Token can reveal whether a guessed prompt prefix matches cached data. The documentation describes cache salting as a mitigation so different security principals do not blindly share the same prefix-cache namespace.
That is a remarkable evolution:
1990s:
"Cache invalidation is hard."
2020s:
"Cache invalidation affects cost."
2026:
"Cache design can become a confidentiality boundary."
The cache went from:
optimization
to:
systems problem
to:
security surface
without changing its basic purpose.
The third trap: bigger caches are not automatically better
There is a temptation in infrastructure engineering to think:
More cache = more performance.
Not necessarily.
A cache consumes resources.
Memory.
Metadata.
Bandwidth.
Eviction logic.
Synchronization.
Invalidation traffic.
Management overhead.
And in LLM inference, the KV cache can become large enough that memory capacity and bandwidth directly influence performance.
Recent research continues to describe LLM inference as strongly constrained by memory movement. One 2026 AAAI paper, for example, focuses specifically on asynchronous KV-cache prefetching to reduce HBM-memory bottlenecks.
Another recent line of work studies the KV cache as an increasingly important bottleneck as context lengths expand, because the cache footprint grows with sequence length and puts pressure on both memory capacity and bandwidth.
This creates an interesting inversion.
Caching exists to save computation.
But caching itself costs memory and data movement.
So eventually you ask:
Am I saving compute by spending bandwidth?
And sometimes:
Did the cache just become the bottleneck I created while trying to remove the original bottleneck?
That is peak computer science.
Prefix caching is basically memoization for giant models
One of the most intuitive ways to understand modern LLM prefix caching is to think about a function:
f(prefix)
If you already calculated:
f("A very long shared prefix...")
why calculate the same thing again?
You can memoize it.
Modern inference engines can do something structurally similar with prefix blocks.
vLLM's automatic prefix caching documentation gives examples such as repeatedly querying the same long document or continuing a multi-turn conversation. In those workloads, the system can reuse the previously processed prefix instead of recomputing that section for every request.
It is a very old trick wearing very new clothes.
The scale changed.
The underlying idea did not.
And then we discover the off-by-one joke was accidentally relevant
Remember the expanded version?
cache invalidation
naming things
off-by-one errors
The off-by-one part looks like a joke stapled onto another joke.
But in systems that divide sequences into cache blocks, boundaries matter.
For example:
Tokens:
[0 1 2 3][4 5 6 7][8 9 10 11]
If your system's indexing, block accounting, token offsets, or ownership logic is wrong by one unit, you may get:
wrong block
wrong lookup
wrong reuse
wrong eviction
wrong attention state
Modern cache managers therefore have to care about details that feel hilariously low-level compared with the headline:
“We built a frontier AI system.”
Underneath that headline you might still find:
hash table
LRU queue
block index
reference count
memory allocator
offset
eviction policy
Humanity invented trillion-parameter models and then immediately had to debug a linked list.
I love this industry.
Why this matters for the cost of inference
This is the part I think gets missed most often.
People talk about AI inference economics as if the only important variable is:
FLOPs
But real inference is constrained by a much richer system:
Model size
+
GPU memory
+
HBM bandwidth
+
interconnect bandwidth
+
batching
+
sequence length
+
KV cache memory
+
scheduler efficiency
+
cache hit rate
+
request shape
+
latency requirements
Caching can reduce repeated work dramatically.
But it cannot magically eliminate the underlying data.
It can also fail to help if requests do not share prefixes.
And it can introduce memory pressure.
And it can introduce security boundaries.
And it creates lifecycle questions.
So saying:
“Cache invalidation is preventing AI inference from getting cheap”
would be too simplistic.
The more accurate statement is:
Caching is one of the mechanisms making inference cheaper, but cache efficiency itself becomes a first-class systems constraint as AI workloads become more repetitive, longer-context, and more multi-tenant.
That distinction matters.
The problem is not that caching is failing.
The problem is that the cost structure of AI increasingly depends on getting caching right.
The hidden economics of a cache miss
Imagine an AI application processing 1 million requests.
Every request has:
8,000 common input tokens
+
500 unique input tokens
Now imagine the common 8,000-token prefix can be reused.
Without useful caching:
8,500 tokens
x
1,000,000 requests
With a strong prefix cache:
8,000 tokens processed once
+
500 unique tokens
x
1,000,000
The exact economics depend on the model, API pricing, cache policy, workload shape, and serving architecture.
But the conceptual difference is enormous.
You are no longer paying to repeatedly understand the same thing.
You pay once, then reuse.
That is why companies expose explicit or automatic caching mechanisms.
OpenAI advertises reduced input-token prices for cached prompts. Anthropic advertises savings of up to 90% on suitable workloads. Google similarly documents cost savings for cache hits and offers both implicit and explicit context-caching mechanisms.
So the old joke has quietly moved into the P&L.
A cache miss can now have a direct inference-cost attached to it.
But here's the paradox
Caching is supposed to make systems cheaper.
Yet modern AI systems sometimes need increasingly sophisticated infrastructure to make the cache useful.
You might need:
hashing
+
block management
+
memory pools
+
eviction policies
+
prefix matching
+
scheduler integration
+
tenant isolation
+
cache salting
+
metrics
+
observability
+
versioning
So the system saves compute by adding complexity.
That trade is often worth it.
But it leads to a broader rule:
The more valuable your cache becomes, the more dangerous it becomes to get the cache wrong.
A cache nobody cares about can be dumb.
A cache that saves millions of dollars becomes infrastructure.
And infrastructure needs invariants.
The old distributed-systems nightmare comes back
Traditional distributed systems have always wrestled with consistency.
You have:
Source of truth
|
+--> Cache A
|
+--> Cache B
|
+--> Replica
|
+--> Search index
Now AI applications often add:
Source
|
+--> Retrieval index
|
+--> Embedding store
|
+--> Prompt cache
|
+--> KV cache
|
+--> Tool state
|
+--> Agent memory
A modern agent may effectively operate over several layers of remembered state.
That is an important conceptual shift.
The AI system is no longer just:
prompt -> model -> answer
It starts looking more like:
+-------------------+
| Source Data |
+---------+---------+
|
+------------+------------+
| | |
v v v
Retrieval Context Agent
Cache Cache Memory
| | |
+------------+------------+
|
v
Model
|
v
KV Cache
|
v
Output
Now ask:
What happens when the source data changes?
That question is no longer academic.
A useful way to think about invalidation
Instead of thinking:
“Delete the cache.”
Think:
“Which facts changed, and which derived artifacts depend on those facts?”
For example:
Document v17
|
+--> embedding v17
|
+--> retrieval index v17
|
+--> prompt context v17
|
+--> generated summary v17
When the document becomes v18, you do not really have a “cache deletion” problem.
You have a dependency graph problem.
document v17
X
|
+---- invalid
|
+--> embedding v17
+--> retrieval result
+--> summary
Then:
document v18
|
+--> new embedding
+--> new retrieval result
+--> new summary
That is why sophisticated systems increasingly use versioning, content hashes, immutable artifacts, timestamps, or explicit cache namespaces instead of trying to manually chase every stale object.
Sometimes the best invalidation strategy is not invalidation
This is one of the strangest and most useful lessons from systems engineering.
You can try to determine exactly when a cache entry is stale.
Or you can design the system so that staleness is naturally bounded.
A common strategy is TTL:
cache entry
|
+--> expires after 5 minutes
Another is versioned keys:
document:123:v17
document:123:v18
Another is content addressing:
hash(contents) -> artifact
Another is immutable data.
Another is write-through or write-behind.
Another is simply refusing to cache data that is too difficult to invalidate safely.
Caching is not a religion.
Sometimes:
Do not cache this
is the best design decision.
vLLM shows what happens when old systems ideas meet LLM workloads
The engineering behind modern inference frameworks is a perfect example of this entire article.
vLLM's PagedAttention treats KV cache more like memory pages than like one giant contiguous array. Its later prefix-caching system extends that idea by hashing KV-cache blocks so identical prefixes can be recognized and reused.
That is a beautiful evolution:
Virtual memory
↓
PagedAttention
↓
KV block management
↓
Prefix caching
↓
Cross-request reuse
The AI revolution keeps producing systems that look suspiciously like classic operating-system research.
Because at some point, a GPU is still a machine with:
finite memory
finite bandwidth
finite time
finite buses
finite queues
No amount of AI hype changes that.
The uncomfortable implication
There is a tendency in AI engineering to think of optimization as a race:
better model
better GPU
more FLOPs
longer context
But there is another race happening underneath:
better memory management
better scheduling
better batching
better cache locality
better reuse
better data movement
And some of the biggest wins are not coming from making the model mathematically smarter.
They come from not doing the same work twice.
That sounds boring.
It is not boring.
At scale, “don't calculate the same thing twice” is an economic strategy.
A cache is basically a memory of what the system believed yesterday
This is the philosophical part.
A cache is not truth.
A cache is a copy of truth.
And copies create divergence.
The interesting thing about AI is that modern models already work with representations instead of the raw world.
Then we add:
retrieved data
+
summaries
+
embeddings
+
prompt caches
+
KV caches
+
agent memory
Now we are building layers of representations on top of representations.
Every layer can become stale.
Every layer needs provenance.
Every layer needs invalidation rules.
The further away you get from the source of truth, the harder it becomes to know whether the thing in your hand is still trustworthy.
That is exactly why cache invalidation has stayed difficult for so long.
And maybe the joke has a modern version
The old version was:
cache invalidation and naming things.
The 2026 version might be:
cache invalidation, naming things, and convincing a 400-line AI agent that the thing it cached yesterday is no longer true.
The joke is funny because it is only slightly exaggerated.
What developers building AI systems should actually care about
Not every application needs an exotic cache architecture.
But if you are building a serious LLM product, it is worth explicitly documenting:
What is being cached?
Be precise.
tokens?
embeddings?
KV blocks?
retrieval results?
model responses?
tool results?
documents?
What makes two requests equivalent?
For example:
same model
+
same prefix
+
same tool configuration
+
same tenant
What invalidates the cached object?
content change
model change
tool change
permission change
tenant change
time
What happens when the cache is wrong?
This is the question many systems forget.
Some stale data is annoying.
Some stale data is catastrophic.
Can one tenant observe another tenant's cache?
In multi-tenant inference, that is now a security question, not just a performance question. vLLM's documentation explicitly addresses cross-request prefix-cache side channels and cache salting for isolation.
What is the economics of a hit vs a miss?
Measure it.
A cache that sounds impressive but produces a 12% hit rate may not matter much.
A cache that consistently eliminates enormous repeated prefixes may be one of the most important pieces of infrastructure in the application.
The metrics I would actually watch
For an LLM caching system, “cache enabled: true” is nearly useless.
I would want to know:
Cache hit rate
Cache miss rate
Reusable token count
Cached token count
Bytes stored
Bytes evicted
Eviction rate
TTFT cold vs warm
Prompt processing time
Decode time
GPU memory usage
HBM traffic
Cost per request
Cost per cached request
Cost per uncached request
And ideally:
hit rate by tenant
hit rate by workload
hit rate by model
hit rate by prefix length
hit rate by request type
Because aggregate statistics can lie.
You might have:
90% average hit rate
and still have:
0% hit rate
for your most expensive requests.
Numbers need context.
Why this matters even more as context windows grow
Longer contexts sound like a pure model-capability improvement.
But long context also means:
more tokens
more memory
more memory movement
larger KV cache
more data to manage
Recent research continues to focus on KV cache compression, offloading, heterogeneous memory, and prefetching precisely because the cache becomes increasingly significant as context lengths increase. For example, RocketKV studies substantial KV-cache compression for long-context inference, while other recent work examines dynamic placement and memory-bandwidth bottlenecks.
So there is a weird feedback loop:
Longer context
↓
More useful information
↓
More reusable information
↓
More reason to cache
↓
Larger cache
↓
More memory pressure
↓
More cache engineering
AI did not eliminate the old problem.
It amplified it.
The biggest misunderstanding about “cheap inference”
People sometimes imagine that AI inference becomes cheaper mainly because GPUs become faster.
Yes, hardware improvements matter enormously.
But software can also make a huge difference by increasing reuse.
The fastest operation is often:
the operation you do not perform.
That principle is ancient.
Compiler optimization uses it.
CPU caches use it.
Databases use it.
Operating systems use it.
Web browsers use it.
CDNs use it.
LLM inference uses it.
And every time you choose reuse, you inherit the problem of proving that the reused thing is still valid.
That is the price of memory.
So... did AI finally solve cache invalidation?
No.
And that is exactly why this is interesting.
Modern AI infrastructure has become sophisticated enough to expose many of the same old problems at a scale where they directly affect:
- inference latency
- GPU utilization
- memory capacity
- bandwidth
- API pricing
- cloud bills
- multi-tenant isolation
- system correctness
Prompt caching has become a product feature.
KV caching has become a fundamental inference mechanism.
Prefix caching has become an optimization strategy.
Cache isolation has become a security concern.
And cache hit rate can become an economic variable.
The joke survived because the underlying problem survived.
The deeper lesson
The AI industry loves to describe itself using words like:
scaling
intelligence
reasoning
agents
context
autonomy
But underneath all of that, the computers are still asking extremely old questions:
Where is the data?
How much memory does it take?
Can I reuse it?
Is it still valid?
Who owns it?
When can I delete it?
Who else can observe it?
What happens if I am wrong?
Those questions are not fashionable.
They are foundational.
And that might be the most interesting thing about modern AI engineering.
We keep inventing systems that look revolutionary from the outside.
Then we open the hood.
And there is still a cache.
And it is still lying to us.
TL;DR
Cache invalidation did not become less important because of AI. It became more important.
Modern LLM systems increasingly depend on several forms of caching, including prompt/context caching, KV caching, and prefix caching. OpenAI, Anthropic, and Google now expose caching mechanisms aimed at reducing repeated-input cost and latency, while inference systems such as vLLM use sophisticated KV-cache management and prefix reuse.
The challenge is that cached AI state can depend on huge amounts of context.
That turns cache invalidation into a problem of:
dependency tracking + consistency + memory management + economics + security.
And the really weird part?
The same old systems ideas that engineers were fighting decades ago are now sitting underneath trillion-token, GPU-heavy AI infrastructure.
So yes:
there are only two hard things in computer science.
Apparently we just decided to make the cache larger.
Frequently Asked Questions
What is cache invalidation?
Cache invalidation is the process of determining when cached data is no longer valid and must be removed, refreshed, or replaced. The difficulty comes from keeping cached copies consistent with the underlying source of truth.
Who said “cache invalidation and naming things”?
The statement is commonly attributed to Phil Karlton. The historical record is based largely on recollections and later documentation rather than a definitive contemporaneous publication. Tim Bray reported hearing the quote in the 1990s, and Martin Fowler documented the phrase and its later variations.
Did Phil Karlton invent the “off-by-one” version?
Not according to the commonly cited history. Martin Fowler attributes the later three-part variation to Leon Bambrick, while preserving Phil Karlton as the source of the original two-part version.
What is prompt caching?
Prompt caching allows an AI system to reuse processing associated with repeated input context, reducing repeated computation and potentially lowering latency and input-token costs. OpenAI, Anthropic, and Google all provide versions of this capability.
What is a KV cache?
A KV cache stores attention key/value information from previous tokens so an autoregressive model can avoid recomputing the entire prior context during generation. Managing this data efficiently is a major part of high-throughput LLM serving.
Does caching always make LLM inference cheaper?
No. Caching only produces major benefits when requests share reusable context or state. Cache effectiveness depends on workload structure, cache-hit rates, memory capacity, eviction behavior, and serving architecture. Google's documentation explicitly notes that prompt structure affects implicit-cache hit rates, while vLLM notes that prefix caching mainly reduces prefilling work rather than decode time.
Can LLM caches create security vulnerabilities?
Yes. Shared prefix caches can create timing side channels in multi-tenant inference, allowing an attacker to infer information about another request's cached prefix. vLLM's security documentation describes this problem and cache-salting mitigations.
Keywords / Topics
cache invalidation, Phil Karlton, cache invalidation problem, prompt caching, LLM caching, KV cache, KV cache optimization, prefix caching, LLM inference cost, AI inference optimization, vLLM, PagedAttention, prompt cache, context cache, cache consistency, distributed systems, GPU memory, HBM bandwidth, inference latency, AI infrastructure, LLM serving, cache security
Suggested DEV / CoderLegion SEO metadata
SEO Title:
Cache Invalidation Never Died. AI Just Made It Expensive Again.
Meta Description:
The famous cache-invalidation joke is suddenly relevant to AI. Explore prompt caching, KV cache, prefix caching, LLM inference costs, memory bottlenecks, and cache security.
Suggested Slug:
cache-invalidation-llm-prompt-kv-prefix-caching
Suggested Tags:
AI, LLM, Programming, Performance, System Design
Primary Search Intent:
Why is cache invalidation hard, and how does it affect modern AI inference?
Semantic topics for search / GEO:
cache invalidation explained, prompt caching explained, KV cache explained, prefix caching, LLM inference optimization, vLLM cache, AI inference cost, cache consistency, multi-tenant LLM security
Sources
Martin Fowler — “Two Hard Things”
Historical discussion of the Phil Karlton quote and its later variants.Tim Bray — “On XML Language Design” / archival references
Early public attribution of the quote to Phil Karlton and evidence of its circulation in the 1990s.David Karlton / Skeptics Stack Exchange discussion
First-hand family recollection regarding Phil Karlton's use and likely origin of the quote.OpenAI — “Prompt Caching in the API”
Official description of automatic prompt caching and cached-input pricing.Anthropic — “Contextual Retrieval in AI Systems”
Discussion of prompt caching and its potential latency and cost reductions.Anthropic Claude Cookbook — “Prompt caching through the Claude API”
Technical description of automatic and explicit caching approaches.Google AI for Developers — “Context caching”
Current Gemini documentation covering implicit and explicit caching, cache-hit behavior, and prompt-structure guidance.Kwon et al. — “Efficient Memory Management for Large Language Model Serving with PagedAttention”
Research introducing PagedAttention and the vLLM serving system.vLLM documentation — “Automatic Prefix Caching”
Technical explanation of KV-cache block reuse and hash-based prefix caching.vLLM security documentation — Prefix Cache Timing Side-Channel Mitigation
Discussion of multi-tenant prefix-cache leakage and cache salting.Dong et al. — “Accelerating LLM Inference Throughput via Asynchronous KV Cache Prefetching”
Research on memory-bandwidth bottlenecks during LLM inference.Xu, Khaira & Singh — “KV Cache Optimization Strategies for Scalable and Efficient LLM Inference”
Recent review of KV-cache capacity and bandwidth challenges in long-context inference.Behnam et al. — “RocketKV”
Research on KV-cache compression for long-context LLM inference.
Top comments (0)