Operating systems solved a problem decades ago:
How do you make a limited amount of fast memory appear much larger to applications?
LLM inference is now facing a surprisingly similar problem.
Except instead of CPU processes and memory pages, we have inference requests and KV-cache blocks.
Instead of:
CPU → RAM → Disk
we increasingly have:
GPU HBM → CPU DRAM → NVMe → Remote Memory
And instead of simply executing instructions, the system must continuously decide:
- What state should stay in GPU memory?
- What can be shared?
- What should be evicted?
- What should be prefetched?
- Which request should run next?
- Where should its state live?
Here is how LLM inference is turning into an operating-system problem.
1. The KV Cache Turns Inference Into a Memory-Management Problem
To understand everything else, we first need to understand the KV cache.
Transformers use self-attention.
For every token, attention produces:
- a Query (Q)
- a Key (K)
- a Value (V)
During autoregressive generation, previously generated tokens don't need to have their Keys and Values recomputed every time.
So inference engines store them.
That's the KV cache.
Conceptually:
Token 1 → K1, V1
Token 2 → K2, V2
Token 3 → K3, V3
Token 4 → K4, V4
...
Token N → KN, VN
When generating token N+1, the model can reuse:
K1...KN
V1...VN
instead of recomputing them.
This dramatically reduces computation.
But it creates a new problem:
The longer the context, the more memory the KV cache consumes.
A simplified KV-cache memory calculation is:
Mₖᵥ = 2 × L × T × Hₖᵥ × D × B
where:
- (L) = number of transformer layers
- (T) = number of tokens
- (H_{KV}) = number of KV heads
- (D) = head dimension
- (B) = bytes per element
- 2 = Key + Value
Notice the important variable:
Double the context length and you roughly double the KV memory.
Now imagine thousands of simultaneous requests.
Request A → 10K tokens
Request B → 50K tokens
Request C → 2K tokens
Request D → 100K tokens
...
The model weights may be static.
The KV cache is dynamic memory.
And that is where the operating-system analogy begins.
2. PagedAttention Turns KV Cache Into Pages
A naive inference engine might allocate one large contiguous region of memory for every request.
For example:
GPU Memory
┌─────────┬─────────┬─────────┬─────────┐
│ Request │ Request │ Request │ Request │
│ A │ B │ C │ D │
└─────────┴─────────┴─────────┴─────────┘
But requests don't behave nicely.
Request A may finish.
Request C may grow.
Request D may be paused.
A new request may arrive.
Eventually memory becomes fragmented.
You can have plenty of free memory but not enough contiguous memory for a large allocation.
Operating systems solved a similar problem with paging.
Instead of requiring a process to occupy one contiguous region:
Process A:
[A][A][A][A][A]
the OS divides it into pages:
[A0][A1][A2][A3][A4]
and stores those pages wherever free physical frames are available:
Physical memory:
[A0][C2][B1][A3][D4][A1][FREE][A2]
A page table maps:
Virtual page → Physical frame
Modern LLM serving systems use a similar idea for KV cache.
The KV cache is divided into blocks:
Request A:
[A0][A1][A2][A3][A4]
Those blocks can be placed independently in GPU memory.
Conceptually:
A0 → GPU block 17
A1 → GPU block 42
A2 → GPU block 5
A3 → GPU block 91
The original vLLM/PagedAttention work explicitly uses the virtual-memory analogy to manage KV cache efficiently.
The important idea is:
Logical token state doesn't need to correspond to one contiguous physical memory region.
That single change makes KV memory dramatically easier to manage.
3. KV Cache Sharing Turns Into Shared Memory
Now consider multiple requests with the same prefix.
For example, thousands of users might send requests beginning with:
You are an expert financial analyst.
Here is the company's annual report:
...
The beginning of every request is identical.
Without sharing:
Request A:
[P][P][P][A]
Request B:
[P][P][P][B]
Request C:
[P][P][P][C]
The same prefix is stored three times.
That's wasteful.
Instead, we can share the prefix:
┌── Request A
│
[P][P][P] ────┼── Request B
│
└── Request C
One physical set of KV blocks.
Multiple logical requests.
This is conceptually similar to shared memory.
It also resembles copy-on-write:
Shared prefix
│
┌────┼────┐
A B C
As long as requests only read the shared blocks, they can reuse them.
If they diverge, new blocks can be allocated.
Systems such as SGLang's RadixAttention exploit prefix reuse to avoid recomputing common KV state.
Now the inference runtime isn't simply asking:
"How much memory does this request need?"
It can also ask:
"How much of this request can I reuse from memory I already have?"
That is a much more sophisticated memory-management problem.
4. KV Eviction Turns Into a Cache-Management Problem
GPU HBM is limited.
Eventually:
HBM
████████████████████
100% FULL
Something must leave.
This introduces KV-cache eviction.
Suppose we have:
[A][B][C][D][E][F][G][H]
and a new request needs another block.
Which one should we remove?
Possible policies include:
LRU
LFU
FIFO
TTL
Priority
Reuse probability
Cost-aware eviction
The simplest approach is something like LRU:
Evict the least recently used block.
But LLM workloads are more complicated.
Imagine:
Request A → active
Request B → paused
Request C → active
Request D → paused
Request B might not have been used recently.
But perhaps B is an agent waiting for a database query.
In two seconds it may resume and need its entire KV history.
So blindly evicting B could be expensive.
There are now multiple possible costs:
KEEP
↓
consume valuable HBM
EVICT
↓
reload later
RECOMPUTE
↓
use GPU compute
TRANSFER
↓
move from another memory tier
The runtime is therefore solving something like:
Bottleneck = min(Cₘₑₘₒᵣᵧ, Cᵣₑₗₒₐd, Cᵣₑcₒₘₚᵤₜₑ, Cₜᵣₐₙₛfₑᵣ)
This is no longer simply "run the neural network."
It is cache management.
5. Hierarchical KV Cache Creates a Memory Hierarchy
Once HBM becomes insufficient, we need somewhere else to put KV state.
Modern systems can increasingly use multiple memory tiers.
Conceptually:
GPU
│
┌─────▼─────┐
│ HBM │
│ HOT │
└─────┬─────┘
│
┌─────▼─────┐
│ DRAM │
│ WARM │
└─────┬─────┘
│
┌─────▼─────┐
│ NVMe │
│ COLD │
└─────┬─────┘
│
┌─────▼──────┐
│ Remote │
│ Storage │
└─────────────┘
This looks remarkably similar to a computer memory hierarchy:
CPU cache
↓
RAM
↓
SSD
Except now:
GPU HBM
↓
CPU DRAM
↓
NVMe
↓
Network
Each level has different:
- latency
- bandwidth
- capacity
- cost
So the runtime needs to decide:
Where should this KV block live?
Hot blocks:
HBM
Warm blocks:
DRAM
Cold blocks:
NVMe / remote storage
SGLang's HiCache is an example of a system explicitly exploring multi-level KV-cache management.
At this point, LLM inference is starting to look less like a simple GPU program and more like a memory hierarchy manager.
6. KV Prefetching Turns Into an I/O Problem
Moving KV state between memory tiers can be expensive.
Suppose a request needs a block currently stored in CPU memory.
A naive implementation does:
Need KV
↓
Request transfer
↓
Wait
↓
KV arrives
↓
Compute
The GPU is waiting.
A better system tries to predict what will be needed next.
COMPUTE
│
│
▼
┌─────────┐
│ GPU │
└────┬────┘
▲
│
PREFETCH
│
│
┌────┴────┐
│ DRAM │
└─────────┘
While the GPU is computing, the runtime asynchronously moves future KV blocks closer.
This is classic prefetching.
The objective is to hide:
T_{memory}
behind:
T_{compute}
Ideally:
T_effective ≈ max(T_compute, T_memory)
rather than:
T_{effective}= T_{compute} + T_{memory}
Modern inference research is actively exploring asynchronous KV-cache prefetching because memory movement increasingly becomes a bottleneck.
The inference engine now needs to answer:
What will this request need next?
That's a prediction problem.
And prediction determines memory performance.
7. Cache-Aware Scheduling Turns Into an OS Scheduler Problem
Traditional request scheduling might look like:
Queue:
A
B
C
D
E
The scheduler chooses which request runs next.
But with KV caching, requests aren't independent.
Suppose:
A shares prefix with B
A shares nothing with C
A shares nothing with D
Running:
A → B
might be much more efficient than:
A → C → D → B
because A and B can reuse the same KV state.
So the scheduler now needs to consider:
GPU utilization
+
request latency
+
batch size
+
KV residency
+
prefix reuse
+
memory pressure
The scheduling problem becomes:
``
Choose(request)= f(latency,compute,memory,cache reuse)
``
This is very similar to an OS scheduler making decisions based on resource availability.
But now the scheduler is also trying to maximize memory locality.
That's a major conceptual shift.
The inference scheduler isn't just scheduling compute. It's scheduling data locality.
8. Prefill/Decode Disaggregation Turns Into Resource Scheduling
LLM inference has two very different phases.
Prefill
The model processes the user's prompt.
For example:
10,000 input tokens
↓
parallel processing
↓
KV cache generated
This tends to be relatively compute-intensive.
Decode
The model generates output one token at a time:
Token 1
↓
Token 2
↓
Token 3
↓
Token 4
Decode is much more sensitive to memory bandwidth and latency.
So the same GPU is being asked to serve two fundamentally different workloads.
This leads to prefill/decode disaggregation.
Conceptually:
Requests
│
┌──────┴──────┐
│ │
PREFILL DECODE
│ │
▼ ▼
GPU cluster GPU cluster
│ │
└──────┬──────┘
│
KV state
Now the system can optimize the two phases separately.
You can dedicate resources to:
Prefill:
compute throughput
and:
Decode:
memory bandwidth
latency
This is another classic systems principle:
Different workloads should not necessarily compete for the same resource pool.
It is similar to how operating systems and distributed systems isolate different classes of work.
9. Agentic Workloads Turn KV Management Into a Process-Lifetime Problem
This is where things get really interesting.
A normal chatbot looks like:
Request
↓
LLM
↓
Response
↓
Done
An agent looks more like:
LLM
↓
Tool call
↓
Wait
↓
LLM
↓
Database
↓
Wait
↓
LLM
↓
Browser
↓
LLM
The request can be inactive for seconds or minutes.
But inactive does not mean finished.
Suppose the agent has accumulated:
50,000 tokens
and is now waiting for a tool.
Should we keep all 50K tokens in HBM?
Maybe.
But if thousands of agents are simultaneously waiting:
Agent A → waiting
Agent B → waiting
Agent C → waiting
...
Agent 10,000 → waiting
we cannot keep everything in HBM.
So we need to understand request lifetime.
This starts looking like process management.
Traditional OS:
RUNNING
↓
WAITING
↓
RUNNABLE
↓
RUNNING
↓
TERMINATED
Agent:
GENERATING
↓
TOOL WAIT
↓
READY
↓
GENERATING
↓
TOOL WAIT
And memory policy now needs to understand those states.
A request that is waiting might have a high probability of becoming active again.
Therefore:
P(\text{reuse soon})
becomes relevant to eviction.
This is fundamentally different from simple LRU.
Agentic AI is therefore pushing inference toward application-aware memory management.
10. Distributed KV Turns Inference Into a Distributed Operating System
Now scale beyond one GPU.
A production system may have:
GPU 0 ─── GPU 1
│ │
│ │
GPU 2 ─── GPU 3
│
│
CPU RAM
│
│
Network
│
▼
Distributed storage
Now KV cache might exist:
GPU HBM
CPU DRAM
another GPU
another machine
NVMe
remote storage
The runtime needs to understand:
capacity
latency
bandwidth
topology
locality
contention
Moving a KV block from local HBM might be cheap.
Moving it from another machine could be much more expensive.
So:
``
Cost(block) =f(distance,bandwidth,latency,contention)
``
Now memory management and network scheduling become connected.
This resembles distributed operating systems and NUMA systems, where memory isn't uniformly accessible.
The runtime must answer:
Where should this state live, and where should computation happen?
That is a distributed-systems problem.
The Bigger Picture
Put all ten layers together:
LLM APPLICATION
│
AGENT
│
INFERENCE RUNTIME
│
┌──────────────┴──────────────┐
│ │
SCHEDULER MEMORY MANAGER
│ │
│ ┌───────┴───────┐
│ │ │
│ HBM DRAM
│ │ │
│ NVMe Network
│ │ │
└──────────────┬──────┴───────────────┘
│
GPU
And the ten transformations are:
1. KV Cache
↓
Dynamic memory
2. PagedAttention
↓
Paging
3. Prefix sharing
↓
Shared memory
4. KV eviction
↓
Cache management
5. Hierarchical KV
↓
Memory hierarchy
6. KV prefetching
↓
I/O management
7. Cache-aware scheduling
↓
OS-style scheduling
8. Prefill/decode disaggregation
↓
Resource scheduling
9. Agentic workloads
↓
Process lifetime management
10. Distributed KV
↓
Distributed memory management
This is why I think the phrase “inference OS” isn't as crazy as it initially sounds.
We are building increasingly sophisticated software to manage:
- compute
- memory
- state
- scheduling
- locality
- data movement
- sharing
- eviction
- prefetching
- distributed resources
The model is only one component.
The Most Important Difference
There is one major difference between traditional operating systems and LLM inference.
Operating systems mostly manage passive data.
LLMs generate their own working state.
Every token can create more KV cache:
Token
↓
KV state
↓
More memory
↓
More pressure
↓
Eviction / transfer
↓
Potential recomputation
So the workload itself continuously changes the memory requirements.
That's unusual.
And it means the inference runtime must understand both:
computation
and
memory.
Conclusion
We spent decades optimizing:
CPU → Memory → Storage
Then AI gave us:
GPU → HBM → CPU Memory → Storage → Network
And now the most interesting part of LLM infrastructure may not be the model itself.
It may be the system managing the model's working state.
Paging.
Caching.
Prefetching.
Eviction.
Scheduling.
Memory hierarchies.
Shared state.
Resource allocation.
Distributed memory.
These are all classic operating-system and systems concepts.
But they're being applied to something new:
the runtime state of a neural network.
The model generates tokens.
The tokens generate state.
The state consumes memory.
The memory determines scheduling.
The scheduler determines throughput.
And the entire system feeds back into the model's performance.
So perhaps the next generation of AI infrastructure won't just be:
better models + faster GPUs
It will be:
better inference operating systems.
The model is the intelligence.
The runtime is the machine that makes that intelligence usable.
Top comments (0)