Hello, I'm Shrijith Venkatramana. I'm building git-lrc, an AI code reviewer that runs on every commit. Star Us to help devs discover the project. Do give it a try and share your feedback for improving the product.
Many AI demonstrations look impressive.
One developer sends one prompt. One GPU generates one answer.
A production system is different.
One GPU serves hundreds of users at the same time. One user sends a short question. Another user uploads a 50-page PDF. One request finishes in 100 ms. Another request generates thousands of tokens. GPU memory fills. Latency increases. Throughput decreases.
In 2023, researchers at UC Berkeley studied this problem. The researchers found that the main bottleneck was no longer matrix multiplication.
The bottleneck was memory management.
The researchers built vLLM to solve this problem. Today, vLLM is one of the most widely used open-source systems for LLM serving.
The key idea did not come from machine learning.
The key idea came from operating systems.
1. Running One Model Is Easy. Serving Many Users Is Hard.
Many developers first use an LLM with a small Python program.
from transformers import AutoModelForCausalLM
output = model.generate(...)
This program works well during development.
A production service has different requirements.
Consider a customer support chatbot.
- User A sends 20 tokens.
- User B sends 5,000 tokens.
- User C receives streamed output for 30 seconds.
- User D cancels the request before completion.
All users share one GPU.
A modern GPU is expensive. An NVIDIA H100 can cost tens of thousands of dollars. Cloud providers also charge several dollars per hour for one GPU.
Suppose one GPU has only 20% utilization.
Then about 80% of the GPU investment produces no useful work.
The engineering question changes.
Instead of asking,
Can the model generate the correct answer?
the engineer asks,
Can the system keep the GPU busy every millisecond?
vLLM focuses on this second question.
2. The Hidden Memory Cost
Many developers think that model weights use most GPU memory.
During inference, another data structure also uses a large amount of memory.
This data structure is the KV cache.
Each generated token creates key tensors and value tensors.
Future attention operations reuse these tensors.
Without the KV cache, the model must process the complete prompt again for every generated token.
The KV cache removes this repeated work.
Consider a book.
If the writer has no notes, the writer must read the complete manuscript before writing the next paragraph.
The KV cache acts like these notes.
The KV cache grows during generation.
Suppose one request contains:
- 8,000 prompt tokens
- 1,000 generated tokens
The KV cache stores information for about 9,000 tokens.
Now multiply this memory by hundreds of active requests.
The available GPU memory decreases quickly.
Each request also grows at a different rate.
Some conversations finish after 50 tokens.
Some conversations continue for thousands of tokens.
Memory becomes fragmented.
This fragmentation became one of the main limits for high-throughput LLM serving.
3. The Operating System Idea
The Berkeley team saw a familiar problem.
The KV cache behaves like computer memory.
A process starts.
A process grows.
A process ends.
Operating systems solved this problem many years ago.
A modern operating system does not require one continuous block of RAM for every process.
Instead, the operating system divides memory into fixed-size pages.
Virtual memory maps logical addresses to physical pages.
An application does not know the physical location of each page.
The vLLM team applied the same idea to the KV cache.
This design became PagedAttention.
The KV cache is divided into many small blocks.
The blocks can exist anywhere in GPU memory.
A lookup table records the physical location of every block.
Traditional
Request
-------------------------
| Continuous KV Cache |
-------------------------
PagedAttention
Logical Cache
Block 1
Block 2
Block 3
Block 4
↓
GPU Memory
Page 7
Page 2
Page 19
Page 5
The model sees one logical KV cache.
The GPU stores many separate physical blocks.
This design is the same general idea that operating systems use for virtual memory.
4. Continuous Batching
PagedAttention solves one problem.
Scheduling solves another problem.
Traditional batching works like a bus.
The bus waits for all passengers.
Then the bus starts.
LLM inference originally behaved in a similar way.
The system created one batch.
The system waited.
The system completed the batch.
Then the system created the next batch.
Continuous batching behaves more like an elevator.
When one request finishes, a new request immediately enters the batch.
The GPU does not wait for the remaining requests.
Time →
Request A ███████████████
Request B █████
Request C ███████████
Request B finishes first.
The scheduler immediately inserts another request.
GPU utilization stays high.
Latency decreases.
Throughput increases.
This scheduling method is one reason why vLLM achieves much higher throughput than earlier serving systems.
5. Small Performance Gains Have Large Financial Value
Suppose one GPU serves 80 requests every second.
An improved serving system increases throughput to 120 requests every second.
This change increases throughput by 50%.
Now consider a service that processes 12 million requests every day.
Before optimization:
120 GPUs
After optimization:
80 GPUs
The service removes 40 GPUs.
Suppose one GPU costs $3 per hour.
40 × $3 × 24
≈ $2,880 each day
≈ $1 million each year
A small increase in utilization can produce a large reduction in infrastructure cost.
For many AI companies, inference now costs more than model training.
6. Why vLLM Matters
vLLM was developed by researchers including Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Lianmin Zheng, Joseph Gonzalez, Hao Zhang, and Ion Stoica at UC Berkeley.
The work grew from the LMSYS research group, which also created Chatbot Arena and Vicuna.
The researchers operated public LLM services.
The researchers saw that model quality improved quickly, but serving systems improved more slowly.
The team presented the paper Efficient Memory Management for Large Language Model Serving with PagedAttention at SOSP 2023, one of the leading conferences in computer systems.
This detail is important.
The paper is a systems paper.
The paper does not change the transformer architecture.
The paper changes resource management.
Today, many LLM serving systems use similar ideas.
7. The Bigger Lesson
Many people expect the next breakthrough to come from a larger model.
vLLM shows another path.
The transformer stayed the same.
The model weights stayed the same.
The prompts stayed the same.
The improvement came from better memory management, better scheduling, and better resource use.
This pattern appears many times in computer science.
Hardware improves.
Algorithms improve.
Then systems engineering becomes the next bottleneck.
LLM serving has reached this stage.
Future progress will depend on both better models and better systems.
What do you think?
Most discussion about AI focuses on larger models, better prompts, and fine-tuning.
Will the next 10× improvement come from better models, or will it come from better systems such as memory management, scheduling, caching, and GPU utilization?
*AI agents write code fast. They also silently remove logic, change behavior, and introduce bugs -- without telling you. You often find out in production.
git-lrc fixes this. It hooks into git commit and reviews every diff before it lands. 60-second setup. Completely free.*
Any feedback or contributors are welcome! It's online, source-available, and ready for anyone to use.
HexmosTech
/
git-lrc
Free, Micro AI Code Reviews That Run on Git Commit
| 🇩🇰 Dansk | 🇪🇸 Español | 🇮🇷 Farsi | 🇫🇮 Suomi | 🇯🇵 日本語 | 🇳🇴 Norsk | 🇵🇹 Português | 🇷🇺 Русский | 🇦🇱 Shqip | 🇨🇳 中文 | 🇮🇳 हिन्दी |
git-lrc
Free, Micro AI Code Reviews That Run on Commit
GenAI today is a race car without brakes. It accelerates fast -- you describe something, and large blocks of code appear instantly. But AI agents silently break things: they remove logic, relax constraints, introduce expensive cloud calls, leak credentials, and change behavior -- without telling you. You often find out in production.
git-lrc is your braking system. It hooks into git commit and runs an AI review on every diff before it lands. 60-second setup. Completely free.
In short, git-lrc helps Prevent Outages, Breaches, and Technical Debt Before They Happen
At a glance: 10 risk categories · 100+ failure patterns tracked · every commit…
Top comments (0)