Hello, I'm Maneshwar. I'm building git-lrc, a Micro AI code reviewer that runs on every commit. It is free and source-available on Github. Star git-lrc to help devs discover the project. Do give it a try and share your feedback.
Quick experiment. Open ChatGPT, ask it something, and watch the words spill out.
Now do the same in Gemini.
Notice anything?
The text does not always arrive at the same pace.
One feels like it is sprinting, the other feels like it is thinking out loud over coffee.
Here is the part that trips people up: that speed is not really about which model is "smarter."
A big chunk of it comes from the thing running underneath the model.
The plumbing. The engine. And once you see how that plumbing works, a lot of mysterious LLM behavior suddenly makes sense.
Now let's walk through it.
We are going to talk about inference, why the same model can run at wildly different speeds, and the clever operating-system trick that made modern LLM serving actually affordable.
First, what is "inference" anyway?
When people say training, they mean teaching the model.
Inference is the other half: the model is done learning, and now it has to actually do the job and generate tokens for you.
Every time you hit send, you are paying for inference.
We measure how fast this goes in tokens per second.
That is the speedometer for the whole conversation.
Higher number, snappier feel.
But generation is not one big magic step.
It happens in two phases, and understanding them is the key to everything else.
- Prefill. The model reads your entire prompt in one go and builds up its internal "notes" about it.
- Decode. The model then produces the answer one token at a time. Each new token depends on every token that came before it, which is why we call it autoregressive. It literally re-reads its own output to decide what comes next, like a writer who rereads the last sentence before writing the one after.
See that loop? That is where most of the time goes. And that little box called the KV cache is the secret villain of this whole story.
The KV cache: small idea, giant appetite
Every token the model has seen so far gets summarized into two vectors, a Key and a Value.
Storing them means the model does not have to recompute all that work for every single new token.
Without the cache, generating token number 500 would mean redoing the math for the previous 499 tokens. Ouch.
So the cache is great.
The catch is that it is hungry and it grows.
The longer the conversation, the bigger it gets, and it lives on the GPU where memory is precious and expensive.
For a 70B model, the KV cache for a single long request can run into tens of gigabytes.
Multiply that by a room full of users and you see the problem.
Now here is where older serving systems fell flat.
Because nobody knows in advance how long your answer will be, the naive approach was to reserve memory for the worst case, the maximum possible length, for every request.
Ask for a haiku? Congrats, the system still booked you the memory of a novel, just in case.
That leads to two flavors of waste, and both have proper names worth knowing:
- Internal fragmentation: you reserved a huge slab, used a sliver of it, and the rest just sits there reserved and useless.
- External fragmentation: after lots of requests grab and release different-sized slabs, GPU memory ends up looking like Swiss cheese. There is plenty of free space in total, but no single contiguous chunk big enough for the next request. So new requests get turned away while the GPU is technically half empty.
How bad was it? The vLLM team measured that older systems wasted roughly 60 to 80 percent of KV cache memory. That is not a rounding error. That is most of your very expensive GPU sitting idle while you pay for it.
Enter PagedAttention: stealing a trick from operating systems
Here is the "aha" that makes this fun. The vLLM folks at UC Berkeley looked at this mess and realized it was not a new problem at all.
Operating systems solved almost exactly this decades ago with virtual memory and paging.
Think about how your OS handles RAM.
Your program thinks it has one long, tidy stretch of memory.
In reality, the OS chops memory into fixed-size pages scattered all over the place, and a page table keeps track of which logical page maps to which physical spot.
Your program never notices. It just sees a clean, continuous view.
PagedAttention does the same thing to the KV cache.
Instead of one giant contiguous block per request, it slices the cache into small fixed-size blocks (16 tokens each by default).
Blocks get handed out on demand as the answer grows, and they can live anywhere in the memory pool.
A block table plays the role of the page table, mapping each logical block of a sequence to whatever physical block it landed in.
The payoff is huge.
Since blocks are tiny and allocated only when needed, the only waste left is the half-empty last block of each sequence, at most 15 unused token slots.
Memory waste drops from that ugly 60 to 80 percent down to under 4 percent.
Pack the memory tighter, fit more requests at once, and your throughput jumps 2 to 4 times on the exact same hardware.
No new GPUs required.
That is basically free money, which in GPU-land is a rare and beautiful thing.
Paging Dr. Memory, indeed.
The other half: keeping the GPU off the couch
Efficient memory is only useful if you actually put it to work.
This is where continuous batching comes in, and it pairs with PagedAttention like chips and salsa.
Old-school batching would group a bunch of requests, run them together, and wait for the whole batch to finish before starting the next one.
Problem: requests finish at different times.
The short ones sit there twiddling their thumbs while the GPU waits on the one guy asking for a 2000-word essay.
GPU utilization tanks.
Continuous batching is smarter.
The moment one request in the batch finishes, its slot is freed and a new waiting request slides right in.
The GPU never gets to nap.
Because PagedAttention makes it cheap to add and drop sequences on the fly (just update a block table, no giant memory shuffles), the two techniques together are what let a single box serve a genuinely useful number of concurrent users.
Plot twist: vLLM is not the answer to everything
Okay, so vLLM sounds like the hero. And it is a fantastic default.
But if you walk away thinking "just use vLLM for everything," you have missed the real lesson.
vLLM is optimized for one specific thing: high throughput when you are serving many users at once.
That is a great goal for a public API.
It is not the only goal, and other engines exist because other goals matter.
Let me introduce the rest of the lineup, because this is a genuine ecosystem and each member earned its seat:
- llama.cpp (and friendly wrappers like Ollama): the portability champ. Runs on CPUs, laptops, Macs, potatoes, whatever you have got. It leans on clever quantization to squeeze big models into small RAM. Perfect for single-user local tinkering. Not built for a stampede of concurrent traffic.
- TensorRT-LLM: NVIDIA's speed demon. It compiles your model into a highly tuned engine specific to NVIDIA hardware, and it tends to win raw tokens per second at sustained high load. The tax is a chunky compilation step and being locked to NVIDIA. Great when you are squeezing every last drop and your model is stable.
- SGLang: shines when your requests share long common prefixes, think agents, tool-calling loops, and RAG. Its RadixAttention reuses that shared prefix instead of recomputing it, which can pull ahead of vLLM on prefix-heavy traffic.
- Hugging Face TGI: an influential early production server, tightly wired into the Hugging Face ecosystem. Worth knowing that as of 2026 the project moved into maintenance mode and now points new users toward the others, which tells you how fast this space moves.
- LMDeploy: another solid serving option with its own bag of optimization and quantization tricks.
The honest takeaway is that the workload picks the engine, not a star rating on some leaderboard infographic.
Ask where it runs, how many users hit it, what your latency target is, and whether your requests look alike.
So what should you actually remember?
If you take three things away from our little chat, make it these:
- The engine matters as much as the model. Same weights, different serving stack, very different tokens per second. That speed gap you feel between chatbots is often an engineering choice, not an intelligence gap.
- Memory is the real bottleneck, and the KV cache is the culprit. PagedAttention borrowed OS paging to turn 60-to-80 percent waste into under 4 percent, which is why serving LLMs at scale stopped being financially terrifying.
- There is no universal "best" engine. There is only the best engine for your workload. Match the tool to the job and you will save yourself real money and real headaches.
The next time an LLM streams an answer at you, you will know there is a whole storage-Tetris tournament happening behind the curtain, packing blocks and juggling batches so your tokens show up fast.
Not bad for something you never see.
If you want to go deeper, the vLLM project on GitHub is very readable, and the original PagedAttention paper is worth a slow read on a rainy afternoon.
AI agents write code fast. They also silently remove logic, change behavior, and introduce bugs — without telling you. You often find out in production.
git-lrc fixes this. It hooks into git commit and reviews every diff before it lands. 60-second setup. Completely free.
Any feedback or contributors are welcome! It's online, source-available, and ready for anyone to use.
⭐ Star it on GitHub:
HexmosTech
/
git-lrc
Free, Micro AI Code Reviews That Run on Git Commit
| 🇩🇰 Dansk | 🇪🇸 Español | 🇮🇷 Farsi | 🇫🇮 Suomi | 🇯🇵 日本語 | 🇳🇴 Norsk | 🇵🇹 Português | 🇷🇺 Русский | 🇦🇱 Shqip | 🇨🇳 中文 | 🇮🇳 हिन्दी |
git-lrc
Free, Micro AI Code Reviews That Run on Commit
GenAI today is a race car without brakes. It accelerates fast -- you describe something, and large blocks of code appear instantly. But AI agents silently break things: they remove logic, relax constraints, introduce expensive cloud calls, leak credentials, and change behavior -- without telling you. You often find out in production.
git-lrc is your braking system. It hooks into git commit and runs an AI review on every diff before it lands. 60-second setup. Completely free.
In short, git-lrc helps Prevent Outages, Breaches, and Technical Debt Before They Happen
At a glance: 10 risk categories · 100+ failure patterns tracked · every commit…
Top comments (0)