I kept hearing the same phrase over and over again: "LLM inference is memory-bound." It appeared in conference talks, benchmark reports, GitHub discussions, and almost every conversation about serving large language models. I understood the words, but I never truly understood what they meant. The answer wasn't inside PyTorch or model.generate(). It was inside the GPU itself.
As I started learning what happens between that single line of Python and the hardware, everything began to click. Threads execute in parallel, data flows through multiple layers of memory, and the movement of bytes often matters more than the amount of computation being performed.
The biggest realization wasn't that GPUs are incredibly powerful. It was that many modern inference optimizations, including continuous batching, quantization, PagedAttention, and FlashAttention, are all solving different pieces of the same hardware problem.
In this article, we'll build that mental model. Instead of looking at these optimizations as isolated techniques, we'll understand the hardware principles that connect them and why they fundamentally shape the performance of every large language model.
Why GPUs Exist in the First Place
Before we talk about LLM inference, we need to answer a much simpler question: why do GPUs exist at all? If CPUs have been powering computers for decades, why wasn't that enough for modern AI?
The answer lies in the kind of work each processor is designed to do. A CPU is built to execute a wide variety of tasks as quickly as possible. It has a small number of powerful cores optimized for low latency, making it excellent at running operating systems, databases, web servers, and application logic where every instruction may be different from the last.
AI workloads are different. During inference, the same mathematical operations are repeated millions or even billions of times across large matrices. Instead of a few complex tasks, the hardware is asked to perform an enormous number of simple calculations simultaneously. That's exactly the problem GPUs were designed to solve.
Rather than using a handful of powerful cores, a GPU contains thousands of smaller execution units that work together on the same operation. This design sacrifices individual core performance in exchange for massive parallelism, allowing GPUs to process thousands of independent calculations at the same time.
This difference is the foundation of modern AI. The goal isn't to make each calculation faster. It's to perform as many calculations as possible in parallel. Once you understand that trade-off, concepts like warps, Tensor Cores, memory bandwidth, and even LLM inference start to make much more sense.
Two Numbers That Explain Almost Everything
If there's one mental model worth remembering from this article, it's this: every GPU has two finite resources, compute and memory bandwidth.
Compute tells you how quickly a GPU can perform mathematical operations. Memory bandwidth tells you how quickly it can move data between memory and the compute units. A model can only run as fast as the slower of these two resources allows.
Think of it like a professional kitchen. Hiring more chefs won't help if ingredients arrive too slowly. Likewise, a faster delivery truck won't improve throughput if there aren't enough chefs to cook the food. The kitchen is always limited by whichever resource becomes the bottleneck first.
The same idea applies to GPUs. Modern accelerators can perform an enormous amount of computation every second, but those computations are only possible after the required data reaches the execution units. If the GPU spends more time waiting for data than performing calculations, increasing compute power alone won't make inference any faster.
This is where a simple concept called arithmetic intensity becomes useful. It measures how much computation is performed for every byte of data moved through memory. In practice, this single metric predicts which resource, compute or memory bandwidth, will become the bottleneck before you even begin optimizing a workload.
High arithmetic intensity means the GPU spends most of its time computing. Low arithmetic intensity means it spends most of its time waiting for data.
That single ratio determines whether your workload is compute-bound or memory-bound, and as you'll see next, it explains why different phases of LLM inference behave so differently despite running on the exact same hardware.
The Two Personalities of LLM Inference
Now let's apply this mental model to LLM inference.
Although generating a response feels like one continuous process, the GPU is actually doing two very different kinds of work: prefill and decode.
The prefill phase begins when you send your prompt to the model. Every token in the prompt is processed together, allowing the GPU to execute massive matrix operations in parallel. Thousands of cores stay busy, Tensor Cores remain highly utilized, and the hardware spends most of its time performing computations rather than waiting for data. This is a classic compute-bound workload.
The story changes completely once the first token is generated.
During the decode phase, the model produces one token at a time. Instead of processing an entire sequence in parallel, it repeatedly loads model weights and cached attention data to predict the next token. Each iteration performs relatively little computation, but still requires a significant amount of data movement. The GPU spends more time fetching bytes from memory than doing arithmetic, making decode a memory-bound workload.
This explains why prompt processing often feels surprisingly fast while token generation slows down, especially for long conversations. The hardware hasn't changed, and neither has the model. What changed is the type of work the GPU is being asked to perform.
Once you recognize that prefill and decode stress completely different parts of the GPU, many production optimization techniques begin to make sense. The challenge is no longer making the GPU compute faster. It's finding ways to reduce or hide the cost of moving data.
The Hidden Memory Consumer: KV Cache
If you ask most engineers what occupies GPU memory during inference, the first answer is usually model weights. That's true, but only part of the story.
As soon as a conversation begins, the model starts building something called the Key-Value (KV) Cache. Instead of recomputing attention for every previous token, the model stores the intermediate key and value tensors produced at each transformer layer. Every new token can then reuse this information, making autoregressive generation practical.
The catch is that the KV cache grows with every generated token. A short conversation consumes relatively little memory, but a long conversation, or hundreds of conversations running at the same time, can quickly consume far more GPU memory than most people expect.
This creates an interesting trade-off. A larger context window improves the model's ability to remember earlier parts of the conversation, but it also increases the amount of data that must be read during every decode step. As the conversation grows, so does the memory traffic, reinforcing why decode remains fundamentally memory-bound.
This is also why production inference systems care so much about efficient KV cache management. Techniques such as PagedAttention, KV cache quantization, and continuous batching aren't independent optimizations. They're different approaches to managing the same resource: GPU memory.
Once you understand the role of the KV cache, another question naturally follows. If memory is the bottleneck, how do modern inference systems deliver higher throughput without changing the model itself? That's exactly what we'll explore next.
How Production Systems Push the Hardware Further
Once you realize the bottleneck is moving data instead of performing computation, the optimization strategy changes completely. The goal is no longer to make the GPU "faster." It's to use the available memory bandwidth more efficiently.
One of the most effective techniques is batching. Instead of serving a single request at a time, the GPU processes multiple requests together. Since the same model weights can be reused across the batch, the cost of loading those weights is shared, increasing the amount of useful computation performed for every byte fetched from memory.
Another widely used optimization is quantization. It's often described as a way to reduce computation, but during the decode phase, its biggest advantage is reducing memory traffic. Storing weights in lower precision formats such as FP16, INT8, or FP8 means fewer bytes need to be transferred from memory, allowing the GPU to spend less time waiting and more time generating tokens.
FlashAttention tackles the problem from a different direction. Instead of repeatedly writing intermediate attention results to GPU memory, it restructures the computation so more of the work stays in the GPU's fast on-chip memory. The mathematics remain exactly the same, but the amount of memory traffic is significantly reduced.
Managing the KV cache efficiently is just as important. Traditional memory allocation can leave unused gaps as conversations start and finish at different times. PagedAttention addresses this by borrowing ideas from virtual memory systems, allocating the KV cache in fixed-size pages that can be managed more efficiently. This reduces fragmentation, increases GPU utilization, and enables more concurrent requests without requiring additional hardware.
Modern serving frameworks like vLLM combine these techniques with continuous batching, dynamically merging incoming requests instead of waiting for fixed batches. The result is higher throughput, better GPU utilization, and significantly more efficient inference under real production workloads.
None of these optimizations change how the model thinks. They change how efficiently the hardware is fed with data, which is often the deciding factor in LLM serving performance.
The Real Mental Model
When I started learning GPU internals, I expected to come away with a list of new concepts: warps, Tensor Cores, shared memory, memory bandwidth, and CUDA kernels. Instead, I left with something much more valuable, a different way of thinking about inference performance.
Whenever an LLM feels slow, it's tempting to ask whether the model is too large or whether the GPU is powerful enough. Those are reasonable questions, but they aren't always the most useful ones. A better starting point is to ask what the hardware is waiting for. Is it busy performing computations, or is it sitting idle while data moves through memory?
That simple shift in perspective connects almost everything we've discussed. It explains why prompt processing and token generation behave differently, why the KV cache becomes such a critical resource, and why techniques like batching, quantization, PagedAttention, and FlashAttention exist in the first place. They're all different attempts to make better use of the same hardware.
The next time you read that an LLM inference workload is memory-bound, don't treat it as another piece of AI jargon. Think of it as a clue about what the GPU is actually doing. Once you understand that, many production decisions stop feeling like magic and start feeling like straightforward engineering trade-offs.
The frameworks we use continue to become simpler, but the hardware underneath hasn't become any less important. Understanding that layer won't turn you into a CUDA engineer overnight, but it will make you a better AI engineer the next time you're asked a simple question that rarely has a simple answer:
"Why is my LLM still slow?"
Conclusion
The goal of this article wasn't to teach CUDA programming or cover every optimization used in modern LLM serving. It was to build a mental model for understanding why LLM inference behaves the way it does.
Once you start looking at inference through the lens of compute and memory bandwidth, many techniques that initially seem unrelated begin to connect. Batching, quantization, KV cache management, PagedAttention, and FlashAttention aren't isolated performance tricks. They're engineering solutions to the same underlying hardware constraints.
The next time you come across a slow inference workload, resist the temptation to immediately blame the model or the GPU. Instead, ask a simpler question: What is the hardware waiting for? More often than not, the answer to that question points you toward the real bottleneck.
Understanding GPUs isn't about becoming a CUDA engineer. It's about understanding why modern AI systems behave the way they do. Once you build that perspective, many production optimizations stop feeling like magic and start feeling like logical engineering decisions.
🔗 Connect with Me
📖 Blog by Naresh B. A.
👨💻 Backend & AI Systems Engineer | Distributed Systems · Production ML
🌐 Portfolio: Naresh B A
📫 Let's connect on LinkedIn | GitHub: Naresh B A
Thanks for spending your precious time reading this. It's my personal take on a tech topic, and I really appreciate you being here. ❤️
Top comments (0)