Inside nano-vLLM: What an RTX 3090 Reveals About LLM Serving

0. What I studied and what I found

I studied nano-vLLM as a small LLM serving engine: modeled its prefill and decode costs, traced the implementation, and compared the predictions with Qwen3-0.6B BF16 on one RTX 3090 (24 GB). Its compact runtime makes the connection between scheduling, memory management, and GPU work possible to follow end to end.

Four findings organize this article:

Finding	Evidence at a glance
Cold prefill uses substantial compute capacity	Two representative shapes imply 58.6–60.1 TFLOP/s, about 76–84% of the locally measured BF16 GEMM reference. These compute rates are model-derived from measured token rates.
Single-request decode fits a memory-traffic model	At batch 1, estimated weight + KV traffic is 531–562 GB/s, versus a measured device-copy reference of 843 GB/s.
Batching strongly improves aggregate decode throughput	With 1,024-token prompts and 256-token outputs, 334 tok/s at batch 1 → 4,794 tok/s at batch 128. This is total throughput across requests.
Longer history slows decode	At batch 1, with 1,024 output tokens, prompt length 256 → 3,072 lowers decode throughput from 333.56 → 293.51 tok/s, consistent with growing KV reads.

These experiments connect nano-vLLM’s decode performance to batch size and KV history through a simple cost model:

decode bytes/token ≈ W / B + KV(Tavg) + overhead

W is weight size, B is the active decode batch, and KV(Tavg) is the history read for a request with average context length. In this byte model, overhead means additional memory traffic; scheduling and launch overhead also add time.

Takeaway: Prefill and decode stress different resources. Batching amortizes weights, while longer contexts increase the KV cost.

The rest follows that result backward: what the engine does, why these costs arise, how optimizations change them, and what the experiments actually establish.

1. What is nano-vLLM?

nano-vLLM implements an LLM serving engine in roughly 1,200 lines of Python. Its compact codebase makes the core serving concepts clear and easy to follow, making it a useful starting point for learning how an inference engine works.

It also delivers competitive offline inference performance. In the project’s official benchmark, Qwen3-0.6B on an RTX 4070 Laptop GPU processes 256 requests at approximately 1,434 tok/s with nano-vLLM and 1,362 tok/s with vLLM. That combination of readable code and practical performance makes it an appealing engine to study.

It turns generation requests into scheduled GPU work while managing the state each request needs between tokens. The implementation covers request scheduling, prefill/decode separation, KV cache, paged KV blocks, continuous batching, prefix caching, FlashAttention, and CUDA graphs.

Figure 1. Responsibilities in the runtime. Main arrows show work progression; side links show KV management and access, rather than a literal function-call stack.

The engine coordinates requests. The scheduler selects work. BlockManager tracks KV blocks. ModelRunner prepares tensors and executes the model, while attention uses the persistent KV state. Sampling and postprocessing produce outputs and advance or finish requests.

You can trace a request from scheduling through KV management to model execution in a small set of files. PyTorch, CUDA libraries, and FlashAttention perform much of the numerical work, while nano-vLLM exposes how the serving runtime coordinates it.

Takeaway: The serving engine decides which tokens run next and where their persistent state lives.

2. What did I find? Prefill and decode have different bottlenecks

Before explaining the optimizations, separate the two workloads that they serve.

Figure 2. Compute-heavy prefill and memory-heavy small-batch decode are useful workload hypotheses, not universal labels for every shape.

2.1 Prefill: many token rows, substantial compute work

Prefill processes the prompt and builds its K/V history. Many input tokens give the linear layers large matrix multiplications; attention also processes each prompt's causal history. The final prefill step can sample the first output token.

Two corrected cold-prefill measurements illustrate its compute behavior:

Prompt tokens	Batch	Measured prefill tok/s	Estimated compute rate	Share of local BF16 GEMM reference
1024	16	58,683	58.6 TFLOP/s	76–82%
4096	4	44,487	60.1 TFLOP/s	78–84%

These are cold-prefill results, with prefix reuse disabled. Compute rates are estimated from measured token throughput.

Compared with the 71–77 TFLOP/s local BF16 GEMM reference, these shapes use substantial compute capability. This does not establish that every prefill batch is purely compute-bound: attention, sequence lengths, and kernel shapes still matter.

2.2 Decode: little new input, much existing state

Decode contributes one latest token per active request per iteration. That token still uses the model weights and attends to its request's growing KV history.

In the batch-1 context sweep, increasing the prompt from 256 to 3,072 tokens reduced decode throughput from 333.56 to 293.51 tok/s. Each request produced 1,024 output tokens.

Adding estimated KV traffic to the weight term yields approximately 531–562 GB/s across the sweep, or 63–67% of the 842.87 GB/s device-copy reference. The fairly stable estimate is consistent with a memory-traffic explanation. It is not a direct DRAM measurement or a kernel bottleneck diagnosis.

Takeaway: For these shapes, prefill should be interpreted against compute capability; batch-1 decode is better explained by weight and KV traffic.

3. Why? Build the decode cost model

Start with one decode iteration containing B active requests. Each supplies one token, so the input has B rows:

X: [B, H]
linear layer: [B, H] @ [H, D]

The same weight matrices serve all those rows. A first-order model counts roughly W bytes of weights per iteration. Because the iteration produces B tokens, the weight traffic per output token is roughly W / B.

Attention has a different sharing boundary. Request 0 reads its history KV(T0), request 1 reads KV(T1), and so on. Across unrelated requests, that cost does not disappear when the batch grows. Averaged over the output tokens, it is approximately KV(Tavg).

Figure 3. The byte model. Box widths are conceptual, not proportional to measured traffic.

This gives the article's central model and a bandwidth-limited throughput estimate:

bytes/token ≈ W / B + KV(Tavg) + other traffic
TPS         ≈ effective BW / bytes_per_token

Two consequences follow directly:

• Batch increases → W/B decreases → aggregate decode throughput can improve. With about 1.503 GB of weights, the weight term falls from 1.503 GB/token at batch 1 to about 94 MB/token at batch 16.
• Context increases → KV(Tavg) increases → decode can slow down. Batching shares weight work, but every request still has a history to read.

Actual traffic depends on caching, tiling, layouts, and possible reloads. Scheduling, launch overhead, and finite compute throughput can also limit performance. This model predicts trends; it is not an exact hardware accounting identity.

Takeaway: Batching amortizes weights, but it cannot eliminate the per-request KV history.

4. How nano-vLLM moves the bottleneck

Each optimization changes one part of that cost picture. We can now connect the mechanisms instead of treating them as independent features.

4.1 KV cache — compute becomes persistent memory

Problem and idea. Without KV reuse, generating each token requires recomputing the prefix. Keeping historical K/V changes that into a one-token forward pass plus access to persistent history.

What changes. The avoided computation comes with a memory requirement:

KV(T) ≈ 2 × L × T × Hkv × bytes_per_element

L is the layer count; Hkv is KV heads times head dimension; the factor two accounts for keys and values. For this BF16 model, L = 28, Hkv = 8 × 128 = 1024, and each element uses two bytes. The result is 112 KiB per cached token, across all layers: 112 MiB for 1,024 tokens or 448 MiB for 4,096.

Implementation and evidence. ModelRunner.allocate_kv_cache() allocates the global KV tensor; Sequence.block_table identifies each sequence's blocks. prepare_decode() passes history metadata to attention, which uses flash_attn_with_kvcache(). The context sweep is consistent with the resulting traffic model; it is not a cache-on/off speedup experiment.

Takeaway: KV cache removes repeated prefix computation, but creates a growing KV capacity and bandwidth cost.

4.2 Paged KV — capacity becomes possible concurrency

Problem and idea. Reserving max_seq_len worth of KV for every request ties up memory that short requests may never use. Paged allocation maps logical sequence blocks onto available physical blocks.

Figure 4. Logical continuity does not require physical adjacency. Only referenced pool locations are shown.

What changes. Consider a hypothetical 14 GiB KV budget, a maximum length of 4,096, current lengths of 1,024, and 256-token blocks:

Strategy	KV per request	Requests fitting in this snapshot
Reserve 4,096 tokens	448 MiB	32
Allocate for 1,024 tokens	112 MiB	128

The chain is less over-reservation → more usable capacity → more possible concurrency → larger active B → smaller W/B. Growing requests still need additional blocks, and partially filled blocks still waste some slots. Paging does not directly reduce attention FLOPs.

Implementation and evidence. BlockManager tracks free physical blocks, Sequence.block_table stores the mapping, and prepare_block_tables() packs it for the GPU. I have not directly measured fragmentation or allocated-versus-used KV memory; the capacity table is an analytical example.

Takeaway: Paged KV allocation turns memory efficiency into higher possible concurrency.

4.3 Continuous batching — keep B high

Problem and idea. Suppose four requests finish after 10, 20, 50, and 100 decode iterations. If a fixed group cannot admit replacements until the longest finishes, its average active batch is (10 + 20 + 50 + 100) / 100 = 1.8, despite starting at four. Continuous admission allows new requests to enter as others finish.

Figure 5. Conceptual request-slot occupancy, not a GPU timeline. Replacement requests need prefill, and admission depends on queued demand and available KV capacity.

What changes. Sustaining B preserves weight amortization over time. New work is not free: prefill consumes GPU time and the studied scheduler gives it priority.

Implementation and evidence. Scheduler.waiting holds prefill work, running holds requests ready for decode, and postprocess() removes finished requests and releases KV blocks. The measured batch sweep rises from 334 tok/s at batch 1 to 2,606 at 16 and 4,794 at 128, then falls to 4,553 at 256. Conditions and the full curve appear in Section 6.

That demonstrates the value of batching, not an isolated continuous-batching speedup. A static-versus-continuous test needs the same arrival pattern and request lengths. At 256 submitted requests, extra prefill work also means the point is not a clean fixed-active-batch measurement.

Takeaway: Continuous batching can keep the weight-amortization benefit alive as individual requests finish.

4.4 Prefill batching — larger GEMMs, independent attention

Problem and idea. Prefill has many token rows per request. Prompts of 128, 256, and 64 tokens can share packed linear/MLP work as [448, H] @ W.

What changes. Packing improves matrix shapes and exposes parallel work. Attention must preserve request boundaries: its sequence-length scaling follows 128² + 256² + 64², not 448². Requests do not attend to each other's tokens.

Implementation and evidence. prepare_prefill() flattens input_ids; cu_seqlens_q and cu_seqlens_k preserve boundaries for variable-length FlashAttention. slot_mapping locates new KV stores. The corrected cold-prefill results in Section 2 demonstrate substantial compute use for the selected shapes, rather than an isolated packed-versus-unpacked speedup.

The word corrected matters: repeated prompts initially triggered prefix-cache reuse. Cold-prefill runs required disabling that cache and checking the actual processed-token count.

Takeaway: Prefill batching improves matrix shapes and parallelism while keeping attention sequences separate.

4.5 Prefix caching — reuse exact prefill work

Problem and idea. Requests may repeat the same system prompt or template:

Request A = [prefix P][suffix A]
Request B = [prefix P][suffix B]

Reusing eligible cached blocks of KV(P) avoids recomputing that prefix for each request.

What changes. Prefix caching reduces the amount of prefill work. It is exact computational reuse, not semantic caching: similar meaning does not imply identical token IDs and computational history.

Implementation and evidence. Block hashes incorporate token blocks and preceding prefix state. Cache lookup attaches reusable physical blocks; remaining tokens still need prefill. This reuse also affected my initial benchmarks: repeating the same prompt measured cached prefill rather than fresh prompt processing.

Takeaway: Prefix caching changes required prefill work, not the decode history cost of unrelated requests.

4.6 Chunked prefill — scheduling granularity

Problem and idea. A large prompt job can delay existing decode work. Dividing it into chunks bounds prompt work per scheduled iteration, creating an opportunity for a scheduler to serve decode between chunks:

One large job:          [long prefill................] [decode]
Possible interleaving: [chunk] [decode] [chunk] [decode] ...

What changes. Chunking changes scheduling granularity; it does not remove the total prompt computation. Latency depends on the scheduling policy.

Implementation and evidence. The studied scheduler uses max_num_batched_tokens and tracks progress through num_cached_tokens and num_scheduled_tokens. It returns selected prefill work before entering its decode branch. Consequently, this implementation does not guarantee the interleaving shown above. Chunk support alone is not evidence of improved decode latency.

I have not run a clean chunked-prefill on/off mixed-workload test. That needs time to first token (TTFT), inter-token latency (ITL), and throughput under the same arrivals and prompt lengths.

Takeaway: Chunked prefill is a scheduling mechanism whose latency benefit depends on how the scheduler uses it.

5. How is nano-vLLM implemented?

Follow request state through the runtime instead of reading the repository file by file.

Figure 6. Three paths through the same runtime. The columns are conceptual flows, not three simultaneously running GPU jobs.

5.1 New request → prefill

LLMEngine creates a Sequence and queues it. The scheduler selects prompt work and uses BlockManager to allocate or reuse blocks. ModelRunner.prepare_prefill() converts the selection into tensors; attention writes K/V through slot_mapping.

Watch four pieces of state: num_cached_tokens records progress, block_table locates persistent blocks, slot_mapping identifies new writes, and cu_seqlens preserves attention boundaries. After an incomplete chunk, postprocessing advances prompt progress without accepting an output token as a completion.

Connection to the model: this path materializes the KV state that makes one-token decode possible.

5.2 Running request → decode

The scheduler selects running sequences, ensures space through the block manager, and calls the decode preparation path. Each request contributes last_token; context_lens gives its history length and block_tables locates that history. Attention reads cached K/V, the sampler chooses the next token, and Scheduler.postprocess() updates or finishes the request.

Connection to the model: this is W/B + KV(Tavg) in operation—shared weight work across token rows, with separate historical state per request.

5.3 Prefix hit → remaining prefill

A block hash lookup identifies reusable full prefix blocks. Attaching those blocks and setting num_cached_tokens lets the model process the remaining uncached tokens. Hash state includes the preceding prefix, and reference counts track shared block use.

Connection to the model: the engine skips already materialized prefix KV rather than approximating a similar response.

Takeaway: Trace the selected tokens, block ownership, and state updates; those connect serving policy to attention's inputs.

6. Measurements — does the model match reality?

The three benchmarks test the same story from different directions: compute use, context growth, and batching.

Benchmark	What changes?	What stays fixed?
A: cold prefill	Two representative prompt/batch shapes	Qwen3-0.6B BF16 on RTX 3090
B: decode context sweep	Prompt length: 256–3072 tokens	Batch 1, output: 1024 tokens
C: decode batch sweep	Submitted requests: 1–256	Prompt: 1024 tokens, output: 256 tokens

6.1 Establish local hardware references

The RTX 3090 measured approximately 71–77 TFLOP/s for BF16 GEMM and 843 GB/s for device copy. These give us practical compute and bandwidth references for the machine running the model.

Full inference includes more than matrix multiplication or memory copying, so these are reference points rather than performance targets every workload should reach.

6.2 Experiment A: cold prefill versus the compute reference

Hypothesis: many prompt tokens expose substantial compute work. The measured token rates imply approximately 59–60 TFLOP/s for these two shapes.

Figure 7. Estimated prefill compute rates compared with the measured BF16 GEMM reference.

Interpretation: these two shapes achieve a substantial fraction of local compute capability. Longer prompts can lower tok/s even while estimated FLOP/s stays high, because attention adds work per token.

6.3 Experiment B: context length versus decode throughput

Hypothesis: at batch 1, a longer history increases KV reads per output token.

Figure 8. At batch 1, longer context lowers decode throughput.

The associated estimated KV reads grow from 0.088 to 0.411 GB/output token, while estimated weight-plus-KV bandwidth remains around 531–562 GB/s. That is consistent with the predicted trend: context increases, history traffic increases, and decode throughput falls.

6.4 Experiment C: batch size versus aggregate decode throughput

Hypothesis: increasing active B reduces weight traffic per output token, with diminishing returns as other costs matter more.

Figure 9. Batching helps strongly, then gains flatten. The orange point includes extra prefill work.

The small-to-medium batch gains fit weight amortization. Once that term shrinks, KV traffic, attention, sampling, block-table access, scheduling, and kernel shapes become increasingly relevant. The curve alone cannot rank these costs.

The 256-request run includes extra prefill work, so it is not a clean fixed-batch comparison. Its decline shows a limit worth investigating, without identifying a single cause.

6.5 How to read these numbers

Token throughput is measured; compute rates and KV bandwidth are model-based estimates. These benchmarks support the bottleneck story for this model and GPU, rather than establish a universal performance ranking or a comparison with vLLM.

Takeaway: The measurements support the predicted trends, while leaving specific kernel bottlenecks and isolated optimization speedups unresolved.

7. What I learned and what I would measure next

LLM serving optimizations do not eliminate bottlenecks. They move the bottleneck.

The chain starts with repeated computation. KV cache removes much of that repetition and leaves persistent memory costs. Paged allocation makes capacity more usable. Continuous batching can turn available capacity and queued demand into a sustained batch, reducing the weight cost per output token. Larger batches eventually expose other limits.

Prefix caching acts on a separate lever—the amount of exact prefill work that must run. Chunked prefill changes the scheduling unit; the policy determines whether that opportunity improves latency.

decode bytes/token ≈ W / B + KV(Tavg) + overhead

That model helps choose the next experiment. I would measure:

• Framework A/B: identical models, token IDs, sampling, hardware, and metric definitions for nano-vLLM and vLLM.
• Paged KV efficiency: allocated blocks, used tokens, wasted slots, and behavior as histories grow.
• Chunked-prefill latency: TTFT, ITL, and throughput under mixed prompt/decode traffic.
• Prefix-cache behavior: hit rates, saved tokens, eviction, and mixed reuse workloads.
• Quantization: weight traffic changes first, then KV quantization as a separate experiment.
• Profiling: Nsight Systems for execution gaps and Nsight Compute for kernel-level compute and DRAM behavior.

These are follow-up questions, not prerequisites for using the current results.

Takeaway: Start with the resource an optimization changes, predict the workload where it should matter, and then measure that workload.