Hello, I'm Shrijith Venkatramana, and I'm building LiveReview — a blast-radius aware AI code review built for your business-critical systems. Star us to help devs discover the project, give it a try, and share your feedback to help improve the product.
You can have eight expensive GPUs, a well-optimized model, fast networking, FlashAttention, continuous batching, and a carefully tuned runtime.
And still waste a large fraction of your machine doing absolutely nothing.
The culprit is often a pipeline bubble.
A pipeline bubble is idle capacity created by dependencies between stages. One GPU is ready to work, but the data it needs has not arrived yet. Or the work assigned to another stage has already finished, so that stage sits idle.
This is one of those systems problems that becomes more important as LLMs get larger.
The interesting part is that the solution is rarely "buy a faster GPU."
The solution is usually:
keep the pipeline full.
1. The intuition: think like a factory
Imagine a factory with four workers:
Worker 1 -> Worker 2 -> Worker 3 -> Worker 4
Each product must pass through all four workers.
If there is only one product:
Time --->
W1: [A]
W2: [A]
W3: [A]
W4: [A]
Three workers spend most of the time waiting.
Now split the work into several small batches:
Time --->
W1: [A] [B] [C] [D] [E]
W2: [A] [B] [C] [D] [E]
W3: [A] [B] [C] [D] [E]
W4: [A] [B] [C] [D] [E]
Now the workers operate concurrently.
The factory has not become faster at performing an individual operation.
It has become better at overlapping operations.
That is the central idea behind pipeline parallelism in deep learning.
The historical lineage is worth remembering. GPipe, published at NeurIPS 2019 by Yanping Huang and colleagues at Google, made pipeline parallelism practical for very large neural networks by splitting batches into micro-batches and pushing them through partitions of the network. Their demonstration included a 6-billion-parameter, 128-layer Transformer trained across more than 100 languages. (Google Research)
The problem is that even a pipeline has startup and shutdown costs.
Those costs are the bubble.
2. Where the bubble comes from
Suppose you have:
p = 4 pipeline stages
m = 4 micro-batches
Ignoring communication for a moment, a simplified schedule looks like:
Time
Stage 1: A A A A
Stage 2: A A A A
Stage 3: A A A A
Stage 4: A A A A
At the beginning, only Stage 1 has something to do.
Then Stage 2 wakes up.
Then Stage 3.
Then Stage 4.
The startup period is the fill.
The reverse happens at the end. Work drains out of the pipeline, leaving stages idle.
For the classic GPipe-style schedule, a useful approximation for bubble fraction is:
bubble_fraction = (p - 1) / (m + p - 1)
where:
p = number of pipeline stages
m = number of micro-batches
Suppose you use 8 GPUs and 8 micro-batches:
bubble = (8 - 1) / (8 + 8 - 1)
= 7 / 15
= 46.7%
That is catastrophic utilization.
Increase the micro-batches to 64:
bubble = 7 / (64 + 8 - 1)
= 7 / 71
= 9.9%
The GPUs did not become faster.
You simply gave the pipeline more work to overlap. (arXiv)
This gives a useful engineering rule:
micro-batches >> pipeline stages
A pipeline with 32 stages and only 8 micro-batches is structurally difficult to keep busy.
A pipeline with 8 stages and 128 micro-batches has much more opportunity for overlap.
3. LLMs make the problem more interesting
LLM inference introduces another source of imbalance.
An LLM request has two fundamentally different phases:
Prefill -> Decode -> Decode -> Decode -> ...
Prefill
The model consumes the entire input prompt.
For a 4,000-token prompt, it may process thousands of tokens in parallel.
This is compute-heavy and tends to use the GPU efficiently.
Decode
The model generates one new token at a time.
So a request might behave roughly like:
Prefill: 4000 tokens
Decode: 1 token
Decode: 1 token
Decode: 1 token
...
The hardware characteristics are therefore different.
Prefill tends to be compute-oriented.
Decode tends to be constrained by memory movement and KV-cache access, and can have much lower arithmetic intensity.
Now imagine putting these requests into a pipeline.
One micro-batch might contain a large prefill:
████████████████████
Another might contain several decode steps:
██
The stages no longer have equal workloads.
You can therefore get a second kind of bubble:
Stage 1: ████████████████████
Stage 2: ████████████
Stage 3: █████
Stage 4: ███
The pipeline is technically full of requests.
It is simply poorly balanced.
This is the problem addressed by SARATHI, a 2023 system from Amey Agrawal, Ashish Panwar, Jayashree Mohan and colleagues at Microsoft Research India and Georgia Tech. Instead of treating prefill and decode as unrelated workloads, SARATHI chunks large prefills and combines a prefill chunk with multiple decode requests. The idea is to make the amount of work in successive micro-batches more uniform. (arXiv)
Their experiments are a useful reminder that "more batching" is too vague.
The important question is:
what shape should the batches have?
4. The first real lever: micro-batch size
Suppose a model takes approximately:
T = 10 ms
to process one micro-batch through a pipeline stage.
You have:
p = 8 stages
m = 8 micro-batches
The useful work is roughly:
8 micro-batches * 10 ms
= 80 ms
But the pipeline has to fill and drain.
The idealized total becomes proportional to:
m + p - 1
= 8 + 8 - 1
= 15 time units
The problem becomes much smaller if:
m = 64
m + p - 1
= 64 + 8 - 1
= 71
The fill cost has not changed.
You have simply amortized it over more useful work.
This is the same basic principle behind many queueing systems:
fixed overhead / amount of work
gets smaller as the amount of useful work increases.
But there is a catch.
Micro-batches consume memory.
During execution, intermediate activations or KV-cache state have to remain live for work that is still "in flight."
So we have a systems tradeoff:
more micro-batches
|
+--> smaller bubbles
|
+--> more memory pressure
And there is another problem:
smaller micro-batches
|
+--> worse GPU kernel efficiency
A GPU generally prefers enough work to form large, efficient matrix operations.
So the engineering objective is not:
maximize micro-batches
It is:
maximize useful overlap
subject to
memory + kernel-efficiency + latency constraints
5. The second lever: balance the stages
Imagine four GPUs running these workloads:
GPU 0: 100 units
GPU 1: 100 units
GPU 2: 100 units
GPU 3: 160 units
The pipeline's throughput is determined by GPU 3.
The other GPUs repeatedly reach:
done
waiting
done
waiting
done
waiting
You have effectively purchased a 160-unit machine and attached three 100-unit machines to it.
This is why pipeline partitioning matters.
If the work can be rearranged:
Before:
100 | 100 | 100 | 160
After:
115 | 115 | 115 | 115
the slowest stage moves from 160 to 115.
This improves the entire pipeline.
In mathematical terms, a simple approximation for steady-state throughput is:
throughput ~= 1 / max(T1, T2, ..., Tp)
where Ti is the execution time of stage i.
That max() is important.
The pipeline does not care that the average stage takes 112 ms.
It cares that the slowest stage takes 160 ms.
DeepSpeed therefore exposes explicit mechanisms for partitioning models across pipeline stages, including parameter-based and layer-based partitioning. Its documentation also makes the operational point directly: pipeline performance depends strongly on load balance. (DeepSpeed)
For LLMs this becomes especially relevant because layers are not always identical in real systems.
Different layers can have different memory behavior.
Attention-heavy components can behave differently from MLP-heavy ones.
Communication can differ depending on placement.
A partition that looks balanced by parameter count may still be unbalanced by actual runtime.
The correct metric is usually:
wall-clock stage time
not:
number of layers
6. The third lever: change the schedule
There is another insight from the history of pipeline systems.
The naive strategy is:
Forward all micro-batches
Backward all micro-batches
GPipe's fill-drain approach works this way.
An alternative is to interleave forward and backward operations:
1F1B
1 Forward
1 Backward
1 Forward
1 Backward
...
PipeDream, associated with Aaron Harlap, Deepak Narayanan, Amar Phanishayee, Vivek Seshadri and others, explored this style of inter-batch pipelining for distributed DNN training. The system explicitly targeted better overlap and higher accelerator utilization. (arXiv)
The important systems lesson is broader than the specific algorithm:
the schedule itself is a resource-allocation policy.
Consider:
GPU 0:
F1 F2 F3 F4 F5 F6
GPU 1:
F1 F2 F3 F4 F5 F6
GPU 2:
F1 F2 F3 F4 F5 F6
versus a schedule that starts backward work as soon as dependencies permit.
You are changing when memory is allocated, when communication happens, and when individual GPUs become free.
For inference systems, the same idea appears in different forms.
You may have:
prefill queue
decode queue
waiting requests
KV-cache memory
GPU compute capacity
A scheduler decides which work enters the pipeline next.
That makes scheduling an optimization problem.
A crude objective function might look like:
maximize:
GPU utilization
+ throughput
- latency penalty
- memory pressure
- scheduling overhead
Real systems are more complicated, but this mental model is useful.
7. The economics: pipeline bubbles are cash sitting idle
Suppose you rent:
8 GPUs
$3/hour/GPU
Your infrastructure cost is:
8 * $3 = $24/hour
Now suppose your effective utilization is 60%.
Very roughly, you are paying:
$24/hour
to get something closer to:
8 * 0.60 = 4.8 fully utilized GPU-equivalents
That gives:
effective cost per fully-utilized GPU
= $24 / 4.8
= $5/hour
Improve utilization from 60% to 80% without buying anything:
8 * 0.80 = 6.4 GPU-equivalents
Now:
$24 / 6.4
= $3.75/hour
You have effectively reduced infrastructure cost per unit of useful compute by about:
1 - 3.75/5
= 25%
This is why pipeline-bubble management is an economics problem as much as a GPU programming problem.
The same reasoning applies to latency.
Imagine a service where:
GPU computation = 30 ms
pipeline waiting = 20 ms
network = 5 ms
queueing = 10 ms
The customer experiences approximately:
65 ms
Yet only 30 ms is actual model computation.
Making the matrix multiplication 10% faster gives:
30 -> 27 ms
total:
27 + 20 + 5 + 10
= 62 ms
Removing the 20 ms pipeline bubble gives:
30 + 0 + 5 + 10
= 45 ms
The optimization with the larger effect is not the one involving the GPU kernel.
It is the one involving the system around the GPU kernel.
That is the deeper lesson of pipeline management.
When your LLM system grows, ask:
Where is work waiting?
before asking:
How do I make the work faster?
8. How to debug a pipeline bubble in practice
The easiest mistake is to look at aggregate GPU utilization.
Suppose your dashboard says:
GPU utilization: 72%
That does not tell you whether you have a pipeline problem.
You want a timeline.
For example:
GPU 0: ███████████████████████████████
GPU 1: █████████████████████████████
GPU 2: █████████████████████████
GPU 3: █████████████████████
versus:
GPU 0: ███████████████████████████████
GPU 1: ███████████████████████████████
GPU 2: ███████████████████████████████
GPU 3: ███████████████████████████████
The second is what you want.
For an actual LLM service, instrument at least:
request arrival
queue wait
prefill start/end
decode start/end
stage start/end
communication start/end
KV-cache allocation
KV-cache eviction
request completion
Then calculate:
stage_utilization_i
= busy_time_i / wall_clock_time
and:
pipeline_efficiency
= useful_work / total_pipeline_work
Also measure the variance of stage execution time:
CV = standard_deviation(stage_time) / mean(stage_time)
A high coefficient of variation is often a warning that a nominally balanced workload is not actually balanced.
For inference, measure prefill and decode separately.
A service can have excellent aggregate throughput while having terrible tail latency because long prefills repeatedly interfere with decoding.
This is exactly the kind of workload interaction that motivated SARATHI's attempt to make micro-batches more uniform. (Microsoft)
9. The mental model to keep
There are really four knobs:
Pipeline Performance
|
+--------------+--------------+
| | |
Micro-batches Stage balance Schedule
| | |
fill/drain bottleneck ordering
|
Memory
And for modern LLM inference there is a fifth:
Work composition
prefill <-> decode
The central equation is almost embarrassingly simple:
pipeline throughput ~= 1 / slowest_stage_time
and the central intuition is even simpler:
idle hardware is lost capacity
This is why pipeline-bubble management deserves to be thought of as a first-class LLM engineering discipline.
The famous scaling story of LLMs is usually about FLOPs, parameters, memory bandwidth, attention kernels, quantization and interconnects.
Those matter.
But once you have a sufficiently large distributed system, another question becomes just as important:
how much of the machine is actually doing useful work at every moment?
GPipe showed how micro-batching could make huge models practical. PipeDream demonstrated how scheduling and pipelined execution could keep distributed accelerators productive. SARATHI showed that, for LLM inference, even the composition of a micro-batch can determine how large your bubbles become. (Google Research)
The modern LLM engineer therefore has a slightly different optimization question.
Not:
"How fast is my GPU?"
But:
"For every millisecond of my GPU bill,
how many milliseconds contain useful work?"
That is pipeline-bubble management.
What pipeline bottleneck have you encountered in practice: micro-batch bubbles, unbalanced stages, prefill/decode interference, or communication stalls?
Your team's attention is limited, and the deluge of AI-generated code is making it harder to keep production reliable and secure without slowing you down.
I'm building LiveReview, a blast-radius aware AI code review built for your business-critical systems.
Instead of presenting every diff with equal emphasis, LiveReview scores each change by blast radius — how far its impact reaches through your call graph — so you can focus attention where it actually matters.
Spend code review effort where business risk is highest — not spread evenly across every diff.
⭐ Star it on GitHub:
HexmosTech
/
LiveReview
Blast-Radius Aware AI Code Review for Business-Critical Systems
LiveReview: Blast-Radius Aware AI Code Review for Business-Critical Systems
LiveReview is an AI code reviewer that scores every hunk of a diff by blast radius: how far a change reaches through your call graph, how much persistent state it touches, and how well-tested it is. A 3-line change to a shared auth check can outrank a 300-line UI tweak. Your team's attention goes to the highest-risk code first, not spread evenly across every diff.
blast-radius-demo.mp4
LiveReview's Blast Radius & Review Priority scoring, live in the diff viewer.
The exact math, not a black box
Visualize blast radius at a glance
Every factor that feeds the score
Here's the goal:
- A 3-line fix in a function used by 40 other files, that also writes to a database, should score high.
- A 300-line UI change in one file, fully covered by…
Click below to try LiveReview with your codebase:
Top comments (0)