Hello, I'm Rijul, and I'm building LiveReview — a blast-radius aware AI code review built for your business-critical systems. Star us to help devs discover the project, give it a try, and share your feedback to help improve the product.
When you use an AI chatbot, have you ever wondered how it can respond to thousands of users with a limited number of GPUs?
One technique that makes this possible is continuous batching.
To understand how it works, let's first look at why processing one request at a time is inefficient, and how continuous batching helps LLM servers serve more users with the same hardware.
1. Why Generating Tokens Can Be Wasteful
A language model generates its response one token at a time. For each generation step, the GPU must perform computations using the model's weights.
For many models during token generation, moving these weights from memory can take more time than performing the calculations themselves. This means the GPU's computing power may not be fully utilized.
Think of a chef who walks to a distant pantry to collect ingredients but prepares only one dish per trip. Much of the time is spent walking instead of cooking.
The obvious solution is to prepare several dishes during each trip.
2. The First Fix: Batching
Instead of processing one user's request at a time, an LLM server can process multiple requests together. This is called batching.
The server loads and uses the model's weights to process several requests during the same generation step. Although processing a larger batch requires additional computation, the cost does not necessarily increase in proportion to the number of requests.
The result is that the server can generate tokens for multiple users during the same step.
This improves throughput, which means the total number of tokens generated per second across all users.
However, batching introduces another problem: different users' requests finish at different times.
3. The Problem With Static Batching
A simple batching strategy, called static batching, works like this:
- Collect a group of requests, such as four.
- Process them together.
- Wait until all four requests finish before starting the next group.
Consider these four requests:
| Request | Tokens to generate |
|---|---|
| A | 10 |
| B | 500 |
| C | 40 |
| D | 60 |
Request A finishes after generating 10 tokens. Requests C and D finish at steps 40 and 60, respectively. But request B continues generating tokens until step 500.
In a conventional static batch, the completed requests cannot simply be replaced with new ones while the batch is running. Their slots may remain unused until the longest request finishes.
For most of the generation process, the server could be working on request B while the other three slots sit idle.
Meanwhile, new users may be waiting in a queue, even though some of the batch's capacity is no longer being used.
4. The Solution: Continuous Batching
Continuous batching solves this problem by reconsidering the batch after every generation iteration instead of waiting for the entire group to finish.
The process works like this:
- The server generates the next token for the active requests.
- When a request finishes, the server releases its slot.
- If another request is waiting, the server can schedule it into the available capacity.
- The process repeats while requests continue arriving and finishing.
Let's return to our example.
Request A finishes at step 10. The server can schedule a new request, E, to use the available capacity in a subsequent iteration. When request C finishes at step 40, another waiting request, F, can take its place.
Instead of keeping the original group fixed, the server continuously adapts to the requests that are still running and the requests waiting to start.
The goal is to keep the GPU productively occupied and improve overall throughput.
Think of a restaurant. With static batching, the restaurant waits for every table in a group to become empty before seating another group of customers. With continuous batching, a new party can be seated as soon as a table becomes available.
You may also encounter the terms in-flight batching and iteration-level scheduling. These are commonly used to describe this general approach.
5. What Happens When a New Request Arrives?
A new request does not immediately start generating its answer. It first needs to pass through two distinct stages.
Prefill: Reading the prompt
During prefill, the model processes the user's input prompt. Unlike generation, where the model produces one new token per step, prefill can process the prompt's tokens in parallel within the model's sequence computation.
Long prompts can require substantial computation and memory, so prefill can temporarily compete with the decoding work of existing requests.
Decode: Generating the answer
During decode, the model generates the response one token at a time. Each new token depends on the preceding context.
Continuous batching is especially useful for keeping multiple requests' decode steps moving together.
The challenge of long prompts
Suppose a new user submits a very long document while several other users are waiting for their next tokens.
Processing the entire prompt in one large prefill operation could delay the decoding of existing requests.
One technique used to manage this is chunked prefill. The server splits a long prompt into smaller chunks and schedules those chunks alongside ongoing decode work.
This can help balance prompt processing and token generation, improving scheduling efficiency. The exact behavior depends on the serving system and its configuration.
6. Where Can You Try Continuous Batching?
You generally do not need to implement continuous batching from scratch. Modern LLM serving systems provide batching and request-scheduling mechanisms.
Examples include:
- vLLM
- NVIDIA TensorRT-LLM
- Hugging Face Text Generation Inference (TGI)
The specific implementation and available configuration options differ between systems and versions. If you deploy a model using one of these servers, check its documentation to understand how continuous batching is enabled and configured.
Conclusion
Serving many users with a limited number of GPUs is not just about using a bigger model or adding more hardware. It is also about making better use of the hardware already available.
Static batching groups requests together but can waste capacity when shorter requests finish before longer ones. Continuous batching addresses this by allowing new requests to join as existing requests complete.
By adapting the batch at each generation iteration, LLM servers can use their available capacity more efficiently and improve throughput.
The key idea is simple: don't wait for every request to finish before making room for the next one.
Your team's attention is limited, and the deluge of AI-generated code is making it harder to keep production reliable and secure without slowing you down.
I'm building LiveReview, a blast-radius aware AI code review built for your business-critical systems.
Instead of presenting every diff with equal emphasis, LiveReview scores each change by blast radius — how far its impact reaches through your call graph — so you can focus attention where it actually matters.
Spend code review effort where business risk is highest — not spread evenly across every diff.
⭐ Star it on GitHub:
HexmosTech
/
LiveReview
Risk-Aware AI Code Review for Business-Critical Systems
LiveReview: Risk-Aware AI Code Review for Business-Critical Systems
LiveReview is built for engineering teams that ship often, but also care about keeping production stable for customers. It approves some 40-50% of low-risk changes automatically, and it gives your team the tools to handle the rest:
- Understand risky changes faster.
- Fix problems in those changes, with human attention directed to the work that needs it.
- Enforce your engineering standards.
- Learn from production outcomes, such as incidents, rollbacks, and queue wait times.
- Improve the review process over time and make it more efficient.
- Connect to the major engineering tools that your team already uses.
Auto-Approve: Ship the Safe 50% Without a Human
Auto-Approve Upto 50% of Your Pull Requests, Safely. Risk scoring decides which MRs/PRs are safe enough to ship without a human, so your team only spends review time on risky and critical changes.
demo.mp4
LiveReview auto-approving a low-risk change while…
Click below to try LiveReview with your codebase:
Top comments (0)