Batch and Micro-Batch Size Flags in llama.cpp

#ai #machinelearning #llm #programming

-b defaults to 2048 and -ub to 512, and almost every question about them comes down to one fact: the compute buffer is sized by the second number, not the first.

Two numbers, logical and physical

llama.cpp’s help calls -b/--batch-size the logical maximum batch size and -ub/--ubatch-size the physical maximum batch size, and the distinction is exactly what it sounds like.

The logical batch is the unit of scheduling — how many tokens are submitted to the runtime in one call. On a server it is how many tokens across all active sequences can be gathered into one round of work.
The micro-batch is the unit of execution — how many tokens go through the model in one forward pass. A logical batch of 2048 with a micro-batch of 512 is four passes.

Both only concern prefill: reading a prompt, where every token is known in advance and can be processed in parallel. That is what makes prompt processing arithmetic-bound and fast, in contrast to generation, which is one token at a time by construction — see the KV cache for why the sequential half cannot be batched the same way.

The micro-batch sizes the compute buffer

When llama.cpp reserves its compute graph it does so for min(n_ctx, n_ubatch) tokens. Every activation tensor in that graph is proportional to that token count, so the compute buffer reported at load — compute buffer size = ... MiB — scales with -ub and is essentially independent of -b.

This gives you a clean derivation. Doubling -ub from 512 to 1024 roughly doubles the compute buffer. Doubling -b from 2048 to 4096 does not, because the larger logical batch is still executed in 512-token passes. If a run aborts allocating a buffer, the number to reduce is -ub; if prompt throughput is disappointing on a fast GPU, the number to raise is usually also -ub, because that is what determines how much parallel work each kernel launch gets.

The direction of the trade is the same on every backend even though the magnitudes are not: a larger micro-batch means more arithmetic per weight read, which is what a GPU wants, at the cost of a proportionally larger scratch allocation. llama.cpp’s own llama-bench README shows the shape of the curve in its batch-size example: on a 7B Q4_0 on CUDA, prompt processing of 1024 tokens runs at 1436 t/s at -b 128, 1932 at 256, 2254 at 512 and 2498 at 1024 — steadily improving and steadily flattening. Those are that machine’s numbers, not yours; the flattening is the transferable part.

Three clamps that override what you typed

The values you pass are maxima and requests, not settings, and three rules can quietly change them:

Context clamps the batch. For a causal model llama.cpp uses n_batch = min(n_ctx, requested). Run with -c 512 -b 2048 and your batch is 512. Nothing warns.
The batch clamps the micro-batch. n_ubatch = min(n_batch, requested), and a requested value of 0 means “use the batch size”. So -ub larger than -b is not an error and not honoured.
An oversized submission is an assertion, not an error. Internally, n_tokens_all <= n_batch is a GGML_ASSERT. Code that hands llama.cpp more tokens in one call than the configured batch aborts the process rather than returning a failure — relevant if you are driving the library directly rather than through the server, which chunks for you.

Both defaults print at context creation as n_batch and n_ubatch. Reading those two lines is faster than reasoning about which clamp applied.

Neither flag speeds up generation

This is the misconception these flags attract. Generating a token requires the token before it, so there is exactly one token of work available per step for a single sequence. A micro-batch of 512 does not help; 511 of those slots have nothing to put in them. Generation speed is set by memory bandwidth and by how much of the model is on the fast device, which is the subject of the -ngl page.

The exception is genuine concurrency. With several sequences in flight, their generation steps can be batched together — one weight read serving many tokens — which is why a server saturates a GPU far better than a single chat does. That is the mechanism behind continuous batching, and on llama-server it is governed by the slot count rather than by -b. Speculative decoding is the other way to create parallel work for a single stream, by proposing several tokens and verifying them in one pass.

What to change, and in which situation

Out of memory during load, at a context you need — lower -ub to 256 or 128. The prompt-processing loss is modest; the buffer saving is proportional.
Slow prompt processing on a GPU with memory to spare — raise -ub to 1024 or 2048, and raise -b to match or exceed it. Sweep it rather than guessing: llama-bench -p 2048 -n 0 -ub 128,256,512,1024,2048.
CPU-only inference — the returns from a large micro-batch flatten early, because a CPU has far less parallelism to fill. The default 512 is usually near the top of the curve; thread count matters much more, and the threads page covers finding it.
Server with many short requests — a larger -b lets more sequences be gathered into one scheduling round, which is where it earns its keep. A larger -ub mostly costs you memory that would be better spent on slots.

One caution on benchmarking: raising -ub changes the compute buffer, which changes how much VRAM is left for the KV cache, which can change the context the fitter settles on. Pin -c explicitly when you sweep, or you will be comparing two different configurations and attributing the difference to the batch.