Multi-LoRA Serving: Why 100 Adapters Fit on One GPU

You fine-tuned 40 LoRA adapters — one per enterprise customer — and your infra lead priced out 40 GPU replicas. That estimate is off by roughly 40x. Multi-LoRA serving puts all 40 adapters on one GPU, in the same batch, at close to base-model throughput. The reason is arithmetic, not magic: a rank-16 adapter on a 7B model is about 33 MB, and the base weights are 14 GB.

The part nobody tells you is where it breaks. It does break, and the failure mode is a throughput cliff, not an error.

TL;DR

• Multi-LoRA serving keeps one copy of base weights in GPU memory and applies per-request low-rank deltas inside a single batched forward pass, so N adapters cost ~1 GPU instead of N.
• A LoRA adapter is tiny: rank 16 on q/k/v/o of a 7B model is ~16.8M params (~33 MB in fp16), about 0.24% of the base model.
• The enabling kernel is SGMV (segmented gather matrix-vector, from Punica); S-LoRA adds paged adapter memory so the adapter pool and KV cache share one allocator.
• The real limits are max_loras (adapters per batch), rank padding, adapter swap thrash from CPU/host memory, and LoRA on embed_tokens/lm_head, which is 10-50x larger than attention-only LoRA.
• Merge weights and serve dedicated replicas only when a handful of adapters carry sustained high traffic. For long-tail multi-tenancy, multi-LoRA wins outright.

Why can't you just merge LoRA weights into the base model?

You can — and it's the right call for exactly one adapter. Merging computes W' = W + BA once, giving you a plain model with zero inference overhead. The cost is that W' is a full-size weight tensor. Ten merged adapters means ten 14 GB checkpoints, ten model replicas, ten KV cache pools, and ten idle GPUs whenever tenant traffic is bursty and uncorrelated — which it always is.

Worse, merging destroys batching across tenants. Requests for adapter A and adapter B can no longer share a forward pass, so you lose the one thing that makes GPU inference economical.

Multi-LoRA serving keeps the factorization unmerged and pays a small compute tax per token to preserve batch fusion:

y = xW  +  (x A_i) B_i * (alpha / r)
    ^^^^     ^^^^^^^^^^^^^^^^^^^^^^^^
   shared     per-request, adapter i

The xW term is one big GEMM across the whole batch. The delta term is many tiny GEMMs, one per distinct adapter in the batch.

How much memory does one LoRA adapter actually cost?

Compute it, don't guess. For a matrix of shape d_out x d_in, a rank-r adapter stores A: r x d_in and B: d_out x r, so r * (d_in + d_out) params.

def lora_bytes(hidden=4096, layers=32, rank=16,
               targets=("q_proj", "k_proj", "v_proj", "o_proj"),
               dtype_bytes=2):
    # Llama-2-7B geometry: all four attention projections are 4096x4096
    per_matrix = rank * (hidden + hidden)
    total = per_matrix * len(targets) * layers
    return total, total * dtype_bytes

params, nbytes = lora_bytes()
print(f"{params/1e6:.1f}M params, {nbytes/2**20:.1f} MiB")
# 16.8M params, 32.0 MiB

Thirty-two mebibytes. A hundred of them is 3.2 GiB — less than a quarter of the base model, and it fits alongside base weights and KV cache on a single 80 GB card with room to spare.

Now change one line. Add gate_proj, up_proj, down_proj (MLP, d_ff = 11008 on this model) and the same rank-16 adapter jumps to roughly 4x the size. Add embed_tokens and lm_head with a 128k vocab and you're adding r * (vocab + hidden) per side — a single adapter can cross 500 MB. Teams enable target_modules: all-linear in PEFT out of habit, then wonder why only eight adapters fit.

Rule: attention-only LoRA is what makes multi-tenant serving cheap. Every extra target module is a direct multiplier on how many tenants share a GPU.

How does one batch run 30 different adapters at once?

Naively you'd loop: split the batch by adapter, run a small GEMM per group, concatenate. That's 30 kernel launches of shape [few tokens] x [4096] x [16] — catastrophically launch-bound during decode, where each request contributes exactly one token.

Punica's SGMV kernel fixes this. It takes the full batch, an index vector mapping each row to an adapter id, and a stacked adapter tensor, then does a segmented gather-and-multiply in a single launch. All adapters run concurrently; there is no per-adapter Python loop and no padding of the token dimension.

Two properties follow, and both are counterintuitive:

1. Adapter count barely affects latency. Serving 1 adapter vs 32 adapters in a batch runs the same kernel with a different index vector. What costs you is reading 32 adapters' weights from HBM — bandwidth, not FLOPs.
2. Rank barely affects latency either, until it doesn't. Rank 8 vs rank 32 changes a [B, 4096] x [4096, r] GEMM that is memory-bound on the adapter weights. Doubling rank doubles the bytes read, but from a base of ~0.2% of the model.

The decode-phase overhead of multi-LoRA is real but small — think single-digit to low-double-digit percent throughput reduction versus the bare base model in typical serving configs, not 2x. Prefill dilutes it further, since prefill is compute-bound and the delta GEMM is a rounding error against a 4096-token xW.

What does a working multi-LoRA config look like?

vLLM is the reference implementation. The knobs that matter:

vllm serve meta-llama/Llama-2-7b-hf \
  --enable-lora \
  --max-loras 32 \            # distinct adapters allowed in ONE batch
  --max-lora-rank 16 \        # all adapters padded to this rank
  --max-cpu-loras 256 \       # host-side adapter pool (swap space)
  --max-model-len 8192 \
  --lora-modules \
      acme=/adapters/acme \
      globex=/adapters/globex \
      initech=/adapters/initech

Requests select an adapter by naming it as the model:

from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="x")

resp = client.chat.completions.create(
    model="acme",                      # <- adapter id, not the base model
    messages=[{"role": "user", "content": "Summarize ticket #4412."}],
)

Set VLLM_ALLOW_RUNTIME_LORA_UPDATING=1 and you can register new adapters against a live server via /v1/load_lora_adapter — which is the actual reason this architecture matters for a product. Onboarding a new tenant's fine-tune becomes an HTTP call, not a deploy.

Where does multi-LoRA serving actually break?

Four places, in rough order of how often they bite.

1. max_loras thrash. max_loras bounds distinct adapters per batch, not total adapters. If 64 tenants are concurrently active and max_loras=8, the scheduler can only co-batch 8 adapters' requests per step. The rest wait. Your GPU stays busy, your p99 latency triples, and no metric says "LoRA." Watch queue depth broken down by adapter, not in aggregate.

2. Rank padding. max_lora_rank is a static allocation. One tenant who trained at rank 64 forces every rank-8 adapter to occupy a rank-64 slot in the padded buffer, cutting how many fit in the GPU pool by 8x. Standardize rank across tenants, or run a separate deployment for the outliers. This is a policy decision disguised as a config flag.

3. Cold swap from CPU. Adapters beyond the GPU pool live in pinned host memory and stream in over PCIe on demand. 32 MB over PCIe Gen4 is sub-millisecond, so a warm long tail is fine. A cold tail — thousands of adapters read from disk or object storage on first request — is not. Prefetch on session start, and route requests so the same adapter lands on the same replica. Adapter-affinity routing (hash tenant id to replica) is the single highest-leverage change for a large tenant fleet, because it turns a global working set into a per-replica one.

4. Quantized base with fp16 adapters. Serving an AWQ or GPTQ base alongside fp16 LoRAs means the delta path runs in a different dtype than the base path. It works, but the adapter GEMM no longer rides the quantized kernel's fast path, and the overhead percentage climbs — because you shrank the denominator. If you quantize aggressively for cost, measure multi-LoRA overhead again; the number from your fp16 benchmark does not transfer.

When should you not use multi-LoRA?

Three cases. When one adapter serves nearly all traffic — merge it, serve it dedicated, skip the delta path entirely. When adapters differ in ways LoRA can't express (different vocab, different context length, different base model) — they aren't adapters of the same base, so nothing shares. And when you're on a closed API.

That last one matters more than it sounds. Claude Opus 4.x, Claude Sonnet 4.x, and GPT-5.x don't expose LoRA at all. The closest analog for per-tenant customization is a cached system prefix: a stable, tenant-specific block at the head of the prompt, marked for prompt caching. You get behavioral specialization and amortized prefill cost, but it consumes context and can't shift weights. If your specialization is stylistic or knowledge-shaped, the cached prefix is fine and cheaper to operate. If it's a genuine capability shift — a domain the base model handles poorly — you need weights, which means open models and this whole serving stack.

The summary

Multi-LoRA serving fits 100 adapters on one GPU because a LoRA adapter is a rounding error next to base weights — roughly 33 MB for rank 16 on a 7B model's attention projections, against 14 GB of base — and because SGMV-style kernels apply per-request low-rank deltas inside a single fused batch instead of splitting the batch per adapter. You keep one weight copy, one KV cache pool, and one scheduler. The costs are bounded and predictable: a modest per-token overhead, plus hard limits from max_loras, rank padding, and adapter swap latency. Keep LoRA on attention projections only, standardize rank across tenants, route by adapter affinity, and the economics of per-customer fine-tuning stop being absurd.