DEV Community: Pujun Lun

When "Just Enable the CUDA Feature" Met Two Free GPUs

Pujun Lun — Sat, 31 Oct 2026 21:51:02 +0000

If a Rust LLM inference engine already runs on CPU and Apple Metal, and its tensor framework supports CUDA, how hard could it be to run it on NVIDIA GPUs?

I expected a simple recompile. Instead, moving mini-vllm-rs, written in Rust and built on top of Candle, from my M2 Mac mini to Linux and CUDA exposed two separate F16 failures on the GPU, a profiler return value that broke my scope guards, a Mac-shaped assumption in my CLI, and a CPU optimization that had been silently leaking onto the GPU.

My motivation for the move was simple: Kaggle Notebooks give you two free NVIDIA Tesla T4 GPUs. On my Mac mini, heterogeneous serving was limited to one CPU worker and one Metal GPU worker (see my previous post). With a dual-T4 notebook, I could test two GPU workers concurrently, or combine them with a CPU worker into a larger heterogeneous system.

Before I could test three workers together, however, a single CUDA worker had to pass my end-to-end smoke test.

F16 failed twice on CUDA: first loudly, then silently

The smoke test requests F16 activations. On macOS, Candle's quantized Metal matmul does not support F16 activations, so GPU inference fell back to F32 while CPU inference used F16. On CUDA, I initially let F16 run through.

It failed during model loading with unsupported dtype for dequantize F32.

Even in a 4-bit GGUF model, small tensors such as normalization weights are typically left unquantized, so their stored type may be F32, F16, or BF16 depending on the source model and conversion.

When converting weights to match F16 activations, Candle 0.11.0 takes different paths by device: non-CUDA backends dequantize to F32 and cast to F16, whereas CUDA dispatches to a specialized kernel table that handles a subset of quantized formats and rejects the rest, including unquantized F32 weights.

I simply added a check to route F32 weights through the generic dequantize-then-cast path. That got the model past initialization, and Qwen2.5 0.5B started producing the right results. Hooray! That was the first victory.

However, when I tried a larger model, Qwen2.5 7B, it failed the smoke test in a worse way: F16 activations produced corrupted text on CUDA. The prompt asked why leaves change color in autumn, and the output was:

![](https://i.imgur.com!v/6QZ2p.jpg) ![](!https://i!imgur.com!v!/5Xt4K.jpg) In!!! the!!!!!! autumn! season!! !,!!!!!!!! leaves!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !change!!!!!! !!their!! !!colors!! !from!! !green!! !!to!!!! !!!yellow! !,! !!orange! !,! !and!!! !red! !. !!This!! !is!!!!!!!!! !!!!!!!!!a!!! !wonder!! !!of!!!!! !!!!nature!! !!!that!! !!happens!!!!! !!!!!!!!!!! !!!!!!!!! !!!! !! !! !! [...]

Beneath the exclamation marks, the model was still emitting relevant words, which pointed to numerical errors rather than a broken tokenizer or weight mapping. When I fell back to F32 activations on CUDA, the output returned to normal immediately, and both Llama 3.1 8B and Llama 3.2 1B worked out of the box as well.

Given more time to evolve this engine, I would dig further into that numerical issue and test more data formats. For now, scoping CUDA GPU backends to fall back F32 with a warning got end-to-end inference working reliably, while F16 is still usable on the CPU backend.

Why my profiler ranges popped as soon as they were pushed

On CPU and Metal, specifying --trace-directory exports Chrome traces for Perfetto. On CUDA, Chrome traces show host-side scopes, but not GPU activity such as kernel execution.

To see both without duplicating instrumentation, I hooked NVTX ranges into the same model, layer, attn, mlp, and qmatmul trace scopes, using an RAII guard to end each range automatically.

In the first capture, NVTX labels such as qmatmul operation=attention-query appeared on the timeline, but each range lasted only up to 600 nanoseconds, and even the longest ones never covered the corresponding CUDA kernel launch call.

My first suspicion was that Candle handed the CUDA API work off to another host thread, so the launches were happening outside the thread-local NVTX range. My coding agent chased the QMatMul call path into Candle, but found no evidence of a thread handoff.

The agent next suggested making the thread sleep briefly after the range starts, to see whether the range would stretch to cover the CUDA dispatch.

Instead of trying that, my intuition told me something was wrong with our RAII guard: the range always closed right before the CUDA dispatch, which looked too deterministic to be a race condition. So I went back to check our code:

impl NvtxRangeGuard {
    fn enter(name: &str) -> Option<Self> {
        (cudarc::nvtx::result::range_push(name) >= 0).then_some(Self)
    }
}

impl Drop for NvtxRangeGuard {
    fn drop(&mut self) {
        cudarc::nvtx::result::range_pop();
    }
}

I asked the agent: would enter() ever return None? Checking the CUDA Profiler documentation turned up this warning:

Some of the NVTX functions are defined to have return values. For example, the nvtxRangeStart() function returns a unique range identifier and nvtxRangePush() function outputs the current stack level. It is recommended not to use the returned values as part of conditional code in the instrumented application. The returned values can differ between various implementations of the NVTX library and, consequently, having added dependencies on the return values might work with one tool, but may fail with another.

The agent had written that check, and I had missed it during review. When we logged the return value under Nsight Systems, it was -1 on every call while still recording the range, matching reports from other developers.

Why did that return value produce a tiny range instead of leaving the range open forever?

range_push() recorded the start of the range in Nsight Systems and returned -1.
Because the .then_some(...) helper evaluates its argument eagerly, it constructed the NvtxRangeGuard before checking whether range_push(name) >= 0 was true.
Because -1 >= 0 was false, .then_some(...) discarded that newly created guard and returned None, running Drop immediately inside enter() and calling range_pop() a few hundred nanoseconds after the push.

The range wasn't ending early because of async GPU work. It was being pushed and popped inside enter() before the caller even received the guard.

As the agent put it in its retrospective:

I treated the push return value as a success flag: nonnegative meant “push succeeded, so create a guard”; negative meant “don’t create one.” I hadn’t verified that assumption against the NVTX implementation we were using. It was a mistake to make range behavior depend on that value.

It is a mistake I could easily have made myself, and a good reminder to be careful with external APIs.

Keeping the guard alive regardless of the return value and assigning a distinct color to each trace scope produced the nested hierarchy below: the blue model span breaks down into orange layer blocks, green attn and mlp phases, and individual qmatmul and attention operations right above the CUDA API launches:

From "mixed" mode to one CPU and two GPUs

Alongside the CUDA fixes, I made a few other changes for the multi-GPU environment:

Make --inference-device flag repeatable: My old mixed inference mode assumed one CPU plus Metal GPU 0, each managing their own KV cache and scheduler. Replacing it with a repeatable --inference-device flag lets each gpu entry claim the next accelerator index (0, 1, and so on), supporting one CPU plus multiple GPU workers without changing the orchestrator.
Fix a leaked CPU optimization: While chasing down CUDA dispatch, my coding agent noticed that the contiguous attention path enabled a CPU grouped-query matmul layout by default without checking whether the tensor actually lived on the CPU, so this leaked onto the GPU path.
Fix model cache: In my Kaggle sessions, the model downloader used Hugging Face Hub’s default cache path (~/.cache/huggingface/hub), which wasn’t retained between sessions. I updated it to honor the environment variable HF_HOME, to choose where the cache lives.

Once those were in place, I ran a smoke test sending 3 requests to a heterogeneous setup of one CPU worker and two GPU workers, with each worker running a copy of Qwen2.5 0.5B and round-robin routing sending one request to each backend:

Backend	Input tokens	Output tokens	Prefill throughput (tok/s)	Decode throughput (tok/s)
GPU `0`	1,068	873	2,067.39	14.99
GPU `1`	998	1,024	2,485.12	15.35
CPU	1,047	1,024	3.94	2.87

The CPU backend was definitely much slower than the GPUs, as expected, but all three workers processed their requests concurrently end to end.

What I learned

Moving from a Mac mini to dual T4s tested every boundary I had drawn earlier. The process split and per-backend threads scaled cleanly to a larger heterogeneous system, while the bugs showed where single-device and Metal-specific assumptions had been hiding in plain sight.

There are definitely more optimizations to do. For example, when the paged KV cache is used, my current attention implementation concatenates cache pages before self-attention, which was tolerable on my Mac mini's unified memory but more significantly hurt decode performance on the T4s (smoke-tested with Qwen2.5 7B):

KV cache	Input tokens	Output tokens	Prefill throughput (tok/s)	Decode throughput (tok/s)
Contiguous	128	421	482.1	63.0
Paged	128	479	501.9	38.6

Dedicated GPU memory also opens the door to larger models: on my Mac mini the CPU and GPU share 16 GB of RAM, whereas each T4 GPU has its own 16 GB of VRAM. The orchestration and scheduling algorithms will need to be recalibrated for multi-GPU setups as well.

There is always more work to do, but this marks a steady first step for my inference engine in proving its cross-platform support.

What I Gave Up to Make Speculative Decoding Faster

Pujun Lun — Wed, 28 Oct 2026 00:29:36 +0000

On a creative-writing prompt, my target LLM rejected most tokens proposed by the smaller draft model. Speculative decoding was slower than running the target alone. I wanted to recover that wasted work. How much faster could generation become if I let the draft have more influence over the output?

Lossless speculative sampling preserves the target model’s output distribution. Relaxing that requirement might improve acceptance, but the resulting text would need a closer look.

I explored this tradeoff with mentored decoding in mini-vllm-rs, my Rust inference engine for macOS. Using quantized Qwen2.5 7B as the target and 0.5B as the draft on an Apple M2, I eventually found a faster configuration. Deciding when I would use it turned out to be a different question.

A knob for accepting more draft tokens

In an earlier benchmark, a code-refactoring prompt worked well under greedy decoding. A policy that adaptively adjusted the draft token count usually selected 8-token proposals, achieved 98.5% acceptance, and raised decode throughput by 38.3% over target-only. However, on the creative-writing prompt, the same policy delivered 36.8% lower decode throughput than target-only, also under greedy decoding.

I then explored the creative-writing prompt with multinomial sampling at temperature 0.7 (sampling benchmark). Lossless speculative sampling with a fixed draft token count of 4 achieved only 28.6% acceptance.

Mentored decoding allows a bounded shift from the target distribution toward the draft. I implemented its linear-interpolation construction.

Let $p$ be the target distribution and $q$ the draft distribution, both after temperature and repetition penalties. Their total-variation distance is:

d = TV (p, q) = \frac{1}{2} x \sum ∣ p (x) - q (x) ∣

Given a TV budget $D$ , the interpolation weight is:

θ = min (1, D / d)

If $d = 0$ , the distributions match exactly, and we take $θ = 1$ . The resulting output distribution is:

π = (1 - θ) p + θq

A draft token $x$ sampled from $q$ is accepted with probability:

a (x) = min (1, \frac{π ( x )}{q ( x )})

After a rejection, the sampler draws a replacement from the normalized positive residual $max (p - q, 0)$ . Together, acceptance and correction produce the distribution $π$ .

At TV budget 0, this preserves the target distribution. At TV budget 1, the budget covers every possible TV distance, so $θ = 1$ and $π = q$ . Every draft token is then accepted.

Intermediate budgets limit the per-step distributional deviation from the target. They do not measure how much semantic quality the output loses.

Even full acceptance did not buy a speedup

I first set the TV budget to 1, so every draft token was accepted. Even then, fixed-4 generation took 24.425 seconds, versus 20.971 seconds for target-only.

Rejection was gone, but execution cost remained. Each round still generated 4 draft tokens autoregressively, verified their positions with the target, sampled, and updated both models’ cache state. My implementation emits no extra target bonus token after a fully accepted proposal, so a successful round produced only 4 output tokens.

The draft-only control took 3.372 seconds. At TV=1, target verification was an expensive way to follow the draft distribution. That endpoint helped diagnose the remaining cost, but I wanted a speedup while retaining more of the target distribution.

Finding the break-even point

To find where speculation could beat target-only E2E latency, I tested fixed draft token counts of 6 and 8 at full acceptance. I then lowered the TV budget to find the smallest tested value that still produced a clear speedup (benchmark report).

The prompt and temperature stayed fixed. Every request generated 256 tokens with EOS ignored. After warm-up, each setting had 3 independent untraced timing invocations, each covering 3 random seeds.

Configuration	Acceptance	Avg. rounds/request	Median E2E latency
Target-only	—	—	20.971 s
Fixed-4, TV=1	100%	64	24.425 s
Fixed-6, TV=1	100%	43	20.587 s
Fixed-8, TV=1	100%	32	18.508 s
Fixed-8, TV=0.75	93.4%	34	18.820 s
Fixed-8, TV=0.5	63.5%	46	22.900 s

Fixed-6 was approximately equal to target-only within observed variability. Fixed-8 clearly crossed the baseline. At full acceptance, it needed 32 verification rounds per request instead of fixed-4’s 64, allowing more output to share each verification call.

Verification also crossed a kernel-dispatch boundary, as in my continuous-batching experiments. Here, 4-row verification used row-wise GEMV, while full 6- and 8-row proposals used regular quantized matmul. Proposal length and kernel dispatch changed together, so these measurements do not isolate their individual contributions.

What did I get in exchange?

For fixed-8, increasing the budget from 0.5 to 0.75 reduced average verification rounds from 46 to 34. Generation moved from 9.2% slower than target-only to 10.3% sooner. Each seed’s median improved by roughly 8–10% at TV=0.75. To judge what that speedup bought me, I inspected the creative-writing output.

The prompt places Mara, a mechanic, and Ilyan, an archivist, in a lunar greenhouse abandoned for 20 years to investigate a distress signal. It specifies fresh condensation despite the lack of power, and patches of blue light reflected from Earth.

In one target-only sample, the characters treat the condensation as evidence that someone may have visited or something is changing. In the TV=0.75 output using the same seed, the roof reveals:

“a thin sliver of blue light streaming through—Earth’s infrared, indicating that it was not the end of this system but only an interruption in its normal operation.”

Infrared is not visible blue light. The passage also jumps from that light to a diagnosis of the system without explaining the connection.

At TV=1, the same seed produced an identity error. Immediately after Mara explained the greenhouse, the text continued:

“I'm Ilyan Sato, the archivist from the station,” she added.

Mara had taken the other character’s name and role.

The complete prompt and samples show different adherence or continuity problems across all 3 TV=0.75 outputs. These are qualitative observations from 1 prompt, and the 256-token cutoff limits judgments about plot development.

What I learned

I started by trying to recover rejected draft work. Raising acceptance helped, but even 100% acceptance was insufficient with 4-token proposals. Longer proposals finally crossed the latency baseline, and reducing the TV budget showed that complete acceptance was unnecessary.

The faster setting also changed the output in ways that the timing table could not capture. The TV budget controlled a mathematical distance, while the text exposed physical inconsistencies and broken reasoning. What I gave up was the guarantee of preserving the target distribution. Whether the time saved justified that change required reading the result, not just measuring how quickly it arrived.

When LLM Inference on CPU Beats Waiting for the GPU

Pujun Lun — Sat, 24 Oct 2026 05:52:05 +0000

mini-vllm-rs is a single-node LLM inference engine I built in Rust on top of Candle for macOS. It can run CPU and Metal GPU backends concurrently, routing whole requests between them. My first request routing benchmark favored sending almost everything to GPU: the fastest policies assigned it 11 of 12 requests.

That left a more useful question unanswered: could selective CPU offload improve completion time compared with serving the entire workload on GPU alone? Separating prefill from decode suggested where to look: requests with warm prefixes, arriving behind a growing GPU backlog.

A follow-up experiment tested that idea. Routing 1 of 8 requests to CPU reduced the burst's completion time by 9.8% compared with forcing all 8 onto GPU. With 48 requests, offloading 6 to CPU increased aggregate output throughput by 12.4%.

The phase gap suggested the next experiment

The experiments used a quantized Qwen2.5 0.5B model with F32 activations on an Apple M2. For a 512-input/128-output request, isolated backend profiles showed:

Phase	CPU	Metal GPU	GPU advantage
Prefill	73 tok/s	2,551 tok/s	About 35×
Decode	45 tok/s	293 tok/s	About 6.5×

A long, uncached prompt exposed the CPU's largest disadvantage. Even when GPU already had work, avoiding CPU prefill could justify waiting for it. A warm prefix could remove most of that cost, leaving the smaller decode disadvantage for GPU backlog to outweigh.

Each backend owns its own scheduler and KV cache, and a request stays on the backend selected for it. I seeded an approximately 512-token prefix into both caches. Each measured request used the same 516-token prompt and generated exactly 128 tokens. Warming both sides gave CPU a chance to avoid expensive prefill.

I compared forced-GPU routing against mixed routing, where the router chose between CPU and Metal using estimated pending and incoming work. Both cases used the same scheduler settings, with up to 4 active requests per backend. Requests arrived 50 ms apart without waiting for earlier responses. Each case started with a fresh server and seeded caches, and ran 3 times.

What changed between requests 7 and 8?

The router priced prefill and decode separately:

score = remaining prefill tokens      / prefill rate
      + remaining decode-token budget / decode rate

The totals included existing work plus the incoming request. Decode work was approximated using each existing request's unused max_new_tokens budget and the incoming request's full budget, so the router did not predict actual output lengths or early stopping. The rates came from isolated backend profiles, making this a work-based ranking rather than an exact prediction of batched completion time.

The candidate diagnostic logs show the boundary in run 1:

Incoming request	CPU active / queued	GPU active / queued	CPU score	GPU score	Selection
7	0 / 0	4 / 2	3.118 s	3.027 s	GPU
8	0 / 0	4 / 3	3.118 s	3.444 s	CPU

Estimated routing costs from mixed-routing run 1. All 3 runs made the same choices, with slightly different GPU scores.

Both candidates advertised a 496-token reusable prefix for the incoming 516-token prompt. The idle CPU therefore had only 20 estimated prefill tokens and a 128-token decode budget to account for. Its score stayed at 20/73 + 128/45 = 3.118 seconds.

At request 7, GPU already had 4 active requests and 2 queued requests. Each queued prompt also carried a 496-token cache estimate, leaving 20 prefill tokens apiece. Including the incoming request, GPU's estimated prefill work was therefore 2 × 20 + 20 = 60 tokens. Its decode budget was 256 queued + 496 active + 128 incoming = 880 tokens. The score was 60/2551 + 880/293 = 3.027 seconds, still below CPU's. Occupied GPU slots alone were not enough to justify offload.

About 50 ms later, request 8 reached the router. Request 7 had joined the GPU queue, while the active requests’ published remaining decode budget had fallen by 8 tokens. Including the incoming request, the GPU now had 80 estimated prefill tokens and a combined decode budget of 384 queued + 488 active + 128 incoming = 1,000 tokens. Its score rose to 3.444 seconds, above the idle CPU’s unchanged 3.118 seconds, so the router chose CPU. The warm prefixes kept prefill work small, and the accumulated decode work tipped the decision.

While helping me organize the earlier benchmark data, AI surfaced an accounting bug: queued requests lost their cache discount, inflating GPU prefill work. I decided to fix it and rerun the benchmark. The first CPU assignment moved from request 7 to request 8, and the capacity results here use the corrected runs. Queued requests now retain their routing-time cache estimates, though actual reuse may decrease or increase as the cache changes before execution.

Output-length estimation remains unfinished. For comparison, SGLang's scheduler scales remaining token budgets with an adjustable new_token_ratio to estimate future KV-cache demand for running requests, a heuristic rather than an exact output-length prediction.

Did the decision help in practice?

Run 1 produced the following measurements for request 8 and the burst as a whole:

Measured outcome	Forced-GPU control	Mixed routing
Request 8: time to first token	5.561 s	0.339 s
Request 8: end-to-end latency	11.116 s	3.681 s
Whole-burst completion time	11.486 s	10.361 s

Whole-burst time starts at the first submission, while request latency starts at that request's submission.

The offloaded request finished well before the mixed burst ended. CPU could start useful work while GPU continued serving its assigned requests, and the entire burst also finished sooner. Across all 3 runs, mean completion time fell from 11.47 to 10.35 seconds, a 9.8% reduction.

Mixed routing assigned 0, 1, and 6 requests to CPU, respectively. Wall time excludes startup and prefix seeding.

The benefit continued under a longer GPU backlog. In a 48-request burst, CPU received just 6 requests in every run. Mean completion time fell from 68.43 to 60.88 seconds. With the same total output, this 11.0% reduction in completion time translated into 12.4% higher aggregate output throughput.

Cells show routing estimates, not measured latencies.

Latency gains were not uniform. In the 8-request case, median TTFT improved, but p95 TTFT increased by 3.5%. Under the longer 48-request backlog, both median and p95 TTFT improved, alongside median and p95 end-to-end latency. Selective offload helped capacity without guaranteeing a win for every latency statistic.

A GPU concurrency sweep remains the next baseline check, since these gains were measured against the fixed 4-active-request GPU configuration. Sustained arrivals and varied cache-hit rates would test how far the result carries.

What the experiments changed

The first benchmark suggested where to look: the CPU’s disadvantage was much smaller in decode than in prefill. The follow-up showed that warm prefixes and GPU backlog could make selective CPU offload useful. Backend speed alone did not determine placement: the router also needed to account for the work already waiting and the prefill that cache reuse could avoid. For a broader discussion of the tradeoffs behind heterogeneous execution, see When Does Heterogeneous Inference Pay Off?.

The benchmark report contains the complete aggregate measurements and reproduction details.

Why My Prefix Cache Missed Identical Requests

Pujun Lun — Fri, 16 Oct 2026 18:44:13 +0000

I enabled prefix caching in mini-vllm-rs, my Rust LLM inference engine for macOS, and submitted two identical requests concurrently to a cold cache.

Neither reused a single token. Both computed the same prompt.

Why? My cache published reusable pages only when a request finished. Both requests arrived before either had published anything, so neither found a prefix to reuse. Later requests reused completed prefixes correctly.

Supporting concurrent reuse required more decisions: when should cached pages become visible, should another request wait for them, and when should that wait end?

Benchmarks led me to add incremental publication and single-flight coordination. Deciding when to release waiting requests then exposed another problem: an increasingly complicated AI-generated state machine.

A warm-cache win left a cold-cache problem

I tested a 4-bit quantized Qwen2.5 0.5B model on my Mac mini's Metal GPU, with 16-token cache pages and 3 repeated runs. My first reuse benchmark compared warm reuse with simultaneous cold arrivals:

Workload	Result
Reuse a warm document prefix	Restored 880 of 904 input tokens, reducing time to first token (TTFT) by 96.2%.
4 identical requests arriving to a cold cache	All restored zero tokens and performed full prefill.

That cold-cache result was the problem I encountered at the start of this article.

Publish pages while their owner is still generating

I added incremental publication. After each successful batched model forward, the engine immediately publishes any newly completed pages containing committed tokens, so they become available for reuse before this request finishes.

To test this change, I started one request and waited until it returned its first output token. While it continued generating the rest of its 128-token output, I submitted more requests with the same prompt to see whether they could reuse the cached prefix.

Incremental publication let each follower restore 480 of its 493 prompt tokens. Across groups of 2, 4, and 8 concurrent requests, mean TTFT fell by 48–93% compared with completion-only publication. That average includes every request's wait from submission to first streamed output. For detailed measurements, see the follow-up benchmark report.

However, incremental publication did not eliminate duplicate work for simultaneous cold arrivals. A request can reuse complete prefix pages already published when it starts execution, even if the other request is still prefilling. But it may find only part of the prompt, or nothing yet, and compute the rest itself. Reuse therefore depends on execution order and publication timing.

Avoiding that race required deciding who should compute the prefix.

Let one request compute the prefix, but how long should others wait?

I added exact-input single flight. One request becomes the leader, while other requests with identical input tokens wait as followers. This coordinates prompt computation, not answers. Every follower still generates its own output, as expected: identical prompts can use different generation settings, such as temperature.

My initial implementation made followers wait until the leader's entire request finished. The cache reuse became predictable, but the policy also made them wait through decode.

With 8 simultaneous requests each generating a 128-token output, this policy reduced both mean TTFT and the slowest request's completion latency compared with independent execution. But with only 2 such requests, mean TTFT became worse than independent execution, even though completion latency improved. Avoiding duplicate computation had introduced a new waiting cost.

Why wait through decode when the prompt cache was already ready? I added a second release policy. Once the leader finishes prefill, the engine publishes its completed prefix pages, then makes followers eligible for admission:

sequenceDiagram
    participant S as Scheduler
    participant C as Prefix cache
    participant L as Leader
    participant F as Followers
    S->>L: Admit leader
    S->>S: Block identical followers
    L->>L: Finish prompt prefill
    L->>C: Publish completed prefix pages
    L->>S: Report successful publication
    S->>S: Unblock followers
    S->>F: Admit as capacity allows
    F->>C: Look up matching prefix
    C-->>F: Return shared KV page references
    par Independent generation
        L->>L: Continue decode
    and Follower generation
        F->>F: Generate their own outputs
    end

For simultaneous identical requests generating 128 output tokens each, both policies restored the same number of cached tokens, while releasing followers after prefill reduced mean TTFT:

Concurrent requests	Release at request completion	Release after prefill	TTFT reduction
2	1,357.5 ms	584.7 ms	56.9%
4	1,757.9 ms	617.1 ms	64.9%
8	1,984.4 ms	646.7 ms	67.4%

To compare both policies with independent execution, the table below focuses on the 8-request burst. Each request had a 532-token prompt and a 128-token output:

Strategy	Prompt tokens restored per follower	Mean TTFT change	Slowest-request completion-time change
Independent execution, publish at completion	0	Baseline	Baseline
Single flight, release at request completion	528	47.5% lower	25.8% lower
Single flight, release after prompt publication	528	82.9% lower	32.4% lower

Regular incremental publication is disabled in all rows. The early-release policy explicitly publishes the prompt prefix when prefill finishes.

However, earlier release did not improve every metric. In a separate test, a second request arrived right after the original request entered decode phase. Compared with release-at-completion single flight, early release reduced mean TTFT by 72%, but increased the slowest request's completion latency by 14%.

Why? The original flight had already closed, so the second request started a new flight and reused the warm prefix. The two requests then decoded concurrently rather than sequentially, changing the active batch. That may have contributed to the longer completion time.

The earlier-release policy publishes the required checkpoint itself. Extra incremental publications added little for these coordinated exact-input workloads. They remain useful without coordination, including requests that share only part of a prefix.

Publication decides when data becomes reusable. Coordination decides who waits, and for how long.

What this small implementation leaves open

These are synthetic workloads on one backend with a small model. Exact-input coordination showed no meaningful completion-time regression for distinct inputs in this control, but the benchmark does not establish behavior under production traffic. Some future directions are:

Partial-prefix coordination. Requests may share a long document but end with different questions. How much overlap justifies waiting?
Wait or recompute. A follower's latency goal and the leader's remaining prefill can change that choice.
Different attention states. Sliding-window and hybrid models change retention and reuse. vLLM's hybrid KV cache manager explores those constraints.
Reuse across backends and memory tiers. Routing must weigh prefix locality against speed and load. SGLang HiCache extends caching across GPU, host, and external storage.

The release change also exposed a design problem in the coordinator itself.

Fixing the error, then questioning the state machine

AI helped implement these features. As in my first article, reviewing its designs was part of building a system I could understand and maintain.

Initially, AI deleted a flight when it released the followers, which seemed reasonable because the waiting had ended.

But the first single-flight benchmark exposed a problem: followers later completed and notified the coordinator that no longer recognized their IDs. AI proposed removing the unknown-request check.

I rejected that fix because it would also hide termination notifications carrying genuinely invalid request IDs. The original design had no FlightState. To preserve validation, we added Active and Draining states and tracked released followers until they terminated. Here is that first revision, with methods omitted:

enum FlightState {
    Active { leader_request_id: u64 },
    Draining,
}

struct Flight {
    input_token_ids: Arc<[u32]>,
    state: FlightState,
    follower_request_ids: VecDeque<u64>,
}

The leader had already finished, so Draining tracked only the released followers.

Earlier release made that representation more complicated. The leader could now still be decoding after followers were released. AI retained the same Flight fields but changed the enum:

enum FlightState {
    Active { leader_request_id: u64 },
    Draining { leader_request_id: Option<u64> },
}

The optional leader made me pause. Different release policies required different combinations. If a policy should not retain a leader but the state contained one, would we need another check?

I took a step back: did we really need a Draining state? We had added it to recognize released requests when they eventually terminated. That required remembering their IDs, but not preserving their former flight membership. Why not collect those IDs separately across flights?

I asked AI to separate those responsibilities. We removed FlightState and tracked released request IDs separately:

pub(super) struct PrefixCacheFlightCoordinator {
    flights: HashMap<FlightId, Flight>,
    released_request_ids: HashSet<u64>,
    // Other fields omitted.
}

On release, we remove the flight and its indexes, keeping only the IDs of requests that have not yet terminated. This preserves termination validation without retaining leader/follower relationships that no longer affect execution. Unknown request IDs still produce errors, and a new request with the same prompt starts a new flight.

AI preserved validation once I challenged the deletion. Later, it kept extending that solution until I questioned the abstraction itself.

Human review meant both preserving an invariant and reconsidering the structure used to enforce it. The benchmark showed when requests could stop waiting. Reviewing the code showed when the flight could stop existing.

Why Continuous Batching Made My LLM Inference Engine Slower

Pujun Lun — Sun, 11 Oct 2026 22:16:29 +0000

I had been adding optimizations to mini-vllm-rs, my Rust LLM inference engine for macOS, including prefix caching and speculative decoding. Continuous batching looked like the next easy win. Process requests together, reuse model weights across more tokens, and watch throughput climb. What could go wrong?

I tested two concurrent requests on my M2 Mac mini with 16 GB of unified memory, running a 4-bit quantized Qwen2.5 7B model. Each request generated 512 tokens, with speculative decoding disabled.

Guess what? Boom. Aggregate throughput dropped from 16.8 tokens/s unbatched to 8.7 tokens/s batched, almost cut in half.

Fixing that slowdown led to another surprise: the same kernel-selection threshold could also change generated text, even with greedy decoding.

Two rows, a different kernel

The clue was in Candle, the framework I use for backend execution. Its quantized Metal path contained this tiny decision:

// Candle 0.11.0: candle-core/src/quantized/metal.rs, line 372
if src_shape.dim(D::Minus2)? == 1 {
    return self.fwd_mv(self_shape, storage, layout);
}
// Line 414 calls into call_quantized_matmul_mm_t

Source. The condition checks the number of input rows. One row takes the GEMV path, multiplying the weights by one vector. More than one row continues into GEMM, which handles multiple input vectors together.

Packing two requests changed the input from one row to two. That was enough to switch kernels.

GEMM can reuse weights across rows, but this implementation's overhead outweighed that benefit at such a small batch. My scheduler was doing what I wanted. The backend was choosing a slower way to execute it.

Getting the speed back

I added a fallback: split small inputs into rows, run GEMV for each, then concatenate the results.

At a batch size of two, the fallback brought throughput back to 16.25 tokens/s, compared with an unbatched baseline of 16.15 tokens/s. Relief! But would GEMV still win with larger batches?

Concurrent requests	Unbatched	Batched GEMM	Batched row-wise GEMV	GEMV / GEMM throughput
3	17.41	13.05	17.52	134.3%
4	17.22	16.55	18.07	109.2%
5	17.44	19.92	18.06	90.7%

Aggregate output throughput in tokens/s.

At five requests, the ratio fell below 100%, so GEMM pulled ahead. Separate dispatches and concatenation may have become more expensive than one GEMM call. I chose four rows as the default switching threshold for this workload. See the benchmark report for details.

Problem solved? Not quite.

Same prompt, different story

Later, I tested speculative decoding with two requests, using Qwen2.5 0.5B to propose four tokens at a time. Their overall draft acceptance rate was 32.9% when processed one at a time, versus 37.1% when allowed to run concurrently. Same requests. Same models. Same proposal policy. Greedy decoding in both runs. Why?

A separate check made the numerical issue easier to see. I asked Qwen to write a science-fiction scene about a mechanic and an archivist investigating a distress signal in an abandoned greenhouse on the Moon. Both runs also used greedy decoding. I changed the threshold so that prefill used GEMM in one run and row-wise GEMV in the other.

The first 597 characters matched exactly. Then the continuations split:

Prefill kernel	First different continuation
GEMM	“Mara moved to the second door, where an emergency terminal stood…”
GEMV	“Mara’s flashlight beam cut through the gloom, illuminating rows of dry hydroponic beds…”

Both responses were coherent and followed the same characters and setting. Nothing looked like a completely broken model. That made small numerical differences a plausible suspect.

Tracing the speculative test gave me stronger evidence. At the first differing token, the drafter proposed the same token IDs in both runs, but two leading target scores had swapped order:

Execution	Token 476	Token 323	Selected token
Serial admission	25.571232	25.565775	476
Continuous batching	25.549618	25.562220	323

That tiny change was enough to flip the greedy choice. Every later token then followed a different prefix, which could change draft acceptance too.

The other request had already finished

Here was the part that caught me off guard: when the token choice flipped, the other request was already gone.

So why did its earlier presence still matter?

Here, rows count token positions, not requests. An earlier batch had combined 120 prefill tokens from the new request with four verification tokens from the existing request. Those 124 rows crossed my threshold and selected GEMM instead of GEMV, introducing small numerical differences into the cached state.

By the time the output diverged, the other request had finished, and the remaining request's four-token verification was back on the GEMV path. But switching back did not undo the differences already stored in its KV cache.

Looking only at the iteration where the text diverged would have missed the cause.

I forced all relevant inputs through GEMV in both execution modes. In the shortened comparison, output hashes and acceptance totals matched. This did not establish which kernel was more accurate, but it supported the explanation. The numerical report contains the investigation.

A batching decision changed the kernel selection, and its effects survived in cached state long after the batch changed again.

My longer article follows how a generation loop grew into a serving engine with clear boundaries between scheduling, model execution, and request state. It also explores how AI helped with architecture, coding, and performance optimization, and where I had to challenge its decisions.

From a Rust Inference Engine to Heterogeneous Serving

Pujun Lun — Fri, 09 Oct 2026 16:00:00 +0000

My background spans ML accelerator compilers and OS virtualization, and I now work on accelerator fleet efficiency.

To explore how compilers, runtimes, and hardware interact in LLM serving, I built mini-vllm-rs, a single-node inference engine written entirely in Rust for macOS. It supports continuous batching, paged KV caching, prefix caching, and speculative decoding. CPU and Metal GPU workers can run concurrently, each handling complete requests.

The project and two articles document that learning process (with more to come):

What building the engine taught me

To give you a glimpse of the problems I encountered, two requests generating 512 tokens each achieved an aggregate throughput of 16.8 output tokens/s when run sequentially on an Apple GPU. However, continuous batching reduced it to 8.7 tokens/s.

Why? Batching changed the matrix shape, making Candle, the framework I use for backend execution, switch from a GEMV kernel to a GEMM kernel. GEMM was slower for this small batch, so I added a GEMV fallback for small batches and benchmarked the crossover point to choose the switching threshold.

The first article follows how a simple generation loop grew into an engine with clear boundaries between scheduling, model execution, and request state. It examines how continuous batching required each request’s KV state to be managed independently of model execution, and how prefix caching complicated when that memory could be reused or released. Those boundaries later made adding concurrent CPU and GPU workers possible without a major redesign.

It also reflects on using AI to implement the engine, and the design reviews and experiments I needed to decide which changes to keep.

Why split work when a GPU can run the whole model?

In my project, I considered GPU prefill with CPU decode, or a GPU target model with a CPU drafter. Apple Silicon's shared memory looked helpful, but Candle's device-based memory model complicated cross-device state access. Synchronization and low request concurrency also made keeping both devices busy difficult. Hence, I implemented neither split.

Meanwhile, disaggregation is being explored at a much larger scale: NVIDIA has announced configurations that split inference work between Vera Rubin GPUs and Groq LPX, while AWS has announced prefill on Trainium paired with decode on Cerebras.

The second article asks when specialization can repay communication costs. In one simplified prefill/decode example, transferring 2 GiB of KV cache at 25 GB/s takes about 86 ms. Without compute–communication overlap, specialized accelerator decode must beat GPU decode by more than 0.67 ms/token for a 128-token output to repay that transfer. For a 512-token output, the threshold falls to 0.17 ms/token. The same accelerator could make a long answer faster but a short one slower.

Beyond this example, it compares prefill/decode, attention/FFN, and target/draft separation. It also asks why production might choose a slower path: more KV capacity, larger batches, and scaling each hardware pool independently can improve service capacity even when one request finishes later.

If you are coming to inference from another part of the stack, or mostly work with GPUs and want to understand the alternatives, you may be asking similar questions. I'm sharing the code, experiments, and articles as I learn, and I'd be interested to hear which trade-offs you've encountered.