GPUStack Day 0 Support for Kimi-K3: vLLM vs. SGLang Inference Benchmark on 8 B300 GPUs

This article documents the deployment and benchmarking of Kimi-K3 on a single server equipped with 8×NVIDIA B300 GPUs. It compares vLLM and SGLang under 64K and 200K long-context workloads. The key finding is straightforward: vLLM is faster at 64K, while SGLang with DCP overtakes it at 200K. The difference comes primarily from KV access during Decode—not from Prefill.

Kimi-K3 is a Mixture-of-Experts model designed for complex reasoning and long-context workloads. In this test, GPUStack was used to deploy the model on a single 8×NVIDIA B300 server with both vLLM and SGLang. Kimi-K3-DSpark was enabled as the draft model for speculative decoding.

The most important results are:

• 64K input, concurrency 10: vLLM completed the workload in 100.5 seconds versus 150.8 seconds for SGLang, making vLLM approximately 1.5× faster.
• 200K input, concurrency 10: SGLang completed the workload in 225.3 seconds versus 295.2 seconds for vLLM, making SGLang approximately 1.31× faster.
• From 64K to 200K, stable Decode throughput degraded by 3.29× on vLLM but only 1.25× on SGLang.
• The primary reason for the crossover is that SGLang ran with --dcp-size=8, while vLLM used decode_context_parallel_size=1.
• Prefill performance was broadly comparable; most of the gap appeared during Decode.
• The Random dataset produced extremely low DSPARK acceptance rates, so the speculative-decoding results should not be extrapolated directly to production traffic.

Important: This was not a strictly controlled A/B test. The two engines differed in DCP, memory allocation, kernel backends, and acceptance criteria. The results are most useful for understanding bottlenecks and deployment trade-offs—not for declaring an absolute winner.

Test Environment

Item	Configuration
Server	Single node
GPU	8×NVIDIA B300
Visible memory per GPU	267.69 GiB
Model	Kimi-K3 (MXFP4)
Draft model	Kimi-K3-DSpark
Inference engines	vLLM / SGLang
Input lengths	64K, 200K, and additional lengths
Output length	3K
Concurrency	10
Dataset	Random

Preparing the Model Weights

Download the Kimi-K3 main model and DSpark draft models from ModelScope.

Kimi-K3 Main Model (MXFP4)

modelscope download \
  --model moonshotai/Kimi-K3 \
  --local_dir ./Kimi-K3

DSpark Draft Model for vLLM

modelscope download \
  --model skyai/Kimi-K3-DSpark \
  --local_dir ./Kimi-K3-DSpark

DSpark Draft Model for SGLang

modelscope download \
  --model skyai/sglang-Kimi-K3-DSpark \
  --local_dir ./Kimi-K3-DSpark

Pulling the Inference Images

vLLM

docker pull vllm/vllm-openai:kimi-k3

SGLang

docker pull lmsysorg/sglang:kimi-k3

After pulling the images, add custom vLLM and SGLang versions under Inference Backends in GPUStack.

Deploying vLLM with GPUStack

Create a custom vLLM backend in GPUStack, select the Kimi-K3 model and all eight B300 GPUs, and configure the required environment variables.

The vLLM custom backend and model settings are shown below.

vLLM Environment Variables

VLLM_ENABLE_K3_LATENT_MOE_TAIL_FUSION=1
VLLM_ALLREDUCE_USE_FLASHINFER=1
VLLM_ENGINE_READY_TIMEOUT_S=3600
VLLM_USE_V2_MODEL_RUNNER=1
VLLM_USE_RUST_FRONTEND=1

vLLM Startup Parameters

--trust-remote-code
--load-format fastsafetensors
--moe-backend auto
--gpu-memory-utilization 0.95
--tensor-parallel-size 8
--max-model-len 1000000
--kv-cache-dtype fp8
--attention-config '{"mla_prefill_backend":"TRTLLM_RAGGED","use_prefill_query_quantization":true}'
--max-num-batched-tokens 32768
--enable-prefix-caching
--enable-auto-tool-choice
--tool-call-parser kimi_k3
--reasoning-parser kimi_k3
--max-num-seqs 32
--speculative-config '{"model":"Inferact/Kimi-K3-DSpark","num_speculative_tokens":7,"method":"dspark","attention_backend":"FLASHINFER_MLA","draft_sample_method":"probabilistic","rejection_sample_method":"block"}'

In this test, --gpu-memory-utilization 0.95 overcommitted the KV pool and caused runtime OOM retries. A safer configuration is discussed later.

Deploying SGLang with GPUStack

Create an SGLang custom backend, select all eight B300 GPUs, and configure the model and startup parameters.

The SGLang custom backend and model settings are shown below.

SGLang Startup Parameters

--trust-remote-code
--model-path <Kimi-K3 model path>
--tp-size 8
--dcp-size 8
--disable-custom-all-reduce
--enable-symm-mem
--mem-fraction-static 0.85
--reasoning-parser kimi_k3
--tool-call-parser kimi_k3
--mamba-full-memory-ratio 0.86
--host 0.0.0.0
--port 30000
--speculative-algorithm DSPARK
--speculative-draft-model-path <Kimi-K3-DSpark model path>
--speculative-dspark-block-size 7
--enable-linear-replayssm-spec
--chunked-prefill-size 32768
--max-prefill-tokens 32768
--context-length 1000000
--kv-cache-dtype fp8_e4m3

GPUStack exposes the live service logs, throughput, and KV Cache state after the service starts.

64K / 3K at Concurrency 10

The first comparison used 64K input, 3K output, and ten concurrent requests.

vLLM Result

SGLang Result

The corresponding inference logs are shown below.

Results Across Different Input Lengths

Additional input lengths were tested to observe how context growth affects TTFT, TPOT, ITL, and end-to-end latency.

vLLM

SGLang

Core Result: Performance Crosses Over Between 64K and 200K

Scenario	vLLM	SGLang	Result
64K end-to-end time	100.5 s	150.8 s	vLLM leads by ~1.5×
200K end-to-end time	295.2 s	225.3 s	SGLang leads by ~1.31×
64K stable Decode throughput	~501 tok/s	~335 tok/s	vLLM leads by ~1.5×
200K stable Decode throughput	~163 tok/s	~268 tok/s	SGLang leads by ~1.64×
Decode degradation, 64K→200K	3.29×	1.25×	SGLang scales better

In one sentence: vLLM is faster at shorter contexts because it avoids DCP communication overhead. As context grows, KV bandwidth becomes dominant, and SGLang’s DCP=8 distributes the KV reads across eight GPUs.

Source inspection shows that this is not simply a missing vLLM flag. vLLM explicitly prohibits DCP and DSPARK from being enabled together. SGLang allows them to coexist, but speculative decoding falls back to a non-adaptive verification mode. This is an architectural trade-off rather than a configuration oversight.

Why the Crossover Happens

1. Decode Scalability Is the Decisive Factor

Stable throughput was measured after all ten requests entered Decode.

Aggregate Decode throughput during the stable ten-request phase, in tokens/s, based on engine-reported logs.

vLLM (DCP=1)

• At 64K, stable throughput was approximately 439–537 tok/s, peaking at 537 tok/s.
• Per request, this is approximately 53.7 tok/s, or 18.6 ms/token.
• At 200K, stable throughput was approximately 157.8–166.6 tok/s, averaging about 163 tok/s.
• Per request, this is approximately 16.3 tok/s, or 61.3 ms/token.
• Context length increased by about 3.1× while Decode performance fell by 3.29×, indicating near-linear degradation.

SGLang (DCP=8)

• At 64K, stable throughput was approximately 313–418 tok/s, averaging about 335 tok/s.
• Per request, this is approximately 33.5 tok/s, or 29.9 ms/token.
• At 200K, stable throughput was approximately 252–288 tok/s, averaging about 268 tok/s.
• Per request, this is approximately 26.8 tok/s, or 37.3 ms/token.
• Context length increased by about 3.1× while Decode performance fell by only 1.25×.

At 64K, DCP communication overhead is greater than the reduction in KV reads, so vLLM is faster. At 200K, KV bandwidth becomes the primary bottleneck and SGLang’s DCP advantage becomes visible.

2. Prefill Performance Is Broadly Comparable

Both engines used a 32,768-token Prefill budget. Stable aggregate Prefill throughput was approximately 19K–23K tok/s, indicating comparable MoE + MLA Prefill capability on 8×B300.

The vLLM 64K run was affected by 32 soft OOM retries across eight GPUs and runtime JIT compilation of four kernels. These events occurred early in the test and made vLLM’s first-batch TTFT look worse. They were not reproduced in the 200K run, and SGLang did not report matching OOM or JIT events.

3. SGLang Logs Expose Long-Context Prefill Degradation

For a single 200K request, SGLang reported the following throughput over six chunks:

22265 → 21064 → 19628 → 18475 → 17243 → 16408 tok/s

Throughput returned to approximately 22K tok/s when the next request started. This directly shows the rising attention cost as cached context grows. vLLM reports a coarser ten-second aggregate prompt throughput, so the same pattern is less visible in its logs.

DSPARK Speculative Decoding Added Little Value

The Random dataset is effectively unpredictable, making it difficult for the draft model to produce consecutive accepted tokens. Both engines used Kimi-K3-DSpark with seven speculative tokens, but average accepted length at 200K remained close to one:

• vLLM: approximately 1.05–1.17
• SGLang: approximately 1.05–1.17

In the vLLM 200K run, drafted throughput was about 990 tok/s while accepted throughput was only about 21 tok/s. Approximately 98% of draft-model computation was discarded.

Scenario	Engine	Accepted Length	Acceptance / Draft Hit Rate	Interpretation
64K	vLLM	2.27–2.51	18%–21%	The only range with material benefit
64K	SGLang	1.21–1.55	3%–8%	Much weaker than vLLM on the same data
200K	vLLM	1.05–1.17	1.1%–2.5%	Draft compute is almost entirely wasted
200K	SGLang	1.05–1.17	1%–2%	The same collapse occurs

This run measured a “speculation as pure overhead” scenario. Production validation should use ShareGPT, real business prompts, or another natural-language dataset.

Different Acceptance Criteria

vLLM used probabilistic draft sampling with block rejection, while the SGLang DSPARK path used strict greedy matching. In SGLang, a mismatch on the first token invalidates the remaining tokens in the block. Strict comparison therefore requires the same dataset, sampling settings, gamma, and acceptance criteria.

Client Metrics: Excluding Unreliable Throughput Columns

Metric	vLLM 64K	SGLang 64K	vLLM 200K	SGLang 200K
Total time (s)	100.5	150.82	295.21	225.34
Mean request latency (s)	84.42	141.53	284.58	217.83
Mean TTFT (ms)	34,838.8	41,670.6	125,210.8	59,002.0
TTFT P50 (ms)	34,181.1	40,391.9	125,189.2	54,104.0
TTFT P90 (ms)	36,987.9	53,525.0	126,667.0	103,336.0
Mean TPOT (ms)	12.28	47.18	45.21	72.6
Mean ITL (ms)	9.94	33.3	3.47	52.96
Failed / incomplete	0 / 0	0 / 0	0 / 0	0 / 0

Some client-reported throughput fields cannot be reconciled with token counts, concurrency, and elapsed time. For example, the vLLM 200K run reported 1111.61 output tok/s, while 10 × 3000 ÷ 295.21 gives a physical upper bound of about 101.6 tok/s. The discrepancy is consistently close to 11×, so this article uses latency metrics from the client and throughput from engine logs.

TTFT Semantics Must Also Be Aligned

With --reasoning-parser kimi_k3, reasoning output is emitted in delta.reasoning, not delta.content. A benchmark that starts TTFT only at the first content token systematically inflates TTFT for both engines. For reasoning models, a better definition is:

time-to-first-any-delta

The timer should stop on the first reasoning or content delta.

Memory Allocation and KV Cache Capacity

Each B300 exposed 267.69 GiB. Model weight memory was similar, but the remaining memory was allocated differently.

vLLM

Item	Value
Model weights and non-torch memory	197.72 GiB
Peak activation memory	12.82 GiB
CUDA Graph	5.13 GiB
KV Cache	43.76 GiB

The KV pool held 2,687,776 tokens, giving a theoretical maximum concurrency of 2.69 at a 1M context. However, the pool was overcommitted. vLLM logs indicated that 38.48 GiB was the safe KV allocation under the configured budget.

Use either:

--gpu-memory-utilization 0.92

or:

--kv-cache-memory 41317885440

SGLang

Item	Value
Model weights	195.86 GiB
Full-attention KV Cache	11.80 GiB
Draft-model KV (K–V)	8.74 GiB
Mamba / SSM state	10.93 GiB
Remaining space	17.08 GiB

The full-attention pool held 916,672 tokens, below the configured one-million-token context. A full 1M request therefore cannot fit without adjusting the pool. max_running_requests also fell from 48 to 39 because of the Mamba state cache.

Cold-Start Cost

Stage	vLLM	SGLang	Notes
Container created	17:06:21	18:01:45	GPUStack timestamp
Weight loading	163.8 s	141.0 s	12 vs. 96 shards
Weight size per GPU	194.54 GiB	195.86 GB	MXFP4 MoE
Kernel warmup	CuTeDSL + KDA Triton	KDA / FA4 / RoPE	K3-specific JIT on both
CUDA Graph capture	85–88 s, 51 sizes	~5.5 min, 67 sizes	SGLang captured more shapes
Engine initialization	409.61 s	~370 s	SGLang inferred from timestamps
Container-to-ready	~10.6 min	~10.2 min	Same order of magnitude

Both engines required roughly ten minutes. Frequent autoscaling should therefore use warm instances, staged expansion, or prewarming.

The Configurations Were Not Fully Equivalent

Item	vLLM	SGLang	Potential Impact
Decode context parallelism	DCP=1	DCP=8, comm=a2a	Dominant at long context
All-Reduce	FlashInfer + Custom + SymmMem	Custom AR disabled; NCCL fallback	High for small-batch Decode
MLA Prefill backend	TRTLLM_RAGGED	trtllm_mla	Medium
MLA Decode backend	FLASHINFER_MLA	cutedsl_mla	Medium
Linear attention	FlashKDA Prefill	Triton Decode/verify/extend	Medium
MoE backend	FLASHINFER_TRTLLM_MXFP4_MXFP8	flashinfer_mxfp4	Medium
KV page size	32, aligned to 1536	64	Medium
Compile path	Disabled by breakable CUDA Graph	Not enabled	Medium
Memory fraction	0.95	0.92	Low–medium
Concurrency limit	max_num_seqs=64	48→39	Not reached in this test

Warnings Worth Addressing

FP8 KV Cache Scaling Factors

Both engines ran FP8 KV Cache with a fallback scale of 1.0. Performance conclusions should be paired with a fixed quality evaluation to confirm that FP8 KV Cache does not introduce unacceptable accuracy loss.

Slow Tokenizer

Both engines reported a slow tokenizer. At 64K and 200K, tokenization time contributes directly to TTFT. A fast tokenizer should be enabled where possible.

SGLang Device Identification

One log entry identified the device as NVIDIA L20D, while other entries correctly reported B300/GB300 and SM100/SM103. Because DCP communication backend selection can depend on the device name, the selected a2a path should be verified.

vLLM Memory Utilization

The 0.95 setting overcommitted KV memory. Reducing it to 0.92 or explicitly setting the log-recommended KV cache size removes the repeated runtime allocation retries.

Potentially Inactive vLLM Fusions

Some reported fusions are Inductor passes. Kimi-K3 enables breakable CUDA Graph mode, which can force compilation mode to NONE; the fusions may therefore not have been applied. FlashInfer autotuning was also disabled, meaning this vLLM result may be below its best achievable performance.

Source-Level Root Cause: DCP vs. Speculative Decoding

This is the strongest conclusion in the analysis, and it cannot be bypassed with a simple flag change.

vLLM structurally disallows DCP and DSPARK from running together. When decode_context_parallel_size > 1 is combined with K3DSparkModel, startup fails. To gain long-context KV sharding in vLLM, speculative decoding must be disabled.

SGLang permits DCP and DSPARK together, but dcp_size > 1 forces static ragged verification. Adaptive SPS cost tables, STS calibration, confidence planning, and block acceptance estimation are no longer active.

Capability	vLLM	SGLang	Deployment Implication
DCP + speculative decoding	Not allowed	Allowed, but non-adaptive	Both require a trade-off
Long-context KV sharding	Requires disabling speculation	Can coexist with speculation	Structural source of the 200K advantage
Adaptive speculation	No equivalent planner	Disabled when DCP is enabled	Neither offers both benefits together
Adjustable acceptance	Two explicit parameters	Threshold hard-coded in kernel	SGLang has no ready tuning knob

Additional Findings from Source Inspection

SGLang’s Fused KDA Verify Kernel Probably Did Not Run

The dispatcher assigns the verify kernel to Triton unless strict runtime shape constraints allow the CuTe path. Logs ended with verify=TritonKDAKernel, leaving additional optimization potential.

Identity of SGLang’s “Second KV Pool”

7,333,376 = 916,672 × 8, exactly matching dcp_size. The pool is the draft worker’s MHA KV pool addressed in DCP’s virtual token space, not a second architectural KV type.

Why vLLM Uses a 1536-Token Attention Block

The hybrid model requires an attention page to be at least as large as a Mamba page. The block grows from 32 to 1536, with approximately 0.70% Mamba-page padding. This coarser boundary also influences chunk alignment and can contribute to Decode starvation.

Source of vLLM’s 2.69× Concurrency Limit

The scheduler reserves KV for the full maximum sequence length at admission time. Reducing --max-model-len from 1M to the actual production requirement immediately increases concurrency.

Deployment Recommendations

Inputs Up to 64K, Throughput First

Prefer vLLM. It delivered approximately 1.5× higher Decode throughput and approximately 1.5× lower end-to-end time in this test.

200K and Longer Document Workloads

Prefer SGLang with DCP. It delivered approximately 1.64× higher Decode throughput at 200K and degraded much more slowly as context increased. Increase the KV pool to at least the target context length before production.

Long Context Plus High Concurrency

The current configuration favors vLLM’s capacity, but this conclusion requires retesting. SGLang should first rebalance KV and Mamba memory, then be compared with a vLLM configuration that disables speculation and enables DCP.

Next-Round Retest Plan

#	Action	Owner	Rationale
1	Set --long-prefill-token-threshold 4096	vLLM	Prevent one long Prefill from consuming the entire Decode budget
2	Replace Random with ShareGPT or real long documents	Benchmark platform	Current acceptance is only 1%–8%
3	Measure time-to-first-any-delta	Benchmark platform	Include reasoning deltas in TTFT
4	Fix throughput accounting	Benchmark platform	Current vLLM output throughput is inflated by ~11×
5	Reduce memory utilization to 0.92 or pin KV memory	vLLM	Remove runtime OOM retries
6	Add an SGLang DCP-off group	SGLang	Isolate DCP and static-verification effects
7	Set max model length to the business requirement	vLLM	Avoid unnecessary full-length KV reservation
8	Expand warmup to cover runtime JIT shapes	vLLM	Remove first-batch compilation noise
9	Re-enable FlashInfer autotuning	vLLM	Allow better MoE tactics
10	Re-enable custom All-Reduce and diagnose multicast	SGLang	Reduce small-batch communication overhead
11	Diagnose the KDA CuTe verify fallback	SGLang	Recover unused optimization potential
12	Provide FP8 KV scaling factors and run quality regression	Both	Add an accuracy basis to performance conclusions
13	Increase Mamba memory ratio or use bf16 SSM	SGLang	Remove the 39-request limit and raise the KV pool
14	Add P99 and concurrency ≥64	Benchmark platform	Exercise scheduler and concurrency limits

Conclusion

The most valuable result of this 8×B300 test is not a simple ranking. It shows how the bottleneck changes with context length:

• At 64K, communication overhead matters more, favoring vLLM.
• At 200K, KV read bandwidth dominates, allowing SGLang with DCP to overtake.
• Prefill is not the main differentiator; Decode scalability is.
• Speculative decoding is ineffective on Random data.
• Memory allocation, TTFT semantics, and client-side throughput accounting can materially change the apparent result.

Production selection should therefore evaluate context length, concurrency, KV capacity, DCP, real request distributions, and output quality together—not rely on a single TPS number.