SGLang for Multi-Turn Agents: Prefix-Cache Wins, Speculative-Decoding Risks

Why I Am Evaluating SGLang for Agent Serving

The production problem I want SGLang to solve is repeated work: an agent sends its instructions, repository context, previous answers, and tool results back to the server on every turn. Much of that prompt has already been processed. Paying to prefill it again wastes capacity that could serve another session.

Validation before deployment

No GPU server or AgentX traces were run locally, and the retrieved AgentX results are unofficial; validate cache reuse, latency, throughput, and correctness before claiming savings.

SGLang’s RadixAttention makes reusable token prefixes central to serving. Speculative decoding attacks a different cost: generating output by proposing candidate tokens and verifying them with the target model. I evaluate these as separate optimizations, not as one “agent acceleration” switch.

The evidence boundary matters: I reviewed the available installation instructions, launch examples, and AgentX optimization material, but I did not launch a GPU server or replay AgentX traces for this article. I have no local cache-hit, TTFT, or throughput measurements to present. The commands below are reproduction recipes, not a claim of completed deployment.

That distinction is especially important here. The supplied InferenceX AgentX optimization page includes an explicit warning that the retrieved copy is unofficial and its data may be modified or outdated. I use its results as leads for validation, not as a basis for buying fewer GPUs.

SGLang’s architecture still warrants evaluation. Multi-turn serving stresses more than attention kernels:

• Prefixes must survive eviction between turns.
• Routing must find the worker holding useful state.
• Tool serialization must preserve reusable token sequences.
• Draft-model state must survive scheduling and transfer.
• Cache reuse must preserve answer correctness.

A strong single-prompt throughput number cannot establish any of those properties. For an agent service, I care about the time to the next useful answer after a tool call, at the concurrency and session churn the business actually expects.

Proposed Setup and Multi-Turn Test

I would begin with one worker, one target checkpoint, and no disaggregation. That isolates prefix reuse before introducing host offload, routing, or draft models.

For my setup plan, I selected these installation paths from the SGLang repository:

# Python environment installation.
python -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip uv
uv pip install --prerelease=allow sglang

# Alternative: the published container image.
docker pull lmsysorg/sglang:latest

These are discovery commands, not reproducible production pins. Before benchmarking, I would record the installed package version or container digest and preserve the dependency list:

python -m pip freeze > requirements.resolved.txt
python -m sglang.launch_server --help > launch-server-help.txt

docker image inspect lmsysorg/sglang:latest \
  --format '{{json .RepoDigests}}'

I have not executed these checks here. In particular, I would not equate a flag appearing in an online example with support in an arbitrary installed wheel. The server-argument reference belongs beside the captured --help output during setup.

For my planned speculative-decoding comparison, I selected the Llama 3.1/EAGLE3 pairing. With compatible hardware and access to the target weights, the baseline recipe is:

python -m sglang.launch_server \
  --model meta-llama/Meta-Llama-3.1-8B-Instruct \
  --mem-fraction-static 0.7 \
  --cuda-graph-max-bs-decode 8 \
  --dtype float16 \
  --host 127.0.0.1 \
  --port 30000

For the speculative arm, I would restart the server and add the documented EAGLE3 configuration:

python -m sglang.launch_server \
  --model meta-llama/Meta-Llama-3.1-8B-Instruct \
  --speculative-algorithm EAGLE3 \
  --speculative-draft-model-path jamesliu1/sglang-EAGLE3-Llama-3.1-Instruct-8B \
  --speculative-num-steps 3 \
  --speculative-eagle-topk 4 \
  --speculative-num-draft-tokens 16 \
  --mem-fraction-static 0.7 \
  --cuda-graph-max-bs-decode 8 \
  --dtype float16 \
  --host 127.0.0.1 \
  --port 30000

I would verify the exact checkpoint revisions before either run. This example also requires model access where applicable; it is not a promise that every checkpoint downloads anonymously.

For my configuration, I would either leave all three sizing parameters unset for automatic selection or set all three explicitly when tuning; I checked this constraint in the speculative-decoding guide. I would test the explicit configuration above against automatic selection rather than mixing hand-picked and implicit values.

A runnable multi-turn smoke test

The following client sends two sequential turns and carries the first response into the second request. It requires a running server but no Python packages beyond the standard library.

import json
import urllib.request

URL = "http://127.0.0.1:30000/v1/chat/completions"
MODEL = "meta-llama/Meta-Llama-3.1-8B-Instruct"

messages = [
    {
        "role": "system",
        "content": "Answer briefly. Preserve identifiers exactly.",
    },
    {
        "role": "user",
        "content": (
            "The deployment identifier is amber-47. "
            "Acknowledge it in one sentence."
        ),
    },
]

for turn in range(2):
    payload = {
        "model": MODEL,
        "messages": messages,
        "temperature": 0,
        "max_tokens": 64,
    }
    request = urllib.request.Request(
        URL,
        data=json.dumps(payload).encode("utf-8"),
        headers={"Content-Type": "application/json"},
        method="POST",
    )
    with urllib.request.urlopen(request, timeout=180) as response:
        result = json.load(response)

    answer = result["choices"][0]["message"]["content"]
    print(json.dumps({"turn": turn + 1, "answer": answer}))
    messages.append({"role": "assistant", "content": answer})

    if turn == 0:
        messages.append({
            "role": "user",
            "content": "What deployment identifier did I give you?",
        })

Illustrative output, not captured server output:

{"turn": 1, "answer": "The deployment identifier is amber-47."}
{"turn": 2, "answer": "amber-47"}

This establishes only a basic client workflow. It does not measure cache reuse, prove that either answer will have this wording, or reproduce AgentX.

For a real AgentX replay, I still need the traces, harness revision, request scheduling, model configuration, and expected outputs. Those artifacts were absent from the available material. I would not substitute two chat requests and label the result “AgentX.”

My evaluation matrix would separate a cache-disabled control, the ordinary cache-enabled baseline, and cache plus speculation. I would confirm the cache-control option in the pinned build rather than invent a flag. Each arm needs cold-cache and warm-cache runs, plus repeated trials under session churn.

I would capture:

• Token-weighted prefix reuse: cached prompt tokens divided by prompt tokens, using verified server telemetry.
• TTFT: request dispatch to the first generated token, excluding empty streaming events.
• Output throughput: generated output tokens divided by wall time, with GPU count explicit.
• Session behavior: live sessions, active requests, branch fan-out, and tool-result sizes.
• Correctness: identifier retrieval, structured output validity, and task-specific assertions.

A non-streaming response time is not TTFT. Likewise, textual prefix similarity is not a measured cache-hit rate.

What I Could Not Verify and What I Would Test

The immediate roadblock in this review was missing execution evidence, not a demonstrated SGLang crash. I could inspect recipes, but I could not establish local performance results or verify a specific vLLM issue. I keep those gaps visible because otherwise the most persuasive part of an infrastructure review becomes fiction.

The technical failure modes still give me a concrete validation checklist.

Window traffic can compete with durable prefixes. In my evaluation, I would stress the shared pool used by sliding-window allocations and prefix pages, where transient window activity can displace reusable history under memory pressure. I would test this with interleaved sessions and enough pressure to force eviction. Repeating one prompt immediately after itself is insufficient.

My mitigation order would be to verify the allocator fixes in the selected build, measure reuse under churn, and then consider offload. Increasing memory allocation without understanding eviction can hide the problem rather than solve it.

Offload has model-specific state requirements. I interpret the hybrid-cache discussion narrowly: full-attention KV can be offloaded while a short sliding-window tail is reconstructed. Recurrent state requires its own checkpointing support; it cannot simply be treated as another reconstructible window.

I would not generalize that path to every supported model architecture.

Host-memory accounting also deserves an explicit budget. For my capacity estimate using the GLM AgentX configuration, I would budget --hicache-size in host-tier GB per rank. At 180 GB across each of four ranks, I would therefore allocate approximately 720 GB for those host cache tiers, not 180 GB for the whole deployment. I would reserve additional host capacity for the operating system and other processes.

Release labels do not prove feature availability. When reviewing the GLM deployment recipe, I found two version gates to check before launch: the dated 2026-10-25 nightly includes checkpoint-loading change #40776, while the v0.5.22 release does not; an AMD monokernel pin predating sgl-amd-mono#60 rejects FP8 KV cache and causes startup to fail.

My workaround would be an immutable image digest and a startup smoke test for the actual checkpoint, attention backend, and KV dtype. A package version alone is not enough.

Speculation changes scheduling as well as decoding. I would not use NGRAM as a neutral, no-extra-weights substitute for EAGLE3. In my comparison, I would account for NGRAM's CUDA-only support, incompatibility with --attn-dp-size > 1, and disabling of overlap scheduling and mixed chunked prefill. That changes the experiment.

Cache correctness must survive transfer. In my correctness review, I identified misplaced cache offsets during heterogeneous-disaggregation staging as a regression case to test. I would use the 127,500-token shared-prefix case, with 2 correct needles out of 128 before the fix and 128 out of 128 afterward, as a validation target. I treat that as an external regression-test lead, not my result or a universal failure rate.

Before introducing disaggregation, I would require correctness tests across cold, partially cached, and heavily reused prefixes. A fast wrong answer does not qualify as throughput.

Finally, I had no specific vLLM issue to inspect, so I cannot identify an affected version or recommend an issue-specific workaround.

Scale, Latency & Cost vs. Alternatives

My comparison starts with workload fit, not a blanket ranking.

Serving choice	Why I would evaluate it	Main evaluation risk	Required decision evidence
SGLang with prefix reuse	Repeated multi-turn history	Eviction or routing destroys reuse	Warm/cold TTFT and token-weighted cache reuse
SGLang with EAGLE3 or model-specific MTP	Decode-heavy responses	Draft and verification work outweigh accepted tokens	Acceptance behavior, latency, throughput, and memory at target load
vLLM with prefix caching	Existing vLLM deployment and operational familiarity	An unmatched cache or scheduler configuration creates a misleading comparison	Same traces, weights, precision, load, and correctness checks
Managed inference API	Lower infrastructure ownership	Limited cache controls and variable billing economics	Actual session-level spend, latency, and service guarantees

I would not describe this as RadixAttention beating PagedAttention. PagedAttention concerns KV-memory management; automatic prefix caching concerns reuse. A meaningful vLLM comparison must enable and validate the relevant reuse behavior rather than compare a tuned SGLang configuration against an unrelated default.

In my performance review, I separated the externally reported 26.75% output-throughput improvement and 36.25% reduction in mean TTFT at concurrency 384 from speculative-decoding gains: the change replaced context-length-driven kernel specialization with a runtime scalar, reducing compilation overhead rather than accelerating computation. These are not local benchmark results or measured benefits of enabling speculative decoding.

I also treated the FlashInfer GDN recurrent-state checkpointing result—47,771 to 53,004 tokens/s/GPU at 92.4% cache-hit rate—as a separate, configuration-specific comparison. I cannot transfer that result to the Llama smoke-test recipe, a different GPU, or an ordinary chat workload.

For capacity planning, I would use SLO-compliant completed sessions, not peak output tokens alone. For my AgentX replay plan, I would count concurrency as live sessions and track active requests separately, because subagent branches can create multiple simultaneous requests within each session.

Practical break-even calculation

My deployment decision would use:

Monthly benefit = removed GPU capacity + avoided external inference spend − added operating costs.

For an illustrative budget—not measured SGLang performance—assume:

• One removable GPU allocation costs $3 per hour.
• The service runs 730 hours per month.
• Engineering and maintenance cost $6,000 initially.
• Extra host resources and monitoring cost $400 monthly.

Removing one continuously allocated GPU saves $2,190 monthly before the added operating costs. Net savings are $1,790, giving a simple payback of about 3.4 months.

But a benchmark improvement does not guarantee that an allocation can be removed. If the service still requires the same replica count for peak demand, failover, or tail latency, immediate GPU-bill savings may be zero.

I would approve the change only after demonstrating that the smaller deployment meets the same TTFT and completion-latency objectives with acceptable correctness and failure behavior.

My Recommendation: Evaluate SGLang Before Deployment

My verdict is shortlist SGLang for a controlled agent-serving evaluation; do not claim validated savings from this review alone.

I see a credible architectural fit between reusable prefixes, cache-aware routing, and long-running agent sessions. I also see enough state-management complexity that a generic throughput benchmark would give me false confidence.

I would deploy after validation if:

• My production traces contain substantial repeated token prefixes.
• Warm-cache gains survive realistic interleaving and session churn.
• The pinned image supports the exact target and draft checkpoints.
• Speculation improves my operating point without unacceptable memory pressure.
• Correctness remains stable across cache hits, eviction, and transfer.
• The improvement removes billable capacity or meets an otherwise missed SLO.

I would hold off if:

• My prompts rarely share reusable prefixes.
• My evidence consists only of immediate repeated requests.
• I cannot identify the fixes included in the deployed build.
• I need hybrid-cache or disaggregation behavior I have not tested.
• My claimed savings depend on applying unrelated AgentX percentages.
• I lack the traces needed for a fair comparison with my existing server.

The next useful artifact is a pinned replay bundle: request traces, launch commands, runtime identity, cache telemetry, streaming timings, and correctness assertions. That is what turns an optimization story into a deployment decision.

For adjacent infrastructure options, browse our tools collection. If the harder problem is designing the replay, routing, and capacity gates around your service, talk with our engineering team.