Many enterprises have idle GPUs and want to use them to deploy and serve models, but optimizing model performance can be challenging. As a result, they may not get the most out of their existing compute resources. Inference performance depends not only on GPU compute power, but also on how requests are scheduled, memory is managed, and computations are executed.
In our multi-turn agent benchmark, Alibaba Cloud Smart Studio increased Qwen3.8-Flash-Next’s input throughput from 49,351 to 162,296 tokens per second, reaching 3.3× the baseline throughput. Average time to first token (TTFT) fell from 8.36 to 1.99 seconds, a 76% reduction.
Model Benchmark Results
We measured input throughput and average TTFT for four optimized model deployments, each against its own baseline.
| Optimized model | Input throughput: baseline → optimized | Average TTFT: baseline → optimized |
|---|---|---|
| Qwen3.8-27B-FP8-FAST | 16,645 → 38,020 tok/s (+128%) | 8.97 → 3.26 s (−64%) |
| Qwen3.8-Flash-Next-FP8-FAST | 49,351 → 162,296 tok/s (+229%) | 8.36 → 1.99 s (−76%) |
| DeepSeek-V4.1-Flash-FAST | 32.8k → 69.1k tok/s (+110%) | 64.8 → 23.2 s (−64%) |
| GLM-5.3-Flash-FAST | 32,043 → 87,775 tok/s (+174%) | 20.41 → 6.34 s (−69%) |
Results are from the supplied multi-turn agent conversation evaluations. They depend on the model, hardware, workload, and baseline configuration.
The benchmarks show how much performance improved on the tested workloads. To see where we made those improvements, let’s follow a request through the inference stack.
An Overview of the Inference Stack
A request typically goes through two stages. During prefill, the model processes the input and builds a KV cache. During decode, it uses that cache as it generates output tokens.
The architecture diagram below illustrates how the components of Alibaba Cloud Smart Studio’s inference stack fit together. The following sections dive into the principles behind each optimization area and how our stack implements them.
Scheduling
Prefill and decode have distinct compute and memory profiles. Under shifting traffic, a static GPU allocation often leaves one stage queuing while the other sits idle.
Alibaba Cloud Smart Studio separates prefill and decode onto dedicated workers, dynamically scaling replicas and tuning the prefill-to-decode ratio based on real-time load. Across tested production models, this reduces P99 TTFT by 30% and scales throughput to 15.2× at 16 replicas (95% of ideal linear scaling) with a shared KV pool.
KV Cache
Multi-turn conversations and agent workflows repeatedly send shared prefixes—such as system prompts, history, and tool outputs. Smart Studio’s KV-Pool caches and reuses these prefix states so only new tokens are processed, eliminating redundant prefill computation.
To support long contexts at high concurrency, it tiers KV cache across GPU HBM, host DRAM, and distributed DRAM/SSD, automatically balancing locality and data movement overhead. On a coding agent benchmark, this delivers up to 4.5× higher input throughput—read our KV-Pool article for implementation details.
Speculative Decoding
Alibaba Cloud Smart Studio integrates draft model approaches such as EAGLE-3 and DFlash to propose candidate tokens, and optimizes verification using backends such as FlashMLA and FlashAttention. It also dynamically adjusts its speculation strategy based on workload conditions, including batch size and queue pressure.
Quantization and Operator Optimization
Quantization
For quantization, we use lower-precision formats such as FP8 and NVFP4 to reduce memory usage and data movement, while evaluating model quality and inference performance.
Operator Optimization
At the GPU execution level, we profile workloads to identify performance bottlenecks. Based on the results, we select suitable operator backends and apply fusion and compilation to reduce execution overhead.
Getting Started
Alibaba Cloud Smart Studio helps enterprises deploy and serve open-source models on their existing GPU infrastructure. Explore Smart Studio to learn more about model serving and deployment.
- 👉 Ready to start? Activate Your Service Now!
- 🧾 Curious about pricing? Check out our Pricing Page
Exploring the Commercial Edition or Need a Demo?
- ✉️ Email Us: SmartStudio@alibabacloud.com
- 💬 Contact Sales: Contact Us / Request a Demo






Top comments (0)