DEV Community

Alibaba Cloud Smart Studio
Alibaba Cloud Smart Studio

Posted on AI-assisted

Alibaba Cloud Smart Studio Reaches 3.3 Input Throughput on Qwen

Many enterprises have idle GPUs and want to use them to deploy and serve models, but optimizing model performance can be challenging. As a result, they may not get the most out of their existing compute resources. Inference performance depends not only on GPU compute power, but also on how requests are scheduled, memory is managed, and computations are executed.

In our multi-turn agent benchmark, Alibaba Cloud Smart Studio increased Qwen3.8-Flash-Next’s input throughput from 49,351 to 162,296 tokens per second, reaching 3.3× the baseline throughput. Average time to first token (TTFT) fell from 8.36 to 1.99 seconds, a 76% reduction.

Model Benchmark Results

We measured input throughput and average TTFT for four optimized model deployments, each against its own baseline.

Optimized model Input throughput: baseline → optimized Average TTFT: baseline → optimized
Qwen3.8-27B-FP8-FAST 16,645 → 38,020 tok/s (+128%) 8.97 → 3.26 s (−64%)
Qwen3.8-Flash-Next-FP8-FAST 49,351 → 162,296 tok/s (+229%) 8.36 → 1.99 s (−76%)
DeepSeek-V4.1-Flash-FAST 32.8k → 69.1k tok/s (+110%) 64.8 → 23.2 s (−64%)
GLM-5.3-Flash-FAST 32,043 → 87,775 tok/s (+174%) 20.41 → 6.34 s (−69%)

Results are from the supplied multi-turn agent conversation evaluations. They depend on the model, hardware, workload, and baseline configuration.

Qwen3.8-27B

The benchmarks show how much performance improved on the tested workloads. To see where we made those improvements, let’s follow a request through the inference stack.

An Overview of the Inference Stack

A request typically goes through two stages. During prefill, the model processes the input and builds a KV cache. During decode, it uses that cache as it generates output tokens.

The architecture diagram below illustrates how the components of Alibaba Cloud Smart Studio’s inference stack fit together. The following sections dive into the principles behind each optimization area and how our stack implements them.

Inference Stack

Scheduling

Prefill and decode have distinct compute and memory profiles. Under shifting traffic, a static GPU allocation often leaves one stage queuing while the other sits idle.

Alibaba Cloud Smart Studio separates prefill and decode onto dedicated workers, dynamically scaling replicas and tuning the prefill-to-decode ratio based on real-time load. Across tested production models, this reduces P99 TTFT by 30% and scales throughput to 15.2× at 16 replicas (95% of ideal linear scaling) with a shared KV pool.

Scheduling

KV Cache

Multi-turn conversations and agent workflows repeatedly send shared prefixes—such as system prompts, history, and tool outputs. Smart Studio’s KV-Pool caches and reuses these prefix states so only new tokens are processed, eliminating redundant prefill computation.

To support long contexts at high concurrency, it tiers KV cache across GPU HBM, host DRAM, and distributed DRAM/SSD, automatically balancing locality and data movement overhead. On a coding agent benchmark, this delivers up to 4.5× higher input throughput—read our KV-Pool article for implementation details.

KV Cache

Speculative Decoding

Alibaba Cloud Smart Studio integrates draft model approaches such as EAGLE-3 and DFlash to propose candidate tokens, and optimizes verification using backends such as FlashMLA and FlashAttention. It also dynamically adjusts its speculation strategy based on workload conditions, including batch size and queue pressure.

Speculative Decoding

Quantization and Operator Optimization

Quantization

For quantization, we use lower-precision formats such as FP8 and NVFP4 to reduce memory usage and data movement, while evaluating model quality and inference performance.

Operator Optimization

At the GPU execution level, we profile workloads to identify performance bottlenecks. Based on the results, we select suitable operator backends and apply fusion and compilation to reduce execution overhead.

Operator

Getting Started

Alibaba Cloud Smart Studio helps enterprises deploy and serve open-source models on their existing GPU infrastructure. Explore Smart Studio to learn more about model serving and deployment.

Exploring the Commercial Edition or Need a Demo?

Top comments (0)