In week one of production deployment, systems rarely fail because a model isn't smart enough. They fail because of an unhandled HTTP 429, a streaming response that drops the final usage frame, or a monthly bill that doesn't match local token estimates.
In this benchmark (2026-11-17, 240 direct requests against dashscope.aliyuncs.com/compatible-mode/v1 with enable_thinking=false and zero SDK wrappers), we put the cloud inference gateway layer under a microscope.
Summary Scorecard (DashScope, 240 Requests)
| Dimension | Measured Result | Sample & Scope |
|---|---|---|
Streaming c_ttft |
qwen3.8-max: P50 0.585s / P95 0.857s; qwen3.8-flash: P50 0.521s / P95 0.762s
|
20 sequential requests/model, max_tokens=64
|
| Total Request Latency |
qwen3.8-max: P50 1.349s / P95 1.794s; qwen3.8-flash: P50 1.155s / P95 1.599s
|
Same as above |
Streaming usage Frame |
40/40 returned complete usage in the final chunk with stream_options.include_usage
|
20 streams/model |
tools Protocol |
5/5 per model returned standard tool_calls (query_order, ORD-77) |
Non-streaming |
json_object Mode |
5/5 per model returned valid json.loads() output (amount_cents=19950) |
Non-streaming |
| 10 -> 100-Way Burst |
180/180 HTTP 200 on qwen3.8-flash (0 HTTP 429s) |
max_tokens=16, stopped at 100-way burst |
| Rate-Limit Headers | Only x-request-id and x-dashscope-timeout present; no x-ratelimit-* or Retry-After
|
Observed on HTTP 200 responses |
3 Production Findings You Won't See on Pricing Pages
1. Identical Prompts Bill Different prompt_tokens Across Models
Across all 40 sequential T1 requests with the exact same prompt:
-
qwen3.8-maxbilled22–23prompt_tokens -
qwen3.8-flashbilled35–36prompt_tokens
Never estimate costs using a local tokenizer or a single model's token count. Always compute spend from each endpoint's returned usage object.
2. 100-Way Burst Passed—Which Means You Must Build Your Own Circuit Breaker
Stepping concurrency from 10 -> 20 -> 50 -> 100 concurrent short requests yielded 180/180 HTTP 200s (100-way wall time: 2.200s, c_ttft P50 0.565s / P95 1.005s, max 2.187s).
Because no x-ratelimit-* headers are returned on 200s and default burst headroom is high, a runaway Agent loop will burn through your budget long before hitting a platform 429. Per-key rate limits and hard daily spend caps must live in your own gateway.
3. Batch Channels Cut Offline Compute Bills by 50%
On DashScope's published pricing, qwen3.8-max Batch File inference is priced at half of real-time (6 / 18 RMB per 1M input/output tokens vs 12 / 36 real-time; context-cached input drops to 1.5). Routing overnight summarization and ETL jobs through real-time endpoints doubles your bill unnecessarily.
Pre-Launch Gateway Checklist
- ☐ Isolate every service with a dedicated sub-account API key and hard daily budget cap.
- ☐ Log and bill strictly from the API's returned
usagepayload. - ☐ Enforce client-side circuit breakers and exponential backoff with jitter.
- ☐ Route non-interactive batch workloads to Batch / low-priority endpoints.
Next in **Production AI Benchmarks: Testing 128K Long-Context "Needle in a Haystack" & Multi-Hop Contract Audits.
Reproducible Code & Benchmarking Collaborations
-
Open-Source Benchmark Suite (
p02_gateway): github.com/wangguode/llm-prod-bench - Independent Evaluations & DevRel Inquiries: Open to reproducible production benchmarks for LLMs, inference gateways, and Agent infra. Reach out via GitHub (@wangguode) or X (@sansaxypy).
Top comments (0)