DEV Community

王果ai
王果ai

Posted on

Production MaaS Gateway Benchmark: Streaming TTFT, 100-Way Burst Concurrency & Token Billing Quirks

In week one of production deployment, systems rarely fail because a model isn't smart enough. They fail because of an unhandled HTTP 429, a streaming response that drops the final usage frame, or a monthly bill that doesn't match local token estimates.

In this benchmark (2026-11-17, 240 direct requests against dashscope.aliyuncs.com/compatible-mode/v1 with enable_thinking=false and zero SDK wrappers), we put the cloud inference gateway layer under a microscope.

Summary Scorecard (DashScope, 240 Requests)

Dimension Measured Result Sample & Scope
Streaming c_ttft qwen3.8-max: P50 0.585s / P95 0.857s; qwen3.8-flash: P50 0.521s / P95 0.762s 20 sequential requests/model, max_tokens=64
Total Request Latency qwen3.8-max: P50 1.349s / P95 1.794s; qwen3.8-flash: P50 1.155s / P95 1.599s Same as above
Streaming usage Frame 40/40 returned complete usage in the final chunk with stream_options.include_usage 20 streams/model
tools Protocol 5/5 per model returned standard tool_calls (query_order, ORD-77) Non-streaming
json_object Mode 5/5 per model returned valid json.loads() output (amount_cents=19950) Non-streaming
10 -> 100-Way Burst 180/180 HTTP 200 on qwen3.8-flash (0 HTTP 429s) max_tokens=16, stopped at 100-way burst
Rate-Limit Headers Only x-request-id and x-dashscope-timeout present; no x-ratelimit-* or Retry-After Observed on HTTP 200 responses

3 Production Findings You Won't See on Pricing Pages

1. Identical Prompts Bill Different prompt_tokens Across Models

Across all 40 sequential T1 requests with the exact same prompt:

  • qwen3.8-max billed 22–23 prompt_tokens
  • qwen3.8-flash billed 35–36 prompt_tokens

Never estimate costs using a local tokenizer or a single model's token count. Always compute spend from each endpoint's returned usage object.

2. 100-Way Burst Passed—Which Means You Must Build Your Own Circuit Breaker

Stepping concurrency from 10 -> 20 -> 50 -> 100 concurrent short requests yielded 180/180 HTTP 200s (100-way wall time: 2.200s, c_ttft P50 0.565s / P95 1.005s, max 2.187s).

Because no x-ratelimit-* headers are returned on 200s and default burst headroom is high, a runaway Agent loop will burn through your budget long before hitting a platform 429. Per-key rate limits and hard daily spend caps must live in your own gateway.

3. Batch Channels Cut Offline Compute Bills by 50%

On DashScope's published pricing, qwen3.8-max Batch File inference is priced at half of real-time (6 / 18 RMB per 1M input/output tokens vs 12 / 36 real-time; context-cached input drops to 1.5). Routing overnight summarization and ETL jobs through real-time endpoints doubles your bill unnecessarily.

Pre-Launch Gateway Checklist

  • ☐ Isolate every service with a dedicated sub-account API key and hard daily budget cap.
  • ☐ Log and bill strictly from the API's returned usage payload.
  • ☐ Enforce client-side circuit breakers and exponential backoff with jitter.
  • ☐ Route non-interactive batch workloads to Batch / low-priority endpoints.

Next in **Production AI Benchmarks: Testing 128K Long-Context "Needle in a Haystack" & Multi-Hop Contract Audits.

Reproducible Code & Benchmarking Collaborations

Top comments (0)