Every week, another model claims #1 on public leaderboards. Yet the moment you wire these models into a production Agent loop—facing dirty ERP logs, concurrency race conditions, multi-turn tool error recovery, 60s API gateway timeouts, and runaway reasoning_tokens bills—public benchmarks stop being predictive.
To measure what actually breaks in production, I built Suite v2: 16 deterministic engineering tests across 6 dimensions, executed directly against Alibaba Cloud DashScope's OpenAI-compatible endpoint (dashscope.aliyuncs.com/compatible-mode/v1) with zero proxy wrappers or client caching (test date: 2026-11-16).
Methodology: 4 Engineering Rules
- Direct, unwrapped Python probes: Pure standard-library HTTP calls so SDK quirks cannot hide gateway behavior.
-
Split TTFT (
r_ttftvsc_ttft): For reasoning models, we separate reasoning first-token latency from content first-token latency (c_ttft). Downstream JSON parsers only start working atc_ttft. -
100% deterministic assertions: All 15 functional tasks use code assertions (
json.loads(), integer-cent checks, AST/keyword checks,tool_callsprotocol verification)—zero LLM-as-a-judge. -
6 Test Dimensions:
-
A (Structured Extraction,
A1–A3): Dirty ERP financial logs, nested SKU discount math, and raw Nginx/Java 502 stack traces (strict JSON, zero Markdown fences). -
B (Production Code Audit,
B1–B4): TOCTOU race conditions + idempotency (B1), SQL N+1 & cursor pagination (B2),asyncioblocking + reentrant deadlock (B3), SQLi + SSRF (B4). -
C (Agent Tool Calling,
C1–C4): Unauthorized write interception (C1), first-hop dependency routing (C2), 2-turn error recovery to backup tool (C3), strict integer-cent argument construction (C4). -
D (16K Long-Context Noise,
D1–D2): Mid-document contract amendment conflict (D1) and distributed timeout bottleneck (D2). -
E (Anti-Hallucination & Negative Constraints,
E1–E2): Refusing a fabricated closed-source function (E1) and strict line/word/zero-English constraints (E2). - F (5-Way Burst Concurrency): Wall-clock throughput and SDK parameter compatibility.
-
A (Structured Extraction,
Overall Scorecard
| Metric | qwen3.8-max |
deepseek-v4-pro |
glm-5.3-prime |
kimi-k3.1-pro |
|---|---|---|---|---|
| Script Auto-Pass Rate |
14/15 (93.3%) |
14/15 (93.3%) |
14/15 (93.3%) |
13/15 (86.7%) |
| Human-Audited Pass Rate |
14/15 (C4 missed call) |
15/15 |
15/15 |
15/15 |
| Total Latency (15 Tasks) |
1,184.0s (19.7 min) |
309.9s (5.1 min) |
226.1s (3.7 min) |
824.7s (13.7 min) |
| Total Tokens Consumed |
69,144 (highest) |
29,809 (lowest) |
52,931 |
49,203 |
| 5-Way Burst Wall Time |
3.891s (5/5) |
19.519s (5/5) |
4.909s (5/5) |
11.225s (5/5) |
Key takeaway: On correctness, all four models cluster between 93% and 100%. What separates them in production is a 5.2x latency gap, a 2.3x token cost gap, and silent tool-calling failures.
Deep Dive: Where Production Systems Actually Break
1. The 572-Second Gateway Timeout Bomb (Dimension B)
Most enterprise API gateways (Nginx, AWS ALB, Kong) enforce a 60s–90s proxy_read_timeout. On heavy code audits (B1–B4), reasoning chains blew past that limit:
| Audit Task | glm-5.3-prime |
deepseek-v4-pro |
kimi-k3.1-pro |
qwen3.8-max |
|---|---|---|---|---|
B1_toctou_idempotency |
62.853s |
64.218s |
159.170s |
572.793s (9.5 min) |
B2_sql_n_plus_one |
47.968s |
65.595s |
164.431s |
175.706s |
B3_async_deadlock |
42.482s |
37.120s |
74.381s |
109.418s |
B4_security_sqli_ssrf |
33.995s |
67.063s |
298.862s |
228.445s |
| Dimension B Total | 187.3s |
234.0s |
696.8s |
1,086.4s (18.1 min) |
If you wire qwen3.8-max or kimi-k3.1-pro synchronously behind a GitHub/GitLab Code Review webhook, the connection will time out before the model finishes thinking. Meanwhile, deepseek-v4-pro solved all four audits in 29,809 total tokens across all 15 tasks—less than half of qwen3.8-max (69,144).
2. Silent Tool-Call Drops & Error Recovery (Dimension C)
-
Unauthorized Write Trap (
C1): All 4 models respected the read-only boundary (2.098songlm-5.3-primeto8.522sonkimi-k3.1-pro). -
2-Turn Error Recovery (
C3): Whenquery_orderreturnedPRIMARY_RATE_LIMITwith a hint to usebackup_query_order, all 4 models switched cleanly (glm-5.3-primecompleted both turns in2.912s). -
Strict Type Check (
C4Failure): Instructed to post250.00yuan as integer cents (amount_cents: 25000),glm-5.3-prime(2.999s),deepseek-v4-pro(6.325s), andkimi-k3.1-pro(6.542s) all emitted validtool_calls.qwen3.8-maxreturned an empty tool call (no call,5.361s), outputting plain text instead—a classic state-machine deadlock if your Agent loop doesn't handle text-only responses.
3. Note on Sampling Parameter Compatibility
In this run, the benchmark harness stripped the temperature parameter for kimi-k3 requests beforehand (suite_v2.py lines 109–110), so no temperature-related HTTP error was triggered or recorded in the raw logs. Cross-model parameter compatibility differences will be tested separately with raw request/response payloads.
Production Routing Matrix
| Workload Role | Best Fit | Evidence from 16-Task Run |
|---|---|---|
| High-Frequency Agent Routing & ETL | glm-5.3-prime |
Fastest overall (226.1s total), reliable JSON & tool recovery, 4.909s 5-way burst |
| Code Audit & Cost-Sensitive Reasoning | deepseek-v4-pro |
Lowest token burn (29,809 total), all B1–B4 tasks under 67s
|
| High-Concurrency Short Tasks / Async Queues | qwen3.8-max |
Fastest 5-way burst (3.891s), but heavy reasoning requires async queues + no call fallback |
| Long-Logic Fallback & Anti-Hallucination | kimi-k3.1-pro |
Strongest hallucination refusal; configure 300s+ async timeouts for heavy audits |
Next in **Production AI Benchmarks: Cloud MaaS Gateway Latency, 100-Way Burst Concurrency, and Real Billing Audits on Alibaba Cloud DashScope.
Top comments (0)