DEV Community

王果ai
王果ai

Posted on

Benchmarking 4 Chinese Flagship LLMs Under Real Agent Load (16 Deterministic Production Tests)

Every week, another model claims #1 on public leaderboards. Yet the moment you wire these models into a production Agent loop—facing dirty ERP logs, concurrency race conditions, multi-turn tool error recovery, 60s API gateway timeouts, and runaway reasoning_tokens bills—public benchmarks stop being predictive.

To measure what actually breaks in production, I built Suite v2: 16 deterministic engineering tests across 6 dimensions, executed directly against Alibaba Cloud DashScope's OpenAI-compatible endpoint (dashscope.aliyuncs.com/compatible-mode/v1) with zero proxy wrappers or client caching (test date: 2026-11-16).

Methodology: 4 Engineering Rules

  1. Direct, unwrapped Python probes: Pure standard-library HTTP calls so SDK quirks cannot hide gateway behavior.
  2. Split TTFT (r_ttft vs c_ttft): For reasoning models, we separate reasoning first-token latency from content first-token latency (c_ttft). Downstream JSON parsers only start working at c_ttft.
  3. 100% deterministic assertions: All 15 functional tasks use code assertions (json.loads(), integer-cent checks, AST/keyword checks, tool_calls protocol verification)—zero LLM-as-a-judge.
  4. 6 Test Dimensions:
    • A (Structured Extraction, A1–A3): Dirty ERP financial logs, nested SKU discount math, and raw Nginx/Java 502 stack traces (strict JSON, zero Markdown fences).
    • B (Production Code Audit, B1–B4): TOCTOU race conditions + idempotency (B1), SQL N+1 & cursor pagination (B2), asyncio blocking + reentrant deadlock (B3), SQLi + SSRF (B4).
    • C (Agent Tool Calling, C1–C4): Unauthorized write interception (C1), first-hop dependency routing (C2), 2-turn error recovery to backup tool (C3), strict integer-cent argument construction (C4).
    • D (16K Long-Context Noise, D1–D2): Mid-document contract amendment conflict (D1) and distributed timeout bottleneck (D2).
    • E (Anti-Hallucination & Negative Constraints, E1–E2): Refusing a fabricated closed-source function (E1) and strict line/word/zero-English constraints (E2).
    • F (5-Way Burst Concurrency): Wall-clock throughput and SDK parameter compatibility.

Overall Scorecard

Metric qwen3.8-max deepseek-v4-pro glm-5.3-prime kimi-k3.1-pro
Script Auto-Pass Rate 14/15 (93.3%) 14/15 (93.3%) 14/15 (93.3%) 13/15 (86.7%)
Human-Audited Pass Rate 14/15 (C4 missed call) 15/15 15/15 15/15
Total Latency (15 Tasks) 1,184.0s (19.7 min) 309.9s (5.1 min) 226.1s (3.7 min) 824.7s (13.7 min)
Total Tokens Consumed 69,144 (highest) 29,809 (lowest) 52,931 49,203
5-Way Burst Wall Time 3.891s (5/5) 19.519s (5/5) 4.909s (5/5) 11.225s (5/5)

Key takeaway: On correctness, all four models cluster between 93% and 100%. What separates them in production is a 5.2x latency gap, a 2.3x token cost gap, and silent tool-calling failures.

Deep Dive: Where Production Systems Actually Break

1. The 572-Second Gateway Timeout Bomb (Dimension B)

Most enterprise API gateways (Nginx, AWS ALB, Kong) enforce a 60s–90s proxy_read_timeout. On heavy code audits (B1–B4), reasoning chains blew past that limit:

Audit Task glm-5.3-prime deepseek-v4-pro kimi-k3.1-pro qwen3.8-max
B1_toctou_idempotency 62.853s 64.218s 159.170s 572.793s (9.5 min)
B2_sql_n_plus_one 47.968s 65.595s 164.431s 175.706s
B3_async_deadlock 42.482s 37.120s 74.381s 109.418s
B4_security_sqli_ssrf 33.995s 67.063s 298.862s 228.445s
Dimension B Total 187.3s 234.0s 696.8s 1,086.4s (18.1 min)

If you wire qwen3.8-max or kimi-k3.1-pro synchronously behind a GitHub/GitLab Code Review webhook, the connection will time out before the model finishes thinking. Meanwhile, deepseek-v4-pro solved all four audits in 29,809 total tokens across all 15 tasks—less than half of qwen3.8-max (69,144).

2. Silent Tool-Call Drops & Error Recovery (Dimension C)

  • Unauthorized Write Trap (C1): All 4 models respected the read-only boundary (2.098s on glm-5.3-prime to 8.522s on kimi-k3.1-pro).
  • 2-Turn Error Recovery (C3): When query_order returned PRIMARY_RATE_LIMIT with a hint to use backup_query_order, all 4 models switched cleanly (glm-5.3-prime completed both turns in 2.912s).
  • Strict Type Check (C4 Failure): Instructed to post 250.00 yuan as integer cents (amount_cents: 25000), glm-5.3-prime (2.999s), deepseek-v4-pro (6.325s), and kimi-k3.1-pro (6.542s) all emitted valid tool_calls. qwen3.8-max returned an empty tool call (no call, 5.361s), outputting plain text instead—a classic state-machine deadlock if your Agent loop doesn't handle text-only responses.

3. Note on Sampling Parameter Compatibility

In this run, the benchmark harness stripped the temperature parameter for kimi-k3 requests beforehand (suite_v2.py lines 109–110), so no temperature-related HTTP error was triggered or recorded in the raw logs. Cross-model parameter compatibility differences will be tested separately with raw request/response payloads.

Production Routing Matrix

Workload Role Best Fit Evidence from 16-Task Run
High-Frequency Agent Routing & ETL glm-5.3-prime Fastest overall (226.1s total), reliable JSON & tool recovery, 4.909s 5-way burst
Code Audit & Cost-Sensitive Reasoning deepseek-v4-pro Lowest token burn (29,809 total), all B1–B4 tasks under 67s
High-Concurrency Short Tasks / Async Queues qwen3.8-max Fastest 5-way burst (3.891s), but heavy reasoning requires async queues + no call fallback
Long-Logic Fallback & Anti-Hallucination kimi-k3.1-pro Strongest hallucination refusal; configure 300s+ async timeouts for heavy audits

Next in **Production AI Benchmarks: Cloud MaaS Gateway Latency, 100-Way Burst Concurrency, and Real Billing Audits on Alibaba Cloud DashScope.

Top comments (0)