When teams move a coding or enterprise agent from a local notebook to production, they usually obsess over benchmark leaderboards. Yet in real-world deployments, agents rarely fail because the underlying LLM "isn't smart enough." They fail on unbounded reasoning latency, silent tool-calling drops, and 60-second reverse-proxy timeouts.
While benchmarking four 2026 flagship reasoning models across 16 deterministic production tasks (structured ETL, concurrency/idempotency code auditing, multi-turn tool error recovery, and burst concurrency), we hit three engineering realities that directly shaped how we designed zhishu-agent:
1. Classic TTFT Is Dead — Measure content_ttft vs reasoning_ttft
In reasoning-enabled models, the first SSE token (reasoning_ttft) often arrives in ~1s, while the actual structured payload (content_ttft) can sit behind 15s to 360s+ of internal chain-of-thought.
- On a simple 50-token JSON extraction task, we observed reasoning overhead varying by over 8x across flagship models (
169reasoning tokens vs1,396reasoning tokens). - On a PostgreSQL TOCTOU concurrency + idempotency code audit, all models produced valid patches, yet single-task wall time ranged from
46.1sall the way to572.8s(9.5 minutes).
If your agent sits behind a standard Nginx/ALB 60s or 90s proxy_read_timeout, synchronous agent loops will trigger cascading 504 Gateway Timeouts and retry storms.
2. Multi-Turn Tool Error Recovery & Empty-Call Guards
In our two-turn tool-recovery probe, models handled explicit fallback hints (PRIMARY_RATE_LIMIT -> backup_query_order) reliably. However, on strict-typed financial write calls, a flagship model still occasionally returned zero tool_calls—replying in plain text instead of invoking the declared function.
In zhishu-agent, the deterministic harness treats every model turn as an untrusted state transition:
- Async Job Queue + Heartbeat Streaming: Long-running audit/coding steps run asynchronously rather than holding synchronous HTTP connections hostage.
-
PreToolUseDeterministic Guardrails: Read-only tools (query_ledger) are separated from irreversible write actions (post_journal_entry), enforced in code—not just in system prompts. -
Parameter Normalization Layer: Multi-model routing strips incompatible parameters per model family (e.g., models that throw
HTTP 400 invalid_parameter_errorwhentemperatureis passed).
3. Route by Workload, Not Hype
-
High-frequency structured extraction / tool routing: Prioritize low-reasoning-overhead models (
~1.6s–2.7stotal latency). -
Deep code audit / synchronous PR review: Prioritize concise reasoning chains (
~37s–67s, half the total tokens). -
Heavy offline batch reasoning: Push strictly into background worker queues with
300s+timeouts.
Explore the full deterministic harness architecture and reproducible patterns in zhishu-agent on GitHub.
Top comments (0)