My LLM router routed by prompt length — a 12-word request broke a production agent loop

When I built my own OpenAI-compatible chat gateway, I needed routing tiers: cheap models for trivial calls, premium ones for real reasoning. The first heuristic was embarrassingly simple — total prompt length. Under ~500 characters, go eco tier; above, go premium. It passed every test I wrote.

Then an agent in production started looping. Same request, twenty times, never converging. I pulled its trace: the user prompt was tiny — "summarize this and call the report tool" — a dozen words. But the request carried six tool definitions and a JSON-mode flag. My router saw a short prompt and shipped it to the cheapest model in the fleet, which was fine at prose but unreliable at tool calls. Half the time it answered in natural language instead of invoking anything, the agent treated that as failure, retried, and looped.

The lesson: prompt length measures cost, not capability. A request with tool definitions, structured-output constraints, or image inputs has a hard capability floor no matter how short the text is. A budget model might comply 70% of the time — and a 70% success rate inside an agent loop compounds into never.

The fix took an afternoon. The router now checks capability flags first — any tools declared? JSON schema? images? — and those requests go to a verified minimum tier. Length only decides within a capability class. I also added a response validator: if a request declares tools and the reply contains no tool call, it retries one tier up instead of returning garbage to the caller.

Result: the loop rate on tool-using agents dropped from a steady trickle to effectively zero, and eco-tier spend barely moved, because most genuine traffic is chat-length prompts with no tools attached.

I ended up packaging the router as my LLM Chat API — one OpenAI-compatible endpoint with auto/eco/premium tiers (https://x402.freeq.one/tools/llm_chat.html). The auto tier encodes exactly this rule: capability first, length second. If you're building an agent loop with tool calls, don't let a cost heuristic route on prompt size alone.