We tested 5 LLMs on negative constraints and traps. Here is what we learned about silent empty responses, latency, and deterministic refusal.
https://arthur.matourdecontrole.fr/
Choosing a language model for production pipelines today usually feels like a gamble: vendor-reported latencies, irrelevant leaderboard scores, and an overwhelming halo effect around the newest flagship release[cite: 7].
When designing an autonomous agent system, the priority changes[cite: 7]. An agent doesn't need to write Shakespeare; it needs to be reliable. A confident hallucination costs significantly more than a refusal or a silence.[cite: 7]
We ran a benchmark with a simple premise: test models on their ability to handle constraints they cannot fulfill[cite: 7].
Here is what we observed, the architectural traps we hit, and the metrics recorded on our local benchmark[cite: 7].
The Protocol: Three Tests, Zero Mercy
We evaluated five setups (including DeepSeek, local Qwen 2.5-3B, and Gemini 2.5 Pro) across three distinct workloads[cite: 7, 9]:
- A simple classification task (structural extraction)[cite: 7].
- A pure mathematical calculation[cite: 7].
- An information trap: a prompt where the only objectively correct answer is "I don't know" or "I cannot determine this"[cite: 7].
When facing the trap, most generalist models attempt to invent plausible context with complete confidence[cite: 7]. They fill the void rather than declaring missing context[cite: 7].
Finding #1: The Silent Empty Response Bug
During stress testing, one model consistently returned completely blank responses ("") without throwing any standard API error code[cite: 7].
-
The cause: When the output constraint (
max_tokens) is set too low for an instruction-tuned model that tries to write verbose reasoning, it aborts output generation entirely if it cannot close its tokens cleanly[cite: 7]. - The impact: The caller script received a HTTP 200 with an empty payload[cite: 7]. In an agent pipeline expecting structured data, the parser broke with zero debugging signals[cite: 7].
Operational lesson: Never assume a model will naturally trim its answer[cite: 7]. You must explicitly configure strict token ceilings alongside fallback handlers for zero-byte payloads[cite: 7].
Finding #2: The Concision Ceiling
Without strict limits on output length (max_tokens clamped tight) and an explicit constraint on token concision, instruction-following degrades rapidly on negative constraints[cite: 7, 9].
The moment a model is allowed to output verbose Markdown or explain why it is processing the prompt, its attention drifts away from negative instructions (e.g., "Do NOT invent missing details"). Enforcing a minimal output length directly improved negative constraint adherence across the board[cite: 7, 9].
Benchmark Results (Measured Run)
Recorded on an isolated test machine (metrics dated September 10, 2026)[cite: 7]:
| Model | Trap Score | Output Integrity | Latency Delta |
|---|---|---|---|
| Model A (Verbose) | Failed (Hallucinated) | Passed | Baseline (fastest) |
| Model B (Uncapped) | Failed (Empty token edge-case)[cite: 7] | Failed on tight budget[cite: 7] | +42 ms |
| Model C (Deterministic Refusal) | Passed (6/6 explicit refusals)[cite: 7] | Passed | +163 ms[cite: 7] |
The model that won our negative-constraint benchmark consistently arrived at an explicit refusal[cite: 7]. However, it proved 163 ms slower than its nearest competitor[cite: 7].
Why? Because parsing negative boundaries, verifying premise validity, and deciding not to generate speculative tokens takes more computational steps than immediately generating the most probable next token[cite: 7].
Architectural Takeaways for Agents
- Refusal is a valid state: A system that admits missing information in 100 ms prevents cascading failures downstream across autonomous tools[cite: 7, 9].
- Don't outsource routing to text generators: If a decision can be resolved by local code, an embedding look-up, or a finite state machine, keep it off the LLM layer[cite: 10].
-
Hard limits over soft prompts: System prompts like "be concise" fail under edge cases. Tight
max_tokenscoupled with fallback guards are mandatory for stability[cite: 7, 9].
Discussion
How do you handle negative constraints in your production pipelines? Do you enforce strict programmatic guardrails, or do you rely on prompt engineering to suppress hallucinations[cite: 9]?
Top comments (0)