Reasoning models have made LLMs much more useful for coding, math, and other multi-step tasks. Instead of returning the first plausible answer, they generate an internal step-by-step path and try to work through the problem more carefully.
That extra reasoning is valuable, but it also creates a new failure mode: some models can be pushed into extremely long, unproductive output loops. New research presented at ICML 2026 shows that this “overthinking” behavior is not just a curiosity — it can be exploited as a denial-of-service style attack against commercial AI systems.
For developers building on top of reasoning models, the practical takeaway is simple: if your product depends on these systems, you need to think about prompt-level abuse the same way you think about rate limits, token budgets, and API costs.
What the researchers found
The study came from researchers at Zhejiang University and Alibaba. They tested several current reasoning models, including:
- DeepSeek-R1
- Alibaba’s Qwen3-Thinking
- OpenAI’s GPT-o3
- Google’s Gemini 2.5 Flash
Their core finding was that logically inconsistent prompts can trigger severe overthinking. In some cases, the model keeps trying to reason through a problem that is fundamentally unsolvable instead of stopping early and rejecting it.
This behavior matters because longer outputs are expensive. More tokens mean more compute, more latency, and more load on the provider’s infrastructure. At enough scale, that creates a real operational problem.
On the MATH benchmark, the largest reported jump was from DeepSeek-R1, which produced outputs up to 26.1 times longer than its responses to the original, unmodified questions.
The attack idea: break the logic, not the model
What makes this research interesting from a builder’s perspective is that the attack does not require access to model internals.
The researchers used an evolutionary algorithm to corrupt the logical structure of prompts. Instead of trying to exploit weights, gradients, or hidden states, the method works by reshaping the question until the model gets trapped in reasoning loops.
The process started with 940 problems from three math benchmark datasets. The team used an LLM to decompose each problem into:
- premises
- a final question
Then they mutated those pieces in different ways:
- swapping premises between problems
- adding extra premises
- deleting premises
- swapping final questions between premise sets
After each mutation round, the prompts were scored on two things:
- how many words the target model produced
- how often the response contained signs of overthinking, such as words like “but,” “wait,” “maybe,” or “alternatively”
The best prompts were kept, then mutated again. The process repeated for five generations.
That is an important detail: the attack is not a one-shot trick. It is a search process that intentionally evolves prompts toward maximum confusion and maximum token generation.
Why this is a security issue for AI products
If you are shipping an app that uses a reasoning model behind an API, you probably already care about:
- request volume
- token consumption
- latency
- timeout handling
- rate limits
- abuse prevention
This research shows that a user does not necessarily need high request volume to cause trouble. A single carefully designed prompt can make one request much more expensive than expected.
That creates a few risks:
1. Cost amplification
If a prompt causes the model to generate far more tokens than normal, your cost per request can spike quickly.
2. Latency spikes
Long reasoning chains can slow down responses, which affects user experience and can create backpressure in your own system.
3. Shared service degradation
For providers serving many customers, abusive prompts can tie up compute that should be going to legitimate traffic.
4. Abuse detection is harder than it looks
Because this attack works through normal text input, it can be harder to flag than obvious scraping or flood attacks.
Why closed-source models are still exposed
One of the most useful parts of the study is that it does not rely on white-box access. The malicious prompts were generated by querying the target model, which means the method can apply to closed-source commercial services as well.
The researchers also showed that a smaller, cheaper model could be used to generate the malicious prompts, and those prompts still transferred to stronger targets. That increases feasibility because the attacker does not necessarily need to burn expensive target-model queries during the whole process.
From a defensive point of view, that means the attack surface is broader than “users who already know how a given model thinks internally.”
What this does and does not prove
The authors are careful not to claim that this is a turnkey large-scale DoS technique.
They note that real-world effectiveness depends on factors like:
- pricing model
- rate limiting
- context window size
- other provider-side defenses
In other words, the research does not prove that any one production service is trivially vulnerable. What it does prove is that the vulnerability exists and can be induced across several modern reasoning models.
That distinction matters. For builders, the right response is not panic. It is engineering.
Practical defenses to consider
The paper is a reminder that token output is a resource. If your application uses reasoning models, consider designing for adversarial prompts, not just helpful ones.
Some practical controls to look at:
Cap output and reasoning depth
Set strict maximum output lengths where possible. If a request starts ballooning, you want a hard stop.
Detect unsatisfiable or inconsistent prompts
If a prompt contains conflicting premises or appears logically incomplete, reject it or route it to a safer fallback.
Add prompt normalization and validation
For applications that accept structured input, validate fields before they reach the model. Don’t let malformed logic become a free-form reasoning exercise.
Track cost and latency anomalies
Monitor requests that produce unusually long completions or unusual token-to-answer ratios. These can be early warning signs of abuse.
Use defense-in-depth
Combine rate limiting, per-user quotas, timeout policies, and circuit breakers. No single control is likely to be enough.
The bigger lesson for developers
Reasoning models are better at solving harder problems, but they also expose a new class of failure: the model can become too willing to keep thinking.
That sounds harmless until you view it through the lens of infrastructure. A model that cannot recognize a broken premise is not just a correctness issue. It is a performance and cost issue.
If you are building AI products today, especially products that depend on expensive reasoning endpoints, you should treat logically inconsistent prompts as a security concern. The model does not need to be hacked in the traditional sense to become a liability.
It just needs to keep talking.
Top comments (0)