Reasoning models are supposed to be better at hard tasks because they “think” before answering. In practice, that extra step-by-step process creates a new attack surface: if you can push the model into overthinking, you can make it spend far more tokens than normal and slow it down for everyone else.
That’s the core finding from a recent study presented at ICML 2026 by researchers from Zhejiang University and Alibaba. They show that carefully constructed, logically inconsistent prompts can trigger reasoning models into long, unproductive internal loops. The result is not just a quality issue. It looks a lot like a denial-of-service vector against commercial AI systems.
For builders shipping LLM-powered products, this matters because the failure mode is tied to cost, latency, and throughput, not just correctness.
What changed with reasoning models
Older LLMs tend to answer directly. Newer reasoning models generate an internal monologue, break a problem into steps, and then produce the final response. That has made them much more capable on coding, math, and other structured tasks.
But prior research had already shown that these models can “overthink” — producing long reasoning chains that don’t actually improve the answer. The new work takes that from an accidental behavior to a deliberate attack strategy.
Instead of asking a model to solve a normal problem, the researchers feed it a problem whose logical structure has been corrupted. The model keeps trying to reconcile premises that do not fit together, and in the process it keeps generating more text.
For an API provider, more generated text means more compute, more latency, and more load on infrastructure. If enough requests are shaped this way, legitimate users feel the impact.
The attack idea: mutate the prompt until it causes overthinking
The interesting part of this research is that it does not rely on model internals. The attack works by querying the target model from the outside, which makes it relevant even for closed-source commercial systems.
The researchers started with 940 math problems from three benchmark datasets. They used an LLM to decompose each problem into its premises and final question, then applied an evolutionary algorithm to mutate that structure in different ways:
- swapping premises between problems
- adding extra premises
- deleting premises
- swapping final questions between premise sets
After each round, the candidate prompts were scored on two things:
- how many words the target model generated
- whether the output showed markers of overthinking, such as “but,” “wait,” “maybe,” or “alternatively”
The highest-scoring prompts survived to the next generation, and the process repeated for five generations.
That’s a useful pattern to understand from a security engineering perspective. The attacker is not trying to confuse the model once. They are searching for prompt structures that maximize output length and keep the model trapped in reasoning mode.
What the researchers observed
The attack worked across several modern reasoning models, including:
- DeepSeek-R1
- Alibaba’s Qwen3-Thinking
- OpenAI’s GPT-o3
- Google’s Gemini 2.5 Flash
In some cases, the response length increased dramatically. The most extreme result reported was on DeepSeek-R1 using the MATH benchmark, where the longest output became 26.1 times longer than the longest response to the unmodified questions.
That is the kind of multiplier that should get the attention of anyone designing a metered inference service.
The team also tested beyond math and saw significant output growth in coding, scientific reasoning, and dialogue tasks. So while the benchmark setup was math-heavy, the behavior is not confined to a single domain.
Why this is a real operational risk
The immediate reaction might be: “So what? The model just outputs more text.” But for deployed systems, that “just” hides several concrete problems.
1. Cost amplification
Token generation is not free. If a prompt causes a model to emit several times more output than expected, the cost per request rises sharply. At scale, that can become expensive fast.
2. Latency spikes
Longer outputs mean slower responses. Even if the model is healthy, the user experience degrades because requests take longer to complete.
3. Throughput reduction
A small number of pathological requests can occupy model capacity and reduce the number of legitimate requests the system can serve.
4. Closed-source exposure
Because the attack does not need access to model weights or internal activations, providers can’t assume that secrecy alone protects them.
The practical limitation: the attack is not free
The researchers are careful not to overclaim. Building the malicious prompts requires repeated queries to expensive reasoning models, which can make the search process costly.
That said, they also showed that a smaller, cheaper model can be used to generate prompts that still induce the target model to overthink. That transferability makes the attack more feasible and lowers the barrier for an attacker who wants to automate prompt discovery.
So the right takeaway is not “this breaks all reasoning models tomorrow.” It is that the attack surface is real, and the economics may already be bad enough to matter in production settings.
What providers can do about it
The study does not present a full defense package, but it points to several practical controls that should be part of any reasoning-model deployment:
- Rate limiting to reduce the impact of repeated probing
- Output caps to bound worst-case token generation
- Pricing models that discourage abuse and reflect generation cost
- Context window controls to limit pathological prompt growth
- Detection for inconsistent or unsatisfiable prompts
- Monitoring for abnormal reasoning length as an operational signal
There is also a product-design question here: if a system routes every ambiguous task into a high-cost reasoning path, it may be more vulnerable than one that selectively uses reasoning only when needed.
Why this belongs on every AI builder’s threat model
A lot of AI security discussions focus on prompt injection, jailbreaks, and data leakage. Those are important, but this research highlights a different class of issue: resource exhaustion through pathological reasoning behavior.
That matters because modern AI systems are not just models. They are infrastructure. Anything that increases token usage unpredictably is an availability risk, and availability is a security property.
If you are building with reasoning models, this is worth adding to your threat model now:
- inputs can be logically inconsistent
- reasoning loops can become expensive
- output length is part of your attack surface
- black-box APIs are not immune
The main lesson is simple: as models get better at thinking, they may also get better at thinking themselves into trouble.
Top comments (0)