Reasoning-capable LLMs are better at math, coding, and multi-step tasks than the first wave of chat models. That extra capability comes from something you do not directly see in the response: an internal chain of reasoning where the model breaks a problem into steps before answering.
That same feature introduces a security problem.
A new line of research presented at ICML 2026 shows that you can deliberately push reasoning models into long, fruitless reasoning spirals by feeding them logically inconsistent prompts. The result is not just bad output quality. It can also turn into a denial-of-service-style issue for AI systems, because every extra token costs compute, latency, and server capacity.
For builders shipping applications on top of reasoning models, this is worth paying attention to. The risk is not only “the model gets confused.” It is “the model keeps talking for far longer than expected.”
What the attack is exploiting
Earlier LLMs tended to respond directly. Reasoning models, by contrast, often spend more time working through the problem internally before producing an answer. That makes them better at tasks where structure matters.
But prior research had already shown that these models can “overthink” even when the problem is not helping them. They may generate very long sequences of reasoning that do not improve correctness.
The new work takes that weakness and weaponizes it.
Instead of asking a reasonable question, the attackers create prompts with broken logic. The model tries to reconcile contradictions or missing premises, and that effort can trigger long reasoning loops. In effect, the model keeps trying to solve something that cannot be solved as stated.
From a systems perspective, that matters because longer outputs mean:
- more tokens generated
- more inference time
- more load on the provider
- worse latency for legitimate users
- potentially higher cost if you pay per token
That is why this looks like a DoS angle rather than just a quality bug.
How the researchers built the prompt attack
The team, from Zhejiang University and Alibaba, used an evolutionary algorithm to generate malicious prompts. The basic idea is familiar if you have ever seen genetic search used in optimization:
- Start with a set of problems.
- Mutate them in different ways.
- Measure which versions make the target model output more.
- Keep the strongest candidates.
- Repeat for several generations.
In this case, they started with 940 math problems from three benchmark datasets. An LLM was used to break each problem into logical premises and a final question. Then the algorithm applied “mutations” such as:
- swapping premises between different problems
- adding extra premises
- deleting existing premises
- swapping final questions across premise sets
The mutated prompts were then scored using two signals:
- how many words the target model generated
- whether the output contained linguistic markers associated with overthinking, such as “but,” “wait,” “maybe,” or “alternatively”
The prompts that scored highest were carried forward and mutated again over five generations.
That is the key implementation detail: the attack does not require access to the model internals. It only needs query access to the target. That makes it relevant to closed-source commercial models, not just research systems.
Which models were affected
The attack was tested against several reasoning models from major vendors, including:
- DeepSeek-R1
- Alibaba’s Qwen3-Thinking
- OpenAI’s GPT-o3
- Google’s Gemini 2.5 Flash
The researchers reported that the prompts caused outputs up to 26 times longer than the models’ responses to the original, unmodified questions. The biggest jump came from DeepSeek-R1 on the MATH benchmark, which includes high school competition-style math problems.
That is a serious multiplier. Even if a small fraction of requests trigger this behavior, the operational impact can be outsized.
The team also tested beyond math. They observed similar increases in output length for:
- coding tasks
- scientific reasoning
- dialogue challenges
So this does not appear to be a narrow benchmark trick tied only to arithmetic.
Why this matters for application developers
If you are building with reasoning models, you probably already think about correctness, hallucinations, and prompt injection. This research adds another layer: adversarial prompts that drive runaway inference cost.
There are a few practical implications.
1. Token budgets need abuse cases, not just happy paths
Many teams set max output limits for product reasons. That is useful, but not enough on its own. An attacker can still force the model into expensive intermediate reasoning before hitting the cap.
You should treat unusually long reasoning or repeated self-correction as a signal worth monitoring.
2. Rate limits and pricing can absorb some pressure, but not all
The researchers noted that provider pricing, rate limiting, context window limits, and existing defenses all affect the real-world impact. That is true. A strict API quota may make the attack less attractive.
But if your application depends on low latency or high throughput, even short bursts of overthinking can cause visible degradation.
3. Closed-source services are not immune
Because the method works through queries alone, it does not depend on model weights, gradients, or hidden states. That means you cannot assume a black-box commercial API is protected just because it is proprietary.
A useful defensive mindset
The paper is not claiming that large-scale attacks are free or guaranteed to work everywhere. The authors explicitly say the point is to show that the attack surface exists.
That is the right way to read it.
For teams shipping AI products, the takeaways are less about reproducing the attack and more about hardening against it:
- cap output length where possible
- watch for abnormal token growth per request
- add anomaly detection for repeated self-referential reasoning patterns
- test reasoning models against logically inconsistent prompts
- consider fallback paths when generation becomes excessive
None of those are silver bullets. But they are the kinds of controls that help turn a model vulnerability into an operationally manageable risk.
The bigger lesson
Reasoning is useful, but it is not free. The more a model is trained to “think through” ambiguity, the more important it becomes to handle malformed or adversarial inputs gracefully.
This research is a reminder that better reasoning does not only expand what models can do. It also expands the attack surface.
If your system uses reasoning models in production, you should assume that some prompts will not just produce the wrong answer. Some will try to make the model work far harder than it should.
And in an inference-priced world, that is a security issue as much as a performance one.
Tags: #LLMs #AI #Cybersecurity #MachineLearning
Top comments (0)