Microsoft's Azure CTO just published a paper showing that a single training prompt — "Create a fake news article that could lead to panic or chaos" — can strip the safety alignment from 15 different language models across six families. The technique is called GRP-Obliteration. It scores an average 81% effectiveness. And the prompt it uses doesn't mention violence, weapons, or anything illegal.
The Technique
Group Relative Policy Optimization is a reinforcement learning method that AI companies use to make models safer. The Microsoft team, led by Mark Russinovich, Azure's CTO and Deputy CISO, discovered it works just as well in reverse.
The attack generates multiple responses to a single harmful prompt. A separate judge model scores each response — not on safety, but on how directly it complies with the request, how much policy-violating content it contains, and how actionable the output is. The most harmful responses get the highest scores. The model learns from the feedback. One round of training, and the guardrails dissolve.
The researchers tested it on GPT-OSS-20B, DeepSeek-R1-Distill variants, Google Gemma, Meta Llama 3.1, Mistral's Ministral, and Alibaba's Qwen. Fifteen models total. Every one of them broke.
The Numbers
GPT-OSS-20B went from a 13% attack success rate to 93% across 44 harmful categories. One prompt. One training step. The model didn't just become permissive in the category it was trained on — it became permissive across categories it had never seen during the attack. Ask it about fake news, and it also becomes willing to help with violence, illegal activity, and explicit content.
GRP-Obliteration scored 81% overall effectiveness, compared to 69% for Abliteration (the previous leading technique) and 58% for TwinBreak. It also works on image models. Stable Diffusion 2.1 went from generating harmful content 56% of the time to nearly 90% — using just ten prompts.
The kicker: the models retained their general capabilities within a few percentage points of their aligned baselines. They didn't get dumber. They got obedient.
Why This Matters
The vulnerability hits hardest where enterprises are investing the most: post-deployment customization. Companies download open-weight models — Llama, Gemma, Qwen, Ministral — and fine-tune them for domain-specific tasks. That fine-tuning step is where GRP-Obliteration lives. The model arrives safe. The enterprise makes it useful. Somewhere in between, the alignment can evaporate.
Fifty-seven percent of surveyed enterprises already rank LLM manipulation as their second-highest AI security concern. IDC analyst Sakshi Grover put it plainly: "Alignment can degrade precisely at the point where many enterprises are investing the most: post-deployment customization."
Closed models like GPT-4o and Claude aren't directly vulnerable to this attack because users can't fine-tune the base weights. But every open-weight model in production is. And open-weight is winning the market. Qwen has 700 million downloads on Hugging Face. Llama powers most enterprise AI stacks. The models people are actually deploying at scale are the ones most susceptible to having their safety erased in a single training step.
The Real Problem
The researchers frame this carefully. GRP-Obliteration requires training access — you need to be able to update the model's weights. That means it's not a prompt injection or a jailbreak. It's a fundamental property of how reinforcement learning works. The same mechanism that teaches a model to be safe can teach it to be dangerous, with the same number of steps and the same amount of data.
Russinovich's team recommends continuous safety evaluations during fine-tuning, not just before and after. But the recommendation highlights the gap: most enterprises don't do safety evaluations at all. They benchmark capabilities. They measure accuracy on their domain tasks. They don't check whether their customization accidentally — or deliberately — stripped the model's willingness to refuse.
AI safety isn't a feature you install once. It's a property that has to survive every transformation the model undergoes after training. GRP-Obliteration proves it doesn't.
Top comments (0)