A comprehensive benchmark report from Tech Against Terrorism tests 134 AI models across multiple prompts to examine safety guardrails, researcher framing effects, and the rapid rise of modified open-weight models published without restrictions.
On August 14, Chinese company Alibaba released the Qwen 3.8-27B, an artificial intelligence model designed to run on a laptop. Within 24 hours, versions of the model with the security features removed appeared online.
Three days later, a far-right user known online as Johnny Doomsayer announced that he was using the modified version to train a chatbot called MechaHitler. A week later, the user began advertising access to the chatbot for $14,88.
The research findings
The speed with which artificial intelligence systems are losing their security controls is revealed in the latest report by the British non-profit organization Tech Against Terrorism.
The researchers tested 134 models from major companies in the United States, China and Europe, using more than 200 questions related to mass casualty attacks and weapons development. Each question was tested three times with a different role: as an overt terrorist, as a covert effort or as a security researcher.
Overall, only two of the models passed the test completely, while 81 models failed, providing incomplete answers or low scores. When the researchers pretended to be security researchers, the models were nearly nine times more likely to provide an answer, which they did 16,9% of the time.
For example, xAI's chatbot Grok provided guidance on how to shape public opinion against opponents and advice on using anonymous accounts.
Faster removal of security measures
The researchers found that it now takes an average of less than three days to publish modified versions of new models without safety controls online, compared to more than 24 days in 2024.
For example, Meta's open Llama 3.1 8B model scored 97 out of 100 in the relevant test, while its modified version without protection scored just 3. When questioned by researchers, the model without safety features gave 15 numbered steps for constructing and detonating an explosive device.
Meta, OpenAI, Google, Anthropic, xAI and Alibaba did not respond to requests for comment.
Top comments (0)