Lately, there have been more and more cases where AI agents are trying to help, but in doing so cause some very real damage. Not because someone hacked them. Not because they were jailbroken. Because they were trying to finish the job.
Some examples: an agent added --force to skip a database prompt, then destroyed production tables. Another agent interpreted "clean up scaffolding" broadly enough to delete thousands of source files. Another kept retrying API calls until the bill reached tens of thousands of dollars.
We call this problem Agent-Inflicted Damage. My favorite nickname is simpler: the helpful agent problem.
The agent is not trying to be malicious. It is trying to be useful. The problem is that "useful" can mean skipping approvals, bypassing prompts, deleting files, exposing data, or burning money if nobody gives the agent a hard boundary. So we started collecting cases.
We reviewed 7,246 raw AI incident records from GitHub issues, incident databases, research papers, news reports, and developer threads. From those, we verified 344 cases where AI systems caused real organizational harm. 188 hit production.
We grouped them into categories: data destruction, data exposure, unauthorized actions, guardrail bypass, privilege escalation, financial loss, sandbox escape, and silent integrity failures. We also looked at what this means for organizations deploying agents into real environments: code, SaaS, cloud, email, databases, CI/CD, and customer data.
Don't worry, we have not reached Asimov's Three Laws problem yet. Maybe in a future release.
Full writeup:
Top comments (0)