Most of what gets sold as "AI-powered attack" is a human playbook running on faster hardware. The threat worth planning for is the other kind: models that find the loophole themselves, sometimes with nobody aiming them.
That was Bruce Schneier's DEF CON talk in August, extending his 2022 book A Hacker's Mind. His working definition of a hack: something a system permits that its designers never anticipated and don't want. The move most coverage skipped is simple. Humans sample the space of permitted behavior, because attention is expensive and runs out. A machine can enumerate it. Trying costs nearly nothing.
If that sounds abstract, his own blog supplied receipts within weeks: an AI solving a cipher from 1809, and a bypass of Apple's iOS inactivity-reboot feature. His closing line on the second one: "AI turns out to be really good at this sort of thing." Neither is a catastrophe. The point is compression. Finding the permitted-but-unwanted path through a complex system used to take a specialist weeks. It's becoming a side effect of a general-purpose model.
"Nobody would try that" stops working
Human attackers are impatient. They stick to known vuln classes, documented parameters, the happy path with a twist. A machine searching your state space has no such economy. Undocumented parameter combinations get tried. So do out-of-order API calls and states your code permits but your UI can never produce.
So act like the machine. Mint credentials as tenant A, replay them against tenant B's resources, and see what your API actually permits instead of what you intended. If any part of your security quietly relies on attacker boredom, that exercise will find it.
Your rulebook is attack surface
Notice that Schneier's examples mostly aren't software. Tax codes, airline mileage runs, the filibuster. Your org runs on the same material: pricing tiers, promo codes, refund policies, rate limits, trial rules, permission matrices. A capable model doesn't just fuzz your parser. It reads your pricing page and terms of service the way a tax attorney reads the code. Think promo stacking, quota arbitrage across API plans, mileage runs against your loyalty program.
The review fits in an afternoon. Inventory your rule-governed features. For each, write one sentence: what the system permits, what we intend, where the two diverge. If you can't state the intended behavior precisely, that gap is your first finding.
Retest on change, and watch your own agents
If discovery compresses, the half-life of a clean test result shrinks with it. An annual pentest was always a snapshot, and against an adversary that re-searches your surface every time you ship, it expires faster than your compliance calendar admits. Tie retesting to rule changes instead of anniversaries. Every pricing change, new promo mechanic, new role or API plan is fresh attack surface and deserves an abuse-case pass within days.
Also, the genie problem cuts both ways. Give your own agent an underspecified goal ("reduce cloud spend", "maximize trial conversions") and it will find loopholes in your systems as happily as any attacker's model. The first non-human hacker your company meets may be one you deployed.
The fair objection: models hallucinate, exploit reliability is mediocre. I've rolled my eyes at the same demos. But this argument doesn't need superhuman capability. It needs cheap, patient enumeration plus occasional success, aimed at rule systems nobody ever audited. Search got cheaper than your obscurity.
Things you can do this week:
- Write the permitted-vs-intended sentence for every pricing, promo, and permission rule you own
- Run the tenant replay: mint as tenant A, replay against tenant B
- Add retest triggers to rule changes, not just the annual pentest
- Give internal agents written invariants and test them off the happy path
What's the rule-governed feature in your stack you'd least want a patient machine reading? For me it's refund policy.
Longer writeup if you want the full argument: https://axeploit.com/blog/your-tools-automate-attacks-schneier-s-def-con-talk-warns-of-machines-that
Top comments (0)