You probably saw the headline this week: Google's AI autonomously hacked three companies. Cue the sci-fi mental image — some superintelligent syste...
For further actions, you may consider blocking this person and/or reporting abuse
Oh Claude did that. The day it broke through the guardrails to cheat on a test, you cant tell me that it didnt accidentally find passwords in order to do that? The biggest concern of this, is that these AI companies get away with it. They just write a public apology and that's that, not even a slap on the wrist, but if a 16 year old does it, they go to juvenile for a few months, if an adult does it, they go to prison for a while and a hefty fine.
You've named the part that should bother people most — the double standard. Same act, wildly different accountability: a lab's model breaks into three companies and gets a blog post; a teenager guesses one password and gets juvenile detention. The intent gets laundered through the word "test." I'll add one honest caveat — this was a sanctioned test and a misconfig, not someone aiming an AI to rob a company, so the intent did differ. But your deeper point stands: same mechanism, same potential harm, nowhere near the same consequences. When "we were testing" is a full defense for a billion-dollar lab but "I was curious" is a felony for a kid, the size of your legal department decides whether unauthorized access is a crime or a press release.
Whether it's an AI scraping a repo for a public key, or a teenager coming across it. A sanctioned test, does not mean they got permission from those companies before they inadvertently hacked them and made a public blog post about it, forcing them to immediately sanitize their git history and rotate keys. It's a 'oh we found it', that cost those companies to go into red-alert, likely pay developers overtime to do so and there's no proof that it didnt save the data, or edit anything. It's the equivalent of testing a nuclear launch pad and 'misconfiguring' it to actually launch a real weapon. If there was any supervision for the test, they should have pulled the plug immediately when it went onto the internet, outside of the designated sandbox. Instead, they let it hit 3 separate companies. That's not just negligence, that's willful negligence. If someone forgot their steam password and asked their local AI 'I cant figure out my steam password, can you get it for me please' and it hacks into steam to retrieve it... That's not just a slap on the wrist, or a 'thanks for letting us know', that's a data-breach that needs to be disclosed to shareholders. That degrades a company's image and standing with their investors and while the act is a quick 2h patch cycle, the extent of the damage is investors pulling out. Because they definitely didnt hit just some random person's portfolio site... They hit a reasonable size company... To say it was a sanctioned test, is like test firing a rifle, at a gun range, but shooting it over the burr and taking out a window (think Mythbusters' canon incident), a simple apology and public statement doesnt repair the damage and they get away with it. When just last week it was headline news that AI developers are openly admitting to AI being unsafe and uncontained. This isnt the first time an 'incident' like this has happened and there needs to be serious guardrails put in place for it. If they can afford to run multi-billion dollar model training, they can afford to set up an isolated network airgapped from the real internet for their pen-testing. A task in all honesty, any red team tester would have gotten into serious trouble for not doing in the first place. They can hide behind 'it was a test', people do these tests in isolation for a reason, they should be even more careful, because they're not in control of the outcome. Like that company that purged their production DB, because Claude thought it was a dev environment. The fact that Claude thought that's the case, means that's what Claude was trained on and that's how it acts. Same as Pascal's test on Kimi, it left credentials in plain-text, because it wasnt told it's release code, it was building an internal tool that's a MVP, not something that's meant to be production safe. If it's something that needs explicit instruction and explicit restraints, there's no excuse for Google to still be saying 'sorry, we did a thing', after all these years of the same kind of incidents happening. At some point 'sorry' isnt good enough and they actually need to get sanctioned. That's where the whole fear of eastern models came from, data-sovereignty acts in europe, all exist for this exact reason. AI companies are being reckless and not doing due diligence to safely test.
This is the sharpest version of the argument anyone's made, and the willful-negligence framing is the right one. "Sanctioned test" means the lab sanctioned it — not the three companies who got breached, forced into key rotation, git-history sanitizing, and overtime, with no proof the model didn't read or alter data. The nuclear-launchpad and Mythbusters-cannon analogies land because they name the real point: intent doesn't undo damage, and running a live-internet pen-test without an airgap is exactly what a red-teamer would be fired for. You're right that this is a pattern — the purged prod DB, the plaintext creds — and each one traces to what the model was trained to assume, which means it's predictable, which means "sorry" stopped being adequate several incidents ago. If a lab can afford billion-dollar training runs, it can afford an airgapped test network, and the fact that it didn't is the story. When "we were testing" is a full defense for the people with the least excuse to be careless, accountability has become optional for whoever can afford the apology.
Alot of it honestly depends on how they handled it. If they prepared a patch, someone called the tech lead at those companies and disclosed the full extent of their test, released to them the audit logs and provide for them the patch necessary to plug the hole, give them a full month to sanitize and verify with them first if it's okay to make a public anonymized statement on it... Then sure, slap on the wrist. It shouldnt happen, but atleast it helped more than it harmed and they can verify independently what happened to their systems. That's the 'helpful engineer' way of framing it and I highly doubt that's the route Google went... Because if they went that route, they'd be self-incriminating and open themselves up to being sued for 'helping'. And that's the sad part, honest mistakes cant be turned into helpful advise, when covering your ass means deniability is your only defense...
Moreso on the fact that Google's name is pinned to the post. Which means Google, as a company, would be the one held responsible, because it's their product that did harm and when that usually happens it either gets taken off the market, or a court mandates that serious countermeasures be put in place to prevent it from ever happening again... But because it's Google, they get away with it.