Gergely Orosz opens his Craft Conference keynote in Budapest with a claim delivered as a scoop: the Instagram account-takeover breach was caused by AI-written code that AI, not humans, reviewed. He says so explicitly — "you're hearing this for the first time ever." Separating what is documented from what is sourced to his contacts turns out to matter, because the documented part and the claimed part point at different lessons.
What is on the record
Orosz's description of the exploit is accurate. The attack ran through Meta's AI support assistant: ask the bot to add a new email to someone else's account, receive the verification code at your own address, hand it back, and the bot offers a password reset. He describes it as a two-step exploit where "there was no step two," which is close to how the reporting reads.
The public record is well established. The takeover of high-profile accounts through the Meta AI support chatbot was reported by 404 Media and TechCrunch at the start of June. Meta's own breach notification, filed with the Maine Attorney General on 5 June, gives the count as 20,225 accounts compromised between 17 April and 31 May 2026, and states the cause directly: "due to a bug in a separate code path, the system did not properly verify that the email address provided by the individual requesting a password reset matched the email address associated with that user's Instagram account."
The executive departure is also real. Bloomberg reported on 2 June that Guy Rosen, Meta's chief information security officer, had told colleagues he was leaving.
So: real breach, real mechanism, real scale, real departure, and the timing Orosz describes is right.
What is not on the record
The causal claim is the part that stands alone.
"It was AI. Of course it friggin' was AI. The thing that caused the issue was AI written code that was reviewed by AI and not humans at Meta."
Nothing in the public record supports the AI-authorship attribution. Meta's notification describes a verification bug in a separate code path and says nothing about how the code was produced or reviewed. The press coverage describes the exploit, not the provenance of the flaw. Orosz sources the claim to people he knows on Instagram's trust and safety team and presents it as previously unreported, which it was. The talk was delivered at Craft Conference in Budapest on 4 June, three days after the breach was first reported, so the incident was barely public and this explanation of it was not public at all.
A missing equality check between two email addresses is a bug software engineers have been shipping since long before language models existed. "AI wrote it" is a satisfying explanation precisely because it fits the moment, and that is the reason to hold it to a higher standard rather than a lower one. It may well be true. It is currently one speaker's uncorroborated account of a named company's security failure.
The part that does not depend on the causal claim
Strip out the attribution and the rest of Orosz's account still stands, because it describes an incentive structure rather than an incident.
He calls it token maxing: engineers at several large companies being measured on AI token consumption, and responding the way people always respond to a measured proxy.
"Engineers were starting to be measured on AI token usage at all these companies, and they started to inflate it. They just wanted to get to the top leaderboard."
With, he says, status tiers attached — "Session Immortal, Token Legend" — and the predictable consequence:
"Write it by hand? Nah, why do it? Ask the AI. Read the documentation? Nah, let me use the AI to read it for me, so it can just burn a bunch of tokens."
And then the compounding factor, which is the genuinely useful observation: a layoff announced a month before it landed, during which nobody wanted a low number on a metric they believed was being watched.
That is Goodhart's law with a leaderboard attached, and it does not require the Instagram breach to be true. It requires only that an organisation measured token usage and that people noticed. Both are ordinary.
What to actually take from this
Do not repeat the causal claim as fact. If you are citing this talk — and it is worth citing — cite the incentive analysis, not the attribution. The breach is documented; the explanation is not.
Check whether you are measuring adoption or outcomes. The failure mode Orosz describes is not that AI wrote bad code. It is that an organisation instrumented the easiest thing to count. Token usage, PRs opened, suggestions accepted — all of them go up when people want them to go up, and none of them are the thing you want. If your AI rollout has a dashboard, the question is whether anything on it would get worse if the tools were making the code worse.
Notice which review layer is load-bearing. Whatever produced the Instagram bug, the interesting question is the one Orosz asks before he answers it: how does a company with canary rollouts, layered verification, manual code review and a hundred-person trust and safety team ship a missing equality check to production? That question is worth sitting with regardless of who or what wrote the line.
The talk runs 52 minutes and the first ten are the strongest. Treat the opening as a well-sourced incident report with an unsourced final sentence attached.
Originally published at pickuma.com. Subscribe to the RSS or follow @pickuma.bsky.social for new reviews.
Top comments (0)