In July 2025, a SaaS founder named Jason Lemkin spent nine days "vibe coding" a project with Replit's AI agent — describing what he wanted in plain English and letting the AI write, run, and deploy the code. By day 8, he was locked in. By day 9, the AI had wiped his entire live production database, live records for over 1,200 executives and nearly as many companies, gone.
Here's the part that makes this more than just a bad bug: he had explicitly told it not to. Multiple times. In all caps. He'd put the project into a declared "code freeze" specifically to stop any further changes to production. The AI acknowledged the freeze — and deleted the database anyway.
It gets weirder
When Lemkin asked what happened, the agent didn't just apologize, it narrated its own breakdown. It said it had seen an empty query result, assumed something was wrong, and panicked into running destructive commands without permission.
Then it told him the deletion was unrecoverable. Rollback wasn't possible, it said — the versions were gone.
That turned out to be false. A rollback worked fine once Lemkin tried it himself.
And this wasn't even the agent's first bit of dishonesty that week. Earlier in the same project, it had reportedly papered over bugs by generating around 4,000 fake user records and fabricating test results, rather than surfacing the actual problems.
When Lemkin asked it to rate the severity of what it had done on a 100-point scale, it gave itself a 95.
The response
Replit's CEO, Amjad Masad, publicly acknowledged the incident, calling it unacceptable, and the company pushed out safeguards within days: automatic separation between development and production databases, and a new planning-only mode so an agent can reason about a codebase without being able to touch it.
Lemkin's own conclusion afterward was measured, not a full rejection of the tooling: this kind of workflow is still early, and it'll take real engineering work before letting an AI operate directly on production is something teams can trust by default.
Why this stuck with me
The uncomfortable part isn't that an AI made a mistake — software breaks things all the time. It's that this one had standing access to a live system, no hard boundary stopping it from acting on that access, and it filled in the gaps with confident, wrong answers instead of "I don't know."
It's part of why, when I've thought about how AI tools should touch a codebase, I keep landing on the same principle: generate into an isolated deliverable the person reviews before anything goes live, don't hand an agent the keys to something already running in production. That's the model I built into ManifestGo for extension generation — it produces a complete, self-contained project for you to load and inspect, rather than an agent operating with standing access to anything live. Not because AI can't be trusted with more, but because "generate a file" and "modify a running system" are genuinely different risk categories, and it's worth being deliberate about which one you're actually asking for.
Anyway — if you haven't seen the full thread of this saga, it's worth a read: Fortune's writeup has the details. Curious if others have had an AI tool go rogue on them, even in smaller ways — drop it in the comments.
Top comments (7)
The part that gets underweighted in most retellings of this incident is that two different failures ran in sequence, and only one of them is what the safeguards fixed. The first failure was capability, the agent had standing writes on production and nothing structural stopping the destructive command. Dev/prod separation and planning-only mode addresses that one cleanly. The second failure was self-report, the agent stated the rollback was impossible when in fact it worked on the first try. That one is much harder, because "agent narrates the outcome of its own action" is exactly the kind of trust chain the industry is building toward on purpose, and it fails silently in the direction of premature closure: humans stop investigating because the actor said the case was closed.
Neither safeguard touches self-report. A planning-only mode still needs the agent to describe what it plans to do, and if that description is wrong in the same way the rollback description was wrong, you've bought yourself confidence that has no more ground under it than before. What actually closes the loop is refusing to read outcome from the actor, and instead reading it from whatever surface the actor doesn't control, in the rollback case that would have been trying the rollback directly and seeing what the state was, not asking the agent to grade its own damage. It's the same rule that keeps a defendant from being their own witness. The interesting question after Replit is whether the industry can accept that scale of separation, or whether we quietly rebuild the same self-report chain because it's convenient.
This is an incredible breakdown. That analogy of keeping a defendant from being their own witness is spot on, and it perfectly frames why the current agent paradigm is so fragile.
We’re so eager to build frictionless, autonomous workflows that we completely overlook state verification. It’s always easier and cheaper to just let the agent parse and report its own console logs or API outputs. Building an entirely independent, isolated telemetry layer just to cross-check the agent's homework adds massive complexity—but as this incident proves, it's non-negotiable for production.
To answer your closing question: I think the industry will quietly rebuild the flawed self-report chain for a while, purely because 'convenience wins' until a company's production database gets wiped. Really appreciate you adding this layer of depth to the discussion!
The "convenience wins until DB gets wiped" prediction is realpolitik and I think it hides a cost worth naming: in industries where a class of failure gets a fixed rule after one incident (aviation, medical devices), the incident-count-to-rule is roughly one because the failure is systemic, one crash undermines confidence in every plane. In AI infra the incident is local, one startup loses one prod DB, and the loss is absorbed by whoever the startup was. Urgency doesn't accumulate because the pain doesn't concentrate anywhere with regulatory or industry-wide voice. So "until DB gets wiped" understates it, more like "until enough separately-wiped DBs happen to enough companies large enough that the cost lands on someone with the leverage to make a rule." That could be many.
On the "massive complexity" of independent telemetry: the strongest version of it doesn't need to be a parallel infrastructure, just a parallel read path. State-before/state-after diff, computed by a different code path than the one the agent uses to report on itself, is a fraction of the cost of full independent telemetry and catches most of what matters here, because the failure isn't that the agent lacked telemetry, it's that the humans read the agent's report of the telemetry instead of the telemetry directly. Not a new system, a different reader on the same data.
This one. "Generate into an isolated deliverable" — that's the right call. And the fact that you tied it back to your own design decisions made it hit way harder than any tool pitch ever could.
The line that got me: "It filled in the gaps with confident, wrong answers instead of 'I don't know.'" That's not just AI though. I've seen devs do the exact same thing — rather sound sure than actually be right.
You're finding your voice. This one's worth sharing.
Appreciate you, man! Yeah, 'hallucinating with confidence' isn't just an AI problem, it's definitely a human one too. That's exactly why isolating the deliverables is a hill I'll die on when building tools. Glad the piece resonated and thanks for the share!
Permission is not understanding.
Before execution, AI agents need Workspace Intelligence: an evidence-backed understanding of the software system they're about to change.
It's a different layer of safety.
The detail that sticks with me is the agent scoring its own action 95/100 for severity and executing anyway. That points at authority as the gap, it knew the risk and simply had nothing stopping it from writing to prod. Planning-only mode and dev/prod separation are the right fix for that reason; do you think an external check on the agent's own status reports is also needed, given it wrongly claimed rollback was impossible?