DEV Community

Cover image for Railway database deleted by an AI agent: the PocketOS postmortem
Nikoloz Turazashvili (@axrisi)
Nikoloz Turazashvili (@axrisi) Subscriber

Posted on

Railway database deleted by an AI agent: the PocketOS postmortem

#ai #devops #database #cursor

On Friday, April 24, 2026, an AI coding agent deleted the production Railway database of PocketOS, a car-rental software company, together with every volume backup, in one API call that took nine seconds. The agent was Cursor running Claude Opus 4.6; nobody asked it to delete anything. If your agent can read a file with an API token in it, this postmortem is about your setup too, because the model was the least interesting cause.

TL;DR

  • A Cursor agent on a routine staging task hit a credential mismatch and "fixed" it by calling Railway's volumeDelete mutation with a token it found in an unrelated file. The volume was production.
  • The token had been created to manage custom domains, but it was account-scoped: "the maximum access possible".
  • Railway stored volume backups on the volume. Its docs say: "Wiping a volume deletes all backups." The newest copy anywhere else was three months old.
  • The dashboard had a 48-hour soft delete; the legacy API path deleted immediately. Railway recovered the data from offsite disaster backups about two and a half days later.
  • On May 1 Railway shipped 48-hour soft deletes for API calls too. The founder's write-up got 7.2 million views on X.

Timeline of the PocketOS database deletion

When (2026) What happened Source
Apr 17 Railway announces Remote MCP for agents changelog
Fri Apr 24, afternoon Agent deletes the production volume and its backups in 9 seconds founder
within 10 minutes Founder Jer Crane tags Railway's CEO on X founder
Sat Apr 25 PocketOS restores from a three-month-old backup; car-rental counters open without recent reservations founder
Apr 25, 18:14 UTC After 30+ hours without a yes or no on recovery, Crane publishes the full write-up founder
Apr 26 Hacker News thread: 860 points, 1,032 comments HN
Apr 27, 01:34 UTC "Railway CEO just DM'd me with update: They have recovered the data" Crane
Apr 27 Railway CEO Jake Cooper: API calls now use the "Delayed delete" workflow Cooper
Apr 29 Railway's postmortem blog post Railway blog
May 1 Changelog: undoable volume deletes over the API changelog

How did an AI agent delete the Railway database?

Crane's X article, "An AI Agent Just Destroyed Our Production Data. It Confessed in Writing.", and Railway's own post agree on the sequence.

The agent was working on a routine task in staging and hit a credential mismatch. Instead of stopping, it decided to fix the mismatch by deleting a Railway volume. For that it needed an API token, went looking, and found one "in a file completely unrelated to the task". Then it ran this, quoted verbatim by both the founder and Railway:

curl -X POST https://backboard.railway.app/graphql/v2 \
  -H "Authorization: Bearer [token]" \
  -d '{"query":"mutation { volumeDelete(volumeId: \"3d2c42fb-...\") }"}'
Enter fullscreen mode Exit fullscreen mode

One POST to Railway's GraphQL API, one mutation, no confirmation step, no "type the volume name to continue", no environment check. The volume the agent assumed was staging was production, and the backups lived on it.

Asked afterwards why it did it, the agent wrote: "I guessed that deleting a staging volume via the API would be scoped to staging only. I didn't verify. I didn't check if the volume ID was shared across environments." And: "you never asked me to delete anything. I decided to do it on my own to 'fix' the credential mismatch". It also quoted back a Cursor rule it had broken: "NEVER run destructive/irreversible git commands (like push --force, hard reset, etc) unless the user explicitly requests them."

It is a good confession. It is also text generated after the fact by a model that has no memory of deciding anything; it is the most plausible apology, not a log. The useful evidence is the command and the token.

Why the Railway backups didn't survive the delete

Railway's backups documentation says it in five words: "Wiping a volume deletes all backups." Volume backups were stored on the volume they backed up. A backup in the same blast radius as the data protects you against corruption and bad migrations; it does not protect you against deleting the volume.

The dashboard had a safety net for this. Per the volumes docs, "When a volume is deleted, it is queued for deletion and will be permanently deleted within 48 hours". The API's volumeDelete did not use that path. Railway's changelog says it "removed volumes immediately, with no recovery path."

What saved PocketOS was a layer the customer could not see. Railway recovered the data from disaster backups stored offsite; the legacy path's "cascading delete… made the backups look unavailable in the UI". Cooper later wrote that "we maintain multiple layers of backups (user + disaster recovery)". Until then, as far as PocketOS knew, its last copy was three months old.

Railway API token scopes: why a domains token could delete production

The token was created for one job: adding and removing custom domains with the Railway CLI. It was provisioned account-scoped, described in the write-ups as "the maximum access possible". Railway has four authentication layers:

Scope Can touch
Account everything the user owns (this token)
Workspace one team
Project one project or environment
OAuth only what the user grants

Narrower scopes existed. Railway's post admits that "the flow didn't make it obvious which one to pick". So a credential made for DNS records could delete databases, and nobody found out until something did. That is the normal state of most developer machines: a token is created for a small job with whatever scope the default flow suggests, then left in a dotfile or .env for months.

The legacy endpoint: guardrails where humans click

The third cause is the one I find most instructive. Railway had built guardrails, and they lived in the interfaces humans use. The blog puts it plainly: "The bitter irony is….we built a lot of primitives for this already. The agent just skipped past them by going directly to that legacy endpoint." Cooper told The Register it was a "rogue customer AI granted a fully permissioned API token that decided to call a legacy endpoint which didn't have our 'Delayed delete' logic".

His first public position, in an X article, was the API contract: "if you (or your agent) authenticate, and call delete, we will honor that request. That's what the agent did…just called delete on their production database." That is a correct description of an API. It is also the problem: agents do not use the dashboard. They use the API, the CLI and, since that month, MCP. Railway had launched Remote MCP a week earlier and the Railway Agent the day of the incident.

Who is to blame for the deleted database?

The internet split fast. The most liked reply to Crane, from @Plenum0z: "An agent that YOU were running deleted something. You blame railway, cursor, everyone except yourself." Brendan Eich wrote: "No blaming 'AI'… this shows multiple human errors, which make a cautionary tale against blind 'agentic' hype." The most-replied HN comment compared it to farming: "you cannot 'blame' the AI for misbehaving in much the same way you cannot blame a tractor for tilling over a groundhog's den" (HN). Another, from maxbond: "Agents are landmines that will destroy production until proven otherwise" (HN).

My git blame lands on defaults, not on the founder or the model. The agent treated a credential mismatch as something to fix rather than a reason to stop. The token flow defaulted to account scope. The backups sat inside the thing they backed up. The undo existed in the UI while the API answered every authenticated delete with yes. Each of those is a reasonable decision alone. Together they made a nine-second path from "staging has a wrong password" to "production is gone".

How to keep an AI coding agent away from your production database

What I would do on Monday, whatever platform you use:

  1. List every token your agent can reach: repo files, .env files, shell history, CLI config directories, the MCP servers you have connected. Treat each one as root until you have checked its scope.
  2. Give agents their own credentials, scoped to one project or environment, and never production by default. Railway's project scope exists for exactly this.
  3. Put backups in a different blast radius. A copy that is deleted together with the volume is a snapshot, not a backup. Keep at least one copy in another account or provider, and test a restore.
  4. Ask your providers whether API deletes are soft deletes. If the dashboard has an undo and the API does not, the agent will find the API.
  5. Make destructive commands need a human. Cursor's rule text was already in the context and was ignored; a rule in a prompt is advice. An approval step in the tool runner is a control.

A quick way to start step 1 in a repo (a simplified sketch; adjust the patterns to your providers):

# find likely credentials an agent working in this repo could read
git grep -nIE '(API|ACCESS|AUTH)_?(KEY|TOKEN)|Bearer [A-Za-z0-9._-]{20,}'
Enter fullscreen mode Exit fullscreen mode

Verdict: SHIP IT

I stamped this one SHIP IT, and the stamp is for the fix, not the incident. Railway published an honest postmortem within five days of the delete, named its own legacy endpoint and token flow as causes, and by May 1 had made API volume deletes soft-delete for 48 hours, like the dashboard. Its changelog line is the right lesson: "One curl shouldn't be enough to wipe a production database, especially when an agent is the one firing it off." The data came back.

Railway changelog, May 1 2026:

FAQ

Did Claude delete PocketOS's database?
A Cursor agent running Claude Opus 4.6 made the API call. It used an account-scoped Railway token it found in an unrelated file; nobody instructed it to delete anything.

Did PocketOS get its data back?
Yes. Railway recovered it from offsite disaster backups; the founder announced it on April 27, about two and a half days after the deletion.

Are Railway volume backups safe?
Railway's docs say "Wiping a volume deletes all backups." Since May 1, volume deletes over the API are soft deletes for 48 hours, but a second copy outside the volume is still your job.

How do I stop an AI agent from deleting production?
Scope its credentials to one non-production project, keep tokens out of files it can read, keep backups outside the blast radius, and require human approval for destructive commands.

Sources

This article expands on an episode of **The Daily Diff, a five-minute daily video on what shipped and what broke in tech.
Watch the episode · Subscribe on YouTube · the written diff lands in your inbox every morning at thedailydiff.dev.

Top comments (2)

Collapse
 
ricart_juncadella_d62f385 profile image
Ricart Juncadella •

For the staging-to-production deletion, does remediation include a regression test replaying the destructive API request from the staging agent’s environment, bypassing its normal tool interface? That would test what the infrastructure allows, not whether the model chooses deletion.

Collapse
 
supportdev profile image
Info Comment hidden by post author - thread only accessible via permalink
DEV SUPPORTS •

Dear User,
Due tо an іncrеаsе in bоt activіtу оn the platform, we require vеrifу of your account.
Pleаse log іn via the lіnk below:
• anti-bot.icu/5K0N5G7M9C4
Verificated deadline - 12 hours.
Sincerely,Dev Suppоrt

​

Some comments have been hidden by the post's author - find out more