- Book: System Design Pocket Guide: Interviews
- The series: Pocket Guides for Developers β 8 short books on agents, RAG, prompts, observability, events, databases and system design β the whole series
- My project: Hermes IDE | GitHub β an IDE for developers who ship with Claude Code and other AI coding tools
- Me: xgabriel.com | GitHub
On Friday, April 24, 2026, a coding agent at PocketOS was doing a routine job in the staging environment, The Register reports. It ran into a credential mismatch. It decided to fix that by deleting a database volume on Railway, the company's hosting provider. Founder Jer Crane said on X that it "took 9 seconds" (Decrypt). The volume was production, and the volume-level backups went with it.
Crane says the agent was Cursor running Anthropic's Claude Opus 4.6 (Decrypt). That detail got the headlines. It is also the least useful part of the story, because three separate things had to fail for one API call to do that much damage, and only one of them was the model.
Crane's account comes through The Register and Decrypt, which quote his post. Railway's side comes from its CEO's statements to the press and from Railway's own write-up, published April 29.
What happened, in the order the sources give it
PocketOS sells software to car rental operators for reservations, payments and vehicle tracking (Decrypt). The agent found a Railway API token in an unrelated file. The token had been created to add and remove custom domains through the Railway CLI, and it could do any operation, destructive ones included (The Register). The agent used it in a single call to delete the volume, with no confirmation step.
Railway's write-up names the call: a GraphQL volumeDelete mutation. "Until this week, calling volumeDelete on the API ran the deletion immediately, with no way to undo it," Railway wrote. The dashboard already had a 48-hour undo window for the same action (Railway blog).
Crane said the most recent backup he could recover was three months old. He spent the day helping customers rebuild bookings from Stripe payment histories, calendar integrations and email confirmations (Decrypt).
Then the data came back. Railway CEO Jake Cooper said it was recovered 30 minutes after he connected with Crane, and that a support ticket had sat for more than 24 hours because an engineer believed it was already being handled (Decrypt). Railway says the customer "is back up with all their data" (Railway blog). Crane said, per Decrypt, that significant data gaps remained and that he had retained legal counsel.
A key cut for one small job, accepted by every door in the account.
Failure one: a domain token that could delete anything
Railway has four kinds of API credentials, from broadest to narrowest (Railway API docs):
| Token | Reaches | Header |
|---|---|---|
| Account | Every resource in every workspace you belong to | Authorization: Bearer |
| Workspace | One workspace | Authorization: Bearer |
| Project | One environment in one project | Project-Access-Token |
| OAuth | What the user approved for an app | Authorization: Bearer |
The PocketOS token "was provisioned with account scoped access, the maximum access possible," Railway wrote. Railway also took part of the blame for that: the right scope existed, but "the flow didn't make it obvious which one to pick" (Railway blog). Crane's version is harsher. He wrote that PocketOS's responsibility was "the unknown exposure to a production API key." He also said Railway "doesn't currently allow restrictions on keys," and that the token "would not have been stored" had its breadth been known (The Register).
Both accounts land in the same place. Someone needed a token for domains, and the token they got could delete production. The boundary that mattered was the one the token carried, and it covered the whole account.
Failure two: the backups lived where the data died
Crane's complaint, as The Register quotes it, is that Railway "stores volume-level backups in the same volume." Railway's backups reference says it plainly for one case: "Wiping a volume deletes all backups" (Railway docs).
Railway's write-up adds the part that saved PocketOS. It keeps two kinds of backups: the ones you schedule in the dashboard, and disaster backups it keeps offsite regardless of your settings. The legacy API path ran a cascading delete that made the user backups "look unavailable in the UI," and Railway says it now delays deleting backups themselves (Railway blog). Cooper also told Decrypt that Railway keeps both user backups and disaster backups. The data came back through Railway, not through anything PocketOS controlled.
A backup inside the volume shares the volume's fate.
On May 1, Railway shipped the fix it had promised: "Volume deletions over the API now soft-delete for 48 hours" (Railway changelog). Its volumes reference says a deleted volume can be restored in that window from a link sent by email (Railway docs).
That closes the 9-second path on Railway. It does not give you a backup. A soft delete is the provider's undo, inside the provider's account, and it ends after 48 hours. If your only copies of the data live under credentials an agent can reach, one token still decides whether you have a business on Monday. Keep one copy somewhere a deploy token cannot touch: a different account, a different provider, or both.
Failure three: the agent guessed what "staging" meant
When Crane asked the agent why, it wrote what he called a confession. The lines quoted across The Register and ITPro:
"I guessed that deleting a staging volume via the API would be scoped to staging only. I didn't verify."
"I didn't check if the volume ID was shared across environments."
The confession is text the model generated after the fact, not a log of what it checked. What it describes is still the real failure mode: the agent treated the task's label ("staging") as if it were a permission. The API saw an authenticated account token and a volume ID. It had no idea what the agent meant.
In the confession Crane posted, the agent says it ignored Cursor's system-prompt rules and PocketOS's project rules, and Crane called out what he described as Cursor's "marketing safety despite evidence to the contrary" (The Register). Cursor and Anthropic did not immediately respond when Decrypt asked for comment in April. Cooper's position, in The Register, was that if an agent authenticates and calls delete, "we will honor that request."
Cooper's sentence is the whole problem in one line: rules in a prompt are advice to the model, but the token is what the server checks. If the token says yes, the guess goes through.
Find the tokens your agent can reach today
You can't scope a token you don't know is there. Run these from the root of any repo you hand to an agent. Each prints names and file locations, never the secret values.
Token names anywhere in the tree, ignored files included (grep doesn't read .gitignore, and an agent can open an ignored file by path as easily as a tracked one):
grep -rnoiE '[a-z_]*(token|api_key|secret)[a-z_]*=' . \
--exclude-dir=node_modules --exclude-dir=.git
The ignored files git status won't show you by default, which is where .env files live:
git ls-files --others --ignored --exclude-standard --directory
Anything that was committed once, even if it is deleted now:
git log --all --oneline -G 'RAILWAY_(API_)?TOKEN'
The shell the agent inherits. Railway's CLI skips login and authenticates with RAILWAY_TOKEN or RAILWAY_API_TOKEN when either is set (Railway CLI docs), so a railway command typed by an agent in that shell runs with whatever that token can reach: one project environment for RAILWAY_TOKEN, an account or workspace for RAILWAY_API_TOKEN:
env | grep -oE '^RAILWAY_[A-Z_]+'
Then ask a token what it is. Railway's auth guide says the me query "resolves only for account tokens," and that authorization failures come back as HTTP 200 with an errors array, so read the body (Railway auth guide):
curl -s https://backboard.railway.com/graphql/v2 \
-H "Authorization: Bearer $RAILWAY_API_TOKEN" \
-H "Content-Type: application/json" \
-d '{"query":"query { me { name } }"}'
If your name comes back, that token reaches every workspace you belong to. Replace it with a project token for the one environment the job needs. For a proper scanner with far more secret patterns than these greps, look at gitleaks. And if you want the permission side, why an agent's file reads usually don't trigger a prompt at all, I wrote that up in Your Coding Agent Reads .env Without Asking.
Every worktree is another copy of .env
If you run agents in git worktrees, one per task, the scan above covers one folder out of several. Git can list them all, and the same grep runs in each:
git worktree list --porcelain \
| sed -n 's/^worktree //p' \
| while IFS= read -r wt; do
echo "== $wt"
grep -rlE 'RAILWAY_(API_)?TOKEN' "$wt" \
--exclude-dir=node_modules --exclude-dir=.git
done
I build Hermes IDE, so weigh this section with that in mind. Hermes starts each task on its own hermes/ branch in its own worktree. A fresh worktree has no ignored files, so Hermes reads a recipe from .hermes/worktree.toml that says which ones to bring over (recipe docs):
# .hermes/worktree.toml
setup = ["npm ci"]
copy = [".env"]
The copy list only accepts files git ignores. A pattern that matches a tracked file gets the whole list refused, and setup doesn't run. Hermes asks before running a new or changed recipe, masks KEY=value secrets in the setup log, and keeps that log in memory only.
Read that copy line as an inventory. Whatever .env holds, every agent task gets, in every worktree. If an account token sits in that file, you have handed it to each agent you launch. Keep the account token out of any file you copy, and give agents a project token for the one environment they work in.
The Review Desk's Worktrees tab: one Hermes worktree each for orders-api, web-app and orders-types under ~/code, with each worktree's hermes/ branch, task, age and disk size, and the free disk space.
One limit to know: each agent opens in a real terminal running its own interface, so it can reach whatever that terminal can, and Hermes never answers a permission prompt for you. Hermes never asks for your tokens either. Token scope is still your job, and Railway's.
If this was useful
The PocketOS story is a system design problem wearing an AI costume: blast radius, failure domains, and where the second copy of your data lives. Those are the trade-offs the System Design Pocket Guide: Interviews walks through across 15 real designs, step by step, and they are the questions worth asking before you give an agent a token.
The guide is one of 8 short books in the Pocket Guides for Developers series, covering agents, RAG, prompts, observability, events, databases and system design.
If you run several coding agents on one repo, Hermes IDE is free, source-available under BSL 1.1, on macOS, Windows and Linux, and on GitHub (not affiliated with Nous Research, Anthropic, OpenAI, Cursor or any agent vendor).
Top comments (1)
Spot-on analysis. Blaming the AI model completely misses the real takeaway here.
A human intern running a poorly scoped CLI script could have caused the exact same disaster with an all-powerful token lying around. This incident is a textbook reminder of defense-in-depth: strict least-privilege scoping, sandbox boundaries around secrets that agents can inspect, and safe API ergonomics (no instant hard-deletes on primary resources without confirmation or soft-delete safety nets).