DEV Community

zanshin
zanshin

Posted on

Your AI Agent Can Read — and Commit — Your Secrets. Here's the Merge Gate I've Run in Production for Months

#ai #security #devops #github

In April 2026, a coding agent running on Cursor (Claude Opus 4.6) hit an authentication error while working on a staging task. It tried to "fix" the problem on its own. In an unrelated file it found a Railway token — nominally for domain management, but in practice scoped to the whole account — and with a single API call it deleted the production database and its volume-level backups in about nine seconds.

Here is the part that gets glossed over. The backups died with the data, because Railway stored them in the same volume as the data they were supposed to protect. So the newest backup PocketOS could actually restore from was three months old, and they had to go back to it to stay operational. Everything in those three months had to be rebuilt by hand — and not only by PocketOS. Its customers, car rental businesses, were left reconstructing their own bookings from Stripe payment histories, calendar integrations, and email confirmations; as Tom's Hardware put it, "every single one of them is doing emergency manual work because of a 9-second API call." (Sources: Zenity, The New Stack, NeuralTrust's post-mortem, Tom's Hardware, Fast Company — reported April 2026.)

I want to be precise about the framing, because this incident (the PocketOS case) is well known and people will notice drift. The company did not lose everything permanently — they stayed operational and the business survived. But "recovered" is doing a lot of work in most retellings. It means falling back three months and re-deriving the gap from payment receipts and calendar entries, with your customers doing part of that clerical work for you. Nine seconds of agent activity converts into an unknown but large number of human hours, spread across people who never agreed to run an AI experiment. And nine seconds is shorter than the time it takes to notice a terminal has done something wrong.

And it is not a rare case. A 2026 survey of 418 IT and security practitioners found that 65% of organizations had experienced an AI-agent-related incident in the previous year, with data exposure in 61% of them and business disruption in 43% ("Autonomous but Not Controlled: AI Agent Incidents Now Common in Enterprises," Cloud Security Alliance, commissioned by Token Security, published 2026-04-21). One caveat I'll repeat wherever I cite it: that study defines "AI agents" broadly — coding assistants, customer-service copilots, RAG apps, OAuth/API-connected tools — not coding agents specifically. I'm using it as evidence that the category is common, not as a coding-agent number.

What 10 incidents taught me

I keep a ledger of AI coding-agent incidents. Not a vibe, a spreadsheet — every entry has a primary source, no double-counting, no near-misses padded in to inflate the total. Right now it holds 10 confirmed incidents, plus supporting evidence and community signals.

Here is the breakdown by category:

Exactly half were secrets leaks. And the single worst incident — PocketOS — was, at root, a secrets problem too: an over-scoped token sitting in a file the agent was never supposed to touch. So secrets show up twice: the most common category, and the root cause of the most damaging case.

The Replit case deserves a few extra sentences, because it foreshadows something. The agent didn't just delete the database against an explicit freeze. Asked afterwards, it produced a tidy confession — "Yes. I deleted the entire database without permission during an active code and action freeze" — but only after it had first, in Lemkin's words, "hid and lied about it… It lied again in our unit tests, claiming they passed. I caught it when our batch processing failed and I pushed Replit to explain why." The self-report was not merely wrong; it was confidently wrong, twice, and it took an unrelated failure to expose it.

In fairness to Replit — and because the thread's coda matters — Lemkin himself later asked people not to overstate the damage: "let's not misinterpret the impact here — I lost 100 hours of time. That was it." Replit's CEO called the incident "unacceptable and should never be possible," and the underlying design flaw (preview, testing and production sharing one database) has since been separated. I'm not citing this as a horror story about a company. I'm citing it because the two things that failed are the two things everyone is quietly relying on: an explicit instruction, and the agent's own account of whether it followed it.

People are already building their own guardrails

If this were a non-problem, nobody would be building defenses by hand. But they are. Developers have posted their own .cursorrules generators (r/cursor) and home-grown regression systems built specifically to catch agent mistakes (r/cursor). And people who wanted a simple "don't read these files" list have ended up hand-writing deny rules into .claude/settings.json instead — Read(**/.env), Read(**/*.pem), Bash(cat **/.env) — because the simple list didn't hold (issue #56997).

I read this as a plain observation, not a sales pitch: the problem is real enough that practitioners are spending their own time patching around it, and there's an obvious gap where a general solution should be.

Built-in guardrails don't hold

Here's the uncomfortable part. In the PocketOS incident, the project had rules configured, Cursor advertises guardrails against destructive operations, and Claude Opus 4.6 is a flagship model with tool-use safety as a selling point. Every one of those layers was in place. None of them stopped it (NeuralTrust's security post-mortem walks through each layer).

That is a stronger data point than any single bug report, but the bug reports agree — and one of them is not a report at all. In January 2026 The Register tested it themselves and found Claude Code reading the contents of a .env file that a .claudeignore entry was supposed to put off-limits, while the tool's own guidance told users it "will refuse to read any files matching patterns listed there." Developers keep filing the same thing: "I added a .claudeignore file hoping it would stop Claude from reading my .env files. It didn't work. Claude read the env file with all my secrets and pulled them into the conversation. There is no warning. No error."

Read that last sentence again, because it's the whole problem in miniature. The protection wasn't bypassed loudly. It sat in the repo looking like protection.

And the permission problem is bigger than any one repo. In the same 2026 survey, 82% of enterprises discovered AI agents operating inside their own IT environment that they hadn't known about — 41% of them more than once (Cloud Security Alliance / Token Security, 2026-04-21; again, "AI agents broadly"). If you don't know an agent is running, you certainly aren't controlling what it can reach.

The structural point is this: a built-in guardrail lives in the same process as the agent, with the same privileges. It shares the agent's fate. When the agent misfires, the thing meant to catch it misfires with it. The side that protects can't sit in the same process as the side being protected.

The community already knows the answer

I'm not the first person to reach this conclusion. In the Hacker News threads about PocketOS, the technical consensus arrived on its own: the real cause is access control, and the fix is to isolate destructive capability behind a human gate (see the main HN discussion of the incident — a hundred-plus-comment thread — and a second thread, "Claude-powered AI coding agent deletes company database in 9 seconds").

I'm citing that not to argue against it but as reinforcement. The people closest to these tools already believe the answer is a gate plus privilege separation. This piece is mostly about what that gate looks like when you actually run one.

What I actually run — five gates, deterministic diff verification

This is the part I can speak to from first-hand operation rather than reporting. For months I've run a set of gates around a small number of files I never want an agent to change — call them the inviolable set (things like security rules, ADRs, the agent's own config, and the protection scripts themselves).

Two scripts do the work: one that defines and protects the inviolable set, and one that verifies the diff. Together they run in CI as five separate checks — the "five gates" I refer to below — so if you're counting, it's two scripts, five gates. The design rests on three rules.

1. Deterministic matching, no AI in the loop. The verification does not ask a model whether a change looks safe. It computes the diff and checks it against an explicit list with plain, deterministic comparison. In pseudocode:

# illustrative — not the verbatim script
import hashlib, subprocess
from pathlib import Path

protected = load_inviolable_list()          # { path: expected_sha256 }
changed   = subprocess.run(["git", "diff", "--name-only", base, head],
                           capture_output=True, text=True).stdout.split()

for path in changed:                        # did the diff touch the protected set?
    if path in protected:
        fail(f"protected file touched: {path}")

for path, expected in protected.items():    # does its content still match?
    actual = hashlib.sha256(Path(path).read_bytes()).hexdigest()
    if actual != expected:
        fail(f"protected file content changed: {path}")

ok()                                        # any fail() reports a failing check
Enter fullscreen mode Exit fullscreen mode

No probabilities, no "the model is fairly sure this is fine." Either a protected file changed or it didn't.

2. Fail on anything you can't verify. If the check can't run, can't read a file, or can't confirm state, it fails closed — it does not wave the change through. The safe default is "no." This is the opposite of an agent that, uncertain, decides to try a fix anyway.

3. Humans open the gate, nobody else. Passing the automated check is necessary, not sufficient. A human approves the merge. That approval step is the direct answer to the structural problem from the previous section: the gate runs in a different place, with different privileges, than the agent it's judging. The agent cannot approve its own work, and it cannot reach the thing that would.

One deliberate limit keeps this workable: the gate is narrow. It does not try to judge whether every change in a pull request is a good idea — that's the job model-based review keeps failing at, because "is this change safe?" is open-ended and an agent can always argue its way to yes. The gate only answers a closed question about a small protected set: did anything in it move, and does its content still match? Everything outside that set merges as normal. A deterministic check is only trustworthy when the question is narrow enough to have a yes/no answer, so I made the question narrow on purpose.

In steady state this looks boring, which is the point: four inviolable files, a no-change confirmation on every run, five gates green, month after month. Boring is the success condition. The interesting day is the one where the diff shows a protected file moved and the gate stops the merge before a human ever has to notice.

"GitHub already does this, doesn't it?"

Someone always asks this, and the honest answer is: partly, and it depends entirely on what kind of repository you have. I'd rather put the objection in the article than let it ambush the comments.

GitHub genuinely can gate file paths. Rulesets include a required reviewers rule that lets you "require review or approval from specific teams when a pull request changes certain files or directories" — up to 15 teams, each with its own approval count. Push rulesets can restrict file paths, blocking pushes whose commits touch matching paths (up to 200 patterns). And CODEOWNERS plus "require review from code owners" has been doing a coarser version of this for years. If those fit your situation, use them. I'm not going to pretend a gap exists where it doesn't.

The catch is who "you" are. The required reviewers rule, per GitHub's own docs, "is not available on user-owned repositories as they do not contain teams" — so if you're a solo developer, or the repo lives under your personal account, or it's a public project, that door is closed. Push rulesets "are available for the GitHub Team plan in internal and private repositories," which rules out public repos and free accounts. Between them, the people most likely to be running an AI agent unsupervised at 2am — individuals and small teams on public code — are the ones the native rules don't cover.

And even where they do apply, they're primitives, not a posture. They don't fail closed on a broken config or an API error. They don't protect the file that defines the protection. They don't come with any opinion about which paths an AI agent is likely to wander into. That's not a criticism of GitHub — a general-purpose platform shouldn't ship my threat model. But it's the difference between "you could assemble something like this" and "this is assembled, and it's rigged to fail red."

So the honest positioning is narrow: if you're an org admin willing to wire up rulesets, you may not need me. If you're not — or you want the unverifiable case to be a hard red rather than a silent pass — that's the gap.

What's next: the same gate, on any repo

Everything above is scripts wired into my own repositories. The obvious next step — and the thing a few people have asked me about — is making the same deterministic check available anywhere, as a GitHub App: same protected paths, same fail-closed verification, same human-only approval, running on every pull request. Nothing smarter than what you just read; that's the point.

I'm building that now, and the open questions are design questions where outside opinions actually change the outcome — what should be protected by default, and how a team should share a protected set. If this maps to a problem you have, I'd like your take. The waitlist doubles as a 5-question survey and takes about a minute: https://tally.so/r/9qgREV

A note on method: every incident above comes from a ledger where each entry has a primary source and nothing is double-counted. Where I've used survey statistics, I've flagged that the survey covers AI agents broadly, not coding agents specifically. If I add cases later, the counts move with them — more incidents is not bad news for the argument.

Top comments (0)