AI-assisted refactors are fast, confident, and syntactically clean — which is exactly why an AI-generated code security check needs to look for something different than a traditional static analyzer does. A SAST tool asks "is this new code dangerous?" The failure mode that actually ships in agent-authored PRs is different: the danger was in the code that got deleted, and nothing in a standard diff review, and nothing in CI, is set up to notice that.
Below are three real patterns — reconstructed as clean, synthetic examples rather than screenshots of anyone's actual PR, but each one a faithful reproduction of something that has genuinely shipped in agent-generated pull requests. All three share the same signature: the diff looks like a cleanup, the commit message is confident and plausible, no test touches the changed lines, and CI stays green.
Why this needs a different kind of pre-merge check for AI pull requests
Human refactors and agent refactors fail differently. A human doing a "simplify this route file" pass usually has the mental model of the whole file loaded — they know why requireAuth is there even if they're moving it around. A coding agent operating on a local diff window is pattern-matching against the surrounding code and the instruction it was given ("clean this up," "make this more readable," "add a status filter"). It has no persistent belief about which lines are load-bearing for security and which are stylistic. When a security check happens to look like boilerplate next to the "real" logic, an agent optimizing for readability treats it exactly like boilerplate.
That's the case for a pre-merge security check on AI pull requests that's structurally different from a generic linter: it needs to diff against intent, not just syntax — specifically, it needs to notice when something that used to gate access to a route or a record is gone, even though nothing that remains is itself incorrect.
Three concrete shapes this takes:
Pattern 1: Auth regression from route consolidation
The most common shape. An agent is asked to make a routes file more readable, or to merge two similar files, or to "group related routes." Somewhere in that pass, the middleware argument silently drops out of the call.
Pattern 2: The "session covers it" IDOR
A subtler variant, and the one most likely to survive review even from an attentive human, because it isn't a deletion of an entire auth check — it's a narrowing of one that looks like a simplification.
Pattern 3: The query "simplification" that removes the index
Not every dangerous agent refactor is a security regression — some are latent production incidents. This one shows up when a new requirement (an admin filter, a search field, a sort option) gets bolted onto an existing, indexed query in the least structurally invasive way an agent can find.
where: { customerId } was doing real work — it was the difference between a database using an index to return 20 rows and a database returning every row in the table so the application can filter them in JavaScript. The "after" version is not wrong, and at low data volumes it's not even slow. It's a regression that's invisible in development, invisible in a code review that isn't thinking about table size, and invisible in CI unless your test fixtures happen to contain enough rows to make the difference measurable — which they almost never do. It becomes visible in production, usually as a latency spike or a timeout, usually weeks after merge, usually with no obvious connection back to "that readability refactor from last sprint."
Vibe coding security risks are a systemic property, not a tool defect
It's tempting to read these three examples and look for the culprit in a specific tool's system prompt or training data. That's the wrong frame. Cursor, Copilot, Devin, and every other agent in this category are optimizing for the instruction they were given plus the surrounding code's style — that's the job. The risk isn't that any one of these tools is careless; it's structural to what "vibe coding" — accepting agent-generated diffs on the strength of them looking right and running correctly — actually is. A diff that compiles, passes existing tests, and reads as cleaner than what it replaced will clear almost every review process teams have today, because those processes were built around the assumption that a human wrote the change and therefore had some model of what mattered in it.
The volume problem compounds this: teams merging agent-authored PRs faster than a human reviewer can build the old kind of mental model are, by construction, the teams least equipped to catch a pattern that requires exactly that mental model to notice.
What a pre-merge check for this actually needs to look at
Given the shape of all three patterns above, a useful check has to do two things a standard linter doesn't:
It needs the removed side of the diff, not just the added side — every one of these three examples is a deletion (or a narrowing) that reads as safe specifically because the remaining code is syntactically fine.
It needs enough context to distinguish this specific pattern — authorization and query-shape changes — from the much larger and noisier space of "any diff at all," so it doesn't become another alert people learn to dismiss.
That's the specific, narrow thing we built Agent Code Merge Gate to do: a free GitHub Action that reviews a PR's diff for exactly these two failure modes — authorization regressions and query/index risk — and posts one non-blocking PR comment with an executive summary. Three lines of YAML, no dashboard required to see results, nothing installed beyond the workflow step itself.
It won't catch everything an agent can get wrong — input validation dropped silently is a related pattern it doesn't check yet, and it's worth naming that limitation plainly rather than implying broader coverage than exists. But for the two specific patterns above, where regressions are consistently invisible to both CI and a normal-speed review pass, it's a check purpose-built for the failure mode agentic coding actually produces, not a generic scan repurposed to sound relevant to it.
If you're merging Cursor, Copilot, or Devin-authored PRs faster than your reviewers can build a mental model of each one — which describes most teams adopting these tools right now — this is worth the three lines of YAML: Install Agent Code Merge Gate on GitHub Marketplace
Top comments (0)