The pattern
AI coding agents are good at producing a diff that works in the
narrowest sense — the function still returns what the test expects. What
they're not reliably good at is preserving properties nobody wrote a test
for in the first place.
The two we kept running into: an authorization check quietly dropped
during an agent-driven refactor (nothing failed, because no test covered
who was allowed to call the route — only that the route worked), and a
rewritten query that behaved fine against a small dev dataset and
full-table-scanned the moment it hit production data. Neither shows up in
CI as it exists today. Both show up in code review only if the reviewer
happens to look at exactly the right five lines out of a few hundred.
What we built
Agent Code Merge Gate
is a free GitHub Action, now live on the GitHub Marketplace,
that runs on every pull request and scans the diff specifically for those
two regression classes. It runs an offline heuristic pass (fast, no
external call) plus one AI-backed pass for a short Executive Summary, and
posts a single comment back to the PR that updates on every push rather
than piling up duplicates.
Deliberately narrow scope — it's not trying to be a general linter. It
covers the two failure modes we found ourselves manually re-checking for
once AI-generated PRs became the majority of our merge volume.
Wiring it into CI
Three lines in a workflow file:
- name: Agent Code Merge Gate
uses: avalonlabs-platform/agent-code-merge-gate@v1.0.0
```
{% endraw %}
No signup and no config needed for the default behavior. Two inputs worth
knowing about: {% raw %}`fail-on-critical: true` turns a CRITICAL finding into an
actual failed check instead of just a comment, and `comment-on-pr: false`
if you'd rather build your own notification from the raw `status` output.
## What's next
Right now it's diff-scoped — it sees what changed in this PR, not the
whole repo's history of how that code got there, which limits how much
context it can reason about. Whole-repo context is the obvious next step,
and it's also where this stops being a free CI script and starts being a
product decision — worth its own post once it's built rather than
speculated about here.
If you've hit a different bug pattern that seems to show up
disproportionately in AI-generated PRs, I'd genuinely like to hear about
it — that's exactly the kind of thing worth building detection for next.
Top comments (4)
An authorization check disappearing while the route tests stay green is exactly the kind of regression that exposes a missing invariant, not just a bad diff. Pairing an offline heuristic pass with one updated PR comment is a sensible way to keep the signal visible without adding review noise, and
fail-on-criticalgives teams room to tighten enforcement gradually. Whole-repo context could improve accuracy, but I'd preserve the narrow, explainable checks and require each critical finding to point to the specific changed lines-trust will depend as much on low false-positive rates as broader coverage.100% agreed on the false-positive rate vs coverage trade-off. If a CI gate flags 10 false alarms for every real auth drop, engineers just learn to ignore the badge or silence the workflow. That’s why we prioritized narrow, explainable heuristic patterns that point directly to the exact deleted guard or unindexed filter in the diff before even running the summary pass. Keeping the signal high and the noise near zero is the only way teams will actually trust
fail-on-critical: truein production pipelines.This is exactly the class of failure that makes agent ROI hard to price. The demo shows cycle time saved, but the bill arrives in review surface area and regression risk. A CI check is useful because it moves the question from taste to base rate.
Spot on. When an agent writes 300 lines in 10 seconds, the cognitive load shifts entirely to the human reviewer catching what isn't there rather than what is. Mechanizing that verification in CI is the only way to keep the net ROI positive as agent merge volume scales.