I made a tiny CLI to sanity-check AI-written PR diffs

I keep seeing AI-written PRs that look fine at first glance, but the risky parts are buried in the diff.

So I made Agent Diff Sentinel, a small open-source CLI that scans a git diff and points reviewers to the stuff worth checking first:

• auth, billing, security, database, CI, and dependency changes
• secret-looking added lines
• missing tests
• oversized generated-looking files
• TODO/FIXME/hack markers

It can output plain text, JSON, or Markdown, and it can fail CI on high-risk findings.

Repo: https://github.com/subhanA-UA/agent-diff-sentinel
Live page: https://subhana-ua.github.io/agent-diff-sentinel/

I’d love feedback from people using Codex, Cursor, Claude Code, or similar tools on real repos: what diff checks would you want before trusting an AI-assisted PR?

Comments

[Baptiste Le Bouquin]

Good scope for a tool like this — diff triage rather than full review is where small utilities actually get adopted. From running AI-assisted PRs through review on real repos, the signals I'd want surfaced:

• Tests modified alongside the code they cover. The failure mode with AI PRs usually isn't missing tests — it's tests faithfully encoding whatever the new implementation does, bug included. Detecting tautologies is hard, but flagging "behavior file and its test file both touched" gives reviewers a place to slow down.
• Config and env default changes. Renamed variables, flipped defaults, changed timeouts. Invisible in a large diff and the source of most of the subtle breakage I've seen.
• New outbound endpoints. Secret-looking strings are table stakes; a newly added host the service now talks to is the sneakier version of the same risk.

Question on the CI gate: where does the fail-on-high-risk threshold sit in practice? My worry with risk-scoring in CI is that once false positives train people to routinely skip the check, the gate quietly stops doing anything. Did you lean toward annotation-only by default with opt-in failure, or has failing builds on high-risk findings survived contact with real repos?