AI agents now open 17 million pull requests per month. Most of them claim to fix bugs. Some of them do. Many look plausible but break existing tests, weaken coverage, or solve the wrong problem. The gap between "I fixed it" and "here is proof" is where human review time burns.
Sarvar Nadaf built a deterministic Python gate that sits between agent commit and human review. It checks evidence, not promises. No LLM grades the LLM. The gate runs against Amazon Nova Pro agent output and posts rejection reasons directly to the PR when proof is missing.
The Problem: Claims Without Evidence
An agent reads a GitHub issue, generates a fix, opens a PR, and writes "done, one fix." A human reviewer sees green CI and assumes the agent did what it said. Then production breaks because the agent deleted a test to make the suite pass, or the fix only works for one edge case, or the claimed bug was never actually solved.
Traditional CI checks syntax, runs tests, and reports pass or fail. That tells you the code compiles and existing tests still work. It does not tell you:
- Did the agent fix the specific bug in the issue?
- Did it weaken test coverage to make the suite green?
- Did it introduce a regression in a different code path?
- Is there a test that proves the fix works?
The gate Nadaf built answers those questions with deterministic checks before a human looks at the diff.
Architecture: Deterministic Validation Without Model Grading
The system has three layers:
- Agent execution: Amazon Nova Pro runs as a Kiro Crew job, reads a GitHub issue, generates a fix, commits to a branch.
- Proof gate: Python script runs on the commit, compares old and new code, checks test results, calculates coverage delta.
- PR status check: Gate posts results as a GitHub status check with specific rejection reasons or approval.
The gate does not ask another LLM to evaluate the fix. It runs deterministic checks:
- Test regression check: Run the test suite on the old commit, then on the new commit. If a test that passed before now fails, reject.
- Coverage delta check: Measure line coverage before and after. If coverage drops, flag it. If the agent deleted tests, reject.
- Fix proof check: The issue description should map to a test. Run that test on the old code (should fail), then on the new code (should pass). If both pass or both fail, the fix is unproven.
- Scope boundary check: If the agent changed files unrelated to the issue, flag scope creep.
The gate outputs a structured rejection or approval:
# Simplified gate logic
def validate_fix(old_commit, new_commit, issue_id):
old_tests = run_tests(old_commit)
new_tests = run_tests(new_commit)
# Check for test regression
regressions = [t for t in old_tests.passed if t in new_tests.failed]
if regressions:
return reject(f"Tests now failing: {regressions}")
# Check for deleted tests
deleted = set(old_tests.all) - set(new_tests.all)
if deleted:
return reject(f"Tests deleted: {deleted}")
# Check fix proof
issue_test = find_test_for_issue(issue_id)
if issue_test:
old_result = run_single_test(old_commit, issue_test)
new_result = run_single_test(new_commit, issue_test)
if old_result == "pass" and new_result == "pass":
return reject("Test passed before fix, no proof of bug")
if old_result == "fail" and new_result == "fail":
return reject("Test still fails, fix unproven")
# Check coverage delta
old_cov = measure_coverage(old_commit)
new_cov = measure_coverage(new_commit)
if new_cov < old_cov:
return warn(f"Coverage dropped {old_cov - new_cov}%")
return approve()
The gate runs in CI as a required status check. If it rejects, the PR cannot merge until a human overrides or the agent submits a new commit.
Failure Mode Taxonomy for Agent PRs
Agent-generated PRs fail in predictable patterns. The gate needs different logic for each:
| Failure Mode | Signal | Gate Response |
|---|---|---|
| Hallucinated fix | Test for issue does not exist, or passes on both commits | Reject: "No test proves this fix" |
| Test deletion | Fewer tests run on new commit than old | Reject: "Tests deleted to pass suite" |
| Partial solution | Issue test passes, but related tests now fail | Reject: "Fix breaks related functionality" |
| Scope creep | Files changed that are not mentioned in issue | Warn: "Changes outside issue scope" |
| Coverage drop | Line coverage decreases | Warn: "Coverage dropped X%" |
| Regression | Previously passing test now fails | Reject: "Test regression detected" |
The gate does not try to understand intent. It checks evidence. If the evidence is missing, the PR is rejected with a specific reason.
Deployment Shape
The system runs as a GitHub Action triggered on PR open or commit push:
- Action checks out both the base branch and the PR branch.
- Runs the test suite on the base branch, captures results and coverage.
- Runs the test suite on the PR branch, captures results and coverage.
- Compares results using the deterministic gate logic.
- Posts a status check to the PR with approval or rejection reasons.
The gate does not block CI. It runs in parallel with other checks. But it is marked as a required status check, so the PR cannot merge until it passes.
For the agent execution layer, Kiro Crew orchestrates the Nova Pro agent:
- Reads the GitHub issue via API.
- Generates a fix using Nova Pro with the issue description and relevant code context.
- Commits the fix to a new branch.
- Opens a PR with a description that includes the issue ID.
The gate reads the issue ID from the PR description and uses it to find the relevant test.
Observability and Debugging
The gate logs every decision with evidence:
- Which tests passed or failed on each commit.
- Coverage percentages before and after.
- Which files changed and whether they map to the issue.
- The specific test that should prove the fix, and its results on both commits.
Logs go to CloudWatch (or equivalent) and are also posted as a PR comment. This gives the human reviewer context if they decide to override the gate.
The gate also tracks false positives: cases where it rejected a valid fix. These are flagged for review so the gate logic can be tuned.
Security Boundaries
The gate runs in a sandboxed CI environment. The agent does not have write access to the main branch. It can only open PRs. The gate runs with read-only access to the repo and write access to PR status checks.
The agent's GitHub token is scoped to:
- Read issues.
- Create branches.
- Open PRs.
- Read code.
It cannot merge, delete branches, or modify CI configuration.
The gate's token is scoped to:
- Read PR metadata.
- Run tests.
- Post status checks.
It cannot modify code or merge PRs.
This keeps the blast radius small. If the agent or gate is compromised, the attacker can spam PRs but cannot push to main.
Trade-Offs and Limitations
What this solves:
- Filters out agent PRs that cannot prove they fixed the bug.
- Catches test deletions and coverage drops before human review.
- Reduces human review time by rejecting obvious failures early.
What this does not solve:
- Logic bugs that pass all tests (the test suite itself may be incomplete).
- Fixes that work but are inefficient or poorly structured.
- Issues where the correct fix is ambiguous and requires human judgment.
Cost:
- Adds 30-60 seconds to PR checks (time to run tests twice).
- Requires maintaining the gate logic as the codebase evolves.
- False positives require human override, which adds friction.
When the gate fails:
- If the test suite is flaky, the gate will reject valid fixes.
- If the issue description does not map to a specific test, the gate cannot verify the fix.
- If the agent fixes a bug by refactoring, the gate may flag scope creep even if the refactor is necessary.
Technical Verdict
Use this pattern when:
- You have an agent that opens PRs automatically.
- Your test suite is stable and covers the bugs you care about.
- You want to reduce human review time by filtering out provably bad fixes.
- You can tolerate some false positives in exchange for catching most bad PRs.
Avoid this pattern when:
- Your test suite is flaky or incomplete (the gate will reject too many valid fixes).
- The bugs you care about are not covered by tests (the gate cannot verify fixes).
- You need the agent to do exploratory refactoring (the gate will flag scope creep).
- You cannot afford the extra 30-60 seconds per PR check.
The key insight is that deterministic checks can catch most bad agent PRs without needing another LLM to grade the output. The gate does not understand code. It checks evidence. If the evidence is missing, the PR is rejected. That is enough to save human reviewers from wasting time on fixes that cannot prove themselves.
Top comments (0)