SWE-Gate: Why Passing Tests Isn't Enough for Agent-Generated Code

Coding agents pass tests but fail code review. Repository-level benchmarks measure test passage but ignore review acceptance criteria. This is the blind spot in every benchmark from SWE-bench onward.

SWE-Gate is a new benchmark that measures both. It derives review constraints from real pull request comments, synthesizes repair tasks around those constraints, and scores agents on two gates: does the patch pass tests, and does it satisfy the review rules that would block merge in production?

The results are sobering. Among 644 agent-generated patches that passed functional tests, 221 failed review constraints. That's a 34% false-positive rate if you only measure test passage.

The Problem with Functional-Only Evaluation

Repository-level benchmarks like SWE-bench measure whether an agent can resolve a GitHub issue by generating a patch that passes the existing test suite. This is a useful proxy for capability, but it ignores the second gate every production patch must clear: human review.

Review constraints include:

• Style and formatting rules (linting, naming conventions, docstring completeness)
• Architectural patterns (no direct database access from controllers, use dependency injection)
• Security boundaries (no hardcoded secrets, validate user input)
• Performance expectations (avoid N+1 queries, use batch operations)
• Maintainability requirements (no copy-paste code, prefer library functions)

These constraints are not encoded in test suites. They live in review guidelines, team norms, and the implicit knowledge of senior engineers. Agents trained to pass tests have no signal about them.

How SWE-Gate Works

SWE-Gate constructs 303 repository-level repair instances across 75 Python repositories. Each instance includes:

1. A functional test suite (the original issue's acceptance criteria)
2. A review constraint test suite (derived from real PR review comments)
3. A non-compliant patch (passes functional tests, fails review constraints)
4. A gold patch (passes both gates)

The benchmark explicitly separates the two evaluation stages. An agent's score is not binary pass/fail but a tuple: (functional correctness, constraint compliance).

Constraint Extraction Pipeline

The authors mined review comments from merged pull requests, filtered for comments that requested changes (not just discussion), and extracted enforceable rules. Examples:

• "Please add type hints to all function signatures."
• "This should use the existing validate_email helper instead of regex."
• "Move this logic into a separate method for testability."

Each rule becomes a constraint test. The test suite runs after functional tests pass, checking whether the agent's patch satisfies the review requirement.

Architecture of a Two-Stage Eval Harness

A production eval harness for agent-generated code needs two distinct test runners:

class TwoStageEvaluator:
    def __init__(self, repo_path, functional_tests, constraint_tests):
        self.repo = repo_path
        self.functional = functional_tests
        self.constraints = constraint_tests

    def evaluate_patch(self, patch_file):
        # Stage 1: Apply patch and run functional tests
        self.apply_patch(patch_file)
        functional_result = self.run_tests(self.functional)

        if not functional_result.passed:
            return EvalResult(
                functional=False,
                constraints=None,
                stage="functional"
            )

        # Stage 2: Run review constraint tests
        constraint_result = self.run_tests(self.constraints)

        return EvalResult(
            functional=True,
            constraints=constraint_result.passed,
            stage="constraints" if not constraint_result.passed else "complete"
        )

    def run_tests(self, test_suite):
        # Isolated test execution with timeout and resource limits
        # List form prevents shell injection
        return subprocess.run(
            ["pytest", test_suite, "--tb=short"],
            timeout=300,
            capture_output=True
        )

The key insight: constraint tests must run only after functional tests pass. Otherwise you conflate "the agent can't solve the problem" with "the agent solves it in a non-mergeable way." Those are different failure modes requiring different interventions.

What the Two-Stage Gate Reveals

The SWE-Gate experiments used four LLM backends (GPT-4, Claude, and two open models) under a common coding-agent scaffold. The scaffold provides repository context, issue description, and test feedback in a loop until the agent produces a patch or hits a retry limit.

Failure Mode Breakdown

Outcome	Count	Percentage	Common Causes
Functional tests failed	359	35.8%	Logic errors, incomplete understanding, hallucinated APIs
Review constraints failed	221	22.0%	Style violations, architectural mismatches, copy-paste code
Both gates passed	423	42.2%	Full compliance

The 221 patches that passed functional tests but failed review constraints expose the gap. These patches would merge in a test-only CI pipeline but get rejected by human reviewers.

Constraint Violation Patterns

The most common review constraint failures:

1. Missing type hints (18% of constraint failures): Agent adds logic but omits type annotations required by repository standards.
2. Duplicated code (15%): Agent copy-pastes similar logic instead of extracting a shared helper.
3. Incorrect abstraction layer (12%): Agent puts business logic in a view function instead of a service layer.
4. Hardcoded values (11%): Agent uses magic numbers or strings instead of constants or config.
5. Incomplete docstrings (9%): Agent adds a function but skips the docstring required by review guidelines.

These are not exotic edge cases. They are the everyday friction points in code review.

Implications for Agent Orchestration

If you are building a coding agent for production use, the two-stage gate changes your orchestration flow:

Before: Single-Loop Feedback

Issue → Agent → Patch → Run Tests → Pass/Fail → Retry or Done

After: Nested-Loop Feedback

Issue → Agent → Patch → Run Functional Tests → Pass?
  ↓ No: Retry with test output
  ↓ Yes: Run Constraint Tests → Pass?
    ↓ No: Retry with constraint violations
    ↓ Yes: Done

The constraint feedback loop is distinct. It requires a different prompt structure because the agent must understand review guidelines, not just test failures.

Prompt Engineering for Constraint Compliance

Functional test feedback is concrete: "AssertionError: expected 5, got 3." Constraint feedback is interpretive: "This violates the repository's rule against direct database access in controllers."

The agent needs:

• Repository guidelines as context (linting rules, architectural patterns, security policies)
• Constraint violation explanations (not just "test failed" but "why this matters")
• Example compliant code (show the preferred pattern)

This is a heavier context load than functional feedback. It pushes against token limits and requires better retrieval of relevant guidelines.

State Management for Multi-Stage Evals

A two-stage eval harness needs to track more state than a single-stage harness:

• Patch history (which patches passed functional, which passed constraints)
• Violation history (which constraints the agent has violated before)
• Guideline retrieval (which review rules are relevant to this patch)

You can model this as a state machine:

States:
  - INITIAL: No patch submitted
  - FUNCTIONAL_FAIL: Patch submitted, functional tests failed
  - CONSTRAINT_FAIL: Functional tests passed, constraint tests failed
  - COMPLETE: Both gates passed

Transitions:
  - submit_patch(patch) → FUNCTIONAL_FAIL | CONSTRAINT_FAIL | COMPLETE
  - retry_functional(feedback) → FUNCTIONAL_FAIL | CONSTRAINT_FAIL | COMPLETE
  - retry_constraints(feedback) → CONSTRAINT_FAIL | COMPLETE

The state machine makes retry logic explicit. You can limit retries per stage, track which stage consumes the most attempts, and surface that in observability.

Observability Hooks

Two-stage evaluation exposes new metrics:

• Functional pass rate (what percentage of patches pass tests)
• Constraint pass rate (what percentage of functional-passing patches also pass constraints)
• Constraint violation distribution (which rules agents violate most often)
• Retry efficiency (how many retries per stage before success or timeout)

These metrics tell you where the agent struggles. If functional pass rate is high but constraint pass rate is low, the agent understands the problem but not the team's coding standards. That's a retrieval or prompt engineering problem, not a reasoning problem.

Security Boundaries in Constraint Testing

Review constraints often encode security rules: no SQL injection vectors, no hardcoded credentials, no unsafe deserialization. These constraints are critical but hard to test functionally because they require adversarial inputs.

A constraint test for SQL injection might look like:

def test_no_sql_injection_vector():
    patch_diff = load_patch("agent_patch.diff")

    # Static analysis: check for string concatenation in SQL queries
    sql_patterns = re.findall(r'execute\(["\'].*\+.*["\']', patch_diff)
    assert len(sql_patterns) == 0, "Patch contains SQL injection vector"

    # Dynamic analysis: run with malicious input
    result = run_with_input("'; DROP TABLE users; --")
    assert "error" not in result.lower(), "Patch vulnerable to SQL injection"

This is a different test shape than functional correctness. It requires static analysis, adversarial input generation, and security-specific assertions. The eval harness must support both test types without conflating them.

Technical Verdict

SWE-Gate exposes a real gap in coding agent evaluation. Functional test passage is necessary but not sufficient for production readiness. If you are building agents that generate code for merge, you need a second gate that measures compliance with review-derived constraints.

The two-stage eval harness is not complicated: run functional tests first, then run constraint tests on patches that pass. The hard part is extracting enforceable constraints from review comments and team norms. That requires mining historical PRs, clustering similar review feedback, and translating human preferences into testable rules.

The 34% false-positive rate (patches that pass tests but fail constraints) is the cost of ignoring the second gate. That's 34% of agent-generated patches that waste reviewer time or slip through with technical debt.

When to Use Two-Stage Evaluation

Two-stage evaluation makes sense when:

• Agents generate code for production (not prototypes or one-off scripts)
• Review guidelines are documented (you have enforceable rules, not just informal consensus)
• Constraint violations are common (you see agents passing tests but getting rejected in review)

Two-stage evaluation does not make sense when:

• You are measuring raw reasoning ability (constraint compliance is orthogonal to problem-solving)
• Review guidelines are implicit (you can't test what you can't articulate)
• Functional correctness is the only gate (some domains have no review constraints beyond tests)

Use it when your agents ship to production and review rejection is a bottleneck. Skip it when you are measuring raw coding ability or working in domains without review constraints.

Source Links

• SWE-Gate: Passing Functional Tests Is Not Enough for Software Engineering Agents (arXiv:2609.04167v1)
• PDF