Postmortem: The Review Gate That Approved Everything

A bad default can turn a review gate into a rubber stamp.

When a rule fails to parse, many gates fall back to approve. That one-liner converts a guardrail into a deployment pipeline. This post reconstructs that failure, walks a timeline, and ends with a negative-test fix.

The scenario below is a reconstruction. The code is runnable, and the failure is reproducible from the snippet alone.

The Failure Pattern

Production broke 90 minutes after merge. The patch touched auth/session.go. The review gate was configured to deny auth/**. It did not stop the merge. The config file still contained the rule.

The real bug lived in the loader, not in policy.

rules:
  - id: protect-auth
    paths: "['auth/**']"
    action: deny

YAML produced a JSON array turned into a string. The paths value was a string, not a list. The parser rejected it, and the loader converted corruption into an empty rule set.

import json

def parse_paths(raw: str) -> list[str]:
    try:
        return json.loads(raw)
    except json.JSONDecodeError:
        return []   # BUG: corruption becomes "no rules"

def decide(files: list[str], rules: dict) -> str:
    patterns = parse_paths(rules["paths"])
    for path in files:
        for pattern in patterns:
            if pattern in path:
                return "deny"
    return "approve"

Parse errors returned an empty list. The gate evaluated no patterns. Every path reached the approve default.

Silent corruption. No log. No error. Just a green check.

Repro

Run the snippet below to see silent approval:

python - <<'PY'
import json

def parse_paths(raw):
    try:
        return json.loads(raw)
    except json.JSONDecodeError:
        return []

def decide(files, rules):
    patterns = parse_paths(rules["paths"])
    for path in files:
        for pattern in patterns:
            if pattern in path:
                return "deny"
    return "approve"

rules = {"paths": "['auth/**']"}
print(decide(["auth/session.go"], rules))
# "approve" -- wrong. Expected: "deny"
PY

The gate denied nothing and still reported success. One config change did that.

Timeline (reconstruction)

Time	Event
00:00	Config migration writes the list as a JSON string.
00:01	json.loads fails; loader returns [].
00:02	Gate evaluates no rules and returns approve.
01:30	CI reports the gate as green.
01:31	Merge queue completes.
03:10	Production shows a 500 spike after session changes.
03:40	Rollback happens.
04:15	Team discovers every patch has been approved for hours.

The failure was not a flaky model. It was an error-handling decision made long before merge.

Contributing Factors

• Default-approve semantics. The gate treats "no rule hit" as safe.
• Parse errors fail silent.
• The gate had zero negative tests.
• Merge treats missing feedback as a green check.

Any one of those factors is tolerable. Together they form a deploy permit.

Fix 1: Fail Loud

Parse failure must crash the gate, not become an empty list.

def parse_paths(raw: str) -> list[str]:
    parsed = json.loads(raw)
    if not isinstance(parsed, list):
        raise ValueError("paths must be a JSON array")
    return [p for p in parsed if isinstance(p, str)]

Errors now fail the gate. No data, no approval.

Fix 2: A Negative Test Matrix

The gate needs known-denied paths. Run this in CI.

# test_gate.py
import json
import pytest
from gate import decide

CASES = [
    (["controllers/user.go"], "approve"),
    (["db/migrations/0042.sql"], "approve"),
    (["auth/session.go"], "deny"),      # negative case
    (["auth/middleware.go"], "deny"),   # negative case
]

@pytest.mark.parametrize("files,expected", CASES)
def test_gate_matrix(files: list[str], expected: str):
    rules = {"paths": json.dumps(["auth/**"])}
    assert decide(files, rules) == expected

The matrix knows that deny must exist. It turns review-gate expectations from prose into assertions.

Fix 3: Merge Must Block on Missing Checks

Some merge systems treat a missing required check as success. Disable that behavior.

A silent gate needs to leave the queue blocked, not completed.

required_status_checks:
  - review-gate

If the review gate cannot report, do not merge. Missing is not green.

Cheap Infrastructure for Probing

Gate tests need disposable runtimes. MonkeyCode offers free model access and a free server option; both fit throwaway probe workloads well. Spin up, run the matrix, tear down. Keep production decisions on infrastructure with contractual availability, not an unpaid hobby box. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The harness itself is vendor-neutral. Any compatible endpoint can run it. The fix is the matrix, not the backend.

Who Should Not Use This

• If review rules rarely change, the lightest fix is still a small negative matrix.
• If an AI agent rewrites config daily, the matrix must run on every change.
• The free server option is not a production firewall. Availability changes. Keep it for probes, not auth paths.

The Durable Fix

• Parse errors fail the gate.
• Negative tests run in CI.
• The merge queue stops when the gate cannot report.
• A green gate is not a tested gate.

Run the self-test before the next patch. The assumption that kept the rubber stamp alive is gone.