It Passed CI. That Still Doesn't Make the Patch Safe.

Last week I merged a model-written patch in four minutes.

CI was green. Review said 'lgtm'. The diff was twelve lines.

Two days later a background job stopped retrying. Nobody noticed for a while.

That is the whole problem in one story. A green check feels like a verdict. It is not one.

I generate patches with models every day now. It changed how I merge them.

Below is the FAQ I keep answering, a table you can steal, and a gate script you can run before lunch.

The short version

Claim	What the check actually proves	Corrected model
'CI is green'	the tests you already had still pass	green means no regression in covered paths
'The diff is small'	few lines moved	risk lives in the files touched, not the line count
'I added tests'	assertions exist	a test written by the same author grades itself
'It looks idiomatic'	style matched the codebase	idiom hides semantics, and semantics fail

Keep that table open. Now the details.

Claim 1: 'Green CI means the change works.'

No. CI proves that your existing suite still passes on the new commit.

So ask yourself a sharper question. Does any test actually execute the lines you added?

Most repos cannot answer that quickly. A coverage percentage is not a diff view.

Two commands get you closer in ten seconds.

git diff --stat origin/main...HEAD
git diff --name-only origin/main...HEAD | grep -v -E '(^|/)(tests?|__tests__)/'

Read the second list carefully. Those are changed files with no obvious test neighbor.

If one of them touches retries, auth, or money, slow down.

The corrected model

Green means 'nothing I already test broke'. It never means 'the new path is correct'.

Treat CI as a regression alarm, not a correctness proof.

Claim 2: 'Small diff, small risk.'

Line count is a bad proxy for danger. Blast radius is the real measure.

A three-line change to a default timeout can take down every worker you have.

Ask these four questions instead.

1. Which files does the diff touch?
2. Do those files sit on a hot path?
3. Does it change a default, a limit, or an error branch?
4. What happens on the second failure, not the first?

Failure paths are where model patches go soft. Retries, fallbacks, except blocks, silent skips.

The corrected model

Risk scales with the semantics you touched. Count branches, not lines.

Claim 3: 'I added a test, so it is tested.'

Who wrote the test? If it was the same model, you have a self-graded exam.

The test asserts what the patch does. It does not assert what the patch should do.

I use one boring rule here. The test must fail on the parent commit.

git switch -c check/parent origin/main
# apply ONLY the new test file, then run it
git checkout patch/retry-fix -- tests/test_retry.py
pytest tests/test_retry.py   # expect: FAIL

If the test passes without the patch, it is not testing the patch.

That single check has caught more bad patches for me than any linter.

The corrected model

A test is evidence only if it can fail for the right reason.

Claim 4: 'Idiomatic code is safe code.'

Python has a few shapes that read fine and behave badly.

• except Exception: pass — silent failure, forever.
• def f(x, acc=[]) — one list, shared across every call.
• if x == None — works, then breaks on __eq__ overrides.
• for i in range(len(items)) — index math where you wanted items.

None of these are syntax errors. All of them survive code review.

Models reproduce them because the training data is full of them.

The corrected model

Idiom is about reading. Safety is about failure behavior. Check them separately.

The artifact: a patch gate that runs in one second

I keep this script in tools/. It needs plain Python 3 and git, nothing else.

It does five jobs.

1. Lists changed files against a base ref.
2. Flags lines that mention auth, retries, schema, or outbound I/O.
3. Flags four common smell patterns.
4. Blocks risky changes with no test file in the diff.
5. Exits non-zero, so it can sit in front of merge.

#!/usr/bin/env python3
# patch_gate.py - deterministic checks before merging a patch.
from __future__ import annotations

import re
import subprocess
import sys
from dataclasses import dataclass, field

BASE = sys.argv[1] if len(sys.argv) > 1 else 'origin/main'

TEST_HINT = re.compile(r'(^|/)(tests?|__tests__)/|(_test\.py$|\.test\.(js|ts)$|_spec\.rb$)')

RISK = {
    'auth': re.compile(r'password|secret|token|session|oauth', re.I),
    'retry': re.compile(r'retry|backoff|timeout|idempot', re.I),
    'schema': re.compile(r'ALTER TABLE|DROP COLUMN|CREATE INDEX|migrat', re.I),
    'io': re.compile(r'requests\.|httpx\.|fetch\(|socket\.', re.I),
}

SMELL = [
    (re.compile(r'except\s*:\s*(#.*)?$'), 'bare except'),
    (re.compile(r'except\s+Exception\s*:\s*pass'), 'swallowed exception'),
    (re.compile(r'def\s+\w+\([^)]*=\s*(\[\]|\{\})'), 'mutable default arg'),
    (re.compile(r'==\s*None|!=\s*None'), 'equality check against None'),
]


def git(*args: str) -> str:
    result = subprocess.run(['git', *args], capture_output=True, text=True, check=True)
    return result.stdout


def added_lines(path: str) -> list:
    diff = git('diff', '-U0', f'{BASE}...HEAD', '--', path)
    return [ln[1:] for ln in diff.splitlines() if ln.startswith('+') and not ln.startswith('+++')]


@dataclass
class Finding:
    path: str
    risks: set = field(default_factory=set)
    flags: list = field(default_factory=list)


def main() -> int:
    files = [f for f in git('diff', '--name-only', f'{BASE}...HEAD').splitlines() if f.strip()]
    tests = [f for f in files if TEST_HINT.search(f)]
    print(f'base={BASE} files={len(files)} test_files={len(tests)}')

    blocking = 0
    for path in files:
        if TEST_HINT.search(path):
            continue
        finding = Finding(path)
        for line in added_lines(path):
            for name, pattern in RISK.items():
                if pattern.search(line):
                    finding.risks.add(name)
            for pattern, label in SMELL:
                if pattern.search(line):
                    finding.flags.append(label)
        if finding.risks and not tests:
            finding.flags.append('risky change, no test file in diff')
            blocking += 1
        risks = ','.join(sorted(finding.risks)) or '-'
        flags = '; '.join(finding.flags) or '-'
        print(f'{path}: risk={risks} flags={flags}')

    return 1 if blocking else 0


if __name__ == '__main__':
    raise SystemExit(main())

Run it on the branch, not on main.

git switch -c patch/retry-fix origin/main
# ... apply the model patch ...
python3 tools/patch_gate.py origin/main; echo "exit=$?"

The sample output below is illustrative, not a benchmark.

base=origin/main files=3 test_files=0
src/worker/retry.py: risk=retry,timeout flags=risky change, no test file in diff
src/worker/queue.py: risk=- flags=-
docs/retries.md: risk=retry flags=-
exit=1

Running the gate inside a free-model loop

The gate is half the loop. The other half is what you send back to the model.

My sequence is short.

1. Generate the patch against a fresh branch.
2. Apply it, then run the gate and save the output.
3. Paste the gate output back verbatim, with a constraint.
4. Re-run the gate on the new diff.
5. Only then read the diff as a human.

python3 tools/patch_gate.py origin/main > gate.txt 2>&1
echo "exit=$?" >> gate.txt

The follow-up prompt I use, unchanged:

Fix only the flagged issues. Do not refactor untouched code.
Add or update one test that fails on origin/main.
Return a unified diff. No prose.

I run that loop with MonkeyCode. Its free model access covers the patch and repair steps. The free server option holds the scratch clone, so the gate runs next to the code it inspects. Both are availability claims from the product team, not numbers I measured.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Where this gate is wrong

Be honest about the edges. The script is regex, so it reads text, not meaning.

• A docs page mentioning tokens will trip the auth rule. Expect noise.
• It cannot see runtime behavior, feature flags, or production data.
• The smell list is Python-shaped. Port it before trusting it elsewhere.
• If origin/main is not your real base, the diff is wrong.
• Exit 1 is a prompt to look, not an automatic merge veto.

Who should not use this

• Teams where any automated gate becomes a rubber stamp within a month.
• Repos dominated by generated or vendored code, where every file looks risky.
• Codebases whose real risk sits in performance, migrations, or infrastructure.
• Anyone who cannot fetch the base ref in CI.

If that is you, keep the parent-commit test check. It needs no script.

What actually changed in my habits

Two habits, both small.

First, I read the file list before the diff. Every time.

Second, I run the parent-commit check on every model patch. No exceptions.

Green CI stopped being a finish line for me. Now it is a starting gun.

If you run the gate on your repo, which flag fires first?