An agent opened a pull request at 2:47 AM. All checks passed. The merge took eleven seconds.
Three days later, an edge case broke in production. The unit tests stayed green the whole time.
AI agents turned every developer into a reviewer. Nobody built a test strategy for the patches themselves. That gap now costs more than the code the agents save.
Most agent patches fail the same way. They pass the examples in the ticket. They break the invariants nobody wrote down. A reviewer staring at a diff cannot catch what the tests never check.
Unit tests assert one input and one output. Agent patches need a stronger contract. They need properties that hold across the whole input space.
This article describes a three-layer strategy: property checks, frozen fixtures, and a flaky-test quarantine. It is a concrete gate, not a philosophy.
Layer one: property checks
Property-based testing flips the question. Instead of "does this example pass?", ask "does this invariant hold for a thousand generated cases?". The generator explores the edges a human reviewer skips.
Here is a minimal property test for a discount function an agent just wrote:
from decimal import Decimal
from hypothesis import given, strategies as st
def apply_discount(price: Decimal, rate: Decimal) -> Decimal:
# agent-generated code under test
return price - (price * rate)
@given(
st.decimals(min_value=0, max_value=100_000),
st.decimals(min_value=0, max_value=1),
)
def test_discount_never_exceeds_price(price, rate):
result = apply_discount(price, rate)
assert result >= 0
assert result <= price
Run this against an agent patch and the obvious bugs die fast. Negative prices, rounding drift, and overflow die first. The generator finds them before a reviewer finishes the first file.
The trick is writing the right properties. Good candidates come from the domain: totals stay non-negative, balances stay consistent, IDs stay unique, timestamps stay monotonic. Bad candidates are the ticket examples rewritten as assertions. Those are already covered.
Layer two: frozen fixtures
Property checks cover the input space. Fixtures cover the real world. Agent patches often break on data shapes the ticket never mentioned.
A fixture freeze is a contract. You snapshot a realistic dataset and hash it. Every agent patch runs against the same frozen world. When a test changes behavior, the patch caused it, not the data.
# scripts/freeze_fixtures.py
import hashlib
from pathlib import Path
FIXTURE_DIR = Path("tests/fixtures/data")
def freeze() -> None:
for path in sorted(FIXTURE_DIR.rglob("*")):
if path.is_file():
digest = hashlib.sha256(path.read_bytes()).hexdigest()
print(f"{path.relative_to(FIXTURE_DIR)}: {digest}")
if __name__ == "__main__":
freeze()
Commit the output. Compare it in CI. If a fixture changes, the diff must be explicit and reviewed. Otherwise an agent can "fix" a test by editing the data instead of the code. That is the most common silent cheat in agent-generated patches.
Layer three: flaky freeze
The third layer is discipline. Agent patches multiply the number of test runs. A flaky test that passed once now fails randomly at 3 AM. The worst response is rerunning until green.
A flaky freeze is simple: quarantine, don't retry. When a test fails without a code change, move it to a quarantine suite. It stops blocking the pipeline. It also stops hiding real regressions.
# scripts/detect_flaky.py
"""Run a test file three times. Report any test that flips."""
import subprocess
import sys
TEST_FILE = sys.argv[1]
seen: dict[str, list[bool]] = {}
for run in range(3):
result = subprocess.run(
["pytest", TEST_FILE, "--tb=no", "-q"],
capture_output=True, text=True,
)
failed = {
line.split("::")[-1]
for line in result.stdout.splitlines()
if "FAILED" in line
}
for test in set(seen) | failed:
seen.setdefault(test, []).append(test in failed)
for test, results in seen.items():
if len(set(results)) > 1:
print(f"FLAKY: {test} -> {results}")
Run this on the agent's branch before review. A test that fails once in three runs is noise, not a signal. Quarantine it. Make the agent fix the flake separately, or reject the patch.
The gate, in order
The full gate runs in three steps. Property checks first, because they are fast and brutal. Fixture hashes second, because they catch silent data edits. The flaky detector last, because it protects the first two from noise.
Only then does a human reviewer read the diff. The reviewer's job shrinks to judgment. Does the approach fit the codebase? Do the properties match the product's real invariants?
Where free infrastructure fits
Running this gate continuously needs compute and model calls. MonkeyCode is an open-source project that pairs free model access with a free server option. The server runs the gate on a schedule. The model access drafts candidate properties and summarizes failure clusters for the reviewer.
Disclosure: This article was prepared as part of MonkeyCode's product outreach.
The gate does not require a specific vendor. Any CI runner and any test framework will do. The free tier just lowers the cost of starting.
Limitations
This strategy has real limits. Property checks only test the properties you write. If the invariant is wrong, the generator happily validates the wrong behavior. Fixtures rot when real data drifts. The freeze protects stability, not relevance.
Who should not use this? Small throwaway scripts. One-off migrations. Demos that die next week. The setup cost is real. If the patch touches code with no invariants worth protecting, the gate is overhead, not safety.
A green gate is not a proof of correctness. It is a filter. It raises the cost of a bad merge. It does not make bad merges impossible. The human reviewer still owns the final call.
The next time an agent opens a PR at 2:47 AM, let the properties argue with it first. If you want to see this gate running on free infrastructure, MonkeyCode's open-source project and free server option are a reasonable place to start.
Top comments (0)