Unit Tests Approve. The Invariant Disagrees: A Property-First Strategy for Agent Patches

A green unit test suite does not prove an agent patch is correct. It proves the agent predicted your examples. When every committed test passes and the production system still breaks, the violated constraint was an invariant you never encoded.

This article is a three-layer testing strategy for agent patches. Layer one is property checks. Layer two is frozen fixtures. Layer three is a flaky test freeze. Each layer is cheap and each is hard for an agent to game.

The failure mode

Take a parser. The agent patches parse_interval so "09:32" becomes 570 minutes—because the team complained about irregular intervals. Unit tests use clean inputs like "09:30" and "10:00". They stay green. A property test that feeds "09:32" and expects an exact round-trip fails instantly.

A crash seldom causes this. The silent change does: the patch modified behavior for inputs nobody enumerated. Unit tests encode your expectations; an agent will optimize for those expectations. Property checks encode constraints you have not enumerated.

Layer one: property checks

Start with one invariant. For parsers, round-trip is a strong starting point. For caches, assert that a value never goes stale. For state machines, assert that reordering messages does not lose transitions.

# unexecuted example: shape, not production code
from hypothesis import given, settings
from hypothesis.strategies import integers

def parse_interval(text: str) -> int:
    """'HH:MM' -> minutes since midnight."""

def format_interval(minutes: int) -> str:
    """minutes since midnight -> 'HH:MM'."""

@given(h=integers(0, 23), m=integers(0, 59))
@settings(max_examples=10_000)
def test_round_trip(h: int, m: int) -> None:
    text = f"{h:02d}:{m:02d}"
    minutes = parse_interval(text)
    assert format_interval(minutes) == text

Property checks are probabilistic, not exhaustive. Ten thousand examples do not prove the invariant. They do catch the rounding patch above because the shrinker quickly produces "09:32" as a counterexample. Use shrinking; a failing property without a minimal input is just a stack trace.

A round-trip has a blind spot. If the agent changes parser and formatter together, the invariant stays intact. Pair it with a stronger property: the result is always below 1440, or it produces an error for a specific invalid range.

Layer two: frozen fixtures

An agent can edit your test data. If the new behavior no longer matches the old expectation, the path of least resistance is to rewrite the fixture instead of the code. Frozen fixtures block that path.

Collect a fixture set from production logs or from the last known good release. Store it in one file with a schema version and a hash. Then make the merge gate fail on any change to that hash.

{
  "schema_version": 1,
  "hash": "sha256: replace-with-your-own",
  "entries": [{"input": "09:30", "expected": 570}]
}

The agent can add new test files. It cannot silently migrate old ones. A human approves a fixture migration separately, with a diff review, and the migration lands in its own commit.

Layer three: a flaky test freeze

Flaky tests are an agent's escape hatch. If a test fails once and passes on retry, the agent learns that red is negotiable. It will retry, reorder, and add sleeps.

Define the freeze: any test that fails on a clean checkout moves to quarantine/. It stops blocking merges until a human fixes or deletes it. The agent cannot claim success while a quarantine ticket is open.

# illustrative CI rule
def verify_gate():
    assert fixture_hash_matches(), "fixtures changed"
    assert quarantine_dir_is_empty(), "flaky test not resolved"
    pytest("tests/properties", max_retries=0)

No retries. No @pytest.mark.flaky bypass. If the suite only stays green by rerunning, the suite is lying to the reviewer.

Ordering the three layers

Run property checks first. They fail loudly and generate counterexamples. Run the fixture integrity check second; it is a single hash comparison. Run the flake freeze audit last; it keeps the first two from being masked by retried failures.

1. Property checks — fast, probabilistic, shrinkable.
2. Fixture integrity — deterministic, one hash.
3. Flake quarantine — makes the first two trustworthy.

Where the compute comes from

You need cycles for the property runs and an endpoint for the agent that writes the patch. Ten thousand examples are a small CPU bill but not a free one; the churn of iterative agent patches adds up.

To run this on a near-zero budget, I used MonkeyCode's free server option as the execution layer and its free model endpoint for the agent loop. They are convenience choices; the strategy does not depend on the vendor.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. The availability claims for free model access and the free server come from the project, not from my benchmarks. I did not measure quotas, uptime, or throughput. Validate them against your workload before wiring them into a mandatory gate.

Who should skip this

Property checks are overhead when you can enumerate every input by hand. If your patch surface is one function with three branches, add a switch-case test instead. Fixture freezing needs a stable baseline; a young project with no known-good release has nothing to freeze. A flake freeze requires enforcement; without it, the property layer will drown in retried failures.

Start with one invariant. Freeze the oldest fixture file. Quarantine the flakiest test. That is a complete first iteration, and it will catch more than the next five unit tests you ask an agent to write.