The Patch Passed All Tests. The Error Contract Didn't.

A green suite is a minimum, not a verdict. When an agent rewrites a function, the happy path can stay intact while the error path silently degrades. I've seen this enough times to stop trusting unit tests alone after an agent patch. The fix isn't a bigger suite. It's a layered check: property tests that enumerate behavior, fixtures that freeze real-world inputs, and a rule that flaky tests are disabled instead of retried. This article shows the workflow with a tiny Python example.

The case that exposed it

Here's a parser that raises a custom error when settings is missing:

class ConfigError(Exception):
    pass

def parse_config(text: str) -> dict:
    data = json.loads(text)
    if "settings" not in data:
        raise ConfigError("missing settings")
    return data["settings"]

An agent patch "simplified" it:

def parse_config(text: str) -> dict:
    data = json.loads(text)
    return data["settings"]  # KeyError when missing

Every unit test passed because every fixture contained settings. The caller, however, catches ConfigError. After the patch, a malformed payload produces KeyError, which escapes to the top level and kills the service. No test caught it.

Step 1: Encode the contract with properties

Property tests don't care about a single example. They generate many inputs and assert an invariant. The invariant here is simple: if parse_config raises, it must raise ConfigError.

from hypothesis import given, strategies as st
import json

@given(st.dictionaries(st.text(), st.integers()))
def test_parse_config_never_raises_keyerror(data):
    text = json.dumps(data)
    try:
        result = parse_config(text)
    except ConfigError:
        return  # expected when "settings" absent
    except Exception as exc:
        raise AssertionError(f"Unexpected type: {type(exc).__name__}: {exc}")
    assert "settings" in data

Hypothesis will hit both cases. When settings is absent, it checks that the exception type is correct. When present, it verifies the return value. Any drift in the error contract now fails the suite. You can run this with pytest -k property and get a minimal failing example when the contract changes.

Why not just mock the error? Mocks test the code you wrote, not the behavior the caller needs. Property tests exercise the actual function with a variety of inputs, which is closer to production.

Step 2: Freeze real inputs as fixtures

Properties generate from a spec. Fixtures capture reality. I keep a small corpus of payloads collected from logs and API calls, each marked with the expected outcome.

[
  {"body": "{\"settings\":{\"theme\":\"dark\"}}", "should_raise": false},
  {"body": "{\"defaults\":{\"theme\":\"light\"}}", "should_raise": true}
]

Then a fixture test loops over them:

import json
from pathlib import Path

def test_config_payload_fixtures():
    cases = json.loads(Path("fixtures/config_payloads.json").read_text())
    for case in cases:
        try:
            parse_config(case["body"])
        except ConfigError:
            assert case["should_raise"], f"Unexpected error: {case['body']}"
        else:
            assert not case["should_raise"], f"Expected error: {case['body']}"

Fixtures are the ground truth your agent patch is measured against. They don't need to be large. Ten carefully chosen payloads beat a thousand generated strings. The trick is to mark each fixture with the expected result, so the test can tell a wrong success from a wrong failure.

Step 3: Freeze flaky tests, don't retry them

In the same patch, a test that hit the network started failing. The temptation is to retry it or mark it as "sometimes". That hides regression. The rule I use: if a test fails without a deterministic reproduction, it's frozen.

import pytest

@pytest.mark.skip(reason="Flaky: freezes on network timeout; see issue #412")
def test_external_parse_endpoint():
    ...

A frozen test isn't forgotten. It becomes a visible TODO that either gets fixed or gets removed. The rest of the suite stays reliable. If you want to recover it, turn it into a property or a fixture test with a controlled input. Then there's nothing to retry.

Running this on a free server

Generating property tests from scratch takes time. I used MonkeyCode's free model access to create initial property stubs from the original function signature, then edited them to match the actual contract. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The generated stubs were a useful starting point, but they were not the final answer; every test needed a human check.

The full matrix — properties plus fixture cases — becomes part of every patch review. I've been using MonkeyCode's free server option to run this on their side, so the matrix isn't competing with local builds during long agent loops. I still run a quick smoke test locally before pushing. The free server is not a substitute for understanding the test output.

Limitations

Property tests are only as good as the invariants you write. If the invariant is wrong, the test reinforces the bug. Fixtures go stale quickly; prune them when they stop reflecting production. The free server may time out on very large suites, so keep the matrix small and pipeline-friendly. This workflow also assumes you have a reliable error contract to assert. If the original behavior is undocumented, write the contract first, then compare the agent patch against it.

Don't use this approach when the contract is still being designed. If the expected error behavior is contentious, locking it into a property test will stall every patch that tries to improve the API. Let the contract settle first, then encode it.

The bigger lesson is not about any tool. A green suite after an agent patch is a starting point. Add a layer that checks error contracts, freeze a few real inputs, and stop retrying flaky tests. That's how you find the regression before your users do.