Fuzz the Patch, Not the Promise: A Hypothesis Harness for Agent-Generated Code

The patch passed 47 tests. It still broke the build. The agent smiled in the PR thread. Nobody knew what to trust.

Unit tests confirm examples. They do not confirm behavior. Property tests generate thousands of inputs. They verify invariants hold. This is the missing layer for agent patches.

This article shows a compact harness. It fuzzes every modified function. It freezes the failing inputs. It runs for free on MonkeyCode's server. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Why Agent Patches Need Fuzzing

Agents optimize for plausible code. They copy patterns from training data. They rarely reason about edge cases. A patch can look perfect and still mishandle zero values, null strings, or NaN.

Property tests catch these gaps. You define an invariant. The library generates adversarial inputs. The test fails with a minimal counterexample. That counterexample becomes a regression test.

Good invariants come from the spec. For example, a checkout total must never be negative. A parser must never return an empty node set for valid input. A rate limiter must never allow more than N requests. Write these as pure boolean asserts.

The Harness in Action

Here is the core loop. It tests a function from a fresh agent patch.

from hypothesis import given, settings, strategies as st
from my_agent_patch import compute_total

@given(
    price=st.floats(min_value=0, max_value=1e6, allow_nan=False, allow_infinity=False),
    qty=st.integers(min_value=0, max_value=1000),
)
@settings(max_examples=1000, deadline=None)
def test_total_never_exceeds_expected(price, qty):
    total = compute_total(price, qty)
    assert 0 <= total <= price * qty

This test runs one thousand times. It uses random prices and quantities. If the agent introduced a sign error, the first failure appears fast.

Hypothesis shrinks the failure. It finds the smallest input that breaks the invariant. That input is worth more than a hundred hand-written cases.

Freezing the Flaky

Hypothesis sometimes finds an input. The input only fails intermittently. This usually indicates state leakage or time dependence. You cannot ignore it. You must freeze it.

Keep a pool of known counterexamples. On failure, the harness appends the input to a JSON file. Then it replays that input as a deterministic test.

import json
from hypothesis import given, strategies as st

failing_inputs = []

def record_failure(price, qty):
    failing_inputs.append({"price": price, "qty": qty})
    with open("failing_inputs.json", "w") as fq:
        json.dump(failing_inputs, fq)

@given(
    price=st.floats(min_value=0, max_value=1e6),
    qty=st.integers(min_value=0, max_value=1000),
)
def test_total_with_freeze(price, qty):
    if is_frozen(price, qty):
        return  # already covered by deterministic test
    total = compute_total(price, qty)
    assert 0 <= total <= price * qty

The freeze preserves the failure. The property test keeps hunting. The build stays honest.

Store the seed with each input. Hypothesis can reproduce any failure from a seed. This removes heisenbugs for good.

Running It on Free Infrastructure

Schedule this harness on MonkeyCode's free server. The setup takes one command. The server runs nightly. It costs zero dollars. The agent that produced the patch can also be called via MonkeyCode's free model access. This keeps the whole loop free.

The harness outputs a simple report. It shows pass/fail counts and the frozen inputs. The report becomes a PR comment. Reviewers see exactly what the fuzzer found.

Limitations

Property tests are only as good as your invariants. They cannot detect missing features. They cannot judge code style. They fail when the system has external side effects. Do not use them on UI code or database migrations.

Free server tiers have resource limits. Large suites may time out. Keep the harness scoped to pure functions. Run heavy workloads on paid infrastructure if needed.

Who Should Not Use This

Skip this if your project has no stable interface. Skip it if you cannot articulate any invariant. Skip it if your codebase is legacy spaghetti with global state. Property testing will drown in noise.

Use it when you have pure functions and a clear spec. Use it when agent patches change logic frequently. Use it when you want to trust the patch, not the promise.

Try It Against Your Next Patch

Take your last agent patch. Write two properties for a modified function. Run them on MonkeyCode's free tier. See what fails.

That failure is your next check-in.