The Formatter Looked Clean. Property Testing Found the Missing Zeroes.

An agent patch can change behavior without failing a single unit test. The tests encode the new author's expectations. They do not encode the old code's promises. A formatter that drops zero-padding looks clean in the diff. The output still changed for every value under ten.

Here is the concrete case. A function formats seconds as "HH:MM:SS". An agent patch replaced the hand-written formatter with a shorter version. The unit test covered 3661 seconds and passed. The new formatter silently dropped zero-padding for hours, minutes, and seconds under ten.

Nobody noticed until a video player started showing "1:2:3" instead of "01:02:03".

The failure pattern

The old formatter used :02d for every component. That is a behavioral contract. The new formatter kept the arithmetic and dropped the padding. The diff looked like a simplification. The semantics changed for a whole class of inputs.

This is the pattern that matters: the patch and its tests stay consistent with each other. Both drift away from the previous behavior. The diff is clean. The contract is broken.

The workflow

The fix is to extract behavioral invariants from the old implementation and test them against the new one. The workflow runs as a merge gate for every agent patch that touches a pure function.

1. Check out the old implementation from git.
2. Generate a list of invariants from the old code.
3. Convert each invariant into a property test.
4. Run the properties against both versions.
5. Block the merge if a property holds on old but fails on new.

Step two is where a free model endpoint fits. Reading a function and listing what it promises is pattern matching with judgment. It is exactly the kind of task a model does well and a regex cannot.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. I used MonkeyCode's free model access for step two and its free server option to run the property gate without standing up a paid CI worker.

Turning old code into invariants

The model reads the old function and answers three questions. What inputs does it accept? What does it return for edge cases? What format does it promise? The answers become properties.

For the formatter, the answers were: every component is zero-padded to two digits, hours can exceed 24, and the separator is always a colon. The first answer became three properties. The other two were already true in both versions.

Ask for the invariants as a list, not prose. A list converts directly into test functions. Prose requires interpretation, and interpretation is where the agent's bias sneaks back in.

The runnable gate

Here is a self-contained version. It defines both formatters, then runs properties that encode the old behavior.

# behavior_gate.py
from hypothesis import given, settings, strategies as st

def old_format(total: int) -> str:
    h = total // 3600
    m = (total % 3600) // 60
    s = total % 60
    return f"{h:02d}:{m:02d}:{s:02d}"

def new_format(total: int) -> str:
    return f"{total // 3600}:{(total % 3600) // 60}:{total % 60}"

@given(st.integers(min_value=0, max_value=86399))
@settings(max_examples=100)
def test_hours_are_zero_padded(total):
    hours = new_format(total).split(":")[0]
    assert hours == f"{total // 3600:02d}"

@given(st.integers(min_value=0, max_value=86399))
@settings(max_examples=100)
def test_minutes_are_zero_padded(total):
    minutes = new_format(total).split(":")[1]
    assert minutes == f"{(total % 3600) // 60:02d}"

@given(st.integers(min_value=0, max_value=86399))
@settings(max_examples=100)
def test_seconds_are_zero_padded(total):
    seconds = new_format(total).split(":")[2]
    assert seconds == f"{total % 60:02d}"

if __name__ == "__main__":
    test_hours_are_zero_padded()
    test_minutes_are_zero_padded()
    test_seconds_are_zero_padded()
    print("all properties passed")

Running this produces a clear failure:

Falsifying example: test_hours_are_zero_padded(total=5)
new_format(5) = '0:0:5'
expected hours = '00'

One property, one failure, one blocked merge. The agent patch is not wrong in isolation. It is wrong relative to the contract the old code established.

Why this beats adding more unit tests

Unit tests are examples. Properties are contracts. An example proves one input. A property proves a class of inputs. When an agent rewrites a function, the risk is not the tested input. The risk is the untested class.

The model-generated invariant list is a cheap second opinion. It reads the old code and asks: what does this function promise? The answer becomes executable.

A property test that fails gives you a counterexample. A counterexample is a gift. It tells you the exact input where the contract breaks. You can paste it into a bug report, send it back to the agent, or turn it into a regression test. The unit test that passed gave you none of that.

Choosing the right gate

Gate	Catches	Cost	Best for
Unit tests	Known examples	Low	Happy paths
Property tests	Behavioral drift	Medium	Pure functions
Mutation testing	Weak assertions	High	Test quality
Differential testing	Output changes	Medium	Reference implementations

Property testing sits between unit tests and mutation testing. It costs more than a unit test and less than a full mutation run. It catches the failure class that matters most for agent patches: the contract that existed before the patch.

How to run this on every patch

1. Keep the property file next to the function it protects.
2. Run it in CI as a separate job from the unit suite.
3. Fail the build on any property that holds for old but fails for new.
4. Send the failing example back to the agent as a review comment.
5. Re-run after the follow-up patch.

The loop costs almost nothing. The properties run in milliseconds. The model call is one request per function. The free server option keeps the whole gate out of the paid tier.

The five-step loop is the same loop you already use for human review. The difference is that the property gate runs in seconds and never gets tired. It does not skip the file because the diff looked small. It does not trust the agent's summary. It runs the contract.

Limitations

This approach only works for pure functions. Stateful code, I/O, and concurrency need a different gate. The invariant list is only as good as the model's reading of the old code. Treat it as a draft, not gospel.

Property tests also inherit the usual risks. A weak generator can miss the failing class. A too-strict property can block legitimate changes. Review the property list before you trust it.

Who should skip this

Teams with stable functions and human-written patches do not need this gate. The overhead pays off when patches arrive fast and tests are written by the same agent that wrote the code. That is the exact situation where the tests and the patch agree with each other, and both are wrong.

If your merge queue is full of agent patches, add this gate. It will find the first behavioral drift within a week. Run it before merge, and let the agent see the failure instead of your users.