A Verification Pipeline for Agent Patches: Property Checks, Fixture Pins, and a Flaky Freeze in One YAML

The cheapest way to trust an agent patch is to refuse trust until three specific checks pass. A green unit suite is not enough. You need invariants, stable fixtures, and a working definition of flaky, all enforced before the patch lands.

I built a small verification pipeline around those three gates and put them in a single YAML file. I used MonkeyCode's free model access to draft the property checks, and its free server option to run the pipeline in a disposable environment. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Why a YAML? Because verification rules become part of the repo. Reviewers, CI, and the agent itself can all read the contract.

The three gates

• Property checks verify what should always be true, not just the examples the agent saw.
• Fixture pins ensure the data used by tests is byte-for-byte identical to what the patch was validated against.
• Flaky freeze quarantines tests that fail too often, forcing the agent to stop retrying and to look for the root cause.

Each gate is trivial in isolation. Together they form a lightweight acceptance procedure.

The YAML contract

Here is the file I place at patch_verify.yaml:

version: 1
fixtures:
  - path: 'tests/fixtures/orders.json'
    sha256: '9f2c4d...'
properties:
  - module: 'tests.properties'
    test_class: 'OrderProperties'
flake_freeze:
  max_retries: 2
  quarantine_file: '.flake_quarantine.txt'
  reset_after_days: 7

The pipeline reads this file and executes each section in order. If any gate fails, the patch is rejected with a clear message.

Step 1: Draft property checks with a free model

Free model access helps here because property tests are often hard to write from a blank editor. I ask the model: 'List invariants this function must always satisfy.' The output is a draft, not a contract.

For an Order class, a draft looks like this:

from hypothesis import given, strategies as st

@given(st.lists(st.tuples(st.text(), st.integers(min_value=1)), min_size=1))
def test_total_is_never_negative(items: list[tuple[str, int]]) -> None:
    order = Order(items)
    assert order.total >= 0

The model may produce 80% useful ideas and 20% noise. I keep only the invariants I can defend. Free models sometimes hallucinate edge cases that do not exist; human review is non-negotiable.

Step 2: Pin fixtures

Fixture drift is the silent killer of agent patches. A developer updates a JSON fixture to make a new feature pass, and suddenly old tests are meaningless. Pinning catches that.

Before any test runs, the pipeline computes the SHA-256 of every fixture listed in YAML:

#!/usr/bin/env bash
expected=$(sha256sum tests/fixtures/orders.json | awk '{print $1}')
required='9f2c4d...'
if [ "$expected" != "$required" ]; then
  echo 'Fixture mismatch. Agent patch verified against different data.'
  exit 1
fi

For large fixtures, pin a digest of the digest. For small ones, the raw hash is fine. The point is simple: verify the data before you trust the suite.

Step 3: Freeze flaky tests with an expiry

Flaky tests give agent patches a way to fail randomly, which poisons the review process. Our freeze works on two counters: consecutive failures and elapsed time.

A test enters quarantine after max_retries consecutive failures with no code change. It stays frozen for reset_after_days days, then the pipeline re-enables it for one vote. If it fails again, it goes back into quarantine with a longer term.

Here is a minimal Python helper:

# freeze.py
from datetime import date, timedelta

def should_skip(test_id: str, fails: int, last_fail: date, max_retries: int, reset_after: int) -> bool:
    if fails < max_retries:
        return False
    if date.today() - last_fail >= timedelta(days=reset_after):
        return False  # expired, allow retry
    return True

This is not a permanent pardon. It is a cooling-off period that prevents the agent from burning tokens on the same unreliable test.

Step 4: Run everything on a disposable server

The free server option gives us a clean environment for every patch. No stale caches, no local state, no 'works on my machine' excuses. The pipeline does:

1. git clone the patch branch.
2. Verify fixture hashes.
3. Run property tests.
4. Apply the flaky freeze list.
5. Reject or approve.

A typical command inside the disposable server:

python -m pytest --tb=short -m 'not frozen' tests/

If that command exits zero and the fixture checks passed, the patch has a fair chance. Not proof, but a fair chance.

Limitations and who should skip this

This pipeline only checks invariants and data integrity. It does not validate business rules hidden in conversations, nor does it inspect whether the agent solved the right problem.

Avoid this approach when:

• Your codebase has almost no property-based tests and you expect the YAML to fix that overnight.
• Your team treats the quarantine file as a graveyard instead of a temporary holding area.
• You need real external dependencies such as databases or network calls in the test run; the free disposable server often cannot provide them.

Keep the strategy small. Each gate must fail fast and produce an actionable message. Otherwise, it becomes yet another layer of ceremony.

The YAML file is a contract between you, the agent, and the next reviewer. Make it explicit, make it small, and review it as often as the code it protects.

If you have a similar verification contract for agent patches, I would love to compare YAML structures with you.