Treat a Flaky Freeze as Merge Debt. Score the Agent Patch.

A green CI run under an active flaky freeze is not a pass. It is a deferred assertion. Agent patches make that debt worse because the model can keep the suite green by widening a skip, rewriting a matcher, or depending on a freeze that a human opened for an unrelated flake. Score the patch before merge. Do not read the test names.

Three signals are enough. Property-hash independence. Fixture-digest stability. Freeze-budget consumption. If any one of those moves in the wrong direction, the patch is not evidence. The rest of this article is a reproducible scorecard, not a philosophy note.

What the scorecard actually measures

Most gates still ask a binary question: did the suite exit 0? That question is the wrong unit. An agent can satisfy it while weakening the oracle. The scorecard asks three independent questions instead.

1. Did any file that defines an invariant change in the same diff as production code?
2. Did any locked fixture byte change, even if the test function name stayed the same?
3. Did the patch add, extend, or rely on a freeze entry whose expiry is in the future?

A pass is a triple, not a checkbox. Freeze consumption is not a skip. It is merge debt with a due date.

Inventory the freeze list as a ledger

Keep freezes out of pytest.mark.skip. Put them in a ledger the gate can parse. The ledger is the only place a freeze is legal.

# freeze_ledger.yaml
version: 1
budget:
  max_active: 3
  max_added_per_patch: 0
entries:
  - id: FZ-014
    test_nodeid: tests/test_parser.py::test_roundtrip_unicode
    reason: "clock skew on CI runner; not a product bug"
    opened_by: human
    opened_at: "2026-09-12T09:00:00Z"
    expires_at: "2026-09-26T09:00:00Z"
    witness: "repro seed 7f3c; fails 3/20 locally, 11/20 on shared runner"
    allows_agent_rely: false

allows_agent_rely: false is the important field. Humans may open a freeze. An agent patch may not spend it. If the patch's coverage of that nodeid is the only reason CI is green, the score is a reject.

Lock properties and fixtures on a different path

Store properties in a tree the patching agent cannot write. Hash that tree in CI. Store fixtures the same way. Names are not the lock. Bytes are the lock.

oracles/
  properties/
    parse_roundtrip.py
    idempotent_normalize.py
  fixtures/
    corpus.sha256
    corpus/
      001.json
      002.json
  MANIFEST.sha256

MANIFEST.sha256 is a sorted list of path, size, and sha256. Recompute it on every pipeline. If the agent diff includes oracles/, the scorecard fails closed. That rule is older than this article. The new part is combining it with freeze debt in one number.

Artifact: a merge scorecard you can run locally

The following script is a labeled, runnable example. It does not claim production metrics. Point it at a git diff, a ledger, and a manifest. It prints a JSON object a gate can consume.

#!/usr/bin/env python3
"""score_agent_patch.py — labeled example, not a published benchmark."""
from __future__ import annotations

import hashlib, json, subprocess, sys
from pathlib import Path

ORACLE_PREFIXES = ("oracles/", "tests/properties/")
LEDGER = Path("freeze_ledger.yaml")

def git_changed(ref: str = "HEAD") -> list[str]:
    out = subprocess.check_output(
        ["git", "diff", "--name-only", f"{ref}^", ref], text=True
    )
    return [line for line in out.splitlines() if line]

def sha256_file(path: Path) -> str:
    h = hashlib.sha256()
    h.update(path.read_bytes())
    return h.hexdigest()

def load_ledger(text: str) -> dict:
    # Minimal YAML subset: avoid a hard dependency in the example.
    import yaml
    return yaml.safe_load(text)

def score(ref: str = "HEAD") -> dict:
    changed = git_changed(ref)
    oracle_hits = [p for p in changed if p.startswith(ORACLE_PREFIXES)]
    fixture_hits = [p for p in changed if p.startswith("oracles/fixtures/")]
    ledger = load_ledger(LEDGER.read_text())
    active = [e for e in ledger["entries"] if e.get("allows_agent_rely") is False]
    freeze_file_touched = any(p == str(LEDGER) for p in changed)

    debt = 0
    if oracle_hits:
        debt += 4
    if fixture_hits:
        debt += 3
    if freeze_file_touched:
        debt += 5
    if len(active) > ledger["budget"]["max_active"]:
        debt += 2

    verdict = "reject" if debt > 0 else "accept"
    return {
        "verdict": verdict,
        "debt": debt,
        "oracle_hits": oracle_hits,
        "fixture_hits": fixture_hits,
        "freeze_ledger_touched": freeze_file_touched,
        "active_no_rely_freezes": [e["id"] for e in active],
        "manifest_sha256": sha256_file(Path("oracles/MANIFEST.sha256"))
        if Path("oracles/MANIFEST.sha256").exists() else None,
    }

if __name__ == "__main__":
    print(json.dumps(score(*sys.argv[1:]), indent=2))

Run it against the last commit:

python3 score_agent_patch.py HEAD
git diff --name-only HEAD^ HEAD | sort
sha256sum oracles/MANIFEST.sha256

A non-zero debt is a reject. Do not compensate with extra unit tests in the same diff. Extra tests written by the same agent are not an independent oracle.

Decision table

Use the table as the gate contract. Put it in the repo next to the ledger. Reviewers should not improvise.

Patch touches	Freeze ledger	Property hash	Fixture digest	Verdict
Production only	Unchanged, no rely	Unchanged	Unchanged	Accept
Production only	Unchanged, relies on no-rely freeze	Unchanged	Unchanged	Reject
Production + tests	Unchanged	Unchanged	Unchanged	Review: tests are not oracles
Production + oracles/	Any	Changed	Any	Reject
Ledger only, human-opened, expiry set	Added within budget	Unchanged	Unchanged	Accept with debt log
Ledger extended by the agent	Any	Any	Any	Reject
Fixtures only	Unchanged	Unchanged	Changed	Reject until human re-signs

"Review: tests are not oracles" means a human must classify the new tests. Classification is a different article. This gate only scores independence and debt.

Draft properties on a separate machine

The patching agent should not propose the invariant it will later satisfy. Split the jobs. Production edits stay in the working tree. Property drafts land in a queue the gate never auto-merges.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. If you need a second process that can read a public API surface and emit candidate properties without write access to oracles/, MonkeyCode's free model access and free server option are one way to run that drafter off the patch path. The server is not the oracle. The human still hashes the accepted file.

A minimal isolation pattern:

# labeled workflow — not a product benchmark
mkdir -p /tmp/prop-queue
# 1. export the public signatures only
rg -n "^def " src/ > /tmp/prop-queue/api_surface.txt
# 2. ask a drafter (separate process, no write to oracles/) for candidates
# 3. human edits, then:
install -m 0444 /tmp/prop-queue/accepted.py oracles/properties/
( cd oracles && find properties fixtures -type f | sort | xargs sha256sum > MANIFEST.sha256 )
chmod 0444 oracles/MANIFEST.sha256

The drafter can be wrong. That is expected. Wrong drafts die in the queue. They never sit in the same commit as the production patch.

Property shape the gate can hash

Keep properties small and side-effect free. The example below is a proposal, not a measured corpus.

# oracles/properties/idempotent_normalize.py
from hypothesis import given, settings
from hypothesis import strategies as st
from product.normalize import normalize

@settings(max_examples=80, deadline=None)
@given(st.text(min_size=0, max_size=200))
def test_normalize_idempotent(s: str) -> None:
    once = normalize(s)
    twice = normalize(once)
    assert once == twice

CI should fail if this file's hash changes in a patch that also changes product/normalize.py. Changing both in one diff is how tautologies sneak in. Split the commits. Re-sign the manifest in a follow-up that contains no production hunks.

CI steps, numbered

1. Compute git diff --name-only against the merge base.
2. Reject if any path starts with oracles/ and any path starts with src/ or product/.
3. Recompute oracles/MANIFEST.sha256 and compare to HEAD. Mismatch is a reject unless the commit is oracle-only and human-signed.
4. Parse freeze_ledger.yaml. Reject expired entries still marked active. Reject agent diffs that edit the ledger.
5. Run property tests with a fixed seed file, not a floating default.
6. Run the remaining suite. If a failure's nodeid is in the ledger with allows_agent_rely: false, do not count the run as green for an agent patch.
7. Publish the scorecard JSON as a CI artifact. Do not collapse it into a single emoji.

Seed file example:

export HYPOTHESIS_SEED=20260920
pytest oracles/properties -q --hypothesis-seed=$HYPOTHESIS_SEED
python3 score_agent_patch.py HEAD

The date in the seed is a pin, not a claim about flake rates. Change it only in an oracle-only commit.

What this does not prove

The scorecard does not prove the product is correct. It proves the agent did not spend oracle integrity or freeze budget to look correct. Those are different statements. Conflating them is how green suites rot.

It also does not replace mutation testing, contract tests against a real backend, or a human reading the diff. A property can be independent and still weak. A fixture can be locked and still unrepresentative. The ledger can be honest and still too large. Budget max_active: 3 is a policy choice. Tune it per repo. Do not copy it as a universal constant.

Who should not use this

Do not use this workflow if the repo has no split between production paths and oracle paths. The score is meaningless when tests and code share a folder the agent can rewrite. Do not use it if every flake is frozen without a witness or an expiry. An infinite freeze is a deleted test. Do not use it as a reason to skip reading agent diffs that touch parsers, auth, or money. The gate is a filter. It is not a reviewer.

Teams that already require oracle-only commits and refuse skip markers may only need the freeze-budget field. Add that field first. The rest of the script can wait.

Closing constraint

If you adopt one rule, adopt this one: an agent patch may not close freeze debt that a human opened, and it may not open new debt to hide a failure. Property hashes and fixture digests stay on a path the patch cannot write. The suite then reports a fact about the product, not a fact about the agent's talent for staying green.

If you want a isolated drafter for candidate properties, try MonkeyCode's free model access on a free server and keep its output in a queue the merge gate never auto-applies.