Split the Flake Before You Freeze It

A freeze on a whole test is a deleted oracle. After an agent patch, treat flake as a split problem first: keep a property core that still fails on contract breaks, pin I/O behind fixtures, and freeze only the timing tail that still needs a witness.

That order matters. Agents often leave the business rule intact and scramble the envelope around it. Clocks, map iteration, retry jitter, and shared temp paths show up as red CI. Freezing the entire case then hides both the envelope and the rule.

This article is a proposed review workflow, not a production study. No pass rates or latency numbers are claimed. The artifact is a ledger format, a pytest gate, and a decision table you can run on the next agent diff.

What this is not

It is not a receipt of seeds and digests. It is not a three-bucket green-test plan. It is not metamorphic checking when you lack an oracle. Those are useful, and they answer different questions.

This workflow answers one question: when an agent patch makes a test intermittent, what still has to stay mandatory?

The three layers of a flaky case

Most flaky tests after an agent refactor are stacked, not atomic. Name the layers before you touch a freeze marker.

1. Property core. Pure, repeatable claims: round-trip, idempotency, bounds, ordering of a sorted view, conservation of counts. These must not enter a freeze list.
2. Fixture belt. Bytes, JSON, and filesystem layout that the core reads. Lock them so the agent cannot rewrite goldens as a side effect of a rename.
3. Timing tail. Sleeps, wall clocks, unordered logs, live ports, and process start order. Only this slice is freeze-eligible, and only with a witness.

If you cannot name the three layers in the failing file, you do not have a freeze candidate. You have an undiagnosed failure.

A witness is the price of a freeze

A skip with a comment is not a freeze. A freeze records how the flake was produced, under which patch, and which property remains in the gate.

Proposed ledger, checked in next to the suite:

# flake_ledger.yaml
version: 1
entries:
  - id: PAY-4417
    test: tests/itest_payment_webhook.py::test_retry_then_ack
    patch_ref: agent/pr-8821
    first_seen: "2026-09-07"
    layer: timing_tail
    witness:
      command: "pytest -q tests/itest_payment_webhook.py::test_retry_then_ack --count=8"
      fail_pattern: "AssertionError: ack lag"
      env_hash: "py3.12-linux-pytest8"
      notes: "fails when retry jitter exceeds 50ms; order of log lines not stable"
    remaining_property: tests/test_payment_properties.py::test_ack_idempotent
    remaining_fixtures:
      - tests/fixtures/webhook_v1.json
    expires: "2026-09-21"
    owner: payments-reviewer

Reject the freeze if any of these are missing: layer, witness.command, remaining_property, expires. The remaining property is the point. Without it, the ledger is a skip list.

Numbered workflow

Run this on the agent diff, not on the whole repository.

1. Reproduce once under a tight command. Do not start with the full suite. Pin the node, the file, and a repeat count. If it fails 8/8, it is not a flake. Fix or reject the patch.
2. Classify the layer. Read the assertion. If it checks a business rule, it is core. If it checks bytes on disk, it is fixture. If it checks elapsed time or log order, it is tail.
3. Extract the core into a property test. Move the rule out of the integration module. Keep the original file only for the envelope.
4. Lock fixtures used by the core. Hash or copy the inputs. Fail CI if an agent commit changes fixture bytes without a human-edited ledger note.
5. Write the freeze entry only for the tail. Point remaining_property at the new test. Set an expiry measured in days, not quarters.
6. Fail the gate if the core is absent. A freeze marker with no live property is a build error, not a warning.

Commands that keep the loop small:

# 1. Is it actually intermittent?
pytest -q tests/itest_payment_webhook.py::test_retry_then_ack --count=8

# 2. After the split, the core must be deterministic.
pytest -q tests/test_payment_properties.py::test_ack_idempotent --count=20

# 3. Fixture drift should be visible in the diff, not buried in an agent commit.
git diff --stat -- tests/fixtures/

If step 1 is all-fail, stop. Freeze policy does not apply to solid regressions.

Artifact: a gate that refuses bare freezes

Label the following as a proposed pytest hook. It is a sketch. Wire it to your real paths before you trust it.

# conftest.py
from pathlib import Path
import yaml
import pytest

LEDGER = Path(__file__).parent / "flake_ledger.yaml"


def _load_entries():
    if not LEDGER.exists():
        return []
    data = yaml.safe_load(LEDGER.read_text()) or {}
    return data.get("entries") or []


def pytest_collection_modifyitems(config, items):
    frozen_ids = {}
    for entry in _load_entries():
        if not entry.get("remaining_property"):
            raise pytest.UsageError(
                f"freeze {entry.get('id')} has no remaining_property"
            )
        if entry.get("layer") != "timing_tail":
            raise pytest.UsageError(
                f"freeze {entry.get('id')} is not limited to timing_tail"
            )
        frozen_ids[entry["test"]] = entry

    for item in items:
        node = item.nodeid
        for test_id, entry in frozen_ids.items():
            if node.endswith(test_id.split("::", 1)[-1]) or test_id in node:
                item.add_marker(pytest.mark.skip(
                    reason=f"timing tail frozen as {entry['id']}; "
                           f"core lives in {entry['remaining_property']}"
                ))

Pair it with a property that cannot be skipped by the same ledger:

# tests/test_payment_properties.py
from hypothesis import given, strategies as st

from payments.ack import apply_ack


@given(st.binary(min_size=1, max_size=64), st.integers(min_value=1, max_value=5))
 def test_ack_idempotent(payload, repeats):
    state = apply_ack(b"", payload)
    for _ in range(repeats):
        again = apply_ack(state, payload)
        assert again == state

The original integration test can keep checking retry timing. That is the tail. If it flakes, the ledger may skip it. test_ack_idempotent stays in the gate.

Fixture belt: lock inputs, not screenshots of time

Agent patches like to "update fixtures" when a timestamp format changes. That is how a freeze spreads into the core.

Keep a small lock file for fixture paths that properties read:

# tests/test_fixture_lock.py
import hashlib
from pathlib import Path

LOCKED = {
    "tests/fixtures/webhook_v1.json": "8f3c...replace_with_real_sha256",
}


def test_locked_fixtures_have_not_drifted():
    for rel, digest in LOCKED.items():
        data = Path(rel).read_bytes()
        got = hashlib.sha256(data).hexdigest()
        assert got == digest, f"{rel} changed without a fixture-lock edit"

Update LOCKED in a human commit. If the agent needs a new schema, that is a review item, not an automatic golden refresh.

A fixture lock is not a freeze. It fails closed. That is the opposite of a skip.

Decision table

Use the table in review comments. Do not negotiate it in chat after CI is red.

Observation	Layer	Action	Freeze allowed?
Fails 8/8 on the agent branch, passes on main	unknown / likely core	Reject or fix the patch	No
Fails 2/8, assertion is elapsed < 50ms	timing tail	Split; keep duration check optional	Yes, with witness
Fails 2/8, assertion is status == "acked"	property core	Extract property; do not skip	No
Diff rewrites tests/fixtures/*.json and tests go green	fixture belt	Restore fixtures; re-run core	No
Log line order changed, counts unchanged	timing tail	Assert counts/properties; freeze order check	Yes, with witness
No remaining property can be named	unclassified	Do not merge	No
Expiry date is in the past	process	Unskip and re-classify	No

The last row is load-bearing. An expired freeze that stays skipped is a skip list with extra YAML.

Where a free coding server fits

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The split is mechanical enough that a coding agent can propose it. The review is not. MonkeyCode's free model access and free server option are relevant when you want a candidate extraction: property stub, fixture list, and a draft ledger row, without standing up your own inference box. The server does not own the freeze. You still require a witness command and a remaining property that a human can read.

Do not ask the agent to invent a digest, a flake rate, or an expiry that "feels safe." Those fields are review data. Generated code that skips test_ack_idempotent is a defect in the workflow, not a convenience.

If you use that path, keep the prompt narrow: split this failing test into a property core and a timing tail; do not add skip markers. Then run the commands above. The value is the split, not the tool.

Limitations

The gate only as strong as remaining_property. A tautology such as assert True will satisfy the YAML and protect nothing.

Hypothesis examples are not a substitute for the original integration path. They miss binding, IAM, and real retry storms. The tail still needs a scheduled unskip.

Shared mutable fixtures across tests will make the ledger lie. If two modules write the same JSON file, the lock test becomes flaky and you will be tempted to freeze the lock. Do not.

This workflow also assumes you can run a single test repeatedly. If the suite only works as a giant shuffle, fix isolation before you freeze anything.

Who should not use this

Do not use a freeze ledger to land agent patches in safety-critical or money-moving paths when the failing assertion is the contract. Split does not mean postpone the rule.

Do not use it when you cannot produce a witness command. "It failed in CI once" is not a witness.

Do not use it as a default for every red test after an agent run. Solid failures are cheaper to reject than to classify.

Teams that already freeze by ticket number with no expiry will make this worse. The YAML will grow, the cores will not, and the suite will look green while the tail stays dark.

Close

Freeze policy is a budget. Spend it only on the timing tail, and only after a property core and a fixture lock are in the gate. If a freeze entry cannot point at a live test, it is not a freeze. It is a skip, and the agent patch is untested at the only layer that still meant something.