A green suite after an agent patch is a claim about the fixture version that just ran, not about the fixture you reviewed last month. If the patch moved the fixture, the passing test is a different test. This article describes a gate for that one surface: the support files that decide what an assertion actually sees.
Prerequisites: a git history where the patch is a diffable commit or range, a test layout you can enumerate by path, and the willingness to treat fixtures as reviewed code. Nothing here depends on a specific test runner. The code below is illustrative and was not executed — treat it as a shape to adapt, not a tool to install.
The failure mode, stated precisely
A test result is a conjunction of three things: assertion logic, fixture state, and wiring. Agent-produced patches can edit all three in one diff, and reviewers tend to read the first and skim the other two.
There are two vacuity routes. Assertion tautology makes a check unable to fail. Fixture vacuity makes a check unable to observe anything — the data it inspects became empty, degenerate, or was replaced by a value the assertion expects.
A property like "the sum of line totals equals the order total" holds trivially on an empty order. If a patch changes a factory default from a seeded two-line order to an empty one, the property still passes and now proves nothing. The assertion did not change. The meaning did.
Step 1: Enumerate the fixture surface by path
Do not split on directory names like tests/. Split on intent: does this file change what an assertion observes? Keep the glob list in the repository so the gate itself is reviewable.
tests/fixtures/**
tests/factories/**
tests/baselines/**
**/__snapshots__/**
tests/helpers/**
tests/conftest.py
Anything outside that list is application code and follows the normal review path. Anything inside it is a claim about test meaning and needs an explicit justification in the patch description.
Step 2: Classify the diff, not the file
The useful unit is the (category, change kind) pair. A new factory method is not the same risk as a changed factory default. This table is the decision rule to put in a PR checklist.
| Fixture category | Change kind | Question to ask | Evidence required before merge |
|---|---|---|---|
| data | added case | Does the new case exercise the new branch? | Name the branch and the assertion covering it |
| data | removed or shrunk case | Which failure used to be caught here? | One test that still fails without the patch |
| factory | new method | Is the default deliberately minimal? | Assertion shown to fail on that minimal default |
| factory | changed default | Does any existing property become vacuous? | List of properties reading this factory, each with a non-vacuous case |
| baseline / snapshot | regenerated | Was the delta human-readable, or a mass rewrite? | Diff summary reviewed line by line |
| wiring (conftest, plugins) | added hook | Can it skip or xfail silently? | Skip report showing zero new skips |
| assertion helper | loosened | Can the helper still return a value that is false? | Helper self-test |
The right-hand column is the part most gates skip. "Tests pass" is not evidence about a fixture change; it is the outcome the fixture change was supposed to make true.
Step 3: Pin properties to a fixture digest
Compute a digest over the fixture surface at the patched tree and record it next to every property result. This makes the coupling explicit and cheap to re-check later.
# Illustrative, unexecuted. Stdlib only. Adapt globs to your layout.
import hashlib
from pathlib import Path
FIXTURE_GLOBS = [
'tests/fixtures/**/*', 'tests/factories/**/*', 'tests/baselines/**/*',
'**/__snapshots__/**/*', 'tests/helpers/**/*', 'tests/conftest.py',
]
def fixture_files(root: Path) -> list[Path]:
seen = {}
for pattern in FIXTURE_GLOBS:
for p in root.glob(pattern):
if p.is_file():
seen[p.relative_to(root)] = p
return [seen[k] for k in sorted(seen)]
def fixture_digest(root: Path) -> str:
h = hashlib.sha256()
for p in fixture_files(root):
h.update(str(p.relative_to(root)).encode())
h.update(b'\0')
h.update(p.read_bytes())
h.update(b'\0')
return h.hexdigest()[:16]
Two rules follow from the digest. First, a property result is only comparable to another result recorded at the same digest. Second, a flaky-test freeze measured at digest A does not carry over to digest B — it expires on fixture change, not on a calendar date.
That second rule is the narrow delta to a general quarantine policy. Calendar expiry assumes time is what makes a freeze stale. On agent patch queues, it is usually the fixture that moved.
Step 4: Order the gate
Order matters, because each step can invalidate the previous one.
- Compute the fixture surface for the patch range.
- Classify each entry as (category, change kind); reject unclassified support-file edits.
- Compute the fixture digest for the patched tree.
- Look up existing freezes for the touched tests and drop any recorded at a different digest.
- Run property checks, labelling each result with the digest.
- Only then run test selection — selection keyed on a stale digest selects the wrong suite.
Skipping step 4 is the common mistake. A freeze recorded before the fixture moved will silently suppress exactly the test the fixture change broke.
Step 5: An illustrative walkthrough
Hypothetical; no real repository, patch, or run is implied.
A patch adds a discount path for line items and also edits tests/factories/order.py. Step 2 classifies that edit as (factory, changed default). The reviewer asks the Step 2 question — which properties read this factory — and finds the totals property from earlier. On the patched tree the order is empty, so the property passes without touching the discount code at all. Step 3 shows a new digest, so the previously frozen totals test no longer carries its freeze and runs. It fails. The patch is not rejected for being wrong; it is rejected for arriving with a fixture that hid the failure.
That is the whole value of the gate: it converts "the suite is green" into "the suite was green against these fixtures, and here they are."
Where a shared workbench changes the picture
If your team runs agent-produced patches through a shared workbench, the fixture surface is also a permissions surface. MonkeyCode (open-source repository: github.com/chaitin/MonkeyCode) is described in its current README as a team AI workbench covering office, coding and design, with shared team resources, configurable models, MCP integrations, access management, and self-hosting.
Disclosure: This article was prepared as part of MonkeyCode's product outreach.
Two of those documented properties interact with this gate honestly. Shared team resources mean a fixture, template, or baseline may be reachable by more than one person's agent run, so a fixture diff can originate outside the patch author's own review. Access management is the control that decides who may modify that shared surface — which is the same question Step 2 asks in prose. If you automate fixture edits at all, the permission boundary and the review boundary should name the same files.
One naming caution: the repository also contains README.monkeycode.md, which documents the historical MonkeyCode AI development platform. That is a different document from the current workbench README; do not assume the two describe an identical feature set. For self-hosted deployments, note as well that external model and tool providers may still receive data through configured integrations — self-hosting is not by itself a data-locality guarantee.
Limitations and who should not use this
- Monorepos with fixtures inside application packages break the glob split. You need an explicit manifest, and the manifest becomes the thing to review.
- Generated fixtures defeat byte-level digests; digest the generator inputs instead.
- Snapshot-heavy frontends produce large regenerated diffs, and the Step 2 evidence requirement becomes real review work rather than a checkbox.
- No stable base commit means digests are not comparable; fix the baseline before adding the gate.
- Small, human-authored, fixture-light patches do not justify the tooling. Use the Step 2 table as a checklist and skip the rest.
How to check it works on your repository
Do not trust the classifier because it looks right. Plant three fixture-only changes on a throwaway branch — a shrunk data case, a changed factory default, a regenerated baseline — and confirm each is classified and each produces a different digest. Then confirm a freeze recorded on the base commit is dropped for the patched tree. If any of the three slips through, the globs are wrong, and no amount of downstream assertion review will catch it.
Top comments (1)
Testing vacuity is subtle when a fixture changes, but another blind spot in property-based testing is fixture cardinality compression. When a patch updates a factory to simplify setup, it often drops combinatorial corner cases (such as multiple currencies in a ledger or duplicate line item IDs). The assertion continues to test the property, but the state space collapses to the trivial path.
In financial reconciliation pipelines, we pair structural fixture digests with property coverage assertion bounds. If a property passes but exercises fewer distinct branching paths or entity states than the baseline recorded, the CI gate flags it as a degenerated fixture run.
The point about dropping freezes on digest changes is also spot on. Carrying over quarantine flags across modified support files defeats regression testing entirely.