The Friday patch looked small, tidy, and safe. An agent renamed a retry helper without touching public signatures. The first CI run returned a clean green badge.
The second run failed one idempotence property on identical inputs. The third run passed that same property again without a code change. The merge still landed before the Friday lunch break.
Monday's ledger showed duplicate charges on one customer account. The badge had treated unstable noise as proof of safety. This opening is a composite incident, not a measured outage recorded by this account.
A nearby testing question is whether a retry should count as proof. Better model drafts do not repair an oracle that flips on identical inputs. The gate has to separate those two failures before anyone merges.
Who may vote
Treat every automated check as a voter with one ballot. A property may block merge after two identical failures. A property may allow merge only after a stable pass.
A frozen flake may not cast either kind of vote. Its old failures stay filed as history, not as a fresh block. Its old passes also stay silent until a human repairs the oracle.
Example tests still help when you debug a single path. They do not certify behavior across generated inputs and boundaries. That split stays useful when an agent patch is wide.
Pin fixture bytes before you trust any oracle result. Hash the fixture pack and store the digest beside the patch. Commit that digest with the fixtures, outside the agent's unreviewed diff.
If that digest drifts, stop the run and refuse every vote. A moving fixture can counterfeit stability across two repeats. Drift is a stop condition, not a warning to click past.
Do not reuse a flake lease as this control. A lease expires and then invites another retry. A freeze removes the voter until a human repairs the oracle.
Those controls fail in different ways and should stay separate. Write the choice down so the next reviewer does not invent a third rule. The scorecard below is the written form of that choice.
Run the scorecard
The script is an unexecuted proposal, not a production result from this account. Copy it into a scratch repository and point it at your fixtures. Expect to replace the property names with invariants from your own service.
# proposal: unexecuted property gate
import hashlib, json, pathlib, subprocess, sys
ROOT = pathlib.Path(__file__).resolve().parent
FIX = ROOT / 'fixtures'
FREEZE = ROOT / 'freeze_set.json'
PROPS = ['idempotent_write', 'bounded_retry', 'no_secret_in_log']
def digest(folder: pathlib.Path) -> str:
h = hashlib.sha256()
for p in sorted(folder.rglob('*')):
if p.is_file():
h.update(p.relative_to(folder).as_posix().encode())
h.update(p.read_bytes())
return h.hexdigest()
def load_freeze() -> dict:
if not FREEZE.exists():
return {'frozen': []}
return json.loads(FREEZE.read_text())
def run_once(prop: str) -> int:
cmd = [sys.executable, '-m', 'pytest', f'properties/test_{prop}.py', '-q']
return subprocess.run(cmd, cwd=ROOT).returncode
def vote(prop: str, runs: int = 2) -> str:
frozen = set(load_freeze()['frozen'])
if prop in frozen:
return 'frozen'
codes = [run_once(prop) for _ in range(runs)]
if any(c != codes[0] for c in codes):
return 'freeze'
return 'pass' if codes[0] == 0 else 'fail'
def main() -> int:
expected = (ROOT / 'fixture_digest.txt').read_text().strip()
if digest(FIX) != expected:
print('fixture drift; no vote')
return 2
ballots = {p: vote(p) for p in PROPS}
freeze = load_freeze()
for name, ballot in ballots.items():
if ballot == 'freeze' and name not in freeze['frozen']:
freeze['frozen'].append(name)
FREEZE.write_text(json.dumps(freeze, indent=2) + '\n')
print(json.dumps(ballots, indent=2))
if any(v == 'fail' for v in ballots.values()):
return 1
if any(v in {'freeze', 'frozen'} for v in ballots.values()):
print('frozen voters abstain; human repair required')
return 3
return 0
if __name__ == '__main__':
raise SystemExit(main())
Pair the gate with one property that states a numeric cap. The example below is a proposal and has not been executed here. Swap the billing import before you treat a failure as real.
# proposal: unexecuted property, not a measured result
# import billing # replace with the module under test
def test_bounded_retry(monkeypatch):
calls = []
def send():
calls.append(1)
raise TimeoutError('synthetic')
monkeypatch.setattr('billing.transport.send', send)
try:
billing.charge(amount=10, retries=3)
except TimeoutError:
pass
assert len(calls) == 3
Two repeats are a starting probe, not a statistical certificate. Raise the repeat count when the property touches clocks, networks, or files. Record that count in the scorecard so reviewers can see the budget.
A stable failure blocks the merge with exit code one. A mixed result freezes that property and writes its name. A frozen name abstains on every later run of the gate.
Abstention is not a pass and must not hide inside zero. The script returns three when any voter is frozen or newly frozen. CI should fail closed until that name is repaired in review.
Pin the digest before the agent starts editing the tree. Run the hasher once on a clean fixture directory. Store the printed line in fixture_digest.txt and commit it with the pack.
python - <<'PY'
import hashlib, pathlib
root = pathlib.Path('fixtures')
h = hashlib.sha256()
for p in sorted(root.rglob('*')):
if p.is_file():
h.update(p.relative_to(root).as_posix().encode())
h.update(p.read_bytes())
print(h.hexdigest())
PY
Then invoke the gate on the agent patch and print the exit status. Let the script own the repeat count on ordinary patch runs. Run an extra manual pass only when you are testing the gate itself.
python property_gate.py
echo exit:$?
The process speaks in four codes, like a traffic light with a closed lane. Code two means the fixture bytes moved, so nobody votes. Code one means a stable property failed, so the merge stops.
Code three means a voter abstained and still needs a human owner. Code zero means every voting property passed with no abstention left. Wire CI to accept only zero if the badge must mean clean.
Inspect the freeze file after a noisy night of reruns. Each name in that file needs a human owner and a repair note. Do not delete a name merely to paint the CI badge green.
python -m json.tool freeze_set.json
The script has no waiver flag and no hidden retry button. Removal of a frozen name belongs in a reviewed repair commit. That commit should show new stable runs beside the removal.
How the patch enters
Give the agent a sealed task, not a blank repository. State the invariants that must remain true after the edit. Forbid edits under properties/ and fixtures/ inside the same patch.
A patch that rewrites its own judge is not evidence of safety. If the agent touched a property, stop and split the change. Land the oracle fix with human tests before the behavior patch faces this gate.
A coding assistant can draft text without becoming a voter. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The operator describes MonkeyCode as an open-source assistant with free model access.
A free server option is the other availability claim supplied for this draft. This article does not state quotas, model names, hardware, or duration. It also states no benchmark numbers and no promise that the offer remains.
Use free model access as a drafter, not as a judge of the patch. Paste a stable failure log and ask for three hypotheses only. Pin any new property yourself before it joins the voter list.
The free server option can host the repeat probe away from a laptop. If that server is unavailable, the local scorecard still stands alone. Availability is a convenience claim, not a correctness proof.
Read a stable miss
A stable failure is a product signal, not an inconvenience to rerun away. Open the property file and name the broken invariant in one sentence. Compare the agent diff with that sentence before you blame the test.
Many agent patches pass narrow examples and still break a boundary. Suppose bounded_retry fails twice on the same fixture pack. The helper now retries four times instead of the pinned cap of three.
The example test used a stub that stopped early and hid the cap. The property called the real limit and failed on both repeats. Keep the property, reject the patch, and request a smaller diff.
Suppose that same property flips between pass and fail. Freeze it and stop counting it as a voter. The cap may be fine while the clock inside the oracle is not.
A sleep inside the oracle is a common source of mixed ballots. Replace the sleep with a fake clock and rerun the same probe. Unfreeze the name only after identical inputs agree twice.
Who should skip it
Skip the gate when you have no pinned fixtures to hash. A property without fixed inputs is a story, not a vote. Skip it when every check needs a live vendor network.
Jitter will fill the freeze file and hide real regressions underneath. Skip it for prototypes that will be deleted within the week. The ceremony then costs more than the patch itself is worth.
Teams with a mature mutation suite should not treat this gate as a replacement. It does not replace mutation work or production monitoring either. It answers whether this patch broke a named invariant or the oracle wobbled.
Limits to keep visible
Two repeats will miss rare flakes that appear only weekly. A freeze file can grow until almost nothing still votes. Review that file each week and assign a named owner.
A green exit with many abstentions is a warning, not a release certificate. Hashing fixtures will not catch a property that encodes the same bug. If the agent and the oracle share a wrong assumption, both runs pass.
Human review of every new property remains mandatory before it votes. Generated cases also need a seed logged beside the ballot. Without that seed, you cannot replay the failure on demand.
Install pytest before you trust any ballot from this script. A missing runner looks like a stable failure, not like noise. Fix the environment first, then rerun the same pinned fixtures.
Free model drafts can look precise and still be wrong. Free server runs can differ from production clocks and filesystems. Treat both as optional tools around the scorecard, not above it.
Need a second draft for one frozen property name. MonkeyCode's free model access can propose wording, not a ballot. Keep the vote on your tree, and show stable runs before unfreezing.
Top comments (0)