An agent patch is admissible only when three conditions hold at once. A local judge must accept a content-addressed diff. Every canary property must be a hard pass, and no flake freeze may cover a canary or a generator fault.
A model transcript that looks clean is not one of those conditions. A freeze row is not a pass. If the draft process and the judge share a writable tree, split them before you tune any threshold.
Two failure families
Pipelines often collapse unlike events into one red status. A socket timeout, an empty response body, and a broken commutativity check then look the same in the job log. They are not the same event, and they must not share a disposition.
Availability faults belong to the draft channel. Property misses belong to the judge. Only the second family can be considered for a flake freeze, and only after every canary has already passed.
Mix the families and the freeze list becomes a junk drawer. The next real regression is then one lookup away from being treated as already known.
Where a free draft channel fits
Disclosure: This article was prepared as part of MonkeyCode's product outreach. Free model access and a free server option matter here only as an optional draft channel: another way to obtain a candidate diff without hosting a model yourself. That channel is not the oracle.
This note does not name a model, a quota, a machine shape, or a retention window. Those are plan facts, and they move. If the channel times out, returns a non-success status, or returns an empty body, mark the trial INFRA and stop. Do not append a freeze row for that event.
Classify, then decide
The labels below are a harness contract. They are not a vendor error catalog, and they are not measured rates.
| Class | Produced by | Freeze allowed? | Effect on admission |
|---|---|---|---|
BLOB_OK |
Local hash of a non-empty diff | No | Required before checks |
INFRA |
Draft timeout, empty body, disconnect, non-success status | No | Stop the trial |
CANARY_FAIL |
Local canary | No | Reject |
PROP_FAIL |
Local non-canary | Only with an unexpired row for that test id and overlapping paths, and only if the test is quarantine-eligible | Reject otherwise |
PROP_PASS |
Local check | No | Counts in the vector |
FREEZE_HIT |
Matching unexpired row | Already stored | Hold, never admit |
A hit means the miss is held, not healed. Promotion still needs a later vector with an empty held list, or a human decision recorded outside this harness.
Five sketched trials fix the vocabulary. They are specifications for the example module, not field observations and not a sample of production runs.
| Trial | Diff | Canaries | Other properties | Matching freeze | Verdict |
|---|---|---|---|---|---|
| T1 | non-empty | pass | pass | none | admit |
| T2 | non-empty | one fail | freeze hit | yes | reject |
| T3 | non-empty | pass | one fail | none | reject |
| T4 | empty | not run | not run | none | reject |
| T5 | non-empty | pass | one eligible freeze hit | yes, unexpired | hold |
T2 is the case that gets skipped under time pressure. A freeze on a slow integration test must not rescue a failed canary. T5 is the strongest result a freeze can buy, and it is still not admission.
The judge module
The module is a local proposal. It performs no network I/O. It encodes the branches so a review can rerun them, and it does not claim a flake rate, a latency figure, or a comparison against any hosted runner.
import hashlib
import json
from dataclasses import dataclass
from enum import Enum
class Kind(Enum):
CANARY_FAIL = "CANARY_FAIL"
PROP_FAIL = "PROP_FAIL"
PROP_PASS = "PROP_PASS"
FREEZE_HIT = "FREEZE_HIT"
@dataclass(frozen=True)
class Check:
test_id: str
canary: bool
quarantine_eligible: bool
kind: Kind
def blob_id(diff: bytes) -> str:
return hashlib.sha256(diff).hexdigest()
def admit(diff: bytes, checks: list[Check], freeze_ids: set[str], channel_infra: bool = False) -> dict:
if channel_infra:
return {"blob": blob_id(diff) if diff.strip() else None, "verdict": "reject", "reason": "infra"}
if not diff.strip():
return {"blob": None, "verdict": "reject", "reason": "empty_blob"}
bid = blob_id(diff)
canary_bad, illegal, uncovered, held = [], [], [], []
for c in checks:
if c.canary and c.kind != Kind.PROP_PASS:
canary_bad.append(c.test_id)
continue
if c.kind == Kind.PROP_PASS:
continue
freeze_ok = (
c.test_id in freeze_ids
and c.quarantine_eligible
and not c.canary
and c.kind in (Kind.PROP_FAIL, Kind.FREEZE_HIT)
)
if freeze_ok:
held.append(c.test_id)
elif c.kind == Kind.FREEZE_HIT:
illegal.append(c.test_id)
else:
uncovered.append(c.test_id)
if canary_bad or illegal or uncovered:
return {
"blob": bid,
"verdict": "reject",
"canary_bad": canary_bad,
"illegal_freeze": illegal,
"uncovered_fail": uncovered,
}
return {"blob": bid, "verdict": "hold" if held else "admit", "held": held}
if __name__ == "__main__":
diff = b"--- a/calc.py\n+++ b/calc.py\n@@\n-return a - b\n+return a + b\n"
t1 = [
Check("parse_diff", True, False, Kind.PROP_PASS),
Check("no_secret_marker", True, False, Kind.PROP_PASS),
Check("add_commutes", False, True, Kind.PROP_PASS),
]
t2 = [
Check("parse_diff", True, False, Kind.CANARY_FAIL),
Check("slow_integration", False, True, Kind.FREEZE_HIT),
]
print("T1", json.dumps(admit(diff, t1, set())))
print("T2", json.dumps(admit(diff, t2, {"slow_integration"})))
print("T4", json.dumps(admit(b" ", [], set())))
Run the specification on a machine you control:
python3 judge.py
test -s patch.diff && git apply --check patch.diff
sha256sum patch.diff || shasum -a 256 patch.diff
T1 must print admit. T2 must print reject and list parse_diff under canary_bad, even though slow_integration is a freeze hit. T4 must print empty_blob. If T2 admits, the branch is wrong. Fix that before any discussion of how long a freeze may live.
Canaries for a small library patch stay short. They also stay ineligible for quarantine, which is a property of the check, not a mood on review day.
CANARIES = ("parse_diff", "apply_check", "no_secret_marker", "public_signature")
def public_signature(before: str, after: str) -> bool:
# Example only: exported names may not vanish.
# Swap this scan for the parser the repository already trusts.
return "def add(" in before and "def add(" in after
A non-canary property stays seeded. The seed list is part of the trial record, beside the blob id, not inside a chat log.
def add_commutes(op, pairs=((0, 0), (1, -2), (10**6, 3))):
return all(op(a, b) == op(b, a) for a, b in pairs)
A pass on this list does not license an unrecorded pair. Widening the list mid-trial creates a different check. Store the new list under a new record, and do not reuse the old verdict.
Numbered run
Pin the base revision. Copy fixtures to a directory the generator cannot write, then hash each fixture path on its own. If any hash changes during the run, reject the trial. The diff has not earned a grade against moving inputs.
Accept a candidate only as
patch.diff. A free model call may create that file, and a free server may host the draft step when a local model process is awkward. Neither process writes fixtures. Neither process returns the verdict.Refuse an empty file before properties start.
test -s patch.diffis the first gate. An empty body from the draft channel isINFRA, not a property miss, so it does not extend the freeze list and it does not consume a canary result.Apply-check against the pinned base with
git apply --check patch.diff. Failure is the canaryapply_check. Do not open a freeze, and do not ask the draft channel for a repair inside the same trial. A changed diff is a new blob id, even when the request id matches.Bind later rows to
sha256(patch.diff)plus test id plus touched paths. Two drafts that differ by one line are different trials. The draft request id is a side-file field, not a lookup key.Run canaries in a fixed order: parse, apply-check, secret-marker scan, public signature. Short-circuit on the first miss. Later properties are not partial credit, and they are not run just to fill the log.
Run quarantine-eligible properties only after canaries pass. Keep their timeout on a different clock from the draft channel. A slow judge is not
INFRA. Reserve that label for the channel that was supposed to deliver a diff.Consult freeze rows last. A usable row has the same test id, overlapping touched paths,
quarantine_eligible=true, and an expiry still in the future. Drop any one of those fields and the miss staysPROP_FAIL. There is no implicit match.Emit one JSON object: blob id, base revision, hash of the seed list, verdict, and the check vector. Store the draft transcript beside it, not inside it.
holdis not mergeable.admitis a review input, not a merge instruction.
Fixture discipline
Use one small fixture per property. A snapshot of a whole service response fails for too many reasons at once. Unclear failures are how freeze rows get approved when a queue is long.
Do not let the generator refresh fixtures. If a property needs the network, take it off the admission path or replace the call with a fixture a reviewer has already read. Timing noise from a socket is an availability problem wearing a test name.
Record the paths a check opened, not one digest of the fixture root. A root digest can move because an unread file changed, and a narrow path hash stays tied to the bytes that check actually used. The verdict can defend only the narrower claim.
Limits, and who should skip this
The function rejects an empty blob, a channel fault, a canary miss, an illegal freeze, and an uncovered property fail. It does not prove the patch is correct. Properties that were not listed do not run.
A secret-marker scan is a filter, not a security review. It will miss secrets that do not match the marker you coded. Treat that gap as a reason for human review, not as a prompt to weaken the canary.
No flake rate appears in this note. Collect one only with a sample plan and a clock you control. Do not paste a figure from a status page, a dashboard tile, or another article into the freeze policy. A stale number is worse than an explicit gap.
Skip the workflow if the judge cannot run on hardware you control. Skip it if one model call both writes the diff and asserts success. Skip it for authentication, cryptography, production credentials, and data migrations: those changes need a person and a stronger suite than four canaries plus a commutativity check.
Also skip it when the freeze store cannot store expiry and test id together. A freeze without expiry is a standing waiver. A standing waiver is a different policy, and this harness should refuse to load it.
Leave the grade local
Use an external draft channel only to widen the set of candidate blobs. If that channel is MonkeyCode's current free model access or free server option, read the live terms, then keep judge.py on the runner that already holds the fixtures.
Cheap drafts do not relax the rule. A canary miss still blocks the patch before any freeze is consulted.
Top comments (0)