Type the Agent-Patch Verdict Before You Freeze Anything

A boolean CI result is the wrong type for an agent patch. Safety breaches, timeouts, sampler variance, and harness faults are four different events. Collapsing them into red versus green is how teams freeze the wrong test and ship the wrong diff.

The fix is a tagged verdict, not a longer skip list. Property checks stay in the pipeline. Fixtures stay content-addressed. Freeze credits are issued only to harness faults, with a hard cap and an expiry. Everything else stays visible.

This is a layout, not a war story. The runner below is a worked example. It is not a production measurement and it does not claim a pass rate.

Four events, one red pixel

Agent patches fail tests for reasons that do not commute. A patch that deletes a row it does not own is a safety failure. A patch that never returns is a liveness gap. A patch that answers 7/10 trials inside a statistical band is sampler noise. A fixture loader that throws FileNotFoundError is a harness fault.

CI that only stores pass or fail cannot tell those apart. The next on-call then freezes the test name. The freeze hides the class of event, not the flake.

Treat the result as a small algebra instead.

Verdict =
  Pass
| FailSafety
| FailContract
| InconclusiveTimeout
| InconclusiveSample
| HarnessError

FailSafety is never freezeable. InconclusiveSample is not a failure of the patch until the inconclusive rate itself exceeds a budget. HarnessError is the only constructor that may draw a freeze credit.

1. Write the verdict type first

Start with the type. Do not start with pytest markers. A marker is a comment. A verdict is data the gate can branch on.

from dataclasses import dataclass
from enum import Enum
from typing import Optional


class Kind(str, Enum):
    PASS = "pass"
    FAIL_SAFETY = "fail_safety"
    FAIL_CONTRACT = "fail_contract"
    INCONCLUSIVE_TIMEOUT = "inconclusive_timeout"
    INCONCLUSIVE_SAMPLE = "inconclusive_sample"
    HARNESS_ERROR = "harness_error"


@dataclass(frozen=True)
class Verdict:
    kind: Kind
    property_id: str
    fixture_digest: str
    detail: str
    trials: int = 1
    hits: int = 0

Keep constructors closed. If a helper wants to return None on timeout, reject it. Timeout is InconclusiveTimeout, not an absent result. Absent results become retries, and retries become accidental freezes.

2. Stratify the properties

Not every property deserves the same failure mode. Split the suite into three bands before the agent is allowed to edit code.

1. Safety. Predicates that must hold on every trial: no extra writes, no auth bypass, no schema drop, no secret in logs. One counterexample is a FailSafety. Zero retries. Zero freezes.
2. Contract. Deterministic relations on a locked fixture: output shape, status code class, idempotent replay of the same digest. A miss is FailContract.
3. Statistical. Predicates that need N independent trials: ranking quality, extraction recall, tool-choice rate. A miss below the band is InconclusiveSample, not a red build, until the window rate trips.

Timeouts sit beside the bands, not inside them. A 2.0s deadline that fires is InconclusiveTimeout. Promote it to FailSafety only when the property is "this call must not hang a checkout path," and document that promotion in the property id.

SAFETY = {
    "no_unmapped_write",
    "no_secret_in_stdout",
    "authz_unchanged",
}
CONTRACT = {
    "json_shape_v3",
    "status_class_stable",
}
STATISTICAL = {
    "tool_choice_rate",
}

If a property cannot be placed, it does not run against the patch. Unclassified checks are how freeze lists grow.

3. Address fixtures by digest, not by filename

A fixture file named happy.json is a magnet for silent edits. The agent, or a well-meaning teammate, can retune the input so the contract passes. Hash the bytes. Bind the property to the digest. Refuse to run if the file moved under the same name.

import hashlib
from pathlib import Path


def digest(path: Path) -> str:
    return hashlib.sha256(path.read_bytes()).hexdigest()[:16]


MANIFEST = {
    "json_shape_v3": {
        "path": Path("fixtures/invoice_v3.json"),
        "sha256_16": "9c1a0e77ab44d2b1",
    },
    "no_unmapped_write": {
        "path": Path("fixtures/write_probe.json"),
        "sha256_16": "e2b1c0aa19f30d44",
    },
}


def load_fixture(property_id: str) -> bytes:
    spec = MANIFEST[property_id]
    got = digest(spec["path"])
    if got != spec["sha256_16"]:
        raise RuntimeError(
            f"fixture drift {property_id}: got {got}, want {spec['sha256_16']}"
        )
    return spec["path"].read_bytes()

A digest mismatch is HarnessError, not FailContract. The patch did not break the relation. The suite lost its domain. That distinction is the entire point of typing the verdict.

Do not golden-file the agent output into this manifest. The manifest is input domain. Expected bytes that the diff can rewrite are not an oracle.

4. Issue freeze credits only to harness faults

A freeze is a quota on infrastructure, not a skip on a property. The ledger records property_id, remaining credits, expiry, and the exact HarnessError detail. It does not record "flaky."

import time
from dataclasses import dataclass


@dataclass
class FreezeCredit:
    property_id: str
    remaining: int
    expires_unix: int
    last_detail: str


LEDGER: dict[str, FreezeCredit] = {}
HARNESS_BUDGET = 2
HARNESS_TTL_S = 6 * 60 * 60


def maybe_freeze(v: Verdict) -> Optional[str]:
    if v.kind is not Kind.HARNESS_ERROR:
        return None
    now = int(time.time())
    slot = LEDGER.get(v.property_id)
    if slot is None or slot.expires_unix <= now:
        LEDGER[v.property_id] = FreezeCredit(
            property_id=v.property_id,
            remaining=HARNESS_BUDGET - 1,
            expires_unix=now + HARNESS_TTL_S,
            last_detail=v.detail,
        )
        return "freeze_opened"
    if slot.remaining <= 0:
        return "freeze_exhausted"
    slot.remaining -= 1
    slot.last_detail = v.detail
    return "freeze_consumed"

Rules that keep the ledger honest:

1. FailSafety and FailContract cannot enter the ledger. A comment in YAML cannot override that.
2. InconclusiveSample cannot enter the ledger. Raise the trial count or fix the sampler.
3. Credits expire. A frozen loader that still throws after six hours is a broken gate, not a flaky property.
4. Exhaustion fails the job as HarnessError. It does not fail the patch.

If the team needs a longer TTL, the TTL is a config change with review. It is not an annotation next to an assertion.

5. Map verdicts onto CI without flattening them

The job exit code can stay binary. The artifact cannot. Write one JSON line per property. Fail the job only for the kinds that mean "do not merge."

import json
import sys

BLOCKING = {Kind.FAIL_SAFETY, Kind.FAIL_CONTRACT}
SAMPLE_WINDOW = 20
SAMPLE_INCONCLUSIVE_MAX = 4


def job_status(verdicts: list[Verdict]) -> int:
    inconclusive = 0
    for v in verdicts:
        sys.stdout.write(json.dumps(v.__dict__) + "\n")
        if v.kind in BLOCKING:
            return 1
        freeze = maybe_freeze(v)
        if freeze == "freeze_exhausted":
            return 2
        if v.kind is Kind.INCONCLUSIVE_SAMPLE:
            inconclusive += 1
        if v.kind is Kind.INCONCLUSIVE_TIMEOUT:
            # Timeouts accumulate; they do not freeze.
            pass
    if inconclusive > SAMPLE_INCONCLUSIVE_MAX:
        sys.stdout.write(
            json.dumps(
                {
                    "kind": "sampler_budget_exceeded",
                    "inconclusive": inconclusive,
                    "window": SAMPLE_WINDOW,
                }
            )
            + "\n"
        )
        return 3
    return 0

Exit 1 blocks the patch. Exit 2 blocks on a broken harness. Exit 3 blocks on a sampler that is too noisy to measure the patch. Those three exits are not interchangeable, and the log lines are what make them auditable.

Worked loop

The loop below is labeled as an example. Swap apply_patch and run_probe for the real sandbox. Keep the verdict constructors unchanged.

import subprocess
import tempfile
from pathlib import Path


def run_probe(workdir: Path, payload: bytes, timeout_s: float) -> bytes:
    proc = subprocess.run(
        [sys.executable, "probe.py"],
        cwd=workdir,
        input=payload,
        capture_output=True,
        timeout=timeout_s,
    )
    if proc.returncode != 0:
        raise RuntimeError(proc.stderr[-400:])
    return proc.stdout


def check_safety_write(stdout: bytes) -> bool:
    return b"UNMAPPED_WRITE" not in stdout


def check_shape(stdout: bytes) -> bool:
    return stdout.strip().startswith(b"{") and stdout.strip().endswith(b"}")


def evaluate(property_id: str, workdir: Path) -> Verdict:
    try:
        payload = load_fixture(property_id)
    except Exception as exc:
        return Verdict(Kind.HARNESS_ERROR, property_id, "missing", str(exc))

    digest_id = MANIFEST[property_id]["sha256_16"]
    try:
        if property_id in STATISTICAL:
            hits = 0
            trials = 10
            for _ in range(trials):
                out = run_probe(workdir, payload, timeout_s=2.0)
                hits += int(check_shape(out))
            kind = (
                Kind.PASS
                if hits >= 8
                else Kind.INCONCLUSIVE_SAMPLE
            )
            return Verdict(kind, property_id, digest_id, "", trials, hits)

        out = run_probe(workdir, payload, timeout_s=2.0)
        if property_id in SAFETY:
            ok = check_safety_write(out)
            kind = Kind.PASS if ok else Kind.FAIL_SAFETY
            return Verdict(kind, property_id, digest_id, "")
        ok = check_shape(out)
        kind = Kind.PASS if ok else Kind.FAIL_CONTRACT
        return Verdict(kind, property_id, digest_id, "")
    except subprocess.TimeoutExpired:
        return Verdict(
            Kind.INCONCLUSIVE_TIMEOUT, property_id, digest_id, "deadline 2.0s"
        )
    except Exception as exc:
        return Verdict(Kind.HARNESS_ERROR, property_id, digest_id, str(exc))

Run it locally against an unpatched tree first. If the unpatched tree already yields FailSafety, the property is wrong. If it already yields HarnessError, the freeze ledger will burn credits before any agent runs. Both are cheaper to learn without a model in the loop.

Decision table

Event	Verdict	Freeze?	Merge?
Extra filesystem write	FailSafety	No	No
JSON shape miss on locked digest	FailContract	No	No
7/10 trials inside a 8/10 band	InconclusiveSample	No	Yes, unless window rate trips
Probe exceeded 2.0s	InconclusiveTimeout	No	Yes, unless the property was promoted to safety
Fixture digest drift	HarnessError	Credit if budget remains	No, if credits exhaust
Sampler inconclusive count > 4 / 20	sampler_budget_exceeded	No	No

The table is the policy. If a case is not in the table, the gate does not guess.

Where a free model and a free server fit

The runner does not need a paid GPU farm. It needs an isolated tree, a patch, and a process boundary for probe.py.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode provides free model access and a free server option. Those two are enough to generate a candidate patch and to execute this verdict loop on a throwaway machine. They do not replace the oracle, the digest manifest, or the freeze ledger. Model names, quotas, and hardware are out of scope here because they are not required to type the result.

Use the free server as the place the probe runs, not as the place policy is invented. Policy stays in the verdict constructors.

Limitations, and who should not use this

The layout does not prove correctness. Statistical bands are measurements with a budget, not theorems. A Pass on ten trials is still a sample.

Do not use this when any of the following hold:

• There is no oracle independent of the diff. If the patch can edit probe.py, the verdict is self-graded.
• The product is safety-critical and requires deterministic certification. Inconclusive kinds are not acceptable evidence there.
• The team wants a skip list. A freeze credit that can attach to FailContract will hide regressions on purpose.
• Properties are unnamed. Anonymous test_it_works functions cannot be stratified or ledgered.
• Fixture files are regenerated from model output. That is a circular domain.

Timeouts also leak into safety if the deadline is a lie. A 2.0s cap on a job that usually takes 1.8s will manufacture InconclusiveTimeout and train the team to ignore it. Measure the unpatched p95 before picking the number. If that measurement is missing, label the deadline as a proposal and keep the verdict inconclusive.

Close the type hole, then stop

The cheapest defect in an agent-patch pipeline is a result that cannot say what happened. Add the six constructors. Bind fixtures to digests. Spend freeze credits only on harness faults. Leave sampler noise in the log until the sampler itself blows the budget.

Once those three rules hold, adding more tests gets cheaper. Until they hold, extra tests only add more names to freeze.