Unscored Is Not Green: Four States for Agent-Patch Gates

Boolean CI is the wrong type for agent patches. A job that timed out, skipped a frozen flake without a receipt, or never finished a remote review lane did not pass. It was unscored. Merge gates that collapse those cases into green will accept diffs that never faced the properties you think they faced.

The fix is a four-state gate, not a louder linter. Local property checks and fixture locks stay deterministic. Flaky tests freeze as skip-with-receipt, not as mute. Any optional remote review—including a free model on a free server—must return the same four states. Silence is not success.

The four states

Map every scoring lane to exactly one of these values. Do not invent a fifth “mostly fine” bucket.

1. PASS — the lane ran to completion, hit its cardinality floor, and found no violation.
2. FAIL — the lane completed and produced a locked counterexample, a fixture mismatch, or a property violation.
3. SKIP — the lane intentionally did not run a case, and it emitted a freeze receipt that matches the ledger.
4. UNSCORED — the lane did not complete honestly: timeout, empty payload, cardinality miss, missing receipt, or parse failure.

PASS is the only merge-ok value from a required lane. FAIL blocks. SKIP is allowed only when receipts reconcile. UNSCORED blocks or routes to a human. Treat UNSCORED as closer to FAIL than to PASS. The algebra is boring on purpose.

Agent patches make this typing necessary. Models edit tests, rename cases, and “fix” flakes by deleting assertions. A boolean job that exits 0 after those edits is not evidence. It is a missing measurement.

Freeze is a receipt, not a mute

A flaky freeze list that only deletes tests from the run is a green-washing tool. The freeze must produce a countable receipt. CI then checks that receipt count, signatures, and expiry match the ledger. A mismatch is UNSCORED, even if every executed test passed.

Keep the ledger next to the suite, not in a wiki. Signatures beat names. Names drift when an agent renames test_retry_timeout to test_retry_timeout_skipped.

{
  "schema": "freeze-ledger.v1",
  "required_lanes": ["properties", "fixtures"],
  "optional_lanes": ["remote_review"],
  "cardinality_floor": {
    "properties": 64,
    "fixtures": 1,
    "remote_review": 1
  },
  "freezes": [
    {
      "id": "fx-20260922-01",
      "signature": "sha256:7c1a…e91",
      "path": "tests/property/test_retry_budget.py",
      "case_id": "retry_budget_under_jitter",
      "reason": "timing-dependent under shared runners",
      "expires_utc": "2026-10-06T00:00:00Z",
      "owner": "test-infra"
    }
  ]
}

signature is a hash of the failing assertion text plus the minimal fixture bytes, not the test function name. expires_utc is mandatory. An expired freeze that still skips is UNSCORED. A freeze whose signature no longer matches the file on disk is also UNSCORED: the agent may have edited the case out from under the ledger.

Property checks need a cardinality floor for the same reason. If the seed stream was supposed to execute 64 cases and the runner died at 11, those 11 green results are not a PASS. They are a partial sample. Partial samples are UNSCORED.

Numbered workflow

Use this sequence on every agent patch. Do not start with the remote lane.

1. Lock inputs. Pin property seeds, fixture paths, and the freeze ledger at the parent revision. Record the git tree of tests/ before the agent edits anything.
2. Run the deterministic lanes. Execute property checks and fixture comparisons with a wall-clock cap. Write a machine-readable report, not only stdout.
3. Reconcile receipts. For every freeze entry that is still unexpired, require one receipt. Extra skips, missing skips, and signature misses become UNSCORED.
4. Apply the cardinality floor. If executed cases < floor, mark that lane UNSCORED even when failures == 0.
5. Optionally run remote review. Only if every required lane is PASS or a reconciled SKIP, send a bounded packet (diff, freeze receipts, property report) to a remote reviewer.
6. Fold states. Required-lane FAIL wins. Required-lane UNSCORED wins over PASS. Optional-lane FAIL can block or warn; optional-lane UNSCORED must not upgrade the build to green.
7. Publish the score object. CI artifacts should store the four-state object. Dashboards that can only store booleans are not a reason to throw the object away.

The order matters. Remote review of a patch whose property lane never finished is theater.

Reference classifier

The following module is a complete, runnable classifier for lane reports. It does not call a network. Feed it JSON from your test runner. Label any adapter that talks to a vendor API as unexecuted until you wire it.

#!/usr/bin/env python3
"""Classify agent-patch lane reports into PASS|FAIL|SKIP|UNSCORED."""
from __future__ import annotations

import json
import sys
from datetime import datetime, timezone
from pathlib import Path
from typing import Any, Literal

State = Literal["PASS", "FAIL", "SKIP", "UNSCORED"]
RANK = {"PASS": 0, "SKIP": 1, "UNSCORED": 2, "FAIL": 3}


def parse_ts(value: str) -> datetime:
    return datetime.fromisoformat(value.replace("Z", "+00:00"))


def classify_lane(lane: str, report: dict[str, Any], ledger: dict[str, Any], now: datetime) -> State:
    if report.get("status") in {"timeout", "crash", "empty", "parse_error"}:
        return "UNSCORED"
    if report.get("violations"):
        return "FAIL"

    floor = ledger["cardinality_floor"].get(lane, 1)
    executed = int(report.get("executed", 0))
    if executed < floor:
        return "UNSCORED"

    receipts = {r["id"]: r for r in report.get("receipts", [])}
    expected = [f for f in ledger.get("freezes", []) if f.get("lane", lane) in {lane, "*"}]
    # Fixture/property freezes apply to the lane that would have run them.
    if lane in {"properties", "fixtures"}:
        expected = [f for f in ledger.get("freezes", [])]

    for freeze in expected:
        if parse_ts(freeze["expires_utc"]) <= now:
            if freeze["id"] in receipts:
                return "UNSCORED"  # expired freeze still skipping
            continue
        receipt = receipts.get(freeze["id"])
        if receipt is None:
            # Not skipped; that is fine if the case ran. Check it executed.
            if freeze["case_id"] not in set(report.get("ran_case_ids", [])):
                return "UNSCORED"
            continue
        if receipt.get("signature") != freeze["signature"]:
            return "UNSCORED"

    extra = set(receipts) - {f["id"] for f in expected}
    if extra:
        return "UNSCORED"

    if report.get("skipped") and not receipts:
        return "UNSCORED"
    if report.get("skipped") and receipts and executed == 0:
        return "SKIP"
    return "PASS"


def fold(states: dict[str, State], required: list[str]) -> State:
    acc: State = "PASS"
    for name in required:
        acc = acc if RANK[acc] >= RANK[states[name]] else states[name]
    return acc


def main() -> int:
    ledger = json.loads(Path(sys.argv[1]).read_text())
    reports = json.loads(Path(sys.argv[2]).read_text())
    now = datetime.now(timezone.utc)
    states = {
        lane: classify_lane(lane, reports[lane], ledger, now)
        for lane in reports
    }
    required = fold(states, ledger["required_lanes"])
    optional = {
        name: states[name]
        for name in ledger.get("optional_lanes", [])
        if name in states
    }
    out = {"lanes": states, "required_fold": required, "optional": optional}
    print(json.dumps(out, indent=2))
    return 0 if required == "PASS" else 1


if __name__ == "__main__":
    raise SystemExit(main())

A lane report the runner should emit looks like this. Keep it small. Agents should not be allowed to edit the report schema in the same patch they are scored on.

{
  "properties": {
    "status": "ok",
    "executed": 64,
    "violations": [],
    "skipped": false,
    "ran_case_ids": ["retry_budget_under_jitter"],
    "receipts": []
  },
  "fixtures": {
    "status": "ok",
    "executed": 12,
    "violations": [],
    "skipped": false,
    "ran_case_ids": ["oracle_host_lock"],
    "receipts": []
  }
}

Run it as a post-step, after pytest or your property runner, never as a replacement for them.

python score_states.py freeze_ledger.json lane_reports.json
echo "classifier_exit:$?"

Decision table

Use the table as the merge policy. Do not special-case “the model was probably right.”

Required fold	Optional remote review	Merge
FAIL	any	block
UNSCORED	any	block or human queue
SKIP (all required cases frozen, receipts ok)	PASS or UNSCORED	block; a fully frozen required lane is not a test
PASS	PASS	allow
PASS	FAIL	block or hold; do not auto-green
PASS	UNSCORED	allow only if policy marks remote as non-gating; never treat as extra evidence
PASS	missing (lane not run)	allow; optional means optional

Two rows need emphasis. A required lane that is entirely SKIP is not PASS: you froze the evidence. An optional UNSCORED must not be counted as supporting the patch. Absence of extra evidence is not a new kind of evidence.

Where a free remote lane fits

Deterministic property and fixture lanes should stay on your ordinary CI image. They need pinned seeds and file-system locks, not a chat interface.

A second, optional lane can review the patch summary after those lanes are green: does the diff touch freeze receipts, property modules, or assertion text? That lane is latency-noisy. It can return empty bodies, truncated JSON, or nothing at all. Those outcomes are UNSCORED.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. If you already use MonkeyCode, the optional review lane can sit on its free server option with free model access. Do not put required property execution there. Do not invent a pass from a missing reply. The classifier above does not need a vendor SDK; it only needs a JSON report that says status: empty when the remote lane produces no body.

Cap the packet. Send the diffstat, the freeze receipts, and the property report. Do not send secrets, prod fixtures, or the full tree. If the remote lane cannot finish inside your wall-clock cap, write timeout and stop. Retries without a lease just amplify load; they do not create a PASS.

Limitations

This protocol does not prove the patch is correct. It proves you measured what you claimed to measure. Property checks still need well-chosen invariants. Fixture locks still need oracles that are not the code under test. Freeze ledgers still need humans who expire entries.

The classifier cannot detect a tautology if the agent rewrites assert result == result inside a case that still runs 64 times. Pair this gate with a separate assertion-shape check if that is in your threat model. It also cannot detect a freeze ledger that an agent rewrites in the same commit. Protect freeze_ledger.json and score_states.py with CODEOWNERS or a path filter that always UNSCORES patches touching those files until a human acks them.

Wall-clock caps are not scientific samples. A floor of 64 cases on a noisy runner is a policy number, not a confidence interval. Do not publish it as a coverage percentage. Do not compare vendors with it. The date on this article is 22 September 2026; do not copy quota, hardware, or latency numbers from older posts as if they were current measurements. Measure your runner.

Who should not use this

Skip the four-state fold if your compliance pipeline can only ingest a single boolean and you are unwilling to store the JSON artifact beside it. You will be forced to collapse UNSCORED into PASS or FAIL, which is the bug this article exists to prevent.

Do not use a remote model lane as the only scorer for safety-critical, cryptographic, or access-control patches. Do not use it to replace compilers, typecheckers, or unit tests. Do not use freeze receipts as a permanent skip list for tests that fail on every agent patch; that is how a suite dies of neglect.

Teams without an owner for the freeze ledger should not enable SKIP at all. An unowned ledger becomes a junk drawer. In that case, run properties and fixtures only, and treat any skip as UNSCORED.

The useful default is strict: required lanes PASS, freezes rare and expiring, remote review optional and non-promotable. If you already have a MonkeyCode workspace, wire score_states.py on the deterministic lanes first, then point the optional review lane at a free server only after the classifier rejects empty replies.