Do Not Merge on One Job Color. Score Property, Fixture, and Flake Lanes Separately.

A single green check after an agent patch is not a score. It is a collapse. Property failures, fixture drift, and leased flakes occupy different failure classes, and a CI bit that OR-reduces them erases the class. Merge on three ledgers, or the suite is not evidence.

This article proposes a three-lane gate: a cheap property lane on every agent turn, a fixture-replay lane that never shares a process with the patch, and a flake-lease lane that fails closed when the lease expires without a root-cause hash. The artifact is a small Python scorer plus a decision table. Treat the numbers in the sample files as schema, not as measurements from a production fleet.

Why one job color is the wrong unit

Agent patches fail in mixed ways. A property that never falsifies can still miss a mutant. A fixture that matches last week can still be the agent rewriting the expected file. A flake that “sometimes passes” can hide an ordering bug that the agent just made more frequent.

CI products are built to show one circle. That UI pressure is the bug. If the merge bot only reads conclusion == success, the agent is scored against a lossy projection of the suite, not against the suite.

Keep the three lanes in separate files. Score them in one function. Do not let a flake rerun paint a property miss green.

Lane contracts

Property lane. Bounded generative checks. Same seed recorded in the ledger. The lane reports cases run, cases failed, and how many planted mutants the properties killed. Mutants are review tools. They are not a substitute for production traffic.

Fixture lane. Frozen inputs and expected digests, loaded from a path the patch is not allowed to write. Replay is deterministic. A digest mismatch is oracle_mismatch, not flake.

Flake-lease lane. A test may be leased as timing, order, or environment. The lease has an expiry. No root-cause hash by expiry means fail closed. A lease is not a skip.

Those three contracts must not share a pytest exit code. Pytest can still run. The merge bot reads the ledgers.

Artifact: ledger files and a merge scorer

Proposed layout:

ci/ledgers/property.json
ci/ledgers/fixture.json
ci/ledgers/flake_leases.json
ci/score_lanes.py

Sample property ledger (schema only):

{
  "suite_id": "props-parser-v4",
  "seed": 42,
  "cases_run": 200,
  "cases_failed": 0,
  "killed_mutants": 12,
  "mutant_budget": 10,
  "writer": "property-runner"
}

Sample fixture ledger:

{
  "suite_id": "fix-parser-v4",
  "cases": [
    {
      "id": "roundtrip-ascii",
      "input_digest": "sha256:9c1d...",
      "expected_digest": "sha256:4aa0...",
      "actual_digest": "sha256:4aa0...",
      "status": "match"
    }
  ],
  "writer": "fixture-runner"
}

Sample flake lease:

{
  "leases": [
    {
      "test_id": "test_concurrent_parse",
      "class": "order",
      "leased_at": "2026-09-18T00:00:00+00:00",
      "expires_at": "2026-09-25T00:00:00+00:00",
      "root_cause_hash": null,
      "max_reruns": 2,
      "reruns_used": 1
    }
  ]
}

The scorer below is executable Python 3 stdlib. It does not call a network. It does not claim a measured flake rate for your repo.

#!/usr/bin/env python3
"""Score three test lanes. Proposed merge policy, not a vendor lock file."""
from __future__ import annotations

import argparse
import json
import sys
from datetime import datetime, timezone
from pathlib import Path
from typing import Any


class LaneError(Exception):
    pass


def load(path: Path) -> dict[str, Any]:
    with path.open(encoding="utf-8") as fh:
        data = json.load(fh)
    if not isinstance(data, dict):
        raise LaneError(f"{path} is not an object")
    return data


def score_property(doc: dict[str, Any]) -> list[str]:
    fails = []
    if int(doc.get("cases_failed", 1)) != 0:
        fails.append("property: cases_failed != 0")
    if int(doc.get("cases_run", 0)) <= 0:
        fails.append("property: cases_run is 0")
    killed = int(doc.get("killed_mutants", 0))
    budget = int(doc.get("mutant_budget", 0))
    if killed < budget:
        fails.append(f"property: killed_mutants {killed} < budget {budget}")
    if doc.get("writer") != "property-runner":
        fails.append("property: writer is not property-runner")
    return fails


def score_fixture(doc: dict[str, Any]) -> list[str]:
    fails = []
    if doc.get("writer") != "fixture-runner":
        fails.append("fixture: writer is not fixture-runner")
    cases = doc.get("cases") or []
    if not cases:
        fails.append("fixture: empty case list")
    for case in cases:
        cid = case.get("id", "?")
        if case.get("status") != "match":
            fails.append(f"fixture: {cid} status={case.get('status')}")
        if case.get("expected_digest") != case.get("actual_digest"):
            fails.append(f"fixture: {cid} oracle_mismatch")
    return fails


def score_flakes(doc: dict[str, Any], now: datetime) -> list[str]:
    fails = []
    warns = []
    for lease in doc.get("leases") or []:
        tid = lease.get("test_id", "?")
        klass = lease.get("class")
        if klass not in {"timing", "order", "environment"}:
            fails.append(f"flake: {tid} unknown class {klass}")
        expires = datetime.fromisoformat(lease["expires_at"]).astimezone(timezone.utc)
        cause = lease.get("root_cause_hash")
        if now >= expires and not cause:
            fails.append(f"flake: {tid} lease expired with no root_cause_hash")
        if int(lease.get("reruns_used", 0)) > int(lease.get("max_reruns", 0)):
            fails.append(f"flake: {tid} exceeded max_reruns")
        if cause is None and now < expires:
            warns.append(f"flake: {tid} still leased as {klass}")
    return fails, warns


def main() -> int:
    p = argparse.ArgumentParser()
    p.add_argument("--property", type=Path, required=True)
    p.add_argument("--fixture", type=Path, required=True)
    p.add_argument("--flakes", type=Path, required=True)
    args = p.parse_args()
    now = datetime.now(timezone.utc)

    prop = load(args.property)
    fix = load(args.fixture)
    flakes = load(args.flakes)

    fails = []
    fails.extend(score_property(prop))
    fails.extend(score_fixture(fix))
    flake_fails, warns = score_flakes(flakes, now)
    fails.extend(flake_fails)

    verdict = {
        "utc": now.isoformat(),
        "verdict": "fail" if fails else "pass",
        "fails": fails,
        "warns": warns,
        "lanes": {
            "property_suite": prop.get("suite_id"),
            "fixture_suite": fix.get("suite_id"),
            "lease_count": len(flakes.get("leases") or []),
        },
    }
    json.dump(verdict, sys.stdout, indent=2)
    sys.stdout.write("\n")
    return 1 if fails else 0


if __name__ == "__main__":
    raise SystemExit(main())

Run it as a merge-time check, not as a replacement for the test runner:

python3 ci/score_lanes.py \
  --property ci/ledgers/property.json \
  --fixture ci/ledgers/fixture.json \
  --flakes ci/ledgers/flake_leases.json

Exit 0 is merge-eligible. Exit 1 is not. Warnings on stdout are not a pass token.

Decision table

Observation	Lane	Merge
cases_failed > 0	property	fail
killed_mutants < mutant_budget	property	fail
expected_digest != actual_digest	fixture	fail
fixture writer is the agent job	fixture	fail
lease class not in {timing, order, environment}	flake	fail
lease expired, root_cause_hash null	flake	fail closed
reruns_used > max_reruns	flake	fail
lease active, not expired, cause null	flake	pass with warn
all three ledgers closed	all	pass

Do not add a fourth column named “retry until green.” Reruns belong inside a lease, with a cap.

Numbered workflow

1. Split the jobs. Job A runs properties and writes property.json. Job B mounts fixtures read-only and writes fixture.json. Job C only renews or expires leases. The agent patch job may not write any of the three files.

2. Record the seed, not the test title. Property order can change. The ledger stores seed and suite_id. If the agent renames a test, the property lane still scores the same generator.

3. Classify every red before you lease. Timing: clock, sleep, HTTP. Order: threads, sets, hashmap iteration. Environment: DNS, disk, sibling jobs. If you cannot pick a class, it is not a flake. It is a fail.

4. Cap reruns on the lease. Two reruns is a starting policy, not a finding. A third green does not mint a root-cause hash.

5. Expire in public. Put expires_at in the PR body. When the date passes, the scorer fails the merge even if pytest is green. That is the point.

6. Plant mutants on the property lane only. A mutant is a deliberate, reviewed edit in a scratch clone. If properties cannot kill the mutant budget, the lane is too weak to score an agent patch. Do not plant mutants into fixture expected files.

7. Score at merge, not at “tests completed.” The bot calls score_lanes.py. Chat comments that say “all tests passed” are not an input.

Where a free model and a free server belong

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The property lane should stay cheap and local. Fixture replay is often the slow path: large inputs, digest checks, isolation from the patch workspace. MonkeyCode’s free model access and free server option are relevant here only as capacity, not as an oracle.

A free model can draft candidate properties from a diff. Those drafts are hypotheses. They go to a holding file. They do not write property.json, and they do not write fixture expected digests. A reviewer accepts or deletes them. If you cannot name the invariant in one sentence, discard the draft.

A free server can run Job B: mount the fixture volume read-only, replay, write fixture.json, and stop. Keep Job A on the PR runner so a property miss still blocks in minutes. Do not send the patch workspace to the fixture runner in writable form. If the remote job can edit expected files, you no longer have a fixture lane.

Do not treat free access as a SLA. If the extra runner is unavailable, fail closed on the fixture lane rather than merging on properties alone. A missing ledger is a fail, same as a mismatch.

Limitations

The scorer does not detect tautologies. A property that asserts x == x can kill zero interesting mutants and still pass cases_failed == 0 if you set mutant_budget to 0. Set the budget above zero or the property lane is a heartbeat.

Leases do not fix races. They bound how long you may ignore them. An order lease that is renewed twice is a process failure, not a flaky test.

Fixture digests do not encode meaning. Two byte-identical outputs can both be wrong if the frozen expected file was wrong before the agent ran. Ledger isolation does not replace a human oracle owner.

The mutant budget is a local heuristic. It is not a published coverage standard, and it is not comparable across repos without listing the mutant operators.

Who should not use this

Do not use three ledgers if the repo has no owner for fixtures. The files will rot, and the scorer will become a second flaky job.

Do not use flake leases on tests that assert business totals, authz, or money. Those failures are not timing classes.

Do not offload the property lane to a remote runner you cannot inspect. Properties are the fast veto. If they queue for an hour, people will skip them.

Do not point a model at the fixture directory and ask it to “make CI green.” That request is how expected files become the patch.

Close

Score the patch as three verdicts. Property. Fixture. Flake lease. If you already keep generation off the review path, parking fixture replay on a free server is optional capacity, not a new definition of done. The merge bot should read the ledgers. It should not read a single job color.