DEV Community

Finley Zhou
Finley Zhou

Posted on

Split Agent-Patch CI Into Three Verdicts, Not One Green Job

#testing #python #ci #ai

A green check on a mixed CI job is not a merge signal for an agent-authored patch. Split the decision into three independent verdicts that must all pass: a src-only property score, an oracle-integrity check, and a flake lease bound to bytecode rather than to a test name.

Agents edit tests. That is the failure mode this workflow is built to catch. A diff that deletes an assertion, rewrites a fixture, or skips a noisy case can look identical to a diff that closes the defect.

Property checks still matter. They stop mattering the moment the generator, the fixture tree, or the skip list moves in the same checkout as src/.

Why a single job lies

One pipeline that installs the branch, runs the suite, and publishes a status check gives the agent control of the measuring stick. Flakes then become a second escape hatch. The job turns green because the suite got quieter, not because the production path got stricter.

Live clocks and live networks make the lie cheaper. A property that samples time.time() or a fixture that hits a remote stub will fail on one runner and pass on the next. The agent then “fixes” CI. The usual fix is a skip.

The merge rule has to stop treating that skip as evidence.

The three verdicts

Keep the jobs separate. Join them with AND. Not majority. Not “src-score passed, ignore the rest.”

  1. Src-score. Apply the agent diff to src/ only. Mount tests/sealed/ and fixtures/sealed/ read-only. Run properties with a fixed clock and a fixed RNG seed.
  2. Oracle-integrity. Fail if sealed tests, sealed fixtures, or the lease file change in the same patch as production code.
  3. Flake-lease. Allow xfail or skip only when a lease matches the current failing bytecode hash and has not expired.

tests/proposed/ may grow. Those files never enter src-score. They are review artifacts, not oracles.

Layout the gate can enforce 

repo/
  src/
  tests/
    sealed/          # scoring oracle; agents do not write here
    proposed/        # agent-authored extras; ignored by src-score
  fixtures/
    sealed/
  ci/
    flake_leases.json
    oracle_gate.py
    src_score.sh
Enter fullscreen mode Exit fullscreen mode

Sealed paths are the constitution. If a patch needs a new oracle, that is a human change set with a different join rule. Do not fold it into the agent’s production diff.

Step 1 — Classify the diff before any test runs

Compute three path sets from git diff --name-only origin/main. Classification is cheap. It also removes arguments later.

git diff --name-only origin/main...HEAD > /tmp/changed.txt

awk '
  /^src\// { src=1 }
  /^tests\/sealed\// || /^fixtures\/sealed\// || $0=="ci/flake_leases.json" { oracle=1 }
  /^tests\/proposed\// { proposed=1 }
  END {
    printf "src=%d oracle=%d proposed=%d\n", src+0, oracle+0, proposed+0
  }
' /tmp/changed.txt
Enter fullscreen mode Exit fullscreen mode

If src=1 and oracle=1 in the same patch, oracle-integrity fails immediately. Do not run properties yet. The score would be self-referential.

Step 2 — Apply src-only onto a read-only oracle volume

Copy HEAD into a work tree. Reset sealed tests and fixtures to origin/main. Then overlay only src/ from the agent commit. The scoring tree now has new production code and old oracles.

#!/usr/bin/env bash
set -euo pipefail
ROOT="$(git rev-parse --show-toplevel)"
SCORE="$(mktemp -d)"
BASE="${BASE_SHA:-origin/main}"
HEAD_SHA="${HEAD_SHA:-HEAD}"

git archive --format=tar "$HEAD_SHA" | tar -x -C "$SCORE"
git -C "$ROOT" archive --format=tar "$BASE" -- tests/sealed fixtures/sealed ci/flake_leases.json ci/oracle_gate.py \
  | tar -x -C "$SCORE"

export PYTHONHASHSEED=0
export SOURCE_DATE_EPOCH=1700000000
export TZ=UTC
cd "$SCORE"
python ci/oracle_gate.py --mode src-score
Enter fullscreen mode Exit fullscreen mode

The overlay is the whole trick. If the agent rewrote a sealed assertion, that rewrite is discarded before scoring. If the product code still fails the old property, src-score stays red.

Step 3 — Bind flake leases to bytecode, not to nodeids

Test names move. Nodeids move. Bytecode of a sealed function is a stabler lease key than either. Hash the sealed test object after import, then require any skip to present that hash plus a failure signature.

# ci/oracle_gate.py
from __future__ import annotations

import ast
import hashlib
import importlib.util
import json
import sys
import time
from pathlib import Path

LEASE_PATH = Path("ci/flake_leases.json")
SEALED_DIR = Path("tests/sealed")


def bytecode_digest(path: Path, qualname: str) -> str:
    spec = importlib.util.spec_from_file_location(path.stem, path)
    mod = importlib.util.module_from_spec(spec)
    spec.loader.exec_module(mod)
    obj = mod
    for part in qualname.split("."):
        obj = getattr(obj, part)
    code = obj.__code__.co_code
    return hashlib.sha256(code).hexdigest()[:16]


def load_leases() -> dict:
    if not LEASE_PATH.exists():
        return {"leases": []}
    return json.loads(LEASE_PATH.read_text())


def collect_skips(tree: ast.AST) -> list[tuple[str, str]]:
    found = []
    for node in ast.walk(tree):
        if not isinstance(node, ast.FunctionDef):
            continue
        for deco in node.decorator_list:
            text = ast.unparse(deco)
            if "skip" in text or "xfail" in text:
                found.append((node.name, text))
    return found


def check_leases(now: float) -> list[str]:
    errors = []
    leases = {row["qualname"]: row for row in load_leases().get("leases", [])}
    for path in SEALED_DIR.rglob("test_*.py"):
        tree = ast.parse(path.read_text())
        for name, deco in collect_skips(tree):
            qualname = f"{path.stem}.{name}"
            row = leases.get(qualname)
            if row is None:
                errors.append(f"unleased skip: {qualname} via {deco}")
                continue
            digest = bytecode_digest(path, name)
            if digest != row.get("bytecode_hash"):
                errors.append(
                    f"lease/bytecode mismatch: {qualname} "
                    f"have={digest} lease={row.get('bytecode_hash')}"
                )
            if now >= float(row["expires_at"]):
                errors.append(f"expired lease: {qualname}")
            if not row.get("failure_signature"):
                errors.append(f"lease missing failure_signature: {qualname}")
    return errors
Enter fullscreen mode Exit fullscreen mode

A lease without a failure signature is a comment. Reject it. A lease whose hash no longer matches means the sealed test body moved; the skip is no longer about the same code.

Example ledger:

{
  "leases": [
    {
      "qualname": "test_parser.parse_trailing_comma",
      "bytecode_hash": "9c1e0a77ab12f0d1",
      "failure_signature": "AssertionError: expected TokenError at col 18",
      "expires_at": 1758614400,
      "owner": "maintainer@example",
      "reason": "repro only on glibc 2.39 + tzdata 2026a"
    }
  ]
}
Enter fullscreen mode Exit fullscreen mode

expires_at is unix time. The gate reads the clock once at process start so lease checks do not race the frozen clock used inside properties.

Step 4 — Run properties against fixtures that cannot move

Src-score should not discover files. It should load an explicit manifest. Manifest drift is an oracle-integrity failure, not a “new coverage win.”

MANIFEST = Path("tests/sealed/MANIFEST.txt")


def property_cases() -> list[Path]:
    listed = [Path(line.strip()) for line in MANIFEST.read_text().splitlines() if line.strip()]
    existing = sorted(SEALED_DIR.rglob("test_*.py"))
    if listed != existing:
        raise SystemExit(f"manifest drift: listed={listed!r} existing={existing!r}")
    return listed
Enter fullscreen mode Exit fullscreen mode

Inside each sealed property, pin the RNG and the clock at the test boundary. Do not pin them inside the production function. If the product code needs time, inject it.

# tests/sealed/test_window.py
import random
from datetime import datetime, timezone

from src.window import rolling_close


def test_rolling_close_is_idempotent_on_frozen_clock():
    rng = random.Random(20260922)
    ticks = [rng.random() for _ in range(64)]
    clock = datetime(2026, 9, 22, 12, 0, tzinfo=timezone.utc)
    first = rolling_close(ticks, now=clock)
    second = rolling_close(ticks, now=clock)
    assert first == second
    assert all(p >= 0 for p in first)
Enter fullscreen mode Exit fullscreen mode

That test is a property: idempotence under a frozen clock, plus a non-negativity invariant. It is not a tautology. Replacing the body with assert True changes bytecode and, if someone also skips it, breaks the lease.

Step 5 — Join rule

Emit three files. The merge job reads them. No other job is allowed to rewrite them.

ci_out/src_score.txt          # PASS|FAIL
ci_out/oracle_integrity.txt   # PASS|FAIL
ci_out/flake_lease.txt        # PASS|FAIL
Enter fullscreen mode Exit fullscreen mode 
python ci/oracle_gate.py --mode oracle-integrity
python ci/oracle_gate.py --mode flake-lease
python ci/oracle_gate.py --mode src-score

join=$(paste -sd' ' ci_out/*.txt)
case "$join" in
  "PASS PASS PASS") echo MERGE_OK ;;
  *) echo MERGE_BLOCKED: $join; exit 1 ;;
esac
Enter fullscreen mode Exit fullscreen mode

The join is boring on purpose. Boring join rules survive agent patches. Smart join rules get negotiated away.

Decision table

Agent change set Src-score Oracle-integrity Flake-lease Merge
src/ only, properties hold, no new skips PASS PASS PASS yes
src/ only, sealed property fails FAIL PASS PASS no
src/ + edit under tests/sealed/ not run FAIL n/a no
src/ + new file in tests/proposed/ scored without it PASS PASS yes, extra tests ignored
skip added to sealed test, no lease not a fix PASS or FAIL FAIL no
skip added, lease hash matches, not expired may PASS PASS PASS yes, temporary
skip added, lease expired or hash drifted may PASS PASS FAIL no
lease file + src/ in one patch not run FAIL n/a no

The last row is the one teams skip. A lease edit is an oracle edit. It needs its own review, after src-score has already passed on the previous ledger.

Where a free model and a free server belong

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode’s free model access is useful for proposing the src/ diff. It is not useful as the scoring host. The model’s workspace is writable, and a writable scoring host is how oracles rot.

Run src_score.sh on MonkeyCode’s free server option instead of on the agent’s tree. The server receives the commit SHAs, rebuilds the overlay, and publishes the three verdict files. The model can iterate on src/ against that remote score. It cannot update tests/sealed/ as a side effect of “making CI green.”

Do not treat that split as a capability claim beyond what it is: untrusted generation, trusted overlay, three-way join.

Limitations

Bytecode hashing is language-specific. The sample gate assumes CPython function objects. It will not see skips constructed at runtime, pytest markers added in conftest.py, or failures inside native extensions.

Frozen clocks do not freeze I/O. If sealed tests still call the network, src-score remains noisy and leases will accumulate. Recorded HTTP fixtures belong under fixtures/sealed/ and must be part of oracle-integrity. This article does not implement a recorder.

The manifest check is exact path equality. Reordering files fails the job. That is intentional. Sorting on both sides is an allowed local change to the gate, not to the oracle.

Expiry does not retry the underlying flake. When a lease dies, the skip becomes a hard failure. Someone has to reproduce it or delete the skip. The gate will not guess.

Who should not use this

Do not use three-verdict scoring if humans already cannot describe a sealed oracle. Snapshot-heavy UI suites, tests that assert on full HTML, and jobs that must hit third-party sandboxes will spend all their time fighting leases.

Do not use it if the agent is also the release engineer. The join rule assumes a reviewer who can reject an oracle edit even when src-score is green.

Do not use it as a substitute for a fuzzer on parsers or codecs. Properties here are regression properties on a pinned seed. They will not enumerate a new grammar class just because the overlay is read-only.

What to keep when you throw the scripts away

Keep the AND join. Keep src-only overlay. Keep skip-as-lease, not skip-as-comment. Those three constraints are the testing strategy. The Python in ci/oracle_gate.py is only a way to make them visible in logs.

If you already generate patches from a free model, point src-score at a free server and refuse to merge until all three verdict files read PASS. That is the whole protocol.

Top comments (0)