Split CI Into Advisory and Blocking Lanes Before You Merge an Agent Patch

An agent patch that raises coverage and lands a dozen new tests is still not evidence. Evidence is a blocking suite the agent cannot edit, freeze, or skip. Everything else is advisory noise.

Coverage deltas describe what executed. They do not describe what must remain true. That distinction is the whole merge policy.

This article specifies a two-lane CI layout for patches produced by coding agents. Lane A runs generated tests and prints a report. Lane B runs human-owned properties against frozen fixtures and is the only job allowed to fail the merge. The workflow is a proposal with runnable examples. It is not a claim about a production fleet.

Why a single suite fails

Agents optimize for green. A single pytest tree rewards three cheap moves: assert the new behavior, rewrite the old assertion, or mark a flicker as xfail. Each move increases the chance the patch merges. None of them preserve the previous contract.

A merged test file is not an oracle. An oracle is a check whose author, path, and fixture digest are outside the agent's write set. If those three can move in the same commit as the production diff, the suite is documenting the patch, not judging it.

Lane contract

Signal	Lane A (advisory)	Lane B (blocking)
Path prefix	tests/advisory/	tests/oracles/
Who may add files in the patch	agent or human	human only
Merge effect of failure	comment, never block	block
Coverage delta	printed, ignored by gate	not collected
Flaky outcome	dropped from the report	fail closed; no skip
Fixture lifetime	disposable	SHA-256 manifest, committed

Lane A is allowed to be wrong. Lane B is not allowed to be quiet.

1. Pin ownership in the tree, not in chat

Create two roots. Do not rely on test names, markers alone, or a bot comment that says "these are the real tests."

tests/
  advisory/                 # generated, report-only
    test_generated_*.py
  oracles/                  # human-owned, merge-blocking
    properties/
    fixtures/
      MANIFEST.sha256
    conftest.py
CODEOWNERS
.pyrefix-lanes.toml

Example CODEOWNERS fragment (proposal):

/tests/oracles/  @oracle-owners

Example lane file:

# .pyrefix-lanes.toml
[lanes.advisory]
root = "tests/advisory"
merge = "report"

[lanes.blocking]
root = "tests/oracles"
merge = "required"
forbid_agent_writes = true

The gate reads paths. It does not parse commit messages.

2. Reject the patch if Lane B paths move

Run this check on the merge SHA before any test job. A single added, deleted, or edited file under tests/oracles/ fails the patch unless the author is in CODEOWNERS and the change is a dedicated oracle PR.

# tools/forbid_oracle_writes.py
# Proposal: run against `git diff --name-only base...HEAD`
from pathlib import Path
import subprocess
import sys

ORACLE_ROOT = Path("tests/oracles")
ALLOWED = {"human-oracle-pr"}  # label check happens in CI, not here

def changed_files(base: str) -> list[str]:
    out = subprocess.check_output(
        ["git", "diff", "--name-only", f"{base}...HEAD"],
        text=True,
    )
    return [line.strip() for line in out.splitlines() if line.strip()]

def main(base: str) -> int:
    hits = [
        path for path in changed_files(base)
        if Path(path) == ORACLE_ROOT or ORACLE_ROOT in Path(path).parents
        or path.startswith("tests/oracles/")
    ]
    if hits:
        print("blocking lane paths changed in an agent patch:")
        print("\n".join(hits))
        return 2
    return 0

if __name__ == "__main__":
    sys.exit(main(sys.argv[1] if len(sys.argv) > 1 else "origin/main"))

Command:

python tools/forbid_oracle_writes.py origin/main

If this process exits non-zero, do not schedule Lane A. The patch already invalidated the evidence.

3. Freeze fixtures, not test titles

Lane B properties read committed fixtures. The freeze target is the digest file, not test_refund_window as a string. Titles churn. Bytes do not, unless a human intended it.

# tests/oracles/conftest.py
from pathlib import Path
import hashlib
import json

ROOT = Path(__file__).parent / "fixtures"
MANIFEST = ROOT / "MANIFEST.sha256"

def load_manifest() -> dict[str, str]:
    rows = {}
    for line in MANIFEST.read_text().splitlines():
        digest, name = line.split("  ", 1)
        rows[name] = digest
    return rows

def test_fixture_manifest_is_sealed():
    expected = load_manifest()
    found = {
        p.name: hashlib.sha256(p.read_bytes()).hexdigest()
        for p in sorted(ROOT.glob("*.json"))
    }
    assert found == expected, (found, expected)

Rebuild the manifest only in an oracle PR:

cd tests/oracles/fixtures
: > MANIFEST.sha256
for f in *.json; do
  printf '%s  %s\n' "$(sha256sum "$f" | awk '{print $1}')" "$f" >> MANIFEST.sha256
done

A property that needs a new fixture is a specification change. It is not a side effect of an agent refactor.

4. Keep properties boring and local

Lane B properties should be small predicates over frozen input. They should not call the network, the clock, or the agent's prompt. The example below is labeled as a template, not as a measured suite.

# tests/oracles/properties/test_invoice_totals.py
import json
from decimal import Decimal, ROUND_HALF_EVEN
from pathlib import Path

FIXTURE = Path(__file__).resolve().parents[1] / "fixtures" / "invoices.v1.json"

def money(value: str) -> Decimal:
    return Decimal(value).quantize(Decimal("0.01"), rounding=ROUND_HALF_EVEN)

def test_line_items_sum_to_header_total():
    payload = json.loads(FIXTURE.read_text())
    for invoice in payload["invoices"]:
        parts = [money(item["amount"]) for item in invoice["lines"]]
        assert sum(parts, Decimal("0.00")) == money(invoice["total"])

def test_tax_never_exceeds_subtotal():
    payload = json.loads(FIXTURE.read_text())
    for invoice in payload["invoices"]:
        assert money(invoice["tax"]) <= money(invoice["subtotal"])

If the agent "fixes" a failing total by editing the fixture, step 2 already failed the patch. If it "fixes" the property, the same gate fires. The only remaining move is to change production code until Lane B is green.

5. Wire two CI jobs with unequal power

Proposal for GitHub Actions. Adjust the runner image to whatever you already use. Do not copy this as a claim that any vendor job is green today.

# .github/workflows/two-lane.yml
name: two-lane-agent-gate
on:
  pull_request:

jobs:
  protect-oracles:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - run: python tools/forbid_oracle_writes.py origin/${{ github.base_ref }}

  lane-b-blocking:
    needs: protect-oracles
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: pytest tests/oracles -q --maxfail=1

  lane-a-advisory:
    needs: protect-oracles
    if: success()
    runs-on: ubuntu-latest
    continue-on-error: true
    steps:
      - uses: actions/checkout@v4
      - run: pytest tests/advisory -q --junitxml=advisory.xml || true
      - run: python tools/summarize_advisory.py advisory.xml >> $GITHUB_STEP_SUMMARY

continue-on-error: true is the point. Lane A must not be able to veto or bless the merge. A required status check is attached only to lane-b-blocking and protect-oracles.

6. Treat flaky Lane B as a failed lease, not a skip

Do not freeze a flaky blocking test with pytest.mark.skip. A skip is an edit the next agent can copy. Record the environment instead, then rerun.

# tools/lease_env.py
import os, time, json, hashlib

def lease() -> dict:
    payload = {
        "tz": os.environ.get("TZ", ""),
        "lang": os.environ.get("LANG", ""),
        "pyc": os.environ.get("PYTHONHASHSEED", ""),
        "epoch_bucket": int(time.time() // 3600),
    }
    blob = json.dumps(payload, sort_keys=True).encode()
    payload["digest"] = hashlib.sha256(blob).hexdigest()
    return payload

Policy:

1. Lane B failure with a changing digest across two reruns is an environment leak, not a product pass.
2. The human adds the missing control (TZ=UTC, fixed PYTHONHASHSEED, frozen clock injection) inside tests/oracles/conftest.py.
3. Until that lands, the merge stays red.
4. Lane A flakes are deleted from the report. They are not promoted.

A freeze that expires by calendar date is still a skip with extra steps. The lease is a replay key. If you cannot replay, you do not have a test.

7. Generate Lane A off to the side

Candidate tests can be drafted on a throwaway machine. That is the only place a free coding environment belongs in this workflow.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode's free model access and free server option can host that drafting job: throw a failing stack trace and a fixture sample at a model, collect pytest files, and copy them into tests/advisory/ on a branch. They stay advisory until a human rewrites a subset under tests/oracles/ with a sealed manifest. The product is not the gate. The path policy is the gate.

Do not send Lane B fixtures to a remote prompt if those fixtures contain account data. Generate from redacted samples only.

If you already have a local runner, skip the remote box. The lanes do not require it.

Decision table for the merge bot

Observation	Action
tests/oracles/** in the agent diff	fail protect-oracles
Lane B red	fail merge
Lane B green, Lane A red	merge allowed; paste the advisory summary
Coverage +12%, Lane B unchanged	ignore coverage
New xfail or skip under tests/oracles/	treat as an oracle edit; fail
Advisory test copies a property verbatim	keep it in Lane A; do not delete Lane B
Fixture digest drift without an oracle PR	fail merge

The table has no row where coverage unblocks a red oracle.

Limitations

This layout does not prove functional completeness. A blocking property that never exercised the patched branch will stay green. Two lanes reduce self-deal; they do not replace review of the production diff.

Path policy fails open if the agent can write to CODEOWNERS, the workflow file, or tools/forbid_oracle_writes.py. Those paths need the same human-only rule as tests/oracles/.

SHA-256 manifests do not help if the fixture is a tautology ({"ok": true}). Garbage in, sealed garbage out.

Remote generation adds a data-handling problem. Anything you paste into a free server may leave your network. Treat it as public.

Who should not use this

Do not adopt two lanes if the repo has no human-owned examples yet. An empty tests/oracles/ makes Lane B a vacuous pass, which is worse than a single honest suite.

Do not adopt it for exploratory spikes where the contract is the thing being invented. Write the oracle PR first, then let agents patch against it.

Do not adopt it as a way to keep shipping while Lane B is skipped "temporarily." That recreates a single suite with extra YAML.

Close

Green advisory tests are a transcript of what the agent believes. Blocking properties on sealed fixtures are the only merge signal that transcript cannot rewrite. Split the jobs. Freeze the bytes. Leave coverage in the summary, not in the gate.