Split CI Into a Merge Lane and a Soak Lane

A single green check is a weak merge signal. Soak jobs, retries, and agent-drafted required steps should not share it. Split CI into two lanes. Put fixtures in a merge lane with zero retries. Park flakes and long soaks in an advisory lane. An agent may draft jobs. It does not rewrite the lane map.

That split is the whole contract. Everything below is how you enforce it.

What a mixed green check hides

You already know the failure mode. A pull request edits application code, a workflow file, and a test that used to flake. CI retries the flake. A soak job fails and is marked optional. The required job is green because the retry landed on a quiet runner. The merge button lights up.

None of that proved the SHA. It proved that noise and signal shared a status check. Agent-authored diffs make the mix worse. They add jobs quickly. They copy retry blocks from the last workflow that "worked." They promote a soak job to required because the prompt said "make CI pass."

You do not fix that with a longer test matrix. You fix it by making lane membership a reviewed file, not a side effect of YAML edits.

The two-lane contract

Keep two lanes, and only two.

1. Merge lane. Deterministic fixtures. No retries. Fail closed. A red job here blocks merge.
2. Soak lane. Load, timing, known flakes, long suites. Failures are visible. They do not vote on merge.

Unmapped jobs fail closed. If a workflow job is not in the lane map, treat it as merge-lane and fail the PR that introduced it. Promotion is a human edit to the map. Drafting a job is not promotion.

Label this as a proposed contract, not a claim about your current org. You can run it on a throwaway repo today.

Step 1: Write a lane map the workflows cannot ignore

Create ci/lanes.yml. Keep it boring. Boring files get reviewed.

# ci/lanes.yml
version: 1
merge:
  - unit-fixtures
  - contract-diff
  - lane-guard
soak:
  - nightly-load
  - flake-watch
rules:
  merge_retries: 0
  unmapped: fail-closed
  agent_may_edit_lanes: false

Job names in this file must match jobs.<id> in your workflows. If they drift, the guard fails. That is the point.

Step 2: Forbid retries on the merge lane

Retries belong to soak. Merge-lane retries convert flakes into greens. You want the opposite: a flake in the merge lane is a fixture bug or a product bug. Fix the fixture. Do not roll the dice.

# .github/workflows/merge-lane.yml
name: merge-lane
on:
  pull_request:
  push:
    branches: [main]

jobs:
  lane-guard:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: python ci/check_lanes.py --strict

  unit-fixtures:
    needs: lane-guard
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: python -m pytest tests/fixtures -q

  contract-diff:
    needs: lane-guard
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - run: python ci/check_lanes.py --diff origin/${{ github.base_ref || 'main' }}

Soak stays separate. continue-on-error: true is allowed there. It is not allowed in merge-lane YAML. The guard parses both files.

# .github/workflows/soak-lane.yml
name: soak-lane
on:
  pull_request:
  schedule:
    - cron: "17 3 * * *"

jobs:
  flake-watch:
    runs-on: ubuntu-latest
    continue-on-error: true
    steps:
      - uses: actions/checkout@v4
      - run: python -m pytest tests/soak -q --maxfail=1

Step 3: Put the rules in a script you can run locally

CI should not be the first place the contract runs. Run the same file on the laptop and in the workflow. The script below is a complete, labeled example. It is not a production security scanner. It is a lane contract.

# ci/check_lanes.py
from __future__ import annotations

import argparse, subprocess, sys
from pathlib import Path

try:
    import yaml
except ImportError:
    print("pip install pyyaml", file=sys.stderr)
    raise

ROOT = Path(__file__).resolve().parents[1]
LANES = ROOT / "ci" / "lanes.yml"
WORKFLOWS = ROOT / ".github" / "workflows"
AGENT_MARKERS = ("Generated-By:", "Co-authored-by: agent", "Agent-Commit: true")


def load_lanes():
    data = yaml.safe_load(LANES.read_text())
    merge = set(data.get("merge") or [])
    soak = set(data.get("soak") or [])
    overlap = merge & soak
    if overlap:
        raise SystemExit(f"job in both lanes: {sorted(overlap)}")
    return data, merge, soak


def workflow_jobs():
    jobs = {}
    for path in sorted(WORKFLOWS.glob("*.yml")):
        doc = yaml.safe_load(path.read_text()) or {}
        for name, body in (doc.get("jobs") or {}).items():
            jobs[name] = {"file": path.name, "body": body or {}}
    return jobs


def changed_files(base: str) -> set[str]:
    out = subprocess.check_output(
        ["git", "diff", "--name-only", f"{base}...HEAD"],
        text=True,
    )
    return {line.strip() for line in out.splitlines() if line.strip()}


def commit_message() -> str:
    return subprocess.check_output(["git", "log", "-1", "--pretty=%B"], text=True)


def has_retry(body: dict) -> bool:
    text = yaml.safe_dump(body)
    return ("max-attempts" in text) or ("retry" in text and "retry-on" in text)


def main() -> int:
    parser = argparse.ArgumentParser()
    parser.add_argument("--strict", action="store_true")
    parser.add_argument("--diff", default="")
    args = parser.parse_args()

    data, merge, soak = load_lanes()
    jobs = workflow_jobs()
    errors = []

    mapped = merge | soak
    extra = set(jobs) - mapped
    missing = mapped - set(jobs)
    if extra:
        errors.append(f"unmapped jobs fail closed: {sorted(extra)}")
    if missing:
        errors.append(f"lane map names missing jobs: {sorted(missing)}")

    for name in merge:
        body = (jobs.get(name) or {}).get("body") or {}
        if body.get("continue-on-error") is True:
            errors.append(f"merge job {name} cannot continue-on-error")
        if has_retry(body):
            errors.append(f"merge job {name} cannot retry")
        if int(data.get("rules", {}).get("merge_retries", 0)) != 0:
            errors.append("rules.merge_retries must be 0")

    if args.diff:
        changed = changed_files(args.diff)
        lanes_touched = "ci/lanes.yml" in changed
        msg = commit_message()
        agent = any(m.lower() in msg.lower() for m in AGENT_MARKERS)
        if lanes_touched and agent and not data["rules"].get("agent_may_edit_lanes"):
            errors.append("agent commit cannot edit ci/lanes.yml")

    if errors:
        print("lane contract failed:")
        for item in errors:
            print(f"  - {item}")
        return 1
    print(f"lane contract ok: {len(merge)} merge, {len(soak)} soak")
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

Install PyYAML in the job and locally. Then run the same command in both places.

pip install pyyaml
python ci/check_lanes.py --strict
python ci/check_lanes.py --diff origin/main

Step 4: Add a hook so the laptop fails before the PR exists

A hook is cheaper than a round trip through GitHub. Keep it thin. Call the same script.

# .git/hooks/pre-push  (or a tracked script that installs it)
#!/bin/sh
set -e
python ci/check_lanes.py --strict
python ci/check_lanes.py --diff origin/main

chmod +x .git/hooks/pre-push

If your team hates client hooks, run lane-guard as a required check and refuse to merge without it. The hook is for speed. The required check is for truth.

Step 5: Classify every new job before you write assertions

When a PR wants a new job, fill this table in the description. If the row is blank, the lane guard should already fail. The table is for humans. The YAML is for the script.

Job id	Lane	Retry allowed	Blocks merge	Who may edit the lane row
unit-fixtures	merge	no	yes	human
contract-diff	merge	no	yes	human
lane-guard	merge	no	yes	human
flake-watch	soak	yes	no	human
nightly-load	soak	yes	no	human

Promotion is a row change in ci/lanes.yml, not a comment in the PR. If the agent drafted a soak job that looks stable, a person moves the name from soak to merge after the fixture is deterministic. Not before.

Where a free coding agent belongs

Drafting fixtures is useful. Promoting jobs is not. Keep the agent in the directories that feed the merge lane: tests/fixtures, golden files, and small contract tests. Keep it out of ci/lanes.yml.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode is an open-source coding assistant with free model access and a free server option. That pairing matters here for one reason. You can generate fixture drafts on the free server, pull them into a worktree, and run python ci/check_lanes.py --strict before GitHub ever sees the branch. The server does not become CI. CI still rebuilds the merge lane. The agent still does not vote.

Point the agent at a failing fixture, not at the workflow that marks the job required. If it rewrites .github/workflows and ci/lanes.yml in one commit, the guard should fail. That is a successful contract, not a rude model.

If you already have a local agent session, use it to propose soak tests. Leave lane promotion for review. One place to start is the project's own repo when you want that draft loop without standing up extra hardware.

A 20-minute local drill

Do this on a throwaway clone. Do not run it against production secrets.

1. Copy ci/lanes.yml, ci/check_lanes.py, and the two workflow files into an empty repo.
2. Add a passing tests/fixtures/test_ok.py and a soak test that fails half the time.
3. Run python ci/check_lanes.py --strict. It should print lane contract ok.
4. Add retry YAML to unit-fixtures. Run the script again. It should fail.
5. Commit with a body line Generated-By: demo-agent and edit ci/lanes.yml to move flake-watch into merge. Run --diff origin/main. It should fail.
6. Revert the lane edit. Keep the soak failure. Open a PR. Merge lane stays green. Soak stays visible and non-blocking.

If step 4 or 5 stays green, the contract is not installed. Fix the script path before you trust the badge.

Limitations, and who should skip this

This does not detect every retry mechanism. GitHub Actions has more retry styles than one string scan. Extend the parser if you use a reusable workflow or a third-party retry action. Until you do, assume the guard is incomplete.

It also does not prove tests are deterministic. Zero retries only expose flakes. You still need fixtures that do not sleep on the wall clock. Soak-lane continue-on-error can hide a broken product if nobody reads the advisory job. Name an owner for soak failures or they become wallpaper.

Skip this approach if you ship a single workflow with one job, or if every suite is already short and deterministic. Skip it if your merge queue requires one check name and you cannot split required versus advisory. Do not use it as a substitute for secret scanning or workflow review. Lane maps do not make untrusted YAML safe.

The merge button should mean the merge lane passed once, on this SHA, with no retries. Anything else is a soak result. Keep the agent in the fixture directory. Keep the lane file in human hands.