Do Not Let the Required Job Write the Fixture It Trusts

The required job should fail if it can still write a fixture. A green check from a job that shares that file with a draft run is not a merge signal. You want the merge job read-only on owned fixtures, and you want every model write confined to a scratch tree. Branch protection waits on the read-only job.

You do not need a headline to justify the split. If an agent can open a pull request, the job that unlocks merge needs a permission boundary. Color on the draft job is optional noise.

What a retry is hiding

You see this on a persistent runner, or in a test that repairs a missing JSON file. The draft rewrites fixtures/owned/paid.json until the new assertion matches. Pytest goes green. The next pull request still has the old bytes, so it fails in a way that looks like order.

Someone adds a retry. The retry passes because a later test put the file back. You did not fix a flake. You hid a shared writable fixture behind a second attempt.

Do not spend retries on that failure. Remove the write permission, and the second run stops being the control. The first run was never allowed to edit the file it claimed to trust.

Two trees

Keep trusted bytes under fixtures/owned/. Keep agent output under scratch/agent/. The merge job may read the first tree and must not write it. The draft job may write only the second tree.

Tests you intend to merge live in tests/merge/ and reference owned paths only. This is a directory contract plus a permission bit. You can rehearse it in a temp folder. Nothing in the gate calls a model.

Step 1: Abort the merge session if a fixture is writable

Drop this file beside the merge tests. It is a proposal until you run it. On pytest_sessionstart it refuses a missing catalog, an empty catalog, a writable path, or a scratch reference in a test module.

# tests/merge/conftest.py
import os
from pathlib import Path

OWNED = Path("fixtures/owned")

def pytest_sessionstart(session):
    if not OWNED.is_dir():
        raise SystemExit("missing fixtures/owned")
    files = [path for path in OWNED.rglob("*") if path.is_file()]
    if not files:
        raise SystemExit("fixtures/owned has no files")
    writable = [str(path) for path in files if os.access(path, os.W_OK)]
    if os.access(OWNED, os.W_OK):
        writable.append(str(OWNED))
    if writable:
        raise SystemExit("owned fixtures are writable:\n" + "\n".join(writable))
    leaked = []
    for test_file in Path("tests/merge").rglob("test_*.py"):
        text = test_file.read_text(encoding="utf-8")
        if "scratch/agent" in text or "scratch.agent" in text:
            leaked.append(str(test_file))
    if leaked:
        raise SystemExit("merge suite references scratch:\n" + "\n".join(leaked))

Scan test_*.py only. conftest.py has to name the forbidden path in order to reject it, so including that file makes the probe fail itself. If the probe can write, the suite stops. You do not get a skip, and you do not get a green collection.

Step 2: Clear the write bit in the required job

Git will not store this mode reliably. The executable bit is the permission Git tends to keep. The read-only bit is the job's responsibility, every run. Apply it after checkout and before pytest.

#!/bin/sh
# scripts/merge_suite.sh
set -eu
test -d fixtures/owned
find fixtures/owned -type f -exec chmod a-w {} +
find fixtures/owned -type d -exec chmod a-w {} +
find fixtures/owned -type d -exec chmod a+x {} +
python3 -m pytest tests/merge -q --maxfail=1

Do not point find at scratch/agent/. The draft job needs that tree writable. One combined root, and the split is gone.

A root checkout can change the bit back and then pass a probe you ran too early. Drop privileges before the probe, or mount fixtures/owned read-only. If you can do neither, do not mark the job required. A green root shell is not a lock.

Step 3: Leave the draft job off the required list

Sketch the two jobs in the CI system you already run. Pin actions to SHAs your org audits. The image name below is a placeholder, not a version claim.

jobs:
  agent-draft:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@<pinned-sha>
      - run: |
          mkdir -p scratch/agent
          python3 scripts/draft_test.py --out scratch/agent/proposed_test.py
  merge-gate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@<pinned-sha>
      - run: sh scripts/merge_suite.sh

Require merge-gate only. A red draft must not block merge. A green draft must not satisfy it. Put that sentence in the pull request template so the extra bubble stops looking like evidence.

draft_test.py can be a stub that writes a proposal under scratch. The merge suite does not import it. If you want the proposal kept, move a reviewed copy into tests/merge/ and add the fixture under fixtures/owned/ in a commit that does not also weaken the probe.

Step 4: Catch the scratch import before the pull request

Match the probe locally with a pre-push hook. It does not compute coverage, and it does not retry a failed job. It only blocks one path leak.

#!/bin/sh
set -eu
if git grep -n -E 'scratch/agent|scratch\.agent' -- tests/merge ':!tests/merge/conftest.py'; then
  echo "tests/merge must not reference scratch/agent"
  exit 1
fi

Use the hook only if the repo already runs one. A hook nobody installs is a note. The session probe remains the gate that branch protection can see.

Step 5: Rehearse a pass, a write failure, and a leak

Run this in a temp directory on a POSIX shell. Copy the conftest.py from step 1 into tests/merge/ first. You want three different endings. If the writable case still exits 0, the probe is not on pytest_sessionstart.

mkdir -p fixtures/owned scratch/agent tests/merge
printf '%s\n' '{"id":"paid-1","status":"paid"}' > fixtures/owned/paid.json
printf '%s\n' \
  'def test_paid_fixture_is_readable():' \
  '    text = open("fixtures/owned/paid.json", encoding="utf-8").read()' \
  '    assert text.startswith("{")' \
  > tests/merge/test_paid.py
chmod a-w fixtures/owned/paid.json
python3 -m pytest tests/merge -q

That run should pass if the file is read-only and the test only reads. Then force the failure you care about.

chmod u+w fixtures/owned/paid.json
python3 -m pytest tests/merge -q

Expect a non-zero exit and the line owned fixtures are writable. Restore the bit, then plant a leak.

chmod a-w fixtures/owned/paid.json
printf '%s\n' 'PATH = "scratch/agent/proposed_test.py"' > tests/merge/test_leak.py
python3 -m pytest tests/merge -q

The third run must name test_leak.py and stop. Delete that file when you are done. Do not commit a mode bit and call it the control. Commit the probe, and let CI apply chmod.

Where a model may run

Only agent-draft may call a model. The probe, the mode bits, and the merge pytest run stay offline.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

If your account includes MonkeyCode's free model access and free server option, bind both to the draft job and to nothing required. Check the current terms in the account before you depend on them. This article does not state a quota, a model name, a machine size, or a duration. When that free server is down, skip the draft job. Still run merge-gate.

A required check that needs a live model will go red for reasons outside the diff. Keep that call off the merge path. The draft job is the bench. It is not the signal.

How to read a noisy pull request

What you see	Merge?	Why
Draft green, merge job missing	No	The green job was allowed to write
Merge green, probe not started	No	You never proved the write bit was off
Probe lists a writable owned file	No	The suite could edit the answer
Merge test mentions scratch/agent	No	The two jobs share a tree again
Probe clean and merge pytest green	Only after review	The required job could not write fixtures

Review stays in the last row on purpose. The probe understands permission and path. It does not understand whether the JSON is the right scenario.

Limitations

os.access follows the current user. If that user is root, a cleared write bit is advisory, because root can set the bit again. The probe is a tripwire for normal CI users, not a sandbox against a hostile root step in the same job.

The scratch check is a literal substring. A test that concatenates path fragments will pass the scan. Review generated tests. Do not treat git grep as a type checker.

A read-only file can still be the wrong file. You can lock a bad payload and merge a matching bad assertion. Read fixture diffs. The gate will not do that for you.

POSIX chmod is part of the contract. On Windows runners, os.access and a-w do not mean the same thing. Use an ACL you have actually tested, or do not copy this script into that job and call it done.

Persistent self-hosted workspaces will fail the probe when a previous job left files writable. Clean the workspace, or stop using a dirty checkout for merge-gate. That failure is the point.

Who should not use this

Skip the split if the merge suite must regenerate fixtures and you have not extracted that step. People will delete the probe to get green, which is worse than no probe.

Skip it on a spike that cannot reach the default branch. The scratch rule is overhead until merge is real.

Skip any hosted model if policy forbids sending repo contents to a third party. A drafting bench does not rewrite that rule. Run the draft locally, or do not run it.

Do not treat a drafting bench as an availability plan, and do not treat the probe as a review replacement. One locks a path. The other can be skipped.

Close

Require the job that cannot write fixtures/owned/. Keep proposals in scratch/agent/. Ignore a green draft. You still read the assertion. You stop calling a shared writable file a passing test.