Extract the CI Matrix Before a Model Reviews Your OSS Patch

A first-time contributor cloned a popular Python library on a Friday night and reproduced the failing unit test in minutes. The local virtualenv used Python 3.12, the newest type-checker extra, and a freshly generated lockfile from the default branch. The patch looked clean, the test passed twice, and an AI review of the diff called the change ready for maintainers. The pull request then failed on the oldest GitHub Actions matrix cell, where Python 3.9 rejected a match statement.

The model never lied about the diff. It simply never saw the runtime the project actually ships. Agentic coding tools still tend to assume one interpreter, one OS image, and one install command unless those facts are extracted first. The useful workflow is therefore mechanical: pin the CI matrix, replay the cheapest failing cell locally, and only then ask a free model to review the patch against that ledger.

The local green check is not the project runtime

Most open-source Python, Node, and Go repositories encode their real contract in .github/workflows, not in README badges. Matrix jobs pin language versions, operating systems, extra extras, and env vars that a laptop virtualenv will never recreate. A model that reviews only git diff will praise syntax the oldest cell cannot parse, or skip an install extra the lint job requires.

Maintainers bounce these pull requests for a boring reason. The contributor optimized for a personal toolchain, while the workflow file still describes four older toolchains and a lint job with a different working-directory. Free models help after that contract is written down. They do not replace the extraction step.

Artifact: an assumption ledger plus a one-cell replay

The portable artifact is a short markdown ledger checked in beside the patch branch, plus a replay script that runs a single matrix cell. The ledger is the review context. The script is the evidence. Neither file should contain tokens, deploy keys, or pull_request_target secrets from the upstream project.

Create ASSUMPTION_LEDGER.md with four blocks before any model sees the diff:

• Workflow files: path, trigger (pull_request, push), and job names that run on forks.
• Matrix cells: language versions, OS images, and fail-fast behavior.
• Install truth: the exact pip, npm, or go commands the job uses, including extras.
• First cell to replay: the oldest language version on ubuntu-latest, because that cell rejects new syntax first.

A filled ledger looks like the following example. The versions are illustrative and must be copied from the cloned workflow, not remembered from last month.

# ASSUMPTION_LEDGER.md
- repo: example/httpx-utils (local clone, default branch main)
- workflow: .github/workflows/ci.yml
- on: pull_request
- jobs: test, lint
- matrix.python-version: ["3.9", "3.10", "3.11", "3.12"]
- matrix.os: [ubuntu-latest]
- fail-fast: false
- test install: pip install -e ".[test]"
- lint install: pip install -e ".[lint]"
- test command: pytest -q
- lint command: ruff check src tests
- first replay cell: python 3.9 / ubuntu / test job
- local evidence: ./scripts/replay_matrix_cell.sh 3.9

Step 1: copy the workflow, then extract the matrix

Clone the upstream repository with a named remote so later git fetch does not rewrite the ledger by accident. Stay on a topic branch that contains only the bugfix commits intended for the pull request.

git clone https://github.com/example/httpx-utils.git
cd httpx-utils
git checkout -b fix/timeout-on-empty-body
ls -l .github/workflows

The extractor below is a labeled helper, not a complete GitHub Actions engine. It reads a simplified strategy.matrix mapping and prints language versions. Reusable workflows, YAML anchors, and matrix.include rows still need a human pass.

# scripts/extract_ci_matrix.py
# Proposal / helper: handles a flat matrix mapping only.
from pathlib import Path
import sys

try:
    import yaml
except ImportError:
    sys.stderr.write("pip install pyyaml before running this helper\n")
    sys.exit(2)


def load_workflow(path: Path) -> dict:
    data = yaml.safe_load(path.read_text(encoding="utf-8"))
    if not isinstance(data, dict):
        raise ValueError(f"{path} is not a mapping")
    return data


def iter_matrix_jobs(workflow: dict):
    jobs = workflow.get("jobs") or {}
    for name, job in jobs.items():
        if not isinstance(job, dict):
            continue
        strategy = job.get("strategy") or {}
        matrix = strategy.get("matrix") or {}
        if isinstance(matrix, dict):
            yield name, matrix, bool(strategy.get("fail-fast", True))


def main() -> None:
    path = Path(sys.argv[1] if len(sys.argv) > 1 else ".github/workflows/ci.yml")
    workflow = load_workflow(path)
    print(f"workflow\t{path}")
    print(f"on\t{workflow.get('on')}")
    for job_name, matrix, fail_fast in iter_matrix_jobs(workflow):
        print(f"job\t{job_name}\tfail-fast={fail_fast}")
        for key, values in matrix.items():
            print(f"matrix.{key}\t{values}")


if __name__ == "__main__":
    main()

Run it against every workflow that triggers on pull_request. Record jobs that install extras the test job does not install, because lint and type-check cells reject patches that tests accept.

python scripts/extract_ci_matrix.py .github/workflows/ci.yml
python scripts/extract_ci_matrix.py .github/workflows/lint.yml

Step 2: replay the oldest cell before touching the model

The replay script should use the same install extra and the same test command as the job, not the contributor's preferred tools. Python 3.9 is the usual first cell for libraries that still advertise it. Replace the version with the oldest value printed by the extractor.

# scripts/replay_matrix_cell.sh
set -euo pipefail
PY_VER="${1:-3.9}"
VENV=".venv-ci-${PY_VER}"

python"${PY_VER}" -m venv "${VENV}"
# shellcheck disable=SC1090
source "${VENV}/bin/activate"
python -m pip install -U pip
python -m pip install -e ".[test]"
pytest -q

chmod +x scripts/replay_matrix_cell.sh
./scripts/replay_matrix_cell.sh 3.9

If the project ships a container image or a devcontainer.json, prefer that image for the replay when the laptop cannot install the old interpreter. The point is not perfect GitHub parity. The point is to fail on the same syntax and dependency surface the matrix will use.

Capture the command, exit code, and first failing assertion in the ledger. That block becomes the only runtime the later model is allowed to assume.

replay: ./scripts/replay_matrix_cell.sh 3.9
exit: 1
first failure: tests/test_timeout.py::test_empty_body SyntaxError: invalid syntax
constraint: no match statements, no tomllib, no 3.10+ type unions in src/

Step 3: patch against the ledger, not against the laptop

Write the failing test in the oldest syntax the ledger allows, then implement the fix in the same subset. After the edit, run the oldest cell again, then one newer cell if the matrix is wide. Do not promote a 3.12-only helper into src/ because the local editor auto-imported it.

# tests/test_timeout.py
import pytest

from httpx_utils import fetch_with_timeout


def test_empty_body_does_not_raise_on_204(monkeypatch):
    class DummyResponse(object):
        status_code = 204
        content = b""

        def json(self):
            raise ValueError("no body")

    def fake_fetch(url, timeout):
        return DummyResponse()

    monkeypatch.setattr("httpx_utils.raw_fetch", fake_fetch)
    payload = fetch_with_timeout("https://example.test/empty", timeout=1)
    assert payload is None

Keep public helpers on syntax the oldest cell accepts. The following shape is verbose on purpose, because brevity is how 3.12-only syntax sneaks into a library still claiming 3.9.

# src/httpx_utils.py
def fetch_with_timeout(url, timeout):
    response = raw_fetch(url, timeout=timeout)
    if response.status_code == 204:
        return None
    if not response.content:
        return None
    return response.json()

Re-run the replay script after the patch. If the oldest cell is green and ruff still fails, that is a second job in the ledger, not a reason to skip lint locally.

./scripts/replay_matrix_cell.sh 3.9
python3.9 -m pip install -e ".[lint]"
python3.9 -m ruff check src tests

Step 4: send the ledger and the diff, never the whole clone

Only after the oldest cell passes should a model review the change. The prompt below is a proposal, not an executed production evaluation. Paste the ledger, the workflow excerpt, and git diff main...HEAD. Omit secrets, private runner labels, and upstream issue reports that are not already public.

You are reviewing an OSS patch against a CI assumption ledger.
Treat ASSUMPTION_LEDGER.md as the only allowed runtime.
Reject syntax, stdlib modules, and extras not installed in the first replay cell.
Ignore style opinions that ruff and the lint job do not enforce.
Return: (1) matrix risks, (2) missing tests, (3) commands still unrun.
Do not assume Python 3.12, macOS, or network access during pytest.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. A contributor who needs a disposable review workspace can load that ledger and the diff into MonkeyCode using its free model access on the free server option, then keep the replay script on the laptop as the source of truth. The model is there to read constraints the diff does not contain. It is not there to invent a toolchain.

Decision table for the next command

Use the table after each replay. The action is a command or a ledger edit, not another prompt.

• SyntaxError on the oldest cell: rewrite the patch; do not ask a model for a 3.12-only rewrite.
• ImportError for a test extra: copy the workflow pip install line into the replay script, then rerun.
• Tests green, lint red: run the lint job install extra; treat it as a second matrix cell.
• Tests green on 3.9 and 3.12, type-checker red: add the type-check command to the ledger before opening the PR.
• Fail-fast true, first cell red: stop; later cells will not run on GitHub either.
• Reusable workflow at uses:: extraction helper is incomplete; open the called workflow file by hand.

git diff --stat main...HEAD
git diff main...HEAD -- src tests .github/workflows

Those two diffs, plus the ledger, are the entire model context. Dumping site-packages or the full clone usually hides the matrix fact that actually failed.

Limitations, and who should skip this loop

The extractor does not evaluate GitHub expressions, service containers, or workflow_call inputs. Self-hosted runners, Windows MSVC jobs, and hardware-specific extras will not replay on a laptop with a bash script. Cached pip wheels can hide a missing extra that a clean CI image will still miss, so delete .venv-ci-* when the install line changes.

This loop is the wrong tool for embargoed security issues, patches that require production credentials, and drive-by refactors of files the issue never named. It is also the wrong tool when the contributor has not run the project's documented test command even once. Free models do not make an unreproducible change reviewable. They only scale a review after the oldest matrix cell has already spoken.

The Friday-night pull request failed because the runtime lived in YAML, and the review lived in a 3.12 diff. Write the ledger, replay one cell, then let a model read those constraints. The green check that matters is still the one GitHub will run on the fork.