A Green Pipeline That Ran Three of Five Shards Is Not Green

Your merge queue trusts a color. It should trust a coverage claim.

A sharded CI run where shard 3 died during setup, shard 5 collected zero tests, and shard 1 passed on its third attempt produces exactly the same green check as a clean sweep of the whole matrix. Nothing in the check name tells you which one you got. The fix is not a smarter retry policy — it is a per-shard manifest plus a gate that refuses to merge anything the manifest cannot prove.

Four ways a shard quietly disappears

1. The matrix shrinks. --splits is computed from a test count that changed between the plan job and the test job. Fewer jobs get scheduled, needs is satisfied, the workflow is green.
2. A partition is empty. Shard 4 is assigned no tests. pytest exits with code 5, and someone already appended || true to keep the pipeline moving.
3. A step is allowed to fail. continue-on-error: true left over from a migration week means a crashed shard reports success at the job level.
4. A retry hides the first attempt. pytest-rerunfailures turns a flake into a pass, and the report contains no field that says "this took three tries."

All four are coverage bugs, not flake bugs. Treat them that way.

The contract: one manifest per shard

Stop reading the job summary. Have every shard write a small JSON file next to its report, and let the gate read only those files.

The manifest needs six fields and nothing more: index, commit, collected, node_ids, status, attempts. Add quarantined if you run a quarantine lane. If a field is missing, the gate blocks — absence is not consent.

Step 1 — Discover once, split deterministically

Collect node IDs in a single job, persist them, and split from that file. Never recompute the split inside the test job.

python -m pytest --collect-only -q | grep '::' | sort > nodes.txt
wc -l < nodes.txt   # the number every later step must agree with

jobs:
  plan:
    runs-on: ubuntu-latest
    outputs:
      shards: ${{ steps.plan.outputs.shards }}
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: '3.12' }
      - id: plan
        run: |
          python -m pytest --collect-only -q | grep '::' | sort > nodes.txt
          n=$(wc -l < nodes.txt)
          echo 'shards='$(( (n + 199) / 200 )) >> "$GITHUB_OUTPUT"
      - uses: actions/upload-artifact@v4
        with: { name: nodes, path: nodes.txt }

Step 2 — Emit the manifest from a hook

Do not parse the terminal output. Use pytest_sessionfinish and derive everything from session.items.

# conftest.py
import json, os, pathlib

def pytest_sessionfinish(session, exitstatus):
    out = pathlib.Path(os.environ.get('SHARD_DIR', 'shard-reports'))
    out.mkdir(parents=True, exist_ok=True)
    nodes = sorted({item.nodeid for item in session.items})
    payload = {
        'index': int(os.environ['SHARD_INDEX']),
        'commit': os.environ['GITHUB_SHA'],
        'collected': len(nodes),
        'node_ids': nodes,
        'status': 'passed' if exitstatus == 0 else 'failed',
        'attempts': int(os.environ.get('SHARD_ATTEMPT', '1')),
        'quarantined': os.environ.get('QUARANTINE') == '1',
    }
    (out / f'shard-{payload["index"]}.json').write_text(json.dumps(payload))

Expose SHARD_INDEX from the matrix, not from a counter inside the process. Counter drift is how two shards end up writing shard-0.json and the gate sees four files where five were expected.

Step 3 — Gate on the manifest, not the color

Stdlib only, no dependencies, no network. It exits non-zero on the first violation it can explain.

# gate_shards.py
import json, os, pathlib, sys

head = os.environ['GITHUB_SHA']
expected = int(os.environ['SHARD_COUNT'])
files = sorted(pathlib.Path(sys.argv[1]).glob('shard-*.json'))

def block(msg):
    print(f'::error::{msg}')
    sys.exit(1)

if len(files) != expected:
    block(f'found {len(files)} manifests, expected {expected}')

indexes, owner, retried = set(), {}, []
for path in files:
    m = json.loads(path.read_text())
    commit, index, status = m['commit'], m['index'], m['status']
    if commit != head:
        block(f'{path.name} tested {commit[:8]}, head is {head[:8]}')
    if index in indexes:
        block(f'shard {index} reported twice')
    indexes.add(index)
    if status != 'passed':
        block(f'shard {index} finished as {status}')
    if m['collected'] == 0:
        block(f'shard {index} collected zero tests')
    if m['attempts'] > 1 and not m['quarantined']:
        retried.append(index)
    for node in m['node_ids']:
        if node in owner:
            block(f'{node} ran in shards {owner[node]} and {index}')
        owner[node] = index

missing = sorted(set(range(expected)) - indexes)
if missing:
    block(f'no result for shard(s) {missing}')
if retried:
    block(f'green only after a retry in shard(s) {retried}')
print(f'OK: {len(owner)} node ids across {expected} shards on {head[:8]}')

Note what the duplicate check buys you: a split bug that sends the same 40 tests to two shards now fails the gate instead of inflating your pass count.

Step 4 — Wire it as its own required check

  gate:
    needs: [plan, test]
    if: always()
    runs-on: ubuntu-latest
    steps:
      - uses: actions/download-artifact@v4
        with:
          pattern: shard-*
          path: shard-reports
          merge-multiple: true
      - env:
          SHARD_COUNT: ${{ needs.plan.outputs.shards }}
          GITHUB_SHA: ${{ github.event.pull_request.head.sha }}
        run: python3 gate_shards.py shard-reports

Two details earn their keep. if: always() stops the gate from being skipped when test fails, and a skipped gate reads like a passed one in most UIs. Overriding GITHUB_SHA with the PR head SHA means the manifest is compared against the code you reviewed, not the merge commit.

Step 5 — Make the retry ledger boring

The attempts > 1 rule is the flake policy, and it is deliberately blunt: a test that only goes green on the second attempt blocks the merge until someone either fixes it or moves it to the quarantine lane with an owner and an expiry date. Rerunning until green is not a decision, it is a coin flip with a paper trail.

Track the top five quarantined files weekly. If the list does not shrink, your quarantine is where tests go to be forgotten.

Where a free model actually fits

Not in the gate. The gate must stay deterministic, offline, and explainable in one line of output.

Triage is a different job: turning twenty scattered failure snippets into three probable root causes is exactly the kind of summarization that saves a reviewer ten minutes. MonkeyCode is an open-source project that offers free model access and a free server option, which makes it a low-friction place to run that advisory step before you decide to pay for anything. The project advertises a 10M-token allowance and a free server tier — treat both as vendor claims and check the current numbers in its console before you build a dependency on them.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

# triage.py — advisory only; never writes status
import json, os, urllib.request, pathlib

fails = []
for path in pathlib.Path('shard-reports').glob('shard-*.json'):
    fails.extend(json.loads(path.read_text()).get('failures', []))
if not fails:
    raise SystemExit('nothing to triage')

prompt = 'Group by shared root cause. Markdown, max 200 words.\n\n' + '\n'.join(fails[:20])
body = json.dumps({'messages': [{'role': 'user', 'content': prompt}], 'temperature': 0}).encode()
req = urllib.request.Request(
    os.environ['MONKEYCODE_BASE_URL'] + '/chat/completions',
    data=body,
    headers={'Authorization': 'Bearer ' + os.environ['MONKEYCODE_API_KEY'],
             'Content-Type': 'application/json'},
)
print(urllib.request.urlopen(req, timeout=60).read().decode())

Read the base URL and model ID from the project's own docs; endpoints move, and a hardcoded one in a tutorial rots the week after publication. Scrub secrets and customer identifiers from fails before it leaves the runner — failure logs are the most common accidental leak in this pattern.

Hard rule: triage output lands in a PR comment. It never sets status, never flips quarantined, never authorizes a merge.

Limits, and who should skip this

• Small suites. Under roughly 200 tests, sharding overhead exceeds the wall-clock win, and one manifest beats five.
• Non-hermetic runners. If your runner image digest can change mid-run, a passed shard is not reproducible and the manifest proves less than you think.
• Regulated audit trails. Don't put a compliance gate behind a free tier whose quota and retention you have not verified in writing.
• Anyone tempted to parse model prose. If you cannot keep the machine summary out of the decision path, don't add the summary.

The checker above is stdlib Python 3.12 and untested against your report format — adapt the status mapping to your runner before you make the gate required. Once it is required, the interesting metric stops being pass rate and becomes coverage: how many node IDs, from how many shards, on which commit.

If you want to try the triage step before wiring a billing account, the project's free model access and free server option are the shortest path to finding out whether it earns a slot in your pipeline.