GitHub Actions Matrix Jobs Need a Failure Map

Matrix jobs are great right up untill four variants fail at once and the whole workflow turns into log roulette. You know the kind: node18-linux failed in one way, node20-macos failed in another, and the only shared output is a red X.

What helped me most was adding a failure map step after the matrix run. Instead of asking every engineer to open each job and compare logs by hand, the workflow collects one small JSON result per variant and prints a single summary table in the parent job. It is not fancy, but it is fast, and fast wins.

Why matrix failures waste time

Most matrix workflows are very good at parallelism and pretty bad at storytelling.

Each variant knows what broke in its own enviroment, but the workflow rarely answers the bigger questions:

• did all failures come from the same API contract drift?
• is one operating system the outlier?
• did only the slowest variants fail after a timeout?
• was the break introduced before the test even reached the interesting step?

GitHub Actions already gives us the building blocks for this pattern: matrix strategy, artifacts, job outputs, and step summaries (https://docs.github.com/en/actions/using-jobs/using-a-matrix-for-your-jobs, https://docs.github.com/en/actions/using-workflows/workflow-commands-for-github-actions). The missing piece is deciding that every matrix leg should leave behind one compact, machine-readable verdict.

That is very similar to the discipline behind release-ready email checks. The point is not more logs. The point is better evidence.

The failure map pattern

The shape is simple:

1. Each matrix job writes result.json.
2. A follow-up job downloads all artifacts.
3. A small script merges them into failure-map.json.
4. The workflow writes a human summary to $GITHUB_STEP_SUMMARY.

I keep each result file tiny on purpose:

{
  "variant": "node20-ubuntu",
  "status": "failed",
  "phase": "contract-check",
  "http_status": 422,
  "duration_ms": 1684,
  "reason": "schema mismatch on billing response",
  "artifact": "result-node20-ubuntu"
}

That one record is enough to group failures by phase, sort them by duration, and spot whether the problem is broad or isolated. If every failing job points to contract-check, you probably have one regression. If failures split between setup, seed, and contract-check, you likely have workflow debt.

I also like this because it makes review calmer. A teammate can scan one summary, open one artifact, and move on. No one has to remember which run had the weird temp mailid branch name in a copied command or which retry log was the one you actualy meant.

A GitHub Actions workflow shape

Here is the basic pattern I reuse:

jobs:
  api-matrix:
    runs-on: ${{ matrix.os }}
    strategy:
      fail-fast: false
      matrix:
        os: [ubuntu-latest, macos-latest]
        node: [18, 20]
    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-node@v4
        with:
          node-version: ${{ matrix.node }}

      - name: Run API checks
        run: |
          mkdir -p out
          node scripts/run-checks.mjs \
            --os "${{ matrix.os }}" \
            --node "${{ matrix.node }}" \
            --out out/result.json

      - name: Upload result
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: result-${{ matrix.node }}-${{ matrix.os }}
          path: out/result.json

  summarize:
    needs: api-matrix
    if: always()
    runs-on: ubuntu-latest
    steps:
      - uses: actions/download-artifact@v4
        with:
          path: artifacts

      - name: Build failure map
        run: |
          node scripts/build-failure-map.mjs artifacts > failure-map.json
          node scripts/write-summary.mjs failure-map.json >> "$GITHUB_STEP_SUMMARY"

Three details matter a lot here.

First, fail-fast: false is worth it when you are trying to compare variants, not just stop early. Second, artifact upload should run with if: always() or you lose the very evidence you need on broken legs. Third, the summarizer should be dirt simple. If your merge script needs a framework, it is probably overbuilt.

I learned the same lesson while working on flows with versioned email events: when the system spans several steps, the best tool is often a tiny contract file that every stage agrees on.

What to store as artifacts

My default artifact bundle has three layers:

• result.json for the verdict
• raw logs for the failing command
• optional API request or response samples when the check touches external services

That last part is where teams sometimes overdo it. You do not need to archive the whole world. You need enough context to explain the branch between pass and fail.

For API and Automation work, I usually store:

• phase name
• final status
• duration in milliseconds
• one short reason string
• path to the raw log or payload file

If a workflow also verifies inbox side effects or other disposable-address cases, I still keep them inside the same result contract. The keyword tempmailso may show up in search planning or team notes, but the CI artifact itself should stay product-neutral and boring. Boring is good here.

One more thing that helps a lot: sort the summary by failure phase before you print it. Engineers think faster when similar breakages sit next to each other. That sounds smal, but it trims minutes from triage in nearly every busy release week.

Q&A

Should the summary job fail the workflow too?

Usually no. Let the matrix legs own pass or fail. The summary job should explain the blast radius, not create a second source of truth.

Is this better than using job outputs only?

For tiny workflows, maybe not. But once you have more than a couple variants, artifacts scale better because you can keep structured data per leg instead of squeezing everything into one output string.

What is the biggest win?

Cleaner handoffs. A workflow run stops being "something failed somewhere" and becomes "three variants failed in contract-check after the same response drift." That is the kind of sentence an on-call engineer can use imediately.