A script that shows what your CI badge is actually testing

Every CI file is a set of claims. "We test on Node 14, 16 and 18." "We install with yarn." "We use actions/checkout." Nobody re-reads them, because the badge is green and green reads as fine.

A while back I read one of these files line by line and found the matrix cell that ran the tests was the only cell setting LANG=jp_JP — a locale that does not exist. The badge had been green for years. (That write-up is here.)

That was one file. The pattern is mechanical, so I wrote a script for it.

The script

ci_truth.py takes owner/repo, pulls .github/workflows/*.yml from the public API, and prints the claims that no longer hold. No token needed for a handful of repos.

#!/usr/bin/env python3
"""ci_truth.py - what a GitHub Actions file claims vs what it actually runs.
Usage: python3 ci_truth.py owner/repo [branch]"""
import json, re, sys, urllib.request

TODAY = "2026-09-21"
EOL_NODE = {12: "2022-04-30", 14: "2023-04-30", 16: "2023-09-11",
            18: "2025-04-30", 20: "2026-04-30"}
ODD_DEAD = {13, 15, 17, 19, 21, 23, 25}            # never an LTS
LATEST = {"actions/checkout": 7, "actions/setup-node": 7}


def get(url):
    req = urllib.request.Request(url, headers={"User-Agent": "ci-truth"})
    return urllib.request.urlopen(req, timeout=30).read().decode()


def major(tok):
    m = re.match(r"v?(\d+)", str(tok).strip())
    return int(m.group(1)) if m else None


def dead(m):
    return m in ODD_DEAD or (m in EOL_NODE and EOL_NODE[m] < TODAY)


def scan(text):
    out = []
    for m in re.finditer(r"node(?:-version)?:\s*\[([^\]]+)\]", text):
        for tok in m.group(1).split(","):
            tok = tok.strip().strip("\"'")
            if major(tok) and dead(major(tok)):
                out.append("matrix tests Node %s - EOL %s" % (
                    tok, EOL_NODE.get(major(tok), "never an LTS")))
    for m in re.finditer(r"uses:\s*(actions/[a-z-]+)@(v?\d+)\b", text):
        if LATEST.get(m.group(1)) and major(m.group(2)) < LATEST[m.group(1)]:
            out.append("%s@%s - %d majors behind" % (
                m.group(1), m.group(2), LATEST[m.group(1)] - major(m.group(2))))
    for m in re.finditer(r"run:\s*(npm|yarn|pnpm)\s+(install|i)\b([^\n]*)", text):
        if m.group(1) == "npm":
            out.append("`npm install` instead of `npm ci` - lockfile drift "
                       "cannot fail the build")
        elif "--frozen-lockfile" not in m.group(3):
            out.append("`%s install` without --frozen-lockfile" % m.group(1))
    return out


def main():
    repo = sys.argv[1]
    branch = sys.argv[2] if len(sys.argv) > 2 else json.loads(
        get("https://api.github.com/repos/" + repo))["default_branch"]
    files = json.loads(get("https://api.github.com/repos/%s/contents/"
                           ".github/workflows?ref=%s" % (repo, branch)))
    total = 0
    for f in files:
        if f["name"].endswith((".yml", ".yaml")):
            found = scan(get(f["download_url"]))
            if found:
                print("\n%s:" % f["name"])
                for x in found:
                    print("  -", x)
                total += len(found)
    print("\n%d claim(s) that no longer hold." % total)


main()

What it prints

On rrule.js (2.2M npm installs a week):

nodejs.yml:
  - matrix tests Node 14.x - EOL 2023-04-30
  - matrix tests Node 16.x - EOL 2023-09-11
  - matrix tests Node 18.x - EOL 2025-04-30
  - actions/checkout@v2 - 5 majors behind
  - actions/setup-node@v2 - 5 majors behind
  - `npm install` instead of `npm ci` - lockfile drift cannot fail the build
  - `yarn install` without --frozen-lockfile

7 claim(s) that no longer hold.

On node-fetch (142M npm installs a week):

ci.yml:
  - matrix tests Node 12.20.0 - EOL 2022-04-30
  - matrix tests Node 14.13.1 - EOL 2023-04-30
  - matrix tests Node 16.0.0 - EOL 2023-09-11
  - actions/checkout@v2 - 5 majors behind
  - actions/setup-node@v2 - 5 majors behind
  - `npm install` instead of `npm ci` - lockfile drift cannot fail the build
...
14 claim(s) that no longer hold.

Node 20 is in that table on purpose: its support window closed on 2026-04-30. As of today, Node 22 and 24 are the LTS lines. If your matrix still lists 18 or 20, part of what the badge proves is that code runs on a runtime that gets no security patches.

What the script does not catch

This is the important part. On node-fetch, the same ci.yml has an exclude block that removes Windows and macOS cells for node: "12.22.3". That version was deleted from the matrix long ago. The exclude has been dead weight ever since — a rule that excludes nothing, in a file that still says it does. A regex does not know what is deliberate and what is dust. That took reading.

Some flags are policy, not rot. A library whose engines field still supports Node 12 should test Node 12; that is a choice. Which flags are choices and which are accidents is the whole job.

Run it on yours

python3 ci_truth.py your-org/your-repo

If it prints nothing, good — your file still means what it says. If it prints something, you now know what your badge is really testing.

If you want the read done properly — the whole file, the Actions history, and a first-change list ordered by what actually matters — I do that for $25.

Free first find: send me one repo. I will send back one verified claim that no longer holds, and where in the file it lives. No charge, no signup. If it is useful, the full pass is there.

[e-mail]