Designing a compliance check that does not cry wolf in CI

Every team has a disabled check. It sits in .github/workflows/, commented out, with a note saying # TODO re-enable after we fix the noise. Nobody ever fixes the noise.

The check did not fail because it was wrong. It failed because it was annoying, and annoying is fatal in a way that wrong is not. A wrong check gets fixed. An annoying check gets deleted.

I spent a while building two small compliance scanners and got this wrong in several instructive ways. Below is what I would tell myself at the start — plus one bug that was much worse than noise, because it made the tool look like it was working while it found nearly nothing.

Start with the bug, because it is the important part

Both tools need to turn a declared version into something comparable against a catalogue. npm gives you ^12.0.0. Docker gives you 3.7-slim. nvm writes v16.20.2. Go modules say v1.19. You want the major, or major.minor.

The first version of that helper used this:

m = re.match(r"^(\d+)(?:\.(\d+))?", v.strip())

re.match anchors at the start of the string, and the pattern demands a digit there. Read that again against ^12.0.0.

The string starts with a caret. The match fails. The function returns the empty string. The component is dropped — silently, with no warning, no error and no log line.

Same inputs through the broken pattern and the fixed one:

input          buggy      fixed
'v16.20.2'     ''         '16.20'     <- what nvm writes into .nvmrc
'^12.0.0'      ''         '12.0'      <- npm's default range syntax
'~3.2'         ''         '3.2'
'>=16 <18'     ''         '16'
'v1.19'        ''         '1.19'      <- go.mod style
'3.7-slim'     '3.7'      '3.7'
'lts/gallium'  ''         ''          <- correctly unparseable

Every row that failed is a string that does not begin with a digit. That is the entire bug, and it happens to cover most of the ways version ranges are actually written in the wild — while leaving bare versions like 3.7-slim working perfectly.

The fix is one word:

m = re.search(r"(\d+)(?:\.(\d+))?", v.strip())

search instead of match: find the first number anywhere in the string rather than demanding it at position zero. Note lts/gallium still yields nothing, correctly — there is no version in it.

Why this is worse than a crash

A crash is self-reporting. A stack trace in CI gets fixed that afternoon.

This produced a plausible report. Here is the same project scanned with the buggy helper and the fixed one — a repo with a .nvmrc containing v16.20.2 and a Dockerfile with FROM python:3.7-slim:

################ FIXED ################
  Found    2 runtime/framework version(s)

  2 COMPONENT(S) PAST UPSTREAM END-OF-LIFE
    Node.js 16.20   (.nvmrc)
      upstream EOL   2023-09-11
    python 3.7   (Dockerfile FROM)
      upstream EOL   2023-06-27
EXIT = 1

################ BUGGY ################
  Found    1 runtime/framework version(s)

  1 COMPONENT(S) PAST UPSTREAM END-OF-LIFE
    python 3.7   (Dockerfile FROM)
      upstream EOL   2023-06-27
EXIT = 1

Both runs exit 1. Both print a real finding with a correct EOL date. Both look like a working tool doing its job. The buggy one lost an end-of-life Node runtime and said nothing about it.

That is the failure mode to fear. The only way to catch it is to already know the answer and compare — which is precisely the knowledge the tool exists to replace.

Three habits that would have caught it:

1. Test the strings the ecosystem actually emits, not the ones you find convenient. My fixtures said 16.20.2. Real .nvmrc files say v16.20.2 and real package.json files say ^16.20.2. The parametrised test that should have existed on day one:

@pytest.mark.parametrize("declared,expected", [
    ("v16.20.2",    "16.20"),
    ("^12.0.0",     "12.0"),
    ("~3.2",        "3.2"),
    (">=16 <18",    "16"),
    ("v1.19",       "1.19"),
    ("3.7-slim",    "3.7"),
    ("lts/gallium", ""),
])
def test_version_extraction(declared, expected):
    assert _major_minor(declared) == expected

A dozen lines. Would have failed on row one.

2. Report your denominator. If a tool sees 40 dependencies and can only extract versions from 3, that ratio is the most important number on screen. Both tools now print what they found and where:

  Scope    /tmp/demojava
           gradle.lockfile (3)
  Unique   3 components

Found 1 versus Found 2 in the runs above is the only visible difference between a working scan and a broken one. Print that number. Any scanner that silently drops unparseable input should be treated as suspect until it tells you how much it dropped.

3. Run it against a known-bad project first. Not to see whether it runs — to see whether it finds the thing you already know is there. If you cannot state the expected output before running, you cannot tell success from silence.

Exit codes: three states, not two

The default instinct is binary — 0 good, non-zero bad. That collapses a distinction that matters enormously in practice. Both tools use:

• 0 — checked, nothing found
• 1 — checked, found something that needs a human
• 2 — could not check

Case 2 is the one people skip, and skipping it is what makes checks get disabled. Consider a monorepo where the security workflow runs against every directory. Some are docs. Some are Terraform. They have no lockfiles and never will. If "no manifests found" returns 1, every one of those jobs goes red, forever, for a reason unrelated to security. Two weeks later somebody adds continue-on-error: true and the check is decorative.

Real behaviour on an empty directory:

$ cra-watch scan .

No dependency manifests found under /tmp/emptyproj

cra-watch looks for: Cargo.lock, Gemfile.lock, Pipfile.lock, composer.lock,
go.sum, gradle.lockfile, npm-shrinkwrap.json, package-lock.json,
pnpm-lock.yaml, poetry.lock, requirements-dev.txt, requirements.txt,
yarn.lock

$ echo $?
2

Exit 2, and it names every file it looked for. If you expected a finding, you can immediately see whether your manifest is on the list. "No data" and "bad news" are different facts and deserve different codes. It also hands the caller a real choice:

- name: KEV screen
  run: |
    set +e
    cra-watch scan .
    code=$?
    set -e
    case $code in
      0) echo "clean" ;;
      1) echo "::error::known-exploited component present"; exit 1 ;;
      2) echo "::notice::no manifests here, skipping" ;;
      *) echo "::warning::scanner failed, code $code" ;;
    esac

A team that wants exit 2 to be fatal — say, a repo that must always have a lockfile — can make it fatal. That is their policy decision, made explicitly, not a default imposed by my tool.

Pre-commit hooks must not block a commit they have nothing to say about

A pre-commit hook is a tax on every commit, and people notice within about two days. The rule: a hook must never block a commit in a repo where it has nothing to check. Not "should rarely". Never. The first time a hook blocks someone's commit over a repo that has no dependencies at all, they run git commit --no-verify, and once that is muscle memory the hook is gone.

pre-commit's files: regex solves this properly. Rather than run and then exit gracefully, do not run at all unless a relevant file changed:

- id: cra-watch
  name: cra-watch (EU CRA Article 14 / CISA KEV screen)
  entry: cra-watch scan
  language: python
  pass_filenames: false
  files: >-
    (?x)^(
      .*package-lock\.json|.*npm-shrinkwrap\.json|.*yarn\.lock|.*pnpm-lock\.yaml|
      .*requirements(-dev)?\.txt|.*poetry\.lock|.*Pipfile\.lock|
      .*go\.sum|.*Cargo\.lock|.*Gemfile\.lock|.*composer\.lock|
      .*gradle\.lockfile|.*packages\.lock\.json
    )$
  stages: [pre-commit, manual]

Now the hook fires when a lockfile changes and is invisible otherwise. Editing a README does not call a remote API. pass_filenames: false is deliberate too: the files: pattern decides whether to run, and the tool then scans the whole project, because a dependency change can affect the whole graph.

Cache the catalogue

Both tools read free, publicly funded APIs — CISA's KEV feed and endoflife.date. Neither charges and neither requires a key, but that is not the same as "hit it as often as you like". A pre-commit hook without caching would fetch a multi-megabyte JSON file on every commit. Multiply by a team, multiply by a year. That is how a free public service gets rate limits.

Cache on disk, respecting the platform convention:

CACHE_DIR = Path(os.environ.get("XDG_CACHE_HOME", Path.home() / ".cache")) / "cra-watch"

Then pick a TTL from how fast the data actually moves. KEV gains a handful of entries in a typical week, so a 6-hour TTL loses nothing. EOL dates are published months ahead, so 24 hours is generous. Always provide --refresh, because the moment you cache something, someone will need to not-cache it. And report the version you used, so a stale cache is visible rather than invisible:

  KEV feed    1709 entries, catalogue 2026.09.11

That line is how you notice you are looking at yesterday's answer.

Fail open on network errors

This one generates arguments, so here is the reasoning rather than the rule.

Your check calls a remote API. The API has an outage. If the tool fails closed, that outage now blocks every deploy at your company — your ability to ship a hotfix depends on the uptime of a third party you have no relationship with. Worse, the failure is unrelated to your risk: nothing about your code changed, no new vulnerability appeared, a webserver in another country returned 503.

So: warn loudly, use the cache if there is one, and exit 0.

except Exception as e:
    if cache.exists():
        eprint(f"warning: could not refresh KEV ({e}); using cached copy")
        return json.loads(cache.read_text())
    raise SystemExit(f"error: could not fetch the CISA KEV catalogue: {e}")

Same pattern for partial failures. When one OSV batch of several fails, the tool says so and continues:

warning: OSV batch 2/3 failed (timeout); those components are unchecked

The scan is now incomplete, and the output says it is incomplete. That is strictly better than either silently pretending everything is fine or blocking the release.

The counter-argument — "fail open means an attacker can bypass the check by breaking the network" — is real, and it is why this is a default, not a law. If your threat model includes that, make it fatal in your own pipeline.

The load-bearing principle

A check that fires constantly gets disabled. A check that fires rarely gets read.

Everything above is downstream of that. Exit code 2 exists so irrelevant repos stay green. The files: regex exists so the hook stays invisible. Caching exists so the check stays fast. Fail-open exists so a third-party outage does not train people to bypass it.

It is also why both tools deliberately answer narrow questions. cra-watch does not list your CVEs — only components on the CISA KEV catalogue, which is typically zero and occasionally two. pld-watch does not rank your tech debt — only components past their upstream end-of-life date. On a healthy project both print "nothing found" and get out of the way.

A security tool's most important output is the quiet one. If a developer sees your check go red, that redness needs to have meant something every previous time, or they will not look this time either.

Checklist

• Three exit states: found nothing (0), found something (1), could not check (2)
• "No data" is never a failure by default; let the caller escalate it
• Pre-commit hooks scope with files: and stay silent otherwise
• Cache public catalogues to disk, and print the catalogue version so staleness is visible
• Fail open on network errors, warn loudly, use the stale cache
• Print your denominator — how many things you found, how many you could parse
• Test against the strings the ecosystem actually emits, not the tidy ones
• Run against a known-bad project and check you get the answer you already knew

The last two caught a real bug. The rest is what keeps the tool installed long enough to matter.

---

Both tools are MIT and single-file: cra-watch, pld-watch. Engineering tooling, not legal advice.

Comments

[Raknaos]

re.match on a caret-prefixed range is a brutal little bug because the tool stays credible the entire time it's lying. Both runs exit 1, both print a real finding with a correct EOL date, and the only difference is one dropped runtime. That's the failure mode that makes a scanner's "nothing found" untrustworthy — the empty result and the silent filter-out are the same bytes in the report.

Two questions: after the parametrised fixtures, did you add a self-check that re-scans those fixtures on every run so a future refactor can't quietly reintroduce the drop? And does CI fail when the finding count moves without an explanation in the commit — that looks like the cheapest guard against a check getting disabled the moment it becomes inconvenient.

[ntoledo319]

Both answers were no. They are yes now, and the first question found something I had wrong.

The self-check. Added tests/test_detection_census.py — it pins the exact set of (slug, version) pairs produced from a fixture tree that exercises every declaration form at once (.nvmrc, go.mod, Dockerfile FROM, requirements.txt, package.json engines + deps). Losing one fails with a message naming which declaration form disappeared, not "expected 7 got 6". Gaining one also fails until it is added to EXPECTED deliberately, because a detector appearing unreviewed is its own problem.

Then I tried to prove it worked by reintroducing the exact bug — and the census passed. That was the useful part.

The reason: detect() strips non-digits before calling _major_minor for package.json dependencies, so caret ranges never reached the broken regex on that path. re.match(r"v?(\d+)...") returns '' for ^12.3.1, ~17.0.2, >=16 <18, but '16.20' for v16.20.2 — so nvm's own format survived too. The bug was latent, not live, and my article implied a blast radius bigger than the one I could demonstrate. The function was genuinely wrong; the silent drop only becomes real the moment a refactor removes that pre-scrub, or a new caller passes a raw range.

So the census now pins the function's own contract independently of what any one caller happens to sanitise today, and the docstring says exactly that instead of the tidier story.

CI. Three jobs: unit tests on 3.9–3.13, a census guard whose failure message tells you to update EXPECTED in the same commit naming the form, and a smoke job that runs the tool against the live endoflife.date catalogue to assert the exit-code contract still holds — 1 on EOL, 0 clean, and specifically 2 (not 1) when there is nothing to check. That last one is the check most likely to get a tool disabled.

On your framing — "the empty result and the silent filter-out are the same bytes in the report" — that is the sentence I should have opened the article with. The only structural defence I have found is making the tool state what it examined, not just what it found, so an empty result is falsifiable. The census is the CI-side version of the same idea.

Commits cb0d348 and 436c5b1. Thanks — this was a better review than the code had before it.