My CI check passed every My CI check passed every build for four days. The JSON said {}

#automation #cicd #github #opensource

This week I found out the GitHub Action for my open-source tool never failed a single build.

The tool is DocsWatcher. It scans code for API calls that already have a shutdown date (OpenAI models, Stripe API versions, that kind of thing) and fails CI when it finds one. So a check that never fails is the one bug it cannot have.

How the Action decides

The CLI is Java, compiled to a native binary with GraalVM. The Action runs it, reads the JSON report, counts the findings whose severity is breaking, and fails the step if that count isn't zero.

That design was deliberate. The CLI exits 1 when a breaking finding exists, and the Action treats exit 1 as "a result, not a failure". It wants the count from the report, not a bare exit code.

What the binary actually printed

For every finding, this:

[
  {},
  {},
  {},
  {}
]

Four findings, no fields. The count of breaking findings was always zero, so the step always passed.

Why

GraalVM builds a closed world. Anything reached only by reflection is dropped unless it is registered. Jackson writes a Java record by reading its accessor methods reflectively, and my Finding record was never registered. So the native binary serialised every finding as an empty object. No exception, no warning.

The fix is a few lines of reachability metadata:

{
  "type": "dev.docswatcher.engine.Finding",
  "methods": [
    { "name": "severity", "parameterTypes": [] },
    { "name": "change", "parameterTypes": [] }
  ]
}

(The real entry lists every accessor.)

Why the tests missed it

The unit tests run on the JVM, where reflection just works. They were green the whole time.
The release smoke test did run the native binary. It checked that a repository with a breaking finding exits 1. It did. The exit code was right all along. Only the JSON was empty.

What changed

The release smoke test now checks the JSON itself, on each platform's own runner:

printf '%s\n' "$json" | grep -q '"severity": "breaking"' \
  || { echo "::error::the findings carry no severity"; exit 1; }

The Action refuses a report whose findings have no severity, instead of counting it as clean. A broken binary now fails loudly.
The Action's v0 tag points at the fixed release, so existing users get the fix without changing anything.

Looking for the same class of bug found a second one: the native binary crashed on any repository where a finding had more than one location, because an array type (Evidence[]) wasn't registered either.

The lesson

Test the artifact you ship, not the build you test. My tests proved the JVM build was right. Nobody had checked the bytes users actually downloaded.

A question for anyone shipping GraalVM native images or other AOT builds: how do you catch missing reflection metadata before your users do? The tracing agent helps, but it only sees the paths your tests exercise.

DocsWatcher is open source: github.com/jameskomo/docswatcher. The fix is in v0.3.0 and later.

DEV Community