Green badge, empty pipeline. Exit code 0 is not evidence.
Somewhere in your CI history there is a green checkmark that verified nothing. Not “verified and found no problems” — verified nothing. Zero files opened, zero tests collected, zero policy rules evaluated. The step ran, the exit code was 0, and everyone moved on.
This is the vacuous pass: a check that succeeds precisely because it never encountered anything it could fail on. Exit code 0 means “I did not fail.” A suite of ten thousand assertions and a suite that collected nothing both report it.
Once you see the pattern, you see it everywhere — especially in infrastructure pipelines, where checks guard the scariest operations you run. Five places it hides in GitHub Actions and Terraform workflows, and the fail-closed fix for each.[
Five checks. Zero files examined. Five green badges.[
Where the five patterns hide in a typical GitHub Actions pipeline — and the one gate that catches them all.
1. The test suite that tested nothing
This is the canonical example, and it’s real:
$ go test ./...
? pkg/billing [no test files]
? pkg/auth [no test files]
ok pkg/util 0.004s
Exit code: 0. go test treats “no test files” as fine, so renaming a directory or breaking a build tag silently shrinks the suite to nothing — and it stays green.
Python’s pytest made the opposite choice: zero collected tests exits with code 5, which fails the step. That single design decision is the whole lesson. The tool that fails closed caught the problem; the tool that passes open hid it.
The fix: never trust the exit code alone — assert that test output contains ok lines, or that the collected-test count is greater than zero:
go test ./... 2>&1 | tee test.log
grep -q "^ok " test.log || { echo "FAIL: no test packages ran"; exit 1; }
2. The loop that never iterated
Shell loops don’t complain about empty input. They just don’t run:
for f in $(find terraform -name '*.tf' -newer .last-scan); do
tflint "$f"
done
echo "lint complete"
If the find matches nothing — wrong directory, clock skew on the timestamp file, a typo in the path — the loop body executes zero times and the step exits 0. “Lint complete.” Lint never started. Any upstream step that produces an empty list turns the downstream check into theater.
The fix: count first, then decide. Make emptiness an explicit, loud branch:
files=$(find terraform -name '*.tf' -newer .last-scan)
[ -n "$files" ] || { echo "FAIL: no files matched, refusing to pass vacuously"; exit 1; }
for f in $files; do tflint "$f"; done
The rule: if a check is supposed to look at N things, it must prove N > 0 before it is allowed to pass.
3. The masked pipe
terraform plan -no-color | grep -q "0 to add, 0 to change, 0 to destroy"
echo "plan is clean"
Looks reasonable. It isn’t. Without pipefail, the exit status of a pipeline is the exit status of its last command. If terraform plan fails (bad credentials, broken config, lock contention), grep still runs, finds nothing, exits 1… and then echo runs anyway and the step exits 0. The plan never completed; the pipeline declared it clean.
GitHub Actions’ default bash shell sets pipefail for you; Jenkins sh steps, GitLab jobs, and plain scripts usually don’t. The moment a pipeline step contains a |, confirm which shell options are in force — “probably pipefail” is not an answer.
The fix: set -o pipefail (with set -euo) at the top of every script CI runs, and treat any pipeline whose failure semantics you can’t state precisely as a bug.
Watch the mechanism in ten seconds:
$ cat > demo.sh <<'EOF'
false | grep -q "clean"
echo "pipeline exit: $?"
EOF
$ bash demo.sh; echo "step exit: $?"
pipeline exit: 1
step exit: 0
The pipeline reports 1 (grep found nothing), but the step reports 0 — because echo ran last and succeeded, laundering the failure. Now the same shape with pipefail:
$ cat > demo2.sh <<'EOF'
set -o pipefail
false | grep -q "clean"
EOF
$ bash demo2.sh; echo "step exit: $?"
step exit: 1
pipefail doesn’t change what your tools do — it stops the shell from laundering their failures through whatever happened to run last.
4. The path filter that filtered everything
This one is GitHub-specific and it bites at the merge-gate level:
jobs:
detect:
runs-on: ubuntu-latest
outputs:
tf: ${{ steps.filter.outputs.tf }}
steps:
- uses: dorny/paths-filter@v3
id: filter
with:
filters: |
tf: 'terraform/**'
security-scan:
needs: detect
if: needs.detect.outputs.tf == 'true'
runs-on: ubuntu-latest
steps:
- run: ./run-tfsec.sh
If the filter matches nothing, security-scan is skipped. And GitHub reports a skipped job to branch protection as a success: success, skipped, and neutral all satisfy a required check. The scan never ran, the gate went green, the merge landed.
The failure is requiring the conditional job itself as the merge gate instead of a summary that knows the difference between “scanned and clean” and “never scanned.”
The fix: the pattern the ecosystem converged on — require one summary job with if: always() that fails unless the scan actually ran:
scan-gate:
name: Security scan gate
needs: [detect, security-scan]
if: always()
runs-on: ubuntu-latest
steps:
- run: |
if [ "${{ needs.security-scan.result }}" != "success" ]; then
echo "FAIL: security scan did not run to success (result: ${{ needs.security-scan.result }})"
exit 1
fi
Require scan-gate in branch protection. Now “skipped” is a failure, and the only green path is a scan that genuinely executed and passed.[
Pattern #4, step by step. Every box above is green in the GitHub UI.
5. The policy check on an empty plan
Terraform pipelines love a policy gate: convert the plan to JSON, evaluate rules, block the merge on violations. The vacuous version:
terraform show -json tfplan > plan.json
jq -e '[.resource_changes[] | select(.change.actions[] | contains("delete"))] | length == 0' plan.json
echo "policy check passed"
If tfplan was never created — the plan step failed but the workflow continued, the artifact came back empty — then plan.json is empty or missing, jq finds zero deletions in zero changes, and the policy “passes.” The check that was supposed to stop destructive applies just waved one through, because there was nothing to object to.
Note the asymmetry: terraform validate and terraform plan themselves fail closed on an empty directory (”No configuration files”). The tools are fine. It’s the glue around them — the JSON conversion, the policy script, the artifact handoff — that passes open.
The fix: assert the plan exists and is non-trivial before evaluating policy against it:
`[ -s plan.json ] || { echo "FAIL: plan.json missing or empty"; exit 1; }
jq -e '.resource_changes | length > 0' plan.json > /dev/null \
|| { echo "FAIL: plan contains no resource changes to evaluate"; exit 1; }
only now run the policy check`[
The same green badge — only one of them means anything.
The vacuous-pass audit
For every check in your most important pipeline, answer three questions:
[ ] Did it run? Not “is the step green” — did the tool actually execute? (Check skip conditions, if: guards, and path filters.)
[ ] Did it look at anything? Assert file counts, test counts, scanned-resource counts. Zero inputs must be a failure, not a pass.
[ ] Would it fail if it found something? Trace one real violation through the whole chain — tool exit code, pipe behavior, artifact handoff, gate job — and confirm red actually propagates.
Audit the irreversible gates first: plan-policy gates, security scans, migration tests, backup verifications. A vacuous unit-test step wastes confidence. A vacuous destroy-guard wastes infrastructure.
A 20-minute audit, worked
Pick your deploy pipeline and walk it job by job. For each check, write down the answers to the three audit questions — did it run, did it look at anything, would it fail — plus the one assertion that would prove it. Twenty minutes, one pipeline, and you usually find at least one of these:
Unit tests: Yes — but did the suite shrink? — Count the ok lines — go test ./... 2>&1 \ — grep -c “^ok “ must be > 0 and match the package list
Lint loop: Yes — Count files before looping — [ -n "$files" ] guard; log the count
Plan summary grep: Yes — but did the plan succeed? — Is pipefail in force? — set -o \ — grep pipefail, or check the workflow’s shell options
Security scan: Only if the path filter matched — Read the job result, not its color — needs.security-scan.result == 'success' in the gate job
Policy on plan.json: Yes — but was the plan real? — Assert the plan is non-empty — [ -s plan.json ] and resource_changes \ — length > 0
The pattern in the last column is the whole post in miniature: every check needs a second, smaller check that testifies the first one did its job. Checks don’t verify themselves. Something has to check the checker — and that something is a one-line assertion, not a second pipeline.
The takeaway
Every one of these patterns has the same root cause: we built pipelines that answer “did anything fail?” when the question we needed was “did anything get checked?” Exit codes collapse those two questions into one number, and the collapse always favors silence.
Count the files. Assert the scan ran. Make skipped a failure. Demand evidence, not absence of failure. A check that cannot prove it looked at something is not a check — it’s a green badge with a story nobody verified.
Honey Kiran · DevOps & Platform Engineering
I write about SRE, monitoring, Terraform, and the unsexy work that keeps systems alive at 3am.
Go audit your most important pipeline this week — and send this to the teammate whose “all checks passed” you’ve always been slightly suspicious of.
Disclaimer
This post describes my personal experience with my own systems and accounts, in my own words. It is a first-person account of what I observed and what I do in my own work — not a statement of any company's intentions, policies, or internal decisions. Where I do not know something, the post says so.
I am not affiliated with, endorsed by, or speaking for any company, product, or vendor named in this post. All product and company names are trademarks of their respective owners and are mentioned only to describe the experience accurately.
Any error messages, documentation excerpts, or figures quoted here are reproduced from my own records or from the public sources linked in the post, believed accurate at the time of writing. Product behavior changes over time; verify against current official documentation before relying on any detail.
Nothing in this post is professional, financial, legal, or security advice. Examples are illustrative — validate in your own environment before running anything in production.
Corrections are welcome: if a company named here publishes an official statement, or a detail above is wrong, I will update this post and note the correction. The goal is an accurate public record of my experience, not a verdict on anyone.
© 2026 Honey Kiran. All rights reserved.
Top comments (0)