Your CI is green and your pipeline produced nothing

ProveDone — Wed, 02 Sep 2026 22:43:33 +0000

Two failures broke an unattended pipeline I ran for a few months, and neither of them looked like an error. I want to describe them precisely, because I spent a long time debugging the wrong thing both times.

Failure one: exit 0, nothing on disk

A publisher posted a blog entry successfully. It then verified the post by rebuilding the URL from parts — including a hardcoded month:

const check = `${base}/2026/08/${slug}.html`;   // August, forever

On September 1st, every publish reported 404. Publishing worked perfectly. The check was broken. Because the check ran inside the same script and swallowed its own error, the job still exited 0.

For a week the dashboard was green and the blog was empty.

Failure two: the work finished, then cleanup died

An image job produced 43 of 45 files, then threw EPERM deleting its own temp directory — a child process still held a handle on Windows. Exit code 1.

The 43 finished images were discarded and a fallback path never ran, because everything downstream keyed off the exit code. Re-running cost forty minutes of compute to undo an rm -rf.

The thing both have in common

An exit code has one bit of information, and the interesting question has two:

	artifacts missing	artifacts present
exit 0	silent failure	pass
exit != 0	failure	false failure

An exit code cannot tell the top-left cell from the top-right one. Both return 0. It cannot separate the bottom two either. It only knows the row, and the column is the half that decides whether you actually shipped.

What I check now

After the command finishes, before deciding anything:

exists — the obvious one
size — ffmpeg exits 0 having written a 0-byte mp4 more often than you'd think
count, recursively — dist/index.js plus dist/assets/*.css is four artifacts, not one. Counting only the top level made a normal bundler build look empty
freshness — not older than N seconds
novelty — and this is the one that actually matters

Novelty is the check everyone skips

A file with the right name and the right size, written yesterday, passes every naive check ever written. A pipeline that has been dead for a week keeps reporting green.

So snapshot mtimes before the command runs:

const before = stamp(dir);          // newest mtime under dir
run(command);
const after = stamp(dir);
if (before !== null && after === before) {
  // the files are there, but this run did not make them
}

One trap: a directory's own mtime does not change when its files are overwritten in place. It changes on create, delete and rename only. If you stat the directory itself, every incremental rebuild that reuses filenames looks like it produced nothing. You have to walk the tree and take the newest mtime among the entries.

I shipped that bug and it flagged healthy builds for a day before I understood why.

Thresholds go stale, so learn them instead

You set minEntries: 30. The job grows to 200 over six months. The day it emits 40, nothing fires.

Recording what each check produced on its healthy runs fixes this:

[X] out/img  <- 5 vs usual 30 (dropped below 50% of baseline)

Nobody configured 30. Two rules keep it honest:

Only passing runs teach the baseline. Otherwise a broken run quietly lowers the bar and the next broken run looks normal.
Never record a measurement you did not finish. If the walk hit its budget and bailed, publishing the partial count as if it were the real one is exactly the lie the whole exercise exists to prevent.

The one that cost me the most

I built a version that, when a tree was too large to walk, "withheld judgement" — which I implemented as ok = true.

A build that produced literally nothing inside a large directory came back PASS, green dashboard, exit 0.

Withholding a verdict is not the same as passing one. It now refuses the check outright and exits 2, and the reason travels all the way to the dashboard and the Slack message. If I can't measure it, I don't get to say it's fine.

Should a false failure fail your build?

I made FALSE_FAILURE exit 0 on purpose. If every artifact is on disk and the process only died in cleanup, re-running a forty-minute render buys nothing.

I'm genuinely unsure this is right, and it's the decision I'd most like to hear arguments against. What do you do — trust the artifacts, or trust the exit code?

I packaged this up as a small tool. The five failure patterns and a runnable two-second demo are free, no signup: Five ways your pipeline lies to you. The tool itself is $19 — but run the demo first, it reproduces every verdict on your machine and installs nothing.

DEV Community: ProveDone