Exit Code 0, Nothing Produced: The Automation Failure I Shipped Five Times

The scheduler log is a column of green. Started, finished, rc=0, started, finished, rc=0, day after day after day. The job that collects candidates for my syndication queue had been running like that for weeks.

The queue was empty. It had been empty the whole time.

I have now shipped this exact bug five separate times in one project, in five different jobs. Every one went unnoticed for weeks, and every one was eventually caught by a human asking "hang on, when did we last actually get anything out of this?" — never by a monitor, never by an alert, never by a failed run. There was nothing to fail.

The pattern deserves a name because it is invisible by construction: the job succeeds and produces nothing.

Why the exit code can't see it

An exit code answers one question: did the process reach the end without raising? That is a statement about control flow, not about work. A script that iterates over zero items and prints a tidy summary exits 0 with total confidence.

The shapes I've actually hit:

• A collector that returns the empty list on failure. except Exception: return [] makes "the search path is broken" and "there was nothing new today" arrive at the caller as the same value. The caller cannot tell them apart, so it picks the cheerful interpretation.
• except: continue inside the loop. One bad item is worth skipping. Every item being bad looks identical from outside — the loop completes, the counter is zero, the function returns.
• An upstream API that answers 200 with an empty array. No transport error, no status to branch on. Auth silently downgraded, a parameter quietly rejected, a filter that now matches nothing — all of it arrives as a well-formed successful response containing no rows.
• Partial harvest. Four of five sources die, the fifth works, the run reports success. This one is the worst, because there is output. Nobody knows what the denominator was supposed to be.

What all four share: nowhere in the system is there a written statement of what this job is supposed to leave behind. Without that statement there is nothing for a machine to check.

The fix: make every job declare its output

I moved the definition of every scheduled job into one JSON file, and required each entry to carry a produces block — a machine-readable claim about what must exist after a healthy run.

{
  "key": "gsc-daily",
  "run_cmd": "scripts/analytics/run_gsc_daily.py",
  "timeout_s": 1800,
  "output_expectation": "conditional",
  "produces": {
    "type": "file_fresh",
    "path": "data/ops/gsc_daily.jsonl",
    "max_age_h": 30,
    "desc": "daily search performance was appended"
  }
}

Then I stopped letting jobs report on themselves. A runner wraps each one, and the runner writes the ledger entry.

Wrapping rather than instrumenting mattered more than I expected. If each script records its own outcome you have to edit five places, and the one you forget is precisely the one that fails quietly. Worse, a script that never starts — bad interpreter path, an import error on line 1, a wrapper that dies before the interpreter is reached — cannot possibly log its own failure. The judgement has to come from outside the process.

Measure the delta, not the total

The runner measures the declared output twice: once before the child process starts, once after it exits.

before = measure(job)
rc, out, err = run_tree(cmd, cwd, timeout_s)
after = measure(job)

if kind == "row_count" and before is not None:
    produced = max(0, int(after) - int(before))
else:
    produced = int(after)   # freshness is a 0/1 state, not a count

The absolute count is a trap. Yesterday's rows are still sitting in the table, so "there are rows" stays true on the day the collector breaks and on every day after. The number that describes this run is the increase.

But the delta is a trap in the other direction, and I walked into that one too. Some jobs upsert: they rewrite one row per day, or refresh a single file in place. Their row count never grows. Judged by delta, a perfectly healthy job reports zero forever, you get an alert every morning, and within a week you have trained yourself to ignore it. So the two kinds are declared separately — cumulative outputs judged by increase, in-place outputs judged by freshness against max_age_h. Collapsing them into one rule produces false alarms, and false alarms are how a monitoring system dies.

Not every zero is a failure

Some jobs are legitimately allowed to produce nothing. My indexing-request job is capped by a daily quota and drains a pending queue; on a day when the queue is empty, zero is the correct answer. So each job also declares output_expectation: every_run means an empty result is a defect, conditional means it may be empty and the thing worth counting is how many consecutive silent days have passed.

The distinction underneath that is the one I'd carry to any scraper or collector:

"We looked and only found things we already have" and "we couldn't look" are different outcomes. Only the second is a failure.

A collector that finds results and discards all of them as duplicates is working correctly. A collector that finds no results at all is not reporting an absence of new material — it is reporting that its search path returned nothing, which for an established source means the path is broken. So the two are counted separately, and only the second returns a non-zero status. Before I made that split, the healthy case and the broken case printed the same line.

Failure is expensive when nothing stops early

Once I could see the failures, a second problem surfaced: broken jobs were burning enormous amounts of wall time on their way to producing nothing.

One collector had its search path blocked. Its response was to try the next seed keyword, hit a 40-second navigation timeout, try the next, hit another 40-second timeout, and continue down the entire seed list. It burned 1,380 seconds and returned 0 candidates. Every one of those timeouts was foreseeable after the third.

fails, FAIL_CAP = 0, 3
for q in seeds:
    try:
        page.goto(url_for(q), timeout=40000)
    except Exception as e:
        fails += 1
        print(f"  [{q}] failed ({fails}/{FAIL_CAP}): {str(e)[:90]}")
        if fails >= FAIL_CAP:
            print("  three consecutive failures - stopping")
            break
        continue
    fails = 0          # any success breaks the streak
    harvest(page)

Two rules, both cheap. A consecutive-failure cap, reset on any success, because a source that refused us three times in a row will refuse the fourth. And a per-job wall-clock ceiling in the job definition, so a job that hangs rather than fails still terminates on schedule instead of still running when tomorrow's copy of itself starts on top of it.

Kill the tree, then verify it died

The wall-clock ceiling introduced its own bug, specific to layered automation.

subprocess.run(timeout=N) kills its direct child. My batch jobs are three generations deep: Python starts a browser driver, the driver starts a browser. Killing the direct child leaves the browser orphaned, holding a profile lock that quietly breaks the next day's run — a silent no-op caused by the very mechanism meant to prevent silent no-ops.

On Windows the job has to be launched into its own process group and killed as a tree:

flags = subprocess.CREATE_NEW_PROCESS_GROUP if os.name == "nt" else 0
proc = subprocess.Popen(cmd, cwd=cwd, stdout=fo, stderr=fe, creationflags=flags)
try:
    proc.wait(timeout=timeout)
except subprocess.TimeoutExpired:
    for _ in range(3):
        subprocess.run(["taskkill", "/F", "/T", "/PID", str(proc.pid)],
                       capture_output=True, timeout=20)
        time.sleep(1.5)
        if not is_alive(proc.pid):        # verify, don't assume
            break
    else:
        note_orphan(key, proc.pid, cmd)   # write it down; nobody notices otherwise

The retry loop exists because of the same class of mistake as everything else here. My first version called taskkill once and recorded its return code as the outcome. A child survived anyway. The parent was gone, the lock was released, the ledger said the timeout had been handled cleanly — and a stray process sat there until I went looking for it. "I issued a kill" and "the process is gone" are two different facts, and only the second is worth recording. When the kill genuinely fails, the PID and its command line go into an orphan list so the next run of that job can clean up after its predecessor.

Logs and artifacts are separate evidence

The last thing that broke my trust in logs: I launched two jobs in the background and later found their log file was 0 bytes, while the ledger showed two successful runs with real output counts.

Both records were accurate. The work happened; the log capture didn't, because output was block-buffered and the redirect never flushed. But the lesson stands on its own: a log is a story the process tells about itself. It can be missing, truncated, or buffered away while the work succeeds — and it can be full of confident green while nothing was produced. Those are independent failure modes. Judge the run by its artifacts, and keep the log as evidence for the postmortem rather than as the verdict. That split is one of the running themes in the operational notes I keep on this.

What to ask your own cron jobs

• For each scheduled job, can you name in one sentence what file or row it must leave behind? If not, it has no success criterion — only an exit code.
• Is that criterion written somewhere a machine reads, or only in your head?
• Are you checking a total or a delta — and if a delta, does that job actually append, or does it upsert in place?
• Can this job distinguish "found nothing new" from "couldn't look," and does it return a different status for each?
• Does every retry loop have a consecutive-failure cap, and every job a wall-clock ceiling?
• When the ceiling fires, do you kill the process tree and then verify it is gone?
• When was the last time this job produced something? If you have to go and dig to answer that, that is the whole problem.

Comments

[Raknaos]

That rc=0 column being green while the queue stays empty is the exact failure I kept hitting on an unattended loop I run on a small VPS. The monitor printed a status line every cycle, the cron marked everything as run, and nothing real ever came out — so nothing ever alerted, because nothing ever failed.

The fix that finally stuck was refusing to treat the exit code as the signal. I made the scheduler assert on the artifact instead: after each run it checks "did the output actually change / did anything get produced," and an empty-but-successful run is reported as a distinct state, not a green tick. I also added a freshness watchdog — if the last real artifact is older than X, that alone trips an alert regardless of what the last job reported.

Your point that "there was nothing to fail" is the whole trap. Success-as-absence is invisible by construction, so you have to invert it: define what a useful run looks like and alert when that doesn't happen, instead of waiting for a failure that will never fire. Appreciate you naming it — I had no clean way to describe this to people until now.