The Scheduled Step Failed on Night One. The Heartbeat Stayed Green.

Originally published on hexisteme notes.

I was writing a briefing about whether it was safe to just wait until September. A governance kill-condition in one of my projects is evaluated over non-overlapping 56-day blocks, and the first block closes on 2026-09-27. Before I could recommend waiting, I wanted to check that the series feeding that condition was actually accumulating. A predecessor note covered a related but different failure — gates whose production call site simply doesn't exist, findable once you know to ask a database instead of a shell. This one is not that. Every static check that note would have run against this system comes back clean.

One row, and it wasn't the schedule's

A nightly job at 03:00 mirrors a corpus of session transcripts to an external volume, then runs a sequence of follow-up steps. One of those steps computes the meter that the kill-condition reads. When I queried the table holding that meter's history, it had exactly one row. That row had been written by hand the previous afternoon, in the session where I built the meter — not by the schedule that was supposed to add one every night. The step had been wired into the job that same day, so the WARN line below is its entire scheduled career: one attempt, one failure.

The archive log already had the explanation sitting in it, in plain WARN text:

[archive_corpus] WARN 2026-08-03T18:00:05Z: decision_trace failed (archive succeeded)
  — ModuleNotFoundError: No module named 'pydantic'

Why an interactive shell always lied to me

The root cause was ordinary once I looked straight at it. launchd hands a scheduled job PATH=/usr/bin:/bin:/usr/sbin:/sbin and nothing else. A bare python3 inside the script resolves, under that PATH, to Apple's system Python — which ships with no third-party packages. The framework Python on the same machine, the one that answers to python3 in an interactive terminal, is on 3.13 and has pydantic 2.12.5 installed. Every time I'd run the script by hand to sanity-check it, it worked, because my shell's PATH was never the scheduler's PATH. The check I trusted and the environment that mattered were two different machines wearing the same hostname.

Three layers of concealment, each reasonable on its own

None of these were sloppy decisions. Stacked, they made a step that had never once succeeded on schedule look exactly like one that had.

1. A sibling step in the same job succeeded every night. It's stdlib-only, so it has no dependency to lose, and it had logged INFO on all four nights the job had been alive. From outside, the job looked alive, because part of it was.
2. The failing step is fail-soft by design, and that design is correct. It must not be allowed to kill the archive run, because the archive is the only long-term preservation path for the corpus. But the failure got absorbed into a single WARN line, and the job's heartbeat file kept reporting "status": "ok" regardless.
3. My own resume notes declared it done. They said, in one sentence: "Running. Wired into the nightly job (03:00)." The wiring claim was true — the step really was in the job. The running claim was false. Both halves sat in the same sentence, and I'd read past it more than once without separating them.

What was actually at stake

Had I not checked, the 56-day block would have closed on 2026-09-27 with a single data point in it, and the kill-condition would have been permanently INDETERMINATE — not failing, not passing, just structurally unable to ever be evaluated, because you can't derive a rate from one measurement. "Wait until September" was, without my knowing it, a plan that produced nothing. The schedule wasn't slow. It wasn't running at all.

I want to be exact about how I found out, because it wasn't a detector. I noticed one night in, on the morning after the only scheduled run that step ever attempted, and only because I'd sat down to answer an unrelated question about whether waiting was safe. There was nothing in the system that was going to raise its hand over the remaining 55 nights. Catching it on night one was luck; the fix below is the part that isn't.

The fix

Two parts. The first closes this specific hole; the second is the one I'd reuse anywhere.

Resolve the interpreter by capability, not by path. Pinning a version path just relocates the failure to the next Python upgrade, which will die the same silent way. Instead, probe for an interpreter that can actually import the dependency you need:

resolve_py_with_deps() {
    local cand
    for cand in /Library/Frameworks/Python.framework/Versions/Current/bin/python3 \
                /opt/homebrew/bin/python3 /usr/local/bin/python3 python3; do
        if command -v "$cand" >/dev/null 2>&1 && "$cand" -c 'import pydantic' >/dev/null 2>&1; then
            command -v "$cand"; return 0
        fi
    done
    return 1
}

If nothing qualifies, that's its own state — nopy — distinct from a computation that ran and failed, and distinct from a step that was deliberately skipped.

Fail-soft is correct design. Fail-silent is not. If a step is allowed to fail without killing its parent job, its state has to land somewhere the parent's own monitor already looks. Don't invent a new channel for it — nothing reads new channels. The heartbeat file already existed, so the state rides along in it:

write_heartbeat "ok" "files=${DST_FILES} new=${NEW_COUNT} shrunk=${SHRUNK_COUNT} ing=${ING_STATE} dt=${DT_STATE}"

Then the monitor has to actually read those keys, and complain when they're absent entirely — because a monitor that can quietly stop monitoring is the identical defect, one level up.

Verifying it for real

This is the part that generalizes past this one job.

1. Reproduce the target environment, not your own shell. Running a script by hand proves nothing about how a scheduler will run it: env -i PATH=/usr/bin:/bin:/usr/sbin:/sbin ./the_job.sh
2. Then run the actual scheduled job, not your reproduction of it — launchctl kickstart -k gui/$(id -u)/com.example.job. In my case the reproduction wrote the second row and the real scheduled run wrote the third, taking the meter's history from 1 row to 3, and the heartbeat came back as files=3773 new=5 shrunk=2 ing=ok dt=ok.
3. Compare the declared cadence against the actual output interval. This is the general-purpose detector, independent of what the job does: SELECT run_id, computed_utc FROM meter_run ORDER BY run_id DESC LIMIT 3; If a job that's supposed to be nightly has one row, and you're the one who wrote it, the schedule is dead — no matter what the code, the crontab entry, or the documentation says.

Why this is a different kind of defect

Two earlier notes in this series were about gates whose production call site didn't exist at all — grep found nothing, and a fill-rate query on the underlying table found the rest that grep missed. This one is close to the inverse. Grep finds the call site here; it's right there in the script. The scheduler entry exists. Run the script yourself and it works. Every static thing you could check about this system is correct. The defect lives at runtime, in the gap between an interactive shell's environment and an unattended scheduler's environment, and a fail-soft design is what keeps it from ever surfacing as a failure. It's the only variety in this family where the monitor doesn't just miss the problem — it actively reports green while the problem is happening.

The constructive complement

Fixing this pushed me to generalize a check that had been hardcoded to look at a single row of a governance table: does every kill-condition in the table actually have a way to be measured? Turning that single-row check into a full-table sweep immediately surfaced two more instances sitting in the same table, neither related to the launchd issue:

• a row whose status cell read "0 events," for a table that doesn't exist anywhere in the schema — vacuously true, and typographically identical to a genuinely measured zero;
• a row claiming its meter had been "included in Phase 1," when Phase 1 was already marked complete and the table that meter needs for its denominator had never existed.

The lesson underneath both: don't try to parse the prose of the condition itself. Condition text mixes real values, object names, and names of things entirely outside the system, so pulling identifiers out of a sentence can't reliably tell those apart — not as an implementation gap, but in principle. The fix instead is a declared meter column, a small controlled vocabulary that a checker validates against instead of inferring from prose:

table:NAME (the table exists and has at least one row) · table?:NAME (the table exists; zero rows is itself a legitimate observation) · guard:NAME · script:PATH · os:... · and n/a: followed by a mandatory reason.

Making that reason mandatory is what removes the silent exemption — you can no longer write n/a and move on without saying why. One aggravating pattern gets flagged automatically: if a condition's declared meter is n/a: while its status cell starts with a number, that's the exact defect above, wearing a different table. Run against the real table, this turned up 13 conditions total, of which 4 had automated meters and 9 were explicitly n/a: with a written reason attached to each. Having that number visible, instead of assumed, was the actual point of doing the sweep. I also wrote seven mutation tests, each one deliberately breaking a single rule — deleting the column, pointing at a table that doesn't exist, an empty reason, a status cell that's numeric where the meter says n/a: — and asserting that the checker fails in each case, plus two controls that assert it still passes on the pristine table and tolerates a legitimately empty one. A checker you've only ever watched pass is itself unverified; so is one you've only ever watched fail.

More notes at hexisteme.github.io/notes.