I fixed one silent failure and stopped looking. A review found three more

#macos #automation #cli #debugging

I had a scheduled script that ran fine by hand and produced nothing on a schedule. Exit code 0, no error in the log, an output file created and empty.

The script calls a CLI tool that reads credentials from the macOS keychain. Run it in my terminal: works. Run it from launchd: Not logged in · Please run /login. The credentials had not expired.

This post is about the fix, and then about the three other places the same shape was hiding — which I did not find. Someone reviewing my code did.

The smallest difference I could reproduce

launchd hands a process a much smaller environment than a login shell does. No .zprofile, no .zshrc, so nothing those files set is present.

I bisected down to a single variable. Both runs below are identical except for one thing:

# A — with USER
env -i HOME="$HOME" USER="myname" PATH="$P" mytool -p "say hi"
# → responds normally

# B — same command, USER removed
env -i HOME="$HOME" PATH="$P" mytool -p "say hi"
# → Not logged in · Please run /login

That is the smallest difference I could reproduce. PATH was already correct in both, HOME was set in both. Remove USER and the tool decides you are not logged in.

I do not know why the tool needs USER. I did not read its source. It might use it to look up the keychain entry, it might be something else. What I can tell you is the reproduction above, because I ran it both ways.

The fix is to set the environment explicitly instead of hoping the scheduler provides it:

export PATH="/usr/local/bin:/opt/homebrew/bin:/usr/bin:/bin"
export HOME="/Users/myname"
export USER="myname"      # without this the CLI reports "not logged in"

The part that mattered more

The login failure exited 0.

My script checked the exit code to decide whether the run succeeded. The run had done nothing, and the exit code said fine. So the real fix is not the export line:

run_job() {
  "$TOOL" -p "$(cat "$prompt")" > "$out" 2>&1

  # exit code is not enough — the login failure exits 0
  if grep -qE "Not logged in|Please run /login|Invalid API key" "$out"; then
    record_incident "auth failed"; return 1
  fi
  # neither is "the file exists" — check it has content
  if [ "$(stat -f%z "$out")" -lt 50 ]; then
    record_incident "no output"; return 1
  fi
}

What I actually verified, so you know how far to trust it: I copied that logic into a standalone script and fed it five files. Three contained one of the error strings each, padded past 50 bytes so the size check could not be what caught them. One was empty. One was 201 bytes of ordinary output. The first four are caught, the last passes. That exercises the guard. It is not a test of the original failure — I did not break a live login to reproduce it.

The three I did not find

I wrote that guard and moved on. Then someone reviewed the rest of my code and came back with three more instances of the same shape. All three are theirs:

A linter I wrote printed an error for an unreadable input file and exited 0. Tested with /dev/null: 0 workflow(s) checked, exit 0. CI would read that as "nothing wrong".
A checker I wrote reported "0 segments, 0 issues" on an empty file — the same exit code as a genuinely clean file.
A pipeline swallowing an exit status, which showed up in two places I had been counting as separate bugs.

The pipeline one, and the correction I owe

I had this filed as two findings: "timeout is missing on macOS and fails silently" and "I read an exit status through a pipe". The first half of that was wrong.

$ timeout 5 echo hi
zsh: command not found: timeout
$ echo $?
127                      # ← correct. A missing command does report failure.

$ timeout 5 echo hi | head -1
zsh: command not found: timeout
$ echo $?
0                        # ← the pipeline reports head's status, not timeout's

A missing command exits 127 on its own. What swallowed it was the pipe: by default a shell pipeline reports the exit status of the last command. timeout was not a silent failure, it was the thing that made the pipe's behaviour visible. Two findings, one bug.

set -o pipefail makes the pipeline return non-zero if any command in it fails — same commands as above, 0 becomes 127. It is worth turning on where you are branching on a pipeline's status, which in an unattended script is usually.

What I check now

An unreadable or empty input never exits 0. It gets its own code, distinct from "scanned and found nothing".
Success is not "exit code 0". It is "the thing I expected to be produced exists and has content".
On macOS, check that timeout exists before a script depends on it. Same for anything else assumed to be standard.
If a status matters, do not read it through a pipe without pipefail.

None of those is the part worth passing on. I found one instance, fixed it, and did not go looking for others. The three above came from someone reading the same code who had not just solved it.

This article was written with the help of AI. The incident, the commands and the outputs are real and from my own machine; the draft was AI-written, then checked against the original logs and corrected — including the timeout explanation above, which the first draft got wrong.

DEV Community