A call for --help got out of hand

I leave small maintenance tasks for AI coding assistants to work on while I sleep. A program reads the waiting tasks and starts assistant sessions to read code, make changes and run checks. Their work is left for me to review in the morning. I've described the full overnight arrangement before, but this story concerns how that program gets started each night.

On my Mac, the operating system's service manager, launchd, starts that program on a schedule. A small installation script writes the configuration that tells launchd what to run and when. Re-running that script applies the configuration by removing the old scheduled job and registering it again. If the second step fails, the configuration file can still exist while the job is no longer registered to run.

On 12 September, one of the overnight assistants was reading the code for this setup. It wanted to understand the installation script. At 03:50, it asked for help. With the private directory removed, the command was:

sh bootstrap/install-queue-drain.sh --help 2>/dev/null | head -3

Command-line programs usually treat --help as a request for usage instructions. This script didn't handle arguments at all, so it ignored the question and ran its normal installation steps.

An assistant trying to read about its own setup had just reconfigured the job that started the night shift. That job was subsequently found missing from launchd. The investigation identified the still-running overnight process as the likely obstacle to registering the job again. The error output was discarded, so the precise reason the reload failed remains unconfirmed.

That discarded output is the part I keep coming back to.

The installer already contained a check for whether launchd had accepted the job. If the check found it missing, the script was written to refuse success, give a command for investigating and exit with a failure status. I would have been pleased to see that error handling in a review.

But the assistant's command included 2>/dev/null. Command-line programs have a separate output stream for errors, called stderr; that redirection throws it away. The | passes ordinary output to head -3, which keeps only the first three lines. The assistant had asked for a short preview, with error messages excluded.

Any complaint from the attempt to register the job, or from the installer's own check, would disappear before the assistant could read it. I can't say which complaint ran: the transcript never received it. The script contained a useful error path, and its caller had removed the channel intended to explain the failure.

All eighteen installation scripts in that directory ignored arguments. They were careful in other respects, including explicit paths and installation checks. Asking for help also looked like an ordinary way to learn about an unfamiliar program. The failure sat between what the caller expected and what the script actually did, with the explanation filtered out on the way back.

I eventually found the missing overnight reports while asking whether the assistants were hitting their session limits. I was trying to understand how much work they could do. The program that would have started them wasn't being started itself.

A separate nightly health check should have helped here. It looked for expected scheduled jobs and complained if any were missing. Its list contained eleven jobs. I also had a daytime run that worked through the same task list, but neither that job nor the overnight one was on the health check's list. The check could find everything it expected while the night shift was absent.

So I had two opportunities to learn what happened: the installation command's error report and a separate check of the resulting state. The first was filtered out. The second didn't know to look.

"It logs an error" now feels like an unfinished answer to me. Failure handling that depends on the caller happening to listen has left part of the system unbuilt. The caller's output limits and redirections belong in the review alongside the message the script prints.

- The caller is part of the error path.

There is a small, harmless way to see how a caller can lose a failure. This example needs Bash and the standard head command. Its pretend installer only prints messages and returns a failure status; it writes no files and touches no scheduled jobs. Paste the whole block into a terminal:

bash --noprofile --norc <<'SH'
set +o pipefail
install_demo() {
  printf '%s\n' 'configuration written'
  printf '%s\n' 'job NOT loaded' >&2
  return 1
}

install_demo 2>/dev/null | head -3
printf 'pipeline status: %s\n' "$?"

if captured=$(install_demo 2>&1); then
  printf '%s\n' "$captured" | head -3
else
  rc=$?
  printf 'installer status: %s\n%s\n' "$rc" "$captured"
fi
SH

An exit status is the number a command returns: 0 conventionally means success, and a nonzero value means failure. The shell keeps the most recent status in $?. With pipefail disabled here, a pipeline (commands joined by |) reports the last command's status. So the first call prints configuration written and reports status 0: head succeeded, even though the pretend installer failed.

The second call captures both output streams and checks the function's own status before shortening anything. It reports status 1 and includes job NOT loaded. An unattended caller would still need to put that failure somewhere its operator checks. Preserving it makes that possible. These are the example's shell settings; the historical caller's settings aren't established. Enabling pipefail can preserve a failure status, but it cannot recover error text already thrown away.

The repair to my installation scripts was small. A shared argument check now answers help requests before installation starts and rejects unknown arguments. A test asks every installer for help, then compares configuration-file sizes and timestamps and the set of registered jobs before and afterwards. The health check now includes the overnight and daytime jobs.

A manually maintained list can still miss the next job, though. And being registered to run doesn't establish that a program finished its work. For these unattended runs, I want a check that asks whether the expected result arrived on time, running somewhere other than inside the worker whose silence it needs to notice.

I had put care into explaining failure. I hadn't followed the explanation all the way to the person who needed it.

Comments

[Raknaos]

The mechanism that bit you is generic, and worse than it looks: --help is the one command every caller is guaranteed to try, so an installer that ignores arguments has its most-tested path be its most destructive one. Your fix has the right shape — a shared argument check that answers help before touching state, plus a test that asks every installer for help and then compares config sizes, timestamps and registered jobs. That last part is the one I'd copy verbatim; "help must be side-effect free" is easy to agree with and almost never asserted anywhere.

The stderr detail is the other half of the story. 2>/dev/null | head -3 makes a failure undetectable twice over: with pipefail off the pipeline reports head's status, and the text that explains why is gone before anyone can read it. Two questions: do those scripts now write their complaint somewhere you actually look, rather than just exiting non-zero, and has the health check moved from "is the job registered" to "did the expected output land by time T"?

[Alkis Yuv]

Yes, failures go to a chat channel I can read on my phone. I still managed to ignore 32 alerts from one mirroring job over four days. I only found out it was broken when I asked about something else.

I've since added a job that reads the message archive and groups recurring failures, ignoring changing timestamps and counts. After the third occurrence it adds a task to the queue the overnight assistants work through. An assistant can then investigate it the next morning, with the usual review. The archive reader only counts and files tasks. It does not diagnose anything or execute instructions from the messages, since those contain arbitrary job output.

On checking whether the work actually finished by a deadline: I haven't built that for the night shift yet. The current check reads each job's last exit status from the scheduler. I added it after finding the worker repeatedly crashing and restarting while the list of registered jobs looked fine.

That already needed a couple of exceptions. The drift checker returns a nonzero status when it finds drift, so it kept reporting itself as failed. A long-running job also reports a termination signal after a normal restart. I had to account for those before the results were useful.

I do check the result directly for my site. A probe runs every five minutes and requests each host from outside the server. It checks a neutral address first to establish that the machine running the probe has a working connection. Two consecutive failures trigger an alert; I get another when the site recovers.

For the overnight work, I'm thinking of checking whether tasks added before a cutoff are still untouched the next morning. An empty queue should pass (some nights there is nothing to do). That would tell me more than whether the worker ran, though it still would not prove that a task was completed properly. So far it's just an idea.

And yes, the installer test should be easy to copy. It records the size and modification time of every scheduled job file, along with the registered job labels. Then it calls all eighteen installers with --help and checks that none of those changed. It also checks that unknown flags are rejected.