On the morning our daily developer email genuinely cannot go out, nobody would have been able to tell it apart from a morning when nothing was wrong, because both endings looked identical from outside. launchd hands a job two files and takes one number back. The property list that runs our morning sender names a path for ordinary output and a path for errors, and that is the entire conversation. Everything the program has to say about how the run went, it says by choosing a number to finish with. Zero means fine. Anything else means broken, and the scheduler keeps the last one.
Stopping early is not the same as breaking
The sender runs a short sequence: fetch the day's ranked stories from the site, pull the confirmed recipient list, hand the batches to Resend in groups of a hundred, write down what went out. There are several points where it can decide to stop before sending anything, and until we changed it, all of them went through one door: print the reason to the error file, finish with a failure.
Some of those reasons are real faults. The shared secret that guards the internal source is missing. The source answers with something other than a success. The mail key is absent, which is worth shouting about, because the recipient list is untouched and the day is simply gone.
One of them is not a fault. When the recipient query comes back with nobody in it, the correct behaviour is to do nothing and say so plainly. Reported as a failure, it teaches the scheduler that this job breaks every morning, and it fills the one file a person opens when something has gone wrong with lines describing a perfectly ordinary morning. Three of the five lines in that job's error file were not errors. They were the sender saying it had nothing to do.
| Nothing to send | Could not send | |
|---|---|---|
| What causes it | the recipient list comes back with nobody in it | the shared secret or the mail key is absent, or the site does not answer |
| Where the line is written | the ordinary output file | the error file |
| How the run finishes now | success | failure |
| How it finished before | failure | failure |
| What the scheduler keeps | a job that ran and had a quiet day | a job that needs a person |
What we changed
There are two functions now instead of one. The one that reports a fault prints to the error file and finishes with a failure. The one that reports having nothing to do prints to the ordinary output and finishes clean, with the reason written above it in the source: an empty list is a normal state, and exiting as a failure made the scheduler report the job as permanently broken and buried real errors underneath it. The product's register of dated claims states the same thing from the other side, that an empty list exits 0 with no send.
The Standup was forked from Front Wire, so both publications run that same sender under their own morning job, 07:40 for one and 07:30 for the other, and both carry the split. The Standup itself is one developer email a day: tagged releases, security advisories, outages and breaking changes, each entry ending on the consequence for a system you own.
The property list did not change. What changed is which of the two files a quiet morning writes to, and the number the job hands back when it is finished.
Read the morning briefing: The Standup, one email a day at 07:40.
A daily email reported failure on the days it simply had nothing to send, so now only real faults count as failures.
One shipped product, taken apart, once a month. What it does, what it cost to build, what the pipeline behind it looks like, and what the numbers did, read off the repository and the live site, not written from memory. Join the list.
Top comments (0)