DEV Community

Cover image for The night the queue ran dry at 1:40
aft3r_d0t
aft3r_d0t

Posted on Originally published at thenightlycrew.com AI-assisted

The night the queue ran dry at 1:40

Here's the rule everything else hangs on: the one who builds a thing never gets to say it's done.

Every piece of work in our house goes through three hands. One writes the spec: what "done" means, in clauses you can check. One builds against it. Then the spec's author returns as the acceptor and measures the result herself, with her own commands, not the builder's word. A builder can be brilliant and quietly wrong; acceptance catches that.

A good acceptance doesn't stop at "the tests pass". It breaks the code on purpose to see whether the tests notice. On one recent feature the acceptor planted ten small bugs in a copy of the code — a dropped check, a flipped reply — and the suite caught all ten. One survivor would have sent the feature back.

Acceptance started on the night machinery itself, before it ran, and found three defects in its own spec — one the "now ≥ 08:00" clause, which read literally would end every night begun the evening before, because a time of day is not a deadline. The suite grew from 43 to 68 as they were fixed.

The ask

In early October I wrote down something simple: stop choosing the night's work by hand every evening. Keep one standing backlog for all projects, give the night a signal that I'm asleep, and let it work until morning. One hard condition: at 08:00, the usage window still open must be at least half free.

The rest of the design followed. One backlog file; every item carries a project, a status, a stops-at line naming the last step we may take without a human, and who added it. The status vocabulary is closed: proposed, ready, in-flight, parked, done, dropped. One rule matters most: only the acceptor may move an item to ready. The runner picks nothing else — that is why a night is safe, and shallow.

There were two triggers: a command on the command line, and a physical tag by the bed. The tag path worked — 112 scans — but the last was in September and none in the 23 days after, so it had quietly left my routine. A tag daemon also kills a bound command after 30 seconds, so the command had to stand on its own. The supervisor runs detached, locked by a file lock in Python.

Each tick is one item, end to end: a spec check, one build, acceptance, a status update, a line in the morning report. Two failed ticks in a row end the night, and a tick that stops to ask for approval counts as failed, not as something to retry quietly.

The budget gate

The runner refuses to start work without knowing what it costs, and a budget gate is only as honest as its gauge. On the evening of the first run there was no gauge at all: the usage script returned "status: unknown", the proxy console an empty five-hour window, and a direct endpoint call failed with a 401.

The cause turned out to be duller than a bug. The script preferred any credentials file on disk, and that one was stale — last written in early September, its token long expired — while the live token sat in the keychain. The 401 came from the dead token, and so did the later rate limit with its long Retry-After. A keychain call answered 200 at once. Two answers, "unknown" and "empty", were one problem in different clothes.

What shipped is a single poller: one job writes one gauge file, and it alone may call the usage endpoint on a schedule. One call per cycle; after a rate limit it waits for the server's Retry-After, otherwise it doubles the wait. Its first run answered 200 with small numbers; the parser read "GO".

The handover between gauge and runner is where the night stumbled. The runner started just after midnight with the variable pointing at a missing gauge file, and its first gate said HOLD: gauge unknown. Restarted with the variable set, it read GO. The report that survives opens at 01:04 — my start, not a fault. One gauge detail stayed unresolved: all three gates read the same numbers, and the reading aged — 30 minutes, then 44, then 58 — though the poller should have refreshed every few minutes.

What happened

Three items were ready. All three ran, exit 0, all ending ready→done, all accepted on the first pass, in 13, 14 and 8 minutes. One fixed a reminder banner that forgot it had been dismissed after a reload. Another packaged a mobile release path and only ran it as a dry run: it reported success and produced an unsigned archive — exactly what a dry run is for.

The report's "waiting on a human" section held two more lines, both fences working: a deployment waiting for a date, and a real upload needing the paid developer team signed in. The night's last line said: 01:40 — queue empty. The deadline was 08:00, and the budget nowhere near a stop row.

The night ended because there was nothing left to do — the design working as written. "Only the acceptor makes things ready" exists so nothing unvetted runs while nobody is watching. The cost: a night goes only as deep as the vetting done in daylight. A gate built to stop a night running too hot never acted at all.

What changed afterwards

The next morning I wrote down: the backlog needs to be thought about much further ahead — research, gathering and writing tasks too, maybe even a book — so that when one task blocks there is something to switch to. Then I narrowed it: an idea to think through together, not an order. Nothing new was opened; by ten I looked at the result and said keep it as it was.

The backlog side came first. Before that pass, ready stood at zero; after it, thirty-five items: three proposed, twenty-two ready, four parked, six done. The ready ones sat in four lanes — five builds, eight research, four gathering, five writing — interleaved so no two neighbours share a project, which a script checked: zero adjacent pairs. A floor of fifteen ready items became the warning line. The non-code items carry their own acceptance floor: sources, recorded statuses, an explicit Unknown list, no decisions taken for me.

The runner side got the fields that make that possible: optional lane, predecessor and builder per item; a lint that errors on a bad lane or on a predecessor naming a missing item; one function shared by the supervisor, the lint and the dry-run, so the three cannot disagree about what is ready. Then the switch rule: a project-wide block — red CI, a dirty tree, an unreachable host, expired auth — parks every ready item of that project in one tick with one waiting line, so the night moves on instead of failing twice and stopping. The suite went from 68 to 90 tests, and the gate stayed byte-for-byte unchanged, checked by comparing syntax trees.

The prediction that didn't hold

At twelve minutes a tick, twenty-two items is roughly four and a half hours; research and writing run longer. After the refill I expected budget, not the queue, to end most nights. The nights that followed all ended like the first: 22 ready and 23 ticks, empty at 05:36; 25 ready and 25 ticks, empty at 04:19; 19 ready and 19 ticks; 31 ready and 31 ticks. Not one stopped on the budget.

A prediction isn't a measurement, and the measurement said the queue was still the wall — the refill bought depth, not immunity. A deeper queue ends the same way, only later.

Top comments (0)