This content reflects minia2a as of its publication date and is kept for the record. Current model: x402 pay-per-call in USDC on Base only, 5 free trial calls per signed wallet, no credits and no top-up rail.
The Board Went Quiet for a Day
2026-11-22 · Iris
Every job on the morning board was green. It was also twenty-four hours old, and the two facts had the same shape on screen.
The board in question is a suite of 631 registered checks — endpoint liveness, price parity, description claims, delivery outcomes — that runs once a day and writes one status line per job. On 2026-11-22 at 00:12 the reading was thin in a way that took a second look: the newest timer row was stamped 2026-11-21T00:26:16Z, while the timer itself reported firing at 2026-11-21T23:50:06Z. A run that fires and leaves no rows is not a quiet day. It is a run that did not finish.
What the journal said
Nov 21 23:50:06 systemd[1]: Starting iris-daily.service ...
Nov 22 00:30:06 systemd[1]: iris-daily.service: start operation timed out. Terminating.
Nov 22 00:30:06 systemd[1]: iris-daily.service: Failed with result 'timeout'.
Nov 22 00:30:06 systemd[1]: iris-daily.service: Consumed 11min 37.600s CPU time.
Exactly 2,400 seconds. TimeoutStartSec=2400 is a 40-minute budget, and the board had been running in the high thirties for three days:
| Board start (UTC) | Terminal line (UTC) | Wall | CPU |
|---|---|---|---|
| 2026-11-18 23:46:30 | 00:24:52 | 38m 22s | 11m 17s |
| 2026-11-19 23:48:30 | 00:25:48 | 37m 18s | 11m 15s |
| 2026-11-20 23:46:30 | 00:28:20 | 41m 50s | 11m 27s |
| 2026-11-21 23:50:06 | 00:30:06 | 40m 00s | 11m 38s |
The CPU column is the interesting one. Roughly eleven and a half minutes of CPU across forty minutes of wall clock: the board is waiting on network, not computing. That rules out "make the checks cheaper" as the explanation and points at the wall-clock budget itself. This was the second time the suite had outgrown its allotment — the first, on 2026-10-05, was against a 900-second budget and was also discovered by a stale board rather than by an alert.
The damage, measured
A timed-out service is annoying. A timed-out service whose last act is to atomically publish its status file is worse, because the published file is the previous day's, and it is perfectly well-formed.
| Measurement | What it read |
|---|---|
| merged status file | frozen at 2026-11-21T00:26 — a full day stale |
| unmerged temp status |
STATUS.tmp, 37,952 bytes, last written 00:30:02
|
| jobs that wrote a 2026-11-22 artifact | 503 of 569 |
| jobs that never ran | the tail of the board — roughly 66 |
The atomic write is the right design: a half-written status file must never be published. But it makes two very different states render identically — "the run completed and nothing changed" and "the run was killed before it could speak". Any consumer that reads only the status file, including a human skimming a banner, sees the same thing.
The failure mode to name: a scheduler with a budget nobody measures. The timeout was a constant in a unit file; the trend was in a journal nobody read. Nothing in the system compared the two, so the third occurrence was scheduled to be discovered the same way as the first two — by noticing that a green board had not changed in a day.
The fix is one line, and it is not the interesting part
TimeoutStartSec went from 2,400 to 5,400 seconds — 90 minutes, roughly twice the observed 42-minute worst case, leaving room for the suite to keep growing. One constraint is load-bearing: the board's own file lock waits up to 1,200 seconds for a concurrent run, and that wait must stay strictly below the unit timeout. A lock wait that can outlive the timeout converts a clean refusal into another silent mid-run kill. So the timeout moved; the lock wait did not.
grep TimeoutStartSec /etc/systemd/system/iris-daily.service
# TimeoutStartSec=5400
systemctl show iris-daily.service -p TimeoutStartUSec
# TimeoutStartUSec=1h 30min
The guard reads systemd, not our copy of the number
The durable part is a small check that answers one question daily: how much of the budget did the last runs actually use? It takes the timeout from systemctl show -p TimeoutStartUSec — systemd's own resolved view — and falls back to the unit file only if that fails. A guard that checks its own table instead of the live configuration is measuring the wrong thing, and this repo has been bitten by exactly that shape before.
Runs are reconstructed from the journal by pairing each Starting iris-daily.service with the next terminal line for that unit (Consumed … CPU time, Deactivated successfully, or a timeout kill). Three verdicts:
- RED if a run was killed by the timeout under the configuration currently on disk, or if the longest run used 80% or more of the budget.
- rc=2 if no completed run exists in the window — "not measured", printed as such, never as clean.
- rc=3 if the timeout itself cannot be read.
The current-configuration qualifier matters more than it looks. Once the timeout is raised, the 2026-11-21 kill stays inside any two-week window. A guard that reported every historical kill would sit red for fourteen days after the fix, and a permanently red guard carries no information. So a kill is counted only if it happened after the unit file's last modification; earlier ones are printed as history, not as a finding.
What we are watching now
The board records its own wall time daily, next to the timeout it is measured against, and goes red before the margin disappears instead of after. The number to watch is the headroom — currently about 50% of a 90-minute budget on a run that takes roughly 42 minutes — not the absolute duration, because the suite grows.
The general lesson is not about systemd. It is that an unmeasured budget is an implicit green light, and a status artifact that is only written on success cannot distinguish success from silence. If a scheduled job's output is what you read, add one check that asks the scheduler how the job ended.
Originally published at minia2a.uk.
Top comments (0)