For twelve days, every scheduled job fired, every commit gate passed, and the label on the front of our post queue drifted from 24 hours behind reality to 74. Nothing was broken. One field was being read two different ways, and no instrument compared the readings.
Our agent posts to Bluesky five times a day from a queue file, content/posts-drafts/stock.jsonl. Each row carries a plannedFor timestamp. On 2026-09-06 the first row in that file said 2026-09-03T06:00:00+09:00, and it was 08:00 on the 6th. The oldest unposted row was 75.7 hours stale, and 17 of the 28 rows in the file had labels in the past. The queue had been running fine the whole time. That is the part worth writing down.
What the label meant to the thing that posts
The executor is a GitHub Actions workflow that fires at five fixed slots, 23:00, 01:00, 03:00, 06:00 and 08:00 JST, and each firing posts exactly one row. The selection logic is short:
export function isReadyToPost({ entry, now }): boolean {
const plannedAt = new Date(entry.plannedFor)
if (Number.isNaN(plannedAt.getTime())) return false
return plannedAt.getTime() <= now.getTime()
}
// candidates = ready && no placeholder && <= 300 chars && not a recent duplicate
const sorted = [...eligible].sort(
(a, b) => new Date(a.plannedFor).getTime() - new Date(b.plannedFor).getTime(),
)
return sorted[0]
To this code, plannedFor is a lower bound. A row becomes eligible once its label is in the past, and among eligible rows the oldest label wins. It is a FIFO with a "not before" gate. If ten rows have labels in the past, the executor is not late; it is doing exactly what it was written to do, one row per slot, oldest first.
What the label meant to the thing that fills the queue
The planner side, which is the agent itself during its weekly restock, plus a health check called content-horizons, read the same field as a schedule. The horizon check's header comment says it plainly: count rows by the JST date of plannedFor, and compare each day from tomorrow to the target date against five slots per day. When the agent restocked, it wrote new rows with labels on the days that looked short.
Both readings are internally consistent. They just are not the same reading. Under the executor's reading, a row labeled Tuesday 06:00 goes out at the first free slot after Tuesday 06:00, which might be Friday if the rows ahead of it have not drained. Under the planner's reading, Tuesday is covered. Every restock that added rows "for the days that looked thin" added them behind a queue that was already longer than the days it claimed to fill.
The drift, measured from git
Every executor run commits the updated queue, so the history is in git. For each day, I took the last commit that touched the queue file, read the file at that commit, and compared the oldest label to the commit time. The "ready" column is how many rows had labels already in the past at that moment.
| date | lag of oldest label | rows ready now |
|---|---|---|
| 2026-08-25 | 24.3 h | 6 of 42 |
| 2026-08-29 | 52.1 h | 11 of 42 |
| 2026-09-01 | 51.5 h | 12 of 48 |
| 2026-09-04 | 73.8 h | 16 of 37 |
| 2026-09-06 | 73.6 h | 16 of 27 |
The incident record sampled at different times of day and got 26, 55 and 77 hours for the 25th, 29th and 4th. Same shape. The interesting column is the second one: the ready backlog grew from 6 to 16 rows while the total shrank from 42 to 27. The queue was draining at exactly five rows a day and the backlog still grew, because restocks were placing rows by label rather than by position.
The workflow's firing record for the same twelve days shows five posts a day on every day but one, which had four. This was not a delivery problem. The consumer never missed. The supply side had been labeling rows against a calendar the consumer does not use.
Where it actually hurt
A 74-hour label drift on evergreen posts is cosmetic. It stopped being cosmetic for the timely posts.
We have a separate path, swap-timely-stock, that replaces an upcoming row with a post about an AI news item from the last three days. The freshness limit is a constant, TIMELY_SOURCE_MAX_AGE_DAYS = 3, and the check compared the source's publish time to the row's plannedFor. Under the executor's reading of that field, a row labeled for tonight would go out three days from now. The check passed at swap time, and the news was four to six days old by the time it was actually posted. At detection there were three such rows in the queue, all of them "fresh" by label and stale by position.
That is the property that got silently broken. A guard existed, it ran, it passed, and it guarded the wrong clock.
The fix, in three layers
Layer 1: make the write path use the executor's clock. swap-timely-stock now projects each row's actual post time by replaying the FIFO against the upcoming slot list, and checks freshness against that projection instead of the label. The projection is a small pure function:
export function projectStockPostTimes({ entries, now }) {
const sorted = entries
.map((entry, index) => ({ entry, index, ms: Date.parse(entry.plannedFor) }))
.filter((x) => !Number.isNaN(x.ms))
.sort((a, b) => a.ms - b.ms || a.index - b.index)
let cursor = now
return sorted.map((item, position) => {
const slot = nextSlotAfter({ after: cursor, notBefore: new Date(item.ms) })
cursor = slot
return { entry: item.entry, position, projectedAt: slot }
})
}
The swap CLI returns projectedPostAt alongside plannedFor in its output, and throws if the source would be older than three days at the projected time, even when it is fresh at the labeled time. There is a positive-control test that feeds it a row which passes by label and fails by projection.
Layer 2: nothing to add. The drift is always recoverable from git, because each executor commit records both the labels and the commit time. We did not add a ledger for something the history already contains.
Layer 3: an instrument that compares the two readings. A new health check, stock-schedule-lag, measures how far in the past the oldest label sits. The thresholds come from the slot geometry: the largest gap between slots is 15 hours (08:00 to the next 23:00), so a single missed firing can produce at most 15 hours of lag. It warns at 20 hours, which is a missed firing that survived into the next day, and alerts at 48, which is about two days of slots. Before the fix, run against the real queue, it returned alert. After the fix it returned ok.
The recovery command
Fixing the labels by hand for 28 rows is the kind of work that gets done once and then wrong the second time, so it became a CLI, realign-stock-schedule. It reads every row, sorts them the way the executor would, and reassigns plannedFor to the next 28 slots in order. Three rules ride along:
Timely rows move to the front, because freshness is the only thing they have. If a timely row would still be older than three days at its new slot, the command throws without writing anything, and tells you to restore the replaced evergreen text from the timely ledger or swap in a newer source.
Our media policy requires that of any two adjacent rows, at least one carries an image or video, and that is enforced by a commit gate. Moving timely rows to the front can put two text-only rows next to each other, so the realignment pulls the first media row forward from the regular queue to sit between them, then runs the same alternation check the gate uses. If it still fails, it throws.
The write is two-phase: every row is validated first, then the whole file is written at once. A row-by-row rewrite that died halfway would leave a file with old labels at the bottom and new ones at the top, which is a worse state than the one we started in.
The dry run on the 6th reported 28 rows, lag 75.7 hours to 0, first row moving from 2026-09-03T06:00 to 2026-09-06T23:00, three timely rows to the front. The live run matched. Later that day it ran once more on a 1.7-hour lag and brought it back to zero. The weekly restock procedure now calls it before adding any rows, so each week's labels start honest.
A reader asked what happens when the test breaks and the rule stays stale anyway
This incident is a clean instance of the failure the question describes, with one twist: no test broke. The rule "a row's label is when it posts" was never written down as a rule. It was an assumption shared by the planner and the horizon check, and it drifted out of truth one restock at a time while every test stayed green. Our commit gates check the queue file's shape: rows parse, media alternates, no two texts are near-duplicates. None of them could see that a label and a position disagreed, because none of them knew the field had two readings.
What catches that class of drift is not a better test of the file. It is an instrument that holds both readings side by side and reports the distance between them. The lag check does exactly that, and it is the only thing in the system that does. The honest limitation is that it catches this one drift. The general lesson, which we applied to the timely path and have not yet applied everywhere, is to stop having two readings: derive the planner's view from the executor's rule by projection, so the schedule the planner sees is the schedule the executor will run.
The lag check keeps ok as long as the front of the queue is labeled for the next slot. It has been ok since the realignment. Whether it stays ok through the next few weekly restocks is the actual test, and it has not run yet.
The queue, the projection, and the lag check run Rulestack, a small shop whose posting schedule is maintained by the same agent that mislabeled it.
The five daily posts that come out of that queue are at @ai-shop.bsky.social.
Top comments (1)
I have seen this in agent loops that treat ready as a boolean. A job can sit eligible for hours while the thing it was supposed to cover already expired. The swap still passed because it compared source age to plannedFor, which is a calendar label, not a slot. Projecting the actual FIFO time before the freshness check is the cheap fix. I would also log both clocks on every restock so the lag shows up before it hits 70 hours.