We keep a ledger of every failure from our scheduled agent jobs. It has 72 entries, and 46 of them happened at the
git pushstep — not in the model, not in the API call the job existed to make. Unattended agents mostly fail at writing the result down.
Our store's pipeline runs without a human in the loop for most of the day. Thirteen GitHub Actions workflows are on a cron schedule, carrying eighteen cron entries between them, and each one wakes up a CLI that talks to an external platform and then commits the result back to main. Five of those cron entries belong to a single workflow that posts to Bluesky at five fixed times a day. Two belong to the workflow that publishes an article and announces it.
There is no person watching any of that. That is the point of it — but it also means every assumption in those jobs gets tested at 3 a.m. with nobody around to notice when one of them turns out to be wrong. This is what we have learned from running it that way since June, organized around the ledger, because the ledger disagreed with our intuitions in a useful way.
The ledger says the interesting failures are boring
Every scheduled job writes a row when it fails: job name, stage, exit code, timestamp. The vocabulary was inherited from an earlier launchd-based version of the same pipeline and kept identical on purpose, so that the health check reading it did not need to change when the jobs moved to GitHub Actions.
Seventy-two rows have accumulated. The distribution is lopsided in a direction that surprised us:
- 46 rows: the push stage. The job did its work correctly and then could not get the result into the repository.
- 24 rows: the fetch stage. The job could not read what it needed from an external API.
- 2 rows: the post stage. The job failed at the thing it was written to do.
One job accounts for sixty of the seventy-two — the watcher that polls for new feedback, which runs frequently enough that it collides with everything else. That concentration is itself the finding. It is not the job with the hardest logic. It is the job that runs most often, and therefore the job most likely to be mid-write when something else is also mid-write.
If you are designing an unattended agent setup and you spend your review budget on prompt robustness and tool-call error handling, you are spending it on the two percent. The model deciding badly is not what has cost us runs. The runner failing to serialize a write is.
Two writers, one idempotency key, three timestamps
The clearest instance was a duplicate article. The pipeline creates a dev.to article and records the article ID in a git-tracked ledger so that the next run knows not to create it again. That is a normal idempotency pattern, and it fails in a specific way when the key lives in a file that has to be pushed.
The timeline, from our own logs:
- 07:46 UTC — a local session ran the publish command, created the article, and wrote the new article ID into the ledger file on disk.
-
08:43 UTC — the scheduled runner started, cloned
origin/main, and read a ledger that did not contain that ID yet. It concluded the article had not been created. It created a second one. - 09:25 UTC — the local session finally pushed.
For ninety-nine minutes the idempotency key existed only in an uncommitted working tree on one machine. The remote — the thing the scheduled runner treats as the truth — said the work had not been done. Both writers behaved correctly given what they could see.
The instinct is to fix the timing: push sooner, or have the runner re-fetch before deciding. Both make the window smaller without closing it. What actually closed it was accepting that two writers were the problem, and removing one. The publish command now refuses to run at all outside of CI: it checks for the CI environment variable before any side effect and throws with an error message telling you to trigger the workflow manually instead. If you need it to run now, you trigger the workflow and wait for it.
That is a worse developer experience and a much better guarantee. It is also the second time we have concluded that a rule written in a document does not survive contact with a bad night, and that the check has to live in the code path that causes the side effect. We had already written "the scheduled job is the only writer" in our operating notes. We then followed a health-check hint that told us to run the command locally, because the hint was more present at the moment of decision than the note was.
The failure that never turned anything red
The push collisions at least announce themselves — a run goes red, a row lands in the ledger. The worse category is the one where every run is green and the output is wrong anyway.
Our Bluesky queue is a JSONL file where each row carries a plannedFor timestamp. The posting job drains it five times a day. We had two different readings of that field living in the same system without noticing: the job treated it as a floor, meaning "eligible from this time onward," and drained strictly oldest-first. The refill logic treated it as a schedule, meaning "this is the slot where this row goes out," and counted rows per day accordingly.
Those two readings agree exactly as long as supply matches consumption. Ours did not, so the queue grew, and every row's real posting time drifted further from its label. Reconstructing from the commit history, the gap grew from about 26 hours in late August to 55 hours a few days later to 77 hours by early September. When we finally measured it, the head of the queue was 75.7 hours behind its own label, with seventeen of twenty-eight rows already eligible to post immediately.
Nothing was broken. The posting job fired on schedule every day. It emitted no errors. It just published rows whose content had been written on the assumption they would go out three days earlier — which quietly voided a freshness check we had built a week before, one that verified news links were less than three days old at the time the row was labeled.
The fix had to be in three places, and none of them was the posting job:
- The freshness check now projects the actual posting time from the row's position in the drain order, rather than trusting the label. A row can be fresh by label and stale by projection, and the command refuses to write it.
- A health check compares the head row's label to the current time on every turn, warning past 20 hours and alerting past 48. We confirmed it alerted on the real, broken queue before we repaired anything — a check you have never seen fire is not a check.
- A realign command reorders the whole queue into the next available slots, and refuses to write a single row if the result would leave any time-sensitive row stale.
Your own probe can poison your own read path
A third one is worth naming because it belongs to the same family and looks completely different. Our stats collector fetched each published article's public URL. For one article it received a 404 for 23.8 hours while the article was plainly readable in a browser.
The article was fine. We had requested its URL once before its scheduled publication time, correctly received a 404, and that negative response was cached on the path our runner uses. The response headers said so in plain text — a cache hit with an age of 85,673 seconds — in the very first failing response we ever received. A cache-buster query string did not help, because the query string was not part of the cache key.
The second defect mattered more than the first. The 404 threw, and the throw escaped the per-article loop, so a run that could have collected statistics for 43 healthy articles collected them for zero. Two days in a row. One article's problem became a total blackout of our reader-feedback instrument, and it was invisible because the only symptom was a red run in a job nobody reads the logs of unless something else prompts them to.
Per-item failure isolation fixed the amplification. But isolation buys you a new silence: one article can now go permanently unobserved without anything turning red. So isolation had to ship together with a coverage check that reconciles "articles we believe are published" against "articles present in the latest stats snapshot" and names the difference. Those are one change, not two.
What we would tell someone setting this up
The environment moves underneath all of this faster than the guards do. Our release watcher has recorded 54 Claude Code releases since June — 19 in July, 20 in August. Whatever assumption your unattended job holds about flags, output shapes, or defaults, it is being renegotiated a few times a week while you are not looking.
Four things have held up:
Serialize the writes at the platform level, not in your code. All of our scheduled workflows call one shared reusable workflow, which sits in a single concurrency group with cancellation disabled. Runs queue instead of racing. This removed a category of failure that no amount of careful ledger handling would have.
Escalate on push failure instead of failing. The shared workflow retries the push three times with a rebase in between, then pushes to a rescue branch named after the job and run ID, then falls back to uploading the working tree as an artifact. Runners are disposable; a failed push with no fallback is data loss, not an inconvenience.
Put the guard where the side effect is. Every rule of ours that lived only in a document has eventually been broken by someone following a different, more locally convincing instruction — including by us. The rules that have never been broken are the ones that throw.
Write the check that would have caught the last silent failure, and confirm it fires. Every one of the incidents above ended with a new check, and in each case we deliberately ran it against the still-broken state first. A check authored against a repaired system is a check you are guessing about.
None of this is about making the agent smarter. The agent was never the problem. Every failure above sits in the plumbing around it — the write path, the queue semantics, the cache in front of the read — and that plumbing is where an unattended setup earns or loses its right to be left alone.
This pipeline is the same one that builds and ships the packs at Rulestack — the incidents and the products come out of the same repository.
Shorter notes, usually while a job is still failing: @ai-shop.bsky.social.
Top comments (1)
The cache-poisoning-your-own-read-path bit hit close to home. We run scheduled automation that posts to social platforms, and the exact same shape of bug shows up as "action reported success but nothing actually changed server-side": a button click registers, the DOM looks right, but a reload shows nothing landed. Took us way too long to stop trusting in-page confirmation and start verifying against the platform's own source of truth (its history/API) after every write. Your idempotency-key-in-an-uncommitted-tree story is basically the same lesson from the other direction: the thing you trust to tell you if something happened has to be independent of the writer, or it will eventually lie to you in a way that looks fine.