DEV Community

prodbymarcu
prodbymarcu

Posted on

A watchdog that finds nothing still pays a full agent turn

cover

Every unattended loop I run carries a cost I never priced until last week's audit, and it comes from ticks where nothing changed. Nothing moved on disk, nothing broke, no news landed, and the loop woke a language model anyway to tell me what I already knew. That floor measured 317 million tokens across ten days on my cron board before any gate sat in front of it, and getting it down took two kinds of gate plus one bug hunt.

What the audit counted

I pulled totals out of usage_audit.jsonl for every scheduled card on my board, one ten-day window, no cherry-picking. 57 cards total, split 32 scheduled, 12 paused, 13 completed one-shots that had never been cleaned up. The three loudest cards by token spend:

article loop        18 runs   avg 6.1M tok    110.7M total
feed curator       477 runs   avg 217k tok    103.7M total
release watcher    139 runs   avg 256k tok     35.6M total
Enter fullscreen mode Exit fullscreen mode

The article loop was doing real work every time it ran, so I left it alone per my own quality-over-savings rule. The other two were watchdogs. They check state, they report state, and most ticks there is no state change to report.

The feed curator is the loudest watchdog number on that board, and here is what it did. It ran every fifteen minutes as a full agent, 477 times inside the ten-day window, averaging 217k tokens per run for 103.7 million total, and 100 of those runs ended silent. Nothing gated the wake, nothing gated the report, so the model paid itself for checking state it then described.

board census

Two gates, one for waking and one for reporting

The cheap half is the wake gate: stop asking an agent to do the tick at all. I moved that card to script mode, a shell script under my profile's scripts/ dir, sourcing what it needed from my zprofile, running the curator, reading the stats file, and pushing the feed digest when it kept anything. Its test run printed this exact line, rc 0:

kept=3 new=3 pushed=yes items_local=3
Enter fullscreen mode Exit fullscreen mode

Then I checked the push landed where a reader would see it, raw GitHub copy returning HTTP 200 with size 2255 and three <item> tags, and the repo log carrying 2026-11-08 curate tick 01:12. Script mode moves the tick off the model entirely, since a shell script that pushes or exits has no reasoning step to bill for.

The expensive half is the report gate, and that one needs more than silence between invocations. A monitor gate only fires when its payload differs from the last payload, so the payload has to be built from state that changes when something actually changed. My release-watcher gate digests four things: one line per tracked slug for its review phase, the done-marker list off the staging directory, any pending post line while shipping without a marker, and a ledger line count. No timestamps in the digest, because a timestamp makes every tick unique and that defeats the gate. Two consecutive runs came back byte-identical, so the digest is deterministic, and luck would have shown up as a difference on the second run.

The homelab watcher took the same recipe one step further, because healthy there means nothing worth saying at all. Two runs came back rc 0 with byte-identical empty output, so the agent stays asleep until something breaks, and its next scheduled tick sat at 2026-11-08T08:23. The figure above is that board, counted before any of it got gated.

The bug that made a gated loop look broken

Converting the feed card exposed a quiet-tick bug that no amount of reading would have found. My manual run printed this line, and then nothing else:

feed tick: no new items but files dirty (not pushing): M stats.json
Enter fullscreen mode Exit fullscreen mode

The branch assumed a dirty repo meant work needed doing, and stats.json carries an at timestamp, so it is dirty on every single tick. Every healthy cycle became a logged complaint. Replacing that branch with an unconditional silent exit when kept and new are both zero gave me two consecutive runs at rc 0 with empty stdout, while git status still showed only that one modified stats file. Dirty does not mean interesting, and a gate has to decide which files carry signal.

Same hunt caught a second liar in my own verification code, this one worse because it failed quietly. I was counting feed items through ElementTree with .//item, and once a feed declares a namespace those tags stop matching that path, so the count came back as zero without raising anything, and the remote leg printed a question mark with no number behind it. A verify step that reports "0 items" for a valid feed is a trap any parser can walk you into. Both legs went to grep -c '<item' and the check now fails unless HTTP 200 AND remote count equals local count. Verified with one item in the local file, one item in the raw GitHub copy, live tick rc 0 with empty stdout, bash -n clean.

Restoring a deleted card mints a new id

One more guard is worth naming because it came out of cleanup work. A job killed by mistake cannot be un-killed by re-adding its prompt from memory, and the replacement gets a fresh id even though the old one never comes back. The restore that worked read the prompt straight off the board's backup file, then diffed the new entry against the old: schedule every 20 minutes, model, provider, delivery on success, delivery on failure, workdir, paused state, all identical, 4,352 characters of prompt byte-for-byte the same. Retyping a 4,352 character prompt would have quietly changed what that card does, and I would have shipped a different automation than the one I meant to bring back.

Cleanup itself shipped 17 graveyard cards removed, 13 completed one-shots, 3 stale paused one-shots, and one cloud keepalive for a decision layer I retired, while two watchdogs got gates and one parked card went back into rotation after I found no reason recorded for pausing it. The board ended at 40 cards. Every gate got checked by running it twice back to back, since one clean run says nothing about a payload that might be varying on its own.

What this decides

Price the tick where nothing changed, then multiply it by how often nothing happens. That product is most of your bill before a single prompt gets rewritten, and mine came out at 317,485,418 tokens over ten days across 57 cards, with one ungated every-fifteen-minute watchdog writing 103.7 million of it from 477 runs at 217k each. Gate the model call, gate the report payload, build that payload out of state that changes only when the thing it watches changes, and let two identical consecutive runs be your proof.

How many of your scheduled jobs are still waking a model on cycles where nothing changed, and what did you pay for that last week? If you can answer that one, you have already found your own worst offender.

Top comments (2)

Collapse
 
aiapikey profile image
于侃 •

The idle floor is two numbers multiplied, and the audit splits them: tick frequency times tokens-per-tick. The first one has a gate that costs zero tokens - a checksum or mtime diff on the input, so the model only wakes when something actually changed. At 477 feed-curator runs where most ticks are no-ops, that is hundreds of free LLM calls a week.

The number worth adding to the audit is cost per event: for the release watcher, 35.6M tokens divided by how many real releases it actually caught. If that is six, you are paying about 6M tokens per catch, and it becomes easy to compare against what missing one would have cost.

That one division turns "watchers are wasteful" into a go/no-go per card.

Collapse
 
jes_p_d4f61f7aa2ec405a418 profile image
Dev_PJS •

Your restore-from-backup story is the same lesson I learned about delegation. Delegation got calm for me when the record replaced my memory as the source of truth. If it isn't in the record, I treat it as not decided yet — that one rule ended most of the 'I thought we agreed' loops with my own agent runs.