We run an agent on an interval. It wakes about a dozen times a day, checks whether anything needs doing, and usually the answer is no. That "usually no" turned out to be the whole cost problem, and it took us longer than it should have to see it.
An interval trigger has no idea whether there is work. It fires exactly as often on a dead Tuesday as during a busy hour. So if the first thing your wake path does is the full sweep, you pay full price for every empty tick. We counted six fires in about two hours and twenty minutes, and half produced nothing but a "nothing changed" log line. Each of those still ran the complete orientation: read state, check five external services, recompute derived metrics, refresh performance snapshots. Minutes of work and real API spend to conclude the world was exactly as we left it.
Put a cheap guard in front of the expensive path
Before doing anything real, answer one question as cheaply as you can: has anything plausibly changed since last time?
For us that is a single indexed query against our own activity log.
SELECT run_session, max(ts)
FROM action_log
GROUP BY run_session
ORDER BY max(ts) DESC
LIMIT 1;
If the last run was under twenty minutes ago, we make exactly one external check, the cheapest signal we have that new work exists. If it comes back empty, we write two rows to the log and exit. No scouting, no recomputation, no metrics refresh.
last_run = db.last_activity_ts()
if now - last_run < timedelta(minutes=20):
if cheapest_external_signal() == 0: # one HTTP call
log("monitor", "no-op")
log("skip", "fast-exit, nothing changed")
return
# ... otherwise fall through to the full pass
The guard reads local state first, because local state is free and the network is not. And the tiebreaker should be the single cheapest external call you have rather than a representative sample of your integrations. The goal is a yes or no answer for close to zero cost, not an accurate picture of the world.
Choose the threshold from how fast your input actually accrues
Twenty minutes is not a magic number. It came from watching how our inbound behaves: replies arrive over hours, so two wakes fifteen minutes apart genuinely cannot differ. If your agent watches a build queue that changes every thirty seconds, twenty minutes is absurd. Match the window to the thing you are watching, and write down why you picked it, because the next person to read the code will assume it was arbitrary.
Do not let the guard trust a cached answer it never verified
This one is subtle enough that we shipped it without noticing.
Our agent keeps a human-readable file of things that are blocked: expired sessions, missing permissions, failing integrations. That part is fine. The bug was that the agent started reading that file to decide what to skip. One entry said an API key lacked write scope, which was true on the day it was written. We skipped that integration on every run afterwards on the strength of the note.
When we finally tested it live, the write succeeded on the first attempt. The key had been fine for who knows how long. Nothing had failed, so nothing had corrected the note, so the agent went on believing it.
The rule we settled on is that a block is a fact with a timestamp rather than a permanent state. If a guard is going to skip work, it has to confirm the reason still holds right now, and that confirmation should be cheap enough that there is no excuse to skip it. A note tells humans what happened. It should never be an input to control flow.
# wrong: the note decides
if "devto" in blocked_notes:
skip("devto")
# right: the note informs, the live check decides
if not devto.can_write(): # one API call
note_block("devto", "write scope missing")
skip("devto")
Make state durable per run, not per tick
The other thing that bites scheduled agents is interruption. A cron-triggered process that dies halfway leaves no trace of how far it got, so the next tick starts from zero and redoes work that already succeeded. If any of that work had side effects, they happen twice.
Persist progress at the step level, keyed by run, and have the wake path check for an unfinished run before starting a new one. Resuming a half-finished pass is almost always cheaper than repeating it, and it turns a crash from a correctness problem into a latency problem.
This is also what separates an agent that can ask a human for approval from one that cannot. If approval lives in memory inside the process, the process dying takes the pending decision with it. If approval is durable state the run can be resumed into, someone can answer twenty minutes later from their phone and the work carries on.
Where we landed
Guard the expensive path with the cheapest check that can answer no. Read local state before touching the network. Confirm a block is still true instead of inheriting it from a note. Keep per-run state durable so an interrupted pass resumes.
The agent is no smarter for any of this. The idle case just costs almost nothing now, and for us the idle case is most of them.
Top comments (0)