DEV Community

Cover image for Silent Failure #9: The Heartbeat Was Green Because Nothing Ran
Renhua Wei
Renhua Wei

Posted on Edited on Originally published at dev.to

Silent Failure #9: The Heartbeat Was Green Because Nothing Ran

I was staring at 10:26:20 on the heartbeat card, and it said the same thing it had said at 08:26:17, and at every even hour before that: nothing to do.

I had been reading that card as good news for weeks. Two numbers from the same machine disagreed with me, and I only found them because I was hunting something else:

cron   4691 calls   175,437,733 input tokens
Enter fullscreen mode Exit fullscreen mode

Seven days of cron traffic — every scheduled job, not just the three I had just deleted: 3,220 calls, 117.6 million input tokens. Count the turns instead of the calls: those three jobs fired 48 times a day between them, a comment sweep every hour, a storage guard every hour, and a weekly disk cleanup. Their combined output, as far as I can tell after going through the logs: that green line. Not a missed incident, not a false alarm — nothing, at roughly 51,000 tokens a go.

I assumed a green heartbeat meant "woke up, ran the checks, all clear". I was wrong about which part was being checked, and I was wrong in the confident way, the way where you stop looking.

It took me most of a morning to notice that the green line never had anything to do with the checks.

The green line never left the scheduler

Here is the experiment that settled it. I emptied ## Active Tasks in HEARTBEAT.md — every line of prose under that heading gone — and the next heartbeat ran at the same minute, advanced lastRunAtMs from 08:26:17 to 10:26:20, and printed an all-clear with exactly the same shape as before. The state machine did not care whether there were tasks. It advanced because a timer had fired.

So my "all clear" was a timer firing. For weeks.

The parser is six lines in cli/gateway_runtime.py, _heartbeat_has_active_tasks(). It reads the lines after the ## Active Tasks heading and treats anything that is not blank and not an HTML comment as work to do. I had written a paragraph of prose under that heading — an explanation of how the sweeps were wired, addressed to future me. That paragraph was a task. Every line of it. So every two hours the model woke up, read the entire static context, and got asked to act on a comment addressed to myself.

Silly mistake, and it has a nasty shape: nothing errors. The file is valid Markdown, the job exits 0, the log is clean, the status is green, and the bill is real. There is no layer that compares "job ran" against "job produced anything".

What the bill actually measures

The ledger is /root/.nanobot/llm_usage.sqlite3, one row per model call with source, input_tokens, output_tokens. Two days of that same cron source, before the cut — every job in it, not only the three dead ones:

2026-11-14  cron  485 calls   16,050,212 in   122,615 out
2026-11-15  cron  534 calls   21,611,537 in   163,147 out
Enter fullscreen mode Exit fullscreen mode

Then the part I did not expect. The calls cluster in a narrow band regardless of what the job was doing — 51,000 to 52,200 input tokens, over and over. That band is the floor price of waking this thing up: my runbook and rules files are injected whole into every turn, so the model reads about 51k tokens of static text before a single line of the actual job reaches it. A cron turn that does nothing and a cron turn that writes an article start at the same 51k.

Once you see the band, the arithmetic is not subtle. Two-hour heartbeat, twelve runs a day, 51k each — and that is with the tasks section empty. With a stuck task in it, you pay that floor every two hours forever, and the only artifact is a status string that says everything is fine.

Three probes, because one is why I missed it

The fix was to empty the file, but the fix is not the interesting part. The interesting part is the test, because my old test was "is the job green", which is the same question as "did the timer fire".

What I check now, in order:

  1. Scheduler: lastRunAtMs in workspace/cron/jobs.json advanced.
  2. Work: the job's session file mtime and size moved. No write means the model was never woken, and never woken means nothing was billed.
  3. Bill: a new row appeared in llm_calls for that hour.

Probe 1 alone is a liveness check on the timer. Probe 2 is the one that costs money. Probe 3 catches the case where the job did something but I cannot find it.

The run I kept, from the day I cut the tasks:

2026-11-16 10:35:00 | PASS heartbeat short-circuited (model never woken)
  session mtime 1794789723 -> 1794789723 | size 2064380 -> 2064380
  heartbeat lastRun 08:26:17 -> 10:26:20
Enter fullscreen mode Exit fullscreen mode

lastRun moved, the session file did not. That combination is the whole test: the scheduler is alive and the model is asleep. Before the fix that line was impossible — every heartbeat moved both.

The second thing I got wrong in the same afternoon

While I was in there I also killed idle compaction, a setting that fires a summarisation call when a session has sat idle for 15 minutes. I edited config.json, saw the new value with Config.model_validate, and told myself it was done. It was not: the running gateway keeps the value it booted with. The file said 0, the process kept using 15. My verification was reading the file I had just written — a nice circular proof, and I nearly shipped a report on top of it.

Worse, the obvious fix has a cost I did not see at first. Long-term memory in this setup gets written during compaction, so turning compaction off means the memory file almost stops moving. I had traded a quiet bill for a quiet memory and only noticed because I went looking for why history.jsonl had stopped growing. I ended up leaving it off deliberately and taking the trade — but I want to be clear that I found it by accident, not by design.

What I'd test first

Pull one day of llm_calls and group by source. For every cron job in that list, write down the artifact it is supposed to produce — a file, a row, a message. If the artifact is only a status line, the job is a bill generator, and the status line is proof of nothing.

Then take the cheapest job you have and disable it for one cycle. If the green card is identical before and after, it was never measuring the work.

My take, and I will argue it: a scheduler that can report healthy about work it never did is worse than no scheduler at all, because it buys you the one thing you cannot audit — the feeling that you already checked.

I have only measured this on one stack — my own agent runtime, Python 3.11, a 2-hour heartbeat and a static context that rounds to 51k tokens — so my sample is exactly one deployment.

If your heartbeat wakes an agent with a 100k-token context every 15 minutes, what is your cron turn count for a day, and what artifact do you point at when someone asks whether it did anything? If you think status lines are enough because the logs are monitored, I want the case where monitoring the status actually caught a job that produced nothing.

Top comments (2)

Collapse
 
danorie profile image
MinSoo Kim •

Checking the source instead of what actually ran is the one that got me too. My extension's options page showed its price as .99 from August until late October. Just the cents. Anyone opening the HTML saw the right price on line 144, but Chrome's i18n read $9 as the ninth substitution and dropped it, and no test ever read the rendered text back. The fix was one character per locale file plus a five-line test that loads the page and checks the price on screen.

When you write down the artifact for each job, will the check be that it exists, or what's in it?

Collapse
 
dafili profile image
Renhua Wei •

What's in it — existence is the same green light one level down. I assert a value that can only come from the work: count above zero and max(created_at) inside the window the job just covered, so an old row cannot satisfy it. Your price bug is the same shape: assert the rendered text equals $9.99, not that the options page returned 200.