For about a week, our backend was killed and restarted by its watchdog roughly every six hours. No crash. No OOM. Exit code 0. Health checks simply stopped answering, and a few minutes later the process came back and behaved perfectly.
I lost more time than I want to admit treating this as an infrastructure problem. The thing that finally cracked it was noticing that the interval was too regular.
What we thought was happening
The first three restarts looked like bad luck. Container platforms restart things. Memory pressure, a flaky host, a network blip — you shrug and move on.
Then I pulled the full watchdog log instead of the last few lines, and the shape was unmistakable:
07-23 04:38 health FAILED 2/2 — restarting
07-23 10:41 health FAILED 2/2 — restarting
07-23 16:43 health FAILED 2/2 — restarting
07-23 22:45 health FAILED 2/2 — restarting
Six hours and two minutes apart, drifting a minute or two each cycle. Twenty-two of them.
Infrastructure failures are not punctual. Anything that regular is something in your own code that runs on a timer. The drift was the restart latency accumulating — which meant each cycle was being scheduled relative to the previous failure, not to a fixed clock.
That last detail turned out to be the whole story.
Why it re-armed itself
Our reflection scheduler restores its clock from the database on boot: it looks at the last recorded run and schedules the next one six hours after that. Perfectly reasonable — it survives restarts without double-running.
But when a cycle dies mid-flight, the last recorded stage is written at the moment of death. So the next boot schedules the next attempt six hours after the crash — landing on exactly the same code path, with exactly the same data, and dying exactly the same way.
The scheduler was faithfully reproducing the crash on a timer. Every run had the same input because no run ever finished.
The actual bug
Deep in the entity-resolution step:
reflection/engine.py
facts = embed_titles_incremental(missing) # ← not awaited, because it isn't async
embed_titles_incremental walks down into an embedding client that does a synchronous httpx.post with a 30-second timeout, plus time.sleep() between retries. On the event loop.
One tenant had 9,126 entities and a cache holding 8,352 of them. Batch size 10. That is several hundred sequential blocking HTTP calls, back to back, on the only thread that answers health checks.
The loop wasn't deadlocked. It wasn't starved of CPU. It was simply not running — parked inside a blocking socket read while asyncio waited politely for control to come back. From the outside: a live process, an open port, and nothing answering.
The 20-minute asyncio.wait_for we had wrapped around this call did nothing at all. A timeout needs a running event loop to fire, and the loop was the thing that was gone.
The detail that made it permanent
The cache of computed embeddings was written after the entire missing set finished. Killed at minute two of a twenty-minute job, we wrote nothing. Next cycle: same 774 missing entities, same doomed walk.
A partial-progress bug and a self-rescheduling bug on their own are each survivable. Together they build a machine that reproduces its own failure forever.
The fixes
Move the blocking call off the loop. One line, and the irony is that the same function was already wrapped correctly at another call site, complete with a comment explaining why. The reflection path was simply missed when that fix went in.
facts = await asyncio.to_thread(embed_titles_incremental, missing)
Checkpoint in chunks. Process 200 at a time and write the cache after each chunk. A kill now costs at most one chunk, and the backlog shrinks monotonically instead of resetting. This is what actually broke the loop — even if something kills the job again, it can no longer make zero progress.
Sweep for siblings. If one blocking call reached the loop, others did too. We found three more: a bare synchronous vector-store query in the retrieval path, a 600-epoch NumPy computation in a scheduled job, and an unbounded connection acquire inside the health endpoint itself — which is a special kind of unfortunate, since it means the check you rely on to notice trouble is one of the things that can hang.
Make the next one self-documenting. A side thread now watches a heartbeat the loop bumps every second. If it goes quiet for more than ten seconds, the thread writes an all-thread stack dump to disk — before any external watchdog gets around to killing the process.
def _watch(self):
while not self._stop.is_set():
if time.monotonic() - self._last_beat > self.threshold:
faulthandler.dump_traceback(file=self._dump_file, all_threads=True)
time.sleep(1)
That last one is the piece I'd install first if I were doing this again. Everything else was a fix. This is the thing that means the next unexplained hang costs an hour instead of a week.
What I'd tell past me
Regularity is a fingerprint. Infrastructure fails at random. Your own scheduled code fails on a schedule. If the interval between incidents is suspiciously round, stop reading platform metrics and go look at what you run on a timer.
A timeout around a blocking call is decoration. asyncio.wait_for cannot interrupt a synchronous socket read. If the thing you're wrapping isn't yielding to the loop, the timeout is a comment.
Grep for the fix you already made. The correct to_thread wrapper existed in this codebase, with a comment explaining exactly this hazard, at a different call site. When you fix a class of bug, search for every other caller the same day — otherwise you've fixed an instance and left the class.
Disclosure: this write-up was drafted with AI assistance from my own incident notes and production logs, then edited and fact-checked by me before publishing.
Top comments (0)