Detecting Frozen Python Bots with StayPresent's heartbeat()

How to detect a python bot deadlock or frozen loop that crash recovery can't catch, using StayPresent's heartbeat() and heartbeat_timeout.

Detecting Frozen Python Bots with StayPresent's heartbeat()

A crashed bot is easy to detect — the process exits, the exit code is non-zero, restart logic kicks in. A frozen bot is a much harder problem: the process is still technically alive, still shows up as "running" in your platform's dashboard, and yet it's doing absolutely nothing — stuck on a deadlock, an API call with no timeout, or a loop that's silently spinning without making progress. Standard crash detection is structurally blind to this, because from the outside, a frozen process looks identical to a working one. StayPresent's heartbeat() closes that gap.

Table of Contents

1. Why Frozen Processes Are Invisible to Crash Recovery
2. How heartbeat() Works
3. Setting heartbeat_timeout
4. What Happens When a Heartbeat Times Out
5. Where to Place heartbeat() Calls
6. Choosing a Timeout Value
7. Full Example
8. Best Practices
9. Common Mistakes
10. FAQs
11. Conclusion

Why Frozen Processes Are Invisible to Crash Recovery

Standard restart logic — restart_on_crash, max_restarts — only reacts to a process actually exiting with a non-zero code. A deadlocked thread, a socket read with no timeout that never returns, or an infinite loop with a broken exit condition never triggers that path at all, because the process simply never exits. It sits there indefinitely, consuming resources, doing nothing, and no restart is ever triggered — because nothing about a hung process looks different from a healthy one to something only watching for exit codes.

How heartbeat() Works

The fix requires the bot itself to periodically confirm it's actually making progress, not just alive:

# worker.py
import staypresent

while True:
    staypresent.heartbeat()
    do_work()

Each call to heartbeat() tells the supervising StayPresent process "I'm still actively working," resetting an internal timer for that bot. As long as heartbeat() gets called regularly, everything proceeds normally.

Setting heartbeat_timeout

The supervising side sets how long it will tolerate silence before treating the bot as hung:

# app.py
import staypresent

staypresent.run("worker.py", heartbeat_timeout=30)

If 30 seconds pass with no heartbeat() call from worker.py, StayPresent treats it as unhealthy — even though the process technically hasn't crashed or exited on its own.

What Happens When a Heartbeat Times Out

The behavior mirrors a real crash: StayPresent logs the issue, terminates the frozen process, and hands it off to the normal restart logic — restart_on_crash, max_restarts, restart_delay all apply exactly as they would for an actual non-zero exit. From the outside, a hang and a crash are treated as functionally the same failure mode, which is exactly the point: you get one consistent recovery path regardless of how the bot failed.

Where to Place heartbeat() Calls

The right placement depends on your bot's structure:

A simple polling loop — call it once per iteration, ideally near the top so a hang anywhere in that iteration's work still gets caught by the next timeout window:

while True:
    staypresent.heartbeat()
    check_for_new_messages()
    process_queue()
    time.sleep(5)

An event-driven bot (discord.py, Pyrogram) — call it from a periodic background task rather than an event handler, since events might not fire at a predictable interval:

async def heartbeat_task():
    while True:
        staypresent.heartbeat()
        await asyncio.sleep(10)

Choosing a Timeout Value

heartbeat_timeout should comfortably exceed your normal loop iteration time, including realistic worst-case latency (a slow API response, a large batch), but still be short enough to catch a genuine hang in a reasonable window. A bot whose iterations normally take 2–5 seconds might reasonably use heartbeat_timeout=30 — generous enough to absorb occasional slow iterations, tight enough to catch a real freeze well before it goes unnoticed for hours.

Full Example

# worker.py
import staypresent
import time

def do_work():
    # fetch, process, etc.
    pass

while True:
    staypresent.heartbeat()
    do_work()
    time.sleep(5)

# main.py
import staypresent

staypresent.web.json({"status": "running"})

staypresent.run(
    "worker.py",
    heartbeat_timeout=30,
    restart_on_crash=True,
    max_restarts=5,
)

Best Practices

• Place heartbeat() before the actual work in each loop iteration, not after — this way a hang during the work still gets caught, rather than only a hang between iterations.
• Set heartbeat_timeout based on measured real-world iteration time, not a guess — log your loop's actual duration for a while before picking a number.
• Combine heartbeat monitoring with restart_on_crash rather than relying on either alone — they cover genuinely different failure modes.

Common Mistakes

• Placing heartbeat() inside a try/except that could swallow the exception preventing it from ever being reached. If an exception happens before the heartbeat call in a given iteration, and it's silently caught elsewhere, the bot can look "hung" from StayPresent's perspective when it's actually stuck in a retry loop — worth checking your exception handling doesn't accidentally mask this.
• Setting heartbeat_timeout too aggressively low, causing false-positive restarts during legitimately slow (but not actually hung) operations, like a large batch job or a slow third-party API.
• Forgetting to call heartbeat() at all and expecting hang detection to work automatically — it's opt-in specifically because StayPresent has no way to know what "still making progress" means for your bot without you telling it.

FAQs

Does heartbeat() need to be called from every bot, or only ones prone to hanging?
Only bots where heartbeat_timeout is set on the run() call need it — if you don't set heartbeat_timeout for a given bot, hang detection simply isn't active for it, and a bare run("bot.py") continues to only monitor for actual crashes.

What if my bot has genuinely variable iteration times?
Set heartbeat_timeout based on the slowest realistic iteration, or call heartbeat() more granularly — e.g. between sub-steps of a long operation, rather than only once per full iteration.

Does this work with multiple bots?
Yes — heartbeat_timeout and heartbeat monitoring apply per bot, so different bots in the same run() call can have different timeout values or no hang detection at all.

Conclusion

A frozen process is a genuinely different failure mode from a crashed one, and standard exit-code-based recovery is structurally incapable of catching it. StayPresent's heartbeat() and heartbeat_timeout close that gap by having the bot itself confirm it's making progress, with a hang handled through the exact same restart pipeline as a real crash.

pip install "staypresent[prod]"