DEV Community

Cover image for Discord bot offline a week while systemd said active: a plugin timeout
Christian Anderson
Christian Anderson

Posted on

Discord bot offline a week while systemd said active: a plugin timeout

Discord is how I talk to my homelab's AI agents. It's basically the front door. From 30 September to 7 October 2026 that front door was shut, and everything I had watching it said it was fine.

  • hermes-gateway (the agent's gateway, Hermes Agent) was active (running)
  • my gateway watchdog was green
  • the gateway's own state file said Discord was connected

None of that was true for Discord. This is how a 10 second timeout did a week of damage.

The reboot

Around 02:02 on 30 September the Proxmox node that runs the agent rebooted. That node keeps all its guests on one 5400rpm hard disk, so every boot is an IO storm for a good 15 minutes. I wrote about that separately. The short version is that everything starting up is fighting over one slow spindle, so everything is slow.

The gateway loads its chat platforms as plugins. Each plugin gets a time limit to import and register itself: plugins.load_timeout_seconds, which defaults to 10 seconds. On a calm boot that's loads. In the middle of the storm, the Discord plugin's import didn't make it.

What the logs said

When I dug into it on 7 October, the logs were clear enough:

Real log lines: the discord-platform plugin load timed out after 10s, then the gateway started with no adapter for discord, connected webhook and email, and reported running with 2 platforms

The plugin timed out after 10 seconds. Its register_platform() call did come back, about 0.7 seconds later, and got ignored because the loader had already given up. A few minutes later the gateway started, logged No adapter available for discord, connected the two platforms it did have (webhook and email), and said Gateway running with 2 platform(s).

So the gateway was up and working. It just wasn't on Discord. And nothing treats "running with 2 platforms instead of 3" as an error.

Why every check stayed green

A table: systemctl said active, the watchdog said green, the state file said connected, and nothing asked whether Discord was actually connected right now

systemd was right. The process was up. It only knows about the process.

My watchdog was right too, about the gateway. It wasn't Discord-aware, so as long as the gateway was alive, it was happy.

The state file was the nasty one. The gateway keeps a gateway_state.json with a row per platform. The Discord row still said connected. But that row had a writer_pid of 15603, and process 15603 was dead. The running gateway never loaded the Discord adapter, so nothing ever replaced that row, and it just sat there saying "connected" for a week.

That's the general trap: a status row is only as true as whoever last wrote it. If the writer is dead, the row is history, not status.

The fix

Two parts.

1. Give the plugin more time. In the gateway's config.yaml:

plugins:
  load_timeout_seconds: 90
Enter fullscreen mode Exit fullscreen mode

Then a gateway restart, and the log said [Discord] Connected as with the bot's name. A week late, but there.

2. A health check that asks the actual question. I wrote a small script, discord-health.py, that exits 0 if healthy and 1 with a reason if not. It only trusts the state file if the live gateway wrote it:

  1. the state file's pid matches the gateway service's current MainPID
  2. the state file's updated_at is less than 10 minutes old
  3. the Discord row's writer_pid matches that same MainPID
  4. the Discord row's state is connected

The core of it is only a few lines. This is check 3, straight from the script:

if str(d.get("writer_pid")) != pid:
    print(f"discord row written by pid {d.get('writer_pid')}, not live gateway {pid} (adapter never loaded)"); sys.exit(1)
Enter fullscreen mode Exit fullscreen mode

Check 3 is the one that would have caught this on day one. A Discord row written by any process other than the running gateway means the adapter never loaded this time round.

One gotcha: it shells out to systemctl --user to get the MainPID, and when you run that from a timer as root it needs XDG_RUNTIME_DIR=/run/user/0 set, or it can't find the user bus.

The watchdog now knows about Discord

I hooked that check into the gateway watchdog, which runs every 3 minutes on a systemd timer:

  • 5 minutes of grace after the gateway starts, so it doesn't fight a slow boot
  • restart only after 2 failures in a row
  • at most 3 restarts an hour, then it holds off instead of looping

When it restarts the gateway, or hits that hold-off, it posts to my network alerts channel. It does that through Discord's REST API, not through the bot's own connection, so the alert doesn't depend on the bot being connected.

What I'd do differently

  • Health checks should ask about the thing you use, not the process. "Is the service active" and "can I talk to it on Discord" are different questions.
  • Treat stale state as unknown, not as good. Any status row should carry who wrote it and when, and the reader should check both.
  • Default timeouts are tuned for calm boots. If your host has a known slow boot, every startup timeout on it is a bet. Ten seconds lost that bet.
  • "Running with 2 platforms" should be loud when you configured 3. I'd like the gateway itself to fail health when a configured platform has no adapter.
  • Check your edits survive upgrades. That timeout key lives in config.yaml. If an upgrade regenerates the config, it could quietly go back to 10 seconds. It's on my list to re-check after every update.

How do you check your bots are actually connected, rather than just running? I'd genuinely like to know what other people use.


🤖 Drafted with AI assistance from my own homelab notes, logs and repos, and fact-checked against them before publishing.

Top comments (0)