DEV Community

HostNest
HostNest

Posted on Fully Autonomous

How to keep an always-on AI agent from breaking when you update it

If you run an AI agent around the clock, on a Mac mini, a home server or a VPS, you have probably had this morning. You updated it last night, and today it doesn't answer. Or it answers, but it has forgotten who it is. Or its scheduled jobs stopped two days ago and nobody noticed.

In the agent forums we read, those three come up every week, whatever the framework. One person updated and found the agent's memory notes gone. Another found their scheduled jobs had stopped two days before they noticed.

None of the fixes below is clever. That's the point: each one turns a bad morning into a non-event.

1. Keep the agent's brain outside the thing you update

Memory notes, instructions and skills should be plain files in a folder the installer never touches. Put that folder under git and commit before every update. If an update resets the workspace, one checkout brings it back. If your framework insists on keeping them in its own folder, move them out and link them back in.

2. Never update without a way back

Take a snapshot before you update: your VPS provider's, or an archive of the install and its config. Better, script the update so it checks that the agent still answers afterwards, and rolls back by itself if it doesn't. A rollback you have never run isn't a rollback. Pin the version, and update on purpose, not whenever something ran last night.

3. Watch the connection, not just the process

Run the agent as a service (systemd on Linux, launchd on a Mac) so it comes back after a crash or a reboot. But the usual failure isn't a dead process. It's a live process whose chat connection quietly dropped. A small watchdog that checks the bot actually responds, every minute or so, and restarts the connection when it doesn't, covers that.

4. Put schedules on the server, not in the conversation

A schedule that lives inside the agent's session dies with the session. Put recurring jobs in the server's own scheduler (cron or a systemd timer) and have it wake the agent with the task. Then make every job leave a trace when it runs, and keep one check that tells you when a trace is missing. Jobs that stop without anyone noticing are a monitoring problem, not an agent problem.

5. Back up off the box, and try a restore

Daily, to somewhere that isn't the server, kept for at least a week. Then once, for real, restore onto a fresh machine and see what's missing. That's when you find the key that only ever lived in your shell history.

6. Hear about it before and after

Have the update tell you: "updating now", then "back, checks passed" or "rolled back, here's why". Silence after an update is the worst signal there is.

Top comments (0)