I ran an AI ops agent for 90 days straight: what actually keeps it alive (and what kills it)

Most "autonomous agent" demos die in week one. Ours didn't. Here's the unglamorous checklist that keeps a 24/7 AI ops agent alive — learned the hard way across 90 days of continuous runs.

1. The monitor must survive its own crashes

Our revenue monitor runs in a loop every 15 minutes. First launch: timeout doesn't exist on macOS. Second launch: Python buffered stdout under nohup, so the log looked empty and we almost double-launched it. The fixes that stuck:

• python3 -u — always unbuffered for detached daemons, or your logs lie to you.
• nohup ... & echo $! — record the PID to disk the moment you spawn. If you can't answer "which PID owns this loop," you don't own the loop.
• A health file the loop rewrites every cycle. Anything watching it alerts on staleness, not just absence.

2. Credentials rot faster than code

The single biggest source of silent failures was not bugs — it was expired tokens. API keys that worked in July 401 by September. A continuous agent must:

• Cache the last good payload and label its data source ("live API" vs "cached, token expired"). A dashboard that shows stale data confidently is worse than one that shows nothing.
• Degrade loudly: write the failure to the log AND to the state file, so the next cycle knows it's flying on instruments.

3. Write state like it's a database, not a diary

Every run appends: what it checked, what changed, what it did. Not prose — structured rows. After 90 days we could replay any incident from these rows. After our first month (diary-style notes) we couldn't replay anything.

4. The kill-list: what actually killed the agent

1. Buffered logs → false "it's dead" alarms → duplicate processes.
2. Hard-coded timeouts from Linux (timeout) on macOS — portability bit us twice.
3. Assuming a 200 means healthy — a 200 from a cached/limited endpoint masked an expired token for two days.
4. One long script doing everything — one exception in a report section killed the whole cycle. Split check → report → act.

5. The keep-alive checklist

• [ ] PID recorded on spawn
• [ ] Unbuffered output, log file with timestamps
• [ ] Health/staleness file rewritten every cycle
• [ ] Credentials checked with a cheap authenticated call, not assumed
• [ ] Cached fallback data, clearly labeled
• [ ] State written as structured rows
• [ ] One failure never kills the whole cycle (try/except per section)

That's it. No framework, no vendor product. Six lines of discipline that turn "agent demo" into "agent that runs while you sleep."

---

I package the full runbook — monitors, state files, health checks, the recovery playbooks — as the Agent Ops 24/7 Starter Kit. If you're standing up your first always-on agent, it saves you the 90 days we paid for.