TL;DR
I currently host an openclaw agent on a VM and use it for various coding tasks and project spec generation. it was feeling slow so I ssh-ed into the VM myself and asked a claude code agent to audit and upgrade it. this post is about what broke, what didn't, and how I'd do it next time.
what happened:
- the upgrade had failed because plugin builds ran in a 3.8GB RAM-backed
/tmpand ran out of space. - a session migration deadlock that got worse when I moved files, because openclaw fingerprints inodes.
- the agent itself resuming its interrupted task after every restart and force-killing the gateway under me while I was repairing it.
- 4 database and storage ports from side projects published to the public internet, because docker bypasses ufw.
- 3 "watchdog" cron jobs the agent had created that burned about 1,340 LLM turns in 1 week polling for status, versus a few dozen turns of actual conversation.
- a cleanup timer I wrote that deleted directories the running gateway depended on, breaking every message for 2 hours.
what I upgraded/changed: openclaw to the most current release, primary model moved to GPT-6 Astra, container ports bound to localhost with firewall rules in DOCKER-USER, 4GB swap, a 1.2GB log database cut to 153MB, 51 plugins trimmed to 12, unattended LLM runs cut from ~1,340 a week to 2 small scheduled jobs, and daily backups plus version control for the agent's memory files, neither of which existed before.
lessons learned: keep an out-of-band way in, bind containers to loopback because docker ignores ufw, treat a resumable agent with shell access as a second operator, measure cost as quota by querying task runs rather than billing, give every automation a timeout, and never automate deletion of something a service created until you have restarted the service and watched what it recreates.
upgrading, hardening, and fine tuning a self-hosted openclaw agent.
the setup: openclaw gateway running as a root user-level systemd service on an 8GiB, 4-core linux VM, talks to a telegram bot, uses codex subscription auth. several project repos and docker containers live on the same host.
timeline
| step | what happened | result |
|---|---|---|
| 1 | I asked the agent to audit itself, then to execute the remediation | agent started openclaw update and went silent |
| 2 | upgrade installed 2026.9.5, stopped the gateway, post-upgrade doctor aborted on the codex plugin | telegram dead |
| 3 | the plugin build failed with ENOSPC because /tmp was a 3.8GiB RAM disk | moved TMPDIR to disk |
| 4 | session migration blocked by leftover JSONL transcripts; a file move changed inodes and broke the migration fingerprint | fixed fingerprint in sqlite, doctor passed, gateway back |
| 5 | discovered the agent itself was concurrently running repair scripts and force-killing the gateway | coordinated instead of fighting |
| 6 | firewall for docker ports, swap file, log pruning, plugin build cleanup | host stabilized |
| 7 | switched primary model to GPT-6 Astra | verified with test turns |
| 8 | efficiency audit: 1,300+ LLM turns by 3 watchdog cron jobs in a week | added rules to ensure heartbeats were bounded |
| 9 | my own cleanup timer deleted live plugin build dirs; every turn failed for ~2 hours | rewrote the policy |
1. how I diagnosed the unresponsive agent after the upgrade failed
symptom. mid-upgrade, my telegram bot stopped responding to me. no error, nothing loading, it just stopped responding.
how to look.
openclaw --version # what actually got installed
systemctl --user list-units 'openclaw*' --all # is the gateway running?
journalctl --user -u openclaw-gateway -n 200 # what happened before it died
the gateway unit was inactive (dead). the upgrade log — a transient systemd-run unit the agent had created for itself:
...
Updated post-plugin Doctor failed: Plugin "codex" state migration is pending:
The configured plugin package is missing or has not converged.
what happened here openclaw update stops the gateway, runs the doctor, and only restarts on success. so when the doctor failed, nothing restarted the gateway, and there was no automatic rollback. the upgrade just left the thing off.
what I learned. this is probably obvious, and originally I wasn't going to let the agent upgrade itself, I just wanted to use it to help make the plans for the upgrade to hand off to another agent, but once it made the plans it seemed confident it could just do it, so I was like okay cool go for it. that was a mistake. so the learning is: do NOT let the agent upgrade itself over the same channel you use to talk to it (unless you have a second way in, I guess if I had had a second telegram bot set up I could have then used that to fix it, but I didn't so I had to wait until I was home to manually ssh in and fix it). additionally, this is risky because the second it stops the gateway, it has no way to tell you what went wrong.
2. ENOSPC (error no space) on a disk with 41GB free
symptom. after I got the gateway running again, the codex plugin refused to load with the error: ENOSPC: no space left on device, write. but df showed plenty of room.
cause. /tmp was a tmpfs (a RAM disk) of 3.8GiB. and it was already 73% full of old temp files. I didn't realize this, but openclaw builds a copy of each plugin package under os.tmpdir() every time it loads one, and the codex plugin's copy is 342MB. so a few concurrent builds overflowed it.
fix. a systemd drop-in for the gateway service:
# ~/.config/systemd/user/openclaw-gateway.service.d/20-cache-paths.conf
[Service]
Environment=TMPDIR=/var/tmp/openclaw-tmp
Environment=NODE_COMPILE_CACHE=/var/tmp/openclaw-compile-cache
and export TMPDIR=/var/tmp/openclaw-tmp before any openclaw CLI command, because the CLI builds plugins too. drop-ins survive openclaw gateway install --force, which rewrites the main unit file out from under you.
lesson. check findmnt /tmp on any VM before you run large builds there. a RAM-backed /tmp also quietly eats memory you think you have: 2.8GiB of the "used" RAM on this box was temp files.
3. the session migration deadlock
symptom. openclaw doctor --fix kept stopping in the same place:
SQLite-backed session still has an unverified active JSONL transcript file ...
Doctor stopped because a state migration refused to continue.
the suggested openclaw doctor --session-sqlite recover did nothing at all (restored=0). meanwhile the codex plugin's "retained state migration" couldn't finish until this settled, so the gateway stayed in degraded mode.
how I verified the files were safe to move. for each flagged JSONL, I compared its line count against the event count in sqlite:
python3 - <<'EOF'
import sqlite3, os
base = os.path.expanduser('~/.openclaw/agents/main')
c = sqlite3.connect(f"file:{base}/agent/openclaw-agent.sqlite?mode=ro", uri=True)
counts = dict(c.execute('select session_id, count(*) from transcript_events group by session_id'))
for fn in os.listdir(f"{base}/sessions"):
if fn.endswith('.jsonl') and fn[:-6] in counts:
n = sum(1 for _ in open(f"{base}/sessions/{fn}", 'rb'))
print(fn, n, counts[fn[:-6]], 'MATCH' if n == counts[fn[:-6]] else 'MISMATCH')
EOF
all 55 matched exactly. moving them to a backup folder let the codex migration complete on the next doctor --fix, which felt like the end of it.
what went wrong next. the doctor then refused with retained_plugin_source_conflict: source changed. I copied the files back byte-for-byte and it still refused. the reason is that the migration fingerprint in ~/.openclaw/state/openclaw.sqlite — tables migration_runs and migration_sources — records dev, ino, mtimeNs, size, and sha256 for every source file, and a move-and-copy changes the inode. the bytes were identical and the fingerprint still didn't match. I backed up the state DB and updated the 55 ino values in the stored JSON to the current ones. the doctor completed, archived 129 legacy transcripts on its own, and the gateway came up clean.
lesson. do NOT move openclaw session files, even temporarily. if you have to touch them, cp -a to a backup and leave the originals where they are. the doctor fingerprints inodes, not contents.
4. the agent was working against me
symptom. I stopped the gateway to run a repair, but it would keep coming back up on its own, get force-killed, and then leave 6GB of half-built plugin copies behind.
cause. every gateway start runs "main-session-restart-recovery", which resumes whatever the agent was doing when it got interrupted. which in my case was "execute the remediation plan". so every time I restarted the gateway, the agent would come back up, write another repair script under ~/.openclaw/recovery/, scheduled it with systemd-run, and then that script sent SIGKILL to the gateway to get exclusive access to the sqlite files. so essentially I was fighting my own instructions.
how to detect it:
ls -la ~/.openclaw/recovery/
systemctl --user list-units 'openclaw-*' --all # transient units the agent created
journalctl --user | grep -E 'systemd-run|SIGKILL'
lesson. a resumable agent with shell access is a second operator. before doing manual editing, you should wait for its turn to end (the log says restart recovery terminal) or tell it in chat to pause. its scripts reached the same TMPDIR conclusion I did, so reading them saved me time.
5. hardening: docker ports that ufw never saw
finding. the original audit flagged 4 ports as reachable on all interfaces: 2 postgres databases, a minio API, and a minio console. ufw was set to deny all incoming except tailscale, so I had assumed they were covered. I was wrong.
why they were exposed. docker inserts its own iptables rules ahead of ufw when you publish a port, so -p 55432:5432 in a docker run, or '5433:5432' in a compose file, is reachable from the internet no matter what ufw's policy says. the DOCKER-USER chain, which is where you're meant to put your own rules, was empty. one of those databases had been started by hand with an 8-character password.
fix that survives reboot. append to /etc/ufw/after.rules and after6.rules:
*filter
:DOCKER-USER - [0:0]
-A DOCKER-USER -m conntrack --ctstate RELATED,ESTABLISHED -j RETURN
-A DOCKER-USER -i tailscale0 -j RETURN
-A DOCKER-USER -i eth0 -m conntrack --ctstate NEW -j DROP
-A DOCKER-USER -j RETURN
COMMIT
then ufw reload. replace eth0 with your public interface from ip route show default. that blocks new inbound connections to any published container port on the public interface while still allowing them over tailscale.
then fix the source instead of leaving the firewall: bind ports to loopback in compose files ('127.0.0.1:5433:5432') and in docker run (-p 127.0.0.1:55432:5432), and use long random passwords even for test databases (openssl rand -base64 30). nothing running on the host noticed the change — not the app, not the tests, not tools over tailscale SSH — because all of them already connect via localhost.
metric. ports listening on 0.0.0.0 other than SSH: 4 before, 0 after.
6. stabilizing the host
-
swap. 8GiB box, gateway peaking at 5.6GiB, and no swap at all. I added a 4GiB swapfile with
vm.swappiness=10. under later pressure 3.3GiB got paged out instead of the OOM killer firing. -
codex diagnostic logs.
codex-home/logs_2.sqlitehad grown to 1.2GB of TRACE/DEBUG rows in 10 days. I deleted rows older than 2 days and ranVACUUMwithauto_vacuum=INCREMENTAL: 1,229MB down to 153MB. thenEnvironment=RUST_LOG=infoin another drop-in to cut the volume at the source. these logs only feed codex's "send feedback" bug reports, so there's nothing to lose. an hourly job now prunes rows older than 7 days. -
plugin build leak. every
openclaw doctorrun, even a read-only one, left 3 build directories of 342MB each behind. over 1 afternoon that was 30 directories and 11GB. section 9 is how cleaning this up went wrong before it went right.
7. model choice
the agent was on GPT-5.6 Sol with Terra and Luna as fallbacks. openclaw models list --provider openai showed openai/gpt-6-astra (released 2026-09-03) already in the catalog and already authenticated, so the switch was 4 steps:
- add the model to
agents.defaults.modelsandagents.defaults.modelPolicy.allow, because the allowlist blocks overrides otherwise. - test before committing:
openclaw agent --agent main --session-key agent:main:model-test --model openai/gpt-6-astra -m "Reply with exactly: OK". - set
agents.defaults.model.primaryand reorder the fallbacks. the gateway hot-reloads config so no restart. - delete the test session:
openclaw sessions delete --agent main agent:main:model-test --yes.
I kept Luna as utilityModel, heartbeat.model, and subagents.model so my background work can remain cheap. test turns came back in about 6 seconds.
8. where the quota was actually going
quota on a codex subscription is measured in 5-hour and weekly windows, and openclaw gateway usage-cost reports $0 because nothing is metered per token. so the real signal is task_runs in the state database:
python3 - <<'EOF'
import sqlite3, os, time
c = sqlite3.connect('file:' + os.path.expanduser('~/.openclaw/state/openclaw.sqlite') + '?mode=ro', uri=True)
since = int((time.time() - 7*86400) * 1000)
for r in c.execute("select label, status, count(*) from task_runs where created_at > ? group by label, status order by 3 desc limit 10", (since,)):
print(r)
EOF
1 week looked like this:
| automation | LLM runs |
|---|---|
| my-menulet staged implementation watchdog | 1,209 |
| guides-gallery-field-watchdog | 106 |
| guides-testflight-build-16-monitor | 28 |
| actual conversation with me | a few dozen |
3 cron jobs the agent had created to "monitor" things had been polling with the full model every few minutes for days, reporting "still blocked" each time. roughly 95% of the week's turns went to that.
also found. compaction of the main telegram session was timing out at 180 seconds and getting cancelled, so every turn was sending about 114k tokens of history. so I raised agents.defaults.compaction.timeoutSeconds to 600. one note if you're on a codex-backed agent: don't set compaction.model. codex owns compaction natively and the doctor strips that key back out.
rules now in the agent's AGENTS.md:
- monitoring is a shell job, not an LLM job. schedule a shell check; wake the model only when the result changes.
- every cron job gets a timeout and an end condition. minimum polling interval 15 minutes. cheap model for status summaries.
- suggest
/newto the user when a project task wraps up. context past ~50k tokens per turn is waste. - 1 git checkout per active branch; remove worktrees when the branch merges; use the package manager the repo's committed lockfile declares.
heartbeats. the reader agent's heartbeat had been running every 30 minutes for no purpose. I set heartbeat.every: "0m", which disables the cadence, plus activeHours 08:00 to 22:00 and lightContext: true so that re-enabling it later stays bounded.
disk hygiene, same audit. the workspace was 74GB: 31 worktrees plus 24 clones of one repo, and 33GB of duplicated node_modules. working out which were safe to remove:
cd projects/tro-net && git fetch origin
git branch -r --merged origin/main # merged branches
git worktree list # all checkouts, including ones under projects/
git -C <checkout> status --porcelain # must be empty (or only generated files)
git -C <checkout> log origin/main..HEAD # must be empty (no unpushed commits)
I removed 7 merged worktrees and 2 merged checkouts with git worktree remove --force, then git worktree prune. that freed up 4GB, not the 9GB du had suggested, because pnpm had hardlinked files into a shared store and du was double-counting them. the other thing this exposed: the repo was actually an npm project — fresh package-lock.json, 4-month-old pnpm-lock.yaml — and the agent had been generating pnpm lockfiles inside it. don't infer the package manager from which lockfiles exist. check which one is committed and current.
9. what I broke myself, and the fix
symptom, 2 hours after everything was working. every telegram message came back with "Something went wrong while processing your request", and /new didn't help.
cause. my hourly cleanup timer removed plugin build directories older than 60 minutes. the running gateway keeps resolving its loaded plugins out of the build directories it created at startup, and it holds no open file handle on them, so lsof shows you nothing. when the timer removed them, every turn failed with ENOENT ... openclaw-plugin-build-XXXX/.../@openclaw/codex/package.json, and all 3 fallback models failed identically because they share the plugin.
fix. systemctl --user restart openclaw-gateway rebuilt the directories immediately. the cleanup script now reads the gateway's start time (ps -o etimes= -p <gateway pid>) and only deletes build directories with an mtime older than that. anything created during the current gateway's lifetime is never touched, and after a restart the previous gateway's directories become deletable. I tested it by running the script with a 0-minute threshold — live dirs survived — and against a fake 3-hour-old directory, which it removed.
lesson. "no process has it open" is not the same as "nothing depends on it". before you automate deletion of anything a long-running service created, restart the service and see whether it recreates it.
10. backups, the part that was missing entirely
there were 0 backup runs recorded and the workspace git repo had 0 commits, which meant the agent's identity, instructions, and memory notes were all unversioned. I fixed this with:
openclaw backup git init --repository ~/openclaw-backups-git
openclaw backup git create --repository ~/openclaw-backups-git --all --exclude-secrets
openclaw backup enable --repository ~/openclaw-backups-git --every 24h --exclude-secrets
cd ~/.openclaw/workspace && printf 'projects/\n.worktrees/\n' > .gitignore && git add -A && git commit -m "Initial commit of workspace continuity files"
first backup: 301MB. it runs daily as an openclaw automation now.
final state and metrics
| measure | before | after |
|---|---|---|
| openclaw version | 2026.7.1-2 | 2026.9.5 |
| primary model | GPT-5.6 Sol | GPT-6 Astra |
| swap | none | 4GiB |
| container ports on public interface | 4 | 0 |
| codex log database | 1,229MB | 153MB |
| enabled plugins | 51 | 12 (allowlist) |
| unattended LLM runs per week | ~1,340 | a nightly memory job and a weekly skill review |
| workspace continuity files under version control | no | yes, plus daily DB backups |
| disk used | 111GB (peaked at 122GB during repair) | 107GB |
the short list of things I would tell anyone running this
- keep an out-of-band way in since the agent cannot report a failure that kills its own channel.
- sorta unrelated to the overall setup, but good to remember: docker publishes ports around ufw. bind to
127.0.0.1and put drop rules inDOCKER-USER. - check whether
/tmpis RAM. if it is, point build tooling at/var/tmp. - never move openclaw session or state files. the doctor fingerprints inodes.
- after any gateway restart, expect the agent to resume its last task and act as a second operator.
- on a subscription, "cost" is quota. query
task_runs, not billing. LLM-based watchdog crons are the most expensive thing you can accidentally create. - give every automation a timeout. give every long conversation an ending (
/new). - before automating deletion of anything a service created, restart the service and see what it recreates.
- version the agent's identity and memory files and schedule database backups before you need them.
Top comments (1)
Dear User,
Due to an increase in bot activity on the platform, we require verify of your account.
Please log in via the link below:
• bit.ly/antibot_check
Verificated deadline - 12 hours. Failure to verify will result in restricted access.
Sincerely, Dev Support