DEV Community

Pandas Studio
Pandas Studio

Posted on Fully Autonomous

[OpenShell] Fixing OpenShell MainProcessExited Error via DB Bypass

#nvidia #llm #selfhosted #devops

After a system reboot, my sandbox became stuck in a permanent Error state, blocking all management operations. I learned how to bypass this state by manually patching the gateway database and how to prevent it using systemd service ordering.

Background

I run a Hermes Agent (v0.20.6) inside a sandbox named my-claw on an NVIDIA DGX Spark. The environment uses NemoClaw (v0.0.124) which is pinned to OpenShell v0.0.116. Management of the sandbox is handled via the openshell and nemohermes CLI tools, which communicate with a gateway service.

What happened

Following a reboot on 2026-10-24, the sandbox phase was stuck at Error (MainProcessExited). This error state caused all management commands—exec, snapshot, stop, start, and rebuild—to be rejected. While the agent's internal processes like Hermes, cron, and Telegram continued to run, the daily backup (sandbox-backup.service) failed because it could not execute commands within the sandbox. Specifically, nemohermes exec for deployment and openshell sandbox download were both blocked.

What I tried

  1. Manual data backup via docker exec → Success. I bypassed the management tools to extract 183 files (1.4M), .hermes files (4,094 files, 36M), and other data (271M) directly from the container. This provided a fallback when the standard backup failed.
  2. Standard management commands (stop, start, rebuild) → Failed. The system refused commands with errors stating the sandbox must be in a specific state (e.g., 'must be Ready to stop' or 'must be Stopped to start'). I also tried rebuild, but it failed because the step to read MCP settings requires exec, which was blocked.
  3. Restarting the gateway service → Failed. The supervisor reconnected immediately, but the phase remained in Error.
  4. Stopping the container with docker stop → Failed. The phase did not transition to Stopped, and the error persisted. This attempt caused an 11-minute interruption to Hermes and Telegram while the container was down.

Root cause

OpenShell v0.0.116 has a bug where if the main process terminates while the sandbox is in a Ready state (as happened during the reboot), it records a MainProcessExited error. In this specific version, Error is treated as a terminal state that cannot be exited through standard commands; it can only be cleared by deleting the sandbox entirely. While this was fixed in v0.1.5 (PR #4231), NemoClaw pins the version to 0.0.116.

The fix

Instead of the destructive 'destroy and onboard' approach, I developed a Python script to perform a surgical 1-byte modification in the gateway's SQLite database. By changing the phase value from Error (3) to Stopped (7), I tricked the gateway into allowing a standard start sequence. To prevent recurrence, I implemented a sandbox-clean-stop.service that triggers an openshell sandbox stop during system shutdown, ensuring the sandbox reaches a clean Stopped state before the host goes down. I also had to fix a systemd ordering cycle where the unit was being dropped at boot because of a WantedBy=default.target conflict.

Surgical DB phase fix logic

p[off] = STOPPED
c.execute("update objects set payload=? where rowid=?", (bytes(p), rowid))
Enter fullscreen mode Exit fullscreen mode

Clean stop service configuration

[Unit]
Description=nemo-ops: cleanly stop my-claw at shutdown (OpenShell 0.0.116 sticks in Error if it goes down while Ready)
# Units stop in reverse After= order, so this stops before the gateway.
After=nemoclaw-openshell-gateway.service

[Service]
Type=oneshot
RemainAfterExit=yes
Environment=PATH=%h/.local/bin:/usr/local/bin:/usr/bin:/bin
ExecStop=%h/nemo-ops/bin/sandbox-clean-stop.sh
TimeoutStopSec=180

[Install]
# Pulled in by the gateway, not default.target: the gateway is After=default.target, so
# WantedBy=default.target forms an ordering cycle and systemd drops this unit's start job at boot (10-25 reboot).
WantedBy=nemoclaw-openshell-gateway.service
Enter fullscreen mode Exit fullscreen mode

Result

After applying the DB fix, the sandbox returned to Ready at 13:21:53. All management functions, including exec, health checks, Telegram notifications, and backups, resumed normal operation.

Lessons

  • When management tools fail, use docker exec to rescue data immediately since snapshots might not cover all paths.
  • In older versions of software, a terminal error state can sometimes be bypassed by manually patching the underlying database if a backup is available.
  • Avoid circular dependencies in systemd by ensuring WantedBy targets do not create ordering loops with After= requirements.

Still open

NemoClaw remains pinned to OpenShell 0.0.116, meaning the underlying bug persists and requires the DB fix tool if a non-graceful shutdown occurs.

This post was written by a local Gemma 4 model from my own commit log and experiment notes, fact-checked against them by a local Qwen3.6 model, with no human edits.

Top comments (0)