DEV Community: Pandas Studio

[OpenShell] Fixing OpenShell MainProcessExited Error via DB Bypass

Pandas Studio — Sun, 25 Oct 2026 15:24:19 +0000

After a system reboot, my sandbox became stuck in a permanent Error state, blocking all management operations. I learned how to bypass this state by manually patching the gateway database and how to prevent it using systemd service ordering.

Background

I run a Hermes Agent (v0.20.6) inside a sandbox named my-claw on an NVIDIA DGX Spark. The environment uses NemoClaw (v0.0.124) which is pinned to OpenShell v0.0.116. Management of the sandbox is handled via the openshell and nemohermes CLI tools, which communicate with a gateway service.

What happened

Following a reboot on 2026-10-24, the sandbox phase was stuck at Error (MainProcessExited). This error state caused all management commands—exec, snapshot, stop, start, and rebuild—to be rejected. While the agent's internal processes like Hermes, cron, and Telegram continued to run, the daily backup (sandbox-backup.service) failed because it could not execute commands within the sandbox. Specifically, nemohermes exec for deployment and openshell sandbox download were both blocked.

What I tried

Manual data backup via docker exec → Success. I bypassed the management tools to extract 183 files (1.4M), .hermes files (4,094 files, 36M), and other data (271M) directly from the container. This provided a fallback when the standard backup failed.
Standard management commands (stop, start, rebuild) → Failed. The system refused commands with errors stating the sandbox must be in a specific state (e.g., 'must be Ready to stop' or 'must be Stopped to start'). I also tried rebuild, but it failed because the step to read MCP settings requires exec, which was blocked.
Restarting the gateway service → Failed. The supervisor reconnected immediately, but the phase remained in Error.
Stopping the container with docker stop → Failed. The phase did not transition to Stopped, and the error persisted. This attempt caused an 11-minute interruption to Hermes and Telegram while the container was down.

Root cause

OpenShell v0.0.116 has a bug where if the main process terminates while the sandbox is in a Ready state (as happened during the reboot), it records a MainProcessExited error. In this specific version, Error is treated as a terminal state that cannot be exited through standard commands; it can only be cleared by deleting the sandbox entirely. While this was fixed in v0.1.5 (PR #4231), NemoClaw pins the version to 0.0.116.

The fix

Instead of the destructive 'destroy and onboard' approach, I developed a Python script to perform a surgical 1-byte modification in the gateway's SQLite database. By changing the phase value from Error (3) to Stopped (7), I tricked the gateway into allowing a standard start sequence. To prevent recurrence, I implemented a sandbox-clean-stop.service that triggers an openshell sandbox stop during system shutdown, ensuring the sandbox reaches a clean Stopped state before the host goes down. I also had to fix a systemd ordering cycle where the unit was being dropped at boot because of a WantedBy=default.target conflict.

Surgical DB phase fix logic

p[off] = STOPPED
c.execute("update objects set payload=? where rowid=?", (bytes(p), rowid))

Clean stop service configuration

[Unit]
Description=nemo-ops: cleanly stop my-claw at shutdown (OpenShell 0.0.116 sticks in Error if it goes down while Ready)
# Units stop in reverse After= order, so this stops before the gateway.
After=nemoclaw-openshell-gateway.service

[Service]
Type=oneshot
RemainAfterExit=yes
Environment=PATH=%h/.local/bin:/usr/local/bin:/usr/bin:/bin
ExecStop=%h/nemo-ops/bin/sandbox-clean-stop.sh
TimeoutStopSec=180

[Install]
# Pulled in by the gateway, not default.target: the gateway is After=default.target, so
# WantedBy=default.target forms an ordering cycle and systemd drops this unit's start job at boot (10-25 reboot).
WantedBy=nemoclaw-openshell-gateway.service

Result

After applying the DB fix, the sandbox returned to Ready at 13:21:53. All management functions, including exec, health checks, Telegram notifications, and backups, resumed normal operation.

Lessons

When management tools fail, use docker exec to rescue data immediately since snapshots might not cover all paths.
In older versions of software, a terminal error state can sometimes be bypassed by manually patching the underlying database if a backup is available.
Avoid circular dependencies in systemd by ensuring WantedBy targets do not create ordering loops with After= requirements.

Still open

NemoClaw remains pinned to OpenShell 0.0.116, meaning the underlying bug persists and requires the DB fix tool if a non-graceful shutdown occurs.

This post was written by a local Gemma 4 model from my own commit log and experiment notes, fact-checked against them by a local Qwen3.6 model, with no human edits.

[Hermes Agent] Fixing Voice Memo Skill Misidentification via pre_gateway_dispatch

Pandas Studio — Sun, 11 Oct 2026 14:24:35 +0000

While testing voice-to-document automation on a DGX Spark, the Hermes Agent repeatedly failed to trigger the intended 'voice-memo' skill. I discovered the model was misinterpreting the trigger word as a request to use its internal memory tool instead of the specialized skill.

Background

The setup involved a Hermes Agent (v0.20.6) running on an NVIDIA DGX Spark with a Qwen 3.6-35B model. The goal was to send voice memos via Telegram, have them transcribed by a local Whisper server, and then have the agent transform them into structured documents saved to /sandbox/notes/memos/ using a specific skill.

What happened

During the second round of testing, despite successful transcriptions, the skill activation rate was 0/3. While the transcriptions correctly started with the word 메모 ("memo"), the Qwen model consistently chose the default memory tool to store the information rather than calling the voice-memo skill. This resulted in the notes being saved to the general MEMORY.md file instead of being processed into structured markdown files.

What I tried

Modifying the skill activation condition to trigger when the transcription starts with the word 메모. → Failed; the model still prioritized the memory tool over the specialized skill.
Adding technical term prompts to the Whisper STT engine to reduce transcription errors. → Partially successful; reduced errors from 6 to 3 in specific test cases.

Root cause

The root cause was a model-level semantic misunderstanding. When the agent received a message starting with 메모, the Qwen model interpreted the intent as 'remember this' and invoked its built-in memory tool. Because the Hermes v0.20.6 gateway only passes the transcribed text as a plain quoted string without specific voice markers, the model had no way to distinguish a general conversational mention of 'memo' from a formal command to trigger the voice-memo skill.

The fix

I implemented a custom Hermes plugin using the pre_gateway_dispatch hook. This hook intercepts the message before it reaches the model. For audio, it performs a preliminary transcription (approx. 0.7s) to check if the first word is 메모. For text, it checks if the message starts with 메모:. If a match is found, the plugin rewrites the message to include a hardcoded instruction: [voice-memo] This message is the author's memo... Do not use the memory tool... Call skill_view with name "voice-memo".... This forces the model to use the correct skill by explicitly forbidding the memory tool.

Voice memo plugin logic for intercepting and rewriting messages

SPOKEN = re.compile(r"^\W*메모(?=[\s,.:!?~…]|$)")   # 전사는 "메모! 어제…", "메모 주간…"처럼 온다
TYPED = re.compile(r"^\s*메모\s*[:：]")
INSTRUCTION = ("[voice-memo] This message is the author's memo (it starts with the word 메모). "
               "Do not use the memory tool and do not continue earlier topics. "
               "Call skill_view with name \"voice-memo\" and follow it exactly: write the document, save it with the file tool "
               "under /sandbox/notes/memos/, then reply with the full document and the 저장: line.")


def on_dispatch(event=None, **_):
    try:
        kind = getattr(getattr(event, "message_type", None), "value", "")
        text = (getattr(event, "text", "") or "").strip()
        if kind == "text" and TYPED.match(text):
            return {"action": "rewrite", "text": f"{INSTRUCTION}\n\n{text}"}
        media = list(getattr(event, "media_urls", None) or [])
        if kind in ("voice", "audio") and media and SPOKEN.match(_transcribe(media[0]).strip()):
            logger.info("voice_memo: 메모 음성 → voice-memo 스킬 지시")
            return {"action": "rewrite", "text": f"{text}\n\n{INSTRUCTION}" if text else INSTRUCTION}
    except Exception as e:   # 훅이 실패해도 메시지는 평소대로 간다
        logger.warning("voice_memo: 판단 실패 %s", e)
    return None

Registering the hook in the plugin

def register(ctx):
    ctx.register_hook("pre_gateway_dispatch", on_dispatch)

Result

In the third round of testing, the fix was successful, achieving a 3/3 success rate for document saving. The memory tool was used 0 times, and all notes were correctly processed into the target directory.

Lessons

When an LLM consistently misinterprets a command as a different tool, do not rely on prompting alone; use a pre-dispatch hook to inject explicit constraints.
For voice-driven automation, performing a lightweight preliminary transcription at the gateway level can provide the necessary context to steer model behavior.

Still open

The system still experiences interrupted turns if a new voice message arrives while the agent is busy (busy_input_mode: interrupt), and some transcription hallucinations (e.g., Whisper adding 'Thank you' at the end of silence) persist.

This post was written by a local Gemma 4 model from my own commit log and experiment notes, fact-checked against them by a local Qwen3.6 model, with no human edits.

[Hermes Agent] Solving Voice Memo Skill Trigger Failures via Plugin Hooks

Pandas Studio — Thu, 08 Oct 2026 23:35:02 +0000

While testing a voice-to-structured-document workflow on an NVIDIA DGX Spark, the Hermes Agent consistently failed to trigger the specific voice-memo skill, opting for the default memory tool instead. This post details how I implemented custom gateway hooks to force correct skill invocation and output validation.

Background

The setup involved a Telegram-based voice interface connected to a Hermes Agent (v0.20.6) running on a DGX Spark. I used a host-side whisper.cpp server with a ggml-large-v3-turbo-q5_0 model for STT (Speech-to-Text). The goal was to have the agent recognize a voice memo, transform it into a structured Markdown document using the voice-memo skill, and save it to /sandbox/notes/memos/.

What happened

In the first round of testing, transcription succeeded 5/5 times, but the voice-memo skill was triggered 0/5 times. In the second round, after changing the trigger word to 메모 (

What I tried

Changing the skill trigger condition in the SKILL.md file to look for the word 메모 (spoken) or 메모: (typed). → The skill still failed to trigger 0/3 times; the Qwen model interpreted 메모 as a command to use the default memory tool rather than the specific skill.
Adding a technical terminology list to the Whisper STT prompt to reduce transcription errors. → Transcription errors for technical terms dropped from 6 to 3 in subsequent tests.

Root cause

The Hermes v0.20.6 gateway only passes transcribed text as a quoted line without any special markers if transcription is successful. The original skill relied on a [The user sent a voice message ...] marker that only appeared during transcription failures. Even when I updated the skill to look for the word 메모, the LLM (Qwen) frequently misidentified the intent as a request to use the standard memory tool for general storage.

The fix

I implemented a custom plugin using two specific Hermes hooks. First, I used pre_gateway_dispatch to intercept messages before they reached the model. For audio, the hook performs its own transcription (taking ~0.7s) to check if the first word is 메모. If it matches, it rewrites the message to include a hard instruction: [voice-memo] This message is the author's memo... Do not use the memory tool.... Second, I added a transform_llm_output hook to intercept the model's response, read the newly saved file, and replace the model's chatty reply with the full document content plus an automated quality check.

Plugin logic for pre-gateway dispatch

if kind == "text" and TYPED.match(text):
    return {"action": "rewrite", "text": f"{INSTRUCTION}\n\n{text}"}
media = list(getattr(event, "media_urls", None) or [])
if kind in ("voice", "audio") and media and SPOKEN.match(_transcribe(media[0]).strip()):
    logger.info("voice_memo: 메모 음성 → voice-memo 스킬 지시")
    return {"action": "rewrite", "text": f"{text}\n\n{INSTRUCTION}" if text else INSTRUCTION}

Automated document quality check logic

def check(doc: str) -> list:
    out = [f"'{s}' 칸 없음" for s in SECTIONS if _section(doc, s) is None]
    summary = _section(doc, "정리") or ""
    missing = [i for i in ITEMS if not re.search(rf"^\s*[-*]\s*\**{re.escape(i)}\**\s*[:：]", summary, re.M)]
    if summary and missing:
        out.append("정리에 빠진 항목: " + ", ".join(missing))
    if (cjk := sorted(set(FOREIGN.findall(doc)))):
        out.append("한자·가나 섞임: " + ", ".join(cjk[:5]))

Result

In the third round of testing, the system achieved 3/3 successful saves and 0 uses of the memory tool. The automated check successfully identified missing sections, the presence of Chinese/Japanese characters, and English words used outside the original context.

Lessons

Don't rely on LLM intent recognition for critical tool selection; use gateway hooks to inject explicit instructions.
When building voice workflows, intercepting the stream before the gateway can provide much more reliable trigger detection.

Still open

The system still experiences 'interrupted' turns if a new voice message arrives while the agent is still processing a previous one due to the busy_input_mode: interrupt setting.

This post was written by a local Gemma 4 model from my own commit log and experiment notes, fact-checked against them by a local Qwen3.6 model, with no human edits.