Self-Hosting an AI Assistant on CasaOS: A Step-by-Step OpenMuse Install Guide (Pitfalls Included)

Murat — Wed, 07 Oct 2026 00:25:00 +0000

How to deploy OpenMuse — a self-hosted AI chat assistant that can also manage your CasaOS server — using SSH and Docker Compose. Every step below exists because we did it the wrong way first.

What you're building

A home server running CasaOS (this guide was tested on v0.4.9) gets a conversational AI assistant living on the same box — one that doesn't just chat, but can list your apps, read logs, and start/stop containers with your approval. Three containers do the work:

| Container | What it is | Port |
|---|---|
| api | The Node API server + agent brain | 8787 |
| web | The chat UI (static site on nginx) | 8081 |
| browser-worker | Sandboxed browser for web tasks | 8790 (loopback only) |

OpenMuse ships no official Dockerfiles and no CasaOS guide, so this post provides both. We'll deploy the casaos-manager-v3 branch, which adds a CasaOS control layer, a document library, and OCR on top of upstream OpenMuse.

Prerequisites

A CasaOS box with Docker and Docker Compose installed (any CasaOS install has both)
SSH access to the box from your own machine (Mac/Linux/Windows terminal)
An OpenRouter API key (or any OpenAI-compatible endpoint)
A CopilotKit Intelligence project key — required in every mode (npx copilotkit@latest login, then npx copilotkit@latest project select)
About 30 minutes, most of it waiting for the build

Step 0: SSH in — don't fight the CasaOS web terminal

The CasaOS dashboard has a built-in terminal, and it will betray you: its xterm.js input silently mangles keystrokes mid-session (typed text arrives as - characters, commands never execute, and retrying doesn't fix it). Do all of the following over SSH from your own machine. If you must use the web terminal, paste whole command blocks — never type long commands by hand.

ssh <user>@<server-lan-ip>

Everything below runs on the server.

Step 1: Make sure ports 8787, 8081, 8790 are free

A previous install attempt (or a half-dead tsx dev server) can leave a stale process holding port 8787. docker compose down does not kill host processes or containers from a different compose project, so check before you start:

ss -tlnp | grep -E '8787|8081|8790'
docker ps -a --format '{{.Names}} {{.Status}}' | grep -i -E 'openmuse|worker'

If anything shows up, stop/remove it now. Every "address already in use" mystery we've seen traced back to this step being skipped.

Step 2: Create the install tree

We'll keep everything under /DATA/AppData/openmuse — the conventional CasaOS app-data location, so backups and future you know where to look.

The Dockerfiles below expect the OpenMuse source tree in a repo/ subdirectory, so clone it there:

sudo mkdir -p /DATA/AppData/openmuse
sudo chown $USER:$USER /DATA/AppData/openmuse
cd /DATA/AppData/openmuse
git clone --branch casaos-manager-v3 https://github.com/Magrebi/openmuse.git repo
mkdir -p data

Why not CasaOS's "Custom Install" button? CasaOS passes the custom-install command field to the container as one single argument. A multi-flag server command (e.g. llama-server -m model.gguf --port 8080) crashes with error: invalid argument. This stack needs real Compose orchestration anyway — SSH + docker compose is the reliable path.

Your tree should look like this:

/DATA/AppData/openmuse/
├── docker-compose.yaml   # you write this (Step 3)
├── Dockerfile.api        # you write this (Step 3)
├── Dockerfile.web        # you write this (Step 3)
├── repo/                 # the OpenMuse source (cloned above)
├── .env                  # you write this (Step 4)
└── data/                 # app database + library files (created at runtime)

Step 3: The three build files

`docker-compose.yaml`

name: openmuse
services:
  api:
    build:
      context: .
      dockerfile: Dockerfile.api
    image: openmuse-api:local
    env_file: .env
    ports:
      - "8787:8787"
    volumes:
      - ./data:/app/.openmuse
    restart: unless-stopped
    healthcheck:
      test: ["CMD", "node", "-e", "fetch('http://127.0.0.1:8787/api/health').then(r=>process.exit(r.ok?0:1)).catch(()=>process.exit(1))"]
      interval: 20s
      timeout: 5s
      retries: 5
      start_period: 60s

  web:
    build:
      context: .
      dockerfile: Dockerfile.web
      args:
        # ⚠️ Baked into the static site at BUILD time.
        # Changing the API address later = rebuilding web. Pick something stable.
        EXPO_PUBLIC_API_URL: http://<server-lan-ip>:8787
    image: openmuse-web:local
    ports:
      - "8081:80"
    restart: unless-stopped
    depends_on:
      - api

  browser-worker:
    build:
      context: ./repo/apps/worker
    image: openmuse-worker:local
    init: true
    restart: unless-stopped
    environment:
      WORKER_HOST: 0.0.0.0
      WORKER_TOKEN: <same-value-as-WORKER_TOKEN-in-.env>
    ports:
      - "127.0.0.1:8790:8790"   # loopback only — the browser never needs LAN exposure
    volumes:
      - worker-data:/data
    tmpfs:
      - /tmp:size=256m,mode=1777
    shm_size: 256mb
    mem_limit: 2g
    pids_limit: 256
    read_only: true
    cap_drop:
      - ALL
    security_opt:
      - no-new-privileges:true
    healthcheck:
      test: ["CMD", "node", "-e", "fetch('http://127.0.0.1:8790/health').then(r=>process.exit(r.ok?0:1)).catch(()=>process.exit(1))"]
      interval: 15s
      timeout: 5s
      retries: 3

volumes:
  worker-data:

Two things worth noticing: the API reaches the worker over Compose's internal DNS (http://browser-worker:8790, set in .env below), and the worker's published port is bound to 127.0.0.1 — browser automation has no business being on your LAN.

`Dockerfile.api`

FROM node:24-bookworm-slim
# Document OCR + PDF text extraction for the library (Tesseract + Poppler).
# Don't pin apt versions: pins that don't exist in this Debian release
# fail the build (we learned this the hard way with tesseract on bookworm).
# If you must pin, verify the version first with `apt-cache policy <pkg>`
# inside the base image.
RUN apt-get update && apt-get install -y --no-install-recommends \
    tesseract-ocr tesseract-ocr-eng tesseract-ocr-tur poppler-utils \
 && rm -rf /var/lib/apt/lists
RUN corepack enable && corepack prepare pnpm@11.19.0 --activate
WORKDIR /app
COPY repo ./
RUN pnpm install --frozen-lockfile
RUN pnpm build:server
ENV PORT=8787 HOST=0.0.0.0 DATA_DIR=/app/.openmuse
EXPOSE 8787
CMD ["node", "dist/apps/server/src/index.js"]

`Dockerfile.web`

FROM node:24-bookworm-slim AS build
RUN corepack enable && corepack prepare pnpm@11.19.0 --activate
WORKDIR /app
COPY repo ./
RUN pnpm install --frozen-lockfile
ARG EXPO_PUBLIC_API_URL
ENV EXPO_PUBLIC_API_URL=$EXPO_PUBLIC_API_URL
RUN pnpm --dir apps/mobile build:web

FROM nginx:alpine
COPY --from=build /app/apps/mobile/dist/web /usr/share/nginx/html
EXPOSE 80

Step 4: The `.env` file — every field explained

Create it on the server with nano (or your editor of choice). Secrets go into this file on the host — never into a chat window, a prompt, or a screenshot.

nano /DATA/AppData/openmuse/.env

WORKSPACE_MODE=live
AGENT_BACKEND=model

# --- Model (OpenRouter example) ---
# "openai/" is OpenMuse's internal routing prefix: everything after it
# is the model id sent to OPENAI_BASE_URL.
MODEL=openai/anthropic/claude-sonnet-4-5
OPENAI_BASE_URL=https://openrouter.ai/api/v1
OPENAI_API_KEY=

CPK_INTELLIGENCE_API_KEY=   # required in every mode

# --- Server ---
PORT=8787
# ⚠️ HOST=0.0.0.0, not your Tailscale or LAN IP.
# Binding to a specific IP broke restarts with EADDRNOTAVAIL when that
# interface wasn't up yet. 0.0.0.0 is the safe choice.
HOST=0.0.0.0
PUBLIC_API_URL=http://<server-lan-ip>:8787
DATA_DIR=/app/.openmuse
ALLOWED_ORIGINS=http://<server-lan-ip>:8081
TASK_WORKER_ENABLED=true
COMPUTER_ENABLED=false

# --- Secrets: generate, don't invent ---
OPENMUSE_ACCESS_KEY=        # ≥24 chars — you type this at first login
TOKEN_ENCRYPTION_KEY=       # 32 random bytes, base64
BROWSER_WORKER_URL=http://browser-worker:8790
WORKER_TOKEN=               # must MATCH the value in docker-compose.yaml

Generate the three secrets on the server:

openssl rand -hex 24     # → OPENMUSE_ACCESS_KEY
openssl rand -base64 32  # → TOKEN_ENCRYPTION_KEY
openssl rand -hex 24     # → WORKER_TOKEN (paste into BOTH .env and docker-compose.yaml)

⚠️ Compose reads .env only when containers are (re)created. Editing .env and running docker compose restart does nothing — you need docker compose up -d so the changed containers are recreated.

Step 5: Build and start

The first build compiles the server and the web UI — expect 10–20 minutes on a modest box. Go make coffee.

cd /DATA/AppData/openmuse
docker compose build
docker compose up -d

Step 6: Verify it's healthy

docker compose ps
curl -s http://127.0.0.1:8787/api/health; echo
curl -s http://127.0.0.1:8790/health; echo

You want: api healthy, web up, browser-worker healthy, and both curl calls answering. If the API container keeps restarting, check docker compose logs api --tail 50 — the usual suspects are a wrong HOST value (Step 4) or a port squatter (Step 1).

Step 7: First login and the CasaOS connection

Open http://<server-lan-ip>:8081 in your browser.
When asked for the access key, enter your OPENMUSE_ACCESS_KEY.
To let the assistant manage CasaOS, connect it in the app's UI (Connections → CasaOS) with your CasaOS username and password. These credentials are encrypted server-side (AES-256-GCM) and verified with a live login before saving — they are never put in .env.

What the CasaOS layer gives you: read-only tools (casaos_list_apps, casaos_app_status, casaos_app_logs, casaos_system_status) plus casaos_start_app / casaos_stop_app / casaos_restart_app — but mutations never execute directly. The tool call only prepares an action; you approve it in the Activity panel, approvals are single-use, and protected apps (OpenMuse itself and anything hosting its network access) are refused outright. Try "which apps are installed?" first, then "stop HandBrake" to see the approval card in action.

The same branch also ships a document library with OCR (Tesseract runs fully on your box — nothing is sent to the cloud) and full-text search over your uploads.

Step 8: Back up before every update

Future updates are: git -C repo pull, docker compose build, docker compose up -d. But back up first, every time:

TS=$(date +%F)
sudo mkdir -p /DATA/AppData/openmuse-backup-$TS
sudo cp -a /DATA/AppData/openmuse/docker-compose.yaml \
           /DATA/AppData/openmuse/Dockerfile.api \
           /DATA/AppData/openmuse/Dockerfile.web \
           /DATA/AppData/openmuse/.env \
           /DATA/AppData/openmuse/data \
           /DATA/AppData/openmuse-backup-$TS/

Two backup lessons from our own scars:

CasaOS's uninstall dialog has a "Delete userdata (config folder)" checkbox. Ticked, it wipes /DATA/AppData/<app> entirely — we lost a whole Jellyfin config this way. Read that checkbox twice before confirming any uninstall.
Rollback is just docker compose down, restoring the backup tree, and docker compose up -d. Because you backed up data/, the app database and library survive the round trip.

Troubleshooting cheat sheet

| Symptom | Root cause | Fix |
|---|---|
| address already in use on 8787 | Stale tsx process or orphan container from an earlier attempt | ss -tlnp \| grep 8787, kill it; docker ps -a for orphans |
| API exits with EADDRNOTAVAIL | HOST bound to an IP that wasn't up at start | HOST=0.0.0.0, then docker compose up -d |
| .env change had no effect | Env is captured at container creation | docker compose up -d (recreate), not restart |
| Web UI talks to the wrong API URL | EXPO_PUBLIC_API_URL is baked at build time | Rebuild web after changing the arg |
| CasaOS dashboard shows OpenMuse as "unknown" | CasaOS control-plane HTTP 500, not your app | Check docker compose ps directly — it's usually fine |
| Script gets Session expired | Raw access key used as a Bearer token | POST /api/session with the key first, use the returned session token |
| apt build fails on a pinned package | That version doesn't exist in bookworm | Drop the pin, or verify with apt-cache policy in the base image |
| error: invalid argument in a container | CasaOS custom-install command passed as one argument | Use docker compose via SSH instead of the CasaOS dialog |

Going further

Remote access: the plan we're settling on is Tailscale Serve with HTTPS (stays on your tailnet, no port forwarding, browser-trusted cert), with the LAN address kept working alongside. That's a post of its own once it's verified end to end.
Second opinions: because this gives an AI a path to mutate server state, we put the CasaOS-manager layer through two independent security reviews before going live. If you extend the tools, get a hostile review before you trust it.

Tested on CasaOS v0.4.9 with OpenMuse built from upstream main. If a step breaks on your box, the troubleshooting table above is where the bodies are buried.

DEV Community: Murat