DEV Community

Cover image for I Made My Self-Hosted AI System Actually Reliable — Building Your Private AI Infrastructure [4/5]
kusunoki
kusunoki

Posted on • Edited on

I Made My Self-Hosted AI System Actually Reliable — Building Your Private AI Infrastructure [4/5]

#security #devops #linux #selfhosted

Free series. All open-source. No DevOps background required. Estimated hands-on time: 60–90 minutes.

There is a moment in every serious build when you realize it is no longer a project. It is infrastructure. You have Nextcloud running. Collabora is live. Four AI models are answering requests through your private proxy. OpenClaw is taking instructions from your phone. The stack works. It does real things. People could depend on this tomorrow.

This part is where a weekend project becomes a production system. Not because the features were missing — Parts 1 through 3 already deliver a working environment — but because production means you can walk away from it on a Friday evening and trust that it will still be running, still be monitored, and still be backed up when you return on Monday morning. That trust is what this part builds.

When you finish today, you will have something that most engineers talk about building and never quite do: a self-hosted, zero-trust, AI-augmented infrastructure that monitors itself, alerts you before failures, backs itself up on three independent schedules, and lets you reach your office workstation from any browser on earth through five layers of authentication. For $35 to $50 a month.

And the thing about building it yourself, one command at a time, is that you understand every piece of it. There is no vendor abstraction you cannot look behind. There is no configuration you cannot change. There is no outage you cannot diagnose because you built the system that is running. That understanding — not the cost savings, not the features — is what changes how you work.

What You Will Complete

Prerequisites: Parts 1–3 complete, all services running. A Gmail account for alerts.

Seven capabilities are added in this part:

1. Apache Guacamole — browser-based RDP gateway. Access any machine on your LAN from any browser, anywhere, through Cloudflare Access.

2. TimeTracker — one-click billable hour logging with CSV export. IRS §6001 contemporaneous records.

3. Calendar/Tasks/Deck — cross-device sync via CalDAV. Single source of truth for all commitments.

4. Accounting API integration — QuickBooks Online, Xero, FreshBooks OAuth 2.0 connections for plain-English financial queries through OpenClaw.

5. Advanced OpenClaw workflows — morning briefings, task-from-email, weekly summaries.

6. Prometheus + Grafana + Alertmanager — metric collection, live dashboard, email alerts before thresholds become incidents.

7. AES-256 encrypted backup — weekly full-system archive. 30-day retention. Documented 2-hour restore.

Step 1: Apache Guacamole

Guacamole is a clientless RDP gateway. It translates Remote Desktop Protocol into browser-renderable HTML5 output. Your Windows desktop appears in a tab, authenticated through five independent security layers:

Browser → WARP encryption → Cloudflare Access OTP → Tunnel → Guacamole on Vultr → your office workstation

mkdir -p ~/guacamole && cd ~/guacamole
nano docker-compose.yml
Enter fullscreen mode Exit fullscreen mode

Paste:

version: '3'
services:
  guacd:
    image: guacamole/guacd
    container_name: guacd
    restart: always

  guacamole-db:
    image: postgres:15
    container_name: guacamole-db
    restart: always
    environment:
      - POSTGRES_DB=guacamole_db
      - POSTGRES_USER=guacamole_user
      - POSTGRES_PASSWORD=REPLACE_WITH_STRONG_PASSWORD
    volumes:
      - guacamole_db_data:/var/lib/postgresql/data

  guacamole:
    image: guacamole/guacamole
    container_name: guacamole
    restart: always
    depends_on:
      - guacd
      - guacamole-db
    environment:
      - GUACD_HOSTNAME=guacd
      - POSTGRESQL_HOSTNAME=guacamole-db
      - POSTGRESQL_DATABASE=guacamole_db
      - POSTGRESQL_USER=guacamole_user
      - POSTGRESQL_PASSWORD=REPLACE_WITH_SAME_PASSWORD
    ports:
      - "127.0.0.1:8888:8080"

volumes:
  guacamole_db_data:
Enter fullscreen mode Exit fullscreen mode

Replace REPLACE_WITH_STRONG_PASSWORD in both locations. Store in Bitwarden.

docker-compose up -d
docker ps
Enter fullscreen mode Exit fullscreen mode

Three containers Up: guacd, guacamole-db, guacamole.

Browse to https://remote.yourdomain.com. Default: guacadmin / guacadmin. Change immediately: username → Settings → Change Password.

Step 2: Preparing the Windows Machine

On the Windows workstation (not your VPS):

Enable Remote Desktop: Settings → System → Remote Desktop → On. (Windows Home: upgrade to Pro ~$100, or install RustDesk as free drop-in.)

Static local IP: Settings → Network → Edit → Manual:

IP: 192.168.1.100   Subnet: 255.255.255.0
Gateway: 192.168.1.1   DNS: 1.1.1.1 / 8.8.8.8
Enter fullscreen mode Exit fullscreen mode

Router port forwarding: forward TCP 3389 to 192.168.1.100. Restrict source to your Vultr IP. Do not set source to "any."

External IP: whatismyip.com. Dynamic IP? Request static from ISP (~$10–15/mo) or use No-IP DDNS (free).

Register in Guacamole: Settings → Connections → New Connection → enter workstation IP/hostname, port 3389, Windows credentials → Save.

Session discipline: Ctrl+Alt+Shift → Disconnect when finished. Closing the tab without disconnecting leaves your desktop unlocked.

Step 3: Time Tracking

Nextcloud → Apps → "TimeTracker" → Enable.

Usage: ▶ Start / ⏸ Pause / ■ Stop. Tag every session. Export CSV monthly.

The IRS requires contemporaneous records under 26 U.S.C. §6001. A tagged time log maintained in real time is materially stronger at audit than a reconstruction from memory. Your future self — and your accountant — will thank you.

Step 4: Calendar, Tasks, and Deck

Already installed in your Nextcloud deployment. No additional installation needed.

Calendar

Create color-coded calendars: Company, Client, Personal, Finance. Pre-populate Finance with the four federal estimated tax dates (Apr 15, Jun 16, Sep 15, Jan 15). Share via three-dot menu → Sharing.

Tasks

Apps → "Tasks" → Enable. Recommended lists: This Week / In Progress / Awaiting Response / Backlog. Syncs to iPhone Reminders and Android OpenTasks via CalDAV.

Deck (Kanban)

Apps → "Deck" → Enable. One board per project. Columns: To Do / In Progress / Review / Done. Due dates auto-appear in Calendar.

Step 5: Accounting API Integration

Standing rule: AI generates drafts. Humans approve. No transaction is committed on AI output alone.

QuickBooks Online

developer.intuit.com → Create App → QBO. Redirect: https://ai.yourdomain.com/callback/qbo. Copy Client ID + Secret.

Append to ~/llm-proxy/.env:

QBO_CLIENT_ID=your-client-id
QBO_CLIENT_SECRET=your-client-secret
QBO_REDIRECT_URI=https://ai.yourdomain.com/callback/qbo
QBO_ENVIRONMENT=production
Enter fullscreen mode Exit fullscreen mode

Auth URL (paste in browser):

https://appcenter.intuit.com/connect/oauth2?client_id=YOUR_CLIENT_ID&redirect_uri=https://ai.yourdomain.com/callback/qbo&response_type=code&scope=com.intuit.quickbooks.accounting&state=secure_random_state
Enter fullscreen mode Exit fullscreen mode

Sign in → Authorize → copy code= from redirect. Exchange:

curl -X POST https://oauth.platform.intuit.com/oauth2/v1/tokens/bearer \
  -H "Accept: application/json" \
  -H "Content-Type: application/x-www-form-urlencoded" \
  -u "YOUR_CLIENT_ID:YOUR_CLIENT_SECRET" \
  -d "grant_type=authorization_code&code=YOUR_AUTH_CODE&redirect_uri=https://ai.yourdomain.com/callback/qbo"
Enter fullscreen mode Exit fullscreen mode

Add access_token, refresh_token, realmId to .env. Verify:

curl -s -H "Authorization: Bearer YOUR_ACCESS_TOKEN" \
  -H "Accept: application/json" \
  "https://quickbooks.api.intuit.com/v3/company/YOUR_REALM_ID/companyinfo/YOUR_REALM_ID"
Enter fullscreen mode Exit fullscreen mode

Expected: JSON with company name.

Xero

developer.xero.com → New App (Web). Redirect: https://ai.yourdomain.com/callback/xero. Same pattern: .env → auth URL → exchange → verify via /Organisation endpoint.

FreshBooks

developer.freshbooks.com → Create Application. Redirect: https://ai.yourdomain.com/callback/freshbooks. Same pattern: .env → auth URL → exchange → verify via /users/me.

Wave

No developer API. Two paths: (1) OpenClaw generates import-ready CSV → review → Wave import, or (2) operate Wave directly through Guacamole remote desktop.

Step 6: Advanced OpenClaw Workflows

With accounting connected and calendar/tasks active, three workflows are immediately practical.

Morning Briefing

Send once to OpenClaw:

"Every weekday 7:30 AM: unread email count + action items, today's calendar, tasks due today, tasks due this week, overdue invoices from [your accounting platform], one-line server status. Deliver to this chat."

Registers and executes daily without further input.

Task from Email

"[Client] requested a revised proposal by April 18. Create task 'Deliver revised proposal' in Client Work, due April 16. Add calendar event."

The obligation moves from inbox to task system. The inbox clears.

Weekly Summary

"Every Friday 5 PM: completed tasks, incomplete tasks due this week, next week's due items, invoices sent/paid, open invoice total, one observation."

Standing rule: EMAIL_DEFAULT = draft_only. OpenClaw drafts email. It never sends autonomously. Configure this before using any email-related workflow.

Step 7: Prometheus + Grafana 

sudo apt install -y prometheus prometheus-node-exporter
sudo apt install -y apt-transport-https software-properties-common
wget -q -O - https://apt.grafana.com/gpg.key | sudo gpg --dearmor -o /usr/share/keyrings/grafana-archive-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/grafana-archive-keyring.gpg] https://apt.grafana.com stable main" | sudo tee /etc/apt/sources.list.d/grafana.list
sudo apt update && sudo apt install -y grafana
Enter fullscreen mode Exit fullscreen mode

Configure:

sudo nano /etc/grafana/grafana.ini
Enter fullscreen mode Exit fullscreen mode

Find and update:

[server]
http_addr = 127.0.0.1

[security]
admin_password = REPLACE_WITH_STRONG_PASSWORD

[users]
allow_sign_up = false

[auth.anonymous]
enabled = false
Enter fullscreen mode Exit fullscreen mode 
sudo systemctl enable --now grafana-server
Enter fullscreen mode Exit fullscreen mode

Browser: https://grafana.yourdomain.com. Connections → Data Sources → Prometheus → URL: http://localhost:9090 → Save & Test. Dashboards → Import → ID 1860 → Load.

Live metrics: CPU, memory, disk, network. Updating in real time. This is the moment your server stops being something you hope is running and becomes something you know is running.

Step 8: Alertmanager

Gmail app password: myaccount.google.com → Security → 2-Step → App passwords → "Alertmanager" → copy 16-char password.

sudo apt install -y prometheus-alertmanager
sudo nano /etc/prometheus/alertmanager.yml
Enter fullscreen mode Exit fullscreen mode

Paste (4 substitutions):

global:
  resolve_timeout: 5m
  smtp_smarthost: 'smtp.gmail.com:587'
  smtp_require_tls: true
  smtp_auth_username: 'youremail@gmail.com'
  smtp_auth_password: '16-char-app-password'
  smtp_from: 'youremail@gmail.com'

route:
  group_by: ['alertname']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  receiver: 'email-alert'

receivers:
  - name: 'email-alert'
    email_configs:
      - to: 'youremail@gmail.com'
        send_resolved: true
Enter fullscreen mode Exit fullscreen mode 
sudo chmod 600 /etc/prometheus/alertmanager.yml
Enter fullscreen mode Exit fullscreen mode

Alert rules:

sudo nano /etc/prometheus/alert.rules.yml
Enter fullscreen mode Exit fullscreen mode 
groups:
  - name: server-alerts
    rules:
      - alert: CloudflaredDown
        expr: up{job="cloudflared"} == 0
        for: 5m
        labels: { severity: critical }
        annotations: { summary: "Cloudflare Tunnel is down" }

      - alert: HighCPU
        expr: 100 - (avg by(instance)(irate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 85
        for: 10m
        labels: { severity: warning }
        annotations: { summary: "CPU above 85% for 10 minutes" }

      - alert: HighMemory
        expr: (1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100 > 85
        for: 10m
        labels: { severity: warning }
        annotations: { summary: "Memory above 85% for 10 minutes" }

      - alert: DiskSpaceLow
        expr: (1 - node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"}) * 100 > 80
        for: 15m
        labels: { severity: warning }
        annotations: { summary: "Disk above 80%" }
Enter fullscreen mode Exit fullscreen mode 
sudo systemctl restart prometheus
sudo systemctl enable --now prometheus-alertmanager
Enter fullscreen mode Exit fullscreen mode

Test:

curl -X POST http://localhost:9093/api/v1/alerts \
  -H "Content-Type: application/json" \
  -d '[{"labels":{"alertname":"TestAlert","severity":"warning"},"annotations":{"summary":"Email alert test — system working"}}]'
Enter fullscreen mode Exit fullscreen mode

Email with subject "TestAlert" arrives within minutes. When it does, monitoring is operational.

Step 9: AES-256 Encrypted Backup 

python3 -c "import secrets; print(secrets.token_urlsafe(48))" > ~/.backup-passphrase
chmod 600 ~/.backup-passphrase
Enter fullscreen mode Exit fullscreen mode

This passphrase is irreplaceable. Lost = encrypted backups permanently unrecoverable. Copy to Bitwarden immediately.

nano ~/backup.sh
Enter fullscreen mode Exit fullscreen mode

Paste (change myadmin if your user differs):

#!/bin/bash
set -euo pipefail
BACKUP_DIR="/home/myadmin/backups"
DATE=$(date +%Y%m%d_%H%M%S)
ARCHIVE="${BACKUP_DIR}/vps-config-${DATE}.tar.gz"
mkdir -p "$BACKUP_DIR"
echo "[$(date)] Encrypted backup started"

sudo tar czf "$ARCHIVE" \
  /home/myadmin/.cloudflared/ \
  /etc/prometheus/ \
  /etc/grafana/grafana.ini \
  /etc/nginx/ \
  /home/myadmin/llm-proxy/.env \
  /home/myadmin/nextcloud/ \
  /home/myadmin/guacamole/ \
  /home/myadmin/.supabase-backup.conf \
  /etc/ssh/sshd_config \
  /etc/ufw/ \
  /etc/fail2ban/ \
  /etc/sysctl.d/99-security.conf \
  /etc/ssl/cloudflare/ \
  2>/dev/null || true

docker run --rm \
  -v nextcloud_nextcloud_data:/data:ro \
  -v "$BACKUP_DIR":/backup \
  alpine tar czf /backup/nextcloud-files-${DATE}.tar.gz -C /data .

gpg --symmetric --cipher-algo AES256 --batch --yes \
  --passphrase-file /home/myadmin/.backup-passphrase "$ARCHIVE"

gpg --symmetric --cipher-algo AES256 --batch --yes \
  --passphrase-file /home/myadmin/.backup-passphrase "${BACKUP_DIR}/nextcloud-files-${DATE}.tar.gz"

rm -f "$ARCHIVE" "${BACKUP_DIR}/nextcloud-files-${DATE}.tar.gz"
find "$BACKUP_DIR" -name "*.gpg" -mtime +30 -delete

echo "[$(date)] Encrypted backup complete"
Enter fullscreen mode Exit fullscreen mode 
chmod 700 ~/backup.sh
~/backup.sh
ls -lh ~/backups/
Enter fullscreen mode Exit fullscreen mode

Expected: two .gpg files — system config + Nextcloud data.

Schedule weekly. Your crontab should now have both:

crontab -e
Enter fullscreen mode Exit fullscreen mode 
# Daily 2 AM — Supabase DB backup (Part 3)
0 2 * * * /home/myadmin/db-backup-to-supabase.sh >> /home/myadmin/db-backup.log 2>&1

# Weekly Sunday 3 AM — encrypted file backup (Part 4)
0 3 * * 0 /home/myadmin/backup.sh >> /home/myadmin/backup.log 2>&1
Enter fullscreen mode Exit fullscreen mode

Recovery from total server loss:

# Restore config
gpg --decrypt --batch --passphrase-file ~/.backup-passphrase \
  ~/backups/latest-vps-config.gpg > /tmp/restore-config.tar.gz
sudo tar xzf /tmp/restore-config.tar.gz -C /

# Restore Nextcloud files
gpg --decrypt --batch --passphrase-file ~/.backup-passphrase \
  ~/backups/latest-nextcloud-files.gpg > /tmp/restore-files.tar.gz
docker run --rm -v nextcloud_nextcloud_data:/data -v /tmp:/backup \
  alpine tar xzf /backup/restore-files.tar.gz -C /data

# Restart
cd ~/nextcloud && docker-compose restart
cd ~/guacamole && docker-compose restart
sudo systemctl restart prometheus grafana-server prometheus-alertmanager nginx cloudflared
Enter fullscreen mode Exit fullscreen mode

Bare server to fully operational: approximately two hours.

Verification Checklist 

docker ps
Enter fullscreen mode Exit fullscreen mode

All containers Up (Nextcloud stack + Guacamole stack).

https://remote.yourdomain.com → Guacamole loads. Click connection → Windows desktop appears.

TimeTracker: Start → 30s → Stop. Session logged. Task with tomorrow's due date → appears on phone. Deck card with due date → appears in Calendar.

Accounting: curl verify returns JSON with company data.

https://grafana.yourdomain.com → live CPU/memory/disk/network.

Alertmanager test email arrives in Gmail.

~/backup.sh && ls -lh ~/backups/
Enter fullscreen mode Exit fullscreen mode

Two .gpg files with current timestamp.

crontab -l
Enter fullscreen mode Exit fullscreen mode

Both cron entries visible.

All checked: Part 4 is complete. The build is finished.

What You Have Built — and What It Changes

Four parts. Two weekends. One stack.

A Vultr server ($12–$48/month) behind eight zero-trust security layers, invisible to the public internet except through Cloudflare's authenticated access. A private file cloud with collaborative editing. Four AI assistants through one portal. An agentic butler that organizes your files, drafts your correspondence, and delivers your briefings while you sleep. Browser-based remote desktop from anywhere on earth. Real-time monitoring with email alerts. Triple-redundant backup with a documented two-hour full recovery.

Total: ~$35–$50/month for a 3–8 person team. The equivalent SaaS stack: $300–$400/month per team — with your data on someone else's servers, under terms that change without notice.

How Your Work Changes

Monday morning: your phone buzzes at 7:30 AM with a briefing you did not write — unread emails categorized, overdue invoices flagged, calendar summarized, tasks prioritized. You scan it in 90 seconds over coffee. Your week is already organized before you sit down.

A client emergency at 2 PM: they need a file from the machine sitting in your office, 30 minutes away. You open a browser tab, authenticate through Cloudflare, and your office desktop materializes on your screen. You send the file. Your client assumes you were at your desk. Total elapsed time: four minutes.

Friday evening: your weekly summary arrives. Tasks completed: 14. Overdue: 0. Open invoices: two, both under 15 days. Server status: all green. Backup completed at 3 AM as scheduled. You close the laptop. The system runs through the weekend without you.

The tax preparer in April: you hand them a CSV of tagged, timestamped billable hours, a Nextcloud folder of YYYYMMDD-named receipts with full version history, and a PostgreSQL audit trail that satisfies 26 U.S.C. §6001. They ask where you found a bookkeeper this organized. You didn't. You built a system.

How Your Business Changes

You stop paying $131–$175 per user per month for tools you do not own, storing data on servers you cannot inspect, under terms you did not negotiate.

You stop wondering whether your VPN is the weakest link in your security — because you eliminated the VPN entirely and replaced it with eight independent layers that follow NIST SP 800-207.

You stop choosing between AI providers — because you have all four, at API pricing, through one portal, matched to each task.

You gain something that SaaS cannot sell you: the knowledge that your data is on your server, your security is under your control, and your infrastructure continues running whether or not any particular vendor survives next quarter.

That is not a feature. That is sovereignty.

What Part 5 Covers

The system is built. Part 5 is the operations manual: monthly and annual maintenance checklists, the complete LLM proxy application code, OpenClaw configuration templates, the emergency response runbook, device loss procedures, and the monthly AI cost audit.

When Part 5 is complete, the series is complete.

Series: Building Your Private AI Infrastructure

Part What It Covers
Part 1 — Architecture Overview Stack, costs, security model
Part 2 — Zero-Trust Server Vultr, Cloudflare, UFW, fail2ban
Part 3 — The Intelligence Layer Docker, Nextcloud, Collabora, AI proxy, OpenClaw
Part 4 — Operations & Monitoring (you are here) Guacamole, Prometheus, Grafana, encrypted backups
Part 5 — The Operations Manual Maintenance, audits, cost optimization, runbook

All five parts are published and free. No paywall, no signup, no follow-up sequence.

Legal Disclaimer

The information provided in this series is for educational and informational purposes only. It does not constitute legal, financial, tax, accounting, cybersecurity, or professional advice. All use is at the sole risk of the user. To the maximum extent permitted by applicable law, including the laws of the State of California (Cal. Civ. Code §§1668, 3513) and the State of New York (N.Y. GOL §5-323), the author disclaims all liability for any damages arising from use of this content. References to all third-party products are for informational purposes only. The author has no commercial relationship with any provider mentioned.

Part 5 — the final installment — is next. Questions, errors, configuration issues: comments. Every one gets read. Technical ones get detailed answers.

— Kusunoki
International Tax Specialist & Systems Builder
Sapporo, Japan | @kusunoki

Tags: devops, linux, security, selfhosted

Top comments (0)