DEV Community

Cover image for Production AI Wallet Monitoring: Docker Healthcheck and Service Recovery
Wallet Guy
Wallet Guy

Posted on

Production AI Wallet Monitoring: Docker Healthcheck and Service Recovery

#ai #docker #devops #selfhosted

Running AI agents with crypto wallets in production means your services need to stay up 24/7. When your Docker containers crash or your daemon becomes unresponsive, your agents lose access to funds and DeFi positions — potentially costing thousands in missed opportunities or liquidations. WAIaaS provides production-grade monitoring and recovery tools to keep your self-hosted wallet infrastructure running smoothly.

Why Production Monitoring Matters

Self-hosting wallet infrastructure gives you complete control over your private keys and eliminates dependency on third-party services. But with that control comes responsibility for uptime and reliability. Unlike hosted solutions where someone else handles service recovery, your self-hosted WAIaaS daemon needs robust monitoring to detect failures and restart automatically.

Production AI trading bots can't afford downtime. A crashed wallet service during high volatility means missed arbitrage opportunities, failed liquidation protection, or stuck DeFi positions. Your monitoring setup becomes as critical as your trading strategy.

Built-in Docker Healthchecks

WAIaaS includes comprehensive healthcheck monitoring in its Docker configuration. The daemon container automatically tests its own health and reports status to Docker's monitoring system.

Here's the healthcheck configuration from the official docker-compose.yml:

services:
  daemon:
    image: ghcr.io/minhoyoo-iotrust/waiaas:latest
    container_name: waiaas-daemon
    ports:
      - "127.0.0.1:3100:3100"
    volumes:
      - waiaas-data:/data
    environment:
      - WAIAAS_DATA_DIR=/data
      - WAIAAS_DAEMON_HOSTNAME=0.0.0.0
    restart: unless-stopped
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:3100/health"]
      interval: 30s
      timeout: 5s
      start_period: 10s
      retries: 3
Enter fullscreen mode Exit fullscreen mode

The healthcheck pings the /health endpoint every 30 seconds. If three consecutive checks fail, Docker marks the container as unhealthy. Combined with restart: unless-stopped, this automatically restarts failed containers.

You can monitor container health status with:

docker compose ps                    # Show service status
docker compose logs -f daemon        # Follow daemon logs
docker inspect waiaas-daemon | grep Health  # Detailed health info
Enter fullscreen mode Exit fullscreen mode

Service Recovery Strategies

Automatic Restart Policies

WAIaaS containers use restart: unless-stopped, which handles most failure scenarios:

  • Process crash → immediate restart
  • Out of memory → restart after cleanup
  • Network connectivity loss → restart when network returns
  • Host reboot → automatic startup

For more aggressive recovery, you can use restart: always:

restart: always  # Restart even if manually stopped
Enter fullscreen mode Exit fullscreen mode

Watchtower Auto-Updates

Keep your WAIaaS daemon current with automatic updates using Watchtower:

# Add to docker-compose.yml
watchtower:
  image: containrrr/watchtower
  volumes:
    - /var/run/docker.sock:/var/run/docker.sock
  environment:
    - WATCHTOWER_POLL_INTERVAL=3600  # Check hourly
    - WATCHTOWER_CLEANUP=true
  restart: unless-stopped
Enter fullscreen mode Exit fullscreen mode

Watchtower pulls the latest ghcr.io/minhoyoo-iotrust/waiaas:latest image and restarts your daemon with zero-downtime updates.

External Health Monitoring

For multi-server setups or external alerting, monitor the health endpoint:

# Simple health check script
#!/bin/bash
HEALTH_URL="http://127.0.0.1:3100/health"
if ! curl -f -s "$HEALTH_URL" > /dev/null; then
    echo "WAIaaS daemon unhealthy - restarting"
    docker compose restart daemon
    # Send alert to your monitoring system
fi
Enter fullscreen mode Exit fullscreen mode

Set this up as a cron job for additional monitoring beyond Docker's built-in healthcheck.

Data Persistence and Recovery

WAIaaS uses named Docker volumes to persist wallet data across container restarts:

volumes:
  waiaas-data:
    driver: local
Enter fullscreen mode Exit fullscreen mode

This ensures your private keys, session tokens, and transaction history survive container updates and restarts. Your wallets remain accessible even after system reboots.

For backup and disaster recovery:

# Backup wallet data
docker run --rm -v waiaas-data:/data -v $(pwd):/backup alpine tar czf /backup/waiaas-backup.tar.gz /data

# Restore from backup
docker run --rm -v waiaas-data:/data -v $(pwd):/backup alpine tar xzf /backup/waiaas-backup.tar.gz -C /
Enter fullscreen mode Exit fullscreen mode

Store encrypted backups offsite to recover from complete hardware failure.

Production Environment Configuration

For production deployments, use Docker secrets to protect sensitive configuration:

# Create secrets
mkdir -p secrets
echo "your-secure-master-password" > secrets/master_password.txt
chmod 600 secrets/master_password.txt
Enter fullscreen mode Exit fullscreen mode

Deploy with the production secrets overlay:

docker compose -f docker-compose.yml -f docker-compose.secrets.yml up -d
Enter fullscreen mode Exit fullscreen mode

This keeps your master password out of environment variables and docker-compose files.

Log Management and Debugging

WAIaaS provides structured JSON logging for production monitoring:

# Set log level
docker compose up -d -e WAIAAS_DAEMON_LOG_LEVEL=info

# Follow logs with timestamps
docker compose logs -f --timestamps daemon

# Export logs for analysis
docker compose logs --since=24h daemon > waiaas-logs.txt
Enter fullscreen mode Exit fullscreen mode

Configure log rotation to prevent disk space issues:

services:
  daemon:
    logging:
      driver: "json-file"
      options:
        max-size: "10m"
        max-file: "3"
Enter fullscreen mode Exit fullscreen mode

Integration with External Monitoring

WAIaaS exposes metrics through its REST API that integrate with monitoring systems:

# Check daemon status
curl -s http://127.0.0.1:3100/health | jq

# Monitor wallet balances
curl -s http://127.0.0.1:3100/v1/wallet/balance \
  -H "Authorization: Bearer $WAIAAS_SESSION_TOKEN"

# Check pending transactions
curl -s http://127.0.0.1:3100/v1/transactions?status=pending \
  -H "Authorization: Bearer $WAIAAS_SESSION_TOKEN"
Enter fullscreen mode Exit fullscreen mode

Integrate these endpoints with Prometheus, Grafana, or your preferred monitoring stack to track wallet performance and detect issues before they impact your AI agents.

Quick Start: Production Monitoring Setup

Here's how to deploy WAIaaS with comprehensive monitoring in 5 steps:

  1. Clone and configure: 
git clone https://github.com/minhoyoo-iotrust/WAIaaS.git
cd WAIaaS
Enter fullscreen mode Exit fullscreen mode
  1. Set up production secrets: 
mkdir -p secrets
echo "your-secure-password" > secrets/master_password.txt
chmod 600 secrets/master_password.txt
Enter fullscreen mode Exit fullscreen mode
  1. Deploy with monitoring: 
docker compose -f docker-compose.yml -f docker-compose.secrets.yml up -d
Enter fullscreen mode Exit fullscreen mode
  1. Verify health monitoring: 
docker compose ps  # Should show "healthy" status
curl -f http://127.0.0.1:3100/health
Enter fullscreen mode Exit fullscreen mode
  1. Set up backup automation: 
# Add to crontab
0 2 * * * docker run --rm -v waiaas-data:/data -v /backup:/backup alpine tar czf /backup/waiaas-$(date +%Y%m%d).tar.gz /data
Enter fullscreen mode Exit fullscreen mode

Your self-hosted WAIaaS daemon now includes automatic restart, health monitoring, secure configuration, and data backup.

What's Next

Production wallet infrastructure requires ongoing monitoring and maintenance, but WAIaaS handles the heavy lifting with built-in Docker healthchecks and recovery mechanisms. Your AI agents get reliable access to funds while you maintain complete control over your private keys.

Ready to deploy production-grade wallet infrastructure? Check out the complete setup guide at GitHub or explore advanced configuration options at waiaas.ai.

Top comments (0)