Monitor websites changes with Firecrawl Observer and Docker

Prerequisites

• Linux/Unix.
• Docker & Docker compose
• Firecrawl Observer
• Convex account with production deployment

Deployment Guide

Step 1: Clone, initialse and prepare the environment

git clone https://github.com/mendableai/firecrawl-observer.git
cd firecrawl-observer
npm install
npx convex dev

Open a new terminal

# Here we set up authentication for production
npx @convex-dev/auth --prod

# Set encryption key (REQUIRED - only run if not already set)
npx convex env set ENCRYPTION_KEY "$(openssl rand -base64 32)" --prod

# Verify all required variables are set
npx convex env list --prod

Required variables should include:

• SITE_URL ✓ (already set)
• JWT_PRIVATE_KEY ✓ (already set)
• JWKS ✓ (already set)
• ENCRYPTION_KEY (set this now)

Deploy Convex functions to production

npx convex deploy --prod

Step 2: Update Environment File. Replace YOUR_PUBLIC_URL with the domain provided by convex.

# Create/update .env file
cat > .env << 'EOF'
NEXT_PUBLIC_CONVEX_URL=YOUR_PUBLIC_URL
NODE_ENV=production
EOF

Step 3: Update Dockerfile

# Stage 1: Build the application
FROM node:20-slim AS builder
WORKDIR /app

# Declare build-time arguments
ARG NEXT_PUBLIC_CONVEX_URL
ARG NODE_ENV=production

# Set environment variables for build
ENV NEXT_PUBLIC_CONVEX_URL=$NEXT_PUBLIC_CONVEX_URL
ENV NODE_ENV=$NODE_ENV

# Install dependencies
COPY package.json yarn.lock* package-lock.json* pnpm-lock.yaml* ./
RUN \
    if [ -f yarn.lock ]; then yarn --frozen-lockfile; \
    elif [ -f package-lock.json ]; then npm ci --only=production; \
    elif [ -f pnpm-lock.yaml ]; then yarn global add pnpm && pnpm i --frozen-lockfile --prod; \
    else echo "Lockfile not found." && exit 1; \
    fi

# Copy source and build
COPY . .
RUN npm run build

# Stage 2: Production runtime
FROM node:20-slim AS runner
WORKDIR /app

# Install curl for health checks
RUN apt-get update && apt-get install -y curl && rm -rf /var/lib/apt/lists/*

# Create non-root user
RUN addgroup --system --gid 1001 nodejs && \
    adduser --system --uid 1001 nextjs

# Copy built application with proper ownership
COPY --from=builder /app/public ./public
COPY --from=builder --chown=nextjs:nodejs /app/.next ./.next
COPY --from=builder /app/node_modules ./node_modules
COPY --from=builder /app/package.json ./package.json

# Switch to non-root user
USER nextjs

# Expose port and set environment
EXPOSE 3000
ENV PORT=3000
ENV NODE_ENV=production

# Health check
HEALTHCHECK --interval=30s --timeout=10s --start-period=40s --retries=3 \
    CMD curl -f http://localhost:3000/api/health || exit 1

# Start application
CMD ["npm", "start"]

Step 4: Update docker-compose.yml (Production Ready)

version: '3.8'

services:
  app:
    image: firecrawl-observer-app
    build:
      context: .
      args:
        - NEXT_PUBLIC_CONVEX_URL=${NEXT_PUBLIC_CONVEX_URL}
        - NODE_ENV=production
    restart: unless-stopped
    env_file:
      - .env
    networks:
      - app-network
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:3000/api/health"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 40s
    # Resource limits (adjust based on your server)
    deploy:
      resources:
        limits:
          memory: 1G
          cpus: '0.5'
        reservations:
          memory: 512M
          cpus: '0.25'

  nginx:
    image: nginx:alpine
    restart: unless-stopped
    ports:
      - "5001:80"
    volumes:
      - ./nginx/nginx.conf:/etc/nginx/conf.d/default.conf:ro
      - ./nginx/nginx.htpasswd:/etc/nginx/conf.d/nginx.htpasswd:ro
    depends_on:
      app:
        condition: service_healthy
    networks:
      - app-network
    # Security and performance
    deploy:
      resources:
        limits:
          memory: 128M
          cpus: '0.1'

networks:
  app-network:
    driver: bridge

# Optional: Volumes for persistence
volumes:
  nginx_cache:

Step 5: Optimize Nginx Configuration

# Update nginx.conf with production optimizations
cat > nginx/nginx.conf << 'EOF'
# Upstream definition
upstream app_backend {
    server app:3000;
    keepalive 32;
}

server {
    listen 80;
    server_name _;

    # Security headers
    add_header X-Frame-Options "SAMEORIGIN" always;
    add_header X-Content-Type-Options "nosniff" always;
    add_header X-XSS-Protection "1; mode=block" always;
    add_header Referrer-Policy "strict-origin-when-cross-origin" always;

    # Gzip compression
    gzip on;
    gzip_vary on;
    gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript;

    # Basic auth for all routes
    auth_basic "Restricted Content";
    auth_basic_user_file /etc/nginx/conf.d/nginx.htpasswd;

    # Health check endpoint (bypass auth)
    location /api/health {
        access_log off;
        proxy_pass http://app_backend;
        proxy_http_version 1.1;
        proxy_set_header Connection "";
    }

    # Main application
    location / {
        proxy_pass http://app_backend;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;

        # Timeouts
        proxy_connect_timeout 60s;
        proxy_send_timeout 60s;
        proxy_read_timeout 60s;

        # Buffer settings
        proxy_buffering on;
        proxy_buffer_size 8k;
        proxy_buffers 8 8k;

        # Cache static assets
        location ~* \.(js|css|png|jpg|jpeg|gif|ico|svg|woff|woff2|ttf|eot)$ {
            expires 1y;
            add_header Cache-Control "public, immutable";
            proxy_pass http://app_backend;
        }
    }
}
EOF

Step 6: Add Health Check Endpoint

# Create health check API route
mkdir -p src/app/api/health
cat > src/app/api/health/route.ts << 'EOF'
import { NextResponse } from 'next/server';

export async function GET() {
  try {
    // Basic health check - can be expanded to check dependencies
    return NextResponse.json({ 
      status: 'healthy',
      timestamp: new Date().toISOString(),
      uptime: process.uptime(),
      environment: process.env.NODE_ENV || 'development'
    }, { status: 200 });
  } catch (error) {
    return NextResponse.json({ 
      status: 'unhealthy',
      error: 'Health check failed',
      timestamp: new Date().toISOString()
    }, { status: 503 });
  }
}
EOF

Step 7: Deploy with Zero-Downtime Strategy

# Stop current containers gracefully
docker-compose down

# Clean up old images (optional, saves disk space)
docker image prune -f

# Build and deploy with improved logging
docker-compose up --build -d

# Monitor startup
echo "Waiting for services to start..."
sleep 10

# Check service health
docker-compose ps
docker-compose logs --tail=50 app

Step 8: Verification and Monitoring. Replace SERVER_IP & PORT with yours

# Test health endpoint
curl -I http://SERVER_IP:PORT/api/health

# Test full application (will prompt for basic auth)
curl -I http://SERVER_IP:PORT/

# Monitor logs in real-time
docker-compose logs -f --tail=20

# Check resource usage
docker stats --no-stream

Production Monitoring Script

# Create monitoring script
cat > monitor.sh << 'EOF'
#!/bin/bash
echo "=== Container Status ==="
docker-compose ps

echo -e "\n=== Health Check ==="
curl -s http://localhost:3000/api/health | jq '.' 2>/dev/null || echo "Health check failed"

echo -e "\n=== Resource Usage ==="
docker stats --no-stream --format "table {{.Container}}\t{{.CPUPerc}}\t{{.MemUsage}}\t{{.MemPerc}}"

echo -e "\n=== Recent Logs ==="
docker-compose logs --tail=5 app
EOF

chmod +x monitor.sh

Automated Deployment Script

# Create deployment script for future updates
cat > deploy.sh << 'EOF'
#!/bin/bash
set -e

echo "Starting deployment..."

# Pull latest changes (if using git)
# git pull origin main

# Deploy Convex functions
echo "Deploying Convex functions..."
npx convex deploy --prod

# Build and deploy containers
echo "Building and deploying containers..."
docker-compose down
docker-compose up --build -d

# Wait for health check
echo "Waiting for application to be healthy..."
for i in {1..30}; do
    if curl -f http://localhost:3000/api/health >/dev/null 2>&1; then
        echo "Application is healthy!"
        break
    fi
    echo "Waiting... ($i/30)"
    sleep 10
done

echo "Deployment complete!"
docker-compose ps
EOF

chmod +x deploy.sh

Security Checklist

• ✅ Non-root containers: App runs as user nextjs
• ✅ Basic authentication: Password protection via nginx
• ✅ Security headers: XSS, CSRF, and frame protection
• ✅ Resource limits: Memory and CPU constraints
• ✅ Health checks: Automated container health monitoring
• ✅ Encrypted secrets: API keys stored in Convex cloud
• ⚠️ HTTPS: Consider adding SSL/TLS for production
• ⚠️ Firewall: Ensure only necessary ports are open

Troubleshooting

Quick Diagnostics

# Full system check
./monitor.sh

# View all logs
docker-compose logs

# Restart specific service
docker-compose restart app

# Force rebuild
docker-compose up --build --force-recreate -d

Common Issues

1. Auth errors: Check Convex environment variables with npx convex env list --prod
2. Build failures: Verify .env file and NEXT_PUBLIC_CONVEX_URL
3. Health check failures: Check if port 3000 is accessible inside container
4. nginx issues: Verify nginx.htpasswd file permissions

If you need a cloud server: $200 Digital Ocean Credit