AI Agent DevOps: Health Checks, Alerts, Runbooks & Cost Control for a One-Person Company

The Pain: Your agent died at 3 AM. No alert, no logs, nobody knew. It quietly failed inside one API call, the error swallowed by a retry loop. The next morning you open the dashboard — "Tool not responding."
What You'll Learn: The full ops stack for AI agents — 3-layer health checks, P0–P3 alert levels, incident runbooks, token & cost governance, and a one-click Prometheus + Grafana deployment you can run on a single server.

---

Agent DevOps closed loop: Detect → Alert → Recover → Learn

0. Prerequisites

Dependency	Version	Install
Ubuntu	22.04 LTS	system install
Python	≥ 3.10	apt install python3
pip package	requests	pip install requests
Docker	≥ 24.x	see §6.1 for install steps

All commands and scripts in this article were verified on Ubuntu 22.04. Other Linux distributions (Debian 12, CentOS Stream 9) may need to adjust package-manager commands. macOS users can use Homebrew instead of apt.

---

1. Why Your Agent Needs Ops — It Dies Silently and Nobody Knows

Let me start with a real incident.

My agent "Yuanbao" ran a data-collection task — every morning at 2 AM it scraped admission data from 91 universities, cleaned it, compared it, and produced a report. The first week went fine. Starting the second week, every report I received was missing data from 3–5 universities.

The investigation was agonizing:

• Checked the code logic — fine
• Checked API limits — normal
• Checked network connectivity — stable

It wasn't until I went through the agent's full execution log that the truth surfaced: university #37 had changed its website's API format. The agent failed to parse the response and fell into an infinite retry loop. On the 8th retry, the token-budget exhaustion policy triggered — but I had configured "budget exhausted" to silently skip. So the agent quietly skipped every remaining university, moved on to the "generate report" step, and produced a report that looked complete but was actually missing 3 universities.

There were several fatal problems here:

Problem	Consequence
No real-time alerting	I found out the next morning — missed an 8-hour fix window
Retry mechanism had no cap	Resources consumed indefinitely
Exceptions silently swallowed	The agent thought everything was normal
No execution-progress tracking	Nobody knew which step went wrong

And this isn't a one-off. Once your agent system grows past 3 independent tasks, "something is wrong somewhere" becomes the norm, not the exception. If your ops posture is still "I'll take a look every few hours," you will get a question from a user at some ungodly hour — or worse, no user will notice, but your API bill will have quietly tripled.

That's why we put full ops monitoring on agents. Not because you're bored — because you'll be forced to do it eventually anyway. Do it proactively and the pain is manageable. Do it reactively and the damage isn't.

---

2. Health Checks — Three Layers: Port / API / Business Logic

The single most important thing in ops is knowing whether your system is alive right now.

In traditional DevOps, health checks are standard: Nginx has a /health endpoint, PostgreSQL has pg_isready, Kubernetes has Liveness and Readiness probes. But most AI Agent projects skip this layer entirely.

They think "an agent isn't a web service, it doesn't need health checks." That's completely wrong.

An agent is a service — its "interfaces" just aren't HTTP. They're tool-call chains, LLM response quality, and the correctness of state-machine transitions. If none of that is checkable, you're flying blind.

I split agent health checks into three layers, each catching a different way of dying.

Layer 1: Port & Infrastructure Health (Liveness)

This is the most fundamental check. No matter how elegant your agent code is, if the infrastructure under it dies, everything is moot.

Check target	Method	Expected state	Failure consequence
Agent process alive	ps aux / systemd status	RUNNING	process crashed
API gateway port	curl localhost:PORT/health	200 OK	route unreachable
Database connection	SELECT 1 succeeds	success	state can't persist
Message queue	produce-consume test	flows normally	backlog or lost tasks
Disk space	df -h	usage < 85%	log writes fail
Memory	free -m	available > 500MB	OOM kills the process

Implementation:

#!/bin/bash
# layer1_liveness.sh — infrastructure liveness check

set -euo pipefail

check_port() {
    local port="$1"
    local name="$2"
    curl -sf --max-time 5 "http://localhost:${port}/health" > /dev/null 2>&1
    local ret=$?
    if [ $ret -eq 0 ]; then
        echo "[OK] $name (port $port) is alive"
    else
        echo "[FAIL] $name (port $port) is DOWN"
    fi
    return $ret
}

check_db() {
    local name="${1:-PostgreSQL}"
    timeout 5 psql -c "SELECT 1" > /dev/null 2>&1
    local ret=$?
    if [ $ret -eq 0 ]; then
        echo "[OK] $name is reachable"
    else
        echo "[FAIL] $name is unreachable"
    fi
    return $ret
}

check_disk() {
    local mount="${1:-/}"
    local threshold="${2:-85}"
    local usage
    usage=$(df -h "$mount" | awk 'NR==2 {print $5}' | tr -d '%')
    if [ "$usage" -lt "$threshold" ]; then
        echo "[OK] Disk $mount usage: ${usage}% (threshold: ${threshold}%)"
        return 0
    else
        echo "[FAIL] Disk $mount usage: ${usage}% exceeds ${threshold}%"
        return 1
    fi
}

check_memory() {
    local threshold_mb="${1:-500}"
    local available
    available=$(free -m | awk 'NR==2 {print $7}')
    if [ "$available" -gt "$threshold_mb" ]; then
        echo "[OK] Available memory: ${available}MB (threshold: ${threshold_mb}MB)"
        return 0
    else
        echo "[FAIL] Available memory: ${available}MB below ${threshold_mb}MB"
        return 1
    fi
}

failed=0

check_port 8080 "Agent API" || ((failed++))
check_port 6379 "Redis" || ((failed++))
check_db "PostgreSQL" || ((failed++))
check_disk "/" 85 || ((failed++))
check_memory 500 || ((failed++))

echo "-----------------------------------"
echo "Total checks failed: $failed"
exit $failed

Key principle: Layer 1 checks must run outside the agent code, independently. If your agent process checks its own ports — when it dies, the check dies with it. Use cron, a systemd timer, or a separate external monitoring service.

Layer 2: API & Dependency Health (Readiness)

This layer checks whether the external services and APIs the agent depends on are reachable and returning what's expected. It's the most overlooked blind spot in agent ops.

An agent isn't like a traditional web service — its dependencies aren't just a database. They include:

• LLM APIs (OpenAI / Claude / local models)
• Search engine APIs
• Third-party data-source APIs
• Microservices called by internal tools
• File storage systems

If any dependency breaks, the agent runs in a "looks normal, actually useless" state.

Check strategy:

#!/usr/bin/env python3
# layer2_readiness.py — API & dependency health check

import os
import sys
import json
import requests
from datetime import datetime

# Read sensitive info from environment variables
OPENAI_API_KEY = os.environ.get("OPENAI_API_KEY", "")

checks = {
    "llm_api": {
        "url": "https://api.openai.com/v1/models",
        "method": "GET",
        "headers": {"Authorization": f"Bearer {OPENAI_API_KEY}"},
        "expected_status": 200,
        "timeout": 10,
    },
    "vector_db": {
        "url": "http://localhost:6333/health",
        "method": "GET",
        "expected_status": 200,
        "timeout": 5,
    },
    "data_source": {
        "url": "https://api.university-data.org/ping",
        "method": "GET",
        "expected_status": [200, 204],
        "timeout": 15,
    }
}

results = {}
overall_pass = True

for name, config in checks.items():
    try:
        resp = requests.request(
            method=config["method"],
            url=config["url"],
            headers=config.get("headers", {}),
            timeout=config["timeout"]
        )
        expected = config.get("expected_status", 200)
        if isinstance(expected, list):
            passed = resp.status_code in expected
        else:
            passed = resp.status_code == expected
        results[name] = {
            "status": "PASS" if passed else "FAIL",
            "http_code": resp.status_code,
            "checked_at": datetime.now().isoformat()
        }
        if not passed:
            overall_pass = False
    except Exception as e:
        results[name] = {
            "status": "FAIL",
            "error": str(e),
            "checked_at": datetime.now().isoformat()
        }
        overall_pass = False

output = {
    "overall": "PASS" if overall_pass else "FAIL",
    "checks": results
}

print(json.dumps(output, indent=2, ensure_ascii=False))
sys.exit(0 if overall_pass else 1)

Practical advice: run Layer 2 checks every 30 seconds. LLM API availability fluctuates a lot — the earlier you detect an outage (or throttling), the sooner you can switch to a fallback model or a degradation strategy.

Layer 3: Business Logic & Data Quality Health (Deep Health)

This layer is unique to agents — traditional web services don't have this dimension. It checks whether the agent is working correctly, not just whether it's alive.

Check	Method	What an anomaly means
Task completion rate	compare "planned executions" vs "actual completions"	agent may be silently skipping tasks
Output format compliance	validate JSON schema after every LLM response	agent produced an unexpected output format
Data quality drift	check null/outlier ratio in output data	silent failure in the collection chain
State-machine distribution	share of tasks stuck in "executing"	deadlock or infinite loop
Token consumption rate	compare against historical baseline	infinite retries or context bloat

Implementation: on every state-machine transition, the agent writes a record to a health-metrics store (Prometheus / InfluxDB / even a plain JSON file). A separate Deep Health Check process queries those metrics periodically and computes a health score.

#!/usr/bin/env python3
# layer3_deep_health.py — business health scoring

import json
import statistics
from datetime import datetime, timedelta

def detect_token_spikes(state_history, window_minutes=5, z_threshold=3.0):
    """Detect anomalous token-consumption spikes"""
    now = datetime.now()
    cutoff = now - timedelta(minutes=30)

    recent_tokens = []
    for s in state_history:
        ts = datetime.fromisoformat(s["timestamp"])
        if ts > cutoff:
            recent_tokens.append(s.get("tokens_consumed", 0))

    if len(recent_tokens) < 6:
        return 0

    mean = statistics.mean(recent_tokens)
    stdev = statistics.stdev(recent_tokens) if len(recent_tokens) > 1 else 1.0
    recent = recent_tokens[-window_minutes:] if len(recent_tokens) >= window_minutes else recent_tokens

    spikes = sum(1 for t in recent if (t - mean) > (z_threshold * stdev))
    return spikes

def compute_health_score(state_history):
    """Compute a business health score (0-100) from state-machine history"""
    score = 100

    total_tasks = len(state_history)
    if total_tasks == 0:
        return 100  # no tasks = healthy

    stuck_tasks = sum(
        1 for s in state_history
        if s.get("state") == "executing" and s.get("duration_min", 0) > 30
    )
    failed_tasks = sum(1 for s in state_history if s.get("state") == "failed")

    # -5 per stuck task
    score -= stuck_tasks * 5
    # -10 per failed task
    score -= failed_tasks * 10

    # Token anomaly detection (5 consecutive minutes > 3σ off baseline)
    token_spikes = detect_token_spikes(state_history)
    score -= token_spikes * 5

    return max(0, min(100, score))

def main():
    """Main entry: read state history, compute and output the health score"""
    import sys

    state_file = sys.argv[1] if len(sys.argv) > 1 else "/opt/agent-monitor/state_history.jsonl"

    state_history = []
    try:
        with open(state_file, "r") as f:
            for line in f:
                if line.strip():
                    state_history.append(json.loads(line))
    except FileNotFoundError:
        print(json.dumps({"health_score": -1, "error": "State file not found"}))
        sys.exit(2)

    score = compute_health_score(state_history)
    result = {
        "health_score": score,
        "total_tasks": len(state_history),
        "checked_at": datetime.now().isoformat(),
        "status": "HEALTHY" if score >= 70 else "DEGRADED" if score >= 40 else "CRITICAL"
    }
    print(json.dumps(result, indent=2, ensure_ascii=False))

    if score >= 70:
        sys.exit(0)
    elif score >= 40:
        sys.exit(1)
    else:
        sys.exit(2)

if __name__ == "__main__":
    main()

Core insight: Layer 3 is what makes agent ops fundamentally different from traditional ops. Traditional ops asks "is the service reachable?" Agent ops must also ask "is the task being done correctly?" And the latter is usually the real disaster — because an agent can be normally doing the wrong thing.

---

3. Alert Levels — P0 Fatal / P1 Critical / P2 Warning / P3 Notice

Health checks are about detection. Alerts are about making someone know. The worst alerting system? Treating every problem identically.

If "database disk nearly full" and "some recipe website returned a 503" go into the same alert channel, your team either drowns in alert fatigue, or ignores everything until something truly breaks.

Adapting the traditional SRE severity model to agent ops, here's the tiering I use:

P0 — Fatal (respond immediately, interrupt current work)

Definition: a core function is completely unavailable, or real losses are occurring (money draining, data loss).

Typical scenarios:

• Agent process fully exited, systemd restart failed
• Core LLM API unavailable for 5+ consecutive minutes (no fallback model)
• Database writes failing → state can't persist
• Current hourly API spend exceeds 30% of the daily budget
• Infinite loop driving token consumption to 10x baseline

Response: immediate, human intervention required. No matter what time of night.
Channel: phone / Feishu-DingTalk voice call / SMS

P1 — Critical (respond within 30 minutes)

Definition: partial damage — core path still running, but will escalate to P0 within an hour if not handled.

Typical scenarios:

• A single external API dependency down (fallback exists but hasn't auto-switched)
• A task stuck in the state machine for 30+ minutes
• Health checks failing 3 times in a row
• Disk usage above 85%
• Token consumption at 3x baseline

Response: acknowledge within 30 minutes, fix or degrade within 2 hours.
Channel: Feishu/DingTalk message with @mention + audio/video call (not phone)

P2 — Warning (respond within 4 hours)

Definition: system running normally, but potential risk or minor anomaly present.

Typical scenarios:

• A single health check fails intermittently (self-recovers within an hour)
• Data-quality detection shows outlier ratio past threshold
• Task queue backlogged but still processing
• Memory usage above 70%
• SSL certificate expires within 7 days

Response: handle within the next business day.
Channel: Feishu/DingTalk regular message + dashboard marker

P3 — Notice (watch but don't act)

Definition: informational — no immediate action needed, but worth recording and watching.

Typical scenarios:

• Daily agent task report (X tasks completed)
• Daily token-consumption report (more/less than yesterday)
• New-version deployment succeeded
• Database backup completed
• Low-priority log-pattern findings

Response: read it, that's it.
Channel: Feishu/DingTalk regular message (no @)

Preventing Alert Fatigue

My one iron rule: if an alert fires and nobody needs to act on it, it's noise. Noise must be eliminated, not tolerated.

Do a monthly "alert audit" — pull all alerts from the past 30 days and check whether every fired alert actually produced an action. If an alert fired 10 times and nobody responded 10 times, either downgrade its priority or fix its false-positive rate.

---

4. Incident Response Runbooks — A Standardized Path From Detection to Recovery

The alert fired. Now what?

This is where most teams fall apart — the alert arrives and nobody knows what to do, or everyone does something but it's duplicated chaos.

A Runbook (incident response playbook) fixes this: it turns "panicked investigation" into "standardized operation executed by procedure."

My Runbook Template

# Runbook: [Incident Name]
Version: v1.0 | Last updated: 2026-07-01

## 1. Incident Identification
- **Alert rule**: [related alert name]
- **Trigger condition**: [under what conditions this fires]
- **Impact scope**: [which features/users are affected]
- **Severity**: [P0/P1/P2/P3]

## 2. Quick Recovery (within 5 minutes)
> Restore service first, investigate root cause later

- [ ] Step 1: Restart the agent process  `systemctl restart agent`
- [ ] Step 2: Verify state machine recovery  `cat STATE.md | head -20`
- [ ] Step 3: Confirm health checks pass  `curl localhost:8080/health`
- [ ] Step 4: Check key dependencies (API/DB/message queue)

## 3. Root Cause Investigation (after recovery)

### Checklist
- [ ] Review the changes from the most recent deployment
- [ ] Check ERROR_LOG for similar historical errors
- [ ] Check the LLM API status page for recent availability
- [ ] Check token consumption for anomalies
- [ ] Check external dependencies for version changes

### Common diagnostic commands
# View recent logs
journalctl -u agent --since "30 min ago" -n 100

# Check process resources
ps aux | grep agent | grep -v grep

# Token usage audit
tail -100 agent.log | grep "token_usage" | jq .

## 4. Recovery Verification
- [ ] Agent returns to normal task processing
- [ ] All P0/P1 alerts cleared
- [ ] Data consistency verified
- [ ] No similar user reports within 15 minutes

## 5. Postmortem
- Incident timeline:
  - Detected at:
  - Responded at:
  - Recovered at:
  - Fixed at:
- Root cause summary:
- Improvement actions:
- Update ERROR_LOG? [yes/no]

Three Real Agent-Failure Runbooks

Runbook #1: LLM API timeout storm
Scenario: OpenAI API returns 5xx or times out (>30s) repeatedly. Severity: P1 (with fallback model) → P0 (no fallback model).
Quick recovery:

• Check the OpenAI status page: https://status.openai.com
• If outage confirmed → switch to the fallback model: agent config set model claude-sonnet-4
• If a local failure cache exists → start degradation mode (process cached tasks only, no new calls) Root cause:
• Check whether the API key recently exceeded its limits
• Check whether batch retries caused rate limiting
• Check historical logs — does it happen at fixed times?

Runbook #2: State-machine deadlock
Scenario: a task stuck in "executing" for 30+ minutes. Severity: P2 (single task) → P1 (3+ tasks stuck).
Quick recovery:

• Force-mark the task as failed: agent task force-fail
• Check for orphaned processes
• If multiple tasks stuck at once → restart the agent process Root cause:
• Check whether all tool calls involved in the task returned properly
• Check whether the LLM emitted an unexpected format (stalling tool parsing)
• Check for deadlock (two tasks waiting on each other's resources)

Runbook #3: Token consumption spike
Scenario: token consumption at 5x baseline for 5 consecutive minutes. Severity: P1 (significant cost possible).
Quick recovery:

• Pause all non-critical tasks: agent task pause --priority=2,3
• Review the last 100 LLM call logs and locate the token hogs
• If it's an infinite loop → force-terminate the task Root cause:
• Check for context-window bloat (stuffing the entire history into every call)
• Check for ineffective retries (re-calling the LLM on the same error trying to "understand" it)
• Check whether model params were accidentally changed (max_tokens set absurdly high)

Runbook Principles

1. Every alert level pairs with a Runbook — P0/P1 alerts must have one.
2. Keep Runbooks where the agent can read them too — I record key steps in the "Fix" field of ERROR_LOG.md, so the agent itself can reference them.
3. Update Runbooks after every incident — the day you find "step 2 of the Runbook is outdated," fix it that day.
4. Drill regularly — run a simulated alert once a month and walk the on-call person (or the agent itself) through the Runbook.

---

5. Cost Control — Token & Spend Tracking

Now that we've covered "how not to die," here's a more practical question — can you afford it?

Agent ops has one core dimension traditional DevOps doesn't: cost ops. Your agent isn't a server with a fixed monthly fee — its spend scales with its workload, and workload can grow 100x with zero human intervention.

Worst case I've seen: someone ran AutoGPT on a data task before bed and woke up to an $800 bill — the agent was stuck in a "find problem → call LLM to understand it → problem unsolved → call LLM again" death loop.

Four Levels of Cost Monitoring

Level 1: Total spend tracking (the baseline)

# cost_tracker.py — simple cost tracking script
import json
import os
from datetime import datetime, timedelta

COST_LOG = os.environ.get("COST_LOG_PATH", "cost_log.jsonl")
DAILY_BUDGET = float(os.environ.get("DAILY_BUDGET", "10"))  # daily budget cap (USD)

def log_token_usage(model, prompt_tokens, completion_tokens, cost_per_1k=0.01):
    """Record the cost of each LLM call"""
    cost = (prompt_tokens + completion_tokens) / 1000 * cost_per_1k
    entry = {
        "timestamp": datetime.now().isoformat(),
        "model": model,
        "prompt_tokens": prompt_tokens,
        "completion_tokens": completion_tokens,
        "cost": round(cost, 6)
    }
    with open(COST_LOG, "a") as f:
        f.write(json.dumps(entry) + "\n")
    return cost

def check_daily_budget():
    """Check whether today's budget has been exceeded"""
    today = datetime.now().strftime("%Y-%m-%d")
    total = 0.0
    try:
        with open(COST_LOG, "r") as f:
            for line in f:
                entry = json.loads(line)
                if entry["timestamp"].startswith(today):
                    total += entry["cost"]
    except FileNotFoundError:
        pass
    return total, total > DAILY_BUDGET

Level 2: Per-task / per-project attribution

Total spend isn't enough — you need to know where the money went. Tag every LLM call with a task_id or project_id and aggregate by it:

-- pseudo-SQL aggregation query
SELECT
    project_id,
    SUM(cost) as total_cost,
    COUNT(*) as api_calls,
    AVG(prompt_tokens + completion_tokens) as avg_tokens
FROM cost_log
WHERE timestamp > NOW() - INTERVAL '7 days'
GROUP BY project_id
ORDER BY total_cost DESC

Level 3: Spend anomaly detection

Build a historical baseline and detect anomalous consumption in real time:

def detect_cost_anomaly(window_minutes=5, z_threshold=3.0):
    """Detect anomalous spend in the last 5 minutes"""
    recent = get_cost_in_window(window_minutes)
    baseline = get_historical_baseline(window_minutes)

    # Compute Z-score
    z_score = (recent - baseline['mean']) / (baseline['std'] + 0.001)

    if z_score > z_threshold:
        # P2 alert: spend unusually high
        alert("P2", f"Token consumption anomaly: {z_score:.1f}σ off baseline")
    elif z_score < -0.5:
        # P3 notice: spend unusually low (maybe tasks aren't running)
        notify("P3", f"Token consumption significantly below baseline, check whether tasks are running")

Level 4: Automatic budget governor

Don't just detect — stop the bleeding automatically:

# cost_governor.py — automatic budget control

class CostGovernor:
    def __init__(self, daily_budget=10.0, hourly_budget=2.0):
        self.daily_budget = daily_budget
        self.hourly_budget = hourly_budget

    def get_today_spent(self):
        """Query today's cumulative API spend (USD)"""
        import datetime
        today = datetime.date.today().isoformat()
        # Aggregate all of today's API call costs from the cost DB/log
        # In production: back it with SQLite/Redis or parse Prometheus metrics
        return self._query_spent(since=today)

    def get_last_hour_spent(self):
        """Query API spend for the last hour (USD)"""
        import datetime
        one_hour_ago = (
            datetime.datetime.utcnow() - datetime.timedelta(hours=1)
        ).isoformat()
        return self._query_spent(since=one_hour_ago)

    def is_critical_path(self):
        """Whether the current call belongs to a critical (non-skippable) task"""
        # Tag by call context: critical / normal / low
        # e.g. health checks, alert delivery, state persistence are critical path
        return getattr(self, '_critical', False)

    def _query_spent(self, since):
        """Internal: aggregate spend for a given period from persistent storage"""
        # TODO: connect a real cost store (Prometheus counter or SQLite)
        # e.g. aggregate of api_calls where timestamp >= since
        return 0.0

    def should_proceed(self, estimated_cost=0.0):
        """Decide whether this LLM call is allowed"""
        daily_spent = self.get_today_spent()
        hourly_spent = self.get_last_hour_spent()

        if daily_spent + estimated_cost > self.daily_budget * 0.8:
            # Past 80% of daily budget → critical tasks only
            if not self.is_critical_path():
                return False, "daily_budget_threshold_reached"

        if hourly_spent > self.hourly_budget:
            # Past hourly budget → pause non-critical tasks
            return False, "hourly_budget_exceeded"

        if daily_spent > self.daily_budget:
            # Past daily budget → pause everything
            return False, "daily_budget_exhausted"

        return True, "ok"

Cost-Control Lessons From the Field

Strategy	Effect	Implementation cost
Hard daily cap	prevents runaway bills	low (one config line)
Per-task budget allocation	high-priority tasks protected from low-priority waste	medium (needs task classification)
Cache identical LLM requests	cuts 20–40% of duplicate calls	medium (add a cache layer)
Dynamic max_tokens control	fewer wasted tokens	low (presets per task type)
Fallback model degradation	>30% daily budget left → GPT-4; <30% → GPT-4o-mini	medium (two-model switch)

My personal rule: don't let your agent spend money faster than you do. The sting you feel looking at the API bill at the end of the month should be controllable and expected.

---

6. Environment Setup — Docker + Prometheus + Grafana

Before deploying the monitoring stack, make sure the server has the required environment. The following applies to Ubuntu 22.04 / Debian 12; adjust the package manager for other distros.

All tools below come from their official GitHub repositories — get them from the source:

Tool	Purpose	GitHub / Official
Docker	container runtime	https://github.com/docker/docker-ce
Prometheus	metrics collection & monitoring	https://github.com/prometheus/prometheus
Grafana	dashboards & visualization	https://github.com/grafana/grafana
OpenAI	LLM API (the cost-tracking subject)	https://github.com/openai/openai-python

6.1 Install Docker

#!/bin/bash
# install_docker.sh — one-click Docker install (Ubuntu/Debian)

set -euo pipefail

echo "=== Installing Docker ==="

# Remove old versions (if present)
sudo apt-get remove -y docker docker-engine docker.io containerd runc 2>/dev/null || true

# Install dependencies
sudo apt-get update
sudo apt-get install -y \
    ca-certificates \
    curl \
    gnupg \
    lsb-release

# Add the official Docker GPG key
sudo mkdir -p /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | \
    sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg

# Add the Docker repository
echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] \
  https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

# Install Docker Engine
sudo apt-get update
sudo apt-get install -y docker-ce docker-ce-cli containerd.io docker-compose-plugin

# Add the current user to the docker group (no sudo needed)
sudo usermod -aG docker "$USER"

# Enable and start on boot
sudo systemctl enable docker
sudo systemctl start docker

# Verify the install
docker --version
docker run --rm hello-world

echo "=== Docker installed. Log out and back in to use docker without sudo. ==="

6.2 Install Prometheus (Docker or binary)

Option A: Docker (recommended, simplest)

# Pull the Prometheus image
docker pull prom/prometheus:latest

# Create config and data directories
sudo mkdir -p /etc/prometheus /var/lib/prometheus

# Test launch (Ctrl+C to stop)
docker run --rm -p 9090:9090 \
    -v /etc/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml \
    prom/prometheus:latest

Option B: Binary install (no Docker)

#!/bin/bash
# install_prometheus_binary.sh — binary Prometheus install

set -euo pipefail

PROMETHEUS_VERSION="2.52.0"
INSTALL_DIR="/usr/local/bin"
CONFIG_DIR="/etc/prometheus"
DATA_DIR="/var/lib/prometheus"

# Download and extract
cd /tmp
curl -LO "https://github.com/prometheus/prometheus/releases/download/v${PROMETHEUS_VERSION}/prometheus-${PROMETHEUS_VERSION}.linux-amd64.tar.gz"
tar xzf "prometheus-${PROMETHEUS_VERSION}.linux-amd64.tar.gz"

# Install binaries
sudo cp "prometheus-${PROMETHEUS_VERSION}.linux-amd64/prometheus" "$INSTALL_DIR/"
sudo cp "prometheus-${PROMETHEUS_VERSION}.linux-amd64/promtool" "$INSTALL_DIR/"

# Create directories and config
sudo mkdir -p "$CONFIG_DIR" "$DATA_DIR"
sudo cp "prometheus-${PROMETHEUS_VERSION}.linux-amd64/prometheus.yml" "$CONFIG_DIR/"

# Create the systemd service
sudo tee /etc/systemd/system/prometheus.service > /dev/null << 'EOF'
[Unit]
Description=Prometheus Monitoring
After=network.target

[Service]
ExecStart=/usr/local/bin/prometheus \
    --config.file=/etc/prometheus/prometheus.yml \
    --storage.tsdb.path=/var/lib/prometheus/ \
    --web.listen-address=0.0.0.0:9090
Restart=always
User=root

[Install]
WantedBy=multi-user.target
EOF

sudo systemctl daemon-reload
sudo systemctl enable prometheus
sudo systemctl start prometheus

# Cleanup
rm -rf "/tmp/prometheus-${PROMETHEUS_VERSION}.linux-amd64"*

echo "Prometheus installed: http://localhost:9090"

6.3 Install Grafana

Option A: Docker (recommended)

# Pull the Grafana image
docker pull grafana/grafana:latest

# Create the data directory
sudo mkdir -p /var/lib/grafana

# Test launch
docker run --rm -p 3000:3000 \
    -v /var/lib/grafana:/var/lib/grafana \
    grafana/grafana:latest

Option B: APT install (Ubuntu/Debian)

#!/bin/bash
# install_grafana_apt.sh — APT install of Grafana

set -euo pipefail

# Add the Grafana APT repository
sudo apt-get install -y software-properties-common wget
sudo mkdir -p /etc/apt/keyrings
wget -q -O - https://apt.grafana.com/gpg.key | sudo gpg --dearmor -o /etc/apt/keyrings/grafana.gpg
echo "deb [signed-by=/etc/apt/keyrings/grafana.gpg] https://apt.grafana.com stable main" | \
    sudo tee /etc/apt/sources.list.d/grafana.list > /dev/null

# Install
sudo apt-get update
sudo apt-get install -y grafana

# Start
sudo systemctl daemon-reload
sudo systemctl enable grafana-server
sudo systemctl start grafana-server

echo "Grafana installed: http://localhost:3000 (default login: admin/admin)"

---

7. Hands-On: One-Click Deploy of the Full Monitoring Stack

Theory done, environment ready — now the real thing. Here's a deployable monitoring stack built entirely from open-source tools, zero cost to start.

Architecture Overview

┌─────────────────────────────────────────────────────────┐
│                     Monitoring Stack                     │
│                                                          │
│  ┌──────────────┐        ┌──────────────┐                │
│  │  Agent       │───────▶│  Prometheus  │                │
│  │  Metrics     │        │ (storage)    │                │
│  └──────────────┘        └──────┬───────┘                │
│                                 │                        │
│  ┌────────────────────┐   │     │    ┌────────────┐      │
│  │  Health Checks     │───┼────▶│    │  Grafana   │      │
│  │  (3-layer script)  │   │     │    │ (dashboards)│     │
│  └────────────────────┘   │     │    └────────────┘      │
│                           │     │                        │
│  ┌──────────────┐         │     │    ┌────────────┐      │
│  │  Alert       │─────────┼────▶│    │  Feishu/   │      │
│  │  Manager     │         │     │    │  DingTalk  │      │
│  └──────────────┘         │     │    │  Webhook   │      │
│                           │     │    └────────────┘      │
│  ┌──────────────┐         │     │                        │
│  │  Cost        │─────────┘     │                        │
│  │  Tracker     │               │                        │
│  └──────────────┘               │                        │
└─────────────────────────────────────────────────────────┘

The One-Click Deploy Script

The following is a complete deployment script: Prometheus + Grafana + three-layer health checks + cost exporter + alert manager. All bash syntax errors fixed and truncated parts restored.

#!/bin/bash
# deploy_monitoring.sh — one-click deploy of the full agent monitoring stack
# Usage: bash deploy_monitoring.sh [agent_name]
# Prerequisites: Docker installed (see section 6), curl/jq available

set -euo pipefail

AGENT_NAME="${1:-my-agent}"
MONITOR_DIR="/opt/agent-monitor/${AGENT_NAME}"
PROMETHEUS_PORT=9090
GRAFANA_PORT=3000
COST_EXPORTER_PORT=9091

echo "====================================="
echo " Deploying agent monitoring: ${AGENT_NAME}"
echo "====================================="

# 0. Environment pre-checks
echo ""
echo "[0/8] Checking prerequisites..."

if ! command -v docker &> /dev/null; then
    echo "[ERROR] Docker not installed. Run install_docker.sh first."
    exit 1
fi

if ! command -v curl &> /dev/null; then
    echo "[ERROR] curl not installed"
    exit 1
fi

if ! command -v jq &> /dev/null; then
    echo "[WARN] jq not installed, installing..."
    sudo apt-get update -qq && sudo apt-get install -y -qq jq
fi

# Make sure ports are free
for port in $PROMETHEUS_PORT $GRAFANA_PORT $COST_EXPORTER_PORT; do
    if ss -tln | grep -q ":${port} "; then
        echo "[WARN] Port ${port} is already in use. Free it and retry."
    fi
done

echo "[OK] Prerequisite checks passed"

# 1. Create the directory structure
echo ""
echo "[1/8] Creating directory structure..."
mkdir -p "${MONITOR_DIR}"/{prometheus,grafana,scripts,alerts,dashboards,data}
echo "[OK] Directories created: ${MONITOR_DIR}"

# 2. Create the Prometheus config
echo ""
echo "[2/8] Creating Prometheus config..."
cat > "${MONITOR_DIR}/prometheus/prometheus.yml" << 'PROMEOF'
global:
  scrape_interval: 15s
  evaluation_interval: 15s
  external_labels:
    monitor: agent-monitor

rule_files:
  - "/etc/prometheus/alerts/*.yml"

scrape_configs:
  - job_name: 'agent-health'
    scrape_interval: 10s
    static_configs:
      - targets: ['host.docker.internal:8080']
        labels:
          agent: 'my-agent'
          env: 'production'

  - job_name: 'agent-cost-metrics'
    scrape_interval: 30s
    static_configs:
      - targets: ['host.docker.internal:9091']
        labels:
          agent: 'my-agent'

  - job_name: 'prometheus-self'
    scrape_interval: 30s
    static_configs:
      - targets: ['localhost:9090']
PROMEOF

# Replace the agent-name placeholder
sed -i "s/my-agent/${AGENT_NAME}/g" "${MONITOR_DIR}/prometheus/prometheus.yml"
echo "[OK] Prometheus config done"

# 3. Create alert rules (P0-P3)
echo ""
echo "[3/8] Creating alert rules..."
cat > "${MONITOR_DIR}/alerts/agent_alerts.yml" << 'ALERTEOF'
groups:
  - name: agent_alerts
    interval: 30s
    rules:
      # P0 - Agent process down
      - alert: AgentProcessDown
        expr: up{job="agent-health"} == 0
        for: 1m
        labels:
          severity: critical
          priority: P0
          channel: phone
        annotations:
          summary: "Agent {{ $labels.agent }} has stopped"
          description: "Agent {{ $labels.agent }} on {{ $labels.instance }} has been unresponsive for over 1 minute. Investigate immediately."
          runbook_url: "https://wiki.example.com/runbooks/agent-process-down"

      # P0 - Database unreachable
      - alert: DatabaseUnreachable
        expr: agent_db_up == 0
        for: 1m
        labels:
          severity: critical
          priority: P0
        annotations:
          summary: "{{ $labels.agent }} database unreachable"

      # P1 - Health check failed
      - alert: HealthCheckFailed
        expr: agent_health_check{check="liveness"} < 1
        for: 3m
        labels:
          severity: high
          priority: P1
          channel: feishu_mention
        annotations:
          summary: "{{ $labels.agent }} health checks failing"
          description: "Health check {{ $labels.check }} has failed for 3 minutes. Investigate."

      # P1 - Disk usage too high
      - alert: DiskUsageHigh
        expr: agent_disk_usage_percent > 85
        for: 5m
        labels:
          severity: high
          priority: P1
        annotations:
          summary: "{{ $labels.agent }} disk usage {{ $value }}%"

      # P2 - Token consumption anomaly
      - alert: TokenCostAnomaly
        expr: rate(agent_token_cost_total[5m]) > 3 * avg(rate(agent_token_cost_total[30m]))
        for: 5m
        labels:
          severity: warning
          priority: P2
        annotations:
          summary: "Token consumption rate unusually high"
          description: "The 5-minute consumption rate is more than 3x the 30-minute average"

      # P3 - Daily cost report
      - alert: DailyCostReport
        expr: agent_daily_cost > 0
        for: 24h
        labels:
          severity: info
          priority: P3
        annotations:
          summary: "Agent {{ $labels.agent }} today's cost: ${{ $value }}"
ALERTEOF
echo "[OK] Alert rules created (6 P0-P3 rules)"

# 4. Create the health check script (all three layers in one)
echo ""
echo "[4/8] Creating health check script..."
cat > "${MONITOR_DIR}/scripts/health_check.sh" << 'HEALTHEOF'
#!/bin/bash
# health_check.sh — three-layer agent health check
# Emits Prometheus textfile-format metrics for the node_exporter textfile collector
# or for pushing via cron + pushgateway

set -euo pipefail

METRICS_DIR="${METRICS_DIR:-/tmp/agent-metrics}"
METRICS_FILE="${METRICS_DIR}/agent_health.prom"
AGENT_NAME="${AGENT_NAME:-my-agent}"

mkdir -p "${METRICS_DIR}"

# Helper: emit a Prometheus metric
emit_metric() {
    local name="$1"
    local value="$2"
    local labels="${3:-}"
    local help="${4:-}"

    if [ -n "$help" ]; then
        echo "# HELP ${name} ${help}" >> "${METRICS_FILE}.tmp"
        echo "# TYPE ${name} gauge" >> "${METRICS_FILE}.tmp"
    fi

    if [ -n "$labels" ]; then
        echo "${name}{${labels}} ${value}" >> "${METRICS_FILE}.tmp"
    else
        echo "${name} ${value}" >> "${METRICS_FILE}.tmp"
    fi
}

# Clear the temp file
> "${METRICS_FILE}.tmp"

# ========== Layer 1: infrastructure health ==========
# Agent process check
if pgrep -f "agent" > /dev/null 2>&1; then
    emit_metric "agent_up" 1 "check=\"process\",agent=\"${AGENT_NAME}\"" "Agent process liveness"
else
    emit_metric "agent_up" 0 "check=\"process\",agent=\"${AGENT_NAME}\"" "Agent process liveness"
fi

# Port checks
for port_info in "8080:agent-api" "6379:redis" "5432:postgresql"; do
    port="${port_info%%:*}"
    service="${port_info##*:}"
    if ss -tln 2>/dev/null | grep -q ":${port} " || netstat -tln 2>/dev/null | grep -q ":${port} "; then
        emit_metric "agent_port_up" 1 "port=\"${port}\",service=\"${service}\",agent=\"${AGENT_NAME}\"" "Port ${port} (${service}) reachability"
    else
        emit_metric "agent_port_up" 0 "port=\"${port}\",service=\"${service}\",agent=\"${AGENT_NAME}\"" "Port ${port} (${service}) reachability"
    fi
done

# Disk usage
disk_usage=$(df -h / 2>/dev/null | awk 'NR==2 {print $5}' | tr -d '%')
if [ -n "$disk_usage" ]; then
    emit_metric "agent_disk_usage_percent" "${disk_usage}" "mount=\"/\",agent=\"${AGENT_NAME}\"" "Disk usage percentage for /"
fi

# Available memory (MB)
avail_mem=$(free -m 2>/dev/null | awk 'NR==2 {print $7}')
if [ -n "$avail_mem" ]; then
    emit_metric "agent_memory_available_mb" "${avail_mem}" "agent=\"${AGENT_NAME}\"" "Available memory in MB"
fi

# Database connectivity
if timeout 5 psql -c "SELECT 1" > /dev/null 2>&1; then
    emit_metric "agent_db_up" 1 "check=\"postgresql\",agent=\"${AGENT_NAME}\"" "Database connectivity"
else
    emit_metric "agent_db_up" 0 "check=\"postgresql\",agent=\"${AGENT_NAME}\"" "Database connectivity"
fi

# ========== Layer 2: API dependency health ==========
# LLM API reachability
if curl -sf --max-time 10 "https://api.openai.com/v1/models" \
    -H "Authorization: Bearer ${OPEN...Y:-}" > /dev/null 2>&1; then
    emit_metric "agent_api_reachable" 1 "target=\"openai\",agent=\"${AGENT_NAME}\"" "External API reachability"
else
    emit_metric "agent_api_reachable" 0 "target=\"openai\",agent=\"${AGENT_NAME}\"" "External API reachability"
fi

# ========== Layer 3: business health (read from the state file) ==========
STATE_FILE="${STATE_FILE:-/opt/agent-monitor/${AGENT_NAME}/data/state_history.jsonl}"
if [ -f "${STATE_FILE}" ]; then
    # Count stuck tasks (>30 min still executing)
    stuck_count=$(grep '"state":"executing"' "${STATE_FILE}" 2>/dev/null | wc -l || echo 0)
    emit_metric "agent_stuck_tasks" "${stuck_count}" "agent=\"${AGENT_NAME}\"" "Number of stuck tasks (>30min)"

    # Count failed tasks
    failed_count=$(grep '"state":"failed"' "${STATE_FILE}" 2>/dev/null | wc -l || echo 0)
    emit_metric "agent_failed_tasks" "${failed_count}" "agent=\"${AGENT_NAME}\"" "Number of failed tasks"

    # Tasks completed today
    today=$(date +%Y-%m-%d)
    completed_today=$(grep "\"${today}\"" "${STATE_FILE}" 2>/dev/null | grep '"state":"completed"' | wc -l || echo 0)
    emit_metric "agent_tasks_completed_today" "${completed_today}" "agent=\"${AGENT_NAME}\"" "Tasks completed today"
fi

# Atomically replace the metrics file
mv "${METRICS_FILE}.tmp" "${METRICS_FILE}"
echo "Metrics written to ${METRICS_FILE}"
HEALTHEOF
chmod +x "${MONITOR_DIR}/scripts/health_check.sh"
echo "[OK] Health check script created"

# 5. Create the cost tracking exporter
echo ""
echo "[5/8] Creating cost tracking exporter..."
cat > "${MONITOR_DIR}/scripts/cost_exporter.py" << 'COSTEOF'
#!/usr/bin/env python3
"""
Prometheus exporter for Agent token costs.
Exposes daily cost and API call count at :9091/metrics.
"""
import json
import os
import sys
from datetime import datetime
from http.server import HTTPServer, BaseHTTPRequestHandler

COST_LOG = os.environ.get("COST_LOG_PATH", "/opt/agent-monitor/cost_log.jsonl")
LISTEN_PORT = int(os.environ.get("COST_EXPORTER_PORT", "9091"))

class CostExporter(BaseHTTPRequestHandler):
    def do_GET(self):
        if self.path == "/metrics":
            self.send_response(200)
            self.send_header("Content-Type", "text/plain; charset=utf-8")
            self.end_headers()
            metrics = self.generate_metrics()
            self.wfile.write(metrics.encode("utf-8"))
        elif self.path == "/health":
            self.send_response(200)
            self.send_header("Content-Type", "text/plain")
            self.end_headers()
            self.wfile.write(b"OK")
        else:
            self.send_response(404)
            self.end_headers()

    def log_message(self, format, *args):
        """Suppress default logging to stderr."""
        pass

    def generate_metrics(self):
        today = datetime.now().strftime("%Y-%m-%d")
        total_cost = 0.0
        total_calls = 0
        cost_by_model = {}

        try:
            with open(COST_LOG, "r") as f:
                for line in f:
                    line = line.strip()
                    if not line:
                        continue
                    try:
                        entry = json.loads(line)
                    except json.JSONDecodeError:
                        continue

                    if entry.get("timestamp", "").startswith(today):
                        cost = entry.get("cost", 0)
                        total_cost += cost
                        total_calls += 1

                        model = entry.get("model", "unknown")
                        cost_by_model[model] = cost_by_model.get(model, 0.0) + cost
        except FileNotFoundError:
            pass

        lines = []
        lines.append("# HELP agent_daily_cost Total daily cost for Agent (USD)")
        lines.append("# TYPE agent_daily_cost gauge")
        lines.append(f"agent_daily_cost {total_cost:.4f}")

        lines.append("# HELP agent_daily_api_calls Total daily API calls")
        lines.append("# TYPE agent_daily_api_calls gauge")
        lines.append(f"agent_daily_api_calls {total_calls}")

        lines.append("# HELP agent_cost_by_model Daily cost breakdown by model")
        lines.append("# TYPE agent_cost_by_model gauge")
        for model, cost in cost_by_model.items():
            safe_model = model.replace('"', '_').replace("'", "_")
            lines.append(f'agent_cost_by_model{{model="{safe_model}"}} {cost:.4f}')

        return "\n".join(lines) + "\n"

if __name__ == "__main__":
    print(f"Starting Cost Exporter on 0.0.0.0:{LISTEN_PORT}")
    print(f"Reading cost log from: {COST_LOG}")
    server = HTTPServer(("0.0.0.0", LISTEN_PORT), CostExporter)
    try:
        server.serve_forever()
    except KeyboardInterrupt:
        print("\nShutting down Cost Exporter...")
        server.shutdown()
COSTEOF
chmod +x "${MONITOR_DIR}/scripts/cost_exporter.py"
echo "[OK] Cost tracking exporter created"

# 6. Create systemd services (health check timer + cost exporter)
echo ""
echo "[6/8] Creating systemd services..."

# Health check timer
cat > "/etc/systemd/system/agent-healthcheck-${AGENT_NAME}.service" << SERVICEEOF
[Unit]
Description=Agent Health Check - ${AGENT_NAME}
After=network.target

[Service]
Type=oneshot
ExecStart=${MONITOR_DIR}/scripts/health_check.sh
Environment="AGENT_NAME=${AGENT_NAME}"
Environment="STATE_FILE=${MONITOR_DIR}/data/state_history.jsonl"
User=root
SERVICEEOF

cat > "/etc/systemd/system/agent-healthcheck-${AGENT_NAME}.timer" << TIMEREOF
[Unit]
Description=Agent Health Check Timer - ${AGENT_NAME}

[Timer]
OnCalendar=*:0/1
Persistent=true

[Install]
WantedBy=timers.target
TIMEREOF

# Cost exporter daemon
cat > "/etc/systemd/system/agent-cost-exporter-${AGENT_NAME}.service" << COSTSVCEOF
[Unit]
Description=Agent Cost Exporter - ${AGENT_NAME}
After=network.target

[Service]
Type=simple
ExecStart=/usr/bin/python3 ${MONITOR_DIR}/scripts/cost_exporter.py
Environment="COST_LOG_PATH=${MONITOR_DIR}/data/cost_log.jsonl"
Environment="COST_EXPORTER_PORT=${COST_EXPORTER_PORT}"
Restart=always
RestartSec=10
User=root

[Install]
WantedBy=multi-user.target
COSTSVCEOF

systemctl daemon-reload
systemctl enable "agent-healthcheck-${AGENT_NAME}.timer"
systemctl start "agent-healthcheck-${AGENT_NAME}.timer"
systemctl enable "agent-cost-exporter-${AGENT_NAME}.service"
systemctl start "agent-cost-exporter-${AGENT_NAME}.service"
echo "[OK] systemd services created and started"

# 7. Start Prometheus (Docker)
echo ""
echo "[7/8] Starting Prometheus..."
# Stop the old container first (if present)
docker rm -f "prometheus-${AGENT_NAME}" 2>/dev/null || true

docker run -d \
    --name "prometheus-${AGENT_NAME}" \
    --restart always \
    --network host \
    -p "${PROMETHEUS_PORT}:9090" \
    -v "${MONITOR_DIR}/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml:ro" \
    -v "${MONITOR_DIR}/alerts:/etc/prometheus/alerts:ro" \
    -v "${MONITOR_DIR}/data/prometheus:/prometheus" \
    prom/prometheus:latest \
    --config.file=/etc/prometheus/prometheus.yml \
    --storage.tsdb.path=/prometheus \
    --web.enable-lifecycle

# Wait for Prometheus to be ready
echo "Waiting for Prometheus to start..."
for i in $(seq 1 30); do
    if curl -sf "http://localhost:${PROMETHEUS_PORT}/-/ready" > /dev/null 2>&1; then
        echo "[OK] Prometheus ready: http://localhost:${PROMETHEUS_PORT}"
        break
    fi
    sleep 2
done
echo "[OK] Prometheus started"

# 8. Start Grafana
echo ""
echo "[8/8] Starting Grafana..."
docker rm -f "grafana-${AGENT_NAME}" 2>/dev/null || true

docker run -d \
    --name "grafana-${AGENT_NAME}" \
    --restart always \
    --network host \
    -p "${GRAFANA_PORT}:3000" \
    -v "${MONITOR_DIR}/grafana:/var/lib/grafana" \
    -v "${MONITOR_DIR}/dashboards:/etc/grafana/provisioning/dashboards" \
    -e "GF_SECURITY_ADMIN_USER=admin" \
    -e "GF_SECURITY_ADMIN_PASSWORD=admin" \
    -e "GF_USERS_ALLOW_SIGN_UP=false" \
    grafana/grafana:latest

# Wait for Grafana to be ready
echo "Waiting for Grafana to start..."
for i in $(seq 1 30); do
    if curl -sf "http://localhost:${GRAFANA_PORT}/api/health" > /dev/null 2>&1; then
        echo "[OK] Grafana ready: http://localhost:${GRAFANA_PORT}"
        break
    fi
    sleep 2
done

echo ""
echo "====================================="
echo " Deployment complete!"
echo "====================================="
echo ""
echo "Prometheus:  http://localhost:${PROMETHEUS_PORT}"
echo "Grafana:     http://localhost:${GRAFANA_PORT} (admin/admin)"
echo "Health check:    ${MONITOR_DIR}/scripts/health_check.sh"
echo "Cost tracking:   http://localhost:${COST_EXPORTER_PORT}/metrics"
echo "Alert rules:     ${MONITOR_DIR}/alerts/agent_alerts.yml"
echo "Metrics dir:     /tmp/agent-metrics/agent_health.prom"
echo ""
echo "Next: add a Prometheus data source in Grafana (http://localhost:${PROMETHEUS_PORT})"
echo "      then import dashboards or create panels manually"

Grafana Dashboard Configuration

After deployment, import four core dashboard panels in Grafana:

• Agent overview panel: current status (Up/Down), task completion rate, average response time
• Health check panel: three-layer health status, failure-count trend, dependency availability
• Alerts panel: P0–P3 alert counts, response time, MTTR (mean time to recovery)
• Cost panel: spend trend, per-task breakdown, budget utilization, cost anomaly events

Adding the data source:

1. Visit http://localhost:3000, log in as admin/admin
2. Left menu → Connections → Data sources → Add data source
3. Select Prometheus, set URL to http://localhost:9090
4. Click Save & Test — a green "Successfully queried the Prometheus API" means it worked

The Minimal Viable Option (if you don't want Docker)

If the full stack feels like too much, the minimal solution is a single file:

#!/bin/bash
# minimal_monitor.sh — minimal monitoring solution
# Usage: add to crontab: */5 * * * * /path/to/minimal_monitor.sh >> /var/log/agent-monitor.log 2>&1

LOG_FILE="${LOG_FILE:-/var/log/agent-monitor.log}"

# 1. Check whether the agent is alive
if ! pgrep -f "agent" > /dev/null 2>&1; then
    echo "[ALERT] P0: Agent process stopped! $(date)"
fi

# 2. Check whether the key API is reachable
if curl -sf --max-time 10 "https://api.openai.com/v1/models" \
    -H "Authorization: Bearer ${OPEN...Y:-}" > /dev/null 2>&1; then
    echo "[OK] OpenAI API reachable - $(date)"
else
    echo "[WARN] P1: OpenAI API unreachable - $(date)"
fi

# 3. Check today's spend
COST_FILE="${COST_FILE:-cost_log.jsonl}"
if [ -f "$COST_FILE" ]; then
    today=$(date +%Y-%m-%d)
    total=$(grep "$today" "$COST_FILE" 2>/dev/null | \
        awk -F'"cost":' '{sum+=$2} END {printf "%.2f", sum}')
    echo "[INFO] Today's cost: \$${total:-0.00} - $(date)"
else
    echo "[INFO] Cost log file not found: $COST_FILE - $(date)"
fi

# 4. Disk space check
usage=$(df -h / | awk 'NR==2 {print $5}' | tr -d '%')
if [ "${usage:-0}" -gt 85 ]; then
    echo "[WARN] P2: Disk usage ${usage}% - $(date)"
fi

Put it in cron, run every 5 minutes:

# Edit crontab
crontab -e

# Add this line
*/5 * * * * /opt/agent-monitor/minimal_monitor.sh >> /var/log/agent-monitor.log 2>&1

The whole thing takes less than 5 minutes — and what you get is visibility into your agent's lifecycle. Before, you were flying blind. Now you at least have an altimeter.

---

8. Verification — Making Sure the Monitoring Actually Works

Deploying isn't the same as monitoring being active. Here are the layered verification steps — every one must be run after deployment.

8.1 Infrastructure Layer

# 1. Confirm Docker containers are running
docker ps --format "table {{.Names}}\t{{.Status}}\t{{.Ports}}" | grep -E "prometheus|grafana"

# 2. Confirm systemd services are healthy
systemctl status "agent-healthcheck-${AGENT_NAME}.timer"
systemctl status "agent-cost-exporter-${AGENT_NAME}.service"

# 3. Confirm ports are listening
ss -tln | grep -E "9090|9091|3000"

# 4. Run a health check manually
bash /opt/agent-monitor/my-agent/scripts/health_check.sh
cat /tmp/agent-metrics/agent_health.prom

8.2 Monitoring Pipeline

# 1. Verify Prometheus is reachable
curl -s http://localhost:9090/-/ready
# Expected: Prometheus Server is Ready.

# 2. Verify Prometheus is scraping targets
curl -s http://localhost:9090/api/v1/targets | jq '.data.activeTargets[] | {job: .labels.job, health: .health}'
# Expected: all targets report health "up"

# 3. Verify Cost Exporter metrics
curl -s http://localhost:9091/metrics
# Expected: agent_daily_cost and agent_daily_api_calls metrics visible

# 4. Verify Grafana is reachable
curl -s http://localhost:3000/api/health
# Expected: {"database": "ok"}

8.3 Alert Pipeline

# 1. Check whether Prometheus loaded the alert rules
curl -s http://localhost:9090/api/v1/rules | jq '.data.groups[] | .name'

# 2. Trigger a test alert manually (stop the agent process)
systemctl stop your-agent-service
# Wait 1-2 minutes, then check alert status
curl -s http://localhost:9090/api/v1/alerts | jq '.data.alerts[] | select(.labels.severity=="critical")'

# 3. Confirm the alert clears after recovery
systemctl start your-agent-service

8.4 End-to-End Verification Checklist

Verification	Command/Method	Expected result
Docker running	docker ps	prometheus/grafana containers Status=Up
Prometheus UI	browser to :9090	targets and graph visible
Grafana UI	browser to :3000	can log in and add data source
Health metrics	cat /tmp/agent-metrics/agent_health.prom	contains agent_up, agent_disk_usage_percent, etc.
Cost metrics	curl localhost:9091/metrics	contains agent_daily_cost
Cron timer	journalctl -u agent-healthcheck-*.timer	logs showing per-minute triggers

---

9. Common Errors & Troubleshooting

I've hit every one of these so you don't have to.

Error 1: docker: command not found

• Symptom: deploy_monitoring.sh reports "Docker not installed"
• Cause: Docker isn't installed, or the current user isn't in the docker group
• Fix:
  1. Run install_docker.sh from section 6
  2. Log out/in or run newgrp docker
  3. Verify: docker ps

Error 2: port already allocated

• Symptom: docker run fails with "port is already allocated"
• Cause: ports 3000/9090/9091 already in use by another process
• Diagnose:

  sudo ss -tlnp | grep -E "3000|9090|9091"

• Fix:

  # Free the port
  sudo kill -9 <PID>
  # Or change the ports in the script
  PROMETHEUS_PORT=9091 GRAFANA_PORT=3001 bash deploy_monitoring.sh

Error 3: Prometheus target shows DOWN

• Symptom: target status is DOWN at http://localhost:9090/targets
• Cause:
  • agent-health target: host.docker.internal doesn't work on Linux
  • cost-metrics target: cost_exporter.py isn't running
• Fix:

  # On Linux, replace host.docker.internal with localhost
  sed -i 's/host.docker.internal/localhost/g' /opt/agent-monitor/*/prometheus/prometheus.yml
  # Restart Prometheus
  docker restart prometheus-my-agent
  # Confirm cost_exporter is running
  systemctl status agent-cost-exporter-my-agent

Error 4: health check script permission denied

• Symptom: systemctl status shows "Permission denied" or exit code 126
• Cause: script lacks execute permission
• Fix:

  chmod +x /opt/agent-monitor/*/scripts/health_check.sh
  chmod +x /opt/agent-monitor/*/scripts/cost_exporter.py

Error 5: curl: (7) Failed to connect / Connection refused

• Symptom: curl checks fail inside the health check script
• Cause:
  • The agent service isn't started
  • The agent doesn't expose a /health endpoint
  • Firewall blocking
• Fix:

  # Confirm the agent is listening
  ss -tln | grep 8080
  # If there's no /health endpoint, switch the health check to pgrep
  # Check the firewall
  sudo ufw status

Error 6: awk: cmd. line:1: runaway string constant

• Symptom: the awk command in minimal_monitor.sh errors out
• Cause: quotes in awk's -F argument weren't escaped correctly
• Wrong:

  awk -F'"cost":' '{sum+=$2} END{...}'

• Correct:

  awk -F'"cost":' '{sum+=$2} END {printf "%.2f\n", sum}'

Error 7: Prometheus rule file path mismatch

• Symptom: Prometheus logs show "rule files not found"
• Cause: paths inside the Docker container differ from the host
• Fix:

  # Confirm the volume mount
  docker inspect prometheus-my-agent | jq '.[0].Mounts'
  # Alert rules should be mounted at the container's /etc/prometheus/alerts/
  # corresponding to the host's /opt/agent-monitor/my-agent/alerts/

Error 8: Grafana shows no data after login

• Symptom: Grafana panels show "No data"
• Cause:
  • No Prometheus data source added
  • Data source URL uses localhost but Docker network isolation applies
• Fix:

  # With host networking, the data source URL should be http://localhost:9090
  # With bridge networking, use http://host.docker.internal:9090
  # Confirm: Grafana → Data Sources → Save & Test shows green

---

10. The Ops Maturity Model — Which Level Is Your Agent?

Finally, a simple maturity model to help you locate yourself:

Level	Characteristics	Signature move
L0 — Running naked	agent runs in a terminal, nobody knows when it dies	only investigates after seeing "tool not responding"
L1 — Basic probing	cron checks whether the process is alive	wrote a ping script into crontab
L2 — Layered checks	three-layer health checks + log collection	glances at logs once a day
L3 — Alerting	P0–P3 graded alerts + notification channels	has been woken up by a Feishu alert
L4 — Auto response	runbook automation + self-healing	agent auto-restarts and records ERROR_LOG
L5 — Predictive ops	trend analysis + capacity planning + cost optimization	gets warned before failures happen

My current agent system sits solidly at L3, with some scenarios at L4. Going L0 → L2 takes about a month. Going L2 → L4 is a matter of discipline, not technology.

Because past L3, the bottleneck stops being "how to implement it" and becomes "when a problem is found, is anyone going to fix it." And that's exactly the hardest part of a one-person company — when you don't have an on-call team, every alert rule, every runbook, and every cost budget is a road you pave in advance for yourself.

Call to action: this afternoon, do three things — 1) add a liveness check to your agent's startup script; 2) open your API provider's console and set a budget cap; 3) create an ERROR_LOG.md and write "2026-07-01: put monitoring on the agent." All three together take under 15 minutes — and you'll sleep a lot better tonight.

Next article: When Tech Selection Isn't a Technical Question — The Infrastructure Bill of a One-Person Company

Share your "running naked" agent stories in the comments — or the sneaky ops tricks you already use.

About the author: Wu Ji (无记) — AI & digitalization practitioner focused on Agent engineering, Loop Engineering, and digital transformation. Practical, hands-on tutorials — follow along and it just works.

---

References

• [1] https://status.openai.com
• [2] https://github.com/docker/docker-ce
• [3] https://github.com/prometheus/prometheus
• [4] https://github.com/grafana/grafana
• [5] https://github.com/openai/openai-python
• [6] https://prometheus.io/docs/
• [7] https://grafana.com/docs/