DEV Community

Violeta Fragosa
Violeta Fragosa

Posted on

Building an AI Agent That Actually Monitors Your APIs

#ai #hermes #agents

The $50,000 Mistake I Almost Made

Last November, I deployed a new API endpoint for our SaaS platform. Everything looked fine in staging. The integration tests passed. The load tests showed acceptable response times. I pushed to production on a Friday afternoon (yes, I know).

By Monday morning, we had burned through 47% of our Anthropic API quota. Not from legitimate traffic — from a retry loop in our webhook handler that nobody caught because our monitoring alerts were configured for HTTP errors, not for API consumption patterns or logical errors in flow control.

The fix took 30 seconds once we found it. But the detection took three days. And by then, we'd spent $2,847 on unnecessary API calls, dangerously close to hitting the hard limit that would have taken down our entire service.

That's when I realized: monitoring HTTP status codes isn't enough. You need an agent that understands your API's behavior, tracks consumption patterns, detects anomalies, and can even draft incident reports or documentation updates based on what it observes.

So I built one.

Why Traditional API Monitoring Fails

Before we dive into the build, let's talk about why Datadog, New Relic, and CloudWatch aren't enough for modern API-driven systems.

Traditional monitoring is reactive. It tells you when something is already broken:

  • 500 errors? Alert fires.
  • Response time over 2 seconds? Alert fires.
  • Error rate above 5%? Alert fires.

But what about:

  • A slow increase in API consumption that indicates a loop or inefficiency?
  • A new endpoint being called in an unusual pattern suggesting misintegration?
  • Changes in request payloads that could indicate a breaking change you introduced?
  • Deprecation warnings from third-party APIs you depend on?
  • Opportunities to optimize based on actual usage patterns?

These are behavioral signals that require reasoning, not just threshold checking. And that's where an autonomous agent comes in.

What We're Building: The API Sentinel Agent

Here's what this agent does:

  1. Continuous Monitoring: Polls your API logs, cloud metrics, and third-party API status pages
  2. Pattern Recognition: Uses LLM reasoning to detect anomalies, not just threshold violations
  3. Consumption Tracking: Tracks API quota usage across multiple providers (OpenAI, Anthropic, Google, etc.)
  4. Documentation Sync: Automatically detects when your API behavior diverges from your docs and drafts updates
  5. Incident Reporting: When something goes wrong, generates a detailed incident report with timeline, impact analysis, and suggested fixes
  6. Cost Optimization: Identifies opportunities to reduce API costs based on actual usage patterns

Unlike my previous post about Hermes Agent for tech radar, this one focuses on operational monitoring rather than information gathering. But the underlying principles are the same: memory, reasoning, scheduling, and action.

The Tech Stack

Here's what I used to build this:

Core Agent Framework: Hermes Agent (same as before, but different skills)
Log Ingestion: Fluent Bit → CloudWatch Logs → Agent API polling
Metrics: Prometheus + custom exporters for API provider quotas
LLM: Claude 3.5 Sonnet (best for reasoning about patterns and anomalies)
Storage: SQLite for agent memory + PostgreSQL for historical metrics
Notifications: Telegram for critical alerts, weekly digest via email

The key architectural decision was making the agent pull-based rather than push-based. Instead of sending every log line to the LLM (expensive and slow), the agent periodically queries aggregated metrics and recent error logs, then reasons about whether patterns are worth investigating.

Step 1: Setting Up the Base Agent

If you haven't already installed Hermes Agent, start here:

# Install Hermes Agent
curl -fsSL https://raw.githubusercontent.com/NousResearch/hermes-agent/main/scripts/install.sh | bash

# Run the interactive setup
hermes setup --portal

# This will:
# - Configure your preferred LLM provider (OpenAI, Anthropic, local, etc.)
# - Set up the SQLite memory database
# - Create your first agent session
# - Configure optional integrations (Telegram, Slack, etc.)
Enter fullscreen mode Exit fullscreen mode

During setup, choose these options:

  • LLM Provider: Anthropic (Claude 3.5 Sonnet has the best reasoning for pattern detection)
  • Memory: Enhanced (you want long-term pattern recognition)
  • Cron: Enabled (we'll schedule monitoring runs)

Once installed, verify it's working:

hermes chat

# In the chat session, try:
# "Remember that I'm building an API monitoring agent"
# "What's the current date and time?"
Enter fullscreen mode Exit fullscreen mode

The agent should respond naturally and confirm it saved the memory about your project.

Step 2: Creating the API Monitoring Skill

Skills are reusable procedures that teach the agent how to perform specific workflows. We're going to create a skill called api-sentinel that contains the monitoring logic.

Create a new file at ~/.hermes/skills/api-sentinel.md:

mkdir -p ~/.hermes/skills
touch ~/.hermes/skills/api-sentinel.md
Enter fullscreen mode Exit fullscreen mode

Here's the skill definition:

# API Sentinel Skill

## Purpose
Monitor API health, consumption, and behavior patterns to detect issues before they become critical.

## Workflow

### 1. Data Collection Phase
- Fetch last 1 hour of error logs from CloudWatch
- Query Prometheus for API latency metrics (p50, p95, p99)
- Check API quota usage for all providers:
  - OpenAI API (via usage API)
  - Anthropic API (via usage API)
  - AWS API Gateway (via CloudWatch metrics)
- Fetch recent deployment events from CI/CD system

### 2. Pattern Analysis Phase
- Compare current metrics against 7-day baseline
- Identify anomalies:
  - Error rate increase >20%
  - Latency increase >50%
  - API consumption increase >30% without traffic increase
  - New error types not seen in past 7 days
- Correlate anomalies with recent deployments

### 3. Investigation Phase
For each detected anomaly:
- Retrieve sample error logs
- Analyze error messages for root cause indicators
- Check if similar issues occurred historically
- Assess impact severity (critical/warning/info)

### 4. Reporting Phase
- Generate incident report if critical issues found
- Update findings document with observations
- Send Telegram notification for critical/warning issues
- Update Prometheus metrics with analysis results

### 5. Documentation Sync Phase
- Compare current API behavior against OpenAPI spec
- Identify divergences (new fields, changed types, deprecated endpoints)
- Draft documentation update if needed

## Memory Keys
- `api_baseline_metrics`: 7-day rolling averages
- `known_error_patterns`: Previously seen and resolved errors
- `quota_thresholds`: Alert thresholds for each API provider
- `last_check_timestamp`: Track monitoring intervals

## Commands Available
- `check_cloudwatch_logs --hours 1`: Fetch recent logs
- `query_prometheus --metric api_latency_p95`: Query metrics
- `check_openai_usage`: Get OpenAI quota usage
- `check_anthropic_usage`: Get Anthropic quota usage
- `send_telegram_alert --severity critical --message "..."`: Send alert
Enter fullscreen mode Exit fullscreen mode

Save this file, then register the skill:

hermes skill load api-sentinel
Enter fullscreen mode Exit fullscreen mode

The agent can now execute this workflow on command or on schedule.

Step 3: Setting Up Data Sources

The agent needs access to your metrics and logs. Here's how to configure each integration:

CloudWatch Logs Integration

Create a script at ~/.hermes/tools/check_cloudwatch_logs.sh:

#!/bin/bash
# Fetch CloudWatch logs for the past N hours

HOURS=${1:-1}
START_TIME=$(($(date +%s) - $HOURS * 3600))000
LOG_GROUP="/aws/lambda/api-handler"

aws logs filter-log-events \
  --log-group-name "$LOG_GROUP" \
  --start-time "$START_TIME" \
  --filter-pattern "[ERROR]" \
  --max-items 100 \
  --output json | jq -r '.events[].message'
Enter fullscreen mode Exit fullscreen mode

Make it executable:

chmod +x ~/.hermes/tools/check_cloudwatch_logs.sh
Enter fullscreen mode Exit fullscreen mode

Prometheus Metrics Integration

Create ~/.hermes/tools/query_prometheus.sh:

#!/bin/bash
# Query Prometheus for specific metrics

METRIC=${1:-http_request_duration_seconds}
PROMETHEUS_URL="http://localhost:9090"

curl -s "$PROMETHEUS_URL/api/v1/query?query=$METRIC" | jq -r '.data.result'
Enter fullscreen mode Exit fullscreen mode

API Usage Tracking

For OpenAI usage tracking, create ~/.hermes/tools/check_openai_usage.py:

#!/usr/bin/env python3
import os
import requests
from datetime import datetime, timedelta

API_KEY = os.environ.get("OPENAI_API_KEY")
ORG_ID = os.environ.get("OPENAI_ORG_ID")

def check_usage():
    """Check OpenAI API usage for the current billing period"""
    headers = {
        "Authorization": f"Bearer {API_KEY}",
        "OpenAI-Organization": ORG_ID
    }

    # Get usage data
    end_date = datetime.now()
    start_date = end_date - timedelta(days=30)

    response = requests.get(
        "https://api.openai.com/v1/usage",
        headers=headers,
        params={
            "start_date": start_date.strftime("%Y-%m-%d"),
            "end_date": end_date.strftime("%Y-%m-%d")
        }
    )

    if response.status_code == 200:
        data = response.json()
        total_tokens = sum(day.get("n_context_tokens_total", 0) for day in data.get("data", []))
        total_cost = sum(day.get("n_context_tokens_total", 0) * 0.00002 for day in data.get("data", []))

        print(f"OpenAI Usage (30 days):")
        print(f"  Total tokens: {total_tokens:,}")
        print(f"  Estimated cost: ${total_cost:.2f}")
        print(f"  Daily average: {total_tokens // 30:,} tokens")

        return {
            "total_tokens": total_tokens,
            "total_cost": total_cost,
            "daily_average": total_tokens // 30
        }
    else:
        print(f"Error: {response.status_code} - {response.text}")
        return None

if __name__ == "__main__":
    check_usage()
Enter fullscreen mode Exit fullscreen mode

Make it executable:

chmod +x ~/.hermes/tools/check_openai_usage.py
Enter fullscreen mode Exit fullscreen mode

Step 4: Configuring the Monitoring Schedule

Now let's set up the agent to run every 6 hours:

hermes cron add \
  --name "api-sentinel-monitor" \
  --schedule "0 */6 * * *" \
  --prompt "Execute the api-sentinel skill to monitor API health and usage patterns. Focus on detecting anomalies and cost optimization opportunities."
Enter fullscreen mode Exit fullscreen mode

You can verify the cron job was added:

hermes cron list
Enter fullscreen mode Exit fullscreen mode

To test the monitoring run manually:

hermes chat --prompt "Execute the api-sentinel skill and provide a detailed report"
Enter fullscreen mode Exit fullscreen mode

Step 5: Setting Up Alerts and Notifications

The agent can send alerts to multiple channels. Let's configure Telegram for critical issues:

Telegram Setup

  1. Create a new bot via @BotFather
  2. Get your bot token
  3. Get your chat ID (send a message to your bot, then visit https://api.telegram.org/bot<TOKEN>/getUpdates)

Configure Hermes with your Telegram credentials:

hermes config set telegram.bot_token "YOUR_BOT_TOKEN"
hermes config set telegram.chat_id "YOUR_CHAT_ID"
Enter fullscreen mode Exit fullscreen mode

Create a notification skill at ~/.hermes/skills/send-alert.md:

# Send Alert Skill

## Purpose
Send formatted alerts to Telegram for API issues

## Workflow
1. Assess severity level (critical/warning/info)
2. Format message with:
   - Timestamp
   - Severity emoji (🔴/🟡/🔵)
   - Issue description
   - Impact assessment
   - Suggested action
3. Send via Telegram API
4. Log alert to memory for future reference

## Example Output
🔴 **CRITICAL: High API Consumption Detected**
Timestamp: 2026-08-06 14:23:15 UTC

OpenAI API usage is 340% above baseline.
Affected endpoint: /api/webhooks/process

Impact: $47/hour burn rate (normal: $12/hour)

Suggested Actions:
- Check for retry loops in webhook handler
- Review recent deployments
- Consider rate limiting

Investigation: [View Logs]
Enter fullscreen mode Exit fullscreen mode

Step 6: Real-World Usage Example

Let me show you what the agent actually does in practice. Here's a real monitoring run output:

$ hermes chat --prompt "Run api-sentinel monitoring"

🤖 Hermes Agent: Executing API Sentinel monitoring...

📊 Data Collection (14:30:15 UTC)
- CloudWatch logs: 23 error events in past 1 hour
- API latency (p95): 245ms (baseline: 180ms) ⚠️ +36%
- OpenAI usage: 2.3M tokens today (baseline: 1.8M) ⚠️ +28%
- Anthropic usage: Within normal range
- Recent deployments: 1 deployment 2 hours ago

🔍 Pattern Analysis
Anomaly detected: Latency spike correlates with deployment

Investigating error logs...
Found 18 instances of: "RetryableError: Anthropic API rate limit"

This suggests the retry logic in src/api/ai_handler.py is not backing off correctly.

🎯 Root Cause Analysis
Issue: Exponential backoff is configured but not being applied
Location: src/api/ai_handler.py:87
Fix: The retry decorator is using linear backoff instead of exponential

Impact Assessment: WARNING
- Current: ~18 failed requests/hour
- If unaddressed: Could escalate to service degradation
- Cost impact: Minimal (failed requests don't consume quota)

📝 Recommended Actions
1. Update retry configuration to use exponential backoff
2. Add circuit breaker for Anthropic API calls
3. Update monitoring to alert on retry rate

Would you like me to:
- Draft a GitHub issue for this?
- Send a Telegram alert?
- Update the API documentation?
Enter fullscreen mode Exit fullscreen mode

The agent didn't just tell me there was an error — it:

  1. Correlated the timing with a deployment
  2. Analyzed the error pattern
  3. Identified the specific code location
  4. Assessed business impact
  5. Suggested concrete fixes

Step 7: Advanced Features

Cost Optimization Analysis

Add a monthly cost review cron job:

hermes cron add \
  --name "monthly-cost-review" \
  --schedule "0 9 1 * *" \
  --prompt "Analyze API usage patterns from the past month. Identify opportunities to reduce costs through caching, batching, or provider optimization. Generate a detailed report with projected savings."
Enter fullscreen mode Exit fullscreen mode

This agent will analyze your API usage and suggest optimizations like:

  • Caching frequently requested data
  • Batching small requests
  • Switching to cheaper models for simple tasks
  • Identifying redundant API calls

Documentation Drift Detection

Create a skill that compares your OpenAPI spec against actual API behavior:

# Documentation Drift Detection Skill

## Purpose
Ensure API documentation stays in sync with actual implementation

## Workflow
1. Parse OpenAPI specification
2. Analyze actual API request/response logs
3. Detect mismatches:
   - New fields not in spec
   - Deprecated endpoints still in spec
   - Changed response types
   - New error codes
4. Draft documentation update PR
5. Notify team via Slack
Enter fullscreen mode Exit fullscreen mode

Anomaly Learning

The agent improves over time by learning from false positives:

hermes chat
> "The latency spike at 3am every day is expected - it's our nightly batch job. Don't alert on it."

🤖 Understood. I've updated my memory:
- Ignore latency spikes between 3:00-3:30am
- Associated with: nightly batch processing
- Pattern: expected operational behavior
Enter fullscreen mode Exit fullscreen mode

Troubleshooting Common Issues

Issue 1: Agent Not Detecting Anomalies

Symptom: Agent runs but reports "no issues found" even when there are obvious problems

Diagnosis: The baseline metrics might not be properly calibrated

Fix:

# Reset baseline and recalibrate
hermes memory delete api_baseline_metrics
hermes chat --prompt "Analyze the past 7 days of API metrics to establish a new baseline"
Enter fullscreen mode Exit fullscreen mode

Issue 2: Too Many False Positive Alerts

Symptom: Getting alerts for normal traffic variations

Fix: Adjust sensitivity thresholds:

hermes config set api_sentinel.latency_threshold 50  # Increase from 30%
hermes config set api_sentinel.error_threshold 25    # Increase from 20%
Enter fullscreen mode Exit fullscreen mode

Issue 3: Missing API Provider Credentials

Symptom: "Cannot fetch usage data" errors

Fix: Ensure environment variables are set:

export OPENAI_API_KEY="sk-..."
export ANTHROPIC_API_KEY="sk-ant-..."
export AWS_ACCESS_KEY_ID="..."
export AWS_SECRET_ACCESS_KEY="..."
Enter fullscreen mode Exit fullscreen mode

Cost Breakdown: Running This Agent

Let me be transparent about what this costs to run:

Infrastructure:

  • Hermes Agent: Free (self-hosted)
  • SQLite storage: Free
  • Prometheus: Free (self-hosted)

LLM Costs (Claude 3.5 Sonnet):

  • Monitoring run: ~5K input tokens, ~2K output tokens
  • Cost per run: ~$0.08
  • 4 runs/day: ~$9.60/month

Data Transfer:

  • CloudWatch API calls: Minimal (~$2/month)
  • Metrics queries: Free (self-hosted Prometheus)

Total: ~$12/month

For reference, that one incident I mentioned at the start cost us $2,847. This agent would have caught it within 6 hours and cost me $0.08.

ROI is pretty clear.

Beyond Monitoring: What Else Can This Do?

Once you have an agent watching your APIs, you can extend it to:

  1. Automatic Incident Response: Not just detect but also remediate (e.g., scale up resources, enable circuit breakers)
  2. Security Monitoring: Detect unusual access patterns that might indicate an attack
  3. Compliance Reporting: Generate audit reports for SOC2, ISO 27001, etc.
  4. Performance Optimization: Suggest code changes based on profiling data
  5. User Behavior Analysis: Understand how your API is actually being used vs. how you designed it

The key is that you have a reasoning engine that can look at data holistically, not just individual metrics in isolation.

Conclusion: Monitoring That Thinks

Traditional monitoring tells you what happened. This agent tells you why it happened and what to do about it.

It's the difference between:

  • "Error rate increased to 5%"
  • vs. "Error rate increased to 5% because the deployment 2 hours ago introduced a retry loop in the webhook handler at line 87. This is costing $35/hour in unnecessary API calls. Fix: change the retry strategy from linear to exponential backoff. Here's the code change: [...]"

I've been running this for three months now. It's caught:

  • 7 potential incidents before they impacted users
  • $3,200 in unnecessary API spending
  • 4 documentation drift issues
  • 2 security concerns (unusual API access patterns)

More importantly, I sleep better knowing that if something goes wrong at 3am, I'll get an intelligent alert that tells me exactly what's wrong and what to do — not just a cryptic "Error rate high" page.

The code is open source. The agent framework is free. The LLM costs are negligible compared to what you'll save in incident prevention.

Build it. Run it. Improve it. And let me know what patterns your agent discovers.

Top comments (0)