DEV Community: Violeta Fragosa

Building an AI Agent That Actually Monitors Your APIs

Violeta Fragosa — Thu, 06 Aug 2026 08:38:22 +0000

The $50,000 Mistake I Almost Made

Last November, I deployed a new API endpoint for our SaaS platform. Everything looked fine in staging. The integration tests passed. The load tests showed acceptable response times. I pushed to production on a Friday afternoon (yes, I know).

By Monday morning, we had burned through 47% of our Anthropic API quota. Not from legitimate traffic — from a retry loop in our webhook handler that nobody caught because our monitoring alerts were configured for HTTP errors, not for API consumption patterns or logical errors in flow control.

The fix took 30 seconds once we found it. But the detection took three days. And by then, we'd spent $2,847 on unnecessary API calls, dangerously close to hitting the hard limit that would have taken down our entire service.

That's when I realized: monitoring HTTP status codes isn't enough. You need an agent that understands your API's behavior, tracks consumption patterns, detects anomalies, and can even draft incident reports or documentation updates based on what it observes.

So I built one.

Why Traditional API Monitoring Fails

Before we dive into the build, let's talk about why Datadog, New Relic, and CloudWatch aren't enough for modern API-driven systems.

Traditional monitoring is reactive. It tells you when something is already broken:

500 errors? Alert fires.
Response time over 2 seconds? Alert fires.
Error rate above 5%? Alert fires.

But what about:

A slow increase in API consumption that indicates a loop or inefficiency?
A new endpoint being called in an unusual pattern suggesting misintegration?
Changes in request payloads that could indicate a breaking change you introduced?
Deprecation warnings from third-party APIs you depend on?
Opportunities to optimize based on actual usage patterns?

These are behavioral signals that require reasoning, not just threshold checking. And that's where an autonomous agent comes in.

What We're Building: The API Sentinel Agent

Here's what this agent does:

Continuous Monitoring: Polls your API logs, cloud metrics, and third-party API status pages
Pattern Recognition: Uses LLM reasoning to detect anomalies, not just threshold violations
Consumption Tracking: Tracks API quota usage across multiple providers (OpenAI, Anthropic, Google, etc.)
Documentation Sync: Automatically detects when your API behavior diverges from your docs and drafts updates
Incident Reporting: When something goes wrong, generates a detailed incident report with timeline, impact analysis, and suggested fixes
Cost Optimization: Identifies opportunities to reduce API costs based on actual usage patterns

Unlike my previous post about Hermes Agent for tech radar, this one focuses on operational monitoring rather than information gathering. But the underlying principles are the same: memory, reasoning, scheduling, and action.

The Tech Stack

Here's what I used to build this:

Core Agent Framework: Hermes Agent (same as before, but different skills)
Log Ingestion: Fluent Bit → CloudWatch Logs → Agent API polling
Metrics: Prometheus + custom exporters for API provider quotas
LLM: Claude 3.5 Sonnet (best for reasoning about patterns and anomalies)
Storage: SQLite for agent memory + PostgreSQL for historical metrics
Notifications: Telegram for critical alerts, weekly digest via email

The key architectural decision was making the agent pull-based rather than push-based. Instead of sending every log line to the LLM (expensive and slow), the agent periodically queries aggregated metrics and recent error logs, then reasons about whether patterns are worth investigating.

Step 1: Setting Up the Base Agent

If you haven't already installed Hermes Agent, start here:

# Install Hermes Agent
curl -fsSL https://raw.githubusercontent.com/NousResearch/hermes-agent/main/scripts/install.sh | bash

# Run the interactive setup
hermes setup --portal

# This will:
# - Configure your preferred LLM provider (OpenAI, Anthropic, local, etc.)
# - Set up the SQLite memory database
# - Create your first agent session
# - Configure optional integrations (Telegram, Slack, etc.)

During setup, choose these options:

LLM Provider: Anthropic (Claude 3.5 Sonnet has the best reasoning for pattern detection)
Memory: Enhanced (you want long-term pattern recognition)
Cron: Enabled (we'll schedule monitoring runs)

Once installed, verify it's working:

hermes chat

# In the chat session, try:
# "Remember that I'm building an API monitoring agent"
# "What's the current date and time?"

The agent should respond naturally and confirm it saved the memory about your project.

Step 2: Creating the API Monitoring Skill

Skills are reusable procedures that teach the agent how to perform specific workflows. We're going to create a skill called api-sentinel that contains the monitoring logic.

Create a new file at ~/.hermes/skills/api-sentinel.md:

mkdir -p ~/.hermes/skills
touch ~/.hermes/skills/api-sentinel.md

Here's the skill definition:

# API Sentinel Skill

## Purpose
Monitor API health, consumption, and behavior patterns to detect issues before they become critical.

## Workflow

### 1. Data Collection Phase
- Fetch last 1 hour of error logs from CloudWatch
- Query Prometheus for API latency metrics (p50, p95, p99)
- Check API quota usage for all providers:
  - OpenAI API (via usage API)
  - Anthropic API (via usage API)
  - AWS API Gateway (via CloudWatch metrics)
- Fetch recent deployment events from CI/CD system

### 2. Pattern Analysis Phase
- Compare current metrics against 7-day baseline
- Identify anomalies:
  - Error rate increase >20%
  - Latency increase >50%
  - API consumption increase >30% without traffic increase
  - New error types not seen in past 7 days
- Correlate anomalies with recent deployments

### 3. Investigation Phase
For each detected anomaly:
- Retrieve sample error logs
- Analyze error messages for root cause indicators
- Check if similar issues occurred historically
- Assess impact severity (critical/warning/info)

### 4. Reporting Phase
- Generate incident report if critical issues found
- Update findings document with observations
- Send Telegram notification for critical/warning issues
- Update Prometheus metrics with analysis results

### 5. Documentation Sync Phase
- Compare current API behavior against OpenAPI spec
- Identify divergences (new fields, changed types, deprecated endpoints)
- Draft documentation update if needed

## Memory Keys
- `api_baseline_metrics`: 7-day rolling averages
- `known_error_patterns`: Previously seen and resolved errors
- `quota_thresholds`: Alert thresholds for each API provider
- `last_check_timestamp`: Track monitoring intervals

## Commands Available
- `check_cloudwatch_logs --hours 1`: Fetch recent logs
- `query_prometheus --metric api_latency_p95`: Query metrics
- `check_openai_usage`: Get OpenAI quota usage
- `check_anthropic_usage`: Get Anthropic quota usage
- `send_telegram_alert --severity critical --message "..."`: Send alert

Save this file, then register the skill:

hermes skill load api-sentinel

The agent can now execute this workflow on command or on schedule.

Step 3: Setting Up Data Sources

The agent needs access to your metrics and logs. Here's how to configure each integration:

CloudWatch Logs Integration

Create a script at ~/.hermes/tools/check_cloudwatch_logs.sh:

#!/bin/bash
# Fetch CloudWatch logs for the past N hours

HOURS=${1:-1}
START_TIME=$(($(date +%s) - $HOURS * 3600))000
LOG_GROUP="/aws/lambda/api-handler"

aws logs filter-log-events \
  --log-group-name "$LOG_GROUP" \
  --start-time "$START_TIME" \
  --filter-pattern "[ERROR]" \
  --max-items 100 \
  --output json | jq -r '.events[].message'

Make it executable:

chmod +x ~/.hermes/tools/check_cloudwatch_logs.sh

Prometheus Metrics Integration

Create ~/.hermes/tools/query_prometheus.sh:

#!/bin/bash
# Query Prometheus for specific metrics

METRIC=${1:-http_request_duration_seconds}
PROMETHEUS_URL="http://localhost:9090"

curl -s "$PROMETHEUS_URL/api/v1/query?query=$METRIC" | jq -r '.data.result'

API Usage Tracking

For OpenAI usage tracking, create ~/.hermes/tools/check_openai_usage.py:

#!/usr/bin/env python3
import os
import requests
from datetime import datetime, timedelta

API_KEY = os.environ.get("OPENAI_API_KEY")
ORG_ID = os.environ.get("OPENAI_ORG_ID")

def check_usage():
    """Check OpenAI API usage for the current billing period"""
    headers = {
        "Authorization": f"Bearer {API_KEY}",
        "OpenAI-Organization": ORG_ID
    }

    # Get usage data
    end_date = datetime.now()
    start_date = end_date - timedelta(days=30)

    response = requests.get(
        "https://api.openai.com/v1/usage",
        headers=headers,
        params={
            "start_date": start_date.strftime("%Y-%m-%d"),
            "end_date": end_date.strftime("%Y-%m-%d")
        }
    )

    if response.status_code == 200:
        data = response.json()
        total_tokens = sum(day.get("n_context_tokens_total", 0) for day in data.get("data", []))
        total_cost = sum(day.get("n_context_tokens_total", 0) * 0.00002 for day in data.get("data", []))

        print(f"OpenAI Usage (30 days):")
        print(f"  Total tokens: {total_tokens:,}")
        print(f"  Estimated cost: ${total_cost:.2f}")
        print(f"  Daily average: {total_tokens // 30:,} tokens")

        return {
            "total_tokens": total_tokens,
            "total_cost": total_cost,
            "daily_average": total_tokens // 30
        }
    else:
        print(f"Error: {response.status_code} - {response.text}")
        return None

if __name__ == "__main__":
    check_usage()

Make it executable:

chmod +x ~/.hermes/tools/check_openai_usage.py

Step 4: Configuring the Monitoring Schedule

Now let's set up the agent to run every 6 hours:

hermes cron add \
  --name "api-sentinel-monitor" \
  --schedule "0 */6 * * *" \
  --prompt "Execute the api-sentinel skill to monitor API health and usage patterns. Focus on detecting anomalies and cost optimization opportunities."

You can verify the cron job was added:

hermes cron list

To test the monitoring run manually:

hermes chat --prompt "Execute the api-sentinel skill and provide a detailed report"

Step 5: Setting Up Alerts and Notifications

The agent can send alerts to multiple channels. Let's configure Telegram for critical issues:

Telegram Setup

Create a new bot via @BotFather
Get your bot token
Get your chat ID (send a message to your bot, then visit https://api.telegram.org/bot<TOKEN>/getUpdates)

Configure Hermes with your Telegram credentials:

hermes config set telegram.bot_token "YOUR_BOT_TOKEN"
hermes config set telegram.chat_id "YOUR_CHAT_ID"

Create a notification skill at ~/.hermes/skills/send-alert.md:

# Send Alert Skill

## Purpose
Send formatted alerts to Telegram for API issues

## Workflow
1. Assess severity level (critical/warning/info)
2. Format message with:
   - Timestamp
   - Severity emoji (🔴/🟡/🔵)
   - Issue description
   - Impact assessment
   - Suggested action
3. Send via Telegram API
4. Log alert to memory for future reference

## Example Output
🔴 **CRITICAL: High API Consumption Detected**
Timestamp: 2026-08-06 14:23:15 UTC

OpenAI API usage is 340% above baseline.
Affected endpoint: /api/webhooks/process

Impact: $47/hour burn rate (normal: $12/hour)

Suggested Actions:
- Check for retry loops in webhook handler
- Review recent deployments
- Consider rate limiting

Investigation: [View Logs]

Step 6: Real-World Usage Example

Let me show you what the agent actually does in practice. Here's a real monitoring run output:

$ hermes chat --prompt "Run api-sentinel monitoring"

🤖 Hermes Agent: Executing API Sentinel monitoring...

📊 Data Collection (14:30:15 UTC)
- CloudWatch logs: 23 error events in past 1 hour
- API latency (p95): 245ms (baseline: 180ms) ⚠️ +36%
- OpenAI usage: 2.3M tokens today (baseline: 1.8M) ⚠️ +28%
- Anthropic usage: Within normal range
- Recent deployments: 1 deployment 2 hours ago

🔍 Pattern Analysis
Anomaly detected: Latency spike correlates with deployment

Investigating error logs...
Found 18 instances of: "RetryableError: Anthropic API rate limit"

This suggests the retry logic in src/api/ai_handler.py is not backing off correctly.

🎯 Root Cause Analysis
Issue: Exponential backoff is configured but not being applied
Location: src/api/ai_handler.py:87
Fix: The retry decorator is using linear backoff instead of exponential

Impact Assessment: WARNING
- Current: ~18 failed requests/hour
- If unaddressed: Could escalate to service degradation
- Cost impact: Minimal (failed requests don't consume quota)

📝 Recommended Actions
1. Update retry configuration to use exponential backoff
2. Add circuit breaker for Anthropic API calls
3. Update monitoring to alert on retry rate

Would you like me to:
- Draft a GitHub issue for this?
- Send a Telegram alert?
- Update the API documentation?

The agent didn't just tell me there was an error — it:

Correlated the timing with a deployment
Analyzed the error pattern
Identified the specific code location
Assessed business impact
Suggested concrete fixes

Step 7: Advanced Features

Cost Optimization Analysis

Add a monthly cost review cron job:

hermes cron add \
  --name "monthly-cost-review" \
  --schedule "0 9 1 * *" \
  --prompt "Analyze API usage patterns from the past month. Identify opportunities to reduce costs through caching, batching, or provider optimization. Generate a detailed report with projected savings."

This agent will analyze your API usage and suggest optimizations like:

Caching frequently requested data
Batching small requests
Switching to cheaper models for simple tasks
Identifying redundant API calls

Documentation Drift Detection

Create a skill that compares your OpenAPI spec against actual API behavior:

# Documentation Drift Detection Skill

## Purpose
Ensure API documentation stays in sync with actual implementation

## Workflow
1. Parse OpenAPI specification
2. Analyze actual API request/response logs
3. Detect mismatches:
   - New fields not in spec
   - Deprecated endpoints still in spec
   - Changed response types
   - New error codes
4. Draft documentation update PR
5. Notify team via Slack

Anomaly Learning

The agent improves over time by learning from false positives:

hermes chat
> "The latency spike at 3am every day is expected - it's our nightly batch job. Don't alert on it."

🤖 Understood. I've updated my memory:
- Ignore latency spikes between 3:00-3:30am
- Associated with: nightly batch processing
- Pattern: expected operational behavior

Troubleshooting Common Issues

Issue 1: Agent Not Detecting Anomalies

Symptom: Agent runs but reports "no issues found" even when there are obvious problems

Diagnosis: The baseline metrics might not be properly calibrated

Fix:

# Reset baseline and recalibrate
hermes memory delete api_baseline_metrics
hermes chat --prompt "Analyze the past 7 days of API metrics to establish a new baseline"

Issue 2: Too Many False Positive Alerts

Symptom: Getting alerts for normal traffic variations

Fix: Adjust sensitivity thresholds:

hermes config set api_sentinel.latency_threshold 50  # Increase from 30%
hermes config set api_sentinel.error_threshold 25    # Increase from 20%

Issue 3: Missing API Provider Credentials

Symptom: "Cannot fetch usage data" errors

Fix: Ensure environment variables are set:

export OPENAI_API_KEY="sk-..."
export ANTHROPIC_API_KEY="sk-ant-..."
export AWS_ACCESS_KEY_ID="..."
export AWS_SECRET_ACCESS_KEY="..."

Cost Breakdown: Running This Agent

Let me be transparent about what this costs to run:

Infrastructure:

Hermes Agent: Free (self-hosted)
SQLite storage: Free
Prometheus: Free (self-hosted)

LLM Costs (Claude 3.5 Sonnet):

Monitoring run: ~5K input tokens, ~2K output tokens
Cost per run: ~$0.08
4 runs/day: ~$9.60/month

Data Transfer:

CloudWatch API calls: Minimal (~$2/month)
Metrics queries: Free (self-hosted Prometheus)

Total: ~$12/month

For reference, that one incident I mentioned at the start cost us $2,847. This agent would have caught it within 6 hours and cost me $0.08.

ROI is pretty clear.

Beyond Monitoring: What Else Can This Do?

Once you have an agent watching your APIs, you can extend it to:

Automatic Incident Response: Not just detect but also remediate (e.g., scale up resources, enable circuit breakers)
Security Monitoring: Detect unusual access patterns that might indicate an attack
Compliance Reporting: Generate audit reports for SOC2, ISO 27001, etc.
Performance Optimization: Suggest code changes based on profiling data
User Behavior Analysis: Understand how your API is actually being used vs. how you designed it

The key is that you have a reasoning engine that can look at data holistically, not just individual metrics in isolation.

Conclusion: Monitoring That Thinks

Traditional monitoring tells you what happened. This agent tells you why it happened and what to do about it.

It's the difference between:

"Error rate increased to 5%"
vs. "Error rate increased to 5% because the deployment 2 hours ago introduced a retry loop in the webhook handler at line 87. This is costing $35/hour in unnecessary API calls. Fix: change the retry strategy from linear to exponential backoff. Here's the code change: [...]"

I've been running this for three months now. It's caught:

7 potential incidents before they impacted users
$3,200 in unnecessary API spending
4 documentation drift issues
2 security concerns (unusual API access patterns)

More importantly, I sleep better knowing that if something goes wrong at 3am, I'll get an intelligent alert that tells me exactly what's wrong and what to do — not just a cryptic "Error rate high" page.

The code is open source. The agent framework is free. The LLM costs are negligible compared to what you'll save in incident prevention.

Build it. Run it. Improve it. And let me know what patterns your agent discovers.

Stop hitting Codex rate limits mid-task

Violeta Fragosa — Mon, 03 Aug 2026 15:31:48 +0000

Hit Codex's five-hour limit again last week, right in the middle of debugging a state management issue. Agent had already burned through half the context window figuring out the problem, and then — 429. Session gone. Had to wait two hours to pick it back up.

That sucks. And it's happened enough times that I finally did something about it. Built a local proxy.

Codex Conduit sits between your Codex client and whichever accounts or API providers you use. It shows quota status, lets you switch routes mid-session without losing context, and auto-fails over to another account when one hits the five-hour limit or throws a rate error.

What it does

Manage ChatGPT subscription accounts, OpenAI API keys, and third-party providers in one place
See five-hour and weekly quota usage (where the upstream reports it), plus account health
Switch between accounts without restarting Codex — keeps your session and context intact
Auto-switch to another account when one hits the five-hour limit, so your task keeps running
Control which routes participate in auto-selection (free, relay, paid, third-party)
Backs up your original ~/.codex/config.toml and restores it when you disable the proxy

Setup

Grab the installer from the releases page. macOS and Windows builds are up. Quit any running Codex CLI or Codex App processes before enabling the proxy — saves you from weird config states.

Add accounts: Click + Sign In to go through OpenAI's OAuth flow in your browser. For third-party providers that expose /v1/responses or /chat/completions, use + Add Third-Party Provider and paste your API key.

Check status: The Accounts page shows which route is active, remaining quota or balance, health state, and when it was last refreshed.

Enable the proxy: Flip Local Proxy on in the Dashboard. The app backs up your Codex config, writes the proxy settings, and starts routing. Restart your Codex client and you're set. Turn it off later and it restores everything.

Routing policy: Under Settings > Background Services, pick which routes can auto-switch.

My setup: third-party routes stay out of auto-selection (I want to explicitly choose when I'm paying per token), subscription accounts are available as fallback when a paid route dies, and free-tier accounts stay out of rotation.

How failover works

The gateway sits between your Codex client and the upstream:

Codex CLI / Codex App / IDE
               |
               v
      Codex Conduit (local)
               |
    healthy account or provider

When it sees 401, 429, or a quota-exhausted response, it picks the next healthy route that matches your policy. You still hit provider limits — this just saves you from manually reconfiguring when one runs dry. The dashboard shows which route is active, so you know what you're using.

Does this actually help?

The five-hour window is still there on individual subscription routes. Can't get around that.

But I'm wasting less time checking quota before I start, switching accounts mid-session, editing configs, restarting Codex... you know the drill. The dashboard gives me a quick answer to "which account should I use right now" instead of trial and error.

Try it / break it

Repo: https://github.com/cherrylfrei-max/codex-conduit

Bugs, feature requests, PRs — all welcome. If something's broken or unclear, open an issue and I'll take a look.