I Tracked 50,000 API Errors — Here Are the 5 Patterns That Matter

Last month, I set up a simple error-tracking system for my APIs. Not Sentry. Not Datadog. Just a SQLite database, a few lines of logging middleware, and a cron job that emails me a summary every morning.

After 30 days and 50,000+ logged errors, patterns emerged that completely changed how I think about API reliability. Here's what I found.

The Setup

Before diving into the patterns, here's what I built. It's embarrassingly simple:

import sqlite3
import time
import traceback
from datetime import datetime

DB_PATH = "api_errors.db"

def init_db():
    conn = sqlite3.connect(DB_PATH)
    conn.execute("""
        CREATE TABLE IF NOT EXISTS errors (
            id INTEGER PRIMARY KEY AUTOINCREMENT,
            endpoint TEXT NOT NULL,
            method TEXT NOT NULL,
            status_code INTEGER,
            error_type TEXT NOT NULL,
            error_message TEXT,
            traceback_text TEXT,
            user_agent TEXT,
            ip_hash TEXT,
            timestamp DATETIME DEFAULT CURRENT_TIMESTAMP
        )
    """)
    conn.execute("CREATE INDEX IF NOT EXISTS idx_error_type ON errors(error_type)")
    conn.execute("CREATE INDEX IF NOT EXISTS idx_endpoint ON errors(endpoint)")
    conn.execute("CREATE INDEX IF NOT EXISTS idx_timestamp ON errors(timestamp)")
    conn.commit()
    return conn

def log_error(conn, endpoint, method, status_code, error, request_headers=None):
    error_type = type(error).__name__
    conn.execute("""
        INSERT INTO errors (endpoint, method, status_code, error_type, error_message, traceback_text, user_agent, ip_hash)
        VALUES (?, ?, ?, ?, ?, ?, ?, ?)
    """, (
        endpoint, method, status_code, error_type,
        str(error)[:500],
        traceback.format_exc()[:2000],
        (request_headers or {}).get("User-Agent", "")[:200],
        "hashed_for_privacy"
    ))
    conn.commit()

That's it. No SaaS subscription. No SDK integration. Just a database and a function call.

The 5 Error Patterns

1. Timeout Errors Are Never About Timeouts (38% of errors)

This was my biggest surprise. 38% of all errors were TimeoutError or ReadTimeout — but almost none were actually about network latency.

The real causes:

• Stuck database connections (connection pool exhausted, waiting for a lock)
• Downstream service hangs (a third-party API that never responds)
• GC pauses (Python's garbage collector freezing a request mid-flight)

The fix wasn't "increase timeouts." The fix was:

• Connection pool monitoring
• Circuit breakers on downstream calls
• Pre-forking worker processes to isolate GC

# Circuit breaker in 10 lines
class CircuitBreaker:
    def __init__(self, failure_threshold=5, recovery_timeout=30):
        self.failures = 0
        self.threshold = failure_threshold
        self.recovery = recovery_timeout
        self.last_failure = 0

    def call(self, fn, *args, **kwargs):
        if self.failures >= self.threshold:
            if time.time() - self.last_failure < self.recovery:
                raise Exception("Circuit open")
            self.failures = 0  # try again
        try:
            result = fn(*args, **kwargs)
            self.failures = 0
            return result
        except Exception:
            self.failures += 1
            self.last_failure = time.time()
            raise

2. 4xx Errors at 3 AM Are Always Bots (27%)

I kept getting paged for 401 and 403 errors in the middle of the night. After checking the user_agent and timing patterns, it was obvious: bots scanning for open endpoints.

Patterns I identified:

• /wp-admin, /.env, /admin/config — classic vulnerability scanners
• Rapid bursts: 50+ requests per second from a single IP
• User agents like python-requests/2.x or totally missing

What I did: Added a simple bot detection layer that checks request velocity and known scanner paths. If triggered, it returns a fake 200 response with no data. Bots think they succeeded and move on. My alerting no longer fires at 3 AM.

BOT_PATHS = {"/wp-admin", "/.env", "/admin", "/config", "/phpmyadmin", "/.git/config"}

def detect_bot(path, user_agent, ip_hash, redis_conn):
    if path in BOT_PATHS:
        return True
    if not user_agent:
        return True
    # Rate check: > 20 req/sec from same IP
    count = redis_conn.incr(f"rate:{ip_hash}")
    redis_conn.expire(f"rate:{ip_hash}", 1)
    return count > 20

3. Validation Errors Mean Your Error Messages Suck (19%)

Nearly 1 in 5 errors were ValidationError — bad JSON, missing fields, wrong types. But here's the thing: most of these weren't malicious. They were integration mistakes.

Developers integrating with my API were getting error messages like:

{"error": "Invalid request body"}

That's useless. They have no idea what is invalid.

I rewrote all validation errors to be specific and helpful:

# Before
raise ValidationError("Invalid request body")

# After
raise ValidationError({
    "error": "validation_failed",
    "details": [
        {
            "field": "email",
            "value": "not-an-email",
            "reason": "Must be a valid email address"
        },
        {
            "field": "age",
            "value": -5,
            "reason": "Must be a positive integer"
        }
    ],
    "help": "See https://docs.example.com/api/users#create"
})

Validation errors dropped 60% in the following week. Not because people got better at sending requests — because they could now fix their mistakes on the first retry.

4. The "Everything Is Fine" Errors (11%)

These are the sneaky ones. Errors that don't actually break anything but clutter your logs.

Examples from my data:

• KeyError in a logging statement (the log itself crashed)
• DeprecationWarning from a library upgrade
• ResourceWarning: unclosed file in a test runner
• Background task failures that retried successfully

These aren't real errors. They're noise. But they made up 11% of my log volume.

What I did: Added severity levels to my error logger and filtered out anything below WARNING from alerts. Now I only page on CRITICAL and ERROR — not warnings, not deprecations, not "everything is fine but I wanted to tell you something."

5. The Cascade Effect: One Bug, 500 Errors (5%)

The remaining 5% were cascading failures. One bad database query would fail, the error handler would try to log the error and fail too, the retry logic would kick in and fail again... Suddenly one real bug produced 500 error entries.

Root cause: Error handlers that make external calls. My logging function was writing to a database that was itself failing. Infinite loop.

# Dangerous: error handler depends on the very system that's failing
def handle_error(error):
    db.insert(error)  # What if the DB is the problem?
    send_alert(error) # What if the network is the problem?

# Safe: error handler is self-contained
def handle_error(error):
    # Always write to stderr first — no dependencies
    import sys
    print(json.dumps({"error": str(error), "time": time.time()}), file=sys.stderr, flush=True)
    # Then try best-effort storage
    try:
        db.insert(error)
    except:
        pass  # Already logged to stderr

What I Changed

After 30 days of tracking, here's what actually moved the needle:

Change	Error Reduction
Circuit breakers on downstream calls	-42% timeout errors
Specific validation error messages	-60% validation errors
Bot detection + fake 200s	-100% 3 AM alerts
Severity-based alerting	-89% alert volume
Self-contained error handlers	-95% cascade events

Total cost: $0. Total time to implement: one weekend.

The Takeaway

You don't need an expensive observability platform to understand your API errors. You need:

1. A place to store them (SQLite works)
2. A way to query them (SQL is fine)
3. The discipline to actually look at the data

That third one is the hard part. I set up a cron job that emails me a daily summary:

Today: 1,247 errors
Top endpoint: /api/users (342)
Top error: ValidationError (233) — mostly missing 'email' field
Bot hits blocked: 127
Cascade events: 0

It takes 30 seconds to read and has saved me from deploying bugs that would have taken hours to debug.

Start tracking your API errors. The patterns will surprise you.

---

What's the most surprising thing you've learned from your API error logs? I'd love to hear in the comments.