Which Commit Made the API Slow? A $0 Performance Detective

Performance regressions are the quiet failures. The API still returns 200s, tests still pass, but users feel the difference. This case study shows a $0 way to catch them: a free server that watches response-time data, flags anomalies with statistics, and uses a free model to point at the most likely culprit commit.

The problem

Your API was fast last week. Today it's 300ms slower on the same endpoint. No test failed. No error was logged. Which commit did it?

Manual hunting is painful. You check the deploy history, diff every commit, and guess. Most teams give up after a few hours.

The goal

Build a detector that runs after every deploy and answers three questions:

1. Is this deploy slower than the baseline?
2. Which endpoints regressed?
3. Which commits are the most likely causes?

The answer must come fast enough to act on, and the budget must stay at zero.

The stack

• Python for the collector and analyzer.
• SQLite for storing response-time samples.
• A cron job on a free server — MonkeyCode's free server option — that runs the analysis after each deploy.
• Free model tokens — MonkeyCode's free tier (10M tokens) — for commit triage.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Step 1: Collect response-time data

The collector pulls response-time percentiles from your existing monitoring (or from access logs if you have none). Store them per endpoint per deploy:

import sqlite3
from datetime import datetime

conn = sqlite3.connect("perf.db")
conn.execute("""
    CREATE TABLE IF NOT EXISTS samples (
        deploy_id TEXT,
        endpoint TEXT,
        p50 REAL,
        p95 REAL,
        p99 REAL,
        collected_at TEXT
    )
""")

def collect(deploy_id, endpoint, p50, p95, p99):
    conn.execute(
        "INSERT INTO samples VALUES (?, ?, ?, ?, ?, ?)",
        (deploy_id, endpoint, p50, p95, p99, datetime.utcnow().isoformat())
    )
    conn.commit()

Step 2: Establish a baseline

A rolling baseline works better than a fixed one. Use the last 20 samples per endpoint, excluding the current deploy:

def baseline(endpoint, current_deploy):
    rows = conn.execute("""
        SELECT p95 FROM samples
        WHERE endpoint = ? AND deploy_id != ?
        ORDER BY collected_at DESC LIMIT 20
    """, (endpoint, current_deploy)).fetchall()
    values = [r[0] for r in rows]
    if not values:
        return None, None
    mean = sum(values) / len(values)
    variance = sum((v - mean) ** 2 for v in values) / len(values)
    return mean, variance ** 0.5

Step 3: Flag anomalies with statistics

A deploy is suspicious if its p95 is more than two standard deviations above the baseline:

def is_anomalous(endpoint, current_p95, current_deploy):
    mean, std = baseline(endpoint, current_deploy)
    if mean is None:
        return False  # not enough data yet
    if std == 0:
        return current_p95 > mean * 1.2  # fallback for stable endpoints
    return current_p95 > mean + 2 * std

Statistics do the heavy lifting. The model never sees raw numbers.

Step 4: Let the free model triage commits

This is where the free model earns its keep. Once anomalies are flagged, collect the commits in the deploy and ask the model to rank them:

TRIAGE_PROMPT = """You are a performance engineer. A deploy caused a p95 regression on {endpoint} (from {baseline_ms}ms to {current_ms}ms).

Commits in this deploy:
{commits}

Return JSON only:
{"ranked_commits": ["hash", "..."], "reason": "one sentence per commit"}
"""

The model's output is a ranked list. The team reviews the top candidates first. The model never blocks a deploy and never reverts a commit — it only shortens the search.

What happened when I ran it

The detector found a regression that had been live for three days. The cause: a commit that added a synchronous HTTP call inside a hot path. No test caught it because the test mocked the external service.

The numbers:

• Detection time: under 5 minutes after the deploy.
• False positives: 2 in the first week, both caused by noisy load tests.
• Time saved: roughly 2 hours of manual diffing per incident.

Limitations

• Statistics need data. A brand-new endpoint with 3 samples has no meaningful baseline.
• The model sees only commit messages and diffs. A commit that says "refactor" can hide a perf disaster.
• Not for real-time alerting. This is a post-deploy check, not a streaming monitor.
• The 10M token budget is enough for hundreds of runs, but not for continuous analysis of every request.

Who should not use this

If your API has no traffic, no baseline, and no deploy cadence, this tool has nothing to watch. If you already have an APM with regression alerts, this adds little. The sweet spot is a small team with a real API and no budget.

Try it

The whole system is about 200 lines of Python. MonkeyCode's open-source project gives you a free server and 10M free tokens — enough to run this detector for a small API for months. Build it, let it watch your next deploy, and see what it finds.