Every AI Patch Deserves a Baseline

Here is my conclusion up front: I almost merged a regression into production.

The AI patch looked clean. It replaced three loops with one query. It removed forty lines. The pull request promised a solid speedup. I read the diff, trusted my eyes, and clicked merge. I never ran a baseline.

A week later, the database pool hit 100% during peak hours. The slowest trace went through the endpoint that patch had touched. My intuition approved that code. The profiler disagreed.

Now every AI-generated change gets measured before merge. No elaborate harness. Just a small script and a text file. That little pair catches most performance lies.

Why the baseline matters

A diff shows what changed. It never shows what got faster. Speed is a measurement, not a property of a deleted loop. So I compare a candidate patch against the commit it replaces.

The script stays boring on purpose:

#!/usr/bin/env bash
# bench_patch.sh <base_commit> <patch_commit> <endpoint>
# Requires a clean tree. The server must listen on :3000.
# Raise the sleep below if your service boots slowly.
set -euo pipefail

BASE="$1"
PATCH="$2"
URL="$3"
RUNS="${RUNS:-200}"
WARMUP="${WARMUP:-20}"
EXPECTED_CODE="${EXPECTED_CODE:-200}"

if [ -n "$(git status --porcelain)" ]; then
  echo "Working tree is dirty. Commit or stash first." >&2
  exit 1
fi

measure() {
  local commit="$1"
  local label="$2"
  local errors=0

  git checkout --quiet "$commit"
  npm ci --silent
  node server.js >/tmp/bench_server.log 2>&1 &
  local server_pid=$!
  sleep 2

  for ((i=0; i<WARMUP; i++)); do
    curl --silent --output /dev/null "$URL" || true
  done

  : > /tmp/bench_latencies.txt
  for ((i=0; i<RUNS; i++)); do
    local meta
    meta=$(curl --silent --no-keepalive --output /dev/null --write-out '%{http_code} %{time_total}' "$URL" || true)
    local http=${meta%% *}
    local secs=${meta##* }
    if [ "$http" != "$EXPECTED_CODE" ]; then
      errors=$((errors+1))
      continue
    fi
    printf '%s\n' "$secs" >> /tmp/bench_latencies.txt
  done

  kill "$server_pid" 2>/dev/null || true
  wait "$server_pid" 2>/dev/null || true

  sort -n /tmp/bench_latencies.txt | awk -v label="$label" -v err="$errors" '
    { a[NR] = $1 }
    END {
      p50 = a[int(NR * 0.50)]
      p95 = a[int(NR * 0.95)]
      printf "%s p50=%.3f p95=%.3f errors=%d\n", label, p50, p95, err
    }'
}

measure "$BASE" "baseline"
measure "$PATCH" "ai patch"

Run it like this:

bash bench_patch.sh "$(git rev-parse HEAD~1)" "$(git rev-parse HEAD)" \
  "http://localhost:3000/api/items?limit=50"

The script checks out each commit. It starts the server, warms it up, then sends 200 requests. It records status and response time per request. Finally, it prints median, tail, and error count.

Why warm up first? Cold caches are real but rare in steady state. Warmup kills the one-time penalty that would pollute the sample.

And why two percentiles? The mean hides the shape. A patch can feel faster on average while hurting your worst users. p50 is the typical case. p95 is the angry customer. You want both.

Sample output

baseline p50=11.238 p95=42.117 errors=0
ai patch p50=8.904  p95=76.302 errors=13

This is the output that changed my mind about AI patches. Median latency improved. Tail latency got worse. The patch batched many small rows into parallel requests. Low load looked fast. Under pressure, connection-pool contention pushed the tail higher. One number would have hidden that story. Two percentiles kept it visible.

The graph you keep

I keep a scatter chart. The x-axis is the commit hash. The y-axis is p95 latency. Every run adds one point. The cloud immediately shows whether the patch drifted right.

The shape matters more than the numbers. Baseline points stacked around 42 ms. Patch points spread from 60 ms to 120 ms. That spread told me what would happen in production. The average never would.

The ledger

A text file is my performance ledger. One run, one line. It is like git log, but for measurements:

31f1d2c p50=11.2 p95=40.1 errors=0 note="baseline load=50 sorted"
91ab3e8 p50=8.9  p95=61.7 errors=13 note="ai-opt parallel batch"

Six fields cover it. Hash, p50, p95, errors, and a note. The note saves future me from guessing. The ledger is more honest than a PR review. A diff can look cleaner. Numbers tell the truth.

When the benchmark fails

When p95 regresses, I send the ledger line back to the model. The diff goes with it. That starts a new attempt loop.

MonkeyCode is the open-source project I run those retry batches through. At the time of writing, its site advertises free model access with a 10-million-token starting quota and a free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I offload the long retry batch there. My laptop stays free for profiling. The loop is simple: measure, retry, measure again. The graph, not the intention, decides the merge.

Limitations

This script sends sequential requests. It does not simulate real concurrency. It ignores cold caches. For proper load testing, run wrk, k6, or hey. Keep this script as a fast safety net.

Do not use it for documentation or dead code removal. Use it for query rewrites, cache layers, and connection-pool changes. Those are the patches that produce dramatic regressions.

Set RUNS=1000 if you need more confidence. Expect the run to take ten times longer. I favor repeatability over speed.

The next time a model says "optimized," ask for proof. Run the baseline. Run the patch. Compare the p95s. Your opinion no longer needs to be based on "looks good." The scatter chart defends you on Friday night when the pages fire.