Your Model Calls Got Slower. Run git bisect Before You Blame the Server

Say your dashboard looks like this. Tuesday: 180 ms per call. Wednesday: 1.4 s. No model upgrade. No endpoint change. Retries grew. The dashboard blamed the server. The server blamed the weekend.

Model latency regressions are rarely model regressions. They are request-shape regressions. A prompt template moved from v2 to v3. A cache key started including a timestamp. A middleware appended four lines to your system prompt. Your dashboard sees a slow server. Your git history sees a guilty commit.

This is a three-myth FAQ plus a script. The myths fix your mental model. The script automates the blame. It works with any OpenAI-compatible chat endpoint, including MonkeyCode's free model routes. When the free server gets noisy, move the probe to a quieter one before calling any commit guilty. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Three myths to drop

1. "Slow calls mean slow infrastructure."

Shared free servers have noisy neighbors. I wrote about that separately. Neighbor noise rarely creates a clean step function. A jump that lines up with a merge is your code, not the host.

Run a two-call probe. First call cold. Second call warm. If the warm call is fast, your cache key changed. Cache keys change inside commits. That is bisectable.

2. "The dashboard will name the commit."

Dashboards aggregate by time, not by commit. They show a spike at 14:02. They do not show the merge at 14:01.

You must build that mapping. Treat dashboards as alarms. Treat git history as the detective.

3. "git bisect only finds compile failures."

False. git bisect run executes any script. Your script exits 0 for good. It exits non-zero for bad. Slow responses are observable states. So are cache misses, token counts, and prompt sizes.

History becomes a binary search. The cause gets isolated in O(log n) checkouts.

The reproducibility script

Save this as ~/bin/latency_bisect.sh. Keep it outside the repository you are probing, or bisect will migrate your test tool into the feedback loop.

#!/usr/bin/env bash
# usage:
#   ENDPOINT=... MODEL=... TOKEN=... THRESHOLD_MS=1200 bash ~/bin/latency_bisect.sh
set -euo pipefail

: "${ENDPOINT:?set ENDPOINT}"
: "${MODEL:?set MODEL}"
: "${TOKEN:?set TOKEN}"
THRESHOLD_MS="${THRESHOLD_MS:-1200}"

now_ms() { perl -MTime::HiRes=time -e 'printf "%d\n", time * 1000'; }
body_file="${TMPDIR:-/tmp}/lb_body_$$.json"

# Optional: use the repo's committed request generator.
if [ -f ./build_request.sh ]; then
  bash ./build_request.sh "$MODEL" > "$body_file"
else
  printf '{"model":"%s","messages":[{"role":"user","content":"say ok"}]}' "$MODEL" > "$body_file"
fi

start=$(now_ms)
if ! curl_out=$(curl -sS -o "$body_file.resp" \
  --max-time 30 \
  -H "Authorization: Bearer ${TOKEN}" \
  -H "Content-Type: application/json" \
  --data @"$body_file" \
  -w "%{http_code} %{time_starttransfer} %{time_total}" \
  "$ENDPOINT"); then
  echo "curl failed on this commit" >&2
  exit 125 # tell git bisect to skip this commit
fi
end=$(now_ms)
read -r http_code ttfb_s total_s <<< "$curl_out"

ttfb_ms=$(awk -v x="$ttfb_s" 'BEGIN{printf "%d", x * 1000}')
total_ms=$(( end - start ))

echo "http=$http_code ttfb_ms=$ttfb_ms total_ms=$total_ms"

# Exit 0 means "good enough"; non-zero means "this commit is bad".
[ "$http_code" = "200" ] && [ "$total_ms" -lt "$THRESHOLD_MS" ]

Why Perl? macOS ships it, while date +%s%N needs GNU coreutils. Why awk? curl prints decimal seconds, and shells cannot do float math. The exit 125 line tells git bisect run to skip a commit when the network fails. Accidentally slow is not the same as untestable.

Assumptions: model names are simple identifiers. For JSON-heavy names, replace the printf with jq -n --arg model "$MODEL" .... I keep the fallback short on purpose.

The three-step bisect

1. Start from a clean worktree. git bisect run does not overwrite local edits.
2. Mark your brackets. Good is a known fast commit. Bad is current main.

git status --porcelain

git bisect start
git bisect bad main
git bisect good HEAD~50
git bisect run bash ~/bin/latency_bisect.sh

1. Read the printed commit id. Save the evidence.

git bisect log > bisect-evidence.txt
git bisect reset

Reading the two numbers

Symptom	Likely cause	Look at
ttfb_ms high	Queueing or server-side load	Quota, neighbor traffic, retry storms
total_ms high, ttfb_ms fine	Longer generation	max_tokens, prompt length, cache misses
First call slow, second fast	Prefix-cache invalidation	Recent template or cache-key commits
Both high everywhere	Network path	DNS, proxy, rotated token

Do not trust one sample. This is a blame divider, not a benchmark. When the free server looks nervous, run the script twice per commit and keep the lower value.

What this script will not catch

Prompt-shape-only changes. The fixed fallback always asks "say ok". If the slow case needs a 4k-token contract, the sample stays fast on every commit. Fix: commit a build_request.sh inside the repo under test. The test script runs it when present, so the old generator travels with old commits.

Neighbor effects. They are time-based, not commit-based. A busy neighbor makes the whole run noisy. Move to a quieter window or a quieter server first.

Slow drift. Bisect needs a step change. If latency creeps upward for months, there is no clean bad commit. Fix the baseline before you bisect.

Throughput and quota curves. This sends one request per commit. It does not stress test. It does not measure concurrency. It finds the first commit that made a single call slower.

Who should not use this

• Teams with no known-good commit. Without a bracket, bisect has no answer.
• Gateway-only users. If you cannot modify the request path, you will bisect someone else's code.
• Anyone storing tokens in the repository. Endpoint secrets belong in env files, not in the history being searched.

The correction

Slow calls are symptoms. Commits are causes. Dashboards summarize. Git bisect decides.

Next time the p50 jumps, run the script first. Let the binary search find the guilty merge. Then blame the server with evidence it cannot argue with.