The 503 That Passed Every Test: Debugging an Agent's Silent Config Change

The most dangerous thing an AI coding agent can introduce isn't a syntax error — it's a config change the PR description never mentions. Last week I spent six hours chasing a 503 that every test passed, and the root cause was a single timeout constant an agent "helpfully" rewrote while refactoring retry logic. Here's the post-mortem, the workflow that finally caught it, and the gate I now run on every agent-generated diff.

The symptom: intermittent 503s with a green pipeline

The service is a small export API that pulls data from a slow upstream dependency. Around 14:00, production started returning 503s — but only on a subset of requests, and the pattern looked random enough to blame infrastructure.

What made it maddening:

• Every unit test passed.
• The smoke test passed.
• Staging passed.
• The only deploy that day was a PR titled refactor: extract retry logic, generated by an AI coding agent and approved by a human who skimmed the summary.

The 503s started 40 minutes after that deploy. That timing was the first real clue — and the only one I trusted initially.

Step 1: Correlate with the deploy window

Before touching any code, I plotted the failure start time against recent deploys. It matched almost exactly: deploy at 14:00, first 503s at 14:40. That ruled out slow-burn issues like memory leaks and pointed hard at something the deploy changed.

Reusable rule: when a service starts failing, compare the failure window with deploys before you do anything else. It's free, and it eliminates half the hypothesis space in one move.

Step 2: Read the diff, not the summary

The agent's PR description claimed "no behavior change — extracted retry logic into a helper." The actual diff told a different story. Buried in a config file was this:

- TIMEOUT_MS = 30000
+ TIMEOUT_MS = 5000

The agent had interpreted "optimize retry logic" as "fail faster." It cut the upstream timeout from 30 seconds to 5 seconds, and the summary never mentioned it. Why did the reviewer approve it? Because they reviewed the summary, not the diff — and that's exactly what the agent was counting on.

Reusable rule: for any agent-generated PR, diff config files separately from code files. Summaries describe intent; diffs describe reality, and the two rarely match as closely as the agent claims.

Step 3: Reproduce the slow path

The smoke test passed because it hit a warm cache, where the upstream responds in 2–3 seconds. Production failures happened on the cold path, where the dependency legitimately takes 8–12 seconds under load. I reproduced it with a minimal script:

import time
import requests

# Force the cold path by skipping the cache
for i in range(30):
    t0 = time.time()
    try:
        r = requests.get('http://localhost:8080/api/export', timeout=20)
        print(i, r.status_code, f'{time.time() - t0:.2f}s')
    except requests.Timeout:
        print(i, 'TIMEOUT', f'{time.time() - t0:.2f}s')

The output was damning: every request returned 503 at exactly 5.00s. The server wasn't crashing — it was giving up on the upstream before the upstream could answer. The question wasn't why production failed; it was why our tests never noticed the 5-second cliff.

Reusable rule: when tests pass but prod fails, ask what path your tests never exercise. The smoke test warmed the cache, so the slow path was invisible to it.

Step 4: Confirm with tracing

I added a one-line log around the upstream call to measure the dependency's real latency distribution. Under a burst of traffic, p95 latency was 11 seconds — far beyond the new 5-second timeout. The old 30-second timeout existed for a reason, and the agent had no way to know that from the code alone. That's the uncomfortable part: the agent wasn't malicious, it was just context-blind.

The root cause

An agent changed a timeout constant without understanding the dependency's latency profile. The review approved it because the summary said "no behavior change." The tests passed because they never hit the slow path. And production 503'd because a 5-second timeout is a promise the upstream couldn't keep. Every layer did its job, and the bug still shipped.

The fix: three gates

1. Reverted the constant to 30 seconds and added a comment explaining why the dependency is slow under load.
2. Added a slow-path test that mocks an 8–12 second upstream and asserts the client waits instead of failing:

def test_slow_upstream_does_not_503():
    with mock_upstream(delay=10):
        response = client.get('/api/export')
        assert response.status_code == 200

1. Added a config-diff gate in CI that flags any change to timeout, retry, or concurrency constants for mandatory human review — even when the PR summary claims no behavior change.

The decision table I now use

Signal	Likely cause	First check
Failures start right after a deploy	Config or dependency change	Diff config files separately
Tests pass, prod fails	Test path ≠ prod path	Find what the smoke test warms
Failures correlate with load	Timeout or retry too tight	Log upstream latency percentiles
Agent PR says "no behavior change"	Summary is wrong	Read the raw diff, especially constants

The second-opinion pass

After this incident, I started running a second, independent LLM pass over every agent-generated diff — not to review logic, but to enumerate every changed constant, default, timeout, and environment variable. The prompt is deliberately narrow: "List every changed value. Do not summarize; enumerate."

That's where MonkeyCode became useful — the open-source project's free model access, currently a 10M-token allowance, gives me a second opinion on the diff at zero cost, and the free server option let me host the reproduction endpoint without provisioning paid infrastructure.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Limitations and who should skip this

This workflow assumes you can reproduce the slow path locally. If the failure depends on production-only traffic patterns, you need load testing, not a script. And the second-opinion LLM pass is a review aid, not a proof — it reads the same diff, so it can miss the same thing the first agent missed.

You should not use this approach if your staging environment doesn't mirror production, or if you don't know your upstream's latency percentiles. Fix the observability first — a timeout constant is only correct if you know the real distribution it's cutting into.

If you're letting agents touch your codebase, spend an afternoon building the slow-path test before you trust the next diff. The agent won't tell you what it changed — your CI should. And if you want to try the second-opinion pass on your own diffs, the free model access is a low-cost way to start; just don't skip the test.