The failure that doesn't show up until week three
Three hundred forty qualified leads went into a CRM that never recorded them. The agent kept reporting success on every call. A bearer token behind the proxy had expired six days earlier, and the proxy was swallowing the real error and returning a generic 502 instead.
That is the pattern with MCP agents in production. They do not crash. They report success while quietly dropping data, and the damage shows up in a pipeline report weeks later, not in a stack trace today.
Your proxy is lying to your agent
Most MCP connection failures are not protocol bugs. They are infrastructure defaults that worked in local dev and broke the moment a proxy or load balancer got involved.
The common failure: a proxy or API gateway normalizes every upstream error into a generic status code, usually a 502 or 500. Your agent's retry logic is written to catch a 401 and trigger a token refresh. It never sees the 401. It sees a 502, assumes the service is down, and either retries blindly or gives up.
Fix the proxy config first, before touching agent code. Pass through the real upstream status code and headers instead of normalizing them:
location /mcp/ {
proxy_pass http://mcp_backend;
proxy_intercept_errors off;
proxy_pass_header WWW-Authenticate;
}
proxy_intercept_errors off stops your gateway from rewriting upstream error pages. Once the real 401 reaches the client, your existing refresh logic will actually fire.
Handle 429s like a server talking to you, not like a crash
A 429 is the server asking you to slow down, not telling you something is broken. Treat it as a crash and your agent either retries so aggressively it gets blocked, or gives up on work it should have finished.
Use exponential backoff with jitter, not a fixed interval. A fixed interval means every client throttled at the same moment retries at the same moment again, and you get a second wave of 429s right on schedule.
import random
import time
def call_with_backoff(fn, max_retries=5):
for attempt in range(max_retries):
response = fn()
if response.status_code != 429:
return response
base = min(2 ** attempt, 30)
time.sleep(base + random.uniform(0, base * 0.3))
raise RuntimeError("Max retries exceeded")
Cap concurrency below the documented rate limit, not at it. A single agent firing twelve parallel tool calls against one MCP server hits the ceiling faster than the same twelve calls routed through a queue with a concurrency cap of three or four. Other traffic on the same key eats into your headroom even when your own code behaves.
One agent, one job
A single agent that qualifies leads, drafts outreach, updates the CRM, and books meetings will fail at the handoffs between those tasks. Split it into narrow agents with a defined contract between them. A lead-qualification agent that outputs a strict JSON schema is trivial to debug. A monolithic agent touching five systems turns every bug into an afternoon of log archaeology, because you cannot isolate which responsibility broke without tracing the whole chain.
Put a human between the agent and the send button
Automate drafting. Gate the actual send, whether that's an email, a CRM update, or a calendar invite, behind human approval for at least the first few hundred executions. Track false positive and false negative rates on what the agent flags, and pull the checkpoint back only as that data earns it. Removing the gate on day one because the demo looked good is how a bad prompt burns through a real list before anyone notices.
Watch tokens per qualified lead, not total spend
Total token spend tells you nothing actionable. Token spend per qualified lead does. A spike usually means one of two things: a prompt is retrying on malformed output, or your context window is accumulating irrelevant history across turns instead of getting trimmed. Check the per-call token count before you check the model. Latency spikes above your SLA on one specific tool call almost always trace back to an external dependency, not the model itself.
Skip the synthetic testing shortcut on edge cases
Synthetic user testing is a fast, cheap pre-filter. It is not a replacement for a support ticket queue. Models tuned to be helpful default to agreeable scoring, so five synthetic personas testing five different flows will often cluster near the top of the scale regardless of which flow is actually worse. Use it to catch obvious breaks before a real user does, not as final validation.
Top comments (1)
Two things in the code blocks that I'd check before anyone copies them.
proxy_intercept_errorsalready defaults tooffin nginx, so adding it to thelocationchanges nothing unless it was set toonin an outer block. The usual cause of a 401 turning into a 502 is different: the upstream timing out or resetting the connection (nginx then makes the 502 itself), or anerror_pagedirective mapping codes to a generic page.nginx -T | grep -n 'intercept_errors\|error_page'and the error log's "upstream" line will tell you which one you have. Also,proxy_pass_header WWW-Authenticateonly matters if the header was being stripped, which it isn't by default.In
call_with_backoff, the loop sleeps after the last failed attempt and then raises, so withmax_retries=5the caller waits an extra ~16 to 21 seconds (2**4 = 16 plus up to 30% jitter) for nothing. It also ignoresRetry-After, which most 429 responses include; when the header is present it should win over the computed delay. A small fix:if attempt == max_retries - 1: breakbefore the sleep, anddelay = float(response.headers.get('Retry-After', base)).The lead story is the real lesson: the agent counted "no exception" as success. Do your tool wrappers check that the CRM write returned a record id, rather than that the HTTP call returned?