A model can pass every functional test and still break your pipeline. The postmortem below is a composite reconstruction. It represents a common failure pattern with free-tier AI servers. Use it as a template for your own incident review.
Incident summary
At 02:14 UTC, a batch summarization job started receiving 503 errors. At 02:31, the retry queue held 12,000 messages. Impact was limited to staging. No customer data was involved. Severity: SEV-2. Total duration: 47 minutes.
The endpoint was a free-tier AI server. It had passed all functional contract tests the day before. Response shape, tone, and format were correct. The team routed traffic and went to sleep.
Timeline
- 01:27 — Batch job starts routing traffic to the new free-tier server.
- 02:14 — First 503 responses appear.
- 02:16 — The retry loop amplifies request volume by 6x.
- 02:23 — The provider's rate limit triggers.
- 02:31 — Queue backlog reaches 12,000 messages.
- 02:34 — An on-call engineer disables the routing flag.
- 02:47 — The queue drains. The service recovers.
Contributing factors
- The model passed functional contract tests. Nobody tested the server under sustained load.
- The retry policy had no backoff and no cap. Every failure created six more requests.
- The health endpoint was never probed before routing.
- The free server had no SLO. The team assumed it behaved like a paid endpoint.
- The alert threshold was set to 'queue non-empty.' It fired 17 minutes after the first error.
Root cause
The failure was a missing operational contract. The team tested what the model returned. It never tested how the server behaved under load, latency, or failure. The model was fine. The server was an unverified dependency.
The durable fix: a server admission test
Before routing traffic to any new endpoint, run an admission test. It checks five properties. Uptime. Latency. Stability. Failure mode. Recovery.
The script below is a minimal version. It runs a health check, a latency probe, an error-rate burst, and a recovery check. Point it at a health or ping endpoint.
#!/usr/bin/env python3
# Server admission test: gate a new AI endpoint before routing traffic.
import json
import sys
import time
import urllib.error
import urllib.request
ENDPOINT = sys.argv[1] if len(sys.argv) > 1 else 'http://localhost:8000/health'
TIMEOUT = 5.0
def probe(payload: dict) -> tuple[float, int]:
req = urllib.request.Request(
ENDPOINT,
data=json.dumps(payload).encode(),
headers={'Content-Type': 'application/json'},
)
start = time.perf_counter()
try:
with urllib.request.urlopen(req, timeout=TIMEOUT) as resp:
return time.perf_counter() - start, resp.status
except urllib.error.HTTPError as e:
return time.perf_counter() - start, e.code
except Exception:
return time.perf_counter() - start, 0
# 1. Health check
lat, status = probe({'text': 'ping'})
assert status == 200, f'health check failed: HTTP {status}'
# 2. Latency probe: 50 sequential requests
latencies = [probe({'text': f'probe {i}'})[0] for i in range(50)]
p95 = sorted(latencies)[int(len(latencies) * 0.95)]
print(f'p95 latency: {p95 * 1000:.0f} ms')
assert p95 < 3.0, 'p95 latency exceeds the 3s budget'
# 3. Error rate under a small burst
errors = sum(1 for _ in range(20) if probe({'text': 'burst'})[1] != 200)
print(f'error rate: {errors / 20:.0%}')
assert errors / 20 < 0.05, 'error rate exceeds 5%'
# 4. Recovery: force a timeout, then verify the next request succeeds
try:
urllib.request.urlopen(urllib.request.Request(ENDPOINT), timeout=0.1)
except Exception:
pass # expected: the endpoint should not hang forever
lat, status = probe({'text': 'after timeout'})
assert status == 200, 'server did not recover after a forced timeout'
print('admission test passed')
Run this before every routing change. Put it in CI or a pre-deploy step. It takes about 90 seconds.
The second fix: a retry budget
The admission test prevents the first mistake. A retry budget prevents the second. The old job retried forever with no backoff. The new policy allows three attempts, then dead-letters the message.
for attempt in 1 2 3; do
curl -sS -f $ENDPOINT && break
sleep $((2 ** attempt))
done
Pair it with a circuit breaker. Open the circuit after a 5% error rate over two minutes. Close it only after a clean admission test.
What the admission test would have caught
The admission test would have caught this incident. The latency probe would have flagged the degradation. The error-rate burst would have exceeded the 5% threshold. The recovery check would have failed after the forced timeout. The team would have kept the old endpoint and opened a ticket instead. That is the point of a gate. It fails before your users do.
Changes made after the incident
The team made five changes after the incident.
- Added the admission test to the deployment pipeline.
- Replaced the infinite retry loop with a three-attempt budget.
- Added a circuit breaker to the routing layer.
- Changed the alert to fire on error rate, not queue length.
- Documented the free-server decision table in the runbook.
When a free server is acceptable
Free servers are not bad. They are unguaranteed. The decision table below is the rule the team now follows. The table is conservative. It errs on the side of caution.
| Use case | Free server OK? | Why |
|---|---|---|
| Prototyping and eval | Yes | Data loss is cheap |
| Batch jobs with dead-letter | Yes | Retries absorb hiccups |
| User-facing real-time | No | No SLO or capacity guarantee |
| SLA-bound pipelines | No | Recovery is best-effort |
Where MonkeyCode fits
Disclosure: This article was prepared as part of MonkeyCode's product outreach.
The open-source MonkeyCode project offers free model access and a free server option. That combination is useful for exactly this workflow. The team re-ran the admission test on a fresh MonkeyCode free server. This happened before any traffic was routed. The free token allowance (10 million at the time of writing) covered hundreds of probe runs. Verify the current terms and quotas before relying on them.
The contract test checked the model. The admission test checks the server. MonkeyCode's free tier made both cheap to run repeatedly. If you want a cheap place to run this experiment, the free tier is a good starting point.
Limitations
This approach does not fit every team. Teams with hard SLAs should not route production traffic through free servers. The admission test is a snapshot, not a guarantee. It cannot predict behavior after hours of sustained load. Treat the results as a floor, not a ceiling. Free tiers change. Quotas, rate limits, and availability can shift without notice.
The lesson
The model was never the problem. The server was an unverified dependency. Treat every free endpoint like an external service with no SLO. Test its behavior before you trust it with traffic. The next time you wire up a free endpoint, run the admission script first. It is cheaper than a 47-minute incident.
Top comments (0)