At 10:42 the agent stopped mid-refactor. Not with an error — with a hang. The status bar spun for four minutes, then the session died. The team's first instinct was to blame the model. The real cause was upstream: the free endpoint had started rate-limiting, and nobody had a way to see it.
Disclosure: This article was prepared as part of MonkeyCode's product outreach.
That scenario is why I stopped writing decision matrices for free AI platform tiers. A comparison table tells you what the vendor promises. A failure drill tells you what your team actually does when the promise breaks. And with a free tier — MonkeyCode's free server and free model allowance (10M tokens in the current published offer) included — the promise is the most fragile part of the stack.
Why a drill beats a spec sheet
A free tier is an availability experiment, not a cost decision. The quota resets on someone else's schedule. The shared server's latency depends on how many other teams are hammering it. The terms can change without your approval.
Spec-sheet comparisons miss all of that. They compare token counts and model names, then declare a winner. What actually decides whether a free tier works for your team is one question: when the endpoint degrades, how long does it take you to notice, and how fast can you recover?
That question is measurable. Here's the drill.
The three injections
The drill simulates the three failure modes that kill free-tier trials:
- Latency — the endpoint responds, but slowly. The agent doesn't error; it just burns time.
- Block — the endpoint is unreachable. The agent fails hard.
- Quota — the endpoint returns 429. The agent may retry, loop, or silently degrade.
Each injection runs for 15 minutes against a local proxy. The agent points at the proxy; the proxy forwards to the real endpoint with the failure applied.
The drill proxy
Save this as drill_proxy.py. It's a minimal HTTP reverse proxy with three failure modes. Treat it as a test fixture, not a production component.
#!/usr/bin/env python3
"""drill_proxy.py — failure-injection proxy for AI platform endpoints.
Test fixture only. Not for production use."""
import argparse
import http.server
import time
import urllib.request
class DrillProxy(http.server.BaseHTTPRequestHandler):
target = None
mode = None
counter = 0
quota = 0
def _forward(self):
body = self.rfile.read(int(self.headers.get('Content-Length', 0)))
req = urllib.request.Request(
DrillProxy.target + self.path,
data=body,
headers=dict(self.headers),
method=self.command,
)
with urllib.request.urlopen(req, timeout=30) as resp:
self.send_response(resp.status)
for k, v in resp.headers.items():
self.send_header(k, v)
self.end_headers()
self.wfile.write(resp.read())
def do_POST(self):
DrillProxy.counter += 1
if DrillProxy.mode == "quota" and DrillProxy.counter > DrillProxy.quota:
self.send_response(429)
self.end_headers()
self.wfile.write(b'{"error":"quota_exhausted"}')
return
if DrillProxy.mode == "latency":
time.sleep(5)
if DrillProxy.mode == "block":
self.send_response(503)
self.end_headers()
self.wfile.write(b'{"error":"service_unavailable"}')
return
try:
self._forward()
except Exception as exc:
self.send_response(502)
self.end_headers()
self.wfile.write(str(exc).encode())
def log_message(self, fmt, *args):
print(f"[drill:{DrillProxy.mode}] {self.command} {self.path} -> {fmt % args}")
if __name__ == "__main__":
parser = argparse.ArgumentParser(description="Failure-injection proxy for AI endpoints")
parser.add_argument("--target", required=True, help="Real endpoint, e.g. https://api.example.com/v1")
parser.add_argument("--mode", choices=["latency", "block", "quota"], required=True)
parser.add_argument("--quota", type=int, default=5, help="Requests allowed before 429 in quota mode")
parser.add_argument("--port", type=int, default=8080)
args = parser.parse_args()
DrillProxy.target = args.target.rstrip("/")
DrillProxy.mode = args.mode
DrillProxy.quota = args.quota
server = http.server.HTTPServer(("127.0.0.1", args.port), DrillProxy)
print(f"Drill proxy on :{args.port} -> {args.target} mode={args.mode}")
server.serve_forever()
Run it:
python3 drill_proxy.py --target https://YOUR_ENDPOINT/v1 --mode latency --port 8080
Then point your agent's base URL at http://127.0.0.1:8080/v1. The agent never knows the difference — until the drill starts.
Running the drill against MonkeyCode's free tier
The setup is identical for any managed AI platform, MonkeyCode's free server included. Point the proxy at the platform's endpoint, configure your agent to use the proxy, and run a realistic task — a code review, a refactor, a test-writing pass.
For each injection, record three numbers:
- Time to detect (T2D): minutes from injection start until someone notices the agent is degraded.
- Time to recover (T2R): minutes from detection until the team switches to a fallback or the endpoint recovers.
- Failure signature: did the agent error, hang, retry, or silently produce garbage?
Run all three injections in one sitting. The whole drill fits in 45 minutes plus a 15-minute debrief.
What the results mean
Here's the scorecard I use:
| Metric | Good | Warning | Dangerous |
|---|---|---|---|
| T2D | < 5 min | 5–15 min | > 15 min |
| T2R | < 10 min | 10–30 min | > 30 min |
| Failure signature | Clear error | Retry loop | Silent garbage |
The scorecard is the comparison. A free tier with a 30-minute T2R isn't cheaper than a self-hosted setup that costs you an hour of ops per week — it's more expensive, in the one currency that matters: engineering time during an incident.
Free managed vs self-hosted, measured in recovery time
| Criterion | Free managed | Self-hosted |
|---|---|---|
| Latency failure | Shared; you don't control it | You own the endpoint; failures are your code, not a neighbor's |
| Quota failure | Fixed allowance; 429s when exhausted | Your budget; you decide when to stop |
| Block failure | Vendor-side; no visibility | Your infra; you can see the network path |
| T2D | Depends on your monitoring | Same — but you can instrument the whole path |
| T2R | Depends on a fallback you must pre-build | Depends on your runbook, not a vendor's status page |
| Ops burden | Near zero | Real, and ongoing |
Paid managed sits between the two: it keeps the zero-ops benefit but replaces the free allowance with a contract. The drill applies identically — point the proxy at the paid endpoint and measure.
The pattern is uncomfortable: the free tier's advantage is ops burden, and its weakness is recovery. If your team can't recover fast, the free tier isn't free — it's a liability with a token allowance.
Who should not use this approach
- Teams with no fallback endpoint. The drill will show you can't recover, and you already knew that.
- Teams that can't run a local proxy — locked-down laptops, air-gapped environments.
- Anyone expecting the drill to certify the vendor. It tests your process, not MonkeyCode's or anyone else's SLA.
Limitations
This is a test fixture, not a security control. The proxy forwards credentials in plain HTTP on localhost — don't expose it beyond 127.0.0.1. The 5-second latency and 5-request quota are arbitrary; tune them to your workload. And one drill run proves nothing about long-term reliability — run it weekly if the free tier is load-bearing.
The token allowance I referenced is the project's current published offer. Verify it from the repository before you rely on it; free terms change, and that's exactly the failure mode this drill is designed to rehearse.
The boundary question
You now have three numbers — T2D, T2R, and a failure signature. Which one belongs in CI? My vote is the failure signature: a regression test that fails when the agent produces output without a successful upstream response. The other two are process metrics, and no CI check can fix a team that doesn't look at its dashboards.
Run the drill once. Publish your scorecard. That's more useful to the community than another benchmark table.
Top comments (0)