DEV Community

jaryn
jaryn

Posted on

Failure-Drill Your AI Platform Trial: Three Injections, One Scorecard

At 10:42 the agent stopped mid-refactor. Not with an error — with a hang. The status bar spun for four minutes, then the session died. The team's first instinct was to blame the model. The real cause was upstream: the free endpoint had started rate-limiting, and nobody had a way to see it.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

That scenario is why I stopped writing decision matrices for free AI platform tiers. A comparison table tells you what the vendor promises. A failure drill tells you what your team actually does when the promise breaks. And with a free tier — MonkeyCode's free server and free model allowance (10M tokens in the current published offer) included — the promise is the most fragile part of the stack.

Why a drill beats a spec sheet

A free tier is an availability experiment, not a cost decision. The quota resets on someone else's schedule. The shared server's latency depends on how many other teams are hammering it. The terms can change without your approval.

Spec-sheet comparisons miss all of that. They compare token counts and model names, then declare a winner. What actually decides whether a free tier works for your team is one question: when the endpoint degrades, how long does it take you to notice, and how fast can you recover?

That question is measurable. Here's the drill.

The three injections

The drill simulates the three failure modes that kill free-tier trials:

  1. Latency — the endpoint responds, but slowly. The agent doesn't error; it just burns time.
  2. Block — the endpoint is unreachable. The agent fails hard.
  3. Quota — the endpoint returns 429. The agent may retry, loop, or silently degrade.

Each injection runs for 15 minutes against a local proxy. The agent points at the proxy; the proxy forwards to the real endpoint with the failure applied.

The drill proxy

Save this as drill_proxy.py. It's a minimal HTTP reverse proxy with three failure modes. Treat it as a test fixture, not a production component.

#!/usr/bin/env python3
"""drill_proxy.py — failure-injection proxy for AI platform endpoints.
Test fixture only. Not for production use."""
import argparse
import http.server
import time
import urllib.request

class DrillProxy(http.server.BaseHTTPRequestHandler):
    target = None
    mode = None
    counter = 0
    quota = 0

    def _forward(self):
        body = self.rfile.read(int(self.headers.get('Content-Length', 0)))
        req = urllib.request.Request(
            DrillProxy.target + self.path,
            data=body,
            headers=dict(self.headers),
            method=self.command,
        )
        with urllib.request.urlopen(req, timeout=30) as resp:
            self.send_response(resp.status)
            for k, v in resp.headers.items():
                self.send_header(k, v)
            self.end_headers()
            self.wfile.write(resp.read())

    def do_POST(self):
        DrillProxy.counter += 1
        if DrillProxy.mode == "quota" and DrillProxy.counter > DrillProxy.quota:
            self.send_response(429)
            self.end_headers()
            self.wfile.write(b'{"error":"quota_exhausted"}')
            return
        if DrillProxy.mode == "latency":
            time.sleep(5)
        if DrillProxy.mode == "block":
            self.send_response(503)
            self.end_headers()
            self.wfile.write(b'{"error":"service_unavailable"}')
            return
        try:
            self._forward()
        except Exception as exc:
            self.send_response(502)
            self.end_headers()
            self.wfile.write(str(exc).encode())

    def log_message(self, fmt, *args):
        print(f"[drill:{DrillProxy.mode}] {self.command} {self.path} -> {fmt % args}")

if __name__ == "__main__":
    parser = argparse.ArgumentParser(description="Failure-injection proxy for AI endpoints")
    parser.add_argument("--target", required=True, help="Real endpoint, e.g. https://api.example.com/v1")
    parser.add_argument("--mode", choices=["latency", "block", "quota"], required=True)
    parser.add_argument("--quota", type=int, default=5, help="Requests allowed before 429 in quota mode")
    parser.add_argument("--port", type=int, default=8080)
    args = parser.parse_args()
    DrillProxy.target = args.target.rstrip("/")
    DrillProxy.mode = args.mode
    DrillProxy.quota = args.quota
    server = http.server.HTTPServer(("127.0.0.1", args.port), DrillProxy)
    print(f"Drill proxy on :{args.port} -> {args.target} mode={args.mode}")
    server.serve_forever()
Enter fullscreen mode Exit fullscreen mode

Run it:

python3 drill_proxy.py --target https://YOUR_ENDPOINT/v1 --mode latency --port 8080
Enter fullscreen mode Exit fullscreen mode

Then point your agent's base URL at http://127.0.0.1:8080/v1. The agent never knows the difference — until the drill starts.

Running the drill against MonkeyCode's free tier

The setup is identical for any managed AI platform, MonkeyCode's free server included. Point the proxy at the platform's endpoint, configure your agent to use the proxy, and run a realistic task — a code review, a refactor, a test-writing pass.

For each injection, record three numbers:

  • Time to detect (T2D): minutes from injection start until someone notices the agent is degraded.
  • Time to recover (T2R): minutes from detection until the team switches to a fallback or the endpoint recovers.
  • Failure signature: did the agent error, hang, retry, or silently produce garbage?

Run all three injections in one sitting. The whole drill fits in 45 minutes plus a 15-minute debrief.

What the results mean

Here's the scorecard I use:

Metric Good Warning Dangerous
T2D < 5 min 5–15 min > 15 min
T2R < 10 min 10–30 min > 30 min
Failure signature Clear error Retry loop Silent garbage

The scorecard is the comparison. A free tier with a 30-minute T2R isn't cheaper than a self-hosted setup that costs you an hour of ops per week — it's more expensive, in the one currency that matters: engineering time during an incident.

Free managed vs self-hosted, measured in recovery time

Criterion Free managed Self-hosted
Latency failure Shared; you don't control it You own the endpoint; failures are your code, not a neighbor's
Quota failure Fixed allowance; 429s when exhausted Your budget; you decide when to stop
Block failure Vendor-side; no visibility Your infra; you can see the network path
T2D Depends on your monitoring Same — but you can instrument the whole path
T2R Depends on a fallback you must pre-build Depends on your runbook, not a vendor's status page
Ops burden Near zero Real, and ongoing

Paid managed sits between the two: it keeps the zero-ops benefit but replaces the free allowance with a contract. The drill applies identically — point the proxy at the paid endpoint and measure.

The pattern is uncomfortable: the free tier's advantage is ops burden, and its weakness is recovery. If your team can't recover fast, the free tier isn't free — it's a liability with a token allowance.

Who should not use this approach

  • Teams with no fallback endpoint. The drill will show you can't recover, and you already knew that.
  • Teams that can't run a local proxy — locked-down laptops, air-gapped environments.
  • Anyone expecting the drill to certify the vendor. It tests your process, not MonkeyCode's or anyone else's SLA.

Limitations

This is a test fixture, not a security control. The proxy forwards credentials in plain HTTP on localhost — don't expose it beyond 127.0.0.1. The 5-second latency and 5-request quota are arbitrary; tune them to your workload. And one drill run proves nothing about long-term reliability — run it weekly if the free tier is load-bearing.

The token allowance I referenced is the project's current published offer. Verify it from the repository before you rely on it; free terms change, and that's exactly the failure mode this drill is designed to rehearse.

The boundary question

You now have three numbers — T2D, T2R, and a failure signature. Which one belongs in CI? My vote is the failure signature: a regression test that fails when the agent produces output without a successful upstream response. The other two are process metrics, and no CI check can fix a team that doesn't look at its dashboards.

Run the drill once. Publish your scorecard. That's more useful to the community than another benchmark table.

Top comments (0)