A nightly summarizer runs on a free model lane. One night the lane returns 403 mid-job. The job catches the error, retries twice, then writes a truncated report. Nobody notices for four days.
That scenario is a composite, built from the four failure shapes below. It is not a customer story. The point is narrower: most teams test whether a free lane works, not how it dies.
This guide covers the second question. It gives a rehearsal you can run in one sitting, a scoring rubric, a ship/degrade/refuse table, and the cases where the whole exercise is a waste of time.
What "free" actually changes
Free access does not change the happy path. A call returns, tokens arrive, the parser is happy. That is why demos pass.
Free access changes the failure model:
- No contractual ceiling. Capacity is shared, and your share is not stated.
- No deprecation window you control. Access can narrow without a ticket in your tracker.
- Error semantics drift. Status codes and retry hints are not a stable contract.
- Partial output is common. Streaming and truncation turn a hard failure into a plausible-looking artifact.
The last bullet is the expensive one. A loud 500 costs you a retry. A quiet truncation costs you trust in a dataset.
The four shapes worth injecting
Do not test "is it down". Test the four ways it degrades.
1. Hard refusal
The endpoint rejects the request as unauthorized, forbidden, or unknown. Your code should surface a typed error and stop, not loop.
2. Soft throttle
The endpoint returns a rate-limit response with a retry hint. The job should requeue or sleep, respect the hint, and never spin.
3. Slow drain
Responses arrive late enough to blow your job deadline. The failure is time, not status code.
4. Mid-stream truncation
Bytes stop early. The JSON is cut in half. Your parser is the only thing standing between you and a corrupt output file.
Each shape maps to a different code path. A single "provider is down" test covers none of them.
A rehearsal harness you can run in a sitting
The cleanest way to inject these is a local proxy. Point your client's base URL at the proxy. Flip the fault mode. Keep the workload unchanged.
Here is a minimal injector. It targets an OpenAI-compatible HTTP surface. It is written for Python 3.11 and was not executed against any specific provider, so treat it as a template.
# rehearse.py — fault-injecting proxy. Python 3.11+.
# Template only: not executed against any production endpoint.
import http.server, json, os, socketserver, time, urllib.request
UPSTREAM = os.environ["UPSTREAM_BASE"].rstrip("/")
MODE = os.environ.get("FAULT_MODE", "none")
PORT = int(os.environ.get("PORT", "8787"))
class Handler(http.server.BaseHTTPRequestHandler):
protocol_version = "HTTP/1.1"
def do_POST(self):
body = self.rfile.read(int(self.headers.get("Content-Length", 0)))
if MODE == "refuse":
return self._send(403, b'{"error":{"message":"injected refusal"}}')
if MODE == "throttle":
self.send_response(429)
self.send_header("Retry-After", "120")
self.send_header("Content-Length", "0")
self.end_headers()
return
if MODE == "drain":
time.sleep(float(os.environ.get("DRAIN_SECONDS", "45")))
req = urllib.request.Request(
UPSTREAM + self.path,
data=body,
headers={"Content-Type": "application/json",
"Authorization": self.headers.get("Authorization", "")},
)
with urllib.request.urlopen(req, timeout=60) as resp:
payload = resp.read()
if MODE == "truncate":
payload = payload[: len(payload) // 2]
self._send(200, payload)
def _send(self, code, payload):
self.send_response(code)
self.send_header("Content-Type", "application/json")
self.send_header("Content-Length", str(len(payload)))
self.end_headers()
self.wfile.write(payload)
def log_message(self, *args):
pass
socketserver.ThreadingTCPServer.allow_reuse_address = True
with socketserver.ThreadingTCPServer(("127.0.0.1", PORT), Handler) as srv:
srv.serve_forever()
The driver is short. Run one proxy per fault mode. Call the real workload through it. Assert on the artifact, not on the log line.
# drill.py — runs each fault mode, then checks what the workload produced.
import os, subprocess, sys, time
FAULTS = ["refuse", "throttle", "drain", "truncate"]
def with_fault(mode, workload, port=8787):
env = {**os.environ, "FAULT_MODE": mode, "PORT": str(port)}
proc = subprocess.Popen([sys.executable, "rehearse.py"], env=env)
time.sleep(0.5)
try:
return workload(f"http://127.0.0.1:{port}/v1")
finally:
proc.terminate()
proc.wait(timeout=5)
def workload(base_url):
# Swap in the real call. Must return (exit_code, artifact_exists).
...
for mode in FAULTS:
code, artifact = with_fault(mode, workload)
print(f"{mode:9s} exit={code} artifact={artifact}")
Run it three times per mode. One clean pass is usually luck.
Score the rehearsal instead of arguing about it
Use a rubric. Every row is a yes or no that a second engineer can check.
| Fault | Pass | Fail |
|---|---|---|
| Refuse | Typed error, non-zero exit, no artifact | Retry loop, or a zero-byte output file |
| Throttle | Honors the retry hint, requeues, one bounded attempt set | Immediate re-send, or a busy loop |
| Drain | Timeout fires before the job deadline, run marked incomplete | Job hangs until the scheduler kills it |
| Truncate | Parser rejects, run marked incomplete | Half-parsed output written and committed |
One fail is not a blocker for all workloads. It is a blocker for the specific workload you tested.
Ship, degrade, or refuse
Read the rubric against your own use case. This table is the decision, not a suggestion.
| Rehearsal result | Internal, reversible | User-facing or billed |
|---|---|---|
| All four pass | Ship on the free lane | Ship, with an exit rehearsal on the schedule |
| Truncate fails only | Fix the parser first | Refuse until fixed |
| Drain fails | Move to a queue with checkpoints | Refuse |
| Refuse or throttle fails | Refuse | Refuse |
Two rules make this hold. First, incomplete runs must be visible in the same place as failed runs. Second, no downstream job may read an output marked incomplete.
Exit criteria worth putting in a ticket
Vague intentions never trigger. Numbers in a tracker do.
- Any single rehearsal mode regresses twice in a week.
- The lane's error surface changes without a note from the provider.
- Your p95 job runtime crosses the budget you set for the workload.
- The workload's output starts feeding a user-facing or financial path.
- You cannot answer "who gets paged" within ten seconds.
Item four is the common one. Workloads migrate from internal to customer-facing without anyone reopening this decision.
Where the rehearsal gets cheap
MonkeyCode provides free model access and a free server option, per the project's own current materials. That combination makes a rehearsal cheaper than a review meeting: you can stand up the proxy and the workload on the same free server and inject faults for an afternoon. Re-check the live limits on the project page before you plan around them, because allowances change.
Disclosure: This article was prepared as part of MonkeyCode's product outreach.
The useful part is not the free capacity. It is that a disposable host is the right place to break things. Do not rehearse against the lane your production traffic already depends on.
When this approach is the wrong one
Be honest about the boundaries.
- Data cannot leave your boundary. No rehearsal helps. The free lane is out, and so is the proxy hop.
- You need a signed contract, DPA, or uptime commitment. Free access does not provide one. Buy the guarantee instead.
- The workload sits on a signed webhook path. Timing and body integrity are part of the contract. Rehearse a different workload first.
- You cannot override the base URL. The proxy has no entry point. Use a feature flag around the call instead.
- The workload is a one-off script. Rehearsal costs more than the failure. Run it, read the output, move on.
- You already know the answer. If a Rehearsal result would not change your decision, skip the drill and write the decision down.
The last point is the anti-use rule for the guide itself. Tests that cannot flip a decision are theater.
The takeaway
A free lane is a dependency with an unknown expiration date. Treat it like one. Inject the four failure shapes, score the results, and let the table decide.
If you run the drill this week, start with truncation. It is the failure that looks like success. The MonkeyCode project is one place to run it without spending anything while you learn what your code actually does.
Top comments (0)