The Fault Injection Workshop: Teaching a Model Call to Fail on Purpose

The Friday demo

Picture a review session in a room with twelve laptops. One student's assistant has answered correctly all week. On demo day the endpoint returns a body that stops mid-sentence, response.json() raises, and the try/except that was wrapped around the network call catches nothing at all. The demo ends early, and the lesson arrives a week late and in public.

That failure is cheap to manufacture. It is a truncation, it takes one line of code, and it needs no flaky provider — only a proxy the students control.

The outline below is a teaching plan, not a report from a delivered class. All code here was written for this article and should be run once by the instructor before anyone teaches from it.

The skill being taught is naming, not handling

"Handle errors" is not a lesson; it is a shrug. Naming them is a lesson, because the name decides the recovery. A rate limit wants a wait. A truncated body wants a re-ask. A 500 wants one bounded retry. A stalled stream wants a shorter cap and a log line. A student who cannot tell a 429 from a truncated 200 will apply the wrong recovery with perfect confidence.

So the exercise gives students a proxy that breaks responses on demand, and asks them to classify what came back before they fix anything. Validation of the schema is a different exercise; here the only deliverable is a label plus an elapsed time.

The artifact: a proxy that corrupts responses on request

Sixty lines, standard library only, no dependencies to install on lab machines. It forwards JSON POSTs upstream and then damages the reply according to an X-Fault header.

#!/usr/bin/env python3
# faultproxy.py — forward model calls, then break them on demand
import http.server
import os
import sys
import time
import urllib.error
import urllib.request

UPSTREAM = os.environ.get("UPSTREAM", "http://127.0.0.1:9000").rstrip("/")
DEFAULT_FAULT = os.environ.get("FAULT", "none")
PORT = int(os.environ.get("PORT", "8787"))


class Handler(http.server.BaseHTTPRequestHandler):
    protocol_version = "HTTP/1.1"

    def do_POST(self):
        fault = self.headers.get("X-Fault") or DEFAULT_FAULT
        body = self.rfile.read(int(self.headers.get("Content-Length") or 0))

        if fault == "ratelimit":
            return self._send(429, b'{"error": "rate_limited"}', {"Retry-After": "2"})
        if fault == "stall":
            time.sleep(30)  # longer than any sane client timeout
            return self._send(504, b'{"error": "stalled"}')

        req = urllib.request.Request(
            UPSTREAM + self.path,
            data=body,
            headers={"Content-Type": "application/json"},
        )
        try:
            with urllib.request.urlopen(req, timeout=20) as resp:
                payload = resp.read()
        except urllib.error.HTTPError as exc:
            return self._send(exc.code, exc.read())
        except (urllib.error.URLError, TimeoutError):
            return self._send(502, b'{"error": "upstream_unreachable"}')

        if fault == "truncate":
            payload = payload[: max(1, len(payload) // 3)]
        elif fault == "quotes":
            payload = payload.replace(b'"', b"'", 3)
        self._send(200, payload)

    def _send(self, status, payload, extra=None):
        self.send_response(status)
        self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length", str(len(payload)))
        for key, value in (extra or {}).items():
            self.send_header(key, value)
        self.end_headers()
        self.wfile.write(payload)

    def log_message(self, fmt, *args):
        sys.stderr.write("proxy: " + fmt % args + "\n")


if __name__ == "__main__":
    print(f"faultproxy :{PORT} fault={DEFAULT_FAULT} upstream={UPSTREAM}", file=sys.stderr)
    http.server.ThreadingHTTPServer(("127.0.0.1", PORT), Handler).serve_forever()

Three design choices carry the lesson. The fault arrives by header, so nobody restarts a process between runs. truncate returns HTTP 200 with a short body, which is the point: the status code is not the health check. stall sleeps for thirty seconds, longer than the client cap, so students measure a timeout instead of reading about one.

The student client: one call, one label

#!/usr/bin/env python3
# classify.py — send one call and name the outcome
import json
import socket
import sys
import time
import urllib.error
import urllib.request

URL = sys.argv[1] if len(sys.argv) > 1 else "http://127.0.0.1:8787/v1/chat/completions"
FAULT = sys.argv[2] if len(sys.argv) > 2 else None
BODY = {"messages": [{"role": "user", "content": "reply with the single word: ready"}]}


def call(timeout=8.0):
    headers = {"Content-Type": "application/json"}
    if FAULT:
        headers["X-Fault"] = FAULT
    req = urllib.request.Request(URL, data=json.dumps(BODY).encode(), headers=headers)
    started = time.monotonic()
    try:
        with urllib.request.urlopen(req, timeout=timeout) as resp:
            status, raw = resp.status, resp.read()
    except urllib.error.HTTPError as exc:
        return {"class": "http_error", "status": exc.code, "ms": _ms(started)}
    except (socket.timeout, TimeoutError):
        return {"class": "timeout", "ms": _ms(started)}
    except urllib.error.URLError as exc:
        return {"class": "transport", "detail": str(exc.reason), "ms": _ms(started)}

    try:
        json.loads(raw)
    except json.JSONDecodeError as exc:
        return {"class": "bad_json", "status": status, "bytes": len(raw),
                "detail": exc.msg, "pos": exc.pos, "ms": _ms(started)}
    return {"class": "ok", "status": status, "bytes": len(raw), "ms": _ms(started)}


def _ms(started):
    return round((time.monotonic() - started) * 1000)


if __name__ == "__main__":
    print(json.dumps(call(), indent=2))

A student can also drive the proxy with curl, which is useful for students who want to see the raw bytes first:

curl -s -H 'X-Fault: truncate' -H 'Content-Type: application/json' \
  -d '{"messages":[{"role":"user","content":"say ready"}]}' \
  http://127.0.0.1:8787/v1/chat/completions | head -c 200

The run sheet

Ninety minutes, pairs at one laptop, instructor circulating.

Minutes	What happens	What the instructor checks
0–10	Baseline call with X-Fault: none through the proxy	every laptop prints "class": "ok"
10–25	Students write a prediction, then run truncate and quotes	predictions exist before the run
25–40	ratelimit and stall with the client cap left at eight seconds	elapsed time comes from measurement
40–60	Each pair adds one new fault to the proxy	their fault breaks a neighbouring client
60–75	Pairs ask which faults a plain retry would fix	answers name the fault, not the fix
75–90	The decision table gets filled from the students' own output	table is evidence, not lecture notes

The prediction step is not decoration. A student who guesses "it will throw an exception" and then sees "class": "bad_json", "status": 200 has learned something a slide cannot teach.

A worked example for the instructor

Start the proxy against any endpoint that speaks JSON, then run the baseline:

UPSTREAM=https://your-endpoint.example python3 faultproxy.py        # terminal 1
python3 classify.py http://127.0.0.1:8787/v1/chat/completions none # terminal 2

The result is "class": "ok" with a byte count worth writing on the board, because every later run is compared against it. Now send the truncation:

python3 classify.py http://127.0.0.1:8787/v1/chat/completions truncate

The reply is "class": "bad_json" with "status": 200, a byte count near a third of the baseline, and a detail that usually reads as an unterminated string or an unexpected end of input. That single result is the whole lesson: the transport succeeded, the payload did not. Run quotes next and the byte count barely moves while the parse still fails, which is how students learn to read bytes and detail together instead of trusting either alone.

Finish on stall. The client returns in roughly eight seconds with "class": "timeout", and the proxy keeps sleeping behind it — a cheap demonstration that an abandoned request does not stop work on the other side.

Where a free option changes the lesson

The exercise needs two things that budget paperwork usually blocks: a model endpoint every student can hit, and one machine that can host the proxy so twelve laptops do not each run their own copy. MonkeyCode's free model access and free server option cover exactly those two needs, which is why the workshop can be run without a card or a quota meeting.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Limits and quotas on a free tier move, so check the project page on the morning you plan a session and keep a fallback endpoint configured in UPSTREAM. If you host the proxy on a shared machine, bind it to a private address or put an access rule in front of it; an open proxy that forwards credentials is a liability, not a teaching aid.

The debrief question that does the work

Ask which faults a blind retry would have fixed. Rate limits and a single upstream 5xx qualify, with a wait and a cap. A truncation never does — retrying a request that produced half a body tends to produce another half a body, and the student who writes except Exception: retry() has just paid twice for the same corruption. That question, answered out loud by the pairs themselves, is worth more than any summary slide at the end.

Limits, and who should skip this

This is a teaching proxy, not a chaos platform. It binds to loopback by default, it does not emulate DNS failures, TLS handshakes, or slow token-by-token streaming, and its faults are deterministic rather than statistically realistic. Anyone who needs production-grade failure testing should reach for established fault-injection tooling instead.

Skip the workshop if the group has never written an HTTP client, and absolutely skip pointing the proxy at an endpoint you do not own or have permission to test. Also skip it if the goal is load testing; thirty seconds of sleeping threads is not a benchmark.

For a classroom-sized first run, the free model access and the free server option are a reasonable place to start, with the current limits checked on the project page before the lesson is planned around them.