The Eviction Drill: Prove Your Agent Survives Losing Its Free Server in 30 Minutes

Your agent runs fine for weeks, and then the free server underneath it disappears at 02:40 on a Tuesday. The failure you get is rarely a clean error; it is forty minutes of silence in a queue nobody is watching, followed by a morning of guesswork. That scenario is cheap to rehearse and expensive to discover, which is why the playbook below exists.

This is a 30-minute eviction drill: a scheduled, reversible rehearsal in which you deliberately break the model endpoint and observe whether your worker degrades or collapses. You get three named roles, a dead proxy you can run on localhost, a timeline format, and a gate script that returns a pass or fail you can paste into a wiki page. Nothing here requires a paid staging cluster, and nothing here asks you to trust a dashboard.

Why rehearse eviction instead of running another load test

A load test answers "how much traffic can this absorb before latency climbs" and it is a useful question. An eviction drill answers a different one: "when the upstream endpoint stops existing, what does a real user see in the next sixty seconds." Those two questions fail in different ways, and teams routinely pass the first while failing the second.

The drill is also intentionally small. You are not testing capacity, model quality, or cost; you are testing one control-flow branch that everyone writes and almost nobody exercises. If your fallback path has never been triggered on purpose, treat it as unverified code rather than as a safety net.

The three roles and the two handoff rules

Assign roles before you touch anything, because the drill is worthless if the same person injects the fault and judges the outcome.

1. Drill Lead owns the clock and the fault injection; only this person starts and stops the drill.
2. Scribe owns the timeline file and writes down wall-clock events as they happen, including the boring ones.
3. Verifier owns the gate script and the final verdict; this person does not inject faults and does not fix code mid-drill.

Two handoff rules keep the rehearsal honest. First, the Lead hands off to the Scribe at injection time and announces the timestamp out loud, so the timeline has an anchor that is not reconstructed from memory. Second, the Verifier receives the timeline file, not a summary of it; if the run needs an explanation to pass, it did not pass.

Step 1: Write the failure budget on a single line

Before any code, agree on one sentence that everyone can read back. A usable example is: "For 30 minutes, no more than 1% of user requests may return 5xx, and a fallback must engage within 60 seconds of the fault."

Write that line at the top of the wiki card and do not negotiate it during the drill. Budgets that get adjusted while the drill is running are how a rehearsal turns into a performance of competence, which is the opposite of what you want.

Step 2: Stand up a dead proxy instead of breaking real infrastructure

You do not need to wait for a genuine outage to test outage behavior. The script below is a tiny TCP server that accepts connections and then misbehaves in one of three ways: it resets the socket, it returns HTTP 503, or it accepts the request and hangs forever. Point your model base URL at it and your agent will believe the provider is gone.

# deadproxy.py - simulate an upstream outage without touching real infrastructure.
# Template only: read it before you run it, and keep it on localhost.
import argparse, itertools, socket, sys, threading, time
from datetime import datetime, timezone

def log(msg):
    stamp = datetime.now(timezone.utc).isoformat(timespec='seconds')
    print(f'{stamp} {msg}', file=sys.stderr, flush=True)

def handle(conn, mode):
    try:
        if mode == 'reset':
            # SO_LINGER with a zero timeout makes close() send RST instead of FIN.
            conn.setsockopt(socket.SOL_SOCKET, socket.SO_LINGER, b'\x01\x00\x00\x00\x00\x00\x00\x00')
            conn.close()
            log('reset one connection')
            return
        if mode == '503':
            conn.recv(65536)
            body = b'{"error":"upstream_unavailable"}'
            head = (b'HTTP/1.1 503 Service Unavailable\r\n'
                    b'Content-Type: application/json\r\n'
                    b'Content-Length: ' + str(len(body)).encode() + b'\r\n'
                    b'Connection: close\r\n\r\n')
            conn.sendall(head + body)
            log('served 503')
            return
        if mode == 'hang':
            log('hanging one connection')
            time.sleep(3600)
    finally:
        try:
            conn.close()
        except OSError:
            pass

def main():
    ap = argparse.ArgumentParser()
    ap.add_argument('--mode', choices=['reset', '503', 'hang'], default='503')
    ap.add_argument('--port', type=int, default=8081)
    ap.add_argument('--host', default='127.0.0.1')
    args = ap.parse_args()

    srv = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    srv.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
    srv.bind((args.host, args.port))
    srv.listen(64)
    log(f'dead proxy listening on {args.host}:{args.port} mode={args.mode}')

    while True:
        conn, _ = srv.accept()
        threading.Thread(target=handle, args=(conn, args.mode), daemon=True).start()

if __name__ == '__main__':
    main()

Run the three modes in separate rehearsals rather than picking a favorite, because a reset, a 503, and a timeout land in different exception branches of most HTTP clients. A team that only rehearses 503 usually discovers on the second drill that their client retries forever on connection resets.

This is also where the choice of host matters for planning. The account behind this article rehearses against MonkeyCode's free model access and its free server option, so the drill cost stays near zero and can be repeated monthly. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Treat the free token grant and the free server option as terms to re-read on the project page before you schedule a drill around them, since availability details are exactly the kind of thing that changes without an announcement.

Step 3: Make your agent emit a timeline, not log noise

A drill you cannot reconstruct is a story. Add a four-line helper that appends one JSON object per event, and call it at the decision points that matter.

# timeline.py - append-only event sink for one drill run.
import json, os
from datetime import datetime, timezone

SINK = os.environ.get('DRILL_LOG', 'drill.jsonl')

def mark(event, **fields):
    row = {'t': datetime.now(timezone.utc).isoformat(timespec='milliseconds'),
           'event': event, **fields}
    with open(SINK, 'a', encoding='utf-8') as fh:
        fh.write(json.dumps(row, sort_keys=True) + '\n')

Then instrument the one branch you actually care about. The names below are deliberately stable, because the gate script in Step 5 greps for them.

from timeline import mark

def answer(prompt, request_id):
    mark('user_request', id=request_id, phase='start')
    try:
        text = call_primary_model(prompt)
        mark('model_ok', id=request_id, path='primary')
    except ProviderUnavailable:
        mark('fallback_engaged', id=request_id, path='fallback')
        text = cached_or_smaller(prompt)
    except Exception:
        mark('unhandled_exception', id=request_id, kind='unknown')
        raise
    mark('user_request', id=request_id, phase='end', status=200)
    return text

If your worker already swallows exceptions in a background loop, add one mark('user_request', status=502) call at that boundary. The goal is a file where every user-visible outcome is present, not a file that only records the happy path.

Step 4: Run the 30-minute rehearsal

Keep the wall clock boring: start the proxy, start the worker, generate steady traffic, then inject the fault ten minutes in and leave it injected for fifteen.

# terminal 1 - the fake provider
python deadproxy.py --mode reset --port 8081

# terminal 2 - your worker, pointed at the fake provider
export MODEL_BASE_URL=http://127.0.0.1:8081
export DRILL_LOG="drill-$(date +%Y%m%d-%H%M).jsonl"

# terminal 3 - steady user-shaped traffic for the whole window
end=$((SECONDS + 1800))
while [ $SECONDS -lt $end ]; do
  ts=$(date -u +%Y-%m-%dT%H:%M:%SZ)
  code=$(curl -s -o /dev/null -w '%{http_code}' \
    "http://127.0.0.1:8000/answer?q=drill-ping" || echo 000)
  printf '{"t":"%s","event":"user_request","id":"curl","status":%s}\n' \
    "$ts" "$code" >> "$DRILL_LOG"
  sleep 5
done

The Scribe writes three lines by hand during the run: injection time, the first fallback they observe, and the end of the window. Those three lines outrank everything else when the Verifier asks whether the timeline is complete.

Step 5: Gate the result with a script, not with opinions

This is the artifact that makes the drill repeatable. It reads the timeline and returns a non-zero exit code when the failure budget is broken.

# evict_gate.py - pass/fail for one eviction drill run.
import json, sys
from datetime import datetime

def load(path):
    rows = []
    with open(path, encoding='utf-8') as fh:
        for line in fh:
            line = line.strip()
            if line:
                rows.append(json.loads(line))
    return rows

def ts(row):
    return datetime.fromisoformat(row['t'].replace('Z', '+00:00'))

def first(rows, event):
    return next((r for r in rows if r['event'] == event), None)

path = sys.argv[1] if len(sys.argv) > 1 else 'drill.jsonl'
budget = float(sys.argv[2]) if len(sys.argv) > 2 else 0.01
rows = load(path)

inj = first(rows, 'fault_injected') or first(rows, 'fallback_engaged')
fb = first(rows, 'fallback_engaged')
checked = [('fault or fallback recorded', inj is not None),
           ('fallback engaged', fb is not None)]
if inj and fb:
    delta = (ts(fb) - ts(inj)).total_seconds()
    checked.append((f'fallback within 60s (observed {delta:.0f}s)', delta <= 60))

reqs = [r for r in rows if r['event'] == 'user_request' and 'status' in r]
fails = [r for r in reqs if int(r['status']) >= 500]
rate = (len(fails) / len(reqs)) if reqs else 1.0
checked.append((f'5xx rate {rate:.2%} <= {budget:.2%}', rate <= budget))
checked.append(('no unhandled exception', first(rows, 'unhandled_exception') is None))

for name, ok in checked:
    print(('PASS ' if ok else 'FAIL ') + name)
sys.exit(0 if all(ok for _, ok in checked) else 1)

Run it as python evict_gate.py "$DRILL_LOG" 0.01. If your worker writes its own timeline and never emits fault_injected, add a Lead-only mark call or accept the fallback timestamp as the anchor, but write down which choice you made so the next drill is comparable.

Reading the result: a decision table

Timeline signature	Most likely cause	Next action
No fallback_engaged at all	Fallback is dead code or catches the wrong exception class	Trigger the fallback in a unit test, then re-run the drill
Fallback engages fast but 5xx exceeds budget	Fallback returns an error for a subset of prompts	Log the fallback's own failures and give it a default response
Fallback within 60s, latency doubles	Cheaper fallback path is slow, often on cold cache	Warm the cache before the window and measure again
One unhandled_exception, then silence	Exception escapes a background loop and kills the worker	Add a supervisor restart and a status=502 marker
Drill passes, production still failed	Real outage involved quota exhaustion or auth expiry, not transport errors	Add a second drill mode for non-transport failures

The last row is the one teams skip, and it is the reason a passing drill is not a warranty. Transport faults are the easy class to simulate locally; quota and credential faults usually require a provider-side control you may not have.

The one-page wiki card to paste before the next drill

# Eviction Drill - <service name> - <date>
Budget: no more than 1% 5xx for 30 min; fallback within 60s of fault.
Roles: Lead=<name>  Scribe=<name>  Verifier=<name>
Rules: Lead injects, Scribe timestamps, Verifier judges. No code edits mid-drill.

Pre-flight
  [ ] quiet window agreed, on-call told, status page note drafted
  [ ] baseline run recorded with the real endpoint (10 min, no fault)
  [ ] DRILL_LOG path agreed and writable by the worker user

Run
  [ ] python deadproxy.py --mode <reset|503|hang> --port 8081
  [ ] export MODEL_BASE_URL=http://127.0.0.1:8081
  [ ] traffic loop started, Scribe notes injection time
  [ ] 15 minutes of injected fault, then proxy stopped

Verdict
  [ ] python evict_gate.py "$DRILL_LOG" 0.01
  [ ] gate output pasted verbatim below
  [ ] one follow-up issue per FAIL line, each with an owner

Gate output:
  <paste here>

Keep the card short enough that someone can read it aloud at the start of the drill. Long cards get skimmed, and skimmed cards produce timelines with missing timestamps.

Limitations and who should skip this

The dead proxy exercises HTTP-level behavior only; it cannot reproduce clock skew, tokenizer changes, or a provider quietly returning 200 with a degraded answer. A green gate therefore proves your degradation path exists, not that your product stays correct under pressure.

Skip this playbook if you are a solo developer with no users yet, because a 30-minute rehearsal with three roles costs more than the outage would. Skip it too if your paying customers depend on a free tier as their only capacity, and fix the dependency before you rehearse around it. And if your drill needs a change window, schedule it rather than sneaking it into a Friday afternoon.

If you want to run this against a free server and a free model grant instead of a paid staging account, that is the setup this account uses and the reason the drill can be monthly rather than annual. Start with the reset mode, because it breaks the most clients and teaches the most in thirty minutes.