The Roll Call Workshop: Telling Throttling Apart from an Outage

Picture a nightly batch job that has been green for eleven nights. On the twelfth it dies at 03:10 with one line in the log: HTTP 429. The retry loop treats it like any other error, backs off politely, tries again, and drains the whole window. By morning the queue is behind and nobody can say whether the endpoint was busy, the key had expired, or the model name had been retired the week before.

To a caller that only asks if response.ok, those three failures are identical. That blindness is the subject of this workshop. It is not a question about answer quality — benchmarking covers that ground well — but about what the caller should do in the ninety seconds after an answer fails to arrive.

The frame is a roll call. At a fixed interval the caller checks in, records enough context to classify what came back, and applies one of three policies: retry, degrade, or stop. A hundred minutes, four exercises, and three artifacts a team keeps afterwards: a probe, a classifier with fixtures, and a decision table.

Why this drill, this week

The loudest AI threads on DEV this week were about organizations rather than syntax: whether teams are building the thing that replaces them, and whether a single internal presentation can move a company. Strip the framing away and both arguments land on the same engineering question. Can the pipeline still behave when the model endpoint does not?

Most teams answer that from memory instead of from data. They know the 429s are occasional and the timeouts are rare, because the only record is a log line that scrolled away. A roll call replaces the memory with a table.

0:00–0:10 — Four failure families

The opening ten minutes go to naming what can go wrong, because a caller cannot classify what it has never listed. Four families matter in practice: throttling, credentials, routing, and transport. Throttling arrives as 429 and usually carries Retry-After. Credentials arrive as 401 or 403 and never fix themselves. Routing arrives as 404, or as a 200-wrapped error body naming a model the account cannot reach. Transport is everything the socket layer reports — timeouts, DNS, TLS, resets — and it is the family most often misread as a provider outage.

Students write one sentence per family describing what their current code does about it. Those sentences are the first artifact, and they are usually humbling.

0:10–0:30 — A probe with no dependencies

Exercise one builds a single-shot probe using only the standard library, so nothing in the drill depends on a framework version. The request path below is illustrative; use whatever path the provider documents.

# canary.py — one request, one JSON line
import json, time, urllib.error, urllib.request

def probe(url, key, model, prompt='ping', timeout=20):
    body = json.dumps({
        'model': model,
        'messages': [{'role': 'user', 'content': prompt}],
        'max_tokens': 8,
    }).encode('utf-8')
    req = urllib.request.Request(url, data=body, method='POST', headers={
        'Content-Type': 'application/json',
        'Authorization': 'Bearer ' + key,
    })
    started = time.monotonic()
    rec = {'ts': time.time(), 'model': model}
    try:
        with urllib.request.urlopen(req, timeout=timeout) as resp:
            rec['status'] = resp.status
            rec['retry_after'] = resp.headers.get('Retry-After')
            rec['raw'] = resp.read(512).decode('utf-8', 'replace')
    except urllib.error.HTTPError as err:
        rec['status'] = err.code
        rec['retry_after'] = err.headers.get('Retry-After') if err.headers else None
        rec['raw'] = err.read(512).decode('utf-8', 'replace')
    except Exception as err:
        rec['status'] = 0
        rec['retry_after'] = None
        rec['raw'] = type(err).__name__ + ': ' + str(err)
    rec['ms'] = round((time.monotonic() - started) * 1000, 1)
    return rec

The 512-byte clamp is deliberate. A probe that stores full completions becomes a data-retention problem on day two, and the first 512 bytes carry everything the classifier needs.

Run it once from the shell, and look at the raw record before writing any logic:

BASE_URL=... API_KEY=... MODEL=... python3 canary.py

0:30–0:50 — The classifier and its fixtures

Classification should be a pure function over the record, tested without a network. That constraint keeps the exercise fast and makes the result reviewable by someone who is not in the room.

def classify(rec):
    status = rec.get('status', 0)
    raw = (rec.get('raw') or '').lower()
    if status == 0:
        return 'transport'
    if status in (401, 403):
        return 'auth'
    if status == 429:
        return 'throttled'
    if status == 404 or 'unknown model' in raw:
        return 'routing'
    if 500 <= status < 600:
        return 'upstream'
    if status == 200:
        return 'ok'
    return 'unknown'

Fixtures come next, one short entry per family. These are constructed examples for the test, not measurements from any service:

FIXTURES = [
    ({'status': 429, 'retry_after': '12', 'raw': 'rate limit exceeded'}, 'throttled'),
    ({'status': 401, 'retry_after': None, 'raw': 'invalid api key'}, 'auth'),
    ({'status': 404, 'retry_after': None, 'raw': 'unknown model: legacy-v1'}, 'routing'),
    ({'status': 0,   'retry_after': None, 'raw': 'TimeoutError: timed out'}, 'transport'),
    ({'status': 200, 'retry_after': None, 'raw': 'pong'}, 'ok'),
]

for rec, expected in FIXTURES:
    assert classify(rec) == expected, (rec, classify(rec))

Work the first fixture by hand in front of the room. Status 429, a Retry-After of twelve seconds, an error string that adds no information — the function returns throttled, and the detail that matters is the twelve, because the gate will spend it. The second fixture is the one that changes behaviour: auth is a permanent class, and a caller that backs off on a 401 has only made the incident slower.

The moment worth protecting is when someone adds a sixth family because their own log shows a shape the classifier calls unknown. That is the drill working as intended.

0:50–1:10 — A gate and a written policy

The gate is a token bucket with jitter, and its only interesting feature is that the server can overrule it.

import random, time

class Gate:
    def __init__(self, per_minute, burst=3):
        self.rate = per_minute / 60.0
        self.burst = burst
        self.tokens = float(burst)
        self.last = time.monotonic()

    def _refill(self):
        now = time.monotonic()
        self.tokens = min(self.burst, self.tokens + (now - self.last) * self.rate)
        self.last = now

    def wait(self, retry_after=None):
        self._refill()
        if self.tokens >= 1:
            self.tokens -= 1
            return 0.0
        delay = (1 - self.tokens) / self.rate
        if retry_after:
            delay = max(delay, float(retry_after))
        delay += random.uniform(0, 0.5)
        time.sleep(delay)
        return delay

Then the table, which is the artifact most teams actually keep after the workshop:

Class	Retry?	First action	Give up after
throttled	yes	wait Retry-After, then jitter	3 attempts, then degrade
auth	never	stop, page the key owner	immediate
routing	never	fail fast, check config	immediate
upstream	yes	backoff from 1s, double each time	5 attempts, then degrade
transport	yes	retry with doubled timeout	2 attempts
unknown	once	record everything, then stop	1 attempt

Degrade gets defined before the exercise ends: serve the last known good answer, or return an explicit unavailable marker. Both beat a stack trace at 03:10.

1:10–1:40 — Leaving the timer running

The last exercise moves the probe off a laptop. A dozen characters of cron is enough, and a systemd timer works just as well:

*/15 * * * * cd /srv/rollcall && /usr/bin/python3 canary.py >> roll-call.jsonl

Two practical blockers stop this step in most workshops: an API key, and somewhere to leave a timer running. The operator of MonkeyCode states that the project offers free model access and a free server option, so students without a paid key can run the whole drill and let the canary accumulate a day of records. Treat both as operator-supplied availability claims rather than guarantees, and read the current allowances, model availability, and terms in the project's own documentation before designing anything around them — free-tier numbers move.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The report at the end is deliberately dull:

import json, sys
from collections import Counter

rows = [json.loads(line) for line in open(sys.argv[1])]
print(Counter(classify(r) for r in rows).most_common())

A day of output turns the endpoint is flaky into 41 throttled, 3 transport, 0 auth — a sentence someone can argue with.

1:40–1:50 — Debrief, and what the numbers cannot say

Four questions close the session. Did any class appear that the table does not cover? Did any retry make things worse? Would the caller have survived the slot using the cached answer it already had? And which of the six rows does nobody on the team own?

The honest limits matter as much as the artifact. A fifteen-minute interval misses short bursts and can misread a provider's rolling window as a pattern. A single key cannot separate the account is throttled from the endpoint is throttled. Free-tier behaviour is not production behaviour, so a table built there should be re-run before anyone trusts it for capacity planning. Nothing in this drill measures answer quality, and no amount of classification will tell you whether the model is good enough for the task.

Skip the drill if you are choosing a model, tuning prompts, or load-testing — the first two need an evaluation harness and the third needs concurrency this probe deliberately avoids. Skip it too if your pipeline already emits per-class error metrics with a named owner for each class, because the workshop exists to create that instrumentation rather than replace it.

For a team that has none of it, a hundred minutes buys the difference between a 429 that pages nobody and a 429 that means something. If the outline is useful, point it at MonkeyCode's free tier, leave the timer on overnight, and read the table in the morning — your own rows will teach more than this page can.