A 200 OK Is Not a Finished Answer: Classifying Model-Call Failures Before You Retry

Last week one generation came back with a 200. My parser died at character 812. So was that a failure? Or a good answer that stopped early?

I keep finding the same bug in client code. Every non-200 becomes 'retry'. Every 200 becomes 'done'. Both assumptions burn tokens and hide the real problem.

Here is the mental model I use now. A model call has three outcomes, not two. It fails, it succeeds, or it stops partway. Most clients only have buckets for the first two.

Myth 1: A 200 means the model answered

A 200 means the server accepted the request. That is all. The body can be empty. The body can be an error envelope dressed up as a 200.

Streaming makes this worse. The 200 arrives before the first token exists. The status line says nothing about the content.

So what does your code check after the status line? If the answer is 'nothing', your parser is doing a validator's job.

Myth 2: The stream ended, so the answer is complete

A stream ends when the connection closes. That is not a finished generation.

The evidence you want is finish_reason. Missing means truncated. A value of length means a token cap cut the answer mid-sentence.

Partial JSON often still parses, if you slice carefully. That is the nasty case. You get a valid object with a missing field and zero errors anywhere.

Myth 3: Retrying a call is free

Retries cost tokens, wall time, and sometimes duplicate side effects. If the call also runs a tool, a retry can run that tool twice.

Retry storms also make rate limits worse. A tight loop on a 429 is you building your own outage. Stop the run instead.

Myth 4: If the JSON parses, the schema is fine

Parsing is not validation. Parsing asks 'is this a JSON document?' Validation asks 'does it have the fields I need?'

Check the keys you depend on. Check their types. Reject early, loudly, with the raw text attached.

Myth 5: A slow call is a broken call

Slow and failed are two different axes. A nine-second success is still a success. Collapsing both into one 'unhealthy' gauge hides which one you actually have.

Record elapsed time per call. Decide about it later, with data instead of vibes.

A small probe that classifies outcomes

Here is the artifact I run before trusting a new endpoint. One file, one log, one row per call.

#!/usr/bin/env python3
"""probe.py - classify model-call outcomes instead of guessing.

    python probe.py --url "$MODEL_URL" --key "$MODEL_KEY" --n 12 --out probe.jsonl

Keep --n small. This is an evidence log, not a load test.
The request body is a chat-completions example. Adapt it to your API.
"""
import argparse
import json
import time
import urllib.error
import urllib.request
from collections import Counter

PROMPT = 'Reply with a small JSON object that has keys ok and n.'
REQUIRED = {'ok', 'n'}


def extract(raw):
    """Pull text and finish_reason out of an unknown envelope."""
    try:
        env = json.loads(raw)
    except json.JSONDecodeError:
        return '', None
    choice = (env.get('choices') or [{}])[0]
    msg = choice.get('message') or {}
    text = msg.get('content') or choice.get('text') or ''
    return text, choice.get('finish_reason')


def classify(status, text, finish_reason, err):
    if err:
        return 'TIMEOUT' if err == 'timeout' else 'NETWORK'
    if status == 429:
        return 'RATE_LIMITED'
    if status >= 500:
        return 'SERVER_ERROR'
    if 400 <= status < 500:
        return 'CLIENT_ERROR'
    if not text.strip():
        return 'EMPTY_CONTENT'
    if finish_reason in (None, '', 'length'):
        return 'TRUNCATED'
    try:
        obj = json.loads(text)
    except json.JSONDecodeError:
        return 'NOT_JSON'
    if not REQUIRED.issubset(obj):
        return 'SCHEMA_MISSING_KEYS'
    return 'OK'


def one_call(url, key, timeout):
    body = json.dumps({
        'messages': [{'role': 'user', 'content': PROMPT}],
        'stream': False,
    }).encode('utf-8')
    req = urllib.request.Request(url, data=body, method='POST')
    req.add_header('Content-Type', 'application/json')
    if key:
        req.add_header('Authorization', 'Bearer ' + key)

    start, status, raw, err = time.monotonic(), None, '', None
    try:
        with urllib.request.urlopen(req, timeout=timeout) as resp:
            status = resp.status
            raw = resp.read().decode('utf-8', 'replace')
    except urllib.error.HTTPError as e:
        status = e.code
        raw = e.read().decode('utf-8', 'replace')
    except (urllib.error.URLError, OSError) as e:
        err = 'timeout' if 'timed out' in str(e) else 'network'

    text, finish = extract(raw)
    return {
        'status': status,
        'elapsed_s': round(time.monotonic() - start, 3),
        'finish_reason': finish,
        'chars': len(text),
        'outcome': classify(status, text, finish, err),
        'raw_head': raw[:200],
    }


def main():
    p = argparse.ArgumentParser()
    p.add_argument('--url', required=True)
    p.add_argument('--key', default='')
    p.add_argument('--n', type=int, default=10)
    p.add_argument('--sleep', type=float, default=2.0)
    p.add_argument('--timeout', type=float, default=60.0)
    p.add_argument('--out', default='probe.jsonl')
    args = p.parse_args()

    rows = []
    with open(args.out, 'w', encoding='utf-8') as fh:
        for i in range(args.n):
            row = one_call(args.url, args.key, args.timeout)
            row['i'] = i
            fh.write(json.dumps(row) + '\n')
            fh.flush()
            rows.append(row)
            print(i, row['outcome'], row['status'], row['elapsed_s'])
            if row['outcome'] == 'RATE_LIMITED':
                print('rate limited: stop, do not tighten the loop')
                break
            time.sleep(args.sleep)

    print('tally:', dict(Counter(r['outcome'] for r in rows)))
    print('log:', args.out)


if __name__ == '__main__':
    main()

Run it, then summarize without eyeballing twelve rows.

python probe.py --url "$MODEL_URL" --key "$MODEL_KEY" --n 12 --out probe.jsonl
jq -r '.outcome' probe.jsonl | sort | uniq -c | sort -rn
jq -r 'select(.outcome=="TRUNCATED") | [.i,.status,.chars,.finish_reason] | @tsv' probe.jsonl

Turn each outcome into a retry rule

Do not write one except. Write one rule per class.

Outcome	Retry?	Wait	Log
OK	no	none	elapsed, chars
EMPTY_CONTENT	once	2s	raw_head
TRUNCATED	once, smaller cap	2s	finish_reason, chars
NOT_JSON	no auto-retry	none	full text
SCHEMA_MISSING_KEYS	no auto-retry	none	parsed keys
RATE_LIMITED	no	stop the run	status, time
SERVER_ERROR	once	5s + jitter	status, raw_head
TIMEOUT	once	5s	elapsed, timeout value
NETWORK	once	5s	error string
CLIENT_ERROR	never	none	status, raw_head

Two rules matter more than the rest. Never retry a 4xx that came from your own bad request. Never auto-retry a schema failure, because it usually repeats.

What the tally actually tells you

Twelve calls in under a minute is a smoke test. It does not measure p50, and you should not report it as one.

It answers a smaller question. Which failure classes does this endpoint produce, for my request shape, right now? That is exactly what your retry code needs to know.

If you see four classes in twelve calls, your old except was hiding all of them. If you see only OK, you learned the happy path and nothing else.

Where I run the probe

My laptop lid closes. That kills a one-minute run halfway through.

For this I want a small box that stays awake for the run and has its own network path. MonkeyCode offers a free server option, and free model access for the model side, so I can keep the whole loop off my own machine. Both are availability claims from the product, not benchmarks from me.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

I treat that box as 'somewhere that is not my laptop'. Not as a scheduler, not as durable storage. The run writes probe.jsonl, and I copy the log out when it finishes.

Keep runs short and spaced. Free access is not a licence to hammer an endpoint, and I do not assume it stays identical forever.

What this probe does not prove

• It is not a benchmark. Twelve calls, one hour, one network path.
• It does not test long prompts, tool calls, or image inputs.
• It cannot tell you an endpoint changed; it only shows what happened today.
• It says nothing about cost, quota, or whether free access continues.

Who should skip this

If you need an SLA number, this is the wrong tool. Use a real load test with a large sample, a fixed schedule, and a proper percentile report.

If you are running at volume, add tracing and per-class metrics in production instead. A probe script is for the day before you trust an endpoint.

If a duplicate tool call would hurt you, fix idempotency first. Retry policy comes second.

The corrected mental model

A response is not one bit. It is a status, a body, a finish reason, and a duration.

Read all four before you decide anything. Then pick a retry rule per class, not per exception type.

If you run the probe, keep --n small, keep the log, and change REQUIRED to match your own schema.