Learn Fallback Logic by Building a Tiny Failure-Observer

Most fallback code looks complete until you test the path where the primary endpoint fails.

This is a tiny Python harness that sends one prompt to a primary model endpoint, catches the failure, and falls back to a second endpoint. The learning question is small: which failures should trigger fallback, and how do you know the fallback actually ran?

Problem

A single model call is easy to reason about:

answer = call_model(PRIMARY_URL, prompt)

Add a fallback and the logic gets harder:

• Which failures should route to fallback?
• How many retries are worth paying for?
• Did the fallback silently send the same prompt twice?
• Can you see token consumption for each attempt?

Without an observed failure path, you are guessing.

Where free tokens fit

The operator-supplied material for this article describes MonkeyCode as an open-source project with free model access, a 30-million-token allowance, and a free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The token allowance and server availability were not independently measured for this article, so treat them as a disposable test environment, not as a production guarantee.

You do not need a paid key to practice the unhappy path. Use a free endpoint for the fallback, send a few failing fixtures, and read the token usage instead of guessing.

Prerequisites

• Python 3.10+
• Two OpenAI-compatible endpoints: one primary, one fallback
• No third-party SDK required
• Environment variables for both base URLs and keys

Build the harness

Save this as fallback_harness.py:

import json
import os
import time
import urllib.error
import urllib.request

PRIMARY_URL = os.environ['PRIMARY_URL']
PRIMARY_KEY = os.environ['PRIMARY_KEY']
FALLBACK_URL = os.environ['FALLBACK_URL']
FALLBACK_KEY = os.environ['FALLBACK_KEY']
MODEL = os.environ['MODEL']


def chat_once(base_url, api_key, model, prompt, timeout=8):
    payload = {
        'model': model,
        'messages': [{'role': 'user', 'content': prompt}],
        'temperature': 0,
    }
    req = urllib.request.Request(
        base_url.rstrip('/') + '/v1/chat/completions',
        data=json.dumps(payload).encode(),
        headers={
            'Authorization': 'Bearer ' + api_key,
            'Content-Type': 'application/json',
        },
        method='POST',
    )
    started = time.time()
    try:
        with urllib.request.urlopen(req, timeout=timeout) as resp:
            body = json.loads(resp.read().decode())
            text = body['choices'][0]['message']['content']
            usage = body.get('usage', {})
            return {
                'ok': True,
                'text': text,
                'prompt_tokens': usage.get('prompt_tokens', 0),
                'completion_tokens': usage.get('completion_tokens', 0),
                'seconds': round(time.time() - started, 2),
                'status': getattr(resp, 'status', 200),
            }
    except urllib.error.HTTPError as e:
        return {'ok': False, 'status': e.code, 'error': e.read().decode()[:120]}
    except Exception as e:
        return {'ok': False, 'status': None, 'error': type(e).__name__ + ': ' + str(e)}


def route(prompt, attempts=2):
    decisions = []
    result = chat_once(PRIMARY_URL, PRIMARY_KEY, MODEL, prompt)
    decisions.append(('primary', result['ok'], result['status'], result['error']))
    if not result['ok'] and PRIMARY_URL != FALLBACK_URL:
        for i in range(attempts):
            result = chat_once(FALLBACK_URL, FALLBACK_KEY, MODEL, prompt)
            decisions.append(('fallback', result['ok'], result['status'], result['error']))
            if result['ok']:
                break
            time.sleep(0.5 * (i + 1))
    return result, decisions


if __name__ == '__main__':
    prompt = 'Say the word ready.'
    result, decisions = route(prompt)
    print(json.dumps({**result, 'decisions': decisions}, indent=2))

Run it with two endpoints configured:

export PRIMARY_URL='https://primary.example'
export PRIMARY_KEY='primary-key'
export FALLBACK_URL='https://fallback.example'
export FALLBACK_KEY='fallback-key'
export MODEL='your-free-model'
python fallback_harness.py

The script sends the prompt to the primary endpoint. If the primary returns a non-200 status or raises a network error, it routes to the fallback and retries there.

Expected output

Force the primary endpoint to fail by setting PRIMARY_URL='http://127.0.0.1:9' (a port that refuses connections). The expected result looks like this:

{'ok': True, 'text': 'ready', 'prompt_tokens': 4, 'completion_tokens': 1, 'seconds': 1.1, 'status': 200, 'decisions': [('primary', False, None, 'ConnectionError: ...'), ('fallback', True, 200, None)]}

The important part is decisions: it shows the primary failed and the fallback succeeded. If you remove that field, the fallback is invisible.

Decision table

Observed condition	Harness action	Why it matters
Primary returns 200 with content	No fallback	Fast path, no duplicate prompt
Primary returns 401 or 403	Fallback	A bad key should not consume retries forever
Primary returns 429	Fallback	Rate limits are a common real failure
Primary times out	Fallback	Network failure is different from model refusal
Fallback also fails	Stop and return error	Retrying forever burns tokens

This table is deliberately small. The harness does not implement every branch; it gives you a visible state machine to reason about.

Common mistakes

• Putting /v1 in PRIMARY_URL and appending /v1/chat/completions, producing /v1/v1/chat/completions.
• Setting timeout=0.1, which turns slow models into false failures.
• Assuming the response always contains usage; some compatible endpoints omit it.
• Reading only the fallback output and never checking whether the primary request actually failed.

What you should understand

After completing this, you should see fallback logic as three states: primary -> fallback -> stop. The harness makes the transition visible and records token usage per attempt.

That small observation is the real skill. Production routers add retry budgets, circuit breakers, and backoff, but if you cannot observe a single fallback transition, those features are just extra code.

Limitations

• The harness is not a load test or a production router.
• Token allowance, free server, and availability are operator-supplied claims; verify them before relying on them.
• The code does not validate model names or enforce quota.
• It does not handle streaming responses or cancellation.

Who should not use this approach

Skip this if you need an approved provider list, security review for production, or multi-tenant quota enforcement. A tiny observer helps you learn failure behavior; it does not replace an operations policy.

Extension exercise

Add a circuit breaker that stops all calls after two fallback failures, then reload the process to confirm the breaker resets. Record the prompt_tokens and completion_tokens for each attempt to a CSV and draw the cost curve for a failing primary.

If you already have a MonkeyCode endpoint, use the free token allowance to run the failure fixtures against two configured bases. If not, any two OpenAI-compatible test endpoints will teach the same state machine.