Prove a Free-Tier Model Fallback Under Load Before Your Queue Depends on It

02:14 on a Sunday. Your primary model endpoint returns 429s. The queue age crosses ninety seconds. The runbook says: fall back to the secondary model. Nobody ever tested that path.

This week's model news will be obsolete in a month. Your fallback drill will not. This tutorial builds a working fallback from zero. You will use MonkeyCode's free model access and free server option. The whole drill takes about an hour.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

What you are building

Here is the topology. Your primary endpoint handles normal traffic. A fallback proxy sits on a free server. It forwards requests to MonkeyCode's free model access. A load-test tool beats on the proxy from your laptop.

laptop (hey) --> free server :8080 (fallback_proxy.py) --> MonkeyCode free model endpoint

MonkeyCode is an open-source project. Its free tier covers model access and a free server option. MonkeyCode currently advertises a free tier with 10M tokens. I will not quote server specs. They change. Check the dashboard before you plan capacity.

Why a free server? Cost is zero. The box is disposable. You can delete it after the drill. That removes the fear of breaking a shared staging environment.

Stage 1: Provision the server and verify the endpoint

Provision the free server from the MonkeyCode dashboard. You get SSH access to a disposable Linux box. Treat it as throwaway. No production data. No secrets.

The free server is not a production host. Do not install your monitoring agent on it. Do not join it to your cluster. Isolation keeps the drill safe.

Verify the server:

ssh user@<server-ip> 'uname -a && nproc'

Expected: a Linux kernel line and a CPU count.

Verify the model endpoint from your laptop:

export MONKEY_BASE_URL="https://<base-url-from-dashboard>"
export MONKEY_KEY="<key-from-dashboard>"

curl -s "$MONKEY_BASE_URL/v1/chat/completions" \
  -H "Authorization: Bearer $MONKEY_KEY" \
  -H "Content-Type: application/json" \
  -d '{"messages":[{"role":"user","content":"Reply with OK"}],"max_tokens":8}'

Expected: a JSON completion with a short reply. If the path differs, check the dashboard docs. If you see an auth error, stop. Fix the key before you build anything.

Stage 2: Deploy the fallback proxy

SSH into the free server. Install the Python dependencies.

ssh user@<server-ip>
python3 -m venv venv
source venv/bin/activate
pip install fastapi uvicorn httpx

Create fallback_proxy.py. It is a pass-through with a timeout and a health check.

import os
import time

import httpx
from fastapi import FastAPI, Request
from fastapi.responses import JSONResponse

app = FastAPI()
UPSTREAM = os.environ["UPSTREAM_URL"]
KEY = os.environ["UPSTREAM_KEY"]
client = httpx.AsyncClient(timeout=10.0)


@app.get("/health")
async def health():
    return {"status": "ok"}


@app.post("/v1/chat/completions")
async def chat(request: Request):
    body = await request.json()
    start = time.monotonic()
    try:
        resp = await client.post(
            f"{UPSTREAM}/v1/chat/completions",
            headers={"Authorization": f"Bearer {KEY}"},
            json=body,
        )
        latency_ms = (time.monotonic() - start) * 1000
        return JSONResponse(
            content=resp.json(),
            status_code=resp.status_code,
            headers={"x-fallback-latency-ms": str(round(latency_ms, 1))},
        )
    except httpx.TimeoutException:
        return JSONResponse({"error": "upstream timeout"}, status_code=503)
    except httpx.HTTPError:
        return JSONResponse({"error": "upstream unreachable"}, status_code=503)

Run it:

export UPSTREAM_URL="$MONKEY_BASE_URL"
export UPSTREAM_KEY="$MONKEY_KEY"
uvicorn fallback_proxy:app --host 0.0.0.0 --port 8080

Verify health from your laptop:

curl -s http://<server-ip>:8080/health

Expected: {"status":"ok"}.

Verify a real completion:

curl -s -X POST http://<server-ip>:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"messages":[{"role":"user","content":"Reply with OK"}],"max_tokens":8}'

Expected: the same JSON as Stage 1, plus an x-fallback-latency-ms header.

The timeout matters. Ten seconds is a guess. Tune it to your deadline slack. If the upstream is slower than your deadline, fail fast.

Stage 3: Load-test the fallback

Declared workload: 200 requests, 20 concurrent, one short prompt. That is enough to expose queueing and timeout problems.

Install hey on your laptop:

go install github.com/rakyll/hey@latest

Run the test:

hey -n 200 -c 20 -m POST \
  -H "Content-Type: application/json" \
  -d '{"messages":[{"role":"user","content":"Reply with OK"}],"max_tokens":8}' \
  http://<server-ip>:8080/v1/chat/completions

Example output from one run. Your numbers will differ.

Status code distribution:
  [200] 200 responses

Latency distribution:
  50% in 812ms
  95% in 1410ms
  99% in 1890ms

Record the p95. That is your fallback latency budget. If p95 is above five seconds, the fallback is too slow. Reject work instead of queuing it.

Run the test three times. Take the median p95. One run can lie. Three runs give you a number you can defend.

Stage 4: Kill the upstream

A fallback that hangs is worse than none. Test the failure path now. Point the proxy at a dead port:

export UPSTREAM_URL="http://127.0.0.1:9"
export UPSTREAM_KEY="dead"
uvicorn fallback_proxy:app --host 0.0.0.0 --port 8080

Re-run the same hey command. Expected output:

Status code distribution:
  [503] 200 responses

Latency distribution:
  50% in 3ms
  95% in 6ms

Fail fast. That is the behavior you want. A fast 503 lets your admission gate reject new work before the queue eats the SLO.

What can go wrong here:

• The proxy hangs instead of returning 503. Your timeout is too high. Lower it.
• The upstream returns 200 with garbage. Add a response validation.
• The load test hits the free tier quota. Wait or use a smaller prompt.

Stage 5: Wire it into your app with thresholds

Now add the fallback to your app. Use a feature flag. Set thresholds before you flip it.

Signal	Action	Threshold
Primary 429/5xx	Send to fallback	>5% of requests in 2 minutes
Primary p95 latency	Send to fallback	>2s for 2 minutes
Queue age	Send to fallback	>30s
Fallback error rate	Reject new work	>10%
Fallback p95 latency	Reject new work	>5s
Deadline slack	Reject new work	<2s

Keep the integration logic dumb. One function decides. The flag controls it. The thresholds live in config, not code.

if fallback_enabled and should_fallback(queue_age, primary_p95, error_rate):
    resp = httpx.post(fallback_url, json=payload, timeout=5.0)
else:
    resp = httpx.post(primary_url, json=payload, timeout=30.0)

Start with these numbers. Adjust after real traffic. Never send to a failing fallback. That turns a slow incident into a confusing one.

Record these fields in your telemetry: status code, latency_ms, upstream status, queue age. You need them for the retrospective.

Cleanup and rollback

The drill is done. Clean up:

ssh user@<server-ip> 'pkill -f uvicorn'

Delete the free server from the dashboard. Rotate the fallback key. Remove the fallback URL from your app config.

Rollback is one flag flip:

fallback_enabled: false

Redeploy and verify primary-only traffic. Keep the proxy script in your repo. You will need it in the next drill.

Limitations and who should skip this

Free tiers are not SLAs. The 10M-token number can change. Verify it before you plan capacity.

Do not send PII or customer data to a free endpoint. Use this for low-stakes traffic only. Skip this approach if you need guaranteed capacity or hard compliance. Buy a paid tier instead.

Use this if you run a demo, an internal tool, or a low-stakes batch job. Use this if you want a failure drill without a cloud bill.

The proxy is a pass-through. It does not rate-limit. Add a limiter if your queue needs one.

Run the drill this week on a free server. The next 429 will not wait for your diagram to become a test.