Ship a Failover Lane for AI Calls in One Weekend (And Skip the Rest)

Saturday, 07:40. My side project's "summarize thread" button returned a 429.

My retry loop made it worse. Every retry burned the same quota I was already out of.

By 08:30 I had a decision to make. Build a full AI gateway, or ship one honest thing by Sunday night.

I shipped the one honest thing. A failover lane with a circuit breaker, a classifier for retryable errors, and five tests that run against fake upstreams. No dashboard. No router config language. No Kubernetes.

Here is the build log: the scope I kept, the code, the tests, and everything I deliberately skipped.

Step 1: Cut the scope before you write code

A weekend dies from ambition, not from bugs. So I wrote the cut list first.

Kept: one breaker per provider, retryable-error classification, a half-open probe, and a test harness. Skipped: distributed state, metrics export, prompt versioning, and cost accounting.

The skipped list is not a roadmap. It is a list of problems I do not have yet. One process, one API key, one user: that is my whole deployment.

Step 2: Decide what counts as a failure

This is the part most side projects get wrong. A 400 from a bad prompt is not an outage. Failing over on it just hides your bug behind another provider.

So classify first. Only retryable failures should move traffic.

# lanes.py
RETRYABLE_STATUS = {408, 409, 425, 429, 500, 502, 503, 504}


def is_retryable(exc: Exception) -> bool:
    status = getattr(exc, "status_code", None)
    if status is None:
        # Network errors, timeouts, connection resets.
        return isinstance(exc, (TimeoutError, ConnectionError))
    return status in RETRYABLE_STATUS

401 and 403 are configuration bugs. 400 and 422 are prompt bugs. Send those to your logs, not to the fallback lane.

Step 3: Write the breaker (about 40 lines)

The breaker has three states. Closed means primary. Open means fallback only. Half-open means one probe gets through.

# breaker.py
import time
from lanes import is_retryable


class Breaker:
    def __init__(self, fail_threshold=3, cooldown_s=20.0):
        self.fail_threshold = fail_threshold
        self.cooldown_s = cooldown_s
        self.fails = 0
        self.opened_at = None

    def state(self):
        if self.opened_at is None:
            return "closed"
        if time.monotonic() - self.opened_at >= self.cooldown_s:
            return "half-open"
        return "open"

    def _trip(self):
        self.opened_at = time.monotonic()
        self.fails = 0

    def _reset(self):
        self.opened_at = None
        self.fails = 0

    def call(self, primary, fallback):
        state = self.state()
        if state == "open":
            return fallback(), "fallback_open"

        if state == "half-open":
            try:
                out = primary()
            except Exception as exc:
                if not is_retryable(exc):
                    raise
                self._trip()
                return fallback(), "fallback_probe_failed"
            self._reset()
            return out, "primary_recovered"

        try:
            out = primary()
        except Exception as exc:
            if not is_retryable(exc):
                raise
            self.fails += 1
            if self.fails >= self.fail_threshold:
                self._trip()
            return fallback(), "fallback_error"

        self._reset()
        return out, "primary"

The second return value is the only observability I allowed myself. It tells you which lane answered, and that is enough for a weekend.

Step 4: Test it with fake upstreams, not real ones

You cannot debug a breaker against a flaky provider. You can debug it against a fake that fails on command.

Five scenarios cover almost everything I care about:

1. Primary succeeds, breaker stays closed.
2. Primary raises 429 three times, breaker opens, fallback answers.
3. Primary raises 400, no failover, error surfaces to caller.
4. Breaker is open, primary is never called at all.
5. Cooldown elapses, probe succeeds, breaker closes again.

# test_breaker.py
import pytest
from breaker import Breaker


class Boom(Exception):
    def __init__(self, status_code):
        self.status_code = status_code


def primary_raises(status):
    def fn():
        raise Boom(status)
    return fn


def test_non_retryable_does_not_failover():
    b = Breaker(fail_threshold=1)
    with pytest.raises(Boom):
        b.call(primary_raises(400), lambda: "fallback")
    assert b.state() == "closed"


def test_opens_after_threshold():
    b = Breaker(fail_threshold=3)
    for _ in range(3):
        out, lane = b.call(primary_raises(429), lambda: "fallback")
        assert out == "fallback"
    assert b.state() == "open"


def test_open_skips_primary():
    b = Breaker(fail_threshold=1)
    b.call(primary_raises(503), lambda: "fallback")
    calls = []

    def spy():
        calls.append(1)
        return "primary"

    out, lane = b.call(spy, lambda: "fallback")
    assert lane == "fallback_open"
    assert calls == []

Scenario five needs a short cooldown. Set cooldown_s=0.05 and the half-open test runs in milliseconds.

Run it: pytest -q test_breaker.py. Five green tests, one afternoon, no API key.

Step 5: Pick a lane you can actually afford to leave on

A fallback lane only helps if it is cheap and available during someone else's outage. That rules out a second premium key for a hobby project.

This is where MonkeyCode fit my build. The operator states that the project offers free model access and a free server option, and that it is open source; I used it as the fallback lane and as the place to run the cron job. Treat those as operator-supplied claims, not as measured results. As of this writing the operator also states a free token allowance in the millions, so check the current number on the project's own pricing page before you size anything around it. Allowances change.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The important design point is not the vendor. It is that your fallback must be a different failure domain from your primary. Two keys in the same account are one outage, not two.

What I skipped, and why it was fine

I skipped distributed state. The breaker lives in one process, so a deploy resets it. For a side project that runs one worker, a reset is a non-event.

I skipped per-endpoint breakers. My app has one model call path. Adding granularity before I had a second path would have been speculative abstraction.

I skipped metrics. The lane string already goes into my request log, and I read that log twice a week.

I also skipped prompt-level retries. If the model returns a wrong answer, a breaker will not save you. That is an eval problem, and it is next weekend's problem.

Should you build this?

Your situation	Build it?	Reason
One side project, one model call path	Yes	40 lines beats a gateway
Multiple processes behind a load balancer	Not yet	In-memory state is per-process
Fallback model quality is untested	Not yet	Fast bad answers are still bad
Hard data-residency requirements	Probably not	Picking a lane is a compliance decision
You already have retries with backoff	Maybe	Check what your retries do on 429

Limitations, stated plainly

Free lanes have their own rate limits. A breaker does not buy you capacity.

Half-open sends one probe. Under real concurrency you may want a token bucket there instead.

And an in-memory breaker forgets everything on restart. If you need it to survive deploys, you need shared state, and that is a different weekend.

Who should not use this

Skip this approach if your fallback model has never passed your own eval set. Skip it if a regulator, not an engineer, decides where your data goes. Skip it if you are already at high request volume, because a single-process breaker will flap under load.

If you want to try the lane I used, start with a throwaway key and the five tests above. Get them green before you point real traffic at anything.

Sunday, 23:10. The button works, the tests pass, and the only new file is 40 lines long.

The scope cut is what shipped the feature. Not the model.