Write the Fallback Path Before You Choose Free Model Access

Here is my conclusion before any comparison table: the question "free, paid, or self-hosted?" is really the question "what does my code do during the thirty seconds when the answer is no?" I have watched teams argue over per-million-token pricing for a week while their retry logic was a bare while True loop with no ceiling and no fallback. Pick the failure behaviour first, and the seat choice becomes almost mechanical.

Free model access is attractive for exactly the reason it is awkward: the capacity is not yours, so you cannot promise anything downstream about it. That is not a defect in any particular vendor, it is a property of anything you do not pay to reserve. The engineering response is not to distrust free tiers, but to make a throttled primary a normal branch in your code instead of an incident report.

I am using MonkeyCode here as the free end of this comparison, since its free model access and its free server option are the two things I can point at without guessing.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. Quotas, hardware and terms change over time, so verify the current offer yourself before you build a plan on it; this post relies only on the operator-supplied claims that free model access and a free server option exist.

Three questions that beat any price table

Before you open a pricing page, write down answers to three questions and put them in the repo, because they decide this faster than any spreadsheet. Who is on the hook at 2:00 a.m. when the endpoint refuses you? What does the user see when a request is throttled — a spinner, a degraded answer, or a refund request? How long can the queued work survive a cold stretch of capacity before the answer stops being useful?

If your honest answers are "me, a degraded answer, and about thirty seconds", you are describing a workload that tolerates free capacity as long as a fallback exists. If they are "a customer, an outage, and never", you are describing a workload that needs a reserved commitment, and no free tier will rescue that. The comparison table is downstream of these answers, not a substitute for them.

Step 1 — Profile the workload in numbers you can defend

Write a small file before you compare anything, so that every later claim has something to be checked against. Mine usually looks like this workload.yaml.

workload: support-triage
calls_per_minute_peak: 40
median_prompt_tokens: 1800
max_prompt_tokens: 9000
latency_budget_ms: 2500
retries_allowed: 2
fallback: deterministic-rules        # must already exist before the primary dies
data_sensitivity: internal
must_run_offline: false

The peak rate sets the ceiling for your limiter, so measure it rather than naming a round number you like. The fallback line is the important one: if it names something you have not built yet, that is your first task, not your last. And data_sensitivity quietly decides whether self-hosting is a lifestyle preference or a hard constraint.

Step 2 — Put an admission controller in front of every call

This wrapper is the real deliverable, and it is small enough to write in an afternoon. It shapes your own traffic with a token bucket, retries with jitter, and returns which path served the request so you can count it later.

# admission.py
from __future__ import annotations

import random
import threading
import time
from dataclasses import dataclass, field


class Retryable(Exception):
    """Raised when the primary refuses us; safe to retry elsewhere."""


@dataclass
class TokenBucket:
    rate: float                     # permits per second, kept below your observed limit
    burst: float                    # how large a spike you are willing to queue
    tokens: float = field(init=False)
    lock: threading.Lock = field(default_factory=threading.Lock)

    def __post_init__(self) -> None:
        self.tokens = self.burst
        self._last = time.monotonic()

    def take(self, n: float = 1.0, timeout: float = 5.0) -> bool:
        deadline = time.monotonic() + timeout
        while True:
            with self.lock:
                now = time.monotonic()
                self.tokens = min(self.burst, self.tokens + (now - self._last) * self.rate)
                self._last = now
                if self.tokens >= n:
                    self.tokens -= n
                    return True
            if time.monotonic() >= deadline:
                return False
            time.sleep(random.uniform(0.05, 0.15))


def complete(prompt, *, primary, fallback, bucket, attempts=3):
    for attempt in range(attempts):
        if not bucket.take():
            break
        try:
            return primary(prompt), "primary"
        except Retryable:
            time.sleep((2 ** attempt) + random.random())
    return fallback(prompt), "fallback"

Notice what the caller never does: it never raises to the user, and it never waits forever on one endpoint. The bucket protects the primary from your own traffic spike, and its rate should be set conservatively below whatever limit you actually observe, because a guess is not a contract. The tuple return is deliberate — label every response with "primary" or "fallback" and you get a real signal about how much of your product is running on the free path.

Step 3 — Force the fallback with a chaos test

A fallback path that has never executed is a comment, so test it before you trust it in production. This test is short and it catches the exact bug that hurts: a fallback that is technically wired up but unreachable because an exception escapes first.

# tests/test_admission.py
from admission import Retryable, TokenBucket, complete


def test_fallback_is_reachable():
    calls = {"primary": 0}

    def primary(prompt):
        calls["primary"] += 1
        raise Retryable("throttled")

    def fallback(prompt):
        return "rules-only answer"

    bucket = TokenBucket(rate=100, burst=10)
    text, path = complete("hi", primary=primary, fallback=fallback, bucket=bucket)

    assert path == "fallback"
    assert text == "rules-only answer"
    assert calls["primary"] == 3

Run it with python -m pytest -q tests/test_admission.py and add one more assertion afterwards: that the bucket itself returns False after you drain it, so a runaway loop cannot hammer anything. Then, in staging, emit a counter with the path label and alarm on a fallback rate you would not accept; five percent might be normal, forty percent means you mis-sized the bucket.

Step 4 — Score the three options against your own profile

Now the comparison is boring, which is the point. Map each line of your workload.yaml onto one of three columns and keep the cells honest.

Signal from your profile	Free model access (e.g. MonkeyCode)	Paid managed API	Self-hosted
Who fixes a bad hour	You, through the fallback you wrote	Vendor on-call plus you	You, on your own box
Cost shape	Zero to start, wrapper is the investment	Metered and forecastable	Fixed hardware, spiky attention
Peak-rate ceiling	Whatever your own bucket allows	Raised by buying quota	Whatever your hardware allows
Sensitive data	Check current terms before sending anything	Contract terms	Stays inside your boundary
What you must build	Retries, fallback, path telemetry	Retries and telemetry	All of the above plus serving
Fits well when	Prototypes, internal tools, batch jobs that can wait	Customer-facing paths with a latency promise	Hard data-boundary or offline needs

The column that decides most real cases is the last lazy one: what you must build. Choosing free access is not choosing zero work, it is choosing to spend the work on resilience instead of on invoices, and that trade is worth it precisely when the fallback answer is acceptable to the user. Choosing self-hosting moves the same work to capacity planning and upgrades, which is a different job with a longer horizon.

Limitations and who should skip this

I would not run this pattern on a path where a degraded answer breaks a promise, such as billing actions or anything a customer paid for a specific latency on. I also would not recommend it to a team with no one available to own the retry budget, because an unowned limiter becomes a silent throttle that nobody notices until a launch. And if your data cannot leave a boundary you control, the free and paid columns are simply off the table, regardless of how good the wrapper is.

None of this depends on a number I cannot verify today, which is deliberate: capacity offers, quotas and machine specifications change, and a plan built on a stale figure is worse than no plan. What does not change is the shape of the code that survives a refusal.

If you want to try the free end of this comparison, MonkeyCode's project is open source and its free model access plus free server option are the parts worth testing first — bring the wrapper, and let the fallback path tell you whether the fit is real.