A Free Model Allowance Is Not an Architecture

Free model access is genuinely useful. Building your stack around it is not.

Every free tier ends differently. Some shrink, some change shape, some just get slower. You cannot predict which one you get. You can make the switch boring.

That is the whole argument. Portability is the part of a free tier you keep. Everything else is rented.

The claim, stated plainly

A free token allowance is a resource, not a design decision. It pays for calls. It does not pay for the coupling you build on top.

Here is the test. If moving to a second endpoint takes a sprint, you never had a free tier. You had a discount with a lock.

I am not arguing against free tiers. I am arguing against spending them before you know your exit.

What coupling looks like in practice

Nobody couples on purpose. It arrives through small conveniences.

• A vendor SDK type appears in your function signature.
• Tool-call schemas get stored in your database, not converted at the edge.
• Retry and streaming logic assumes one response envelope.
• Prompt templates get tuned to one model's quirks and nothing else.
• Your test suite is only ever green against a single host.

Each item saves an hour now. Each one costs a week later.

The artifact: one adapter, two modes

The fix is a single call shape with two implementations. One talks to the network. One reads from disk.

# gateway.py — sketch only; not yet executed against a live endpoint
from __future__ import annotations

import dataclasses
import hashlib
import json
import os
import pathlib
from typing import Protocol

CASSETTES = pathlib.Path(os.environ.get("CASSETTES", "cassettes"))


@dataclasses.dataclass(frozen=True)
class Call:
    prompt: str
    model: str
    temperature: float = 0.0


def fingerprint(call: Call) -> str:
    blob = json.dumps(dataclasses.asdict(call), sort_keys=True)
    return hashlib.sha256(blob.encode()).hexdigest()[:16]


class Provider(Protocol):
    def complete(self, call: Call) -> str: ...


class Replay:
    """Reads recorded responses. No network. No credentials."""

    def complete(self, call: Call) -> str:
        path = CASSETTES / f"{fingerprint(call)}.json"
        if not path.exists():
            raise FileNotFoundError(f"missing cassette: {path.name}")
        return json.loads(path.read_text())["text"]


class Http:
    """One POST, plain JSON, no vendor SDK in your type signatures."""

    def __init__(self, base_url: str, api_key: str) -> None:
        self.base_url = base_url.rstrip("/")
        self.api_key = api_key

    def complete(self, call: Call) -> str:
        import urllib.request

        body = json.dumps(dataclasses.asdict(call)).encode()
        req = urllib.request.Request(
            f"{self.base_url}/v1/complete",
            data=body,
            headers={
                "Content-Type": "application/json",
                "Authorization": f"Bearer {self.api_key}",
            },
        )
        with urllib.request.urlopen(req, timeout=60) as resp:
            return json.loads(resp.read())["text"]

Selection stays in one place. That place is a function you can argue about in review.

def build_provider(mode: str) -> Provider:
    if mode == "replay":
        return Replay()
    if mode == "live":
        return Http(os.environ["BASE_URL"], os.environ["API_KEY"])
    raise SystemExit(f"unknown mode: {mode}")

Two tests keep the design honest. Both run offline and cost nothing.

# test_portability.py
import pytest
from gateway import Call, Replay, fingerprint


def test_fingerprint_is_stable():
    a = Call(prompt="summarise README", model="any")
    b = Call(prompt="summarise README", model="any")
    assert fingerprint(a) == fingerprint(b)


def test_replay_never_touches_network():
    with pytest.raises(FileNotFoundError):
        Replay().complete(Call(prompt="never recorded", model="any"))

The second test is the important one. It fails loudly when someone sneaks a network call into a replay path.

The drill: prove you can switch in under an hour

Run this once per quarter. Write the result in your repository README.

1. Record ten real prompts against your primary endpoint. Store cassettes in git.
2. Run the suite in replay mode with networking disabled. Everything must pass.
3. Point the same suite at a second endpoint. Change no test files.
4. Diff the outputs into a report. Report differences; do not score them yet.
5. Count the lines you changed to switch. That number is your coupling score.
6. Time the whole exercise. Anything over an hour is a backlog item.

pip install -e .
CASSETTES=cassettes python -m gateway --record --suite suites/core.yaml
CASSETTES=cassettes python -m gateway --replay --suite suites/core.yaml

One honest note. The --record and --replay flags are the interface I would want, not a CLI I have run. Wire them to your own runner.

A decision table you can argue with

Situation	Free allowance	Free server option	Paid endpoint
Learning an agent loop	Yes	Yes	No
Recording replay cassettes	Yes	Yes	Only if blocked
Per-commit CI runs	Maybe	Yes, as your runner	Depends on budget
Latency-sensitive traffic	No	No	Yes
Regulated data	No	No	Approved vendors only

Read the table as a default, not a law. The point is that "free" appears in one column, never in your architecture.

Where MonkeyCode fits

MonkeyCode is an open-source project offering free model access and a free server option. It is a good second endpoint for the drill, because a second endpoint is exactly what the drill needs.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

I am not quoting a quota or a server spec here. Those numbers change, and an article is a bad place to pin them. Check the project page for current terms before you plan around them.

The genuine fit is narrow and useful. Keep your primary endpoint where it is. Point the adapter at MonkeyCode, run the replay suite, and measure the diff and the lines changed. If the switch is boring, the free tier did its job.

The free server option covers a different gap. Run the recording proxy and the cassette store there, so the drill costs you nothing while you decide.

Limitations

• Replay proves reproducibility, not quality. A cassette says what happened, not what is correct.
• Cassettes bind to exact prompts. Edit a prompt and you re-record.
• A second endpoint passing your suite does not mean the endpoints are equivalent.
• Free infrastructure is not a benchmark target. Do not publish latency numbers from it.
• This drill detects coupling. It does not remove coupling you already shipped.

Who should not use this

Skip the drill if switching providers is contractually prohibited for you. Skip it if you have a single endpoint and no plausible reason to move. Skip it if your prompts are already versioned and tested, because you have solved this problem.

Also skip it if you want a free tier to be a permanent plan. Free tiers are not permanent. That is the entire reason this article exists.

The close

Write the exit plan first, then spend the allowance. Point the adapter at the free endpoint and run the drill once. If the switch takes an afternoon, you learned something cheap.