The Free-Model Canary: 60 Lines Before I Trust a Free AI Server

Last month I watched a teammate point our internal CLI at a free model endpoint. The env var looked harmless: LLM_BASE_URL. The first request returned a clean answer, so the build moved on.

Two days later the CLI started returning empty strings in the afternoon. No error, no timeout, just blank responses. The endpoint hadn't changed. Our assumptions about it had.

That's the real problem with free model servers: they come with a contract you haven't read. The contract lives in behavior, not in the README. When AI makes code cheap, the dependency surface gets wider, and every new endpoint becomes a piece of technical debt.

One project I've been checking with that mindset is MonkeyCode. It's an open-source tool that gives solo builders free models and a free server, which is a useful combination for a side project that shouldn't cost money to run. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I'm not going to repeat a token allowance or uptime figure from its README, because I haven't benchmarked it yet. That's what the canary below is for.

Why a canary, not a benchmark

A benchmark tells you how fast a server is. A canary tells you whether you can depend on it at all. It makes six small assertions about the way an OpenAI-compatible endpoint should behave, and it fails loudly when a free server breaks the rules.

I run something like this against every new endpoint before I wire it into a project. The script is deliberately small because I want it used. A 200-line load test gets skipped; a 60-line smoke test gets committed.

The checks are boring on purpose. Bad auth should return 401. Empty messages should return 4xx. A valid prompt should return content. Quota headers should be visible. A slow worst-case call should stay under a threshold you choose. These are not performance goals; they are dependency boundaries.

The 60-line canary

Save this as free_model_canary.py:

#!/usr/bin/env python3
'''free_model_canary.py: smoke-test an OpenAI-compatible API before you build on it.'''

import sys
import time

import httpx

ENDPOINT = sys.argv[1]
API_KEY = sys.argv[2]
MODEL = sys.argv[3]
BASE = ENDPOINT.rsplit('/chat/completions', 1)[0]


def post(client, payload, headers):
    return client.post(ENDPOINT, json=payload, headers=headers, timeout=15)


def main():
    failures = []

    with httpx.Client() as client:
        # 1. A bad key must fail with 401, not hang or silently succeed.
        bad_key_headers = {'Authorization': 'Bearer definitely-wrong'}
        bad = post(
            client,
            {'model': MODEL, 'messages': [{'role': 'user', 'content': 'hi'}]},
            bad_key_headers,
        )
        if bad.status_code != 401:
            failures.append(f'auth: expected 401, got {bad.status_code}')

        # 2. An empty message list must be rejected.
        headers = {'Authorization': f'Bearer {API_KEY}'}
        empty = post(client, {'model': MODEL, 'messages': []}, headers)
        if empty.status_code < 400:
            failures.append(f'empty payload: expected 4xx, got {empty.status_code}')

        # 3. A minimal valid request must return content.
        ok = post(
            client,
            {
                'model': MODEL,
                'messages': [{'role': 'user', 'content': 'Reply with exactly BLOOP.'}],
                'max_tokens': 10,
            },
            headers,
        )
        if ok.status_code != 200:
            failures.append(f'valid request: expected 200, got {ok.status_code}')
        else:
            text = ok.json()['choices'][0]['message']['content'].strip()
            if 'BLOOP' not in text:
                failures.append(f'content: expected BLOOP, got {text!r}')

        # 4. Quota headers should be visible so you can watch your budget.
        if not any('ratelimit' in key.lower() for key in ok.headers):
            failures.append('quota headers: no ratelimit header found')

        # 5. A quick latency sample: the slowest of 5 tiny calls stays under 3s.
        samples = []
        for _ in range(5):
            start = time.perf_counter()
            post(
                client,
                {'model': MODEL, 'messages': [{'role': 'user', 'content': 'hi'}], 'max_tokens': 1},
                headers,
            )
            samples.append((time.perf_counter() - start) * 1000)
        worst_ms = max(samples)
        print(f'latency sample (ms): {[round(x) for x in samples]}, worst={round(worst_ms)}')
        if worst_ms > 3000:
            failures.append(f'latency: worst {round(worst_ms)}ms exceeds 3000ms')

        # 6. The model list endpoint should be reachable.
        try:
            models = client.get(f'{BASE}/models', headers=headers, timeout=10)
            if models.status_code != 200:
                failures.append(f'model list: expected 200, got {models.status_code}')
        except httpx.HTTPError as exc:
            failures.append(f'model list: {exc}')

    if failures:
        print('CANARY FAIL')
        for failure in failures:
            print(' -', failure)
        sys.exit(1)

    print('CANARY PASS')


if __name__ == '__main__':
    main()

What the checks catch

The auth check catches the most embarrassing failure mode: a server that accepts any key and logs nothing. If your bad-key request returns 200, your logs will be full of phantom users.

The empty-payload check validates the server's basic contract. An OpenAI-compatible endpoint should reject a message list with nothing in it. Some free servers skip this validation and store the payload, which only delays the problem.

The content check looks for a marker token. Asking the model to reply with exactly BLOOP gives you a fixed string to compare. This catches servers that return an empty choices array, or that stream content when you asked for a single response.

The quota-header check is about observability. Even a rough x-ratelimit-remaining value is enough to schedule your background jobs around the wall. No header means the wall is invisible.

The latency sample is a sanity floor, not a benchmark. Five tiny requests are enough to catch a server that is swapping models or cold-starting containers. It can't tell you about tail latency at scale.

The model-list check confirms the server knows what it is serving. A broken /models endpoint means the API surface is unstable, and your code will chase that instability later.

These six checks take about thirty seconds. For that time you get a written record of what the free model server actually promised you, instead of what its landing page promised.

How to run it

Create a clean environment and run it with three arguments:

python -m venv .venv
source .venv/bin/activate
pip install httpx
python free_model_canary.py https://your-server/v1/chat/completions your-key your-model

A passing run looks like this:

latency sample (ms): [320, 410, 288, 522, 1890], worst=1890
CANARY PASS

If the script prints CANARY FAIL, the server is telling you how it will break your build. A common failure shape is an endpoint that passes connection and JSON checks but hides quota headers. That is the quietest way a free tier fails: no warning, no budget counter, just a 429 at the worst moment.

A decision table for free model servers

A pass is a green light, not a blank one. Here's the table I use:

Where the code runs	Trust a free server after canary?	Extra guardrails
Local experiment or prototype	Yes	Keep a disposable API key
Scheduled job that can retry	Yes	Add a dead-letter queue and alarm on empty responses
Production API with external users	No	Pay for an SLA or keep a self-hosted fallback
PII, health, or financial data	Never	No free tier, no exception

The canary checks the server, not your data. If your workflow touches regulated data, this script is not your approval process.

Where the canary lies

The script is a snapshot. It does not prove uptime over a month, fairness during a flash crowd, or the durability of your prompts. It also cannot tell you whether a free server will change its quota policy next week. Free model tiers can be generous at launch and become unusable before a hackathon ends.

That's why the quota-header check matters. A server that exposes x-ratelimit-remaining is giving you a fighting chance to detect a drain before it becomes a failure. A server that hides the limit is a background surprise.

Who should not use free model servers

If you are building something that people depend on while they sleep, a free server is a dependency without a contract. Run the canary, learn what the endpoint can do, then buy the thing you intend to depend on.

Free model access is a tool for discovery, not a substrate for a product. MonkeyCode's free models and free server make sense for prototypes, internal CLIs, and weekend experiments. My rule is simple: free is fine until the user becomes someone other than me.

If you run this canary against a free model endpoint and it fails on the quota-header check, tell me which headers the server returned. That missing field is the next thing I want to encode in the script.