Free Model Endpoints: A Myth-Busting FAQ With a Probe Script

Developers repeat claims about free model endpoints. Most of those claims are wrong. I know because I tested them.

This is a myth-busting FAQ, not a review. I wrote a small probe script. Then I ran it against a real free endpoint.

The endpoint was MonkeyCode's free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

How many of these myths have you repeated this week? Here are five. Each one gets a claim, evidence, and a corrected mental model.

The five myths

Myth 1: Latency is the only number that matters

The claim: "The endpoint feels fast, so it's healthy."

Why it sounds true: Speed is the easiest thing to measure. A slow endpoint is painful. A fast one feels like progress.

What the probe catches: I send the same prompt ten times. The script prints median latency and correctness. Fast responses can be empty, truncated, or just wrong. What good is a fast answer if it's wrong?

The corrected model: Measure latency, correctness, and throughput separately. One number hides the other two.

Myth 2: The token count in the response is exact

The claim: "The usage field tells me what I spent."

Why it sounds true: The API returns a number. Numbers look precise. Precision feels like truth.

What the probe catches: Tokenizers differ between providers and clients. Some endpoints omit usage fields. Others return approximations. The script compares local counts against the API count. Trusting one number broke my cost estimates.

The corrected model: Count tokens with your own tokenizer. Track input and output separately. Never trust a single usage field.

Myth 3: Retrying a failed call is harmless

The claim: "I'll just wrap it in a retry loop."

Why it sounds true: Retries fix transient failures. That's true for read-only calls. It's false for anything with side effects.

What the probe catches: Retries multiply load. They turn a small burst into an outage. Worse, they duplicate side effects. A retried call can send two emails or insert two rows. How many side effects can one retry create?

The corrected model: Use idempotency keys. Cap your retry budget. Use exponential backoff with jitter.

Myth 4: max_tokens controls your cost

The claim: "Lower max_tokens means cheaper calls."

Why it sounds true: Output tokens are what you see. So they must be what you pay for. Usually, they're not.

What the probe catches: Input tokens usually dominate. A long system prompt costs more than a long answer. Conversation history grows fast. Caching changes the math again. The script prints the input/output ratio.

The corrected model: Trim context, not just max_tokens. Measure the input/output ratio. Cache aggressively when the endpoint supports it.

Myth 5: A working demo means a working integration

The claim: "It worked in my notebook. Ship it."

Why it sounds true: A demo proves the API works. It proves the happy path. Production is not the happy path.

What the probe catches: Demos are short and sequential. Real workloads are concurrent and long. Free endpoints behave differently under concurrency. The script fires ten parallel calls and prints the damage.

The corrected model: Load-test with your real prompt distribution. Test concurrency before you commit.

The probe script

Here's the script I used. It's stdlib-only. No dependencies. Run it against any OpenAI-compatible endpoint.

# myth_probe.py — test five free-endpoint myths with stdlib only
# Usage: python myth_probe.py ENDPOINT MODEL TOKEN
import concurrent.futures as cf
import hashlib, json, statistics, sys, time, urllib.request

endpoint, model, token = sys.argv[1], sys.argv[2], sys.argv[3]
headers = {"Content-Type": "application/json", "Authorization": f"Bearer {token}"}

def call(messages, max_tokens=50, idem=None, timeout=30):
    payload = {"model": model, "messages": messages, "max_tokens": max_tokens}
    if idem:
        payload["idempotency_key"] = idem
    req = urllib.request.Request(endpoint, data=json.dumps(payload).encode(), headers=headers)
    t0 = time.time()
    try:
        with urllib.request.urlopen(req, timeout=timeout) as r:
            return r.status, json.load(r), time.time() - t0
    except Exception as e:
        return None, {"error": str(e)}, time.time() - t0

def text_of(body):
    return body.get("choices", [{}])[0].get("message", {}).get("content", "")

# Myth 1: latency vs correctness
lats, ok = [], 0
for _ in range(10):
    _, body, dt = call([{"role": "user", "content": "Reply with exactly: PONG"}], max_tokens=5)
    lats.append(dt)
    ok += int(text_of(body).strip().upper() == "PONG")
print(f"[myth1] median_latency={statistics.median(lats):.2f}s correctness={ok}/10")

# Myth 2: local count vs API usage
_, body, _ = call([{"role": "user", "content": "Count this sentence. " * 20}], max_tokens=20)
local = len(text_of(body).split())
api = (body.get("usage") or {}).get("completion_tokens", "missing")
print(f"[myth2] local_words={local} api_completion_tokens={api}")

# Myth 3: retries with idempotency keys
for i in range(3):
    key = hashlib.sha256(f"job-{i}".encode()).hexdigest()
    status, body, dt = call([{"role": "user", "content": "Reply with: DONE"}], idem=key)
    print(f"[myth3] attempt={i} status={status} idem={key[:8]} dt={dt:.2f}s")

# Myth 4: input vs output token ratio
_, body, _ = call([{"role": "system", "content": "You are a terse assistant. " * 10},
                   {"role": "user", "content": "Summarize in one line."}], max_tokens=100)
usage = body.get("usage") or {}
print(f"[myth4] input_tokens={usage.get('prompt_tokens', '?')} output_tokens={usage.get('completion_tokens', '?')}")

# Myth 5: concurrency behavior
def one(_):
    _, b, dt = call([{"role": "user", "content": "Reply with: OK"}], max_tokens=5)
    return dt, text_of(b).strip()

with cf.ThreadPoolExecutor(max_workers=10) as ex:
    results = list(ex.map(one, range(10)))
print(f"[myth5] parallel_latency_median={statistics.median(r[0] for r in results):.2f}s ok={sum(1 for r in results if r[1] == 'OK')}/10")

How to read the output

Run each probe three times. Free endpoints change by the hour. One run is a sample, not a verdict.

Probe	What it tells you	Red flag
myth1	Latency vs correctness	Fast median, low correctness
myth2	Token accounting accuracy	usage missing or far from local count
myth3	Retry and idempotency behavior	Status flapping, high dt
myth4	Input/output ratio	Input far bigger than output
myth5	Concurrency headroom	Latency jumps, wrong outputs

Who should not use this

• You need an SLA. A free endpoint won't give you one. This script won't fix that.
• You send one request per minute. The concurrency probe is overkill.
• You have no ground truth. The probe can't judge output you can't verify.

Limitations

• The script tests one endpoint at a time. It doesn't test streaming. That needs a separate probe.
• Token counts are approximate by design. Words are not tokens. This is a heuristic, not a tokenizer.
• Free endpoints change. Treat every result as dated. Re-run the probe weekly.

The corrected mental model

Free endpoints are not magic. They're not scams either. They're systems with different trade-offs.

Latency is not health. Usage fields are not truth. Retries are not free. max_tokens is not a budget. Demos are not proof.

Still think free means flaky? Run the probe. Build your mental model from evidence.

If you probe a free endpoint anyway, the script is yours. Fork it, break it, and tell me what you find.