Free LLM Tier or Self-Hosted? A 5-Question Framework (and a Probe Script)

The hottest take on DEV this week is that constraints make you better. I mostly agree — but only when you pick the right constraint.

Most of us pick our LLM serving tier by vibes. Free API? Self-hosted? Paid API? That's a decision with real cost, latency, and privacy consequences — and we make it like we're ordering coffee. Sound familiar?

Here's the conclusion up front: you don't need a benchmark suite to decide. You need five questions, one probe script, and about 15 minutes. This post gives you all three.

Why the default answer is wrong

Defaulting to "self-host everything" is a status symbol, not a strategy. Defaulting to "free API for everything" leaks data and breaks at the worst moment. Both defaults are lazy.

I've done both. I've burned a weekend wrestling a 7B model into Docker for a script that ran twice a day. I've also watched a free API rate-limit me mid-demo because I ignored the quota math.

The fix wasn't more discipline. It was a decision framework.

The five questions

Ask them in order. Stop when one gives you a clear answer.

1. Where does the data sleep? Proprietary code, customer data, anything a lawyer would frown at? A third-party API is a hard no. Self-hosted, or a private VPC. Non-negotiable.
2. What's the latency budget? Interactive pair-programming wants sub-second responses. A batch job at 2am tolerates 30 seconds. Free APIs add network hops and queueing; local models add hardware variance.
3. What does the traffic look like? Spiky, small, dev-time? A free quota absorbs that beautifully. Steady 24/7 production? The free tier becomes a liability the moment you hit the limit.
4. Does a small model pass your gate? Run your own 12-prompt quality gate against the free model. Pass? You're done. Fail? No amount of free tokens fixes quality.
5. Who pays the ops tax? Self-hosting costs hours: updates, GPU drivers, monitoring, disk. If your week has zero spare hours, that's a real line item.

The decision matrix

Question	Choose free API	Choose self-hosted
Data residency	Public or generic code	Proprietary, regulated, air-gapped
Latency budget	1–5s tolerated	Sub-second or offline required
Traffic shape	Spiky, low volume, dev-time	Steady, high volume, production
Quality gate	Passes your 20-prompt set	Needs fine-tuning or local RAG
Ops hours	Zero to spare	Budgeted in the roadmap

Notice the pattern: the free tier wins on cost and setup, loses on control. Self-hosting wins on control, loses on your calendar. Neither wins on quality — that's question four's job.

The probe script

Questions are cheap. Evidence is better.

Here's a 45-line probe that runs the same four prompts against two endpoints — your free API and your self-hosted model — then prints a comparison table. It's a smoke test, not a benchmark. That's the point: you want gross mismatches, not statistical significance.

# tier_probe.py — smoke-test a free API against a self-hosted model
import json, os, time, urllib.request

TASKS = [
    ("codegen", "Write a Python retry decorator with exponential backoff and jitter."),
    ("refactor", "Add type hints and rename: def f(x): return x*2+1"),
    ("debug", "This loop hangs. Why? while True: pass"),
    ("docs", "Write a one-paragraph API doc for: requests.post(url, json=payload, timeout=10)"),
]

def call(url, key, model, prompt):
    body = json.dumps({
        "model": model,
        "messages": [{"role": "user", "content": prompt}],
        "max_tokens": 512, "temperature": 0.2,
    }).encode()
    req = urllib.request.Request(url, data=body, headers={
        "Content-Type": "application/json",
        "Authorization": f"Bearer {key}",
    })
    t0 = time.time()
    with urllib.request.urlopen(req, timeout=120) as r:
        data = json.loads(r.read())
    return time.time() - t0, data["choices"][0]["message"]["content"]

def score(task, out):
    checks = {
        "codegen": ["def", "retry", "backoff"],
        "refactor": ["def", ":", "->"],
        "debug": ["infinite", "loop", "condition"],
        "docs": ["param", "return", "timeout"],
    }
    return sum(1 for k in checks[task] if k in out.lower())

def main():
    endpoints = {
        "free_api": (os.environ["FREE_URL"], os.environ["FREE_KEY"], os.environ["FREE_MODEL"]),
        "self_hosted": (os.environ["LOCAL_URL"], "no-key", os.environ["LOCAL_MODEL"]),
    }
    print(f"{'task':10} {'endpoint':12} {'sec':>6} {'score':>6}")
    for task, prompt in TASKS:
        for name, (url, key, model) in endpoints.items():
            try:
                dt, out = call(url, key, model, prompt)
                print(f"{task:10} {name:12} {dt:6.1f} {score(task, out):6d}")
            except Exception as e:
                print(f"{task:10} {name:12} FAILED: {e}")

if __name__ == "__main__":
    main()

Run it:

export FREE_URL="https://api.example.com/v1" FREE_KEY="your-key" FREE_MODEL="your-model"
export LOCAL_URL="http://localhost:11434/v1" LOCAL_MODEL="your-local-model"
python tier_probe.py

For Ollama, the key is ignored, so no-key is fine. Swap in any OpenAI-compatible endpoint.

Reading the results

Three outcomes, three decisions.

• Free API is fast and scores close. Ship it. You just saved yourself a weekend of Docker.
• Free API is slow but scores fine. Check the latency budget again. Batch jobs? Still ship it. Interactive? Test with real users first.
• Free API scores badly. Stop. No quota increase fixes a quality miss. Self-host a bigger model or pay for a stronger API.

The score column is crude by design — it checks keywords, not semantics. Replace the checks with your own rubric. The structure is what matters.

A concrete data point: MonkeyCode's free tier

Let me make this less abstract. MonkeyCode is an open-source project that ships two things relevant to this framework: free model access and a free server option.

As of this writing (August 2026), the free model access includes a 10M-token allowance — comfortable for weeks of dev-time usage, wrong for production. The free server option lets you run the agent without provisioning a VPS. Quotas change, so check the repo before you build a workflow around them.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Where does it sit in the matrix? It's the "free API" row. Use it when your data is safe to send, your traffic is spiky dev-time, and your quality gate passes. Skip it for regulated data, hard latency SLAs, or steady production load.

Who should NOT use this approach

Let me be explicit about the edges.

• Data residency rules forbid third-party processing. No free API, no exceptions. The framework doesn't override compliance.
• You need a model the free tier doesn't offer. The framework can't conjure a 70B-class model out of a free quota.
• You have paying customers and no fallback. Free tiers are a feature, not an SLA.
• You treat the probe as a benchmark. Four prompts is a smoke test. It catches gross mismatches, not subtle regressions.

The takeaway

The best tier is the one you can defend with evidence. Run the probe, answer the five questions, let the matrix decide.

If you want to see how MonkeyCode's free tier holds up against your local model, the repo's README has the setup steps. Your results will beat my opinion anyway.