Free AI Server or Self-Hosted? Audit Your Workload Before You Choose

Last month, a platform team I know stood up their own LLM gateway on a fresh cloud VM. Forty-eight hours later, a scanner found the exposed dashboard and started burning their API key. The model bill was never the risk. The operational surface was.

That story repeats every week, and it's why I stopped asking "free or paid?" The real question is: what are you actually optimizing? Token price? Latency? Data boundary? Ops hours?

This article is a decision framework, not a recommendation. You get a 60-line probe that measures your workload against any OpenAI-compatible endpoint — managed or self-hosted — plus a matrix that tells you which tier you should be on.

Why "free vs self-hosted" is the wrong axis

Free tiers fail for reasons that have nothing to do with price. Self-hosting fails for reasons that have nothing to do with GPUs.

I've seen teams pick self-hosted because "it's cheaper at scale," then spend three weeks patching CUDA versions. I've seen teams pick a free managed tier because "setup takes five minutes," then discover their prompts contained customer data that required a DPA.

The axis that matters is constraint. Which resource are you actually short on?

• Short on money → free managed tier wins.
• Short on ops time → free managed tier wins.
• Short on trust in the data boundary → self-hosted wins.
• Short on latency budget → neither; you need a measured answer.

That last one is the trap. You can't know your latency or error budget until you measure your actual prompts against a real endpoint.

MonkeyCode, the open-source AI development platform I've covered before, currently advertises a free managed model tier with a 10M-token allowance (as of August 2026) and a free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I'm using it as a concrete instance of the "free managed" column — the probe works against any OpenAI-compatible endpoint, including your own vLLM or llama.cpp server.

The probe: measure your workload, not the marketing page

Here's the artifact. It reads a JSONL corpus of your real prompts, fires a configurable number of requests at any OpenAI-compatible endpoint, and emits a JSON report with error rate, p50/p95 latency, and token counts.

#!/usr/bin/env python3
"""Probe an LLM endpoint with prompts from your own repo, not a benchmark.

Usage:
    python3 probe_llm_workload.py \
        --corpus prompts.jsonl \
        --endpoint https://<endpoint>/v1/chat/completions \
        --api-key "$KEY" \
        --model <model-name> \
        --samples 20 \
        --max-tokens 256
"""

import argparse
import asyncio
import json
import statistics
import time

import httpx


async def one_call(client, endpoint, api_key, model, prompt, max_tokens):
    payload = {
        "model": model,
        "messages": [{"role": "user", "content": prompt}],
        "max_tokens": max_tokens,
        "stream": False,
    }
    headers = {"Authorization": f"Bearer {api_key}"}
    t0 = time.perf_counter()
    try:
        r = await client.post(endpoint, json=payload, headers=headers, timeout=60.0)
        dt = (time.perf_counter() - t0) * 1000
        r.raise_for_status()
        data = r.json()
        return {
            "ok": True,
            "latency_ms": round(dt, 1),
            "prompt_tokens": data.get("usage", {}).get("prompt_tokens", 0),
            "completion_tokens": data.get("usage", {}).get("completion_tokens", 0),
            "status": r.status_code,
        }
    except Exception as e:  # noqa: BLE001
        return {
            "ok": False,
            "latency_ms": round((time.perf_counter() - t0) * 1000, 1),
            "error": str(e)[:200],
            "status": getattr(getattr(e, "response", None), "status_code", None),
        }


async def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("--corpus", required=True)
    ap.add_argument("--endpoint", required=True)
    ap.add_argument("--api-key", required=True)
    ap.add_argument("--model", required=True)
    ap.add_argument("--samples", type=int, default=20)
    ap.add_argument("--max-tokens", type=int, default=256)
    args = ap.parse_args()

    prompts = [
        json.loads(line)["prompt"]
        for line in open(args.corpus, encoding="utf-8")
    ][: args.samples]

    async with httpx.AsyncClient() as client:
        results = await asyncio.gather(
            *[
                one_call(client, args.endpoint, args.api_key, args.model, p, args.max_tokens)
                for p in prompts
            ]
        )

    ok = [r for r in results if r["ok"]]
    latencies = [r["latency_ms"] for r in ok]
    total_in = sum(r["prompt_tokens"] for r in ok)
    total_out = sum(r["completion_tokens"] for r in ok)

    report = {
        "endpoint": args.endpoint,
        "samples_requested": len(prompts),
        "samples_ok": len(ok),
        "error_rate": round(1 - len(ok) / len(prompts), 3) if prompts else None,
        "latency_p50_ms": round(statistics.median(latencies), 1) if latencies else None,
        "latency_p95_ms": round(sorted(latencies)[int(len(latencies) * 0.95) - 1], 1)
        if latencies
        else None,
        "tokens_in": total_in,
        "tokens_out": total_out,
        "decision": "needs_manual_review",
    }

    print(json.dumps(report, indent=2))


if __name__ == "__main__":
    asyncio.run(main())

This is a probe, not a benchmark. It won't tell you which model is smarter. It will tell you whether an endpoint is usable for your traffic pattern — which is the question that actually blocks adoption.

How to run it

Step one: build a corpus from your real workload. Don't write new prompts; extract old ones.

# From git commit messages (a decent proxy for real dev prompts)
git log --format=%B -n 50 \
  | jq -R -s 'split("\n") | map(select(length > 10)) | .[:20] | map({prompt: .})' \
  | jq -c '.[]' > prompts.jsonl

# Or from your app's request logs (adjust the field path to your schema)
jq -c '{prompt: .request_body.messages[-1].content}' app_logs.jsonl | head -20 > prompts.jsonl

Step two: run the probe against the free managed endpoint.

python3 probe_llm_workload.py \
  --corpus prompts.jsonl \
  --endpoint https://<monkeycode-free-endpoint>/v1/chat/completions \
  --api-key "$MONKEYCODE_KEY" \
  --model <model-name-from-endpoint> \
  --samples 20 \
  --max-tokens 256

Step three: run the same corpus against your self-hosted candidate.

python3 probe_llm_workload.py \
  --corpus prompts.jsonl \
  --endpoint http://localhost:8000/v1/chat/completions \
  --api-key "no-key-needed" \
  --model <model-name-from-endpoint> \
  --samples 20 \
  --max-tokens 256

Step four: compare the two JSON reports. Don't average them in your head. Put them side by side and apply the matrix below.

The decision matrix

Criterion	Free managed tier	Self-hosted / paid
Data class	Non-sensitive, synthetic, public code	Regulated PII, proprietary source, customer data
Ops capacity	No dedicated MLOps/SRE	Team already runs and patches GPUs
Latency requirement	Best-effort is fine	p95 must stay under a hard threshold
Volume	Prototyping, CI smoke tests, weekend spikes	Steady production load, fine-tuning jobs
Reproducibility	Quick experiments, model may change	Pinned model versions, offline eval
Compliance	No formal DPA needed	Audit logs, data residency, signed DPA

Two rows deserve extra attention.

Data class is the one that overrides everything else. If your prompts contain customer PII, a free managed tier without a signed DPA is a compliance incident waiting for an auditor. Self-host, or pay for a tier with a real data-processing agreement.

Ops capacity is the one teams lie about. "We have a DevOps person" is not the same as "we have someone who will patch a GPU driver at 2 AM." If that sentence made you uncomfortable, you're a free-tier candidate.

Reading the output: three scenarios

Scenario one: the free endpoint returns error_rate: 0.0 and p95 under your budget. Ship it. You've just saved yourself a weekend of CUDA debugging.

Scenario two: error rate is fine, but p95 is 3x your self-hosted number. Now you have a real tradeoff — latency vs ops burden. That's a business decision, not a technical one.

Scenario three: the free endpoint fails on 40% of your corpus. That's not a failure; that's the probe earning its keep. You now know the free tier can't handle your workload shape, and you can stop evaluating it.

Who should not use the free option

Be honest about the exclusion list:

• Teams under GDPR/HIPAA with no signed DPA with the provider.
• Air-gapped or regulated environments where traffic can't leave the network.
• Workloads that need fine-tuning or a specific pinned model version.
• Production systems with a hard p95 latency SLA and no fallback path.

The probe has its own limits too. Twenty samples won't catch cold-start variance. Network jitter between your office and the endpoint pollutes latency numbers. And it measures availability, not answer quality — a fast wrong answer is still wrong.

The boundary question

Run the probe, then ask yourself one question: which row of that matrix belongs in your CI pipeline?

My answer: the data-class check. A regex or allowlist that blocks prompts containing secrets or PII from ever reaching a free endpoint is enforceable today, in code, before any human decides. The latency and cost rows are decisions for a weekly review. The data boundary is an invariant.

If you run the probe against MonkeyCode's free tier, I'd genuinely like to see your decision record — especially the rows where the framework says "don't use it." Those are the ones that teach us where the boundary actually is.

MonkeyCode provides free models that can run this workflow.