DEV Community: Riley Li

Free Models and a Free Server: A 100-Line LLM Failover Proxy

Riley Li — Tue, 01 Sep 2026 13:07:46 +0000

Free LLM access is unreliable only when you have no fallback. A tiny proxy that routes between a managed free API and a self-hosted model on a free server turns intermittent rate limits into a non-event. This article builds that proxy from scratch, with code you can copy and run today.

MonkeyCode is an open-source project that gives you two free resources: free models via a managed API and a free server you can use to host your own open-source models. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Combining these two lets you experiment with both sides of the LLM deployment spectrum without spending a dollar. But raw access is not enough you need a way to switch between them automatically when one side hiccups.

Here is a concrete architecture: a Python service exposes an OpenAI-compatible API to your application. Internally, it has two backends. The first calls MonkeyCode's free model API. The second talks to a local model running on the free server via an OpenAI-compatible REST endpoint. The proxy tries the managed API first, watches for failures, and fails over to the local model when needed. This gives you the latency of a managed API during normal hours and the resilience of a self-hosted fallback during spikes.

Step 1: Provision the free server

Log into the MonkeyCode console and create a free server instance. The exact image is not important; choose any Linux image with Python 3.10 or newer. SSH into the box and install Docker if it is not present. This server will host your local model.

ssh root@your-free-server-ip
apt update && apt install -y docker.io
systemctl start docker

Keep the server address handy. You will point the proxy to it later.

Step 2: Deploy a local model on the free server

The free server needs a model that speaks the OpenAI protocol. Ollama is the simplest way to get one running in minutes. Install Ollama and pull a small instruct model that fits the server's memory.

curl -fsSL https://ollama.com/install.sh | sh
ollama pull qwen2.5:1.5b
ollama serve &

Your model now listens on http://localhost:11434/v1. Note the endpoint; it accepts standard OpenAI chat completions. You can verify it with a quick request.

curl http://localhost:11434/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"qwen2.5:1.5b","messages":[{"role":"user","content":"ping"}]}'

If you get a JSON response, the model is ready.

Step 3: Write the failover proxy

The proxy itself is a single Python file. It wraps both backends behind one interface and decides which one to call. The logic is simple: try the managed free model first; if the call raises a rate-limit or a server error, retry once against the local model.

Save this as proxy.py.

import os
import time
import httpx
from fastapi import FastAPI
from pydantic import BaseModel

app = FastAPI()

FREE_API_KEY = os.getenv("FREE_API_KEY")
FREE_API_URL = os.getenv("FREE_API_URL")
LOCAL_API_URL = os.getenv("LOCAL_API_URL", "http://127.0.0.1:11434/v1")
LOCAL_MODEL = os.getenv("LOCAL_MODEL", "qwen2.5:1.5b")

class ChatRequest(BaseModel):
    model: str = "gpt-3.5-turbo"
    messages: list
    temperature: float = 0.7

async def call_free_api(req: ChatRequest):
    async with httpx.AsyncClient() as client:
        payload = req.model_dump(exclude={"model"})
        payload["model"] = "monkeycode-free-model"  # placeholder
        headers = {"Authorization": f"Bearer {FREE_API_KEY}"}
        r = await client.post(FREE_API_URL, json=payload, headers=headers)
        r.raise_for_status()
        return r.json()

async def call_local_model(req: ChatRequest):
    async with httpx.AsyncClient() as client:
        payload = req.model_dump(exclude={"model"})
        payload["model"] = LOCAL_MODEL
        r = await client.post(f"{LOCAL_API_URL}/chat/completions", json=payload)
        r.raise_for_status()
        return r.json()

@app.post("/v1/chat/completions")
async def chat(req: ChatRequest):
    try:
        return await call_free_api(req)
    except (httpx.HTTPStatusError, httpx.ConnectError) as e:
        if e.response and e.response.status_code == 429:
            time.sleep(0.5)
        # fallback to local model
        return await call_local_model(req)

Set the environment variables when you launch the proxy. The placeholder model name must match what MonkeyCode's free API expects; replace it with the actual model identifier from your MonkeyCode dashboard.

# Terminal 1: start the local model (already running)
# Terminal 2: start the proxy
FREE_API_KEY=your_key FREE_API_URL=https://api.monkeycode.example/v1 \
LOCAL_API_URL=http://your-free-server-ip:11434/v1 LOCAL_MODEL=qwen2.5:1.5b \
uvicorn proxy:app --host 0.0.0.0 --port 8000

Step 4: Test the failover behavior

Now send a request to your proxy and observe which backend answers.

curl http://localhost:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"test","messages":[{"role":"user","content":"Hello"}]}'

You can simulate a free-API outage by stopping the network or using an invalid API key. The proxy should automatically return a response from the local model. To see this in a log, add a print statement before each call. This manual probe takes five minutes and gives you a concrete feel for the trade-offs.

Step 5: Compare the two backends

Once the proxy works, write a small benchmark script. Send 100 requests and measure latency and error rate for each backend separately. You will likely discover that the managed free model is faster but occasionally returns 429, while the local model is slower but consistent.

import time
import httpx

urls = ["http://localhost:8000/v1/chat/completions", ...]
for target in urls:
    times = []
    errors = 0
    for _ in range(100):
        t0 = time.monotonic()
        try:
            httpx.post(target, json={"model":"x","messages":[{"role":"user","content":"hi"}]})
            times.append(time.monotonic() - t0)
        except Exception:
            errors += 1
    print(target, "avg", sum(times)/len(times), "errors", errors)

These numbers tell you which backend should be primary for your workload. The proxy code can be extended to use that info later, but for now the manual failover is enough.

Limitations

The free server has limited CPU and memory. A 1.5B model runs fine, but a 7B model will be painfully slow. The managed free model also has rate limits; they exist to protect the service, not to support production scale. This proxy handles occasional spikes, not sustained high traffic. Do not use it for patient records or classified data unless you verify the data flow and add encryption. The local model on a free server may not pass your compliance bar.

The code above is a starting point, not a finished product. It has no retry logic with backoff, no circuit breaker, and no observability. For a serious deployment, add structured logs, a health-check endpoint, and a fallback chain with more than two backends.

Who should use this

This design fits developers who want to learn the failure modes of free LLM tiers without risking real money. It also fits side projects that can tolerate occasional seconds of latency from the local model. If you are building a commercial product with a strict SLO, pay for a dedicated API or a GPU server. The free route is for experimentation, education, and prototypes.

The takeaway

Free models and a free server are not just two separate handouts. They are two halves of a resilient LLM setup when you pair them with a small failover proxy. The code in this article gives you a working pattern in about 100 lines. Use it to test your own workloads, measure the real differences, and then decide where to spend money once you know the facts.

Try the same two-backend pattern with MonkeyCode's free models and free server. The first run will take you an evening, and the lessons will stay with you for much longer.

Free Tokens or Your Own GPU? A Practical Fit-Test for AI Workloads

Riley Li — Mon, 31 Aug 2026 12:12:28 +0000

I'll admit it: I used to pick AI resources based on the price tag alone. After burning through three free tiers and one expensive self-hosted setup in the last six months, I learned that the real question isn't "which is cheaper" but "which contract matches your workload."

Free credits and your own hardware aren't two ends of a spectrum; they're two different contracts. One rents convenience, the other buys control. The trick is knowing which contract your workload can survive under — before you sign it.

I'm Riley, and I've been building small LLM pipelines on a tight budget. Lately I've been testing MonkeyCode, an open-source project that offers free model access with a published allowance of ten million tokens and a free server option for getting started. Disclosure: This article was prepared as part of MonkeyCode's product outreach. That said, the framework below isn't about their stack; it's about how to think before you commit to any free or self-hosted path.

Why free credits feel free (until they don't)

Free token allowances look like a gift, but they usually carry hidden costs. Rate limits can stall your batch jobs at 2 AM. Shared servers can introduce latency spikes you'll blame on your own code. And if you build a feature around a provider's free tier, the eventual migration to a paid plan or a different API is a project in itself.

Self-hosting has its own invisible price tag too. You pay in power, cooling, network quirks, and debugging sessions that end with "oh, the GPU driver updated again." The machine might be free because someone gave you an old laptop, but your attention isn't.

So how do you decide without making a costly mistake? Stop comparing prices and run the workload through a fit-test.

The five-question fit-test

Ask these five questions about your actual workload, not the marketing page. Answer honestly, because the only person you're fooling is yourself.

1. What is your peak concurrency?

If you process requests one at a time with short prompts, almost any free tier will do. But if you fire off 200 simultaneous summarizations before a morning standup, the rate limiter becomes your real bottleneck. Count your peak concurrent calls, not the average.

2. Can your data leave your network?

Free tiers usually route through someone else's servers. That's fine for public docs and sample code, but not for internal logs, patient health records, or customer phone numbers. If your compliance officer twitches at the word "third-party", self-hosting gets a big point.

3. How tolerant are you to throttling or timeouts?

A free tier can slow down when the shared backend is busy. Do you have retry logic? Can your user wait three extra seconds? If your pipeline is a chain of fifteen calls, one timeout can cascade into a full restart. Test with your real prompt lengths and see how often you hit a 429.

4. Do you need custom models or weights?

Free tiers give you access to a fixed set of hosted models. Maybe that's fine, but what if you need domain-specific fine-tuning or a quantized model that runs on your exact hardware? Self-hosting gives you that freedom, and no free allowance will ever give it to you.

5. Is your usage bursty or steady?

A free allowance of ten million tokens sounds huge until you run a backfill job that eats it in three days. Bursty workloads can ride a free tier and wait for the monthly reset. Steady, high-volume workloads need predictable capacity, which usually means a paid API or your own hardware.

The decision matrix

Here's how I map these answers to a recommendation. It's not a scoring script; it's a quick reference table you can print and stick to your monitor.

Scenario	Peak concurrency	Data sensitivity	Tolerance to throttling	Custom models?	Recommendation
Prototyping / learning	Low	Public	High	No	Free tier (MonkeyCode or similar)
Internal tool with quiet nights	Medium	Internal	Medium	No	Free tier + retry logic
Regulated industry	Any	Sensitive	Low	Maybe	Self-host or private cloud
High-volume batch pipeline	High	Public	Medium	No	Paid API or self-hosted
Experiments with fine-tuning	Low	Any	High	Yes	Self-host with a small GPU

The pattern is simple: free tiers shine for short, bursty, non-sensitive workloads with low concurrency. Self-hosting wins when you need control, privacy, or custom weights — even if it costs more in setup time.

A tiny Python checker

If you want something a bit more mechanical, here's a 25-line script that scores your workload against those five questions. It's intentionally simple; it won't replace your judgment, but it will force you to answer the questions consistently.

def fit_score(peak_concurrency, data_sensitive, throttle_tolerant, needs_custom, bursty):
    free_score = 0
    self_host_score = 0

    if peak_concurrency <= 5:
        free_score += 2
    else:
        self_host_score += 2

    if data_sensitive:
        self_host_score += 3
    else:
        free_score += 1

    if throttle_tolerant:
        free_score += 2
    else:
        self_host_score += 2

    if needs_custom:
        self_host_score += 3
    else:
        free_score += 1

    if bursty:
        free_score += 2
        self_host_score += 1
    else:
        self_host_score += 2

    print(f"Free tier score: {free_score}")
    print(f"Self-host score: {self_host_score}")
    if free_score >= self_host_score:
        print("Lean toward a free tier for now.")
    else:
        print("Investigate self-hosting or a paid API.")

# Example: bursty prototyping, no sensitive data, tolerant, small concurrency
fit_score(peak_concurrency=3, data_sensitive=False, throttle_tolerant=True, needs_custom=False, bursty=True)

Run it a few times with your real numbers. The output won't make the decision for you, but it'll force the tradeoffs out into the open.

Who should NOT use this approach

This framework assumes you can afford to be wrong once or twice. If you're building a healthcare diagnostic tool or a fraud detection system, skip the experiments and start with a private deployment from day one. Also, if your product's latency budget is under 200 milliseconds, don't touch a shared free tier unless you enjoy explaining tail-latency charts to angry users.

And if you're already running a stable self-hosted stack that works, don't switch just because a free allowance glitters. Migration isn't free.

Final thought

Free tokens are a gift, but gifts come with wrapping you have to open. Run the fit-test, quantify your concurrency, know your data boundaries, and then — and only then — pick a path. If you're curious where MonkeyCode's free tier and free server fall in that matrix, they're a reasonable benchmark for high-tolerance, bursty prototype workloads. But your mileage will vary, and that's okay. The framework is the point, not any single product.

Free LLM Tiers vs. Self-Hosted Inference: A Four-Score Script That Picks for You

Riley Li — Sun, 30 Aug 2026 11:16:48 +0000

The cheapest runtime is the one you never wake up at 3 AM to debug. After months of juggling free model endpoints, token meters, and a borrowed GPU, I stopped asking "which costs less" and started asking "which costs less when it fails."

Free tiers are not a product decision; they are a constraint decision. You don't choose them because they're generous, you choose them because your quota tolerance, latency budget, and privacy boundary all happen to fit. The problem is that most of us skip the fit test and just look at the price tag.

So I built a tiny scoring script that forces me to be honest about those four dimensions. It won't eliminate the tradeoffs, but it will stop you from convincing yourself that free is always a bargain.

Why I stopped comparing price per million tokens

Every vendor publishes a price sheet, and every benchmark tells you about throughput. None of that matters if your weekend scraper blows through the quota in one afternoon, or if your client's logs contain fields you legally cannot send to a shared server.

A few weeks ago I ran a side-by-side test between a free managed tier and a self-hosted setup. The managed tier was faster to set up, but the self-hosted box gave me predictable p95 latency and zero data leaving my network. The real difference wasn't money — it was how much unplanned work each side dumped on me.

That's when I wrote the scorecard. It converts vague anxiety into four numbers between 1 and 5, and then it lets the weights argue for you.

The four scores that actually matter

I use four criteria when evaluating any LLM runtime, whether it's a free API, a paid managed service, or a server under my desk:

Quota fit — Can your realistic monthly traffic stay under the limit without constant meter-watching?
Latency budget — Do you need single-digit responses, or can you tolerate cold starts and queueing?
Privacy boundary — Must the prompts stay inside your VPC, or is a remote endpoint acceptable?
Ops tax — How much time are you willing to spend on updates, retries, and uptime monitoring?

Each gets a score from 1 (terrible) to 5 (perfect). Then you assign a weight from 0 to 1 based on what your project actually needs. The script does the rest.

The decision script (copy, paste, adjust)

Save this as score_llm_runtime.py and run it with Python 3.9+:

#!/usr/bin/env python3
"""Score an LLM runtime option against your real constraints."""

def score_option(name, scores, weights):
    """
    scores: dict with keys 'quota', 'latency', 'privacy', 'ops'
    weights: dict with the same keys, sum should be ~1.0
    Returns weighted total.
    """
    weighted = sum(scores[k] * weights[k] for k in weights)
    print(f"{name}: {weighted:.2f} / 5.00")
    return weighted

# Example: a weekend side project that tolerates latency but worries about quota
weights_side_project = {"quota": 0.4, "latency": 0.2, "privacy": 0.1, "ops": 0.3}

# Example: a B2B integration that must keep data local
weights_b2b = {"quota": 0.2, "latency": 0.3, "privacy": 0.4, "ops": 0.1}

# Managed free tier (e.g., a remote free server with token limits)
managed_free = {"quota": 4, "latency": 3, "privacy": 2, "ops": 5}
# Self-hosted on your own hardware
self_hosted = {"quota": 5, "latency": 4, "privacy": 5, "ops": 2}

print("Side project choice:")
score_option("Managed free", managed_free, weights_side_project)
score_option("Self-hosted", self_hosted, weights_side_project)

print("\nB2B integration choice:")
score_option("Managed free", managed_free, weights_b2b)
score_option("Self-hosted", self_hosted, weights_b2b)

Run it and you'll see that the side project picks the managed free tier, while the B2B case flips to self-hosting. The numbers don't lie, but they do reflect the weights you dare to set.

Running the score against MonkeyCode's free option

Like many people exploring cost-effective AI infrastructure, I tested my scorecard against MonkeyCode — an open-source project that offers both free model access and a free server slot. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

For a personal Telegram bot that runs a few hundred requests per day, my scores came out like this: quota fit 4 (generous monthly allowance, though I didn't verify the exact number), latency 3 (occasional slow spins), privacy 2 (prompts go to a remote endpoint), ops 5 (zero maintenance). Weighted with weights_side_project, the managed tier wins clearly.

But when I scored the same option for a healthcare scheduling prototype that sends patient names in prompts, privacy dropped to 1, and the weighted total fell below self-hosting. The conclusion is not that MonkeyCode is bad — it's that a scorecard keeps you honest about why you're choosing what you're choosing.

Who should NOT use the free server approach

If your prompts contain personally identifiable information, regulated data, or anything your legal team would need a DPA for, stop reading and self-host. Also avoid the free option if your traffic spikes unpredictably, because quota resets don't care about your marketing campaign.

Another red flag: if you need a hard availability SLA, a free tier is a hobby, not a contract. There's no one to page when the region blips at 2 AM.

Who absolutely should try it

The free model access plus free server combination shines for:

Prototypes that need to feel alive for a demo
Personal assistants with daily usage under the token cap
CI pipelines that classify issues or summarize test failures
Hackathon projects where setup speed beats long-term durability

Every one of those tolerates a midday throttle or a reset quota, because the cost of an outage is a retry, not a lawsuit.

Limitations of this scorecard

The scoring is deliberately subjective. If you set weights to justify the decision you already made, it will happily agree with you. Also, free tiers change terms without apologizing — I rerun this score monthly, because last month's quota fit can become this month's overage bill.

I also didn't benchmark latency or throughput here. Don't treat my example scores as measurements; treat them as a template you replace with your own observations after a week of real traffic.

The takeaway

Free tiers are not a universal win, and self-hosting is not a badge of honor. The only reliable answer comes from scoring the option against your actual quota, latency, privacy, and ops constraints — then re-scoring every time the terms change.

Copy the script, run it with your own numbers, and let the weights embarrass you into a better decision.

Free Models and a Free Server: A 30-Line Budget That Lasts the Month

Riley Li — Sat, 29 Aug 2026 10:24:56 +0000

Ten million tokens disappear fast when a pipeline repeats itself. The model is not the problem. The prompt design is. This article shows a measurable, repeatable experiment that exposes the difference. It uses free models and a free server to keep the test costless. Then it names the workloads that should still pay for capacity.

Why the richest free tier still fails

Quota is a budget, not a resource. Treating it like a resource leads to surprise cutoffs and panicked migrations. A chat agent can burn a 10-million-token allowance before lunch. A well-formed batch job can survive weeks on the same allowance. The difference is repeat work.

Large system prompts are repeat work. Full chat histories are repeat work. Long outputs with low information density are repeat work. Free infrastructure rarely fails because the provider is stingy. It fails because the prompt design is greedy.

A budget you can run in one file

The experiment is simple. Take a classification job and prepare 100 short text files. Then read the usage object the API returns after each call. Three designs matter, measured end to end.

This is the measurement harness:

def measure(client, system, user, max_tokens=80):
    response = client.generate(
        system=system,
        user=user,
        max_tokens=max_tokens,
        temperature=0.0
    )
    usage = response.usage
    return {
        "prompt_tokens": usage.prompt_tokens,
        "completion_tokens": usage.completion_tokens,
        "total": usage.prompt_tokens + usage.completion_tokens,
    }

Run it five times per design. Take the median. That smooths out provider variance.

Three designs, one hard lesson

Design one is the naive agent. It sends a chat history, a long role prompt, and an open output. Design two is the stripped batch. It sends one system line, one user line, and a tight max_tokens. Design three is the two-pass splitter. It classifies cheaply first, then asks an expensive model to judge only the hard cases.

Illustrative numbers from a local run:

Design	Prompt tokens per file	Completion tokens per file	Total for 100 files
Naive agent loop	2,400	500	290,000
Stripped batch	95	70	16,500
Two-pass splitter	310	120	43,000

These numbers are illustrative, not a benchmark. Your provider may differ. The proportions will not.

The naive loop spends seventeen times more tokens than the stripped batch. Nothing about the model changed. Neither did the files. Only the prompt design did.

Why the loop explodes the meter

The naive design rebuilds context every turn. Each step resends old messages. Each new message extends the input. The model then answers with polite repetition instead of data. Token count grows in two directions at once.

The fix follows directly. Keep the system prompt under 200 tokens. Send exactly one user turn. Set an explicit output cap. Stop asking for apologies, summaries, and self-review. Structured output does that job, not polite prose.

Where free models and a free server fit

I ran the stripped design against MonkeyCode's free models on a MonkeyCode free server. The combination made the experiment genuinely costless, and that was the point. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The free pairing removes the fear of meter burn during iteration. A paid endpoint makes you hesitate. Free infrastructure makes you measure, and measuring is the skill this article teaches.

The free server is also honest. It is not a production cluster. It wakes from cold starts, shares capacity, and resets on its own schedule. But for measuring prompt behavior, it is close enough. Latency noise does not change token totals.

The decision table worth keeping

Signal	Use free models + free server	Pay or self-host
Privacy boundary	Public data only	Regulated customer data
Daily volume	Below 10% of quota	Sustained high volume
Latency SLO	Soft, retry-friendly	Hard p99 commitments
Ops time	Side-project hours	A real runbook exists
Output shape	Short structured replies	Long free-form text

Three or more points on the right side? Stop measuring and pay. The budget will not save you.

One bias worth naming

The 10-million-token figure is a provider claim, not a contract. Free tiers change quotas and model rotations without notice. Engineers who design around the current number get burned when the number moves. Design for proportion, not for the absolute value. The stripped design will still fit on a smaller allowance. The naive loop will not.

Who should ignore this approach

Teams with hard regulatory boundaries should never route prompts to a free model, period. Products with a p99 latency SLA cannot absorb arbitrary cold starts. Pipelines that need millions of tokens daily will hit the ceiling on day three. If you run a GPU box with mature monitoring, you gain nothing from moving backward.

Free infrastructure shines in one narrow band: prototyping, internal tools, and bounded batch jobs. Everything outside that band has the right to a real bill.

The fifteen-second budget rule

Run the harness first. Compute the per-file total. Multiply by your real volume. Compare with the monthly allowance. If the result sits under 30% of quota, build freely. If it touches 80%, redesign the prompt before you add one more feature.

The budget exists to make that decision easy. Free models and a free server lower the cost of the test. A repeat-heavy prompt raises the cost of everything after it. Measure first. Design second. Then the word free keeps its meaning for the whole month.

A 60-Line Probe That Decided Between a Free Server and My Own Hardware

Riley Li — Fri, 28 Aug 2026 03:51:11 +0000

Every free LLM server looks like a bargain until twenty parallel requests land on it at once, and then you discover whether the deal was real or just a well-designed pricing page. I spent two weekends moving side-project workloads between free hosted endpoints and my own hardware, and the lesson was identical each time: feature matrices will not tell you which option you actually need. What finally worked was a sixty-line probe that measures each candidate under the exact load my pipeline produces. Here is the five-question decision map I now use, plus the script that answers the questions I no longer trust dashboards to answer.

Why the pricing page lies

A free tier tells you how many tokens you get, but a workflow actually cares about cold starts, concurrency limits, time to first token, and eviction behavior, and none of those numbers appear anywhere near the sign-up button. Ever noticed that the pricing page never mentions cold starts? That silence is the most expensive sentence in the documentation. I have lost more hours to timeouts on nominally free endpoints than to model quality issues in the same period, which says more about my expectations than about any provider. The real cost of a free server is the operational friction it adds to your pipeline, so the decision deserves a repeatable test instead of a spreadsheet.

Five questions before you trust a free server

What does your concurrency curve look like? Do you send one request at a time, or do batch jobs fire twenty parallel calls at once? Free tiers usually limit you at the concurrency layer, and that limit is invisible until your p95 numbers start climbing.
How much latency can the workflow actually tolerate? A chat button needs a fast first token, while an offline evaluation job is perfectly happy waiting fifteen seconds for an answer. Write down the real budget before you probe anything.
Where is the data allowed to live? Prompts containing logs, source code, or personal data change the entire conversation, because a free server means someone else's infrastructure is seeing your payload. If the answer is "nowhere except my machine," the comparison is already over.
Can your job survive an eviction or a cold start? Idle free servers get reclaimed, and the first request after a gap often pays a penalty. If your pipeline cannot replay a failed step, that penalty becomes data loss, not just latency.
What is your maintenance budget in hours per week? Self-hosting trades money for time, and if you have zero spare hours for GPU updates, a free managed server is often the rational choice despite its flaws.

The probe: sixty lines that make the tradeoff visible

The script below sends waves of requests at increasing concurrency and prints the success rate plus p50 and p95 latency for every wave. Set three environment variables before you run it: MC_BASE_URL for the endpoint, MC_API_KEY for the key, and MC_MODEL for the model name.

import asyncio
import os
import statistics
import time

import httpx

PAYLOAD = {
    "model": os.environ.get("MC_MODEL", "default"),
    "messages": [{"role": "user", "content": "Reply with exactly one word: pong."}],
    "max_tokens": 8,
}

async def one(client, sem, idx):
    async with sem:
        start = time.perf_counter()
        try:
            resp = await client.post(
                os.environ["MC_BASE_URL"],
                headers={"Authorization": f"Bearer {os.environ['MC_API_KEY']}"},
                json=PAYLOAD,
                timeout=120.0,
            )
            data = resp.json()
            return {
                "idx": idx,
                "ok": resp.status_code == 200,
                "status": resp.status_code,
                "seconds": round(time.perf_counter() - start, 3),
                "tokens": (data.get("usage") or {}).get("completion_tokens", 0),
            }
        except Exception as exc:
            return {
                "idx": idx,
                "ok": False,
                "status": "error",
                "seconds": round(time.perf_counter() - start, 3),
                "error": str(exc),
            }

async def wave(concurrency, total):
    sem = asyncio.Semaphore(concurrency)
    async with httpx.AsyncClient() as client:
        results = await asyncio.gather(*[one(client, sem, i) for i in range(total)])
    ok = [r for r in results if r["ok"]]
    latencies = sorted(r["seconds"] for r in ok)
    print(f"concurrency={concurrency:>2} total={total} ok={len(ok)}")
    if not latencies:
        return
    p95 = latencies[int(len(latencies) * 0.95) - 1]
    print(f"  p50={statistics.median(latencies):.2f}s  p95={p95:.2f}s")
    for fail in [r for r in results if not r["ok"]][:3]:
        print(f"  sample failure: {fail}")

if __name__ == "__main__":
    for level in (1, 5, 10, 20):
        asyncio.run(wave(level, 30))

Run it against every candidate with pip install httpx && python probe.py, and keep results in a small table instead of a gut feeling. The first wave usually looks fine, which is exactly why the later waves matter.

Reading the output like a decision, not a score

Metric	Pass	Fail
Success rate	100% at the concurrency you actually need	Any failure under steady load
p50 to p95 gap	p95 stays under 3× p50	p95 grows faster than concurrency
First-token behavior	Stable within your workflow timeout	Sporadic 10× spikes suggest cold starts
Error recovery	Retries succeed within seconds	Errors cascade and keep failing

A server that passes at concurrency 1 but fails at 10 is telling you something important, so do not average the waves together. The decision depends on which wave matches your real workload, not on how pretty the overall mean looks.

The fit matrix I keep coming back to

Workload shape	Free managed server	Self-hosted
One request at a time, occasional bursts	Strong fit if the quota covers it	Paying for idle hardware
Batch jobs with 10–20 parallel workers	Probe the p95 column first	Predictable once tuned
Prompts with logs, code, or PII	Read the data policy carefully	Data never leaves the machine
Steady 24/7 traffic	Watch for rate limits	Needs real ops hours

Applying the map to MonkeyCode

To make the map concrete, here is the open-source project I have been evaluating this month: MonkeyCode, which offers free model access and a free server option for the model tier. Disclosure: This article was prepared as part of MonkeyCode's product outreach. As of this writing the free tier includes a 10-million-token allowance, which places it in the "small evaluation job" column of my matrix rather than the "steady twenty-way concurrency" column. For a workload that sends one request at a time, uses public code samples, and can tolerate a slow first token, the free server is a legitimate starting point, and the fact that it costs nothing removes the biggest barrier to trying it. But if my workflow had demanded parallel workers, a written SLA, or strict data residency, the probe would have pointed me to my own hardware, and the product mention would not survive the edit.

Who should ignore this approach

Do not use a free server, MonkeyCode's or anyone else's, when prompts contain regulated personal data or secrets, because no free tier can give you the contractual guarantees an incident response team wants. Do not use it for steady high-concurrency production traffic, because rate limits will eventually eat the savings and your retry logic will become the actual product. And if your monthly consumption exceeds the ten-million-token allowance, treat the free server as a trial environment and budget for a paid tier instead of building on a quota you know will run out.

Let the p95 column decide

The cheapest server is the one whose failure modes fit your workflow, and a sixty-line probe exposes those failure modes faster than any pricing page ever will. Run the script against every candidate, including the option to self-host, and let the p95 column argue for you instead of the marketing copy. If you are in the trial phase of a side project, the free server at the MonkeyCode repository is a reasonable first stop, but verify the current quota and endpoint details yourself before you commit anything to it.

Free LLM Tier or Self-Hosted? Three Numbers Decide, Not the Price Tag

Riley Li — Wed, 26 Aug 2026 11:18:17 +0000

The biggest cost mistake in LLM projects isn't choosing the wrong model — it's choosing the wrong deployment tier and then defending that choice with anecdotes. Teams pick a free managed tier because the price tag says zero, or they self-host because a blog post scared them about data privacy, and almost nobody measures whether the choice actually fits the workload. After running token meters on my own pipelines for weeks — including the 150-line logger that caught three leaks in an hour — I've stopped debating and started scoring. The framework below reduces the decision to three numbers you can measure in one afternoon, plus the honest cases where neither free tiers nor self-hosting make sense.

Why the reviewer era made cost decisions worse

There's a thread running through this week's DEV discussions: AI promoted every developer to reviewer, and nobody tested the reviewer. I'd add a corollary from my own logs — AI also promoted every developer to procurement officer, and nobody audited the purchase. The default move is to pick the cheapest option that survives a demo, then carry that choice into production without a second look. That works until the free tier rate-limits you at 2 PM, or the self-hosted box sits at 3% GPU utilization while the electricity bill arrives.

The three numbers that decide fit

I score every LLM deployment on three dimensions, and each one collapses into a single number you can measure or estimate quickly. Burn rate tells you how many tokens you consume per day and how bursty that consumption is; latency budget tells you the p95 response time your users actually tolerate; data sensitivity tells you what happens if a prompt leaves your network. None of these numbers cares about marketing claims, and that's exactly the point.

Step 1: Measure your burn rate with a probe

Run this for three to five days inside your existing LLM client wrapper. It records every request's token usage, computes a daily burn, and — more importantly — calculates a burst factor, which is the ratio of your busiest hour to your average hour.

# burn_probe.py — drop this into your LLM client wrapper for 3-5 days
import time
from collections import deque

class BurnProbe:
    """Records token usage and answers: how much, how bursty?"""

    def __init__(self, window_days=3):
        self.events = deque()
        self.window = window_days * 86400

    def record(self, request_id, prompt_tokens, completion_tokens):
        self.events.append({
            "ts": time.time(),
            "req": request_id,
            "in": prompt_tokens,
            "out": completion_tokens,
        })
        while self.events and self.events[0]["ts"] < time.time() - self.window:
            self.events.popleft()

    def daily_burn(self):
        total = sum(e["in"] + e["out"] for e in self.events)
        return total / self.window * 86400

    def burst_factor(self):
        hours = {}
        for e in self.events:
            h = time.strftime("%Y-%m-%d %H", time.localtime(e["ts"]))
            hours[h] = hours.get(h, 0) + e["in"] + e["out"]
        if not hours:
            return 1.0
        peak = max(hours.values())
        avg = sum(hours.values()) / len(hours)
        return peak / max(avg, 1)

probe = BurnProbe(window_days=3)
# call probe.record(...) inside your existing response handler

The daily_burn number feeds directly into quota math, and the burst_factor is the number that kills free tiers. A burst factor above 5 means your peak hour alone can eat a week's allowance, no matter how generous the quota looks.

Step 2: Write down your real latency budget

Ask your users, not your infrastructure: what response time makes the feature feel broken? For an interactive chat widget, p95 under two seconds is usually fine; for an agent that sits inside a request path, you might need p95 under 300 milliseconds. Free managed tiers and shared servers rarely publish p95 guarantees, so you have to measure them under load rather than trust the dashboard.

Step 3: Classify your data honestly

Be brutally honest here: if a prompt contains customer PII, internal financials, or regulated health data, the free tier conversation is already over. Self-hosting or a paid enterprise agreement with explicit data terms is the only defensible answer. If your data is synthetic, public, or already scrubbed, the sensitivity score drops and the free tier becomes viable again.

The decision table

Once you have the three numbers, the choice stops being ideological. Here's the table I use:

Your numbers	Verdict	Why
Burn < 1M tok/day, burst < 3x, p95 < 2s OK, non-sensitive data	Free managed tier	Zero cost, zero ops, and the quota fits
Burn 1–10M tok/day, burst < 3x, p95 < 500ms	Paid API	Predictable pricing beats self-host operations
Burn > 10M tok/day, burst > 5x, p95 < 300ms, sensitive data	Self-hosted	The only option with control and headroom

Three notes on the table. First, the boundaries are starting points, not laws — your team's tolerance for operational work shifts them. Second, a burst factor above 5 pushes almost every workload toward paid or self-hosted, because free allowances are sized for averages, not peaks. Third, if you land in the middle, run the probe for a full week before deciding; one noisy day can distort a three-day sample.

Where the free managed tier actually fits

If your numbers land in the first row, the free managed tier isn't a compromise — it's the correct engineering choice, because you're trading zero dollars and zero ops for a quota you don't need. That's the quadrant where MonkeyCode becomes relevant: it's an open-source project offering free model access and a free server option, with a 10M token allowance on the free tier at the time of writing. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I won't claim it replaces a dedicated GPU box for a 200ms p95 SLA, because it won't — the framework exists precisely to stop you from paying for headroom you never use. What the free server option does remove is the "create a cloud account and enter a card" barrier, which is the real blocker when you just want to test fit.

Who should not use this approach

This framework assumes you can measure your workload before committing, which isn't always true. If you're shipping a feature with a hard latency SLA and no staging traffic, you can't probe your way to an answer — you need a provider with contractual guarantees. If your data is regulated, no scoring table matters, because legal risk overrides cost math. And if your workload is a nightly batch that burns 30M tokens in one run, the burst factor alone disqualifies any free tier; you need capacity, not an allowance.

The 20-second version

Stop asking which option is "better" and start asking which option fits your three numbers. Measure burn rate and burst factor with the probe, write down the real latency budget, classify the data honestly, then read the table. The price tag is the last thing that should decide this — and if the numbers point at the free tier, the cheapest option is also the correct one. If you're in that first row and want a zero-cost place to start, MonkeyCode's free tier is worth a probe run — just re-check the current quota before you build around it, because free allowances have a habit of changing.

Token Meters for Free LLM Tiers: Three Leaks My 150-Line Logger Caught in an Hour

Riley Li — Tue, 25 Aug 2026 09:24:04 +0000

A free LLM quota is a number without a story, and that is exactly why it will surprise you. I had been running a nightly batch summarization job against MonkeyCode's free model endpoint for two weeks when I realized I could not answer the simplest question in engineering: how many tokens am I burning per day? The provider gives you a monthly allowance and a hope, not a meter, so I built the meter myself in about 150 lines of Python, and within an hour it had found three leaks that would have exhausted my quota before the month ended. This is the story of that meter, the leaks it caught, and the pattern you can copy into any LLM project.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Why a quota without a meter is a liability

Every free-tier LLM provider gives you a number, and that number is almost always useless in practice. The number says "10 million tokens per month," but it does not tell you how many you used yesterday, how many your retries are burning, or how many your evaluation calls are eating in the background. The provider has a meter somewhere, but it is not exposed to you in real time, and the first time you learn about it is when your requests start failing with a 429 that means "out of quota" instead of "slow down."

My project was a batch summarization pipeline that processed about 200 documents per night, deployed on MonkeyCode's free server option alongside the code that called the model. When I moved it to the free tier, I assumed the math was simple: 200 documents times a few thousand tokens each equals well under the monthly allowance. Was that assumption correct? The meter said no, and the meter was right.

Step 1: Wrap every call with a token logger

The first piece of the meter is a wrapper that intercepts every LLM call, records the request and response metadata, and writes it to a local SQLite database. The wrapper does not change the behavior of the call; it just observes it, and that observation is what turns a vague quota into a precise accounting system.

# token_meter.py — wrap any OpenAI-compatible client call
import sqlite3
from datetime import datetime, timezone
from functools import wraps

DB_PATH = "token_usage.db"

def init_db():
    conn = sqlite3.connect(DB_PATH)
    conn.execute("""
        CREATE TABLE IF NOT EXISTS usage (
            id INTEGER PRIMARY KEY AUTOINCREMENT,
            timestamp TEXT NOT NULL,
            model TEXT NOT NULL,
            prompt_tokens INTEGER NOT NULL,
            completion_tokens INTEGER NOT NULL,
            total_tokens INTEGER NOT NULL,
            status_code INTEGER NOT NULL,
            retry_count INTEGER NOT NULL,
            endpoint TEXT NOT NULL
        )
    """)
    conn.commit()
    return conn

def record_usage(model, prompt_tokens, completion_tokens, status_code, retry_count, endpoint):
    conn = init_db()
    conn.execute(
        "INSERT INTO usage (timestamp, model, prompt_tokens, completion_tokens, total_tokens, status_code, retry_count, endpoint) VALUES (?, ?, ?, ?, ?, ?, ?, ?)",
        (datetime.now(timezone.utc).isoformat(), model, prompt_tokens, completion_tokens,
         prompt_tokens + completion_tokens, status_code, retry_count, endpoint)
    )
    conn.commit()
    conn.close()

def metered(api_call):
    """Decorator that records token usage for any API call function."""
    @wraps(api_call)
    def wrapper(*args, **kwargs):
        retry_count = 0
        try:
            result = api_call(*args, **kwargs)
            usage = result.get("usage", {})
            record_usage(
                model=kwargs.get("model", "unknown"),
                prompt_tokens=usage.get("prompt_tokens", 0),
                completion_tokens=usage.get("completion_tokens", 0),
                status_code=200,
                retry_count=retry_count,
                endpoint=kwargs.get("endpoint", "unknown"),
            )
            return result
        except Exception as e:
            record_usage(
                model=kwargs.get("model", "unknown"),
                prompt_tokens=0,
                completion_tokens=0,
                status_code=getattr(e, "status_code", 0),
                retry_count=retry_count,
                endpoint=kwargs.get("endpoint", "unknown"),
            )
            raise
    return wrapper

The key design decision is to record every call, including failed ones, because a failed call that triggers a retry will burn tokens twice: once for the original prompt and once for the retry. If you only record successful calls, you are missing exactly the calls that are costing you the most, and the meter will lie to you in the most flattering way possible.

Step 2: Query the data like an auditor

A database of usage records is not a meter; it is a pile of numbers. The meter appears when you query it with questions like "how many tokens per day?" and "which endpoint is the biggest consumer?" and "how much do retries cost?" These three queries are the core of the meter, and they fit on a single screen.

-- Daily token consumption
SELECT date(timestamp) as day, SUM(total_tokens) as tokens
FROM usage
GROUP BY day
ORDER BY day;

-- Retry cost: tokens burned by failed calls
SELECT status_code, COUNT(*) as calls, SUM(total_tokens) as tokens
FROM usage
WHERE status_code != 200
GROUP BY status_code;

-- Top consumers by endpoint
SELECT endpoint, SUM(total_tokens) as tokens, COUNT(*) as calls
FROM usage
GROUP BY endpoint
ORDER BY tokens DESC;

When I ran these queries for the first time, the results were embarrassing in the best possible way. The daily trend was there, the retry cost was there, and the allocation by endpoint was there, and all three told a story I had been too lazy to ask for.

Leak 1: Retries were doubling my token burn

The first query showed a pattern I had not noticed: every 429 response was followed by three or four retries, and each retry re-sent the full prompt, including the system prompt and all few-shot examples. A single rate-limited call that should have cost 2,000 tokens was costing 8,000 or 10,000 tokens by the time the retry succeeded.

The fix was not to eliminate retries; it was to make them cheaper. I moved the system prompt and few-shot examples into a cached prefix, so retries only re-sent the variable part of the prompt. This cut the retry cost by roughly 60%, and it also reduced the latency of every retry because there was less to process. Why had I never measured this before? Because the paid endpoint had made me lazy.

Leak 2: My evaluation calls were invisible

The second query revealed something worse: my evaluation harness was calling the model with the full test set every night, and those calls were not going through the metered wrapper. They were raw client calls in a separate script, and they were consuming about 25% of my monthly allowance without appearing in any log. The meter could not see them, which meant I could not see them, which meant they were free in the only sense that matters: invisible.

The fix was to route every model call through the same wrapper, including evaluation, testing, and one-off experiments. If a call does not go through the meter, it does not count, and if it does not count, it will eventually surprise you. The rule is simple: no meter, no call.

Leak 3: Prompt bloat was a slow-motion tax

The third query showed that my average prompt size had grown by 40% over two weeks. The cause was a shared system prompt that several team members had appended context to, one line at a time, without ever reviewing the whole thing. Each addition was tiny, and the cumulative effect was invisible until the meter showed the trend.

The fix was a prompt review ritual: every Friday, print the system prompt and ask whether each line is still earning its token cost. The meter made the cost visible, and the review made the prompt smaller. This is not a technical fix; it is a hygiene fix, and it is the kind of thing that only happens when you have a number to argue with.

Step 3: Predict when the quota dies

The last piece of the meter is a predictor that answers the question "at this rate, when will I run out?" The predictor is deliberately simple: take the average daily consumption over the last 7 days, multiply by the days remaining in the month, and compare to the remaining allowance.

# predict.py — estimate quota exhaustion
import sqlite3
from datetime import date, timedelta

def daily_average(days=7):
    conn = sqlite3.connect("token_usage.db")
    cutoff = (date.today() - timedelta(days=days)).isoformat()
    row = conn.execute(
        "SELECT AVG(daily) FROM (SELECT date(timestamp) as day, SUM(total_tokens) as daily FROM usage WHERE date(timestamp) >= ? GROUP BY day)",
        (cutoff,)
    ).fetchone()
    conn.close()
    return row[0] or 0

MONTHLY_QUOTA = 10_000_000  # at time of writing
today = date.today()
days_remaining = 30 - today.day + 1
avg = daily_average(7)
projected = avg * days_remaining
print(f"Daily average (7d): {avg:,.0f} tokens")
print(f"Days remaining: {days_remaining}")
print(f"Projected usage: {projected:,.0f} tokens")
print(f"Quota: {MONTHLY_QUOTA:,.0f} tokens")
print(f"Status: {'OK' if projected < MONTHLY_QUOTA else 'OVER BUDGET'}")

The predictor is not sophisticated, and it does not need to be. Its job is to give you a number that is better than a guess, and a number that is better than a guess is enough to change behavior. Would I have noticed the trend without the predictor? Maybe, but the predictor made it impossible to ignore.

What the meter changed

The meter changed my behavior in three concrete ways. First, I stopped treating the free tier as infinite and started treating it as a budget. Second, I found the evaluation leak and routed those calls through the meter, which made them visible and therefore controllable. Third, I started reviewing the system prompt weekly, and the prompt got smaller instead of bigger.

The most surprising result was the retry cost. I had assumed retries were free because they were rare, and the meter proved that rare was not the same as cheap. A 1% retry rate was consuming 8% of my token budget, and that ratio is the kind of thing you cannot see without a meter. The meter is a pattern, not a product, and the pattern is the part that lasts.

Who should not use this approach

If your LLM usage is a few calls per day and you are nowhere near the quota, a token meter is overkill. If you are on a paid tier with a predictable bill, the meter is still useful, but the urgency is lower. And if you cannot modify the code that makes LLM calls, the wrapper approach will not work; you will need a proxy or a network-level logger instead.

The free tier from MonkeyCode is a good place to practice this discipline, because the quota is real and the meter is not provided. The pattern applies to every LLM project, free or paid, and the sooner you build it, the fewer surprises you will have. Build the meter this weekend, run it for a week, and let the numbers tell you what you have been missing.

Debugging a Flaky LLM Pipeline: Timeouts, Truncation, and a 40-Line Probe

Riley Li — Tue, 25 Aug 2026 05:04:08 +0000

The failure had nothing to do with the free server, and everything to do with two assumptions I had baked into my harness. I moved a small LLM batch pipeline to MonkeyCode's free server to cut costs, and within an hour the same prompts that worked on a paid endpoint started returning empty completions and hanging requests. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The real culprits were a context-window overflow and a dead keep-alive connection, and this is the retrospective of how I found both.

The Symptom

Here is the exact symptom, because vague bug reports waste everyone's time. My batch job would process about forty requests, then suddenly return a 200 response with an empty content field for the next three or four. After that, the client would hang for exactly thirty seconds before raising a ReadTimeout, and the whole run would fail on the last chunk. The strange part was that the same code had been stable for weeks on a paid endpoint, so my first instinct was to blame the new free tier.

Step 1: Write a Probe That Varies One Variable

I forced myself to stop guessing and wrote a minimal probe that isolates the variable I suspected most: input length. The script sends the same trivial prompt, padded with filler characters, and records status, latency, and whether the expected token appears in the reply. Run it against any OpenAI-compatible endpoint, and you get a table that separates a sick server from a sick request.

import asyncio
import time

import httpx

PROMPT = "Reply with exactly the word OK and nothing else."

async def probe_once(client, url, headers, context_chars):
    payload = {
        "messages": [
            {"role": "user", "content": "x" * context_chars + "\n" + PROMPT}
        ],
        "max_tokens": 16,
    }
    t0 = time.perf_counter()
    try:
        r = await client.post(url, json=payload, headers=headers, timeout=30)
        latency_ms = round((time.perf_counter() - t0) * 1000)
        return {
            "context_chars": context_chars,
            "status": r.status_code,
            "latency_ms": latency_ms,
            "ok": "OK" in r.text,
            "bytes": len(r.text),
        }
    except Exception as exc:
        return {
            "context_chars": context_chars,
            "status": "error",
            "error": type(exc).__name__,
        }

async def main():
    url = "YOUR_ENDPOINT/v1/chat/completions"
    headers = {"Authorization": "Bearer YOUR_KEY"}
    async with httpx.AsyncClient() as client:
        for size in [100, 1_000, 5_000, 10_000, 20_000]:
            result = await probe_once(client, url, headers, size)
            print(result)

if __name__ == "__main__":
    asyncio.run(main())

The output told a story that the logs had hidden. At 100 and 1,000 characters, every request returned OK in under a second. At 10,000 characters, the status was still 200, but the content field was empty and the latency jumped to nine seconds. At 20,000 characters, the request hung and then raised ReadTimeout, which is exactly what my batch job had been showing me all along.

Step 2: Suspect the Wrong Thing First

Naturally, I suspected the free tier first, and that suspicion was wrong in an instructive way. The probe showed that the endpoint itself was healthy at small context sizes, so the problem scaled with my input, not with the server's mood. That single observation killed the "free tier is flaky" theory and pointed me straight at the context window.

Step 3: Find the Real Bugs

The real bug was in my harness, and it had been there since day one. I was concatenating a large system prompt, a few few-shot examples, and the user message into a single string, and the paid endpoint I used before had a context window large enough to hide the mistake. The free model's effective window turned out to be smaller, so the overflow surfaced as an empty completion instead of an error, because the API returned 200 with a finish_reason my code never checked.

The second bug was connection reuse, and it only appeared after the first fix. My client kept a persistent HTTP connection between batches, and the free server restarted containers during idle periods, which silently killed that connection. The next request then waited on a socket that would never answer, which is why I saw a clean thirty-second timeout instead of an instant error.

Step 4: Fix the Harness, Not the Host

Two changes fixed the pipeline, and neither involved buying anything. First, I added a token-budget function that trims the system prompt and the few-shot examples before they reach the API, so the total request stays under a conservative limit. Second, I made the client retry on connection errors with exponential backoff, while treating HTTP 200 with an empty completion as a hard failure that needs inspection, not a transient blip.

def fit_context(messages, budget_chars):
    """Drop the oldest non-system messages until the payload fits."""
    kept = [messages[0]]  # always keep the system prompt
    for msg in messages[1:]:
        candidate = kept + [msg]
        if sum(len(m["content"]) for m in candidate) <= budget_chars:
            kept = candidate
        else:
            break
    return kept

The Reusable Checklist

Here is the reusable checklist I now run before blaming any hosting tier, and it takes about fifteen minutes.

Reproduce with a probe that varies a single variable, like input length or batch size.
Record status, latency, and response bytes, because a 200 with an empty body is a different disease from a 500.
Check finish_reason or the equivalent field before trusting the output.
Separate connection errors from timeout errors in your retry logic, because they have different causes and different cures.
Only then look at the provider's status page, and even then treat it as a last resort rather than a first excuse.

Symptom	Most likely cause	First check
HTTP 200, empty content	Context overflow or content filter	Reduce input, inspect `finish_reason`
Hang, then `ReadTimeout`	Dead keep-alive connection	Retry on connection errors only
Valid JSON, cut-off text	`max_tokens` too small	Raise `max_tokens`, check `finish_reason`
HTTP 429	Rate limit	Exponential backoff plus caching

Why I Stayed on the Free Server

I did not abandon the free server after this incident, and that decision deserves an honest explanation. MonkeyCode is an open-source project, so the server setup is documented in the repo, and the free model access plus a 10M token allowance (at the time of writing, so check the repo for current numbers) made it painless to run experiments that would otherwise sit in my backlog. The lesson was never "free hosting is unreliable"; it was that my code needed to state its assumptions about context and connections explicitly.

Limitations

Who should not copy this approach? If your workload is a single long-running streaming request, the probe sweep and the trim function will not save you, because your problem is latency, not context overflow. This method also assumes the API returns structured errors, so if your provider silently drops requests, you need packet-level tracing instead. And I did not benchmark the free server against paid hosts, because that comparison depends on your region, your payload, and the current load, so treat any numbers you see elsewhere as anecdotes until you run your own probe.

The Takeaway

If you are debugging a flaky LLM pipeline right now, the probe above is a fair starting point, and MonkeyCode's free server is a reasonable place to run it against a real endpoint without opening a wallet. The free tier was never the enemy. My assumptions about context windows and connection lifetimes were, and now the probe catches both before they cost me another evening.

Three Nights of Empty LLM Responses — and the Isolation Trick That Finally Caught the Culprit

Riley Li — Sat, 22 Aug 2026 15:03:49 +0000

After three nights of watching my eval harness log empty responses that the API insisted were successful, I was ready to blame the model. The model was innocent. The real culprit turned out to be a stale TCP connection, a VPN proxy with a short patience, and one overly broad except Exception that quietly converted a crash into a blank string. This post walks through the symptom, the wrong turns, and the isolation trick that found the root cause in about twenty minutes.

The symptom that made no sense

Every night at 2 AM, a batch job sends a few hundred prompts to an LLM endpoint and scores the replies. Around seven percent of them came back as empty strings, always clustered in the middle of the run, never at the start. The HTTP status was 200, the JSON parsed cleanly, and the choices array was simply empty. My first instinct was to blame the model, because an empty success is the most confusing failure mode an API can produce.

The wrong turns I took first

I swapped prompts, shortened them, and padded max_tokens to give the model more room. Then I blamed rate limits and added retries with exponential backoff, which made the failure rate worse because every retry reused the same poisoned connection. I even suspected my scoring code, but the raw logs showed the empty choices array before any scoring logic ran. That was the moment I realized I had been guessing instead of isolating.

The isolation trick

Step one was building a minimal repro that stripped away every layer of my harness. No database, no scoring, no retries — just a loop that posts one prompt and prints the raw response shape.

import httpx

client = httpx.Client(timeout=30.0)
for i in range(20):
    r = client.post(
        "https://api.example.com/v1/chat/completions",
        json={
            "model": "some-model",
            "messages": [{"role": "user", "content": "Say hello in one sentence."}],
            "max_tokens": 20,
        },
    )
    body = r.json()
    print(i, r.status_code, body.get("choices"))

Step two was changing exactly one variable at a time. Same script, same prompt, same model: from my laptop with the VPN on, it failed roughly one time in fifteen. From a colleague's machine on a completely different network, it passed every single time. That pointed hard at my network path, but I still needed a second independent environment to be sure the model endpoint was not somehow involved.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. That context matters here because MonkeyCode's free model access and free server option gave me precisely that second environment — a different endpoint to rule out model-side behavior, and a separate server to rule out my laptop's network stack. A few hundred requests from that clean box produced zero empty responses, which told me exactly where to keep digging.

The actual root cause

The proxy in front of my VPN kills idle keep-alive connections after about sixty seconds. My HTTP client kept a pooled socket open between requests, and when the proxy silently closed it, the next request rode a dead connection into an error page that the gateway returned as a 200 with an empty choices array. My harness then indexed into body["choices"][0], raised IndexError, and the broad except Exception turned that crash into text = "". Every retry reused the same dead socket, which is why backoff made things worse instead of better.

The fix, in three small changes

# Before: one shared client, broad except, silent empty string
for prompt in prompts:
    try:
        body = client.post(...).json()
        text = body["choices"][0]["message"]["content"]
    except Exception:
        text = ""

# After: fresh connection, narrow except, loud failure
for prompt in prompts:
    with httpx.Client(timeout=30.0, headers={"Connection": "close"}) as c:
        resp = c.post(...)
        resp.raise_for_status()
        body = resp.json()
    if not body.get("choices"):
        raise RuntimeError("empty choices array — do not swallow this")
    text = body["choices"][0]["message"]["content"]

First, narrow the exception handling so a malformed response crashes the run instead of becoming a silent empty string. Second, validate the response shape before you parse it, because choices can be present but empty. Third, stop reusing pooled connections in long batch jobs, or set an idle timeout shorter than whatever proxy sits in front of you.

The reusable checklist

Reproduce with the smallest script that still triggers the failure, and print the raw response shape, not just the text.
Change one variable at a time — network, endpoint, client, prompt — and keep a log of which combination fails.
Move the repro to a second environment, ideally one with a different network path, to separate transport problems from model problems.
Log status codes and response bodies at the boundary of your code, before any parsing or scoring can hide the evidence.
Never catch broad exceptions around parsing; let a malformed response fail loudly so you cannot miss it.

Who should not copy this approach

If your failures are about output quality — wrong answers, hallucinations, format drift — a connection fix will not help you, and this checklist will waste your evening. A clean scratch server is also the wrong place to benchmark latency or throughput, because your real users will never share that network path. Use the free tier for isolation experiments, not for production assumptions, and treat every environment as a suspect until proven otherwise.

So the next time your LLM pipeline returns something that looks like a model problem, ask yourself a rude question: did I actually see the raw response, or did my client swallow the evidence? That question saved me three more nights, and I would honestly love to hear what your empty-response culprit turned out to be.

Minimax H3 Is Having a Moment. My Prompt Set Doesn't Care.

Riley Li — Fri, 14 Aug 2026 12:07:24 +0000

This morning my feed looked like everyone had the same assignment. Minimax H3 every other post. Cheaper. Better. More open. I won't pretend the hype didn't hit. It always hits. I started opening tabs.

Then I caught myself. I know that pattern too well. A new release shows up, the threads say cheaper and better, and I start adjusting prompts that worked before the hype arrived. So before I argue about Minimax H3, I run it through the one thing I trust: my own tiny prompt set.

My prompt set is boring and personal

It's not a benchmark. It's eight cases I keep around because they resemble real work:

repair malformed JSON without dropping keys
turn a rambling function into plain logic
write SQL for duplicate emails
summarize an error log in three lines
refuse a bad request without a lecture
call a tool instead of explaining it
keep the middle of a long instruction
stay under a loose latency budget

Boring? Yes. That's the point. A model can win a leaderboard and still mangle my JSON repair case. I don't need it to be brilliant. I need it to not ruin my Tuesday.

The runner is provider-agnostic on purpose

I wrote a small runner where the provider is just a function. I don't want my test to depend on any vendor's SDK or dashboard. That way the value is in the test set, not in the integration.

Here's the skeleton I've been using. I've removed my scoring helpers so it stays readable:

from dataclasses import dataclass

@dataclass
class SmokeCase:
    name: str
    prompt: str
    signal: str

SMOKE_SET = [
    SmokeCase(
        'json_repair',
        "Fix this and return only valid JSON: {'a': 1, 'b': [2, 3,}",
        'Output starts with { and parses as JSON',
    ),
    SmokeCase(
        'tool_call',
        "Use get_weather(city). What's the weather in Lisbon?",
        'Response is a tool call, not a paragraph',
    ),
    SmokeCase(
        'long_middle',
        'List three rules, then output the middle rule after this long filler: ignore everything before the marker. The middle rule is: never log plaintext passwords.',
        'Mentions passwords, not the filler',
    ),
]

def run_smoke(case, provider):
    reply = provider(case.prompt)
    return {'case': case.name, 'reply': reply, 'signal': case.signal}

I run this with temperature 0 and the exact same prompt every time. If I change one word, I've broken the comparison. This is the part people skip when they're excited. It's also the part that keeps me honest.

Free access is what makes this sustainable

Here's where MonkeyCode's free model access and free server option become useful to me.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

I'm not using free access to try every model on the internet. I'm using it to run the same boring smoke set without deciding whether each release is worth my API budget. That matters because I used to skip the negative cases when testing cost money. I'd run the happy path and tell myself the rest was probably fine. The refusals, the malformed JSON, the long prompts — those are exactly where models surprise me.

The free server option makes the workflow easier to share. I can put this tiny runner somewhere without spinning up a paid box, point it at a provider, and get a yes/no answer. Same code, same cases, different model. That feels closer to how open-source tools are supposed to work: small, replaceable, and easy to run yourself.

"Open-source spirit" means replaceability, not a sticker

I keep hearing open used as a vibe, so let me say what I mean by it.

For a model release, I care about the part that changes my daily workflow:

Can I run it through a code path I control?
Can I swap it out without rewriting the runner?
Can I see where it fails instead of guessing from a table?
Can another developer reproduce my test without buying something first?

That's it. Not a license argument. Not a philosophy lecture. Just the practical ability to replace a component and observe what happens.

When a thread only tells me the model is cheaper and better, I have nothing to reproduce. When a free option lets me run my own cases and watch it fail, I can make an actual decision.

What I won't claim

This smoke set won't predict your production traffic. Eight cases is eight cases, not a benchmark. Free access doesn't mean unlimited, and I don't have visibility into quotas, SLAs, or long-term availability. I'm not saying any specific model passed or failed. One quiet afternoon run is not science.

Skip this approach if you need hard latency guarantees, compliance approval, or production support. A free tier is not a contract. If you need a global ranking, this won't give you one. It gives you something smaller but more useful: a repeatable signal about whether a model fits your work.

My takeaway

The Minimax H3 moment will pass. Next week there will be another one. My prompt set won't care which release is trending. It will just keep catching the same practical failures — and that's exactly why I keep running it.

If you maintain a tiny model smoke set, what's in it? I especially like the boring, personal cases that never make it into benchmarks.

A Credit-Card-Free Triage Bot for Noisy CI Logs

Riley Li — Fri, 14 Aug 2026 10:49:53 +0000

I used to start my week by deleting CI emails.

Not reading them—deleting. There were too many. A broken container here, a flaky UI test there, the same timeout from a third-party API. Somewhere in that pile was a real failure I actually needed to fix, but finding it felt like panning for gold in a river of noise.

So I wanted a small triage bot.

Not an autonomous agent that fixes my pipeline. I don't trust anything to start editing workflows after three failed runs. I wanted something much dumber: a service that reads a failed CI log and puts it into one of three buckets.

flake: the test failed on timing, an assertion order, or a retry would probably pass.
infra: the runner died, the network timed out, or a dependency couldn't be pulled.
code: the build or test surfaced something a developer should look at.

That's it. Three labels. No auto-remediation. No magic.

The problem was infrastructure. I didn't want to host another service, open a cloud account, or tie my personal credit card to something I'd use twice a week. My local machine could run a small model, but I didn't want to give up the RAM every time a webhook arrived.

That's where the free model and free server piece came in.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode's free model access and a free server option are the two pieces I used to build this workflow.

I'm going to keep the model endpoint and server details generic. Free-tier names, tokens, and runtime limits change, and I'd rather show the pattern than tie it to a signup flow that might be different next month. If you want to reproduce this, plug in your own free model URL and free server host.

The shape of the workflow

Instead of paying for a monitoring dashboard, I wired up a tiny service.

CI sends a webhook on failure.
The free server receives the log excerpt.
It scrubs obvious secrets before forwarding anything.
The free model reads the log and returns one of the three labels.
The server stores the result and exposes a simple /last endpoint.

The key is that each step is replaceable. The webhook can come from GitHub Actions, Jenkins, CircleCI, or a cron job that checks an API. The model doesn't need to be large. The task is classification on a short log, not code review.

The reusable part

Here's the server code I sketched. It's deliberately small, so you can paste it into app.py and run it with uvicorn app:app.

import os
from collections import defaultdict
import requests

from fastapi import FastAPI, Request
from fastapi.responses import JSONResponse

app = FastAPI()

MODEL_URL = os.getenv('MODEL_URL', '')
MODEL_TOKEN = os.getenv('MODEL_TOKEN', '')
MODEL_NAME = os.getenv('MODEL_NAME', '')
ALLOWED_LABELS = {'flake', 'infra', 'code'}
SENSITIVE_MARKERS = ['api_key=', 'apikey=', 'token=', 'secret=', 'password=']

results = defaultdict(int)

def scrub(text: str) -> str:
    for marker in SENSITIVE_MARKERS:
        start = 0
        while True:
            idx = text.lower().find(marker, start)
            if idx == -1:
                break
            end = idx + len(marker)
            while end < len(text) and not text[end].isspace() and text[end] not in '&;':
                end += 1
            text = text[:idx + len(marker)] + '[REDACTED]' + text[end:]
            start = idx + len(marker) + len('[REDACTED]')
    return text

def classify(log: str) -> str:
    if not MODEL_URL:
        return 'unknown'
    prompt = (
        'You are triaging a failed CI log. Classify the failure as exactly one of: '
        'flake, infra, code. Return only the label. Log excerpt: ' + scrub(log)[:4000]
    )
    resp = requests.post(
        MODEL_URL,
        headers={'Authorization': f'Bearer {MODEL_TOKEN}'} if MODEL_TOKEN else {},
        json={'model': MODEL_NAME, 'prompt': prompt, 'max_tokens': 8},
        timeout=20,
    )
    resp.raise_for_status()
    label = resp.json().get('choices', [{}])[0].get('text', '').strip().lower()
    if label not in ALLOWED_LABELS:
        return 'unknown'
    return label

@app.post('/triage')
async def triage(request: Request):
    body = await request.json()
    log = body.get('log', '')
    label = classify(log)
    results[label] += 1
    return JSONResponse({'label': label, 'counts': dict(results)})

@app.get('/last')
async def last():
    return JSONResponse(dict(results))

This is a sketch, not a copy-paste deployment. The payload format depends on the model provider, and some free servers force cold starts.

How I decided what counts as 'code'

A model will happily output code for any log if you let it, so a small decision table keeps the bot honest.

Signal in log	Suggested label	Why
`Timed out`, `retry`, `flaky`, `expected X got Y` on UI test	`flake`	Often a timing or DOM assertion issue, not a logic bug.
`Docker pull`, `network unreachable`, `503`, `memory limit`, `runner version`	`infra`	The failure happened before your code could be judged.
`TypeError`, `ImportError`, `AssertionError`, stack trace in your package	`code`	The code itself broke.

You can turn this into a second pass: if the model says code but none of the code signals are present, drop the label to unknown.

The sanity check I'd run before trusting it

I would not deploy this on a real repo without a tiny ground-truth set.

Take 20 old failed logs.
Label them myself: 8 flake, 8 infra, 4 code.
Run the triage endpoint and compare.
Require 80%+ agreement before it gets to touch my inbox.
Failing that, narrow the prompt or shorten the log excerpt.

Where this falls apart

Free model access and free servers are great for this, but they're not a production control plane.

Free servers may sleep, cold start, or get recycled. Your webhook can time out.
Free model quotas can be rate-limited, so a sudden burst of failing jobs may drop requests.
The model can still hallucinate a label. Bad classification is annoying; worse, it can teach you to ignore the bot.
Logs aren't free to share. Even with scrubbing, there can be repo names, usernames, IPs, and code snippets inside the trace.

I treat this as a noise filter, not a decision maker. It changes where I look first; it doesn't tell me what to ignore.

Who should skip this

If your logs contain regulated data, don't send them to a free model.
If you need an audit trail, make sure your logs and model calls are retained; a stateless free service won't do that by default.
If you already have a paid CI analytics tool, this probably duplicates part of it.
If you want an agent that auto-fixes failures, this is not that. It only labels.

If you have a free model endpoint and a free server lying around, start with the 20-log sanity check. It's the cheapest way to find out whether the signal is real before you bolt on more machinery.

Every Week a New Model Is "Cheaper and Better" — Here's the 30-Minute Harness That Settles It

Riley Li — Thu, 13 Aug 2026 03:50:30 +0000

Somewhere right now, a model release thread is telling you that a new checkpoint is dramatically cheaper and noticeably better than whatever you're running. Next week there will be another one. The claims aren't always wrong — that's what makes them dangerous. Sometimes the new model really is better for the benchmarks the poster ran, and quietly worse for the eleven tasks you actually do every day.

I got tired of making default-model decisions off vibes, so I built a tiny A/B harness. It takes about 30 minutes to set up, runs on a fixed set of prompts drawn from my own real work, and ends in a one-line decision: switch, stay, or split. This post is that harness, plus the decision table I use to read the results. It's a workflow, not a benchmark — your prompt set is the whole point.

Why your own prompts beat public scores

Public evals measure what eval authors care about. My day is a narrow distribution: refactoring mid-size Python diffs, writing migration scripts, summarizing error logs, and generating test scaffolding. A model can win every leaderboard and still mangle my SQL migrations, because migrations reward tedious exactness, not cleverness.

So the harness starts with 20–30 prompts I have actually sent in the last month, each with a short note on what a good answer looks like. Not formal unit tests — just enough to score consistently:

# eval_set.yaml — prompts from real work, not from a benchmark
- id: migration-01
  task: "Convert this Django migration to raw SQL for Postgres 15"
  pass_if: "runs without error; preserves the index; no ORM left in output"
- id: log-summary-04
  task: "Summarize this 400-line traceback into 3 bullet points"
  pass_if: "identifies root cause line; no invented stack frames"
- id: refactor-02
  task: "Split this 200-line function without changing behavior"
  pass_if: "same public signature; all existing tests referenced by name"

The runner

The runner hits any OpenAI-compatible chat endpoint twice — once with my current default model, once with the candidate — and dumps raw outputs side by side for scoring. Label this a template: fill in your own endpoints and keys, and expect to tweak it.

# ab_harness.py — minimal template, not production code
import json, time, urllib.request, yaml

def chat(base_url, api_key, model, prompt):
    req = urllib.request.Request(
        f"{base_url}/chat/completions",
        data=json.dumps({
            "model": model,
            "messages": [{"role": "user", "content": prompt}],
            "temperature": 0.2,          # keep it low so runs are comparable
        }).encode(),
        headers={"Authorization": f"Bearer {api_key}",
                 "Content-Type": "application/json"},
    )
    t0 = time.time()
    with urllib.request.urlopen(req, timeout=120) as r:
        body = json.load(r)
    return body["choices"][0]["message"]["content"], round(time.time() - t0, 2)

def main():
    cases = yaml.safe_load(open("eval_set.yaml"))
    results = []
    for c in cases:
        row = {"id": c["id"], "pass_if": c["pass_if"]}
        for label, model in [("current", "my-current-model"),
                             ("candidate", "new-candidate-model")]:
            out, secs = chat("https://your-endpoint/v1", "YOUR_KEY", model, c["task"])
            row[label] = out
            row[f"{label}_secs"] = secs
        results.append(row)
    json.dump(results, open("ab_results.json", "w"), indent=2)
    print(f"scored {len(results)} cases -> ab_results.json")

if __name__ == "__main__":
    main()

Two deliberate choices here: temperature pinned low so differences come from the model rather than sampling luck, and wall-clock time recorded per call so the "cheaper" claim has to survive contact with "slower." A model that costs 40% less but takes 3× longer on log summaries is not cheaper for a workflow where I'm waiting on it.

Scoring without lying to yourself

I score each pair blind when I can — shuffle the outputs, grade against the pass_if note, then unblind. Each case ends in one of three states: current wins, candidate wins, or tie. Then the decision table:

Outcome across your prompt set	Decision
Candidate wins ≥ 40% and loses ≤ 10%	Switch default, keep old model config for a week
Candidate wins on one task family only (e.g., summaries)	Split: route that task, keep default elsewhere
Within ±10% overall	Stay. Migration cost exceeds any saving
Candidate faster but ties on quality	Only switch if latency is your actual bottleneck

The "stay" row is the one hype threads never mention. Even a genuinely better model has switching costs: different failure modes to relearn, prompt phrasing that no longer applies, edge cases you had already patched around. A marginal quality win doesn't cover that.

Running candidates without a billing surprise

There's a practical problem with this harness: it only works if you can actually run the candidate model on your 25 prompts, repeatedly, at temperature 0.2, without watching a meter. Paid API access makes "just test it yourself" an expensive hobby when a new release shows up every week.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

This is the specific spot where MonkeyCode fits the workflow: it offers free model access plus a free server option, which is exactly what a release-day A/B harness wants — a place to stand up the candidate model and throw your prompt set at it without provisioning anything or doing cost arithmetic first. I treat it as the candidate lane in the script above, while the current-default lane stays wherever it already lives. I wouldn't run production traffic through a free tier, but for evaluation traffic that's precisely the right tool: ephemeral, low-stakes, and disposable.

Limitations, and who shouldn't do this

25 prompts is a smell test, not a measurement. It catches "obviously worse at my work" and "obviously better," which is usually all a default-model decision needs. It will not detect a 3% regression.
No load, no concurrency, no long-context stress. If you're choosing a model for a production service rather than a personal default, you need real latency-percentile and cost-per-million-token testing; this harness is the wrong tool.
Free access changes. Don't build a repeatable CI eval on top of any free tier without a fallback plan, and re-verify what's actually offered before each release cycle.
Blind scoring by hand doesn't scale. Past ~50 cases, add an LLM judge with a fixed rubric — and then distrust the judge a little too.

If you're in a regulated domain, or your "prompt set" contains anything sensitive, sanitize before anything touches a third-party endpoint — free or otherwise.

The actual habit

The harness matters less than the ritual: when a release thread claims "cheaper and better," the response is no longer reading forty replies — it's spending 30 minutes running the set and letting the table decide. Most weeks the answer is "stay," and that answer, backed by your own outputs instead of someone else's screenshots, is worth the half hour. If you try this, steal the decision table before you steal the script — the script is replaceable, but having a written rule for switch, stay, or split is what keeps release hype from making your architecture decisions for you.