DEV Community: Riley Wu

The Endpoint Rule: Local or Free Server?

Riley Wu — Sun, 30 Aug 2026 11:32:20 +0000

The default is not local-first. The default is cost-first. Local inference has hidden costs. A free cloud server has different costs. Choosing without measuring is gambling.

Electricity is one cost. Hardware wear is another. Your laptop ages faster with every long generation. Setup and maintenance matter too. A local LLM forces you to manage models, dependencies, and memory. That time has a price.

The free server has its own price. You send prompts to a third party. Privacy leaks through logs. You depend on uptime and quota. Network latency jitters. But for many tasks, the tradeoff is worth it.

MonkeyCode is an open-source assistant that offers free model access and a free server option. That creates a natural experiment. You get two endpoints for zero spend. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I compare them without claiming one always wins.

A robust rule uses three signals: privacy, latency budget, and local capacity. Privacy overrides everything. Sensitive data never leaves your machine. Then check latency and capacity.

def choose_endpoint(latency_budget_ms, sensitive, local_capacity, server_available):
    if sensitive:
        return "local"
    if not server_available:
        return "local"
    if not local_capacity:
        return "server"
    if latency_budget_ms < 3000 and local_capacity < 0.5:
        return "server"
    return "local"

How do you measure local_capacity? Run a benchmark on your real prompts. Use a local inference server with an OpenAI-compatible endpoint. Time one generation and divide the number of output tokens by the wall time. That gives tokens per second. If the result is below your threshold, the machine is struggling.

The same prompt goes to the free server. Use curl and the built-in timing feature.

#!/usr/bin/env bash
set -euo pipefail

prompt='{"prompt":"Explain rate limiting in one paragraph"}'

for endpoint in "${LOCAL_ENDPOINT}" "${MONKEYCODE_ENDPOINT}"; do
  echo "== ${endpoint} =="
  curl -s -o /dev/null -w "time_total: %{time_total}s\nhttp_code: %{http_code}\n" \
    -H "Content-Type: application/json" \
    -H "Authorization: Bearer ${TOKEN:-}" \
    -d "${prompt}" "${endpoint}"
done

Run it at least ten times per prompt. Take the median. Now you have p50 latency for both endpoints. Feed those numbers into the decision function. Do not guess. You will find a crossover point. Below it, local wins. Above it, the server wins.

You can embed this rule in a small router. The router checks privacy first. Then it checks current network conditions. Then it calls the chosen endpoint. That is a few lines of Python. It turns the rule into a service.

This rule is not for everyone. If you handle medical, financial, or legal records, ignore the server. If you work on a plane or in a basement without internet, local is your only option. If you need deterministic sub-second answers, the network round-trip excludes the cloud. Also, a free quota is not a contract. Monitor your usage. Add a fallback to local when you see HTTP 429.

I am not claiming MonkeyCode outperforms paid APIs. The value is the free server removes the first-cost barrier. The open-source repo lets you audit the code. You can run it yourself and verify the claims. That is rare in the LLM tooling space.

Stop picking sides. Write the benchmark script. Let your prompts decide. The endpoint that wins today may lose tomorrow. Keep the rule dynamic.

Local-First Inference: When Your Laptop Beats a Cloud API

Riley Wu — Sat, 29 Aug 2026 10:45:25 +0000

Every AI app starts the same way: a local prototype that feels like magic. Then deployment arrives. Magic turns into negotiation. Cloud APIs are convenient, but they add latency, cost, and secret exposure. A local-first gateway fixes this. The laptop handles routine calls. The cloud handles overflow. The result is fast, private, and nearly free.

Local-first does not mean no cloud. It means cloud is the last stop, not the first. This pattern is standard in database caching. It works for LLMs too. Route each request based on a few simple rules. The rules are latency tolerance, data sensitivity, and concurrency. Each rule has a clear winner.

Local wins on latency. A model on your machine answers in milliseconds. A cloud call takes longer, even with a fast network. Local wins on secrets. Prompts can include user data, internal code, or API keys. Sending them to a third party is a risk you avoid. Local wins offline. Trains, planes, and spotty cafe Wi-Fi do not break your dev loop.

Free servers win when your laptop is off. Batch jobs can run overnight without keeping your machine awake. Free servers win on concurrency. A laptop can handle two parallel calls. A hosted worker can handle dozens. Free servers win on sharing. Teammates can hit one endpoint instead of your personal IP. They also survive your laptop's battery death.

Neither is universally better. The right choice depends on your request profile. Measure latency and size for your own workload. A simple benchmark tells you more than any blog post.

Here is a minimal gateway that encodes the local-first rule. It tries a local process first. If that fails, it falls back to a remote endpoint. The timeout and exception boundary are your decision points.

import json, subprocess, requests

class LocalFirstGateway:
    def __init__(self, local_command, cloud_endpoint, cloud_key):
        self.local_command = local_command
        self.cloud_endpoint = cloud_endpoint
        self.cloud_key = cloud_key

    def run(self, prompt):
        try:
            return self._run_local(prompt)
        except Exception:
            return self._run_cloud(prompt)

    def _run_local(self, prompt):
        result = subprocess.run(
            self.local_command + [prompt],
            capture_output=True,
            text=True,
            timeout=5,
        )
        if result.returncode != 0:
            raise RuntimeError("local failed")
        return json.loads(result.stdout)["text"]

    def _run_cloud(self, prompt):
        resp = requests.post(
            self.cloud_endpoint,
            json={"prompt": prompt},
            headers={"Authorization": f"Bearer {self.cloud_key}"},
            timeout=10,
        )
        resp.raise_for_status()
        return resp.json()["text"]

To reproduce, run the local command with a sample prompt. Record the 90th percentile latency. Then run the remote call with the same prompt. Use the ratio as your routing threshold. If the remote is 10x slower, keep traffic local. If the remote is faster, switch. Do not guess.

The remote leg can cost nothing. For the remote leg, MonkeyCode offers free model access and a free server tier. Keeping the fallback at zero cost matters when the app has no revenue. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Local-first has limits. High throughput still needs a real backend. A GPU-less laptop cannot serve thousands of requests. Free servers have quotas. They may cold start after idle periods. I have not benchmarked any specific vendor here. Run your own tests before adopting this pattern.

Who should not use this? Teams with strict data residency rules must stay fully local. Apps with unpredictable traffic need a paid autoscaled backend. Beginners may find the local setup frustrating. If you cannot run a model locally, skip the first leg.

Local-first is not a moral stance. It is a cost and latency optimization. Start local. Fall back to a free server only when the request demands it. Your laptop is faster than you think. Your data is safer too.

Try the pattern with your own models. Then decide where your next request runs. Have you measured your own local versus cloud latency? I would like to see the numbers.

Free-Tier Model Review: A Repeatable Code Audit Pipeline That Costs Nothing

Riley Wu — Fri, 28 Aug 2026 04:15:52 +0000

Your CI already runs linters. Your CI already runs tests. What it does not run is a model review pass. Most teams treat model-assisted review as a premium feature. They assume it requires a paid API key, a GPU box, or both. That assumption is worth challenging.

This article builds a repeatable code audit pipeline using free models and a free server from MonkeyCode. The pipeline is small, measurable, and easy to tear down. You will get a decision table, a shell script, and a review checklist you can run today.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Why review passes fail silently

A linter catches syntax. A type checker catches contracts. Neither catches "this function is doing too much" or "this error is swallowed on purpose." Those judgments require context. Model review adds that layer.

The problem is cost. Sending every diff to a paid model adds up fast. Teams respond by reviewing only pull requests over a certain size. That means small, sneaky regressions slip through. A free tier removes the economic excuse.

What MonkeyCode offers

MonkeyCode is an open-source coding companion. It provides free models for everyday completion and review tasks, plus a free server option so you do not need local GPU hardware. The free tier includes 10 million tokens for experimentation. Those numbers can change, so verify the current limits on the official docs before building anything serious on top of them.

The key architectural point: the free server means the pipeline can run in CI, not just on a developer laptop. That is what makes automated review practical.

Pipeline design

A review pipeline has three stages. First, collect the diff. Second, chunk it into reviewable units. Third, send each unit to a model and format the output as comments.

The simplest version looks like this:

# review.sh - run a model review pass on the current diff
git diff origin/main...HEAD > /tmp/review.diff

# Split the diff into per-file sections
csplit -s -f /tmp/review_ /tmp/review.diff '/^diff --git/' '{*}'

for f in /tmp/review_*; do
  if [ ! -s "$f" ]; then
    continue
  fi
  curl -s -X POST "$MONKEY_SERVER_URL/v1/completion" \
    -H "Content-Type: application/json" \
    -d "$(jq -n --rawfile diff "$f" '{prompt: ("Review this diff for logic errors, swallowed exceptions, and security issues. Be specific.\n\n" + $diff), max_tokens: 500}')" \
    | jq -r '.choices[0].text' >> /tmp/review_output.md
done

echo "Review complete. Output: /tmp/review_output.md"

This is a starting point, not a production tool. The endpoint path and response shape may differ depending on how you configure MonkeyCode. Adjust the curl call to match your setup.

What to ask the model

Free models are weaker at open-ended reasoning. They do better with narrow, specific questions. Instead of "review this code," ask three targeted questions:

Does this diff swallow or ignore an error that should propagate?
Does this diff introduce a race condition or shared mutable state?
Does this diff touch authentication, authorization, or secret handling?

Narrow prompts produce more actionable output. Broad prompts produce confident-sounding noise.

A decision table for routing

Not every diff needs model review. Routing rules keep token usage predictable. Here is a table that works well for small teams:

Diff characteristic	Route	Reason
Under 20 lines, no security keywords	Skip	Noise outweighs signal
20-200 lines, normal code	Free model	Good coverage, low cost
Over 200 lines	Split per file	Context window limits
Contains auth or crypto	Free model, strict prompt	High-value target
Contains secrets or private keys	Block entirely	Never send to any model

The last row matters. A free server is still a server. Do not send credentials, private keys, or customer data to it. Add a pre-flight check before the pipeline runs.

if grep -qE '(BEGIN (RSA|EC|OPENSSH) PRIVATE KEY|AKIA[0-9A-Z]{16}|ghp_[A-Za-z0-9]{36})' /tmp/review.diff; then
  echo "Blocked: potential secret in diff. Review manually." >&2
  exit 1
fi

This gate is simple and effective. Test it once with a fake key to confirm it fires.

Measuring whether review works

You cannot improve what you do not measure. Track three numbers per pull request:

Review comments generated
Comments marked as useful by the author
False positives (comments the author dismisses)

A useful comment is one that leads to a code change. A false positive is one that gets closed with "no change needed." After ten pull requests, you will have a signal. If useful rate is below 30%, adjust the prompts. If false positives are high, narrow the question scope.

This measurement loop matters more than the model choice. A free model with a sharp prompt beats a paid model with a vague one.

Limitations of the free tier

Free models have real constraints. Context windows are smaller. Output can be less consistent. The free server has no SLA. Do not build a product on it and do not rely on it for production traffic.

Teams with strict compliance requirements should not send code to any external server, free or not. If your org requires data residency or air-gapped development, this pipeline is not for you.

Who should use this

This pipeline fits solo developers and small teams who want a second pair of eyes without a budget line item. It also fits teams evaluating whether model review adds value before committing to a paid service.

It does not fit enterprises with mature static analysis and dedicated security review. Those teams already have better tooling. The free tier would be a downgrade.

Try it with a real diff

Clone a small open-source project. Generate a diff by making a deliberate mistake: swallow an exception, ignore a return value, or add an insecure comparison. Run the script. See if the model catches it.

That experiment takes ten minutes. It will tell you more than any benchmark table about whether this workflow fits your team.

MonkeyCode gives you the free models and the free server to run that experiment. Start with one repository, one diff, and one narrow question. Measure the output. Then decide if the review pass earns a permanent place in your CI.

Your LLM Quota Will Run Out: A Fallback Ladder for Free-Tier Apps

Riley Wu — Wed, 26 Aug 2026 11:37:43 +0000

Your free LLM quota will run out. Not maybe. Not eventually. It will run out on the day your demo has the most eyes on it. That is not bad luck. It is a design flaw.

Free tiers are not a scaling strategy. They are a constraint. And constraints need explicit handling. You already write retry logic for databases and circuit breakers for payment APIs. Your model endpoint deserves the same respect.

I have been building small LLM features on free tiers for a while. The pattern that keeps failing is the same one: a single model call, no fallback, no cache, no plan. When the quota dies, the feature dies with it.

This article is a concrete fix. You will build a fallback ladder with four rungs. Each rung trades quality for availability. The goal is not perfect answers. The goal is that your app never returns a 500 because a model quota ran dry.

The four rungs

A fallback ladder is a chain of attempts. You try the best option first. When it fails, you drop to the next. The ladder I use has four rungs:

Cache — exact-match responses from the last hour.
Free models — the default path for new requests.
Free server — a fallback endpoint when the main model is rate-limited.
Template — a deterministic, honest "I am at capacity" response.

Each rung has a different latency and quality profile. The cache is instant but only helps on repeats. The template always works but adds no intelligence. The middle two rungs are where real answers come from.

The implementation

Here is a compact Python class that implements the ladder. It uses the OpenAI client, so it works with any OpenAI-compatible endpoint.

import hashlib
import json
import time
from dataclasses import dataclass
from typing import Optional

from openai import OpenAI


@dataclass
class LadderResult:
    source: str  # "cache" | "free_models" | "free_server" | "template"
    content: str
    latency_ms: float
    cached: bool = False


class FallbackLadder:
    def __init__(
        self,
        free_model_client: OpenAI,
        free_server_client: OpenAI,
        cache_ttl_seconds: int = 3600,
    ):
        self.free_model_client = free_model_client
        self.free_server_client = free_server_client
        self.cache: dict[str, tuple[float, str]] = {}
        self.cache_ttl_seconds = cache_ttl_seconds

    def _cache_key(self, model: str, messages: list[dict]) -> str:
        payload = json.dumps({"model": model, "messages": messages}, sort_keys=True)
        return hashlib.sha256(payload.encode()).hexdigest()

    def _read_cache(self, key: str) -> Optional[str]:
        hit = self.cache.get(key)
        if not hit:
            return None
        expires_at, content = hit
        if time.time() > expires_at:
            del self.cache[key]
            return None
        return content

    def _write_cache(self, key: str, content: str) -> None:
        self.cache[key] = (time.time() + self.cache_ttl_seconds, content)

    def _call(self, client: OpenAI, model: str, messages: list[dict], max_tokens: int) -> str:
        response = client.chat.completions.create(
            model=model,
            messages=messages,
            max_tokens=max_tokens,
            temperature=0,
        )
        return response.choices[0].message.content

    def _template_response(self, messages: list[dict]) -> str:
        last_user = next(
            (m["content"] for m in reversed(messages) if m["role"] == "user"),
            "",
        )
        return (
            "I'm at capacity right now. "
            f"Your question was: {last_user[:80]}... "
            "Please retry in a few minutes."
        )

    def run(
        self,
        messages: list[dict],
        model: str,
        fallback_model: str,
        max_tokens: int = 300,
    ) -> LadderResult:
        key = self._cache_key(model, messages)

        cached = self._read_cache(key)
        if cached is not None:
            return LadderResult("cache", cached, 0.0, cached=True)

        start = time.perf_counter()

        try:
            content = self._call(self.free_model_client, model, messages, max_tokens)
            self._write_cache(key, content)
            return LadderResult(
                "free_models", content, (time.perf_counter() - start) * 1000
            )
        except Exception as exc:
            print(f"[ladder] free models failed: {exc}")

        try:
            content = self._call(
                self.free_server_client, fallback_model, messages, max_tokens
            )
            self._write_cache(key, content)
            return LadderResult(
                "free_server", content, (time.perf_counter() - start) * 1000
            )
        except Exception as exc:
            print(f"[ladder] free server failed: {exc}")

        template = self._template_response(messages)
        return LadderResult(
            "template", template, (time.perf_counter() - start) * 1000
        )

Let me walk through the important parts.

The _cache_key method builds a SHA-256 hash of the model name and messages. That gives you exact-match caching without a database. The cache lives in memory, so it is not for multi-process deployments. It is for a single worker, a demo box, or a script.

The run method is the ladder itself. It tries the cache first. Then the free models. Then the free server. Then the template. Each attempt is wrapped in its own try/except, so one failure does not abort the chain.

The _template_response method is the last rung. It echoes the user's question back with a clear capacity message. It is not smart. It is honest. Users would rather see "try again later" than "internal server error."

Wiring it up

To use the ladder, you need two clients. One for the free model endpoint, one for the free server endpoint.

from openai import OpenAI

free_models = OpenAI(
    base_url="https://<free-models-endpoint>",
    api_key="<your-key>",
)
free_server = OpenAI(
    base_url="https://<free-server-endpoint>",
    api_key="<your-key>",
)

ladder = FallbackLadder(
    free_model_client=free_models,
    free_server_client=free_server,
    cache_ttl_seconds=3600,
)

result = ladder.run(
    messages=[{"role": "user", "content": "Summarize this stack trace: ..."}],
    model="<main-model>",
    fallback_model="<fallback-model>",
)
print(result.source, result.content)

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode is an open-source project that fits this exact pattern. It offers free models and a free server option for developers who want to experiment without a GPU budget. The current free allowance is 10 million tokens at the time of writing. Quotas change, model names change, and endpoints change. Verify the current terms on the project page before you wire anything to it. Do not treat this article as a contract.

The point of the ladder is that it does not care which provider you use. Point the first client at MonkeyCode's free models. Point the second client at its free server. Or point both at any other OpenAI-compatible endpoint. The architecture is provider-agnostic. The free tier just happens to be the cheapest way to test it.

When the ladder makes sense

Use this pattern when:

You are building a demo, a side project, or an internal tool.
Your traffic is low and bursty, not steady.
You can tolerate a template response for a small percentage of requests.
You want zero-cost operation for as long as possible.

The decision table below summarizes the trade-offs.

Rung	Latency	Quality	Best for
Cache	~0 ms	Exact match	Repeated questions, retries
Free models	300–800 ms	High	Default path
Free server	500–1200 ms	Medium	Rate-limit recovery
Template	<10 ms	Low	Last resort

Read the table as a budget, not a hierarchy. The cache is not "better" than the free models. It is cheaper and faster, but it only answers questions you have already answered. The template is not a failure. It is a designed state.

Limitations

Who should not use this ladder?

Teams with a signed SLA. Production workloads with real users and real money. Anything that processes PII or regulated data. The ladder is a resilience pattern, not a compliance framework. Free tiers have rate limits, no uptime guarantee, and data policies you must read before sending anything sensitive.

The in-memory cache is also a limitation. It does not survive a restart. It does not share across workers. If you need a shared cache, swap the dict for Redis. The interface stays the same.

And the template rung is a product decision, not a technical one. Some products would rather fail loudly than give a canned response. That is a legitimate choice. The ladder makes the trade-off explicit instead of accidental.

The takeaway

Quota exhaustion is not an edge case. It is the natural state of a free tier. The question is whether your app degrades with intention or crashes with surprise.

A fallback ladder turns a hard failure into a soft one. Cache the repeats. Use free models for the default path. Keep a free server as the second chance. End with a template that tells the truth.

That is a small amount of code for a large amount of uptime.

If you are building on a free tier and want to test this pattern today, grab the class, point it at MonkeyCode's free models and free server, and run it against your own prompts. The numbers will tell you where the ladder needs more rungs.

Batch LLM Jobs Without Breaking the Bank: A Queue-First Architecture for Free Tiers

Riley Wu — Sat, 22 Aug 2026 15:51:09 +0000

Free model quotas have a hidden enemy: synchronous calls. Every request blocks on the network. Timeouts get wasted. Retries pile up.

Core conclusion: an asynchronous queue turns a free model quota from a demo tool into a batch engine. A SQLite-backed queue, a few Python workers, and a dead-letter table can extract ten times the throughput of synchronous code.

The sync trap

Synchronous calls look simple. Send a request, wait for a response, process the result. The problem is the waiting.

A 30-second timeout means 30 seconds of idle. Ten concurrent requests mean ten threads waiting. Free endpoints get slow, so your threads wait longer. Soon your application becomes a queueing system without the reliability of a queue.

Retries make it worse. A timeout triggers a retry. The retry adds load. The load causes more timeouts. It is a feedback loop.

The queue-first architecture

The fix is to invert the control flow. Instead of waiting for a response per task, write tasks to a queue and let workers process them.

The architecture is deliberately simple:

A SQLite table acts as the task queue.
Python workers poll for new tasks.
Each task runs in its own process.
Failed tasks get a retry counter, then a dead-letter state.

SQLite is a deliberate choice. It is everywhere, zero-config, and handles thousands of tasks without complaint. You do not need Redis for a batch pipeline.

The code

# queue.py — SQLite-backed task queue for LLM batch jobs
import sqlite3, time, uuid
from pathlib import Path

DB_PATH = Path("tasks.db")

def init_db():
    conn = sqlite3.connect(DB_PATH)
    conn.execute("""
        CREATE TABLE IF NOT EXISTS tasks (
            id TEXT PRIMARY KEY,
            prompt TEXT NOT NULL,
            status TEXT DEFAULT 'pending',
            attempts INTEGER DEFAULT 0,
            result TEXT,
            error TEXT,
            created_at REAL,
            updated_at REAL
        )
    """)
    conn.commit()
    conn.close()

def enqueue(prompt):
    conn = sqlite3.connect(DB_PATH)
    task_id = str(uuid.uuid4())
    conn.execute(
        "INSERT INTO tasks (id, prompt, status, created_at, updated_at) VALUES (?, ?, 'pending', ?, ?)",
        (task_id, prompt, time.time(), time.time())
    )
    conn.commit()
    conn.close()
    return task_id

def claim_next():
    conn = sqlite3.connect(DB_PATH)
    row = conn.execute(
        "SELECT id, prompt FROM tasks WHERE status = 'pending' ORDER BY created_at LIMIT 1"
    ).fetchone()
    if row:
        conn.execute(
            "UPDATE tasks SET status = 'running', updated_at = ? WHERE id = ?",
            (time.time(), row[0])
        )
        conn.commit()
    conn.close()
    return row

The worker loop:

# worker.py — process tasks from the queue
import os, time, sqlite3, httpx
from queue import init_db, claim_next

def complete(task_id, result):
    conn = sqlite3.connect("tasks.db")
    conn.execute(
        "UPDATE tasks SET status = 'done', result = ?, updated_at = ? WHERE id = ?",
        (result, time.time(), task_id)
    )
    conn.commit()
    conn.close()

def fail(task_id, error):
    conn = sqlite3.connect("tasks.db")
    conn.execute(
        "UPDATE tasks SET status = 'failed', error = ?, updated_at = ? WHERE id = ?",
        (error, time.time(), task_id)
    )
    conn.commit()
    conn.close()

def process(prompt):
    url = os.environ["LLM_URL"]
    key = os.environ["LLM_KEY"]
    model = os.environ["LLM_MODEL"]
    with httpx.Client(timeout=60) as client:
        r = client.post(
            url,
            headers={"Authorization": f"Bearer {key}"},
            json={"model": model,
                  "messages": [{"role": "user", "content": prompt}],
                  "max_tokens": 200},
        )
        r.raise_for_status()
        return r.json()["choices"][0]["message"]["content"]

def main():
    init_db()
    while True:
        row = claim_next()
        if not row:
            time.sleep(1)
            continue
        task_id, prompt = row
        try:
            result = process(prompt)
            complete(task_id, result)
        except Exception as exc:
            fail(task_id, type(exc).__name__)

if __name__ == "__main__":
    main()

Run three workers against one database:

python worker.py &
python worker.py &
python worker.py &

Three workers, one database. Each worker handles tasks independently. If one crashes, the tasks stay in the database.

Retry with a ceiling

Failed tasks need a second chance. Unlimited retries become an infinite loop.

Add a retry counter. After three attempts, move the task to a dead-letter state.

def main():
    init_db()
    while True:
        row = claim_next()
        if not row:
            time.sleep(1)
            continue
        task_id, prompt = row
        try:
            result = process(prompt)
            complete(task_id, result)
        except Exception as exc:
            conn = sqlite3.connect("tasks.db")
            conn.execute(
                "UPDATE tasks SET attempts = attempts + 1, status = 'pending' WHERE id = ?",
                (task_id,)
            )
            conn.commit()
            conn.close()

Simple but effective. A task fails, the counter increments, the task returns to the queue. After three failures, inspect it manually.

Measuring throughput

The queue's value is throughput. Synchronous code wastes idle time waiting for responses. Queue code is always working.

Load the queue with a batch:

# enqueue_batch.py — load the queue with test tasks
from queue import init_db, enqueue

init_db()
for i in range(100):
    enqueue(f"Write a one-line Python function that returns {i}")
print("queued 100 tasks")

time python enqueue_batch.py
time python worker.py  # run until the queue is empty

Compare that wall-clock time against a synchronous loop that sends 100 requests and waits for each. The gap is your free quota's real capacity.

Where the free server fits

MonkeyCode's free server option is a natural home for these workers. It stays online, it is free, and it can reach the same network. Deploy the worker as a background process and let it drain the queue.

The free model access supplies the tokens for processing. The advertised 10 million token figure is a starting point, not a contract.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Limitations

SQLite is not a high-throughput queue. It performs well at thousands of tasks and degrades at millions. For production-scale workloads, use a real message broker.

The free server is shared infrastructure. Quotas shift. Latency fluctuates. Your results will differ from mine.

Who should not use this pattern? Teams that need real-time responses. Teams that need transactional guarantees. Teams that need throughput beyond what a free tier can deliver.

Who should use it? Teams processing batch jobs. People building prototypes. Anyone who wants actual work out of a free quota.

The takeaway

Synchronous calls are the enemy of free. Queues are the friend of free. Move work out of waiting, into a database, and let workers chew through it.

The pattern is not new. It is just rarely applied to LLM calls. Now you have the code. Run it, measure your throughput, and share the numbers.

The Week After the Eval: A Cost-Aware Routing Harness for New Model Drops

Riley Wu — Thu, 13 Aug 2026 04:14:48 +0000

A new model drops roughly every other week now, and my timeline fills up with the same two claims: "it's cheaper" and "it's better." I've written before about how I gate new models with a small self-written eval deck instead of trusting the hype. This post is about the week after that eval — the part nobody blogs about, where you actually have to decide which model gets which traffic without lighting your budget on fire.

The short version: a single winner-take-all model choice is usually wrong, and the fix is a tiny routing harness with per-task costs, not a bigger eval.

The problem with "the new model is my default now"

After my eval deck blesses a new release, the temptation is to flip a config variable and move on. Every time I've done that, one of three things happened within a month:

The cheap model wasn't cheap on my workload. Sticker price per token means little if the model needs twice the retries, longer prompts, or produces output my downstream parser rejects 12% of the time.
The strong model was wasted on easy tasks. Renaming variables and writing docstrings don't need the frontier model. Paying frontier prices for them is a quiet leak.
A silent regression appeared in one task category — usually the one with the fewest eval cases — and I noticed only because a coworker did.

So the real question isn't "is the new model good?" It's "for each task type I actually run, which model gives the lowest cost per accepted result?" That's a routing question, and it deserves its own artifact.

The artifact: a 60-line routing logger

The harness below is deliberately boring. It runs your task through a candidate model, checks the result against whatever acceptance test you already have (a unit test, a JSON schema, a linter, a diff review), and logs one line of JSONL per attempt. After a few days of real traffic, you have evidence instead of vibes.

# router.py — log cost-per-accepted-result per (task_type, model)
import json, time, hashlib, pathlib

LOG = pathlib.Path("routing_log.jsonl")

# Fill these in from each provider's CURRENT pricing page.
# Prices change; do not hardcode numbers you haven't verified this month.
PRICE_PER_1K_TOKENS = {
    "frontier-model": {"in": None, "out": None},   # verify before use
    "budget-model":   {"in": None, "out": None},   # verify before use
}

def acceptance_check(task_type: str, output: str) -> bool:
    """Your real checks go here. Examples:
    - 'sql':      run the query against a scratch DB
    - 'code':     run pytest on the generated file
    - 'json':     validate against the schema
    - 'summary':  length + keyword-presence heuristic (weakest check!)
    """
    raise NotImplementedError

def run_and_log(task_type: str, model: str, prompt: str, call_fn):
    t0 = time.time()
    resp = call_fn(model, prompt)          # your existing API wrapper
    elapsed = time.time() - t0
    accepted = acceptance_check(task_type, resp["text"])
    record = {
        "ts": int(t0),
        "task_type": task_type,
        "model": model,
        "prompt_hash": hashlib.sha256(prompt.encode()).hexdigest()[:12],
        "in_tokens": resp["usage"]["in"],
        "out_tokens": resp["usage"]["out"],
        "latency_s": round(elapsed, 2),
        "accepted": accepted,
    }
    with LOG.open("a") as f:
        f.write(json.dumps(record) + "\n")
    return resp["text"] if accepted else None

Then a weekly rollup:

# rollup.py — cost per ACCEPTED result, not per call
import json, collections

stats = collections.defaultdict(lambda: {"calls": 0, "accepted": 0, "in_t": 0, "out_t": 0})
for line in open("routing_log.jsonl"):
    r = json.loads(line)
    k = (r["task_type"], r["model"])
    s = stats[k]
    s["calls"] += 1
    s["accepted"] += r["accepted"]
    s["in_t"] += r["in_tokens"]
    s["out_t"] += r["out_tokens"]

for (task, model), s in sorted(stats.items()):
    rate = s["accepted"] / s["calls"]
    # cost = (in_t/1000)*price_in + (out_t/1000)*price_out — plug in verified prices
    print(f"{task:12s} {model:18s} accept={rate:.0%} calls={s['calls']}")

The metric that matters is cost per accepted result: raw token cost divided by acceptance rate. A model that's 5x cheaper per token but passes your checks half as often is only 2.5x cheaper in reality — and that's before you count the engineering time of babysitting retries.

The decision matrix I actually use

After two or three weeks of logs, I fill in this table per task type:

Task type	Frontier accept rate	Budget accept rate	Verdict
Boilerplate codegen / renames	~high	~high	Budget model, always
SQL with schema constraints	high	medium	Budget, retry once, escalate on failure
Novel debugging / root-cause	high	low	Frontier only
JSON transforms with schema test	~high	~high	Budget — the test does the gating
Free-text summaries	???	???	Neither — my acceptance check is too weak to trust

The last row is the honest one. If you can't write an acceptance check for a task type, your routing decision for it is superstition. Fix the check first or route conservatively.

Escalation routing, in one sentence

For the middle row of that table, the pattern is: try the budget model first, and if the acceptance check fails, escalate to the stronger model with the failure message included in the prompt. In my logs this catches most of the gap at a fraction of always-frontier cost, because the escalation prompt is unusually informative — "here's what a weaker attempt got wrong" is great context.

Where free access fits

The expensive part of this workflow isn't production traffic — it's the exploration phase, where you're re-running eval decks and shadow-routing candidates before committing. That's where I've been using MonkeyCode: it offers free access to a set of models plus a free server option, which maps neatly onto the "run the harness against candidates without a billing page open in another tab" phase.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Concretely, my loop when a new release lands is: point call_fn at the candidate via MonkeyCode's free model access, replay last month's logged prompts (the prompt_hash column makes this a clean shadow test on your own distribution rather than a generic benchmark), then read the rollup. The free server option covers the always-on piece — the rollup cron and the small routing sidecar — so the whole gate costs nothing to leave running between releases, which matters because model gating is a recurring chore, not a one-time project. If you want to try the same loop, their docs describe how to point an OpenAI-compatible client at it; the harness above doesn't care which backend call_fn wraps.

Two caveats I'd hold onto: free tiers can change in scope, rate, or availability, so treat them as an evaluation convenience rather than production infrastructure; and confirm which models are actually available there this week before designing around any specific one — the landscape moves fast enough that I now verify model names and pricing on official pages the day I make a routing change, not the day I read about them.

Limitations, and who shouldn't bother

Small samples lie. A task type with 15 logged calls has no accept rate, it has a coin flip. I wait for a few hundred calls per cell before acting on the matrix.
Acceptance checks are the whole game. For tasks where "correct" is subjective (design review, prose), this harness degenerates into measuring which model you like. That's fine to know, but call it preference, not quality.
Latency-sensitive paths (interactive autocomplete, user-facing chat) may not tolerate try-then-escalate. Route those directly.
If you run one model on one task type fifty times a month, the routing infrastructure costs more attention than it saves. The single-winner default is fine for you — just re-run your eval deck when something new drops.
Don't route on this week's headlines. Any claim about a brand-new release — speed, price, "beats X on Y" — deserves a look at the provider's own docs and a run through your deck before it influences traffic.

The eval deck tells you whether a model can do your work. The routing log tells you whether it should. You need both, and the second one is cheaper to build than the first.

Hype Cycles Don't Ship My Code: How I Gate New LLMs with a Self-Written Eval Deck

Riley Wu — Mon, 10 Aug 2026 11:12:48 +0000

Every few weeks a new model dominates the timeline. The launch posts look the same every time: a cherry-picked demo, a benchmark chart, a wall of flame emojis — and no answer to the question that actually matters to me, which is "will this thing handle the boring, weird, half-documented work in my repos?"

I learned this the expensive way. I once adopted a trending model based on launch-day enthusiasm and spent a night cleaning up hallucinated CLI flags it had emitted with total confidence. Since then, every model that wants into my workflow has to pass a small deck of adversarial tasks I wrote myself. This post is the full method.

Launch metrics answer launch questions, not yours

Benchmarks are legitimate science — for the benchmark's questions. Your codebase asks different ones:

Does the model follow your formatter and your naming conventions without being told twice?
Can it deal with unglamorous glue: GitHub Actions YAML, a creaky ORM migration, a Helm values file?
Does it hold together when your prompt is long and messy instead of short and clean?
When it fails, does it fail loudly and catchably, or quietly and plausibly?

A viral screenshot tells you none of this. The fix isn't cynicism — it's a tiny verification loop that you own.

The artifact: a deck of task cards plus a dumb runner

I keep each eval as a card: a prompt file plus a machine-checkable contract, stored as plain files so adding a task never touches the runner.

cards/
  tasks/
    pr-summary.md          # summarize this diff for a reviewer
    migration-plan.md      # propose a zero-downtime schema change
    incident-timeline.md   # reconstruct events from these log lines
  contracts/
    pr-summary.yml
    migration-plan.yml
    incident-timeline.yml

A contract declares what a passing answer looks like:

required_terms: ["rollback", "index concurrently"]
forbidden_terms: ["LOCK TABLE", "truncate"]
max_words: 220
samples: 4
required_success_fraction: 1.0

The runner is one dependency-free file. It speaks to any OpenAI-compatible endpoint, so the model under test is just environment config:

// eval.mjs — Node 18+, stdlib only
import { readdir, readFile } from "node:fs/promises";
import { parse } from "node:util"; // placeholder-free: we hand-roll YAML below

const BASE = process.env.LLM_BASE_URL;  // e.g. any OpenAI-compatible endpoint
const KEY  = process.env.LLM_API_KEY;
const MODEL = process.env.LLM_MODEL;

// Tiny YAML-subset parser (only the keys our contracts use)
function parseContract(text) {
  const out = {};
  for (const line of text.split("\n")) {
    const m = line.match(/^(\w+):\s*(.+)$/);
    if (!m) continue;
    const [, k, v] = m;
    out[k] = v.startsWith("[")
      ? JSON.parse(v.replace(/'/g, '"'))
      : isNaN(+v) ? v.trim() : +v;
  }
  return out;
}

async function complete(prompt) {
  const r = await fetch(`${BASE}/v1/chat/completions`, {
    method: "POST",
    headers: {
      Authorization: `Bearer ${KEY}`,
      "Content-Type": "application/json",
    },
    body: JSON.stringify({
      model: MODEL,
      messages: [{ role: "user", content: prompt }],
      temperature: 0.1,
    }),
  });
  return (await r.json()).choices[0].message.content;
}

function passes(answer, c) {
  if (answer.split(/\s+/).length > c.max_words) return false;
  const low = answer.toLowerCase();
  return c.required_terms.every(t => low.includes(t.toLowerCase()))
      && c.forbidden_terms.every(t => !low.includes(t.toLowerCase()));
}

for (const f of await readdir("cards/tasks")) {
  const id = f.replace(".md", "");
  const prompt = await readFile(`cards/tasks/${f}`, "utf8");
  const c = parseContract(await readFile(`cards/contracts/${id}.yml", "utf8"));

  let ok = 0;
  const lat = [];
  for (let i = 0; i < c.samples; i++) {
    const t0 = performance.now();
    const ans = await complete(prompt);
    lat.push(Math.round(performance.now() - t0));
    ok += passes(ans, c);
  }
  const frac = ok / c.samples;
  console.log(
    `${id.padEnd(22)} ${ok}/${c.samples}` +
    `  p50=${lat.sort((a,b)=>a-b)[Math.floor(lat.length/2)]}ms` +
    (frac >= c.required_success_fraction ? "  PASS" : "  REJECT")
  );
}

(The toy YAML parser is fine for flat contracts; swap in a real parser if your specs grow nesting.)

Three design choices carry most of the weight:

Sampled consistency, not single-shot luck. Four low-temperature runs per card, and the required fraction decides. A model that's right one time in four isn't skilled — it's a slot machine that paid out while you were watching.
Deny-lists for catastrophic modes. For anything infra-adjacent, certain words auto-fail the answer. I don't care how elegant the prose is if the migration plan contains truncate.
Tasks sourced from my own ticket history. Every card is a sanitized version of a real task from the last month. Curated benchmarks and real glue work correlate less than anyone wants to admit.

To run this you need an endpoint for the candidate model — ideally one that costs nothing while you're still in the poking-around phase. For my most recent round I pointed the runner at MonkeyCode, which offers free model access and a free server option; that meant an afternoon of iterating against the new release with no provisioning and no card on file. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

From raw results to an actual decision

A score line is not a verdict. I run each card's output through a short decision tree:

All cards pass, latency fine?
├─ yes → promote to one real, reversible task this week
└─ no  → is the failure concentrated in ONE card category?
    ├─ yes → adopt it everywhere EXCEPT that category
    ├─ no, failures are flaky/random → reject; re-test next checkpoint
    └─ no, failures are systemic → reject

The single-category-failure branch is where the real value lives. Models aren't uniformly good or bad — they have a shape. The goal isn't to crown a winner; it's to find a model whose weak spots never intersect your workload. A model that can't write commit messages but nails log analysis is a fantastic fit for my on-call triage and a terrible fit for my release notes.

Why open availability is the real story

This workflow only exists because more models are now reachable without an enterprise contract. That trend — whatever vendor's release is trending this week — quietly changes the epistemics: "trust the launch post" becomes "verify on your own machine tonight". It's the same deal open source always offered, applied to weights instead of source. Providers that make independent checking cheap (free model access, a free server tier — MonkeyCode's current arrangement is one example) are removing the last excuse for skipping due diligence. Providers that make it expensive are making a statement too.

Honest limitations

This catches dealbreakers, not gradients. A dozen cards can't finely rank models. Never sign an annual contract on this evidence alone.
Term matching is style-blind. An answer can pass every contract and still be ugly. I hand-read two full responses per model, no exceptions.
Free access is a snapshot, not a covenant. Pricing and availability change. That's precisely why the runner only assumes the OpenAI-compatible API shape — switching endpoints is an env var, not a refactor.
Proprietary code needs clearance first. Sending internal snippets to any external endpoint, free or paid, is a decision your security policy must authorize.
If your org already runs a mature eval platform, skip the harness — though I'd argue the forbidden-terms deny-list trick is still worth stealing.

Wrapping up

The next model launch is already scheduled somewhere. You don't have to choose between drinking the hype and ignoring the field — spend one evening turning your recent tickets into task cards, run the deck, and let your own results argue. If you need a zero-cost endpoint to run this against, MonkeyCode's free tier is a convenient place to start. One thing I'd genuinely like to hear: which catastrophic-failure terms land on other people's deny-lists — mine skews heavily toward destructive filesystem and SQL verbs, which is either prudence or scar tissue.

Pick Your LLM With a Scoreboard, Not a Hunch: A Two-File Eval That Costs Nothing to Re-Run

Riley Wu — Mon, 10 Aug 2026 08:55:05 +0000

The decision nobody writes down

Ask a team why they run model X in production and the honest answer is usually "someone tried it in a chat window and it looked good." That works right up until the inputs stop looking like the demo: an edge case arrives, the model bluffs instead of abstaining, and the bad output flows straight into whatever consumes it downstream.

Code doesn't get this luxury. When behavior matters, we encode expectations as tests and re-run them every time something changes. Model selection deserves the same treatment, because "something changed" happens constantly in LLM land: you switch candidates, you edit the prompt, or the vendor ships a silent revision under an unchanged model name.

What follows is a minimal harness for that: a versioned set of labeled inputs, a deterministic grader, and a scoreboard you regenerate whenever you want. It runs against any free model endpoint and lives on any small always-on machine — the whole comparison phase can cost zero while you're still deciding.

Why the evaluation phase is the free phase

The questions you need answered before committing to a model are cheap to ask:

Can anything on the market handle this task shape, even imperfectly?
Which failure shows up first — broken JSON, invented fields, refusal when it should answer?
Is the model weak, or is my prompt just ambiguous?

A free endpoint answers all three. And there's an underrated outcome: if every candidate fails identically, that tells you the bottleneck is task design, not model quality — which saves you from upgrading your way out of a prompt problem.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

In my setup, candidates are served through MonkeyCode's free model access, and the harness runs on its free server option, which also acts as the archive for past scoreboards. Neither choice is load-bearing: the script below pulls the base URL, credential, and model name from environment variables, so any OpenAI-compatible endpoint and any tiny VM work identically. I'm intentionally not quoting model catalogs, rate allowances, or how long the free tier lasts — those terms shift, so verify the current offering yourself before building on it.

The artifact: receipts, gold labels, and a grader that distrusts confidence

The walkthrough task: pull a total amount and currency out of messy receipt text. It's a good eval subject because the output is tiny, structured, and checkable by pure functions — no vibes required.

Three design commitments keep the harness meaningful:

Cases are files, not conversation logs. They live in git, diff like code, and grow every time production surprises you.
Deterministic checks first. Numeric comparison with tolerance, enum membership, JSON parseability. Human judgment is reserved for what a function genuinely cannot decide.
"I can't tell" is a valid answer with a score. A model that fabricates a total from ambiguous text is more dangerous than one that declines — the grader must reflect that asymmetry.

receipts.jsonl:

{"id": "r-001", "text": "COFFEE NORTHSIDE\\n2x latte 9.00\\nTOTAL $9.00\\nthank you", "gold": {"total": 9.00, "currency": "USD"}}
{"id": "r-002", "text": "Boulangerie Marie\\nCroissant 2,40 EUR\\nTotal: 2,40 EUR", "gold": {"total": 2.40, "currency": "EUR"}}
{"id": "r-003", "text": "TAXI RECEIPT\\nfare ...... 18.50\\ntip ....... 3.00\\nCARD ****4412", "gold": null}

That third row is the landmine: the receipt shows numbers but no explicit total. The honest answer is an abstention. A model that sums the lines and reports 21.50 has produced plausible-looking garbage — the exact thing that corrupts a downstream ledger.

receipt_eval.py (a starting point, not gospel):

import json, os, time, urllib.request, urllib.error

BASE  = os.environ["EVAL_BASE_URL"]      # any OpenAI-compatible endpoint
TOKEN = os.environ["EVAL_API_KEY"]
MODEL = os.environ["EVAL_MODEL"]

SYSTEM = (
    "Extract the grand total from the receipt. Respond with a single JSON object "
    '{"total": <number>, "currency": "USD|EUR|GBP"}. '
    "If no explicit grand total is printed, respond with: unclear"
)

def ask(receipt_text):
    payload = json.dumps({
        "model": MODEL,
        "temperature": 0,
        "messages": [
            {"role": "system", "content": SYSTEM},
            {"role": "user", "content": receipt_text},
        ],
    }).encode()
    req = urllib.request.Request(
        BASE.rstrip("/") + "/chat/completions",
        data=payload,
        headers={"Authorization": "Bearer " + TOKEN,
                 "Content-Type": "application/json"},
    )
    with urllib.request.urlopen(req, timeout=60) as resp:
        return json.load(resp)["choices"][0]["message"]["content"].strip()

def judge(case, raw):
    """Return (points, tag). Fabrication is penalized below zero."""
    if raw.strip().lower() == "unclear":
        pred = None
    else:
        try:
            pred = json.loads(raw)
        except json.JSONDecodeError:
            return (-1, "not_json")
    gold = case["gold"]
    if gold is None:
        return (1, "abstained_ok") if pred is None else (-2, "fabricated_total")
    if pred is None:
        return (0, "gave_up")
    total_ok = isinstance(pred.get("total"), (int, float)) and \
               abs(pred["total"] - gold["total"]) < 0.01
    curr_ok = pred.get("currency") == gold["currency"]
    if total_ok and curr_ok:
        return (2, "full_match")
    return (0, "wrong_value" if not total_ok else "wrong_currency")

run = {"model": MODEL, "ts": int(time.time()), "cases": []}
earned = possible = 0
with open("receipts.jsonl") as fh:
    for line in fh:
        case = json.loads(line)
        points, tag = judge(case, ask(case["text"]))
        earned += points
        possible += 2 if case["gold"] else 1
        run["cases"].append({"id": case["id"], "tag": tag, "points": points})
        print(f"{case['id']:>7}  {tag:<16} {points:+d}")

run["score"] = round(100 * max(earned, 0) / possible, 1)
print(f"\nscore: {run['score']}  ({MODEL})")
with open("scoreboard.jsonl", "a") as out:
    out.write(json.dumps(run) + "\n")

No SDK, no framework — one standard HTTP shape, so pointing it at a different provider is an environment-variable change:

EVAL_BASE_URL="https://your-endpoint.example/v1" \
EVAL_API_KEY="..." \
EVAL_MODEL="candidate-a" \
python receipt_eval.py

Two details are doing quiet work here. The weighted tags encode your actual risk model: fabricated_total scores below giving up, because in a bookkeeping pipeline a confident wrong number costs more than a manual review queue. And results append to scoreboard.jsonl with a timestamp, so every run becomes one line of history you can diff later.

Reading a scoreboard without lying to yourself

Aggregates are the beginning of the analysis, not the end. A few habits that keep the numbers honest:

Habit	Why it matters
Diff per case, not per model	Two candidates at 85% can fail on disjoint cases — one might be shippable with a fallback path, the other not
Weight fabrication heavily	A wrong field on a real input is an annoyance; an invented answer on noise is a data-integrity incident
Freeze `temperature=0` and the prompt string	Otherwise you're comparing runs, not models
Require a minimum case count	Under ~20 cases, noise picks winners; grow the set before trusting any ranking

Because every run lands in the same append-only file, "did candidate B actually beat candidate A, and on which inputs?" becomes a jq query instead of a memory exercise.

The boring machine is the point

Nothing in the harness needs orchestration. The useful trick is simply running it again later, which is what the always-on free server is for:

42 5 * * 1  cd /srv/receipt-eval && . ./env.sh && python receipt_eval.py >> weekly.log 2>&1

A weekly re-run against the same model name catches the failure mode no demo ever shows: the vendor revises the model silently, extraction behavior drifts, and your pipeline degrades without a single error being raised. With dated scoreboard lines, drift becomes a visible event with a before-and-after diff, not a mystery you debug for a weekend. One small box executing one small script is precisely the right amount of infrastructure here — the harness stays alive without turning into a second job.

Where this design runs out of road

Free capacity is for exploration, not scale. Rate ceilings, context limits, and evolving terms are normal on no-cost tiers. A few dozen cases fit that world comfortably; a thousand-case nightly gate does not — plan for paid capacity when you get there, and treat that as a success signal.
Function-graded scoring only covers verifiable outputs. Extraction, classification, format compliance: great fit. Summaries, code review comments, creative rewrites: not checkable by equality, and forcing them into this mold produces false confidence. Those need sampled human review or a judge model, with the subjectivity that implies.
Your case file is a theory of production. Every green run means "green on these inputs." Feed sanitized real-world surprises back into the JSONL continuously, or the suite drifts into irrelevance even while it stays green.
Abstention grading assumes your domain allows it. If your product must always answer, re-weight the rubric — but make that trade-off explicit rather than accidental.

Skip this if...

...you call a model interactively a few times a week with nothing automated downstream — the harness is ceremony at that scale. Also skip building cases from scratch if a maintained public benchmark genuinely mirrors your task; run that first. And if your output can't be mechanically checked at all, you want a sampling-and-review workflow, not this one.

But if you're about to wire a model into a pipeline based on playground impressions, one JSONL file and one grading script are all that stand between you and an evidence-based choice. If you want a zero-budget way to stand the loop up, MonkeyCode's free model access together with its free server is one concrete option — the endpoint stays swappable by design, which is exactly how an eval harness should treat its infrastructure. Version the cases, keep the scoreboard history, and let the next silent model update be something you measure instead of something you chase.