Free Models and a Free Server: Building a Zero-Budget LLM Regression Harness

The cheapest way to notice that a language model quietly got worse is to run the same ten prompts against it every week and compare the answers. Most teams skip this because they assume regression testing an LLM requires a paid evaluation platform, a GPU quota, and a data scientist who owns a whiteboard. In practice, a free model endpoint and a free server are enough to catch the embarrassing regressions before your users do. This article shows you how to build a small, repeatable harness that costs nothing to run and produces a number you can put in a CI log.

I built this harness against MonkeyCode's free offering, which includes access to free models and an optional free server for running your own lightweight workloads. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The exact models and quotas change over time, so the code below treats the endpoint as an OpenAI-compatible API and lets you point it at whatever model name is currently available. The point is not to benchmark the free tier; the point is to give you a workflow that turns a free tier into a monitoring tool instead of a chat window.

The core idea is simple: keep a fixed set of prompts with expected behaviors, send them to the model, score each response with a cheap heuristic, and store the scores in SQLite. After a few runs you have a trend line. When a model update or a server-side change shifts the scores, the trend line moves and your script can complain. A single prompt can pass or fail a test just like a unit test, except the assertion is built from string matching and simple semantic checks rather than exact equality.

Here is the complete harness. It uses httpx for requests, sqlite3 for storage, and a scoring function that you can replace with your own rules. The script reads the base URL and API key from environment variables, which keeps secrets out of the source file.

import os
import sqlite3
import time
import httpx

# A tiny regression set: prompt, expected substring, minimum length.
CASES = [
    ("What is the capital of France?", "Paris", 5),
    ("Write a Python one-liner that sums a list of ints.", "sum", 20),
    ("Explain the difference between HTTP and HTTPS in two sentences.", "encrypt", 80),
    ("Fix this buggy code: def f(x): return x + 1", "def f", 10),
    ("List three Python testing frameworks.", "pytest", 20),
    ("Translate 'good morning' into Spanish.", "buenos", 5),
    ("What is idempotency in REST APIs?", "same", 40),
    ("Write a SQL query to select all users older than 30.", "WHERE", 20),
]

def score_response(prompt, response, expected_substring, min_length):
    text = response.strip()
    if len(text) < min_length:
        return 0.0
    if expected_substring.lower() in text.lower():
        return 1.0
    # Partial credit for length only; you can make this smarter later.
    return 0.3 if len(text) > min_length * 2 else 0.1

def run_one(client, base_url, api_key, prompt):
    headers = {"Authorization": f"Bearer {api_key}"}
    body = {
        "model": os.getenv("FREE_MODEL_NAME", "gpt-3.5-turbo"),
        "messages": [{"role": "user", "content": prompt}],
        "max_tokens": 150,
        "temperature": 0.0,
    }
    try:
        r = client.post(f"{base_url}/chat/completions", json=body, headers=headers, timeout=30.0)
        r.raise_for_status()
        return r.json()["choices"][0]["message"]["content"]
    except Exception:
        return ""

def main():
    base_url = os.getenv("FREE_MODEL_BASE_URL")
    api_key = os.getenv("FREE_MODEL_API_KEY")
    if not base_url or not api_key:
        raise SystemExit("Set FREE_MODEL_BASE_URL and FREE_MODEL_API_KEY")

    conn = sqlite3.connect("llm_regression.db")
    conn.execute("""CREATE TABLE IF NOT EXISTS runs (
        id INTEGER PRIMARY KEY AUTOINCREMENT,
        timestamp REAL, model TEXT, pass_rate REAL)""")

    client = httpx.Client()
    scores = []
    for prompt, expected, min_len in CASES:
        response = run_one(client, base_url, api_key, prompt)
        scores.append(score_response(prompt, response, expected, min_len))
        print(f"{prompt[:40]:<42} score={scores[-1]:.2f}")

    pass_rate = sum(1 for s in scores if s >= 0.7) / len(scores)
    model = os.getenv("FREE_MODEL_NAME", "unknown")
    conn.execute(
        "INSERT INTO runs (timestamp, model, pass_rate) VALUES (?, ?, ?)",
        (time.time(), model, pass_rate),
    )
    conn.commit()

    prev = conn.execute(
        "SELECT pass_rate FROM runs WHERE model = ? ORDER BY id DESC LIMIT 1 OFFSET 1",
        (model,),
    ).fetchone()
    conn.close()

    print(f"\nModel: {model}")
    print(f"Pass rate this run: {pass_rate:.2f}")
    if prev:
        delta = pass_rate - prev[0]
        print(f"Change vs previous run: {delta:+.2f}")
        if delta < -0.2:
            print("WARNING: Significant drop detected — investigate before deploying.")
        elif delta < -0.1:
            print("Caution: slight decline, worth a manual review.")
    else:
        print("No previous run for this model; baseline established.")

if __name__ == "__main__":
    main()

Run it once a week with a cron job or a GitHub Actions schedule. The script stores every run, so you can build a chart after a few weeks. You can also extend the scoring function to use an embedding similarity check or a small separate model as a judge, but the beauty of the string-matching version is that it has no moving parts and no extra token cost.

To keep your findings honest, the decision table below maps score changes to actions. It assumes you are using a free tier where occasional latency spikes are normal, but a persistent score decline is not.

Pass rate delta	Meaning	Action
+0.1 or more	Model improved or test too easy	Recheck prompts, keep tests
-0.1 to +0.1	Stable behavior	No action, log the run
-0.2 to -0.1	Mild drift	Rerun once, inspect failing prompts
less than -0.2	Significant regression	Stop rollout, compare outputs manually

There is a legitimate criticism of this approach: your assertion logic can be wrong. A model that answers "Paris" as "paris" still passes, but a model that factually says "Lyon" in a long sentence may also pass if it contains "Paris" somewhere in a clarification. That is why the scoring function returns partial credit and why you should keep the test set small enough to manually review when a drop triggers. The goal is not to prove a model is intelligent; the goal is to catch sudden changes in the behavior you depend on.

The same harness works for code generation tasks if your expected substring is a function name or a language keyword. For multi-step reasoning you would need a better judge, but for many API-backed workflows a substring check is a surprisingly effective smoke test. You can run the exact same script against any OpenAI-compatible endpoint, which means your regression suite is portable across providers.

Who should not use this? Teams that need statistical confidence in model quality, or anyone measuring subtle semantic drift, will find this harness too coarse. It is also not a substitute for a proper evaluation set with human-labeled golden answers. Use this when you want a zero-cost canary that tells you when to pay attention, not when you need certification-grade metrics.

If you are prototyping a side project and want to see whether a free model and a free server can survive a small CI workload, this harness is a gentle introduction. Point it at MonkeyCode's free models, set the free server to run the cron job, and let the SQLite database accumulate a history. You will learn more from three weeks of pass rates than from any single benchmark leaderboard.

Try the harness against your own prompts. Steal the code, change the scoring rules, and make the free tier earn its place in your workflow.