Use a Free Server and 30M Tokens as a Disposable Regression Oracle

A CLI changed last night. I did not want to read 400 lines of diff.

So I asked one question: can a free model act as a poor man's regression oracle?

Not a full judge. Not a replacement for tests. A temporary smell detector I can throw away.

I had two things to work with, both operator-supplied: MonkeyCode's free model access and its free server option. The advertised allowance is 30 million tokens. That is a budget, not a promise. I treat it that way.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

What I built

I built a tiny harness that sends CLI output to a model and asks one binary question:

Did this output change its behavior in a way that matters?

The harness is not smart. It is disposable. That is the point.

Here is the contract:

• 12 test cases
• one CLI command
• one temperature-zero prompt
• strict timeout
• token usage logged after each call

If the harness breaks, I can remove it without losing my real test suite.

Why this angle?

A free server changes the math.

If I run this on a paid endpoint, I hesitate on every call. I worry about cost before I worry about correctness. On a free tier, the decision flips.

I can afford to be wrong.

That is the real workflow: use the free option for high-volume, low-stakes judgment. Keep the expensive checks for release gates.

But a free tier also hides failure differently. Latency may spike. Quotas may reset at odd times. The model may refuse a request and return a 200. I need to see those failure modes before I lean on the tool.

Step 1: Install a minimal client

The client keeps the request shape boring on purpose.

mkdir regression-oracle && cd regression-oracle
python3 -m venv .venv && source .venv/bin/activate
pip install httpx

I use httpx because it gives me explicit timeouts and clean JSON.

Step 2: Write the test fixture

The fixture is a list of small cases. Each case has a command, expected-behavior notes, and a label.

# cases.py
CASES = [
    {"label": "ls_root", "cmd": ["ls", "/"], "watch": "list of top-level directories"},
    {"label": "env_count", "cmd": ["sh", "-c", "env | wc -l"], "watch": "environment size"},
    {"label": "disk_usage", "cmd": ["df", "-h", "/tmp"], "watch": "tmp disk availability"},
    {"label": "git_status", "cmd": ["git", "status", "--short"], "watch": "working tree state"},
]

Twelve cases are enough for a canary. More cases do not make the result more honest. They make the run slower.

Step 3: Add the oracle prompt

The prompt stays fixed. I do not ask the model to fix anything.

You are a regression smell detector.

Given:
- a command label
- the current command output
- a one-line note about expected behavior

Answer only with:
{"changed": true|false, "reason": "one short reason"}

Do not explain. Do not offer fixes.

That is the whole judge. It returns JSON or it fails.

Step 4: Run the harness

The harness catches network failure, timeout, malformed JSON, and token usage.

# oracle.py
import asyncio
import json
import subprocess
import time

import httpx

from cases import CASES

API_URL = "https://your-monkeycode-endpoint.example/v1/chat/completions"
API_KEY = "your-key"  # use environment variable in real use
SYSTEM_PROMPT = """You are a regression smell detector.
Given a command label, current output, and one-line note, answer only with JSON:
{\"changed\": true|false, \"reason\": \"one short reason\"}"""

def run_command(cmd):
    started = time.time()
    proc = subprocess.run(cmd, capture_output=True, text=True, timeout=10)
    elapsed = time.time() - started
    return {
        "stdout": proc.stdout[-2000:],
        "stderr": proc.stderr[-1000:],
        "returncode": proc.returncode,
        "elapsed_ms": round(elapsed * 1000),
    }

async def ask_oracle(client, label, output, watch):
    payload = {
        "model": "CHANGEME",
        "temperature": 0,
        "messages": [
            {"role": "system", "content": SYSTEM_PROMPT},
            {
                "role": "user",
                "content": json.dumps({
                    "label": label,
                    "watch": watch,
                    "output": output,
                }),
            },
        ],
    }
    try:
        resp = await client.post(API_URL, json=payload, timeout=20)
        resp.raise_for_status()
    except Exception as exc:
        return {"status": "request_failed", "error": type(exc).__name__}

    data = resp.json()
    usage = data.get("usage", {})
    text = data["choices"][0]["message"]["content"]

    try:
        parsed = json.loads(text)
    except json.JSONDecodeError:
        parsed = {"raw": text, "parse": "failed"}

    return {
        "status": "ok",
        "parsed": parsed,
        "tokens": usage,
    }

async def main():
    async with httpx.AsyncClient() as client:
        for case in CASES:
            command_result = run_command(case["cmd"])
            oracle_result = await ask_oracle(
                client, case["label"], command_result["stdout"], case["watch"]
            )
            print(json.dumps({
                "case": case["label"],
                "command_ms": command_result["elapsed_ms"],
                "oracle": oracle_result,
            }))

if __name__ == "__main__":
    asyncio.run(main())

This is not production code. It is a canary.

What I measure

I record six things per case:

Metric	Why it matters
command latency	distinguishes slow CLI from slow model
model latency	flags free-tier queueing
token usage	confirms the advertised budget is being consumed
JSON validity	tests whether the oracle can follow a strict shape
refusal mode	catches silent 200s with empty content
changed flag	gives the actual regression signal

One run gives me a decision table.

Sample output shape

The harness prints one JSON line per case. Here is the shape I use, not a fabricated result:

{
  "case": "git_status",
  "command_ms": 18,
  "oracle": {
    "status": "ok",
    "parsed": {"changed": false, "reason": "working tree matches expected state"},
    "tokens": {"prompt_tokens": 142, "completion_tokens": 31, "total_tokens": 173}
  }
}

If status is request_failed, I stop. A free tier that cannot answer is not an oracle. It is a queue.

Step 5: Run a failure fixture

A good canary includes one case designed to fail.

{
    "label": "tmp_usage_anomaly",
    "cmd": ["sh", "-c", "du -sh /tmp/* 2>/dev/null | head -5"],
    "watch": "tmp directory file usage",
}

The point is not for the model to panic. The point is to see whether the JSON contract survives surprising input.

If the model starts explaining instead of returning JSON, I know the oracle is not reliable enough for automation.

What breaks first

I expect these failures, based on how free tiers tend to behave:

1. Timeout on first request after idle. Cold start is real.
2. Malformed JSON when the output contains a stray brace.
3. Token padding when a command returns more than 2,000 characters.
4. Empty content without an error field.
5. Rate-limit response when I re-run too quickly.

I do not fix those failures. I record them. The data tells me whether to trust the oracle for one-off checks or only for batched work.

Who should not use this approach

This is not for you if:

• You need a deterministic pass/fail gate for releases.
• You work with sensitive CLI output that cannot leave your machine.
• You need millisecond-level answers on every request.
• You cannot tolerate occasional malformed JSON.

For those cases, keep your unit tests and a real integration suite.

This harness is for the fuzzy middle: a quick smell check when a teammate changes a command and you want one extra signal before you read the diff.

The rollback plan

The exit is clean.

rm -rf regression-oracle

No repo change. No config file. No dependency in the build pipeline.

That is the whole appeal of a disposable oracle. When it stops being useful, remove it.

What I would try next

One improvement is a diff-aware prompt: send both old and new output, then ask if the new output is materially different. Another is a token ceiling per case so one verbose command cannot eat the whole free allowance.

If you try this, record the first failure you hit with the free server. Was it cold start, malformed JSON, or a rate limit?