Your LLM Eval Pipeline Needs a Queue, Not a Notebook

You spent two hours building a prompt, ran it five times in a notebook, got a decent answer, and closed the tab. Three weeks later a new model version ships, someone bumps a dependency, and your careful experiment silently breaks.

That workflow is a script, not a pipeline. A script stops being useful the moment you stop looking at it. A pipeline keeps working in the background, produces a record, and tells you when something drifts.

In this post I'll show you how to turn a one-off LLM evaluation into a batch job you can schedule on a free server, using a small queue system. I'll use an open-source project called MonkeyCode as the running example because it provides free models and a free server option, which keeps the cost of this whole setup at zero.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Why Notebooks Fail as Eval Infrastructure

A notebook is great for exploration. It is terrible for regression testing. When you rerun a notebook cell, you have to remember to run the cell above it first. You might get a different random seed, a different model response, or a half-updated variable. Nobody audits a notebook the way they audit a test suite.

A queue changes the contract. You submit a unit of work, a worker picks it up, and the result lands somewhere persistent. That structure gives you three things a notebook never will:

1. Replayability — you can rerun the same input against a new model version.
2. Isolation — a failed request doesn't kill the whole run.
3. Auditability — every input and output is saved with timestamps.

Anatomy of a Minimal Eval Queue

You don't need Kafka or Redis for this. A directory of JSONL files and a worker loop is enough for hundreds of cases. Here's the architecture:

• cases.jsonl — one test case per line
• pending/ — queue of case IDs to process
• results/ — one JSONL file per case with input, output, and metadata
• worker.py — long-running process that pulls from the queue
• report.py — aggregates results into a pass/fail summary

The queue is just a text file with one ID per line. The worker reads the first line, processes it, and removes it. No locks needed if you only run one worker at a time.

The Code: A Portable Worker

Here's a complete worker that calls any OpenAI-compatible chat endpoint. It expects the standard environment variables you'd configure for most AI providers.

#!/usr/bin/env python3
import json
import os
import sys
import time
from pathlib import Path

import httpx

BASE_DIR = Path(__file__).parent
CASES_FILE = BASE_DIR / "cases.jsonl"
QUEUE_FILE = BASE_DIR / "pending.txt"
RESULTS_DIR = BASE_DIR / "results"

API_URL = os.environ["LLM_API_URL"]
API_KEY = os.environ["LLM_API_KEY"]
MODEL = os.environ.get("LLM_MODEL", "default")
MAX_ATTEMPTS = int(os.environ.get("MAX_ATTEMPTS", "3"))


def load_cases():
    cases = {}
    with CASES_FILE.open() as f:
        for line in f:
            line = line.strip()
            if not line:
                continue
            obj = json.loads(line)
            cases[obj["id"]] = obj
    return cases


def build_prompt(case):
    return case["prompt"]


def call_model(prompt):
    payload = {
        "model": MODEL,
        "messages": [
            {"role": "system", "content": "Return JSON only."},
            {"role": "user", "content": prompt},
        ],
        "temperature": 0.0,
    }
    with httpx.Client(timeout=60) as client:
        resp = client.post(API_URL, headers={"Authorization": f"Bearer {API_KEY}"}, json=payload)
        resp.raise_for_status()
        data = resp.json()
        return data["choices"][0]["message"]["content"], data.get("usage")


def validate_output(case, content):
    try:
        obj = json.loads(content)
    except json.JSONDecodeError:
        return False, "invalid-json"
    for key in case.get("required_keys", []):
        if key not in obj:
            return False, f"missing-key:{key}"
    return True, "ok"


def process(case):
    for attempt in range(MAX_ATTEMPTS):
        try:
            content, usage = call_model(build_prompt(case))
            ok, reason = validate_output(case, content)
            return {
                "id": case["id"],
                "ok": ok,
                "reason": reason,
                "output": content,
                "usage": usage,
                "attempt": attempt + 1,
                "ts": int(time.time()),
            }
        except Exception as exc:
            last_error = str(exc)
            time.sleep(2**attempt)
    return {"id": case["id"], "ok": False, "reason": f"error:{last_error}", "ts": int(time.time())}


def main():
    RESULTS_DIR.mkdir(exist_ok=True)
    cases = load_cases()
    if not QUEUE_FILE.exists():
        print("No pending.txt found, aborting")
        sys.exit(1)

    lines = QUEUE_FILE.read_text().strip().splitlines()
    if not lines:
        print("Queue is empty")
        return

    case_id = lines[0]
    remaining = lines[1:]
    case = cases.get(case_id)
    if not case:
        print(f"Unknown case: {case_id}")
        QUEUE_FILE.write_text("\n".join(remaining))
        return

    result = process(case)
    out_path = RESULTS_DIR / f"{case_id}.jsonl"
    with out_path.open("a") as f:
        f.write(json.dumps(result) + "\n")

    QUEUE_FILE.write_text("\n".join(remaining) + ("\n" if remaining else ""))
    print(f"Processed {case_id}: {'PASS' if result['ok'] else 'FAIL'} - {result.get('reason')}")


if __name__ == "__main__":
    main()

The Report Script

After the worker has processed a batch, you want a summary. This script reads all result files and prints a table.

#!/usr/bin/env python3
import json
from pathlib import Path

RESULTS_DIR = Path("results")

rows = []
for f in sorted(RESULTS_DIR.glob("*.jsonl")):
    for line in f.read_text().strip().splitlines():
        rows.append(json.loads(line))

passed = sum(1 for r in rows if r["ok"])
print(f"Total: {len(rows)}")
print(f"Passed: {passed}")
print(f"Failed: {len(rows) - passed}")

for r in rows:
    status = "PASS" if r["ok"] else "FAIL"
    print(f"{status}  {r['id']:20} {r.get('reason', '')}")

Wiring It to a Free Server

A free server is perfect for this batch workload. There's no user traffic, no uptime SLA, and you can tolerate occasional cold starts. Just add a cron job:

*/30 * * * * cd /path/to/eval && python worker.py >> worker.log 2>&1

That will process one queued case every 30 minutes. If you have 50 cases, the whole batch finishes overnight. With MonkeyCode's free server option, you can run this without paying for idle compute.

A free server is not for production user requests, but for a nightly eval it's exactly the right tool. You get persistent storage, a public scheduler, and no surprise bill.

Decision Table: Batch Queue vs. Direct Loop

Workload	Recommended approach	Why
1–5 exploratory prompts	Direct script in a notebook	No queue overhead
10–100 cases, run once	Direct loop	Simpler setup
10–100 cases, run weekly	Queue on free server	Reusable, auditable
100+ cases with retry logic	Queue + workers	Parallel, isolated failures
CI gate on model output	Queue + report	Enforces a pass/fail contract

Limitations of This Workflow

Nothing here is magic. A file-based queue assumes a single worker. If you need concurrent workers, you'll want a proper message broker or a lock file. The free tier may also have rate limits that make rapid retries slower, so the exponential backoff in the worker becomes your friend.

More importantly, this workflow only checks structural validity, not semantic quality. Valid JSON is not the same as a correct answer. For higher-stakes evals, add a separate LLM-as-judge step that scores each output against a rubric. That costs extra tokens, so gate it behind the same queue.

Who Should Skip This

If you're building a user-facing product on free infrastructure, stop. Free servers and free models are not designed for production traffic, privacy-sensitive data, or strict uptime. If you need an SLA or HIPAA compliance, pay for a real provider.

But if you're an individual developer who wants a repeatable regression test for model prompts, a queue plus a free server is a solid starting point. You can always migrate to managed queues later without changing your case format.

Start With One Case

Your next prompt experiment doesn't have to be a throwaway notebook. Turn it into a case, push it through a queue, and let a free server remember it.

Try it with MonkeyCode's free models and free server. Submit a case that failed once. Run the worker. Watch it pass the second time. That's the feeling a notebook never gives you.