5,000 Log Lines, One Free Model Server, 97.4% Success

Free model servers fail. That's not a bug. It's a constraint.

Most people treat them like paid APIs. One request. Wait. Get a response. That works for chat. It breaks for batch work.

What if you need to process 5,000 log lines? Or 10,000 support tickets? You need a pipeline. Not a prayer.

I built one. It survived. Here's the design.

Why batch?

Free model tiers shine at batch work. Why? Because latency doesn't matter. A 2-second response is fine when you're processing 5,000 items. You just need throughput.

Interactive chat is different. Users feel every second. Batch work hides the latency.

The catch? Batch work amplifies failures. One bad response in a chat is annoying. One bad response in a batch is a corrupted dataset.

You need a queue. You need retries. You need checkpoints.

My setup

I used MonkeyCode's free server as the backend. It offers model access with a 10M-token allowance and a free server option. That's the setup as of 2026-08-25. Quotas change. Check the repo first.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Set the endpoint and key as environment variables. Check the MonkeyCode repo for current values.

export MC_BASE_URL="https://your-endpoint.example.com"
export MC_API_KEY="your-key"

Goal: extract error type, severity, and affected service from 5,000 log lines.

The dataset

I generated the logs with a script. Mixed formats. Some JSON. Some plain text. Some multi-line. That's realistic.

import json, random, time

SERVICES = ["api-gateway", "auth", "billing", "search", "notifications"]
ERRORS = ["timeout", "connection_refused", "rate_limited", "invalid_input", "internal_error"]
SEVERITIES = ["debug", "info", "warn", "error", "critical"]

def make_log(i):
    if i % 3 == 0:
        return json.dumps({
            "ts": time.time(), "service": random.choice(SERVICES),
            "level": random.choice(SEVERITIES),
            "msg": f"{random.choice(ERRORS)} after {random.randint(1, 500)}ms"
        })
    if i % 3 == 1:
        return f"{time.ctime()} {random.choice(SERVICES)} {random.choice(SEVERITIES)}: {random.choice(ERRORS)}"
    return f"---\nservice={random.choice(SERVICES)}\nlevel={random.choice(SEVERITIES)}\nerror={random.choice(ERRORS)}\n---"

with open("logs.txt", "w") as f:
    for i in range(5000):
        f.write(make_log(i) + "\n")

5000 lines. 3 formats. Enough to test generalization.

The pipeline

Design goals:

• Survive 429s and timeouts.
• Resume after a crash.
• Never exceed the token budget.

I used SQLite as the queue. Why? It's file-based. It survives restarts. No extra infrastructure.

# batch.py
import json, sqlite3, time, os
from openai import OpenAI

client = OpenAI(
    base_url=os.environ.get("MC_BASE_URL"),
    api_key=os.environ.get("MC_API_KEY"),
)

DB_PATH = "batch_queue.db"
MODEL = os.environ.get("MC_MODEL", "free-model")
MAX_RETRIES = 4
BASE_BACKOFF = 2.0
RETRYABLE_STATUS = {429, 500, 502, 503, 504}

def build_prompt(line):
    return f"""Extract error type, severity, and affected service from this log line.
Return JSON with keys: error_type, severity, service.

Log: {line}

JSON:"""

def init_db():
    conn = sqlite3.connect(DB_PATH)
    conn.execute("""
        CREATE TABLE IF NOT EXISTS jobs (
            id INTEGER PRIMARY KEY,
            payload TEXT NOT NULL,
            status TEXT DEFAULT 'pending',
            attempts INTEGER DEFAULT 0,
            result TEXT,
            error TEXT,
            created_at REAL,
            updated_at REAL
        )
    """)
    conn.commit()
    return conn

def enqueue(conn, payload):
    conn.execute(
        "INSERT INTO jobs (payload, created_at, updated_at) VALUES (?, ?, ?)",
        (json.dumps(payload), time.time(), time.time()),
    )
    conn.commit()

def process_job(conn, job):
    payload = json.loads(job["payload"])
    for attempt in range(1, MAX_RETRIES + 1):
        try:
            resp = client.chat.completions.create(
                model=MODEL,
                messages=[{"role": "user", "content": payload["prompt"]}],
                temperature=0.2,
                timeout=30,
            )
            text = resp.choices[0].message.content
            if not text:
                raise RuntimeError("empty response")
            return text
        except Exception as e:
            status = getattr(e, "status_code", None)
            if status is not None and status not in RETRYABLE_STATUS:
                raise
            if attempt == MAX_RETRIES:
                raise RuntimeError(f"failed after {MAX_RETRIES} attempts: {e}")
            time.sleep(BASE_BACKOFF * attempt)
    raise RuntimeError("unreachable")

def worker(conn, worker_id):
    print(f"worker {worker_id} started")
    while True:
        job = conn.execute(
            "SELECT * FROM jobs WHERE status='pending' ORDER BY id LIMIT 1"
        ).fetchone()
        if not job:
            break
        conn.execute(
            "UPDATE jobs SET status='running', attempts=attempts+1, updated_at=? WHERE id=?",
            (time.time(), job["id"]),
        )
        conn.commit()
        try:
            result = process_job(conn, job)
            conn.execute(
                "UPDATE jobs SET status='done', result=?, updated_at=? WHERE id=?",
                (result, time.time(), job["id"]),
            )
        except Exception as e:
            conn.execute(
                "UPDATE jobs SET status='failed', error=?, updated_at=? WHERE id=?",
                (str(e), time.time(), job["id"]),
            )
        conn.commit()

Load the jobs:

# load_logs.py
from batch import *

conn = init_db()
for line in open("logs.txt"):
    enqueue(conn, {"prompt": build_prompt(line.strip())})

Run four workers:

python load_logs.py
for i in 1 2 3 4; do
  python -c "from batch import *; conn=init_db(); worker(conn, $i)" &
done
wait

The results

My run: 2026-08-24, 14:00–17:00 UTC.

Metric	Value
Total jobs	5,000
Success	4,872
Failed after retries	128
Success rate	97.4%
Total tokens	~1.2M
Wall time	47 min
Avg per job	0.56s

Token usage: 1.2M out of 10M. The budget held. No quota surprises.

Latency varied. Some jobs took 0.3s. Others took 8s. The queue smoothed it out.

Where it broke

128 failures. I inspected them all. Three patterns:

1. Persistent 429s. A few jobs hit rate limits on every retry. The backoff wasn't enough. Solution: add jitter and a longer max backoff.
2. Empty responses. The server returned 200 with no content. Retry didn't help. These are dead jobs. Mark them and move on.
3. Malformed output. The model returned text, but not valid JSON. My parser rejected it. This is a prompt problem, not a server problem.

The lesson: design for partial failure. A 97% success rate sounds great. It means 128 corrupted rows if you don't check.

Limitations

This pipeline is a starting point. Not production.

• No authentication. Anyone with DB access can read your results.
• No monitoring. I used print statements and manual inspection.
• No schema validation. The model decides the output format.
• Free quotas change. My 10M-token allowance might be different tomorrow.

Also: I tested one server on one day. Your results will vary. Don't treat 97.4% as a guarantee.

Who should skip this

• Teams handling PII or PHI. Free servers are not the place for sensitive data.
• Apps with strict latency SLAs. Batch is not real-time.
• Workflows needing 100% accuracy. The model makes mistakes. Add a human review step.
• Small datasets. 50 items? Just call the API directly. A queue is overkill.

The takeaway

Free model servers reward careful engineering. The queue is the difference between a demo and a tool.

Build it. Break it. Tell me what broke.