Free Model vs. 40 Lines of Python: Who Reads Build Logs Better?

The build failed at 2:14 AM. The CI log was 41,000 lines long. My first grep found an error on line 12. The real failure lived on line 12,003. My regex never saw it.

That night I stopped trusting my parser. I also refused to trust a model blindly. So I built a benchmark to make them fight fairly.

This post is that benchmark. Copy it, run it on your logs, and get your own numbers in about 30 minutes.

The Question

Can a free model endpoint read build logs better than 40 lines of Python? And if it can, where exactly does it win? I wanted a score, not a vibe.

The Setup

Two contestants. One labeled corpus. One scoring script.

Contestant A is a deterministic parser. Grep-style patterns, zero AI, zero surprises.

Contestant B is a free model endpoint. MonkeyCode is an open-source project with a free tier that includes 10M tokens and a free server option. That made this whole eval free to run.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The Corpus

I built a labeled corpus of 20 log fixtures. Each one has a known answer. The categories matter more than the count:

1. Obvious errors — single-line failures a regex catches instantly.
2. Hidden errors — multi-line failures where the real message sits two lines after the keyword.
3. Decoys — warning lines that contain the word "error" inside a filename.
4. Clean logs — no error at all. The correct answer is "no".

Here is a decoy:

WARNING: build/error_handler.py:12: unused import

A naive grep for error flags this line. It is not an error. A good triage tool must refuse it.

The fixtures file is plain JSON:

[
  {"log": "ERROR: build/main.c:12: undefined reference to `foo`", "error": true},
  {"log": "WARNING: build/error_handler.py:12: unused import", "error": false}
]

Contestant A: The Regex

Forty lines of Python, deliberately naive. Because that is what most of us actually ship:

import re

def regex_parse(log: str) -> dict:
    patterns = [
        (r"\berror\b|fatal|FAILED|Exception", "obvious"),
        (r"undefined reference|No such file|not found", "hidden"),
    ]
    for pattern, kind in patterns:
        m = re.search(pattern, log, re.IGNORECASE)
        if m:
            line = log[:m.start()].count("\n") + 1
            return {"error": True, "line": line, "reason": kind}
    return {"error": False, "line": None, "reason": None}

Fast. Deterministic. And completely line-blind.

Contestant B: The Free Model

The model gets the same log, truncated to the last 2,000 characters. The prompt is strict:

You are a build-log triage tool. Return JSON only.
{"error": true|false, "line": int|null, "reason": "short text"}

Log:
<last 2000 chars>

The harness leaves the client as a stub. Swap in your endpoint's SDK:

def model_parse(log: str) -> dict:
    prompt = f"""You are a build-log triage tool. Return JSON only.
{{"error": true|false, "line": int|null, "reason": "short text"}}

Log:
{log[-2000:]}
"""
    # response = your_model_client.complete(prompt)
    # return json.loads(response)
    raise NotImplementedError("Add your endpoint client here.")

The Scoring Script

Both parsers run against the same fixtures. The scorer counts true positives, false positives, and false negatives:

import json, sys

def score(parser, fixtures):
    tp = fp = fn = 0
    for fx in fixtures:
        pred = parser(fx["log"])
        truth = fx["error"]
        if pred["error"] and truth:
            tp += 1
        elif pred["error"] and not truth:
            fp += 1
        elif not pred["error"] and truth:
            fn += 1
    precision = tp / (tp + fp) if tp + fp else 0
    recall = tp / (tp + fn) if tp + fn else 0
    return {"tp": tp, "fp": fp, "fn": fn,
            "precision": round(precision, 2),
            "recall": round(recall, 2)}

fixtures = json.load(open(sys.argv[1]))
print("regex:", score(regex_parse, fixtures))
print("model:", score(model_parse, fixtures))

Run it:

python compare.py fixtures.json

What the Output Tells You

Here is the output shape:

regex: {'tp': 6, 'fp': 4, 'fn': 4, 'precision': 0.6, 'recall': 0.6}
model: {'tp': 9, 'fp': 1, 'fn': 1, 'precision': 0.9, 'recall': 0.9}

Those numbers are illustrative. Your logs will score differently. What matters is reading the pattern correctly.

Where the regex breaks

• Multi-line errors. Grep is line-oriented. The keyword lands on line 12,003. The real message sits on line 12,005. The regex reports the wrong line.
• Decoys. Filenames containing "error" create false positives.
• New toolchains. Patterns you never wrote will always slip through.

Where the model breaks

• Hallucinated line numbers. The model sometimes invents a line that does not exist.
• Over-confidence. It occasionally upgrades a warning into an error.
• Nondeterminism. Same log, two runs, two verdicts. The regex never does that.
• Latency. Two hundred sequential calls take minutes, not milliseconds.

The Cost Math

Each triage call uses roughly 600 tokens. The truncated log is 2,000 characters, which is about 500 tokens. The prompt and the JSON reply add another 100. At that rate, 10M free tokens covers about 16,000 triage calls. That is the real reason this eval is worth running: the measurement itself is free.

The free server option means you can run the harness away from your laptop. No local GPU. No long-running process eating your battery.

The Decision Table

Situation	Regex	Free model
Known toolchain, single-line errors	✅ First choice	❌ Overkill
Multi-line or obfuscated failures	❌ Misses them	✅ Better recall
Clean logs must stay quiet	✅ Deterministic	⚠️ Watch false positives
Latency-sensitive CI gate	✅ Milliseconds	❌ Seconds per call
Unknown error, need a hint	❌ No imagination	✅ Useful guess

Who Should Not Use This

If your patterns already work, do not add a model call. You are adding latency and nondeterminism for nothing.

If you need reproducible output for an audit, a model is the wrong tool.

If you cannot label even 10 fixtures, you cannot measure anything. Skip the benchmark and fix the regex.

Limitations

Twenty fixtures is a smoke test, not a benchmark. Model behavior changes between versions, so re-run monthly. My labels are my judgment — your logs will disagree somewhere.

The harness is the artifact. The numbers are yours to collect.

Try It

If you want to run the same eval without spending money, MonkeyCode's free tier (10M tokens, free server) is enough for this whole harness. Drop in your logs, label 20 lines, and see which contestant you should actually trust.

Then tell me: which category breaks your parser first — multi-line errors or decoys? That answer decides whether round two is worth building.