48-Hour Field Notes: Cheap Model First, With a Lie Detector

The red run landed at hour three of what was supposed to be a quiet Friday: 180,000 characters of CI log, one failing assertion buried somewhere past the middle, and a model call queued to read all of it. The failure was not the interesting part. The interesting part was the order of operations.

Most log-triage setups I have seen answer in the wrong sequence. They buy the expensive call first, then argue afterwards about whether the answer was good. Over 48 hours I inverted that: reduce the log with deterministic code, ask the cheapest model that will accept the excerpt, and require the answer to clear a check that does not involve a second model. What follows is what survived contact, including the three hours I lost to a cache key.

Hour 0–6: freeze the input, not the output

The first draft kept the whole log in the prompt and asked for a diagnosis in JSON. It was slow, it was expensive, and every run disagreed slightly with the last one. That is a sampling problem, and sampling problems are boring to debug because there is nothing to compare against.

The fix was not a better prompt. It was a fixture: one failing run captured to disk, with its stdout, the exit code, and the exact commit SHA written alongside it. Every later experiment ran against that fixture and nothing else. If a change moved the answer, the input had not moved, so the change was real.

Two ingredients made the loop cheap enough to iterate on twenty times in one evening: free model access, which covered the small tier doing most of the attempts, and a free server option, so the harness ran next to the artifacts instead of on my laptop with a terminal I kept closing by accident.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Treat allowance numbers as a snapshot rather than a spec. The figure given for this round was 10 million free tokens, and the operator describes both the model access and the server as free; confirm current terms on the project page before you size any capacity planning around them, because free-tier numbers are exactly the sort of thing that moves between quarters.

Hour 6–12: the reducer refuses to guess

The most useful code I wrote all weekend does not call a model at all. It finds the failing predicate and ships a numbered window around it, or it returns nothing and says so.

import hashlib, re

ASSERT = re.compile(
    r"^(?P<file>[\w./-]+\.\w+):(?P<line>\d+):?\s+"
    r"(?:AssertionError|E\s+assert)\s*(?P<rest>.*)$",
    re.M,
)

def reduce_log(raw: str, window: int = 40) -> dict:
    hit = ASSERT.search(raw)
    if hit is None:
        return {"ok": False, "reason": "no failing predicate in log"}

    lines = raw.splitlines()
    end_of_hit = raw[: hit.end()].count("\n")
    start = max(0, end_of_hit - window)
    stop = min(len(lines), end_of_hit + window)

    excerpt = "\n".join(
        f"{i + 1:>6}| {lines[i]}" for i in range(start, stop)
    )
    return {
        "ok": True,
        "anchor": hit.group(0).strip()[:200],
        "body": excerpt,
        "sha": hashlib.sha256(excerpt.encode()).hexdigest()[:12],
    }

That numbered gutter is doing quiet work. It gives the model a vocabulary for citing evidence, and it gives the verifier something concrete to check against. Without it, answers arrive as prose about "the assertion" and cannot be falsified.

When ok is false, the pipeline stops and posts the tail of the log to a human. A model asked to diagnose a log with no failing predicate will produce something plausible, and plausible is the failure mode I was trying to delete.

Hour 12–24: route once, escalate once

The router has two tiers and one rule. The small tier gets the excerpt. If the answer clears the gate, we are done and the run costs almost nothing. If it fails, the same excerpt goes to the stronger tier exactly once, with the gate's complaint appended. A second failure ends the run rather than starting a conversation with the model about why it should try harder.

The gate itself is about forty lines and contains no ML:

import re
from pathlib import Path

def accepts(answer: str, fixture: dict) -> tuple[bool, str]:
    shipped = {int(n) for n in re.findall(r"^\s*(\d+)\| ", fixture["body"], re.M)}
    claimed = {int(n) for n in re.findall(r"(\d+)\| ", answer)}

    if not claimed:
        return False, "no line reference in answer"
    if claimed - shipped:
        return False, f"cites lines never sent: {sorted(claimed - shipped)}"

    blamed = Path(fixture["anchor"].split(":")[0]).name
    if blamed not in answer:
        return False, f"does not name {blamed} as the failing file"

    return True, "ok"

This rejects fluent nonsense, and it also rejects correct answers phrased badly. That trade is deliberate. A gate that lets bad answers through is worse than a gate that occasionally sends a good answer to a human, because the second failure is visible and the first one is not.

Fixture shape	Reducer output	Who answers
Assertion with file and line	80-line numbered window	Small tier, gate enforced
Assertion, but truncated stack	Window plus raw tail	Small tier, escalate on gate failure
Timeout or race, no assertion	Nothing	Human, no model call

Hour 24–36: what actually broke

The cache key was wrong in the first version. I keyed entries on the model identifier, which meant every escalation wrote a fresh entry and no run ever hit a warm cache. Keying on fixture["sha"] plus a prompt version string fixed it, and the hit rate went from never to most mornings.

Truncation by character count cut the assertion in half on long lines. Windows are now measured in lines, and the reducer never splits a numbered line, because a cited line that does not exist is worse than a missing citation.

Asking for JSON produced JSON about eighty percent of the time and a friendly paragraph with an embedded blob the rest. I deleted the schema and asked for Markdown that cites line numbers instead. Parsing got duller, which is the goal.

The escalation loop was the expensive one. The gate rejected an answer for a formatting reason, the loop escalated, the stronger tier made the same formatting choice, and the run paid twice for a correct diagnosis. One escalation, hard stop, and the gate's complaint goes into the transcript so the next reader can see why the run ended.

Free-tier limits meant the run had to be resumable. State is written before the call, never after, so a rate-limited morning costs a retry rather than a re-run.

What I would repeat, and who should skip this

Run the whole thing on the fixture before touching a live pipeline. Write the gate before the prompt. Cap escalation at one. Keep the reducer dumb enough that you can read it in a code review.

Three cases where this is the wrong tool. Flaky tests with no assertion produce no anchor, so the reducer abstains and the workflow degenerates into a human reading logs, which is fine but not automation. Logs containing credentials or customer data need redaction upstream of the excerpt, and a free tier is not the place to discover your redaction has gaps. And if you need an auditable explanation for a post-incident review, a model's citation is supporting evidence, not the record; keep the raw log.

If the appeal of this loop is trying it on your own red run without putting a card behind it, the free model access and the free server are the two pieces that make the harness reproducible rather than theoretical. Bring a fixture, and let the gate tell you the truth.