48-Hour Field Notes: Same Log, Five Answers

At 02:14 on a Thursday, nightly-sync failed for the fourth night running. Same truncated trace, different exit code, and the incident field already said "flaky" — the word people write when they have stopped looking. I had a 48-hour window before the release freeze and a hard constraint I set myself: whatever I used to triage this had to cost nothing, because the tooling line was already spent.

That constraint pointed me at MonkeyCode, which its maintainers describe as an open-source project offering free model access and a free server option. Those two availability claims are operator-supplied; I did not benchmark them, and terms like this change, so verify current limits and licensing in the project's own repo and site before you build on them. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The question was stability, not intelligence

Every triage write-up I read asks "was the model right?" That is the wrong first question. A wrong answer you can argue with. An answer that changes between runs cannot be argued with at all, because you never know which run you are arguing with. So the measurement I cared about was agreement: run the same bytes through the same instruction five times, and count how often the conclusion holds.

That is also cheap to test. Five calls is not a load test, and it fits inside a two-day window without touching the release branch.

Hour 0–6: a harness that assumes the first answer is wrong

The first artifact is a script, not a prompt. It pins the log, zeroes the clock, and refuses to let the filename leak into the request.

#!/usr/bin/env bash
# triage-repeat.sh <logfile> [runs]
set -euo pipefail

LOG="${1:?usage: triage-repeat.sh <logfile> [runs]}"
RUNS="${2:-5}"
OUTDIR="runs/$(date -u +%Y%m%dT%H%M%SZ)"
mkdir -p "$OUTDIR"

# Record exactly which bytes were judged.
sha256sum "$LOG" | tee "$OUTDIR/log.sha256"

read -r -d '' HEADER <<'EOF' || true
You are a CI triage assistant. Reply with exactly one line:
FAULT=<one of: network|resource|dependency|config|unknown> EVIDENCE=<log line number>
No prose. No explanation.
EOF

for i in $(seq 1 "$RUNS"); do
  payload=$( { printf '%s\n\n' "$HEADER"; cat "$LOG"; } )
  # TRIAGE_CMD: point this at the client or endpoint you actually use.
  if printf '%s' "$payload" | "$TRIAGE_CMD" > "$OUTDIR/run-$i.txt" 2> "$OUTDIR/run-$i.err"; then
    :
  else
    echo "run $i failed: $(tail -1 "$OUTDIR/run-$i.err")" >> "$OUTDIR/failures.txt"
  fi
  sleep 20   # free access is metered; spacing beats retrying
 done

The sleep 20 is the least glamorous line in the file and the one I would not remove. Free access is metered, and a batch that hammers the endpoint teaches you about the meter instead of the model.

The control matters just as much. Before running anything, I wrote a five-line keyword classifier that maps known substrings to the same five fault classes. It is dumb and deterministic, and that is the point — it gives me a floor. If the model cannot beat grep on agreement, I have learned something more useful than any single answer:

grep -nE 'ECONNRESET|ETIMEDOUT|no space left|OOM|quota exceeded' "$LOG" | tail -n 5

Hour 6–20: what broke

Run 1 and run 2 came back identical: FAULT=resource EVIDENCE=418. Run 3 returned an error instead of an answer, because I had fired it a few seconds after run 2 — my own scheduling bug, not the model's. Run 4 came back FAULT=dependency. Run 5 came back FAULT=resource with a different evidence line.

That is the whole finding, and it took less than a day to earn. Three runs pointed at one class and one pointed at another, on byte-identical input. Whatever I thought I had after run 1 was a coin flip dressed as a diagnosis.

The second failure was structural. My first version of the harness stored only the model's one-line answer, which meant I could count agreement but could not explain disagreement — I had thrown away the only data that would tell me whether the drift came from the instruction, the truncation, or the log itself.

Hour 20–40: the fix that mattered was spacing, not the model

I rebuilt the runner to keep raw responses and errors together, then compared conclusions instead of prose:

import re, sys, pathlib
from collections import Counter

pat = re.compile(r"FAULT=(\w+)\s+EVIDENCE=(\d+)")
rows = []
for p in sorted(pathlib.Path(sys.argv[1]).glob("run-*.txt")):
    m = pat.search(p.read_text())
    rows.append(m.groups() if m else None)

answered = [r for r in rows if r]
if answered:
    top, hits = Counter(answered).most_common(1)[0]
    print(f"answered={len(answered)}/{len(rows)} mode={top} agree={hits}/{len(answered)}")
print("no-answer:", [i for i, r in enumerate(rows, 1) if r is None])

The output I was looking for has this shape — illustrative, not a benchmark, and your numbers will differ because that is precisely what running it is for:

Run	FAULT	EVIDENCE	Counted
1	resource	418	yes
2	resource	418	yes
3	—	—	error, no answer
4	dependency	402	yes
5	resource	431	yes

Mode resource at 3/4 answers, one no-answer. Spacing the calls brought the no-answer rate down; it did not make the conclusions agree, and I stopped pretending otherwise. The free server option is what makes this shape practical at all: a batch of five short calls spread across twenty-minute gaps is a queue job, not a laptop job, and a small always-on box can hold that queue while you sleep. I treat its availability as an operator claim, so before pointing anything at it, read the current terms and keep secrets out of the payload.

Hour 40–48: what I would repeat

Keep the control classifier. Pin the prompt with a hash next to the log hash so a later rerun cannot silently use a different instruction. Store failures as first-class results, because a missing answer is information and an empty file is not. And rerun the same bytes rather than hunting for new logs — variety feels like progress and measures nothing.

When this workflow is the wrong tool

If your logs carry customer IDs, session tokens, or anything regulated, this harness is the wrong first step; redaction is a separate project and free access does not change your obligations. If you need per-decision audit trails, a repeat-and-count script is a diagnostic, not a compliance control. If your triage has hard latency targets, a metered free path with deliberate spacing is the wrong substrate. And if you cannot state ground truth for even five incidents, no agreement rate will mean anything yet — label five logs by hand first.

Observed signal	Stay on the free path	Move off it
5/5 agreement, one log family	yes	—
Agreement holds, evidence line drifts	yes, record the drift	when the line must be auditable
2+ runs disagree	no, it is noise	label by hand, then re-measure
Runs fail mid-batch	no	space the calls, or own the runner
Logs carry anything sensitive	no	redact first, then re-decide

The release shipped with a hand-built timeout around the sync call and a note that night four was a resource problem nobody had scoped. Not elegant, but the decision rests on bytes I can point at. If you want to run the repeat harness against the free tier before you commit to anything, the project's current access terms are on its own site, and the right first input is a log you already understand.