I Logged Every Model Decision My Triage Agent Made. The Log Was the Real Product.

There's a lot of talk right now about remembering decisions, not just data. I've been on that train for a while — my agent's routing policy lives in git, and permission slips get reviewed like code. But last week I learned something simpler. The cheapest way to debug an agent is to log every decision it makes. The second cheapest is hosting that log somewhere you can actually read it. So I built a tiny triage agent, pointed it at a small repo, and ran it for a week on free infrastructure. The bill was zero. The log was the product.

Background

The repo is small: a dozen open issues, a mix of bugs, docs gaps, and questions. Nothing that needs a fancy pipeline. But I have a rule I keep repeating — never let an agent decide anything without leaving a receipt. A receipt means the input, the model, the token count, the verdict, and the timestamp. Without that, a wrong answer is just a ghost you can't interrogate.

The constraint was the interesting part. I wanted the experiment to cost nothing. MonkeyCode's free model access and the free server option made that possible — the whole run sat inside a ten-million-token allowance, and the project being open source meant I could check how the pieces worked before trusting them. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Goal

The goal was boring on purpose. Classify each open issue as bug, docs, question, or enhancement. Write one JSON line per decision. Serve those lines as a readable page. That's it. No vector store, no eval harness, no dashboard with charts. The point was to test a habit, not to build a platform.

Implementation

The core is one script. It reads an issue, builds a prompt, calls the model, and appends a record to a JSONL file. Nothing clever.

# decide.py — one decision, one receipt
import json, os, time, hashlib
from openai import OpenAI

client = OpenAI(
    base_url=os.environ["MODEL_BASE_URL"],
    api_key=os.environ["MODEL_API_KEY"],
)

def decide(issue: dict) -> dict:
    prompt = f"""Classify this GitHub issue. Reply with exactly one label
and one confidence score between 0 and 1.
Labels: bug, docs, question, enhancement.

Title: {issue['title']}
Body: {issue['body'][:2000]}"""

    resp = client.chat.completions.create(
        model=os.environ["MODEL_NAME"],
        messages=[{"role": "user", "content": prompt}],
        temperature=0.2,
    )

    return {
        "ts": time.time(),
        "issue_id": issue["number"],
        "prompt_hash": hashlib.sha256(prompt.encode()).hexdigest()[:12],
        "model": os.environ["MODEL_NAME"],
        "tokens": resp.usage.total_tokens,
        "verdict": resp.choices[0].message.content,
    }

if __name__ == "__main__":
    issue = json.load(open("issue.json"))
    record = decide(issue)
    with open("ledger.jsonl", "a") as f:
        f.write(json.dumps(record) + "\n")
    print(json.dumps(record, indent=2))

The prompt hash is the part I refuse to skip. It lets me reproduce exactly what the model saw, even after the issue text changes. That single field turned a log into an audit trail. Swap in whatever client your endpoint speaks; the shape of the record is the point.

The scheduler is a cron job and a bash loop:

#!/usr/bin/env bash
# triage.sh — fetch open issues, log one decision each
for n in $(gh issue list --state open --json number -q '.[].number'); do
  gh issue view "$n" --json number,title,body > issue.json
  python decide.py
done

Every thirty minutes, the cron runs the loop, and each run appends to the same JSONL file. Idempotent? No. That bug becomes important later.

The server is even dumber. A FastAPI app with two routes: one serves a static HTML page, the other serves the JSONL file. The free server option was enough for this workload, which is exactly what I expected — this is a file read, not a compute job.

# serve.py — the free server only needs to read a file
from fastapi import FastAPI
from fastapi.responses import FileResponse

app = FastAPI()

@app.get("/")
def index():
    return FileResponse("ledger.html")

@app.get("/ledger.jsonl")
def ledger():
    return FileResponse("ledger.jsonl")

The HTML page fetches the JSONL and renders a table with the last twenty records. Fifty lines total. I wrote less code than I used to, and that was the point.

Results

The ledger grew to a few dozen records over the week. The token cost stayed inside the free allowance — ten million tokens is a lot of small classification calls, and this workload barely dented it. The free server option never complained, because there was nothing to complain about.

But the real result was the first thing the ledger caught. Issue #12 was being re-decided on every cron run. Same prompt hash, same verdict, three times a day, forever. The model wasn't wrong — my pipeline was. I was re-processing all open issues instead of only new ones, so every run re-lit the same cigarette. The ledger made that visible in five seconds. A dashboard would have hidden it behind a chart.

The second finding was quieter. Confidence scores clustered at the extremes — the model was either very sure or very unsure, rarely in between. That told me my prompt was doing the deciding, not the model. The labels were so constrained that the model had nothing to be uncertain about. That's a feature if you want consistency, and a trap if you want nuance.

Lessons

Free tiers change what you log. When logging costs nothing, you stop sampling and start recording. Sampling is a habit from paid infrastructure — you log ten percent because a hundred percent is expensive. The free allowance flipped that math for me, and the habit stuck.

The receipt is more valuable than the verdict. The verdict tells you what the model said. The receipt tells you why it might have said it — the exact input, the model, the token count. That's the difference between debugging an agent and guessing about it.

A free server is a feature, not a limitation. It forced me to keep the artifact small. No database, no queue, no orchestration. A JSONL file and a static page. The constraint did the architecture for me.

Who should not do this? If you need an SLA, a free server is not your target. If you're handling sensitive data, think hard about where the log lives. If your workload is a million records a night, a free token allowance will not cover it. This approach is for small tools, experiments, and habits — not for production.

The experiment is over, but the ledger stays. If you want to run the same thing, a free tier is an easy place to start — the habit matters more than the vendor. And if you do, keep the log. I'd genuinely like to see what your receipts catch.