I Counted the Turns That Didn't Need a Model

Half of a convincing agent loop is a state machine wearing a chat template. I did not believe that until I scored turns instead of demos. The model can still be useful. The question is which turns actually required it.

You have seen the other version of this story. A loop, a tool list, a thought field, a screenshot. It looks like software that thinks. Then you watch it radio headquarters to ask whether priority=low and status=open should skip the on-call queue. That is not reasoning. That is a missing if.

So I built a harness that asks a ruder question than "did the agent finish?" Did this turn need a model at all? Could a closed-set rule have produced the same action? Did we skip a tool we already had an id for? Did we refuse to halt after the schema was already complete?

I am not going to sell you a leaderboard I did not run in a lab. This post is the experiment. You point it at a free model endpoint, park the runner on a free server if you want the loop off your laptop, and you get three rates: over-model, under-tool, loop-overrun. Those three embarrass a demo faster than a comment thread.

The ticket is small on purpose

A support ticket arrives as JSON. Allowed actions are a closed set: route_oncall, route_triage, route_docs, lookup_order, draft_reply, halt. Some tickets are fully determined by fields you already have. Product is billing, priority is low, no order id. That is route_docs or draft_reply. No model required. Some tickets carry an order id and a shipping complaint. That is a real lookup. If the loop skips the tool and invents a tracking status, that is under-tool. If it looks up the order and then asks the model whether delivered means delivered, that is over-model. See the difference?

I keep the gold policy in code, not in a prompt. Prompts drift. Functions do not, at least not without a diff.

# gold_policy.py — closed-set oracle. Not a prompt.
CLOSED = {
    "route_oncall", "route_triage", "route_docs",
    "lookup_order", "draft_reply", "halt",
}

def gold_plan(ticket: dict) -> list[str]:
    product = ticket.get("product")
    priority = ticket.get("priority")
    order_id = ticket.get("order_id")
    text = (ticket.get("text") or "").lower()

    needs_lookup = bool(order_id) and any(
        w in text for w in ("ship", "track", "delivery", "package")
    )
    if needs_lookup:
        return ["lookup_order", "draft_reply", "halt"]
    if priority == "high" and product == "payments":
        return ["route_oncall", "halt"]
    if product == "billing" and priority == "low":
        return ["route_docs", "halt"]
    return ["route_triage", "halt"]


def rule_resolvable(ticket: dict) -> bool:
    """True when the route does not depend on free text."""
    plan = gold_plan(ticket)
    return plan[0] != "lookup_order" and "draft_reply" not in plan

The loop under test is allowed to look like an agent. It can call tools. It can write a plan. I do not grade the plan. I grade the action sequence against the gold policy and against a halt oracle. If that sounds harsh, good. Demos already have enough fans.

Score the control flow, not the vibes

# score_loop.py
from dataclasses import dataclass, field
from gold_policy import gold_plan, rule_resolvable, CLOSED

@dataclass
class Turn:
    action: str
    used_model: bool
    tool: str | None = None
    output: str = ""

@dataclass
class Score:
    over_model: bool = False
    under_tool: bool = False
    loop_overrun: bool = False
    unknown_verb: bool = False
    actions: list[str] = field(default_factory=list)
    notes: list[str] = field(default_factory=list)


def score_transcript(ticket: dict, turns: list[Turn], max_turns: int = 6) -> Score:
    s = Score(actions=[t.action for t in turns])
    gold = gold_plan(ticket)
    resolvable = rule_resolvable(ticket)

    for t in turns:
        if t.action not in CLOSED:
            s.unknown_verb = True
            s.notes.append(f"invented verb: {t.action}")

    called_lookup = any(t.tool == "lookup_order" or t.action == "lookup_order" for t in turns)
    if gold[0] == "lookup_order" and not called_lookup:
        s.under_tool = True
        s.notes.append("needed lookup_order, never called it")

    model_turns = [t for t in turns if t.used_model]
    if resolvable and model_turns:
        # A model preface on a closed-set route is still a wasted turn.
        s.over_model = True
        s.notes.append(f"rule-resolvable, but {len(model_turns)} model turns")

    if len(turns) > len(gold) or len(turns) > max_turns:
        s.loop_overrun = True
        s.notes.append(f"turns={len(turns)} gold={len(gold)} cap={max_turns}")

    halted = any(t.action == "halt" for t in turns)
    if not halted:
        s.loop_overrun = True
        s.notes.append("never halted")
    return s

That scorer is the article. over_model fires when the gold policy says the ticket was rule-resolvable and the loop still spent a model turn deciding the route. under_tool fires when an order lookup was required and never happened. loop_overrun fires when we already had a terminal action and the model kept talking. I used to score n-gram overlap on the draft reply. Cute. Useless. The blast was in the control flow.

How do you drive a model without pretending the HTTP client is the insight? You isolate one function: complete(messages) -> str. On a laptop it can be a stub. On a server it can be whoever is hosting the free endpoint this week. I do not need a model name to make this honest. I need a transcript.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. I park this runner on MonkeyCode's free server option and point complete() at their free model access because the experiment is chatty and I did not want the scoring loop itself to become a billing subplot. That is the only product claim I am making: free model access, and a free server you can leave the harness on. No quota theater. No hardware fanfic. If the endpoint is down, the harness fails closed. That is a valid result.

# runner.py — fill in complete(); do not paste keys.
import json, os, urllib.request
from score_loop import Turn, score_transcript
from gold_policy import CLOSED

SYSTEM = (
    "You emit one JSON object per turn: "
    '{"action": "<one of ' + ",".join(sorted(CLOSED)) + '>", '
    '"tool": null or "lookup_order", "output": "..."}. '
    "No preface. Halt when the action is terminal."
)

def complete(messages: list[dict]) -> str:
    url = os.environ["MODEL_URL"]  # your free-model endpoint
    body = json.dumps({"messages": messages, "temperature": 0}).encode()
    req = urllib.request.Request(
        url, data=body, headers={"Content-Type": "application/json"}
    )
    with urllib.request.urlopen(req, timeout=60) as resp:
        payload = json.loads(resp.read().decode())
    # Adapt this one line to whatever envelope your endpoint returns.
    return payload["choices"][0]["message"]["content"]

def parse_action(raw: str) -> dict:
    raw = raw.strip()
    start, end = raw.find("{"), raw.rfind("}")
    if start < 0 or end <= start:
        return {"action": "unknown", "tool": None, "output": raw[:200]}
    return json.loads(raw[start:end + 1])

def run_ticket(ticket: dict, max_turns: int = 6) -> list[Turn]:
    messages = [
        {"role": "system", "content": SYSTEM},
        {"role": "user", "content": json.dumps(ticket)},
    ]
    turns: list[Turn] = []
    for _ in range(max_turns):
        raw = complete(messages)
        parsed = parse_action(raw)
        action = parsed.get("action") or "unknown"
        tool = parsed.get("tool")
        turn = Turn(action=action, used_model=True, tool=tool,
                    output=str(parsed.get("output", "")))
        turns.append(turn)
        messages.append({"role": "assistant", "content": raw})
        if action == "lookup_order" and ticket.get("order_id"):
            tool_obs = json.dumps({"order_id": ticket["order_id"], "status": "delivered"})
            messages.append({"role": "user", "content": "TOOL_RESULT " + tool_obs})
        if action == "halt":
            break
    return turns

if __name__ == "__main__":
    over = under = overrun = n = 0
    with open("tickets.jsonl") as f:
        for line in f:
            ticket = json.loads(line)
            turns = run_ticket(ticket)
            s = score_transcript(ticket, turns)
            n += 1
            over += int(s.over_model)
            under += int(s.under_tool)
            overrun += int(s.loop_overrun)
            print(ticket["id"], s)
    print(f"n={n} over_model={over/n:.2f} under_tool={under/n:.2f} overrun={overrun/n:.2f}")

Run it like this. The fixture is boring on purpose. Boring fixtures keep you from grading vibes.

export MODEL_URL="https://your-free-model-endpoint.example/v1/chat"
python runner.py

{"id": "t1", "product": "billing", "priority": "low", "order_id": null, "text": "invoice looks high"}
{"id": "t2", "product": "shop", "priority": "med", "order_id": "A-1042", "text": "where is my package"}
{"id": "t3", "product": "payments", "priority": "high", "order_id": null, "text": "card charged twice"}

When I walk a constructed transcript through the scorer — and I am labeling this as a constructed example, not a published run — the pattern is almost rude. Ticket t1 is rule-resolvable. The loop still emits a "let me think" preface and a tool-less route. Over-model. Ticket t2 needs lookup_order. The loop drafts a tracking paragraph with a plausible carrier name. Under-tool. Ticket t3 should be route_oncall then halt. The loop routes, then asks the model whether paging a human is "too aggressive," then drafts a reply anyway. Overrun. None of those failures show up if you only read the final English.

That is the analogy I cannot shake. It is a night guard with a radio. Every unlocked door, they call dispatch. Dispatch is a language model. Dispatch will always have something to say. Your job is to count how often they already had the key.

Who is the judge?

Should you trust a free model for this? For scoring control flow, yes, because the gold policy lives in your repo. The model is the system under test, not the judge. Do not invert that. If you let the same free model grade its own halt decisions, you have built a mirror, not an evaluation. I have watched people do that and then quote the mirror as a metric. Please do not.

Want a fourth number? Count unknown verbs. The minute your "agent" invents escalate_via_slack because the prompt said "be helpful," the closed-set assumption is on fire. unknown_verb is the alarm. A trench-coat of if-statements at least has the honesty to name its branches in source control.

def summarize(scores: list) -> dict:
    n = max(len(scores), 1)
    return {
        "n": len(scores),
        "over_model_rate": sum(s.over_model for s in scores) / n,
        "under_tool_rate": sum(s.under_tool for s in scores) / n,
        "overrun_rate": sum(s.loop_overrun for s in scores) / n,
        "unknown_verb_rate": sum(s.unknown_verb for s in scores) / n,
    }

Print that next to the pull request. If the gif still looks magical and over_model_rate is 0.6, the gif does not get a vote. What were you actually shipping — a policy, or a narrator?

Limitations, said in daylight

This harness does not measure code quality, security, or user empathy. It does not know if the draft reply is kind. It will happily pass a monstrous sentence if the action tag is halt and the tool sequence matched. It assumes your tool results are ground truth. If your lookup API lies, the scorer will bless the lie.

It also assumes a closed action set. Open-ended agents undercount here. And a free model on a free server will be slower and noisier than whatever you put on a credit card. Noise is useful in a fuzzer. Noise is not an SLA. I am not giving you tokens-per-second, GPU SKUs, or a promise the free tier lasts forever, because I did not measure those and I will not invent them.

Who should not use this approach? Anyone shipping medical, financial, or on-call routing without a deterministic policy in front of the model. Anyone who needs a published speed chart from this post. There isn't one. Anyone who wants to prove their agent is already better than most developers. That claim is a different fight, and this scorer will not win it for you. If your loop is a single tool call with no branch, you do not need this. You need a unit test.

I still use a model. I use it when the ticket text is actually ambiguous, when the lookup result is messy, when the draft has to quote a field the rules do not know how to phrase. The win is not "delete the model." The win is knowing which turns were theater.

If you run the harness, keep the three rates next to the diff. Over-model, under-tool, overrun. If those numbers move, the narrator in the middle of the loop can finally shut up.