Workshop: Record Agent Loop Exits as Structured Events in 80 Minutes

Agent labs collapse at grading time when the only stop signal lives inside a free-form chat transcript. A structured exit event makes retry policy, tool failure, and completion comparable across students on the same fixture. This workshop records those exits in eighty minutes with a JSONL ledger students can rerun without rereading dialogue. The method is small, local, and deliberately boring, which is the point of a lab that must grade overnight.

Public discussion of agents this week still circles loop control, yet most classroom runners still treat "the model stopped talking" as a finished job. That heuristic hides timeout retries, schema mismatches, and silent max-turn cuts behind the same assistant message. Instructors then compare transcripts instead of comparing reasons, which makes two identical failures look like two different students. The ledger below turns each stop into a typed record before any narrative summary is written.

What this lab actually grades

Students are not asked to invent a better planner. They are asked to make every loop termination observable, replayable, and cheap to assert in CI. The worked example uses a toy weather tool, one retry, and a hard turn cap so the exit taxonomy stays finite. If a runner cannot emit completed, retry_exhausted, schema_mismatch, turn_cap, or tool_error, the lab is incomplete even when the chat looks fluent.

The core conclusion is operational rather than architectural. Loop quality is not the number of tools you registered. It is whether two reruns of the same fixture produce the same exit reason, the same turn count, and the same tool-name sequence.

Timing box (80 minutes)

Use a wall clock. Do not expand the taxonomy mid-lab when a student finds a poetic new failure name.

1. 0–10 min — Failure review. Open two transcripts that both "just stopped" and list what you cannot prove.
2. 10–25 min — Schema lock. Copy the LoopExit record below and refuse extra fields until the tests pass.
3. 25–50 min — Worked runner. Implement run_loop() against the fixture in weather_timeout.json.
4. 50–70 min — Assertions. Add tests for retry exhaustion and turn-cap exits on the same fixture.
5. 70–80 min — Diff drill. Rerun twice, then diff JSONL lines rather than chat Markdown.

Keep phones away from the taxonomy debate. Extra exit reasons are homework, not a live edit to the shared schema.

Artifact: a loop-exit ledger

Treat the ledger as the product of the loop, not as debug residue. Each line is one termination. Partial turns do not get their own files because graders will otherwise sort noise. The schema below is intentionally narrow so a ninety-minute lab cannot invent telemetry religion.

# loop_exit.py
from __future__ import annotations

from dataclasses import asdict, dataclass
from typing import Literal, Optional
import json
from pathlib import Path

ExitReason = Literal[
    "completed",
    "retry_exhausted",
    "schema_mismatch",
    "turn_cap",
    "tool_error",
]

@dataclass(frozen=True)
class LoopExit:
    run_id: str
    fixture_id: str
    reason: ExitReason
    turns: int
    tool_names: tuple[str, ...]
    last_error: Optional[str]
    stopped_on_turn: int

    def to_jsonl(self) -> str:
        payload = asdict(self)
        payload["tool_names"] = list(self.tool_names)
        return json.dumps(payload, sort_keys=True)


def append_exit(path: Path, event: LoopExit) -> None:
    path.parent.mkdir(parents=True, exist_ok=True)
    with path.open("a", encoding="utf-8") as handle:
        handle.write(event.to_jsonl() + "\n")

Students should print the JSONL path in the lab log and nowhere else. Console chatter about "the model seemed done" is not an exit record. If last_error is set, reason must not be completed, which is enforced in the tests later rather than in a custom exception hierarchy.

Minimal loop that is allowed to stop

The runner below is labeled as lab pseudocode with a real control flow. It does not claim production scheduling, distributed locks, or model-specific behavior. Tool calls are injected by a fixture so the exit path does not depend on live weather.

# run_loop.py
from __future__ import annotations

from dataclasses import dataclass
from typing import Any
from loop_exit import LoopExit, append_exit
from pathlib import Path

MAX_TURNS = 4
MAX_RETRIES = 1

@dataclass
class ToolResult:
    ok: bool
    name: str
    payload: dict[str, Any] | None
    error: str | None


def run_loop(run_id: str, fixture: dict[str, Any], ledger: Path) -> LoopExit:
    tool_names: list[str] = []
    retries = 0
    last_error = None

    planned_calls: list[dict[str, Any]] = fixture["planned_calls"]
    for turn, call in enumerate(planned_calls, start=1):
        if turn > MAX_TURNS:
            event = LoopExit(
                run_id=run_id,
                fixture_id=fixture["id"],
                reason="turn_cap",
                turns=turn - 1,
                tool_names=tuple(tool_names),
                last_error=last_error,
                stopped_on_turn=turn - 1,
            )
            append_exit(ledger, event)
            return event

        name = call.get("name")
        if not isinstance(name, str) or not name:
            event = LoopExit(
                run_id=run_id,
                fixture_id=fixture["id"],
                reason="schema_mismatch",
                turns=turn,
                tool_names=tuple(tool_names),
                last_error="missing tool name",
                stopped_on_turn=turn,
            )
            append_exit(ledger, event)
            return event

        result = ToolResult(
            ok=bool(call.get("ok")),
            name=name,
            payload=call.get("payload"),
            error=call.get("error"),
        )
        tool_names.append(result.name)

        if result.ok and call.get("final"):
            event = LoopExit(
                run_id=run_id,
                fixture_id=fixture["id"],
                reason="completed",
                turns=turn,
                tool_names=tuple(tool_names),
                last_error=None,
                stopped_on_turn=turn,
            )
            append_exit(ledger, event)
            return event

        if not result.ok:
            last_error = result.error or "tool_error"
            if retries >= MAX_RETRIES:
                event = LoopExit(
                    run_id=run_id,
                    fixture_id=fixture["id"],
                    reason="retry_exhausted",
                    turns=turn,
                    tool_names=tuple(tool_names),
                    last_error=last_error,
                    stopped_on_turn=turn,
                )
                append_exit(ledger, event)
                return event
            retries += 1

    event = LoopExit(
        run_id=run_id,
        fixture_id=fixture["id"],
        reason="tool_error" if last_error else "turn_cap",
        turns=len(planned_calls),
        tool_names=tuple(tool_names),
        last_error=last_error,
        stopped_on_turn=len(planned_calls),
    )
    append_exit(ledger, event)
    return event

The fixture, not the model, decides whether the second weather call succeeds. That split is the teaching move. Students can later swap the fixture for a live tool without changing the exit schema, which keeps grading stable when inference is shared and slow.

Worked example students can rerun

Save the fixture as fixtures/weather_timeout.json. The first call times out. The second call returns a city temperature and marks the loop final. A third planned call exists only to prove the runner never reaches it after completion.

{
  "id": "weather_timeout_v1",
  "planned_calls": [
    {
      "name": "get_weather",
      "ok": false,
      "error": "timeout",
      "final": false
    },
    {
      "name": "get_weather",
      "ok": true,
      "payload": {"city": "Lisbon", "celsius": 22},
      "final": true
    },
    {
      "name": "get_weather",
      "ok": true,
      "payload": {"city": "Lisbon", "celsius": 22},
      "final": true
    }
  ]
}

Commands for a clean rerun on a laptop. No extra services are required for the fixture path.

python - <<'PY'
import json
from pathlib import Path
from run_loop import run_loop

fixture = json.loads(Path("fixtures/weather_timeout.json").read_text())
ledger = Path("artifacts/exits.jsonl")
if ledger.exists():
    ledger.unlink()
event = run_loop("run-001", fixture, ledger)
print(event.reason, event.turns, event.tool_names)
print(ledger.read_text())
PY

Expected line of reasoning for graders, not a live benchmark: reason is completed, turns is 2, and tool_names is ("get_weather", "get_weather"). If a student implementation emits retry_exhausted here, they incremented retries after success or treated timeout as terminal without applying MAX_RETRIES. If they emit three tool names, the loop ignored final and walked the leftover fixture row.

Tests that pin the taxonomy

These tests are the lab contract. They do not measure tokens, latency, or model quality. They measure whether exit reasons stay stable when the fixture is replayed.

# test_loop_exit.py
import json
from pathlib import Path
from run_loop import run_loop

FIX = {
    "id": "weather_timeout_v1",
    "planned_calls": [
        {"name": "get_weather", "ok": False, "error": "timeout", "final": False},
        {
            "name": "get_weather",
            "ok": True,
            "payload": {"city": "Lisbon", "celsius": 22},
            "final": True,
        },
    ],
}


def test_timeout_then_success_completes(tmp_path: Path) -> None:
    ledger = tmp_path / "exits.jsonl"
    event = run_loop("run-001", FIX, ledger)
    assert event.reason == "completed"
    assert event.turns == 2
    assert event.tool_names == ("get_weather", "get_weather")
    lines = ledger.read_text().strip().splitlines()
    assert len(lines) == 1
    body = json.loads(lines[0])
    assert body["reason"] == "completed"
    assert body["last_error"] is None


def test_retry_exhausted_on_double_timeout(tmp_path: Path) -> None:
    fixture = {
        "id": "weather_double_timeout_v1",
        "planned_calls": [
            {"name": "get_weather", "ok": False, "error": "timeout", "final": False},
            {"name": "get_weather", "ok": False, "error": "timeout", "final": False},
        ],
    }
    event = run_loop("run-002", fixture, tmp_path / "exits.jsonl")
    assert event.reason == "retry_exhausted"
    assert event.last_error == "timeout"


def test_missing_name_is_schema_mismatch(tmp_path: Path) -> None:
    fixture = {
        "id": "bad_schema_v1",
        "planned_calls": [{"ok": True, "final": True}],
    }
    event = run_loop("run-003", fixture, tmp_path / "exits.jsonl")
    assert event.reason == "schema_mismatch"

Run the suite with a single command so CI and laptops share the same gate.

pytest -q test_loop_exit.py

Exercise set after the worked example

Assign these in order. Each exercise must produce a new JSONL line, not a longer chat log.

• Exercise A — Turn cap. Extend the fixture to five failing calls and assert turn_cap with MAX_TURNS = 4.
• Exercise B — Schema mismatch. Drop name from the second call only, then prove the first tool name is still recorded.
• Exercise C — Error text hygiene. Replace a stack trace in last_error with a short code such as timeout and reject strings over 80 characters.
• Exercise D — Double run. Execute run-001 twice into the same file and assert two JSONL lines with identical reasons and tool-name tuples.

Exercise D is the grading trick. Instructors can hash the reason plus tool-name tuple and ignore run_id. Students who embed wall-clock timestamps inside last_error will fail the hash even when the control flow is correct, which is a useful lab bruise.

Where a shared free inference queue fits

The fixture path needs no network. The next teaching hour often does, because students want the planner to propose the planned_calls list instead of reading it from disk. That is the moment a classroom burns a pile of personal keys and then cannot reproduce Friday's exits on Monday. A shared endpoint keeps the ledger comparable if every student points at the same runner and the same fixture identifier.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode's free model access and free server option can host that shared runner so the lab does not depend on each student pasting a billed key into a notebook. The exit schema does not change when the planner is remote. Only the producer of planned_calls changes, which is why the tests above stay fixture-first.

Do not treat a shared queue as unlimited capacity or as a promise about any named model. If the class must pause when the queue is busy, freeze planned_calls to disk and keep grading on the ledger. The educational artifact is the exit record, not the freshness of the planner.

Decision table for stop reasons

Use this table on the projector during the last ten minutes. It prevents students from encoding policy in adjectives.

Condition observed	Required reason	last_error	Typical student mistake
Tool payload accepted and final is true	completed	null	Writing completed after a timeout that later recovered
Failures exceed MAX_RETRIES	retry_exhausted	short code	Counting the successful retry as a failure
Tool object lacks a string name	schema_mismatch	missing tool name	Swallowing the row and calling the next tool
Turn index exceeds MAX_TURNS	turn_cap	prior error or null	Off-by-one when enumerating from zero
Tool returns ok=false with no retry left	retry_exhausted or tool_error	short code	Dumping a full exception into the ledger

If two rows could apply, pick the earlier row in the runner, not the more dramatic row in the transcript. Stability beats expressive failure poetry.

Limitations

This ledger is not a trace backend, not a safety interlock, and not a substitute for tool-result validation before memory writes. JSONL append is not atomic across networked filesystems, so a shared lab directory can interleave lines if two processes use one file without a run-id partition. The taxonomy is incomplete on purpose. It will not encode human escalation, policy refusals, or partial credit for almost-correct tool arguments.

Live planners introduce variance that the fixture path removed. If the remote planner emits a different tool order between reruns, the ledger will correctly show instability, which students may misread as a broken test. Teach that difference explicitly. A changing tool_names tuple is a planner bug or a prompt bug, not a pytest bug.

Who should not use this approach

Skip this lab if you already emit OpenTelemetry span status for every tool call and already grade those spans. Skip it if the agent may log personal data into last_error, because JSONL in a homework folder is not an access-controlled store. Skip it for production stop conditions that must halt dangerous tools in process, since this record is written after the call returns. Skip it for courses that cannot freeze a fixture, because a live-only loop will train students to argue with chat logs again.

Teams shipping multi-tenant agents need request isolation, quotas, and audit storage this workshop does not provide. The eighty-minute box is for instructors who need comparable exits across a class, not for operators who need compliance evidence.

Close the loop on Monday

Rerun the weather fixture, keep the five-reason taxonomy frozen, and grade hashes of reason plus tool_names before anyone opens a transcript. If the next session needs a shared planner instead of a disk fixture, point the runner at one free queue rather than collecting keys, and leave the ledger format untouched. Instructors who want that shared queue without building a billing workshop first can use MonkeyCode's free model access and free server option as the lab endpoint, then keep every assertion on the JSONL file.