An agent loop that only prints iteration=7 is already unbounded. The process can die, two retries can interleave, and a remote model hop can swallow stdout. If the counter and the halt reason are not attributes on a closed span, you do not have a halt condition. You have a log line that hoped someone would read it.
This is not a style complaint. It is a failure-mode complaint. Agent work is a loop with tools, and the loop is the thing that runs away. HTTP services taught us to sample traces. Agent loops punish that habit. A sampled-away iteration is indistinguishable from an iteration that never ran.
The reusable rule is small. Start a parent span for the run. Open a child span for every iteration. Nest each tool call under that child. Write iteration, budget_remaining, and halt_reason as attributes. Close the child before the next hop. If the child never closes, treat the run as a crash, not as a slow success.
Why the print statement lies
Stdout is a stream. A stream has no identity when two workers share a terminal, when a free remote endpoint truncates the response, or when the process is SIGKILLed after the model returns and before print flushes. The line you wanted is the line you lost.
A span is a record with a start, an end, and a parent. That shape survives a killed child. It also survives a model call that leaves the laptop. The remote hop can still fail. It cannot pretend the iteration never existed if the parent already opened a child with a stable work id.
Think of the loop like a checkout line. The receipt is not the chatter at the register. The receipt is the numbered ticket that still exists after the cashier walks away. Agent traces need that ticket. They do not need a louder log level.
A minimum span schema for one loop
Keep the schema boring. Boring is queryable.
run name=agent.run attrs: run_id, goal_hash, max_iters
iter name=agent.iter attrs: iteration, budget_remaining, work_id
llm name=agent.llm attrs: role=plan|act, input_hash, output_hash
tool name=agent.tool attrs: tool, schema_hash, ok, error_class
end name=agent.halt attrs: halt_reason, last_iteration
work_id is the idempotency key for the unit of work, not the retry count. Retries of the same tool input keep the same work_id and increment attempt. That stops a retry storm from looking like progress. schema_hash is a digest of the tool’s argument schema at call time. If the hash changes between iterations, the loop is not looping. It is drifting.
halt_reason is an enum you own: budget_exhausted, tool_error, span_open, schema_drift, done. Anything else is unknown, and unknown is a failed run. Do not encode poetry in that field. Downstream alerts cannot parse poetry.
A local writer you can grep
The code below is a proposed, self-contained writer. It is not a production tracer. It appends JSON lines and refuses to start the next iteration while a child is still open. Label it as a workflow you can run on a laptop, not as a measured benchmark.
# loop_trace.py — proposed local span writer for agent loops
from __future__ import annotations
import hashlib, json, time, uuid
from contextlib import contextmanager
from pathlib import Path
from typing import Any, Iterator
HALT_REASONS = {
"done", "budget_exhausted", "tool_error",
"span_open", "schema_drift", "unknown",
}
def _now() -> float:
return time.time()
def schema_hash(schema: dict) -> str:
blob = json.dumps(schema, sort_keys=True, separators=(",", ":")).encode()
return hashlib.sha256(blob).hexdigest()[:16]
class LoopTrace:
def __init__(self, path: Path, run_id: str, max_iters: int) -> None:
self.path = path
self.run_id = run_id
self.max_iters = max_iters
self._open_iter: str | None = None
self.last_schema: dict[str, str] = {}
path.parent.mkdir(parents=True, exist_ok=True)
self._emit("agent.run", "start", {"run_id": run_id, "max_iters": max_iters})
def _emit(self, name: str, event: str, attrs: dict[str, Any]) -> None:
rec = {
"ts": _now(),
"run_id": self.run_id,
"span_id": uuid.uuid4().hex[:16],
"parent_open": self._open_iter,
"name": name,
"event": event,
"attrs": attrs,
}
with self.path.open("a", encoding="utf-8") as fh:
fh.write(json.dumps(rec, ensure_ascii=False) + "\n")
@contextmanager
def iteration(self, n: int) -> Iterator[dict[str, Any]]:
if self._open_iter is not None:
self.halt("span_open", n)
raise RuntimeError("previous iteration span still open")
if n > self.max_iters:
self.halt("budget_exhausted", n)
raise RuntimeError("iteration exceeds max_iters")
remaining = self.max_iters - n
work_id = f"{self.run_id}:{n}"
self._open_iter = work_id
self._emit("agent.iter", "start", {
"iteration": n,
"budget_remaining": remaining,
"work_id": work_id,
})
try:
yield {"work_id": work_id, "budget_remaining": remaining}
finally:
self._emit("agent.iter", "end", {"work_id": work_id, "iteration": n})
self._open_iter = None
def tool(self, name: str, schema: dict, ok: bool, error_class: str | None = None) -> None:
digest = schema_hash(schema)
prev = self.last_schema.get(name)
if prev and prev != digest:
self.halt("schema_drift", -1)
raise RuntimeError(f"schema drift for {name}: {prev} -> {digest}")
self.last_schema[name] = digest
self._emit("agent.tool", "event", {
"tool": name,
"schema_hash": digest,
"ok": ok,
"error_class": error_class,
})
if not ok:
self.halt("tool_error", -1)
def halt(self, reason: str, last_iteration: int) -> None:
if reason not in HALT_REASONS:
reason = "unknown"
self._emit("agent.halt", "end", {
"halt_reason": reason,
"last_iteration": last_iteration,
"span_still_open": self._open_iter is not None,
})
A thin loop around that writer makes the policy visible. The model call is a function you inject. Keep the injection local so a remote hop cannot close your spans for you.
# run_loop.py — proposed control loop, not an executed benchmark
from pathlib import Path
from loop_trace import LoopTrace
MAX_ITERS = 8
TOOL_SCHEMA = {"type": "object", "properties": {"path": {"type": "string"}}}
def call_model(prompt: str) -> dict:
# Replace with your client. The span must start and end around this call.
return {"action": "tool", "tool": "read_file", "ok": True}
def main() -> None:
trace = LoopTrace(Path("runs/latest.jsonl"), run_id="run-20260911", max_iters=MAX_ITERS)
halt = "unknown"
last = 0
try:
for n in range(1, MAX_ITERS + 1):
last = n
with trace.iteration(n):
decision = call_model(f"iter={n}")
if decision["action"] == "stop":
halt = "done"
break
trace.tool(decision["tool"], TOOL_SCHEMA, ok=bool(decision["ok"]))
else:
halt = "budget_exhausted"
except RuntimeError as exc:
halt = "span_open" if "open" in str(exc) else "unknown"
raise
finally:
trace.halt(halt, last)
if __name__ == "__main__":
main()
The interesting line is not the model client. It is with trace.iteration(n). That context manager is the cap. If call_model blocks forever, the child stays open, and the next process that reads the JSONL can see an agent.iter start with no matching end. That is a cheaper crash signal than a wall-clock timeout you invented after the fact.
Query the run, do not tail the terminal
Once the file exists, the debug loop is a query. No dashboard required.
python run_loop.py
jq -c 'select(.name=="agent.iter") | {event, iteration:.attrs.iteration, work:.attrs.work_id}' runs/latest.jsonl
jq -c 'select(.name=="agent.halt") | .attrs' runs/latest.jsonl
# unpaired starts: iterations that never closed
jq -s '
map(select(.name=="agent.iter"))
| group_by(.attrs.work_id)
| map(select(map(.event)|index("end")|not))
| .[] | .[0].attrs
' runs/latest.jsonl
A proposed regression check belongs next to the writer. It does not prove production behavior. It proves the writer refuses a second start.
# test_loop_trace.py — proposed tests
from pathlib import Path
from loop_trace import LoopTrace
import pytest
def test_open_child_blocks_next_iter(tmp_path: Path) -> None:
t = LoopTrace(tmp_path / "r.jsonl", "r1", max_iters=3)
cm = t.iteration(1)
cm.__enter__()
with pytest.raises(RuntimeError, match="still open"):
with t.iteration(2):
pass
def test_schema_drift_halts(tmp_path: Path) -> None:
t = LoopTrace(tmp_path / "r.jsonl", "r2", max_iters=3)
with t.iteration(1):
t.tool("read_file", {"properties": {"path": {"type": "string"}}}, ok=True)
with t.iteration(2):
with pytest.raises(RuntimeError, match="schema drift"):
t.tool("read_file", {"properties": {"id": {"type": "integer"}}}, ok=True)
Run them with pytest -q test_loop_trace.py. If the first test ever goes green while a child is open, the rest of the workflow is theater.
The remote hop is why this has to be local
Agent loops often send the planning step off-box. That is the moment stdout becomes fiction. The laptop still owns the iteration. The remote endpoint owns a completion. Mix those clocks and you will debug the network instead of the loop.
Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode offers free model access and a free server option, which is enough to exercise the remote hop without standing up a private GPU box. Use that hop as the call_model body. Do not let it own halt_reason. The writer above starts and ends on your side of the request. The free server can return a tool decision. It cannot close a span it never opened.
Keep payloads out of the JSONL if the goal text is sensitive. Store hashes, not prompts. The earlier data-boundary problem still exists; this article does not re-solve it. It only keeps the loop counter from riding along with the prompt into a remote log you do not control.
A practical split looks like this. Local process: open agent.iter, hash the prompt, call the remote model, record output_hash, close agent.iter. Remote process: answer the completion. If the free server is also where you park a tiny collector, ship the JSONL after the halt span, not during the tool call. Export during the call is how you duplicate bodies and blow a quiet size cap you did not measure.
What this does not do
It does not replace OpenTelemetry. If your team already emits OTLP with a real collector, map iteration and halt_reason onto that pipeline and delete this file. It does not give you tail sampling. Every iteration is written because a missing iteration is the bug. If a run can do thousands of cheap tool calls, this writer will grow a large JSONL. Cap max_iters first. Do not “fix” size by dropping ends.
It does not prove the model was right. A halt_reason=done span means the loop stopped. It does not mean the tool did the useful thing. Pair this with a closed-span success gate if you need that check. This piece only makes the counter survive.
Skip this approach if the agent is a single model call with no tools. Skip it if you cannot write to disk on the machine that starts the loop. Skip it for multi-tenant agents that put raw user text into span attributes. Hash or drop those fields. The writer will not redact them for you.
Time-sensitive product numbers change. This workflow does not depend on a published token quota, a hardware SKU, or a durability promise. It depends on one closed child per iteration. If you cannot attest that, the loop is still printing.
The core test is one command after a crash: jq 'select(.name=="agent.halt")' runs/latest.jsonl. Empty output means the run never halted. That is the bug. Fix the writer before you tune the prompt.
Top comments (0)