Gate Agent Success on Closed Spans

The last assistant token is not a status code. If any tool span is still open, the run failed, even when the model wrote a confident wrap-up.

Most agent UIs collapse that distinction. They paint a completed bubble as soon as the model stream ends. The trace underneath is often uglier: a read_file span that started, an apply_patch span that never returned, and a final sentence that assumes both finished. Treating the prose as success is how silent partial work lands on a branch.

This is the same class of bug as an HTTP 200 from a checkout handler while the payment span is still in flight. The outer message completed. The inner contract did not.

What closed means here

A usable agent trace is a small DAG, not a log tail. Each model turn and each tool call is a span with a stable id, a parent id, a start timestamp, and an end timestamp. Status is one of ok, error, or timeout. Missing status is not a fourth kind of success. It is incomplete.

A run is complete only when every span that started has an end, every tool id cited by a model span exists as a tool span, and the root status is derived from children rather than copied from the last token. Incomplete is a failure class of its own. It is not "unknown" and it is not "probably fine."

Free model endpoints and shared free servers make this class common. Idle cuts, process preemption, and dropped streams tend to kill the tool side first and leave the chat side looking finished. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode's free model access and free server option matter here only as a place you can persist those traces; they do not relax the contract. If spans never hit disk, you cannot gate on them.

A proposed checker

The validator below is a proposed gate, not a claim about a private production fleet. It has not been executed against a live workload in this article. Point it at JSON you already collect. The schema is small on purpose so a thin wrapper around any runtime can emit it.

{
  "run_id": "run_7f3a",
  "root_id": "span_root",
  "spans": [
    {
      "id": "span_root",
      "parent_id": null,
      "kind": "run",
      "name": "agent_turn",
      "started_at": "2026-09-06T10:01:00.000Z",
      "ended_at": "2026-09-06T10:01:08.410Z",
      "status": "ok",
      "attrs": {}
    },
    {
      "id": "span_m1",
      "parent_id": "span_root",
      "kind": "model",
      "name": "plan",
      "started_at": "2026-09-06T10:01:00.020Z",
      "ended_at": "2026-09-06T10:01:02.100Z",
      "status": "ok",
      "attrs": {"cites_tool_ids": ["span_t1"]}
    },
    {
      "id": "span_t1",
      "parent_id": "span_root",
      "kind": "tool",
      "name": "apply_patch",
      "started_at": "2026-09-06T10:01:02.120Z",
      "ended_at": null,
      "status": null,
      "attrs": {"tool_call_id": "call_9"}
    }
  ]
}

That sample is a failure. The root claims ok. The model span cites span_t1. The tool span never ended. A human reading the final message would miss it. A checker will not.

#!/usr/bin/env python3
"""Proposed agent-trace completeness gate. Unexecuted example."""
from __future__ import annotations

import json
import sys
from hashlib import sha256
from pathlib import Path
from typing import Any

ALLOWED_STATUS = {"ok", "error", "timeout"}


def load_trace(path: Path) -> dict[str, Any]:
    data = json.loads(path.read_text())
    if not isinstance(data.get("spans"), list) or not data.get("run_id"):
        raise ValueError("trace must include run_id and spans[]")
    return data


def issues_for(trace: dict[str, Any]) -> list[str]:
    spans = {s["id"]: s for s in trace["spans"] if "id" in s}
    problems: list[str] = []
    if trace.get("root_id") not in spans:
        problems.append("missing_root")
        return problems

    for span in spans.values():
        if not span.get("started_at"):
            problems.append(f"no_start:{span['id']}")
        ended = span.get("ended_at")
        status = span.get("status")
        if ended is None or status not in ALLOWED_STATUS:
            problems.append(
                f"open_span:{span['id']}:{span.get('kind')}:{span.get('name')}"
            )
        parent = span.get("parent_id")
        if parent and parent not in spans:
            problems.append(f"orphan_parent:{span['id']}")

    for span in spans.values():
        if span.get("kind") != "model":
            continue
        cited = span.get("attrs", {}).get("cites_tool_ids", [])
        for tool_id in cited:
            child = spans.get(tool_id)
            if not child or child.get("kind") != "tool":
                problems.append(f"cited_missing_tool:{span['id']}->{tool_id}")

    root = spans[trace["root_id"]]
    child_bad = any(
        s.get("status") in {"error", "timeout"} or s.get("ended_at") is None
        for s in spans.values()
        if s.get("id") != root["id"]
    )
    if child_bad and root.get("status") == "ok":
        problems.append("root_ok_with_incomplete_or_failed_child")
    return problems


def fingerprint(problems: list[str]) -> str:
    blob = "|".join(sorted(problems)).encode()
    return sha256(blob).hexdigest()[:12]


def main(argv: list[str]) -> int:
    if len(argv) != 2:
        print("usage: check_trace.py TRACE.json", file=sys.stderr)
        return 2
    trace = load_trace(Path(argv[1]))
    problems = issues_for(trace)
    result = {
        "run_id": trace["run_id"],
        "complete": not problems,
        "problem_count": len(problems),
        "problems": problems,
        "fingerprint": fingerprint(problems) if problems else None,
    }
    print(json.dumps(result, indent=2))
    return 0 if result["complete"] else 1


if __name__ == "__main__":
    raise SystemExit(main(sys.argv))

Wire it as a step that can fail the job. Dashboards are optional. The process exit code is not.

python3 check_trace.py traces/run_7f3a.json
echo $?
# expected for the sample: 1

Exit code 1 means do not merge, do not apply the patch, do not send the email the agent drafted. Exit code 0 means the trace is structurally closed. It still does not mean the patch is correct.

The fingerprint is the part that scales. Incomplete runs cluster. open_span:span_t1:tool:apply_patch plus root_ok_with_incomplete_or_failed_child will hash the same way across nights if the same tool is the one that never returns. Debug the cluster. Do not debug each cheerful wrap-up as a unique prompt failure.

A tiny pytest wrapper makes the same rule visible next to unit tests. Label it as a contract test on fixtures, not as evidence that a hosted agent passed.

# proposed test; fixtures live under traces/
from pathlib import Path
from check_trace import issues_for, load_trace


def test_open_tool_span_fails_the_run():
    trace = load_trace(Path("traces/run_7f3a.json"))
    problems = issues_for(trace)
    assert "root_ok_with_incomplete_or_failed_child" in problems
    assert any(p.startswith("open_span:") for p in problems)

Emit the end event locally

When you wrap a tool, the local process is the source of truth for span closure, not the model. The model can still emit "done" after you kill the child. Record timeout yourself.

# proposed wrapper sketch; timestamps come from the wrapper clock
import subprocess, time
from datetime import datetime, timezone


def utcnow():
    return datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%S.%fZ")


def run_tool(span: dict, argv: list[str], timeout_s: float) -> dict:
    span["started_at"] = utcnow()
    span["kind"] = "tool"
    try:
        completed = subprocess.run(argv, capture_output=True, text=True, timeout=timeout_s)
        span["ended_at"] = utcnow()
        span["status"] = "ok" if completed.returncode == 0 else "error"
        span.setdefault("attrs", {})["exit_code"] = completed.returncode
    except subprocess.TimeoutExpired:
        span["ended_at"] = utcnow()
        span["status"] = "timeout"
    except Exception as exc:
        span["ended_at"] = utcnow()
        span["status"] = "error"
        span.setdefault("attrs", {})["exception"] = type(exc).__name__
    return span

That except path is what turns a dropped connection on a free server into a closed error instead of a fake success. Use the wrapper clock for started_at and ended_at. Do not trust a duration the model narrates. Models round. Wrappers do not have to.

Keep raw logs. They are not the gate. A file diff is also not the gate. An empty diff plus an open apply_patch span is still a failed run, because the agent claimed work it never finished. The order is mechanical: close the spans, then read the diff, then read the model text.

A parent id on every tool is necessary and not sufficient. Ordering among stdout lines is a different bug. Completeness is coarser and cheaper. Did the span end, and did the root inherit that fact.

Limits, and who should skip this

The checker will not catch a tool that returned ok with the wrong bytes. It will not catch a model that never cited a tool it should have called. It will not reconstruct work if you never persisted spans. Cardinality still matters: do not stuff full file contents into attributes or the trace store becomes the bottleneck.

Skip the approach if the session has no tools, if the runtime already fails a parent when a child is open, or if the question you actually have is semantic eval, not structure. A completeness gate is a filter in front of those evals. It is not a substitute.

Vendor quotas, hardware SKUs, and model lists are omitted on purpose. Those figures move. A closed-span rule does not. If you already persist traces from a free model path, run the checker on a day's files before you change prompts. The prompt is rarely the first bug. An open apply_patch span usually is.