Join Tool Exits to Spans Before Trusting the Summary

A model’s closing paragraph is not a trace. The only agent run you can debug is the one where every tool process exit inner-joins to a closed span on a caller-issued invocation_id, and where span status is a function of that exit code rather than of later assistant text.

That rule sounds pedantic until a coding agent shells out twice, retries once, and then writes “ran the tests.” The summary is one sentence. The machine did three things. Two of them may have failed. If those failures never became spans, the debug loop starts from fiction.

Think of baggage tags. A passenger name on a boarding pass does not locate a suitcase. The tag number does. Tool names in a chat transcript are passenger names. invocation_id is the tag. Order of appearance in the log is not a join key. Concurrent tools scramble wall-clock order, and retries mint extra exits that never show up in the last message.

What the join actually asserts

Treat the run as two relations that must agree. Relation E is process exits: one row per OS-level tool invocation, with invocation_id, exit_code, ended_at, and a hash of stdout plus stderr. Relation S is closed spans: one row per span that claimed to wrap a tool, with the same invocation_id, a mapped span_status, and closed_at.

A run is join-complete when three predicates hold. Every row in E matches exactly one row in S. Every row in S matches exactly one row in E. For each matched pair, span_status equals the mapping of exit_code (zero to ok, anything else to error). Fail the run on leak, orphan, or mismatch. Do not “warn and continue.” A warning is how unmatched exits become folklore.

The mapping is deliberately crude. A test harness that prints FAILED and still exits 0 is a product bug in the tool, not a reason to let the model overwrite span status. If you need richer outcomes, put them on span events. Status stays a function of the process.

A proposed local harness

The checker below is a proposed local harness, not a production trace backend. It writes JSONL. It does not sample. Label it as unexecuted in your environment until you run it against a real agent wrapper.

# join_exits.py — proposed checker, file-backed JSONL
from __future__ import annotations

import hashlib, json, os, subprocess, time, uuid
from pathlib import Path

TRACE = Path(os.environ.get("TRACE_JSONL", "run.jsonl"))

def _write(row: dict) -> None:
    row["ts"] = time.time()
    with TRACE.open("a", encoding="utf-8") as f:
        f.write(json.dumps(row, sort_keys=True) + "\n")

def map_status(exit_code: int) -> str:
    return "ok" if exit_code == 0 else "error"

def run_tool(argv: list[str]) -> int:
    invocation_id = str(uuid.uuid4())
    env = os.environ.copy()
    env["INVOCATION_ID"] = invocation_id
    _write({"type": "span_open", "invocation_id": invocation_id, "argv": argv})
    proc = subprocess.run(argv, env=env, capture_output=True)
    payload = proc.stdout + b"\0" + proc.stderr
    digest = hashlib.sha256(payload).hexdigest()
    status = map_status(proc.returncode)
    _write({
        "type": "process_exit",
        "invocation_id": invocation_id,
        "exit_code": proc.returncode,
        "stdout_stderr_sha256": digest,
        "bytes": len(payload),
    })
    _write({
        "type": "span_close",
        "invocation_id": invocation_id,
        "span_status": status,
        "exit_code": proc.returncode,
    })
    return proc.returncode

def load_rows() -> list[dict]:
    if not TRACE.exists():
        return []
    return [json.loads(line) for line in TRACE.read_text().splitlines() if line.strip()]

def join_report(rows: list[dict]) -> dict:
    exits = {r["invocation_id"]: r for r in rows if r.get("type") == "process_exit"}
    closes = {r["invocation_id"]: r for r in rows if r.get("type") == "span_close"}
    leaks = sorted(set(exits) - set(closes))
    orphans = sorted(set(closes) - set(exits))
    mismatches = sorted(
        i for i in set(exits) & set(closes)
        if closes[i]["span_status"] != map_status(exits[i]["exit_code"])
    )
    return {
        "exit_count": len(exits),
        "close_count": len(closes),
        "leaks": leaks,
        "orphans": orphans,
        "status_mismatches": mismatches,
        "join_ok": not (leaks or orphans or mismatches),
    }

if __name__ == "__main__":
    import sys
    cmd = sys.argv[1:]
    if cmd[:1] == ["--report"]:
        report = join_report(load_rows())
        print(json.dumps(report, indent=2))
        raise SystemExit(0 if report["join_ok"] else 2)
    if not cmd:
        raise SystemExit("usage: join_exits.py <tool> [args...] | --report")
    raise SystemExit(run_tool(cmd))

Wrap the agent’s tool runner so it cannot exec except through run_tool. A one-line shell stand-in is enough to prove the join on a laptop:

export TRACE_JSONL="$PWD/run.jsonl"
: > "$TRACE_JSONL"
python join_exits.py python -c "print('ok')"
python join_exits.py python -c "raise SystemExit(1)"
python join_exits.py --report

The second command must produce span_status of error. If your agent later prints “all good,” the report still fails. That is the point. The summary is not a source row in either relation.

How summaries lie without leaking bytes

Three failure modes show up constantly, and none of them require a sophisticated model. The agent retries a formatter, the first process exits 1, the second exits 0, and only the second is wrapped in a span. Relation E has two rows. Relation S has one. The leak is the first exit. The summary mentions a single format step because language models compress.

The inverse also happens. A span is opened for pytest, the process is never started, and the span is closed ok because the orchestrator timed out of the thought loop and needed a status. That is an orphan. It is worse than a leak. A leak hides work. An orphan invents work.

Status mismatch is quieter. The process exits 2. A post-processor, or the model itself, writes span_status: ok because tests were “mostly fine.” The inner join on invocation_id succeeds. The run still has to fail. Id equality is not semantic equality.

If you want a regression hook, do not grep the assistant message. Assert the report:

python join_exits.py --report
test "$(python -c 'import json,sys; print(json.load(sys.stdin)["join_ok"])' < <(python join_exits.py --report))" = True

Better: read the JSON. Gate CI on join_ok, exit_count, and a ceiling on error closes. A bound on tool exits is an observability SLO. It is not a token budget. Token counters live in another pipeline and do not prove that a process was traced.

Where a free model host fits, and where it does not

The join does not need a particular vendor. It needs a wrapper you control. When the loop also needs a model to propose the next tool, standing up paid inference just to debug tracing is a poor trade. MonkeyCode is an open-source project that currently offers free model access and a free server option for that kind of harness work. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Keep secrets off that path. The checker hashes stdout and stderr; it does not redact them at the source. If a tool prints a token or a customer fixture, the JSONL becomes a credential store. Run the free server option only on synthetic repos and fixtures you would paste into a ticket. The join logic itself should stay local, next to the working tree.

Strip the product out and the method is unchanged. You still emit span_open, process_exit, and span_close. You still fail on leaks. That is the usefulness test.

Limitations, and who should skip this

This join assumes tools are processes. In-process function calls that never exec will not appear in relation E unless you fabricate an exit row. Fabricated exits are how orphans are born. If your agent’s tools are Python callables, instrument the callable, not a fake subprocess.

It also assumes one exit per invocation_id. Pipelines that reuse an id across retries will collapse two exits into one dict key in the sample checker. Retries need new ids. Correlate them with a parent_invocation_id field if you must, but do not recycle the primary key.

Do not use this as a substitute for OpenTelemetry if you already have a collector, span processors, and identity at the RPC layer. Dual-writing JSONL and OTLP without a single id space creates a second forest. Do not use it on jobs whose stdout is legally sensitive. Hashing does not erase the bytes you wrote before the hash.

Clock skew is out of scope. The join key is the id, not ended_at ≈ closed_at. If you add a time window as a hint, keep it diagnostic. Never let a 50ms skew decide that an exit “probably” belongs to a span with a different id.

The last assistant message can still be useful after the join passes. Read it as commentary. Commentary is not a row. If the join fails, do not debate the commentary. Fix the wrapper until leaks, orphans, and status_mismatches are empty lists, then read the prose.

If your traces are already JSONL, run join_exits.py --report on the last failed job before you add another log line. Extra logs without a join key are more passenger names. They will not find the bag.