Agent Traces Hide the Fork. Record Ranked Options.

A coding agent returned a green run at 02:11.
The ticket asked for a small test helper.
This scene is a constructed on-call example.

The agent called search_repo and then edit_file.
Both tool spans closed with status ok.
The patch landed in the wrong Python package.

The JSONL export listed every call in order.
It never listed the tools the planner rejected.
Ops could not replay the fork that mattered.

Execution logs are not routing logs

Most agent traces are still plain call journals.
They store name, arguments, latency, and final status.
They omit the ranked options behind the pick.

A green tool span answers one narrow question.
It only asks if the invocation threw an error.
It does not answer why this tool won.

Loop-style agents fail at the fork often.
The wrong search query still returns hits.
The wrong file still accepts a clean edit.

Those runs look healthy in span dashboards.
The user still receives a useless change.
You need a record of discarded alternatives.

What a decision span must capture

Record the choice before the tool starts.
Do not reconstruct intent from later arguments.
Arguments describe the winner, not the field.

Each decision span needs a stable shape.
Keep it small enough to grep in JSONL.
Keep it strict enough to validate in CI.

Use these minimum fields on every decision span:

• Store span_id and parent_id for tree joins.
• Store ts in UTC with second precision.
• Store goal as a short remaining task.
• Store options as scored tool candidates.
• Store chosen matching one option name.
• Store policy naming the selection rule.
• Store margin as winner minus runner-up.

Skip free-text essays in this span.
Put long prompt blobs in a separate event.
Decision records should stay cheap to diff.

A concrete JSONL fragment

Example below is illustrative, not a live export.

{"type":"decision","span_id":"dec_7f3a","parent_id":"run_91","ts":"2026-09-13T02:11:04Z","goal":"add a date-parse test helper","options":[{"tool":"search_repo","score":0.81,"reason":"find existing helpers"},{"tool":"edit_file","score":0.44,"reason":"patch a remembered path"},{"tool":"run_tests","score":0.12,"reason":"no diff exists yet"}],"chosen":"search_repo","policy":"argmax_score","margin":0.37}
{"type":"tool_call","span_id":"call_12","parent_id":"dec_7f3a","ts":"2026-09-13T02:11:05Z","tool":"search_repo","args":{"query":"date parse helper"}}
{"type":"tool_result","span_id":"res_12","parent_id":"call_12","ts":"2026-09-13T02:11:06Z","ok":true,"bytes":842}

Notice the parent link on the tool call.
The call hangs under the decision, not the run.
That join is the entire debugging upgrade.

Validator you can run on a file

The next script is a local example.
It is not a measured production harness.
Save it as check_decisions.py beside a JSONL trace.

#!/usr/bin/env python3
"""Flag tool calls that lack a scored decision."""
from __future__ import annotations

import json
import sys
from dataclasses import dataclass
from typing import Any


@dataclass
class Finding:
    line: int
    kind: str
    detail: str


def load_events(path: str) -> list[tuple[int, dict[str, Any]]]:
    rows: list[tuple[int, dict[str, Any]]] = []
    with open(path, encoding="utf-8") as handle:
        for line_no, raw in enumerate(handle, start=1):
            text = raw.strip()
            if not text:
                continue
            rows.append((line_no, json.loads(text)))
    return rows


def margin_of(options: list[dict[str, Any]], chosen: str) -> float | None:
    scores = {str(item["tool"]): float(item["score"]) for item in options}
    if chosen not in scores or len(scores) < 2:
        return None
    winner = scores[chosen]
    rest = [value for name, value in scores.items() if name != chosen]
    return winner - max(rest)


def audit(path: str) -> list[Finding]:
    findings: list[Finding] = []
    pending: dict[str, dict[str, Any]] = {}
    for line, event in load_events(path):
        kind = event.get("type")
        if kind == "decision":
            options = event.get("options") or []
            chosen = str(event.get("chosen") or "")
            tools = [str(item.get("tool")) for item in options]
            if chosen not in tools:
                findings.append(Finding(line, "chosen_missing", chosen))
            computed = margin_of(options, chosen)
            reported = event.get("margin")
            if computed is not None and reported is not None:
                if abs(float(reported) - computed) > 1e-6:
                    findings.append(Finding(line, "bad_margin", str(reported)))
            pending[str(event.get("span_id"))] = event
            continue
        if kind == "tool_call":
            parent = str(event.get("parent_id") or "")
            tool = str(event.get("tool") or "")
            decision = pending.pop(parent, None)
            if decision is None:
                findings.append(Finding(line, "missing_decision", tool))
                continue
            if str(decision.get("chosen")) != tool:
                findings.append(
                    Finding(line, "choice_mismatch", f"{decision.get('chosen')}->{tool}")
                )
    for leftover in pending.values():
        findings.append(
            Finding(0, "unconsumed_decision", str(leftover.get("span_id")))
        )
    return findings


def main() -> int:
    if len(sys.argv) != 2:
        print("usage: check_decisions.py trace.jsonl", file=sys.stderr)
        return 2
    findings = audit(sys.argv[1])
    for item in findings:
        print(f"{item.kind}\tL{item.line}\t{item.detail}")
    print(f"finding_count\t{len(findings)}")
    return 1 if findings else 0


if __name__ == "__main__":
    raise SystemExit(main())

Run it against a tiny fixture first.

python3 check_decisions.py sample.jsonl

A clean file prints a zero finding count.
A missing parent prints the missing_decision kind.
A rewritten winner prints the choice_mismatch kind.

A debug loop that diffs forks

Do not start with model quality debates.
Start with two traces of the same ticket.
Keep the user prompt identical on both runs.

Work the loop in this fixed order:

1. Confirm both JSONL files parse line by line.
2. Extract the sequence of chosen tools only.
3. Extract the option sets sitting under each choice.
4. Flag any tool call without a parent decision.
5. Flag margins below a local threshold, such as 0.05.
6. Diff the two chosen sequences as plain text.
7. Only then inspect arguments on the first mismatch.

Example commands for steps two and six:

jq -r 'select(.type=="decision") | .chosen' a.jsonl > a.chosen.txt
jq -r 'select(.type=="decision") | .chosen' b.jsonl > b.chosen.txt
diff -u a.chosen.txt b.chosen.txt

If the chosen lists match, the bug is downstream.
If they diverge, stop reading tool payloads.
Read the option scores at the first split.

Low margin is a routing smell, not a proof.
A 0.02 gap means the planner was almost tied.
Tied planners need a policy, not another retry.

How this differs from call diffs

Call diffs tell you arguments changed later.
Decision diffs tell you the menu changed first.
Those are different bugs with different fixes.

A search query tweak is an argument bug.
A missing search_repo option is a prompt bug.
A tiny margin is a scoring bug.

Treat those three cases as separate tickets.
Mixing them produces noisy agent evaluation reports.
Routing quality is not the same as tool quality.

Where a scratch model host helps

You can run the validator on a laptop.
Replaying the agent still needs a model endpoint.
It also needs a quiet machine for two traces.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.
MonkeyCode currently offers free model access and a free server option.
Use that pair only as a scratch replay host.

They are not a substitute for a decision schema.
Add the spans first, then replay on any host you already use.
Keep the product choice secondary to the span contract.

Capture two JSONL files for the same ticket.
Compare chosen lists before you compare patches.
Stop at the first fork that changed rank.

Limitations

Model-supplied scores are not calibrated success probabilities.
Do not treat 0.81 as an 81 percent success chance.
Use scores only to rank options inside one span.

Option reason fields can leak private context.
Redact paths, tokens, and customer names before storage.
Drop reasons in production if your store is wide.

This pattern adds one event per tool call.
Busy loops will grow trace volume quickly.
Sample long exploratory runs, never the decision keys.

Clock skew still disorders events across hosts.
The validator assumes a single writer per file.
Do not merge JSONL from two clocks without ids.

Who should not use this

Skip this if the agent has one hard-coded tool.
There is no fork, so ranking adds noise.

Skip this if you cannot store traces at all.
A schema without a file is documentation theater.

Skip this for binary policy routers with no scores.
Log the rule name, not a fake numeric margin.

Skip this when option text includes secrets.
Fix redaction before you add richer spans.

What to keep in the next incident

When a green agent still ships a bad edit, ask four questions.

• Did every tool call cite a decision parent?
• Did chosen match the tool that actually ran?
• Was the margin too small to trust the pick?
• Did a second run diverge at the same fork?

If you cannot answer those from the file, the trace is incomplete.
Call journals will keep looking green under dashboards.
The rejected options will keep disappearing from review.

Ship the schema before you shop for another dashboard.
A ranked option list is a small, testable artifact.
It survives even if you change models next week.