A Trace Contract and a Triage Table: Debugging Agent Runs in Cost Order

A run fails at 02:14. The trace file has 1,400 spans and 9 MB.

Three engineers open it at once. Each starts somewhere different.

One reads the model output. One reads the tool results. One greps for "error".

This is not a data problem. The trace is complete.

The problem is order. Agent debugging is mostly a scheduling question.

The trace is not the artifact

Teams collect traces and stop there. A file is not a workflow.

What you actually reuse is the sequence of checks. Same order, every time.

Two properties decide your time-to-cause:

• Which fields are guaranteed to exist. Missing fields force improvisation.
• Which check runs first. Cheap checks should never wait behind expensive ones.

Both are design choices. Neither is provided by your tracing library.

Write the contract before the trace

A trace contract is a short list of fields you refuse to live without.

Field	Type	Why it must exist
run_id	string	Groups spans into one replayable unit
seq	int	Orders spans when timestamps tie
span_id / parent_id	string / nullable	Gives you a tree, not a list
kind	string	Separates model, tool, and host events
started_at / duration_ms	number	Enables gap scans
status	string	Makes the cheapest check possible
args_hash / result_hash	string	Compares tool calls without reading payloads

Seven fields. Roughly 200 bytes per span. That is a cheap insurance policy.

The hashes matter most. They let you diff two runs without shipping prompts to a diff tool.

Enforce the contract in CI

A contract nobody checks is a comment. Validate traces on every CI run.

# trace_contract.py — stdlib only, no dependencies
import json, sys

REQUIRED = {
    "run_id": str,
    "seq": int,
    "span_id": str,
    "parent_id": (str, type(None)),
    "kind": str,
    "started_at": (int, float),
    "duration_ms": (int, float),
    "status": str,
}

TOOL_ONLY = {"tool_name": str, "args_hash": str, "result_hash": str}

def check(span, line_no):
    errs = []
    for field, typ in REQUIRED.items():
        if field not in span:
            errs.append(f"line {line_no}: missing {field}")
        elif not isinstance(span[field], typ) or isinstance(span[field], bool):
            errs.append(f"line {line_no}: {field} is {type(span[field]).__name__}")
    if span.get("kind") == "tool":
        for field, typ in TOOL_ONLY.items():
            if not isinstance(span.get(field), typ):
                errs.append(f"line {line_no}: tool span missing {field}")
    return errs

def main(path):
    errs, seqs = [], []
    with open(path) as fh:
        for i, line in enumerate(fh, 1):
            if not line.strip():
                continue
            span = json.loads(line)
            errs += check(span, i)
            seqs.append(span.get("seq"))
    if seqs != sorted(seqs):
        errs.append("seq is not monotonic: spans arrived out of order")
    if len(set(seqs)) != len(seqs):
        errs.append("duplicate seq: two spans claim the same slot")
    for e in errs:
        print(e)
    print(f"{len(seqs)} spans, {len(errs)} violations")
    return 1 if errs else 0

if __name__ == "__main__":
    sys.exit(main(sys.argv[1]))

Two notes on the code. isinstance(True, int) is True in Python, so the bool guard is not decoration.

Second: the seq checks catch silent truncation. A trace that quietly drops the last 30 spans still looks healthy.

Run it in CI against one recorded failure per week:

python trace_contract.py fixtures/run-2026-11-08.jsonl

Failing this check is cheaper than debugging without the fields.

Order your checks by cost

Here is the triage table. Read top to bottom, stop when you find a signal.

#	Symptom	Check	Cost
1	"The output is wrong"	status != ok scan	one pass, no payload read
2	"A step is missing"	seq monotonic + duplicate check	one pass
3	"The tool lied"	args_hash / result_hash diff vs last green run	one pass
4	"It got slower"	gap scan between consecutive spans	one pass
5	"It looped"	count spans per (tool_name, args_hash)	one pass
6	"The model is bad"	all of the above first	expensive, run last

Row 6 is the punchline. Model blame is the most expensive hypothesis and the least often correct.

Each row is one pass over a file. No row needs a GPU, a dashboard, or a meeting.

The loop, in five steps

1. Freeze the input. Record the trace file and the environment fingerprint together. A replay against a different host proves nothing.
2. Validate. Run the contract check. Fix the trace before you fix the agent.
3. Triage. Walk the table in order. Write down which rows you cleared.
4. Localize. Pick the span window between the last good span and the first bad one. Ignore everything else.
5. Verify one hypothesis. Change one thing, replay, diff the same window. Two changes means two stories.

Step 5 is where most loops die. People patch three things and learn nothing.

Gap scanning, concretely

Timing gaps are cheap and reveal surprising things. This script prints gaps over one second.

# gaps.py — largest wall-clock gaps between consecutive spans
import json, sys

spans = sorted((json.loads(l) for l in open(sys.argv[1]) if l.strip()),
               key=lambda s: s["seq"])

for a, b in zip(spans, spans[1:]):
    gap = b["started_at"] - (a["started_at"] + a["duration_ms"] / 1000)
    if gap > 1.0:
        print(f"{gap:7.2f}s  seq {a['seq']} ({a['kind']}) -> seq {b['seq']} ({b['kind']})")

A 40-second gap before a tool span means something waited on the network. A gap before a model span means something waited on a provider.

Same duration, different owner. The trace tells you which.

Where a free tier fits the loop

The loop needs two things that are annoying to supply locally: a clean machine and a model you can call repeatedly.

Operator-supplied availability: the MonkeyCode open-source project offers free model access and a free server option.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Two honest uses:

• The server as a disposable replay box. Run the harness on a host you can delete. Traces from a clean box are comparable; traces from your laptop are not.
• Free model access for the hypothesis step. Batch the failing span window and ask for candidate causes. Label the output as hypotheses, then verify against the hashes.

Do not treat free-tier output as evidence. It generates candidates. The trace decides.

Check the project's current terms yourself. Quotas, models, and lifetimes change, and I am not restating numbers I cannot verify today.

Limitations and who should skip this

• Small scripts. Under ~50 spans, a print statement beats a contract.
• No replay path. If you cannot reproduce a run, fix reproducibility first. This loop assumes replay.
• Sensitive payloads. Hashes reduce exposure, they do not remove it. Hash the args, never log raw tokens or credentials.
• Free tiers and secrets. Never put production credentials on a shared or disposable host.
• Already-complete traces. If coverage is fine and the loop still stalls, your problem is environment drift, not trace shape.

Try it on one real failure

Pick the last agent failure you debugged. Walk the six-row table in order.

Count how many rows you skipped last time. That number is your loop.

The contract check and the gap scanner are about 60 lines total. Run them against one recorded run this week and see what row 1 catches.