Validate the Trace Before You Trust the Trial: A Conformance Linter for Agent Runs

The trial that looks like a regression

You run one task three times on the same cheap model. Two runs agree. The third returns a different plan.

You blame the model. You call it flaky. You reach for a bigger, slower one.

Then you open the raw trace. One span has no parent. Another ends before it starts. A tool output was cut at two kilobytes, and nothing recorded the cut.

Nothing wobbled inside the model. The record was broken. You measured a parser bug and called it variance.

Cheaper models make this worse, not better. Free access and cheap compute turn three trials into thirty. Thirty dirty traces give you thirty confident wrong answers.

Model quality keeps climbing. The plumbing around it does not. That gap is where most "the model is flaky" reports live.

Trace trust comes before trial count

A trace is evidence. Evidence needs a shape.

Most agent harnesses emit spans as JSONL. One line is one unit of work. Each unit carries an id, a parent, a name, and time bounds.

Those fields are trivial to write and easy to corrupt. Async workers drop parents. Clocks disagree across hosts. Truncation goes unflagged. A retry reuses an id.

Here is the rule that keeps every later conclusion honest:

Lint the trace before you interpret it.

A clean trace with a bad answer is a model problem. A dirty trace is an instrumentation problem. Those need different fixes. You cannot tell them apart by reading outputs alone.

Eight invariants a runnable trace must satisfy

1. It parses. Every line is valid JSON. One bad line invalidates the whole run.
2. Fields are complete. span_id, run_id, parent_id, name, start_ns, end_ns.
3. Ids are unique. No span_id repeats inside one run.
4. Time is ordered. end_ns >= start_ns on every span.
5. Parents resolve. Every non-root parent_id exists in the same run.
6. Children are contained. A child starts after its parent and ends before it.
7. Exactly one root. Unless you model fan-out with an explicit join span.
8. Tools report bytes. Every tool span records stdout_bytes and a truncated flag.

Invariants one through six are structural. Seven and eight are the ones people skip, and they cause the worst conclusions.

The artifact: a small linter

The script below uses only the Python standard library. It reads one run and prints findings. It is not tested against any vendor schema, so change REQUIRED to match your exporter first.

#!/usr/bin/env python3
'''trace_lint.py - conformance checks for one agent run in JSONL.

Not tested against vendor schemas. Adapt REQUIRED to your exporter.
Usage: python3 trace_lint.py path/to/run.jsonl
Exit:  0 clean, 1 findings, 2 unreadable input.
'''
import json
import sys
from collections import defaultdict

REQUIRED = ('span_id', 'run_id', 'parent_id', 'name', 'start_ns', 'end_ns')
TOOL_KINDS = {'tool', 'tool_call', 'function'}


def load(path):
    rows = []
    with open(path, 'r', encoding='utf-8') as fh:
        for lineno, raw in enumerate(fh, 1):
            raw = raw.strip()
            if not raw:
                continue
            try:
                row = json.loads(raw)
            except json.JSONDecodeError as exc:
                row = {'_parse_error': str(exc)}
            rows.append((lineno, row))
    return rows


def lint(rows):
    findings = []
    by_run = defaultdict(list)

    for lineno, row in rows:
        if '_parse_error' in row:
            findings.append(('E_PARSE', lineno, row['_parse_error']))
            continue
        missing = [k for k in REQUIRED if k not in row]
        if missing:
            findings.append(('E_MISSING', lineno, 'missing ' + ','.join(missing)))
            continue
        by_run[row['run_id']].append((lineno, row))

    for run_id, spans in by_run.items():
        index = {row['span_id']: (lineno, row) for lineno, row in spans}
        seen = {}
        roots = 0

        for lineno, row in spans:
            sid = row['span_id']
            if sid in seen:
                findings.append(('E_DUP_ID', lineno, sid + ' first seen line ' + str(seen[sid])))
            seen[sid] = lineno

            if row['end_ns'] < row['start_ns']:
                findings.append(('E_NEG_DUR', lineno, sid + ' end before start'))

            pid = row['parent_id']
            if pid in (None, '', 'root'):
                roots += 1
            elif pid not in index:
                findings.append(('E_ORPHAN', lineno, sid + ' parent ' + str(pid) + ' not in run'))
            else:
                parent = index[pid][1]
                if row['start_ns'] < parent['start_ns']:
                    findings.append(('E_PARENT_TIME', lineno, sid + ' starts before parent'))
                if row['end_ns'] > parent['end_ns']:
                    findings.append(('E_CHILD_OUTSIDE', lineno, sid + ' ends after parent'))

            if row.get('kind') in TOOL_KINDS:
                if 'stdout_bytes' not in row:
                    findings.append(('E_TOOL_BYTES', lineno, sid + ' no stdout_bytes'))
                if 'truncated' not in row:
                    findings.append(('E_TOOL_TRUNC', lineno, sid + ' no truncated flag'))

        if roots != 1:
            findings.append(('E_ROOT_COUNT', 0, 'run ' + run_id + ' has ' + str(roots) + ' roots'))

    return findings


def main(argv):
    if len(argv) != 2:
        print('usage: trace_lint.py <run.jsonl>', file=sys.stderr)
        return 2
    try:
        rows = load(argv[1])
    except OSError as exc:
        print('cannot read ' + argv[1] + ': ' + str(exc), file=sys.stderr)
        return 2

    findings = lint(rows)
    for code, lineno, detail in findings:
        print(code + ' line=' + str(lineno) + ' ' + detail)
    print(str(len(rows)) + ' lines scanned, ' + str(len(findings)) + ' findings')
    return 1 if findings else 0


if __name__ == '__main__':
    raise SystemExit(main(sys.argv))

A fixture that must fail

Save these three lines as fixtures/run-003.bad.jsonl. This trace is deliberately broken in four ways.

{"span_id": "r1", "run_id": "run-003", "parent_id": null, "name": "agent", "kind": "root", "start_ns": 1000, "end_ns": 9000}
{"span_id": "s2", "run_id": "run-003", "parent_id": "r1", "name": "plan", "kind": "llm", "start_ns": 1100, "end_ns": 4000}
{"span_id": "s3", "run_id": "run-003", "parent_id": "gone", "name": "read_file", "kind": "tool", "start_ns": 5000, "end_ns": 4800}

Run it:

python3 trace_lint.py fixtures/run-003.bad.jsonl; echo "exit=$?"

Expected output:

E_ORPHAN line=3 s3 parent gone not in run
E_NEG_DUR line=3 s3 end before start
E_TOOL_BYTES line=3 s3 no stdout_bytes
E_TOOL_TRUNC line=3 s3 no truncated flag
3 lines scanned, 4 findings
exit=1

Keep this fixture in CI. A linter that stops detecting known-bad input is worse than no linter at all.

Lint every trace in the comparison set

Run the linter over the whole directory before you compare any outputs.

dirty=0
for f in traces/*.jsonl; do
  if ! python3 trace_lint.py "$f" > /dev/null; then
    echo "DIRTY $f"
    dirty=$((dirty + 1))
  fi
done
echo "$dirty dirty traces"

If any trace is dirty, the comparison is void. Fix the emitter, regenerate the traces, then compare. Score outputs only from clean runs.

On a free server, this loop is a five-minute cron job. Point it at a traces directory and append failures to a file you actually read.

17 3 * * * cd /srv/agent-lab && bash lint_all.sh >> lint.log 2>&1

A failure taxonomy you can act on

Finding	What it usually means	Where to look first
E_PARSE	Interleaved writes from two processes	The log writer, not the agent
E_MISSING	Exporter schema drifted	Emitter version and defaults
E_DUP_ID	Id from a counter that resets per worker	Id source, not the model
E_ORPHAN	Context lost at an async boundary	Worker handoff and trace context
E_NEG_DUR	Wall clock mixed with monotonic clock	Clock source per host
E_CHILD_OUTSIDE	Parent closed on the first child return	Parent span lifecycle
E_ROOT_COUNT	Fan-out with no join span	Orchestrator join logic
E_TOOL_TRUNC	Capture layer dropped the flag	Tool output capture

Two rules make these findings trustworthy. Use time.monotonic_ns() on each host. Store host_id and boot_id next to every span. Wall clock is for humans. Monotonic time is for durations.

Where free models and a free server fit

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode offers free model access and a free server option. The account operator supplied those two availability claims. Quotas, model lists, hardware, and limits change over time, so read the project's own page before you plan around them. The project describes itself as open source; check the repository and the license yourself.

What that changes here is quantity, and quantity is exactly what this workflow needs.

• You need many trials. Model variance separates from bug variance only across a corpus.
• You need a cheap host. A linter plus a nightly replay job wants a small always-on box.
• You need room to iterate on the emitter. Instrumentation bugs burn trials before they burn tokens.

A practical ladder looks like this:

1. Run five trials on the free tier.
2. Lint every trace. Fix the emitter until all five are clean.
3. Re-run the same task five more times on the same free models.
4. Only then compare against a paid model, with clean traces on both sides.

If the free tier survives step four, keep it. Cheap models are not the weak link once your traces are honest. If you want to try that loop without a card, the free tier and the free server option are the cheapest place to start.

Limitations

• The linter checks structure, not meaning. A clean trace can still hold a wrong answer.
• It assumes one run per file and one clock domain per host. Add host_id and boot_id before trusting cross-host durations.
• Missing spans are invisible. If the emitter never wrote a span, no check will find it.
• Aggressive sampling breaks containment checks. Sample deliberately, or lint the unsampled stream.
• The code above is untested against your schema. Treat it as a starting point, not a product.

Who should not use this

• Single-shot scripts with no tool calls. There is nothing to correlate.
• Teams without a frozen span schema. Freeze the schema first, then lint.
• Anyone who wants statistical significance from three runs. A linter is not a sample size.
• Teams that cannot change their emitter. The linter will only report problems you cannot fix.

The five-minute version

Freeze the span schema. Add the linter. Keep a known-bad fixture in CI.

Then run your comparison. If the traces are clean and the answers still diverge, you have a real model finding. That finding is worth acting on. Everything before it was noise.