DEV Community

Riley Wang
Riley Wang

Posted on

Lint Agent Traces Before You Debug the Model

#ai #debugging #observability #python

A Monday CI job went green after the agent ran. The pull request looked small and well scoped. A later job failed on a path the agent touched.

You opened the JSONL trace to reconstruct the tool calls. Span four named a formatter and nothing else. There was no argument digest and no exit code.

You started rewriting the system prompt that morning. The model was not the missing piece here. The tracer had emitted a success-shaped hole in the log.

Incomplete traces impersonate engineering

Hot threads this week treat vibe coding as a taste problem. The sharper failure is missing evidence. A green agent status is not a measurement.

A usable trace is a closed record. Every span kind needs a required field set. If a field is absent, the run is not debuggable.

Stop reading the model output first. Lint the file. Fix the tracer. Only then inspect behavior.

A closed attribute set

Keep four span kinds. Do not invent extra kinds on day one. Reject unknown kinds in CI.

span_kind required fields
run.start trace_id, span_id, ts_utc, task_id, repo_head
model.completion trace_id, span_id, parent_id, prompt_digest, response_digest, schema_digest, duration_ms
tool.call trace_id, span_id, parent_id, tool_name, schema_digest, args_digest, args_json_valid, exit_code, duration_ms, stdout_digest, stderr_digest, result_kind
run.end trace_id, span_id, parent_id, stop_class, ok

schema_digest is the hash of the tool schema the model saw. args_json_valid is a boolean from local JSON Schema checks. result_kind must be ok, exception, timeout, or rejected.

unknown is not allowed. Missing keys are not allowed. Nulls are not allowed for required fields.

Artifact: lint JSONL before anyone reads it

Label: this is a local example, not a production study. Save it as lint_agent_trace.py. Run it on one file.

#!/usr/bin/env python3
"""Fail a trace file when required span fields are missing."""
from __future__ import annotations

import hashlib
import json
import sys
from typing import Any

REQUIRED = {
    "run.start": [
        "trace_id", "span_id", "ts_utc", "task_id", "repo_head"
    ],
    "model.completion": [
        "trace_id", "span_id", "parent_id", "prompt_digest",
        "response_digest", "schema_digest", "duration_ms",
    ],
    "tool.call": [
        "trace_id", "span_id", "parent_id", "tool_name",
        "schema_digest", "args_digest", "args_json_valid",
        "exit_code", "duration_ms", "stdout_digest",
        "stderr_digest", "result_kind",
    ],
    "run.end": ["trace_id", "span_id", "parent_id", "stop_class", "ok"],
}

ALLOWED_RESULT = {"ok", "exception", "timeout", "rejected"}
ALLOWED_STOP = {"completed", "budget", "error", "cancelled"}


def digest(value: Any) -> str:
    blob = json.dumps(value, sort_keys=True, separators=(",", ":"))
    return hashlib.sha256(blob.encode()).hexdigest()[:16]


def lint_line(n: int, row: dict[str, Any]) -> list[str]:
    errors: list[str] = []
    kind = row.get("span_kind")
    if kind not in REQUIRED:
        return [f"L{n}: unknown span_kind {kind!r}"]
    for key in REQUIRED[kind]:
        if key not in row or row[key] in (None, ""):
            errors.append(f"L{n}: {kind} missing {key}")
    if kind == "tool.call":
        if row.get("args_json_valid") not in (True, False):
            errors.append(f"L{n}: args_json_valid must be bool")
        if row.get("result_kind") not in ALLOWED_RESULT:
            errors.append(f"L{n}: bad result_kind {row.get('result_kind')!r}")
        if not isinstance(row.get("exit_code"), int):
            errors.append(f"L{n}: exit_code must be int")
    if kind == "run.end" and row.get("stop_class") not in ALLOWED_STOP:
        errors.append(f"L{n}: bad stop_class {row.get('stop_class')!r}")
    return errors


def lint_file(path: str) -> int:
    errors: list[str] = []
    kinds: list[str] = []
    trace_ids: set[str] = set()
    with open(path, encoding="utf-8") as handle:
        for n, line in enumerate(handle, 1):
            line = line.strip()
            if not line:
                continue
            row = json.loads(line)
            kinds.append(row.get("span_kind"))
            if "trace_id" in row:
                trace_ids.add(row["trace_id"])
            errors.extend(lint_line(n, row))
    if kinds[:1] != ["run.start"] or kinds[-1:] != ["run.end"]:
        errors.append("file must start with run.start and end with run.end")
    if len(trace_ids) != 1:
        errors.append(f"expected one trace_id, found {sorted(trace_ids)}")
    for item in errors:
        print(item)
    print(f"spans={len(kinds)} errors={len(errors)}")
    return 1 if errors else 0


if __name__ == "__main__":
    sys.exit(lint_file(sys.argv[1]))
Enter fullscreen mode Exit fullscreen mode

A failing fixture looks like this. Note the empty tool span.

{"span_kind":"run.start","trace_id":"t1","span_id":"s0","ts_utc":"2026-09-18T12:00:00Z","task_id":"fix-null","repo_head":"abc1234"}
{"span_kind":"tool.call","trace_id":"t1","span_id":"s1","parent_id":"s0","tool_name":"format"}
{"span_kind":"run.end","trace_id":"t1","span_id":"s2","parent_id":"s0","stop_class":"completed","ok":true}
Enter fullscreen mode Exit fullscreen mode

Run the linter. It should exit non-zero.

python3 lint_agent_trace.py bad.jsonl
# L2: tool.call missing schema_digest
# L2: tool.call missing args_digest
# L2: tool.call missing args_json_valid
# L2: tool.call missing exit_code
# L2: tool.call missing duration_ms
# L2: tool.call missing stdout_digest
# L2: tool.call missing stderr_digest
# L2: tool.call missing result_kind
# spans=3 errors=8
Enter fullscreen mode Exit fullscreen mode

A passing tool span carries digests, not raw dumps. Store payloads elsewhere under those keys.

{"span_kind":"tool.call","trace_id":"t1","span_id":"s1","parent_id":"s0","tool_name":"format","schema_digest":"9c1e0b4a2d77f1c0","args_digest":"e3b0c44298fc1c14","args_json_valid":true,"exit_code":0,"duration_ms":41,"stdout_digest":"2c26b46b68ffc68f","stderr_digest":"e3b0c44298fc1c14","result_kind":"ok"}
Enter fullscreen mode Exit fullscreen mode

Wire it as a gate. Do not review a trace that fails this command.

python3 lint_agent_trace.py run.jsonl && echo TRACE_OK
Enter fullscreen mode Exit fullscreen mode

A debug loop that refuses partial evidence

Use the same loop on every incident. Do not skip step one because the reply looks fluent.

  1. Export one JSONL file per agent run.
  2. Lint it. If lint fails, patch the tracer.
  3. Confirm schema_digest is stable for the same tool version.
  4. Confirm args_json_valid is true before you blame the model.
  5. Compare prompt_digest across reproductions of the same task.
  6. Only then read payloads behind the digests.

Step three catches schema drift. The model may call format after the argument shape changed. The name matches. The digest does not.

Step four catches parse theater. The wrapper swallowed a JSON error. The span still said success. The linter now forces a boolean.

Step five tells you whether the prompt moved. If digests differ, you are not comparing the same request.

Cheap reproductions need a complete tracer first

You need volume to learn which fields drop. Local laptops hide that. A shared box makes the gap obvious.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode's free model access and free server option can host the same linted runner without a private GPU queue. That is the only product claim used here.

Keep the server as a recorder. Do not treat it as a judge. Copy lint_agent_trace.py onto the box. Drop incomplete files. Diff the survivors.

mkdir -p traces/ok traces/drop
for f in traces/inbox/*.jsonl; do
  if python3 lint_agent_trace.py "$f"; then
    mv "$f" traces/ok/
  else
    mv "$f" traces/drop/
  fi
done
python3 - <<'PY'
import json, pathlib
from collections import Counter
c = Counter()
for path in pathlib.Path("traces/drop").glob("*.jsonl"):
    for line in path.read_text().splitlines():
        if not line.strip():
            continue
        row = json.loads(line)
        if row.get("span_kind") == "tool.call" and "schema_digest" not in row:
            c[row.get("tool_name","?")] += 1
print(c.most_common(10))
PY
Enter fullscreen mode Exit fullscreen mode

The histogram names wrappers that emit hollow spans. That list is your sprint board. It is not a model ranking.

What the linter cannot claim

A lint-clean trace can still be wrong. Complete fields are not correct fields. Digests do not prove a patch fixed the bug.

This workflow also fails open if you store secrets in payloads. Hash first. Redact first. Then archive. The linter does not replace access control.

Do not use this approach for single-shot chat with no tools. The required set assumes tool spans exist. Do not use it as a model leaderboard. It measures record quality, not answer quality.

Skip it if your org cannot export traces off a laptop. Skip it if you still log with unstructured print statements only. Adopt a schema first.

Practical bounds

Keep duration_ms as an integer millisecond count. Do not mix clocks in one field. Keep one trace_id per file. Split retries into new files if wrappers restart.

Store raw stdout off-trace. The lint file should stay small. Large traces hide the missing key you needed.

Re-run the linter in CI on fixture files. A tracer change that drops args_json_valid should fail the build. That is the whole point.

Close the hole, then argue about the model

Prompt edits are cheap and noisy. Tracer edits are boring and durable. If span four has no exit code, you cannot replay it.

Lint the JSONL. Count missing keys. Repair the emitter. After the file is complete, the model discussion has evidence.

If you want a shared box for those lint-clean reproductions, MonkeyCode's free model access and free server option are enough to start collecting files.

Top comments (0)