The Timeline Has Holes. Record Span Sequence Numbers.

The on-call channel lit up after a staging deploy. The agent report claimed two full pytest runs. The trace view showed four tool spans only. Two pytest spans never reached the collector backend.

This is a reconstructed incident, not a dump. The failure mode is omission, not a wrong token.

Green summaries hide missing spans

Teams debug the prompt when the timeline is incomplete. A missing tool span looks like a skipped step. Operators then change temperature, tools, or system text.

The model may have called the tool correctly. The exporter dropped the span during a batch flush. You cannot tell without a per-run sequence number.

Dashboards also reward short, pretty traces. Sparse timelines look efficient during a review. They can hide work that never left the process.

Sequence is not a timestamp

Clock order was a different failure class. Sequence answers a narrower question than time. Did this run emit span k, yes or no.

Timestamps still matter for latency later. They do not prove the exporter kept every span. A hole can sit between two plausible timestamps.

The contract for one run

One agent run needs one strictly increasing integer. Start at one when the run record is created. Increment after each tool call, model step, and flush.

Store seq before the exporter touches the row. Never renumber after a retry or partial export. Export retries keep the original number on that span.

A real tool retry is a new span. It gets the next integer, not a reused one. Mixing those two retry kinds creates false duplicates.

Fields to persist

Keep the row small and boring:

• run_id: one id for the whole attempt
• seq: integer starting at one, no intended gaps
• span_id: unique id for this span row
• name: tool name or model step label
• export_batch_id: flush batch that held the row
• t_emit_ms: monotonic clock at emit time
• attempt: retry index for that tool name

Reject rows that lack run_id or seq. Incomplete telemetry is a failed run, not a maybe.

Stamp seq in a thin wrapper

Put the counter in process memory, not the model. The wrapper owns seq; the model never sees it. The example below is a local script, not production telemetry.

# span_seq.py — example wrapper, run locally
from __future__ import annotations

import json
import time
import uuid
from dataclasses import dataclass, field
from pathlib import Path
from typing import Any, Callable


@dataclass
class SeqTracer:
    path: Path
    run_id: str = field(default_factory=lambda: uuid.uuid4().hex[:12])
    _seq: int = 0
    _batch: int = 0

    def emit(self, name: str, attempt: int = 0, **fields: Any) -> dict:
        self._seq += 1
        row = {
            "run_id": self.run_id,
            "seq": self._seq,
            "span_id": uuid.uuid4().hex[:16],
            "name": name,
            "attempt": attempt,
            "export_batch_id": f"b{self._batch}",
            "t_emit_ms": time.monotonic_ns() // 1_000_000,
            **fields,
        }
        self.path.parent.mkdir(parents=True, exist_ok=True)
        with self.path.open("a", encoding="utf-8") as fh:
            fh.write(json.dumps(row, ensure_ascii=True) + "\n")
        return row

    def flush_batch(self) -> None:
        self._batch += 1


def wrap_tool(tracer: SeqTracer, name: str, fn: Callable[..., Any]) -> Callable[..., Any]:
    def wrapped(*args: Any, **kwargs: Any) -> Any:
        attempt = int(kwargs.pop("_attempt", 0))
        tracer.emit(name, attempt=attempt, phase="start")
        try:
            result = fn(*args, **kwargs)
        except Exception as exc:
            tracer.emit(name, attempt=attempt, phase="error", error=type(exc).__name__)
            raise
        tracer.emit(name, attempt=attempt, phase="end")
        return result

    return wrapped

Flush after a successful export ACK, not before write. A killed process then leaves a tail gap. That gap is data, not a model mystery.

Fail the debug session on holes

Do not open the prompt until this script exits zero. The checker reads JSONL and groups rows by run_id. It expects seq values equal to one through N.

Any hole, duplicate, or restart fails the run. Save the file and run it against one trace.

# check_span_seq.py — example completeness checker
from __future__ import annotations

import argparse
import json
import sys
from collections import defaultdict
from pathlib import Path


def load_rows(path: Path) -> list[dict]:
    rows = []
    with path.open(encoding="utf-8") as fh:
        for line_no, line in enumerate(fh, 1):
            line = line.strip()
            if not line:
                continue
            try:
                row = json.loads(line)
            except json.JSONDecodeError as exc:
                raise SystemExit(f"bad json at line {line_no}: {exc}") from exc
            if "run_id" not in row or "seq" not in row:
                raise SystemExit(f"missing run_id/seq at line {line_no}")
            rows.append(row)
    return rows


def report_for_run(run_id: str, rows: list[dict]) -> dict:
    seqs = [int(r["seq"]) for r in rows]
    names = {int(r["seq"]): r.get("name", "?") for r in rows}
    last = max(seqs) if seqs else 0
    present = set(seqs)
    expected = set(range(1, last + 1)) if last else set()
    missing = sorted(expected - present)
    dupes = sorted(s for s in present if seqs.count(s) > 1)
    completeness = (len(present) / last) if last else 0.0
    return {
        "run_id": run_id,
        "row_count": len(rows),
        "last_seq": last,
        "missing": missing,
        "duplicates": dupes,
        "first_gap_name_after": names.get((missing[0] - 1), None) if missing else None,
        "completeness": round(completeness, 4),
        "ok": not missing and not dupes and last == len(present),
    }


def main() -> int:
    p = argparse.ArgumentParser(description="Detect span seq gaps in JSONL")
    p.add_argument("jsonl", type=Path)
    p.add_argument("--run-id", default="")
    args = p.parse_args()
    grouped: dict[str, list[dict]] = defaultdict(list)
    for row in load_rows(args.jsonl):
        grouped[str(row["run_id"])].append(row)
    run_ids = [args.run_id] if args.run_id else sorted(grouped)
    failed = 0
    for run_id in run_ids:
        rep = report_for_run(run_id, grouped.get(run_id, []))
        print(json.dumps(rep, indent=2))
        failed += 0 if rep["ok"] else 1
    return 2 if failed else 0


if __name__ == "__main__":
    sys.exit(main())

python3 check_span_seq.py traces/staging-0214.jsonl --run-id staging-0214
echo $?

Exit zero means the sequence is dense. Exit two means you still lack a timeline. Treat that like a red test, not a hint.

Read the gap shape, not the model card

The exit code is the first signal, not the model. Completeness is present integers divided by last_seq. The shape of the hole picks the next probe.

Gap shape	Likely cause	Next probe
1,2,4,5	middle drop or flush race	export_batch_id on neighbors of 3
1,2,3 then stop	tail drop or kill	last flush, process exit code
1,1,2	duplicate export	ACK path versus in-memory buffer
5,6,7	missing head	late attach, wrong run_id
two dense streams	mixed runs	join keys, child trace ids

A middle hole after tool.pytest start is not a skip. It is often a dropped end span. The agent may have finished the tests off-screen.

A tail hole after model.generate often means SIGKILL. Disk full and batch limits look the same here. Inspect the collector log before the weights.

A four-step debug loop

Use the same order on every flaky agent:

1. Confirm one run_id and a dense seq range.
2. Only then compare tool names against the agent summary.
3. Diff payloads after the timeline is complete.
4. Change prompts last, never as the first move.

Skip step one and you debug fiction. The summary text is not an inventory. It is another span you might also have dropped.

Reproducible fixture

Label this fixture as an unexecuted example unless you save it. Three rows should fail with missing: [3].

{"run_id":"staging-0214","seq":1,"name":"tool.read","export_batch_id":"b0"}
{"run_id":"staging-0214","seq":2,"name":"tool.pytest","export_batch_id":"b0"}
{"run_id":"staging-0214","seq":4,"name":"model.generate","export_batch_id":"b1"}

Expected report fields for that file:

• last_seq: 4
• missing: [3]
• completeness: 0.75
• process exit: 2

If your checker prints ok: true here, stop. The tool is wrong, not the agent.

Where a throwaway box fits

You can run the wrapper beside a local collector. A laptop works until the disk fills with JSONL. Overnight reproductions need a box that stays up.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode offers free model access and a free server option. Use the free server to host the collector and wrapper. Use free model access only to reproduce the same traces.

Do not treat that access as a quota, SLA, or model list. If you need a disposable host, try the free server option. Keep the JSONL on that host until completeness stays at one.

Limitations

Sequence numbers do not prove the tool did useful work. They only prove the tracer emitted a row. Semantic bugs still need diffs and exit codes elsewhere.

In-process counters die with the process. A crash before emit looks like a skip. You still need an OS-level audit for that class.

Concurrent workers need a lock or a partition key. Two threads sharing one integer will duplicate seq. This article does not cover distributed id factories.

Sampling in a real collector can drop spans after emit. Your JSONL can be complete while the vendor UI is not. Compare both stores before you trust either.

Who should not use this

Skip this loop if OpenTelemetry already ACKs every span. Skip it if you lack a single run_id today. Skip it for vibe checks and chat demos with no tools.

Do not use it as a model leaderboard. Completeness is not quality. A dense trace can still call the wrong binary.

Platform teams with an existing trace pipeline should extend that pipeline. Do not add a second JSONL dialect without a join key. Two incomplete stores are worse than one.

Tomorrow morning

Pick one flaky staging run, not a new model. Add seq on the wrapper before the exporter. Fail the debug session when completeness is below one.

Only then inspect prompts, tools, and model output. The timeline had holes. Number every span until it does not.

Comments

[Raju Dandigam]

This is a useful distinction: timestamps order what arrived, while a sequence can prove that something never arrived. While working on agent-inspect, I’ve found completeness needs to be captured before any adapter or exporter can drop data. For concurrent branches, I’d be cautious about treating one global sequence as causality; a run sequence plus parent-local child order or emitter ID preserves the gap signal without implying that parallel spans happened serially. Have you tried that shape with fan-out/fan-in agent runs?