Tool Previews Lie. Store stdout_bytes and a Truncation Flag.

A coding agent closed the flaky test ticket.
The JSONL span showed pytest and a short preview.
The next human run failed in the same module.

The wrapper had capped stdout at 8192 bytes.
The assertion text started after that cap.
The model never read the failing lines.

This is not a prompt problem first.
It is missing I/O metadata on the tool span.

The failure mode

Most agent traces store a short text preview.
They also store a loose success flag.
That pair cannot describe a clipped pipe.

Three wrappers hide the real tool result:

1. Head-cap keeps only the first N bytes.
2. Tail-cap keeps only the last N bytes.
3. Pipe-stall drops the stream and stores empty text.

The model then plans from a partial document.
Later debug sessions trust that partial document.
Teams retune temperature while the bytes were absent.

What typical spans already contain

A typical tool span already has a name.
It often includes arguments and a duration.
Some collectors also keep an exit code.

Those fields still miss clipped pipes.
Exit code 0 can sit beside truncated stdout.
Exit code 1 can sit beside truncated stderr.
Both streams can be empty after a full pipe.

Fields to add on every tool span

Record these keys on each tool span:

• stdout_bytes: length before any wrapper cap
• stderr_bytes: length before any wrapper cap
• stdout_cap: wrapper limit in bytes, or null
• stderr_cap: wrapper limit in bytes, or null
• stdout_truncated: boolean after the cap check
• stderr_truncated: boolean after the cap check
• stdout_sha256: full-stream hash when unclipped
• stderr_sha256: full-stream hash when unclipped
• stdout_head_sha256 / stdout_tail_sha256: clipped case
• encoding: utf-8 or binary

Keep the stored preview short on purpose.
Store hashes for bytes you refused to keep.
Never treat the preview as the full stream.

Proposed JSONL schema

The next record is a proposed schema only.
It is not sampled from a production fleet.
Use it as a contract for local collectors.

{
  "span_type": "tool",
  "tool_name": "pytest",
  "cwd": "/work/repo",
  "preview_stdout": "===== test session starts =====\n",
  "preview_stderr": "",
  "stdout_bytes": 24118,
  "stderr_bytes": 902,
  "stdout_cap": 8192,
  "stderr_cap": 8192,
  "stdout_truncated": true,
  "stderr_truncated": false,
  "stdout_sha256": null,
  "stderr_sha256": "b3a1...",
  "stdout_head_sha256": "9c21...",
  "stdout_tail_sha256": "44de...",
  "encoding": "utf-8",
  "ok_preview": true
}

ok_preview is what the agent saw.
It is not a claim about the full process.
Lint that distinction before you debug policy.

A local subprocess wrapper

The wrapper below is a labeled local example.
Run it against one command on your machine.
It is not a production sandbox or jail.

#!/usr/bin/env python3
"""Record truncation metadata around one tool command."""
from __future__ import annotations

import hashlib
import json
import subprocess
import sys
from pathlib import Path

CAP = 8192  # example cap; set from your collector
PREVIEW = 512


def sha256_bytes(data: bytes) -> str:
    return hashlib.sha256(data).hexdigest()


def clip_meta(data: bytes, cap: int) -> dict:
    truncated = len(data) > cap
    kept = data[:cap] if truncated else data
    meta = {
        "bytes": len(data),
        "cap": cap,
        "truncated": truncated,
        "sha256": None if truncated else sha256_bytes(data),
        "head_sha256": sha256_bytes(data[: min(len(data), 256)]),
        "tail_sha256": sha256_bytes(data[-min(len(data), 256) :]) if data else None,
        "preview": kept[:PREVIEW].decode("utf-8", "replace"),
        "encoding": "utf-8",
    }
    return meta


def run_tool(argv: list[str], cwd: str) -> dict:
    proc = subprocess.run(
        argv,
        cwd=cwd,
        stdout=subprocess.PIPE,
        stderr=subprocess.PIPE,
        check=False,
    )
    out = clip_meta(proc.stdout, CAP)
    err = clip_meta(proc.stderr, CAP)
    return {
        "span_type": "tool",
        "argv": argv,
        "cwd": str(Path(cwd).resolve()),
        "returncode": proc.returncode,
        "stdout_bytes": out["bytes"],
        "stderr_bytes": err["bytes"],
        "stdout_cap": out["cap"],
        "stderr_cap": err["cap"],
        "stdout_truncated": out["truncated"],
        "stderr_truncated": err["truncated"],
        "stdout_sha256": out["sha256"],
        "stderr_sha256": err["sha256"],
        "stdout_head_sha256": out["head_sha256"],
        "stdout_tail_sha256": out["tail_sha256"],
        "preview_stdout": out["preview"],
        "preview_stderr": err["preview"],
        "encoding": "utf-8",
        "ok_preview": proc.returncode == 0 and not out["truncated"] and not err["truncated"],
    }


if __name__ == "__main__":
    if len(sys.argv) < 3:
        print("usage: wrap_tool.py CWD CMD [ARGS...]", file=sys.stderr)
        sys.exit(2)
    record = run_tool(sys.argv[2:], sys.argv[1])
    print(json.dumps(record, ensure_ascii=False))

Example command:

python3 wrap_tool.py "$PWD" pytest -q tests/test_billing.py

Redirect that JSON line into run.jsonl.
Keep one file per agent run, not per week.
Do not append unrelated jobs into the same file.

A truncation linter

Feed traces into this checker next.
It fails when flags disagree with byte counts.
It also fails when a preview claims success after a clip.

#!/usr/bin/env python3
"""Lint tool spans for truncation contract breaks."""
from __future__ import annotations

import json
import sys
from pathlib import Path

REQUIRED = (
    "stdout_bytes",
    "stderr_bytes",
    "stdout_cap",
    "stderr_cap",
    "stdout_truncated",
    "stderr_truncated",
)


def problems(span: dict, line_no: int) -> list[str]:
    issues: list[str] = []
    if span.get("span_type") != "tool":
        return issues
    missing = [key for key in REQUIRED if key not in span]
    if missing:
        issues.append(f"L{line_no}: missing {missing}")
        return issues
    for stream in ("stdout", "stderr"):
        n = span[f"{stream}_bytes"]
        cap = span[f"{stream}_cap"]
        flag = span[f"{stream}_truncated"]
        if cap is not None and n > cap and flag is not True:
            issues.append(f"L{line_no}: {stream} clipped without flag")
        if cap is not None and n <= cap and flag is True:
            issues.append(f"L{line_no}: {stream} flag set under cap")
        if flag and span.get(f"{stream}_sha256"):
            issues.append(f"L{line_no}: full {stream} hash on clipped stream")
    if span.get("ok_preview") is True and (
        span["stdout_truncated"] or span["stderr_truncated"]
    ):
        issues.append(f"L{line_no}: ok_preview true while truncated")
    return issues


def lint_path(path: Path) -> int:
    found = 0
    with path.open() as handle:
        for line_no, raw in enumerate(handle, start=1):
            raw = raw.strip()
            if not raw:
                continue
            span = json.loads(raw)
            for item in problems(span, line_no):
                print(item)
                found += 1
    return found


if __name__ == "__main__":
    if len(sys.argv) != 2:
        print("usage: lint_truncation.py run.jsonl", file=sys.stderr)
        sys.exit(2)
    count = lint_path(Path(sys.argv[1]))
    print(f"issues={count}")
    sys.exit(1 if count else 0)

Run it:

python3 lint_truncation.py run.jsonl

A non-zero exit means stop model debugging.
Re-run the clipped tool without a cap first.
Diff that full stderr against the stored preview.

Decision table

Observation	First action	Not the first action
stderr_truncated=true, returncode 0	Re-run tool with cap removed	Raise temperature
stdout_truncated=true on test output	Fetch tail hash, re-run pytest	Swap the coding model
both byte counts zero, returncode 0	Check pipe stall and ulimit	Add more tool descriptions
preview shows pass, linter fails	Treat span as incomplete	Close the incident
binary encoding, hashes only	Skip UTF-8 preview diffs	Parse preview as logs

Use the table during incident review.
Do not turn it into a model rubric.
The artifact is the span contract, not a score.

A debug loop you can reuse

Use this loop on the next opaque agent miss:

1. Export JSONL for that single run only.
2. Run lint_truncation.py run.jsonl.
3. Stop if any span sets a truncation flag.
4. Re-run that one argv with CAP removed.
5. Diff new stderr against preview_stderr.
6. Inspect prompts only after the pipe is honest.

Do not start with a new system prompt.
Do not start with a larger model.
Fix the stream, then re-evaluate the policy.

Where a scratch model and server fit

Batch linting JSONL does not need a GPU.
A small VM can parse traces and re-run tools.
That is the only capacity this method needs.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode provides free model access and a free server option.
Those two are enough to host the linter and cheap re-runs.
The checker does not depend on that product to work.
Remove the product and the span contract still stands.

Limitations

• Hashes cannot rebuild bytes you already dropped.
• Stdout may contain tokens, paths, and secrets.
• Nested shells can apply a second silent cap.
• Binary compilers break naive UTF-8 previews.
• duration_ms still says nothing about clip point.
• Preview diffs fail when logs contain timestamps.

Store redaction rules beside the collector.
Hash first if policy forbids raw command output.
Do not ship full pytest logs to a shared trace bucket.

Who should skip this

Skip this workflow if agents never call tools.
Skip it if your collector already stores full blobs.
Skip it if policy forbids retaining command output.
Skip it for interactive TTY tools with cursor codes.

Close

The model cannot repair bytes it never received.
Record truncation flags before you blame the policy.
Run lint_truncation.py on one failed JSONL file this week.