Tool Output Looks Complete. Record the Injected Slice.

The on-call thread opened at 02:11 UTC.
The agent had already marked the run green.
It claimed production.yaml was read and valid.

Slack still received the old staging port.
The file on disk contained the right value.
The trace even stored the entire file body.

That mismatch is the bug this article names.
The tool span was complete and technically truthful.
The next model turn never received that body.

The lie inside a successful span

Most agent logs keep the tool call name.
They also keep arguments and a result string.
Some keep timestamps and a parent span id.

They rarely keep three different length fields.
Those three numbers are not interchangeable.
Mixing them hides a common production failure.

Here are the three lengths that actually matter:

• observed_bytes counts bytes the tool returned.
• injected_bytes counts bytes copied into context.
• injected_tokens counts tokens the API accepted.

A green span can still drop most of a file.
Your replay then debugs the wrong evidence set.
You will rewrite prompts the model never saw.

Clipping happens after the tool returns

This is not a hang and not a retry bug.
The tool finished. The filesystem read succeeded.
The prompt builder then applied a size cap.

Duration fields will look healthy and boring.
Retry counters will stay at one attempt.
Span trees will show a normal child read.

The loss sits on a later boundary.
That boundary is prompt assembly, not I/O.
Traces that stop at tool return will miss it.

Tool success answers one question only.
It says the executor did not throw.
It does not say the model saw the tail.

A reconstructed incident

Treat this section as a reconstructed pattern.
It is not a named customer outage report.

The agent called read_file on production.yaml.
The tool returned 96412 bytes of YAML text.
The trace writer stored the full result blob.

The prompt builder applied an 8192 token cap.
Only the leading keys reached model context.
The port field lived near the file end.

The model answered from that truncated prefix.
It reused a default port from the system prompt.
The trace still showed a perfect file read.

Diffing two full tool bodies did not help.
Both traces contained identical complete files.
Only the injected slice had changed size.

A later run used a tighter builder policy.
Observed hashes stayed stable across both runs.
Injected hashes diverged, and the port flipped.

Artifact: clip-aware JSONL spans

Store clipping as first-class fields on spans.
Do not infer clipping from sampled prompt dumps.
Prompt dumps are often redacted or dropped.

Use one JSON object per line, as follows.

{
  "span_id": "tool-14",
  "parent_id": "turn-3",
  "name": "read_file",
  "status": "ok",
  "observed_bytes": 96412,
  "injected_bytes": 12040,
  "injected_tokens": 8192,
  "clip_reason": "prompt_token_cap",
  "sha256_observed": "ab12",
  "sha256_injected": "cd34",
  "injected_preview": "app:\\n  name: billing\\n",
  "prompt_builder_version": "cap-v3"
}

Keep clip_reason as a small closed enum.
Do not store free-text excuses in that field.

Closed values worth supporting today:

• none means the full result was injected.
• prompt_token_cap means the builder cut tokens.
• tool_result_char_limit means a char cap fired.
• provider_context_window means the API rejected tail.
• redaction means bytes changed for safety.
• binary_omitted means a non-text payload was dropped.

Flag a span when injected bytes fall below observed.
Flag again when hashes differ at equal lengths.
Redaction can change bytes without a size cap.

Checker script

The script below is a local proposal only.
It reads JSONL from stdin and prints a table.
It does not call any network service.

#!/usr/bin/env python3
"""clip_check.py — flag tool spans with silent truncation."""
import json
import sys

PRIORITY = {
    "silent_clip": 0,
    "incomplete": 1,
    "hash_mismatch": 2,
    "clipped": 3,
    "ok": 4,
}


def load_spans(stream):
    for line_no, raw in enumerate(stream, 1):
        line = raw.strip()
        if not line:
            continue
        rec = json.loads(line)
        rec["_line"] = line_no
        yield rec


def clip_ratio(obs, inj):
    if obs <= 0:
        return 0.0
    return 1.0 - (inj / obs)


def classify(rec):
    missing = [
        key
        for key in ("observed_bytes", "injected_bytes", "injected_tokens")
        if rec.get(key) is None
    ]
    if missing:
        return "incomplete", missing

    obs = int(rec.get("observed_bytes") or 0)
    inj = int(rec.get("injected_bytes") or 0)
    reason = rec.get("clip_reason") or "none"

    if inj < obs and reason == "none":
        return "silent_clip", [f"ratio={clip_ratio(obs, inj):.2f}"]
    if inj < obs:
        return "clipped", [reason, f"ratio={clip_ratio(obs, inj):.2f}"]

    obs_hash = rec.get("sha256_observed")
    inj_hash = rec.get("sha256_injected")
    if obs_hash and inj_hash and obs_hash != inj_hash and reason == "none":
        return "hash_mismatch", ["bytes_match_but_hash_differs"]
    return "ok", []


def main():
    rows = []
    for rec in load_spans(sys.stdin):
        if not rec.get("name"):
            continue
        status, detail = classify(rec)
        rows.append((status, rec, detail))

    rows.sort(key=lambda item: PRIORITY.get(item[0], 9))
    print(f"{'status':<16} {'span':<12} {'tool':<16} {'obs':>8} {'inj':>8} detail")
    for status, rec, detail in rows:
        print(
            f"{status:<16} {str(rec.get('span_id', '-')):<12} "
            f"{str(rec.get('name', '-')):<16} "
            f"{int(rec.get('observed_bytes') or 0):>8} "
            f"{int(rec.get('injected_bytes') or 0):>8} "
            f"{','.join(detail)}"
        )

    silent = sum(1 for status, _, _ in rows if status == "silent_clip")
    incomplete = sum(1 for status, _, _ in rows if status == "incomplete")
    print(f"\n# silent_clip={silent} incomplete={incomplete} total={len(rows)}")
    if silent or incomplete:
        sys.exit(1)


if __name__ == "__main__":
    main()

Run it against one exported run:

python3 clip_check.py < traces.jsonl
echo $?

Exit code 1 means the trace is not trustworthy.
Exit code 0 means every tool span declared clipping.
That zero does not prove the answer was correct.

Emit fields at the tool boundary

Wrap the executor so spans cannot omit lengths.
Hash both strings before the prompt is built.
Write the JSONL line before the next model call.

import hashlib
import json
from pathlib import Path


def sha256_text(text: str) -> str:
    return hashlib.sha256(text.encode("utf-8")).hexdigest()[:16]


def emit_tool_span(path, span_id, parent_id, name, observed, injected, reason, builder_version, token_count):
    rec = {
        "span_id": span_id,
        "parent_id": parent_id,
        "name": name,
        "status": "ok",
        "observed_bytes": len(observed.encode("utf-8")),
        "injected_bytes": len(injected.encode("utf-8")),
        "injected_tokens": token_count,
        "clip_reason": reason,
        "sha256_observed": sha256_text(observed),
        "sha256_injected": sha256_text(injected),
        "injected_preview": injected[:128],
        "prompt_builder_version": builder_version,
    }
    with Path(path).open("a", encoding="utf-8") as handle:
        handle.write(json.dumps(rec, ensure_ascii=False) + "\n")
    return rec

Call emit_tool_span once per finished tool.
Pass the exact string the prompt builder used.
Do not hash the file on disk after the fact.

Disk can change between tool return and replay.
The injected string is the only fair evidence.
That is what the model could have used.

If your builder truncates after encoding tokens, record that.
Byte cuts and token cuts are different operations.
The enum must say which policy actually fired.

Four-step debug loop

Use the same loop on every flaky agent run.
Do not start with prompt rewriting first.

1. Export one JSONL file per agent run.
2. Require clip fields on every tool span.
3. Fail the run if clip_check.py exits 1.
4. Replay only after injected hashes match.

Step four needs the injected slice, not the file.
Keep injected_preview as a short stable prefix.
A 128-byte prefix is enough to compare two runs.

Add these extra fields when you can:

• injected_preview stores 128 escaped leading bytes.
• model_context_limit_tokens stores the active window.
• prompt_builder_version stores the cap policy id.

prompt_builder_version catches silent policy changes.
A new cap can ship without any model change.
Your model regression may be a clip change.

Compare builder versions before you compare models.
A cap change rewrites every downstream token.
Model swap notes become noise after that rewrite.

Tests for the classifier

These tests are local and do not need GPUs.
They lock the silent-clip definition in place.
Run them before you trust a new exporter.

from clip_check import classify


def test_silent_clip():
    status, detail = classify({
        "observed_bytes": 1000,
        "injected_bytes": 120,
        "injected_tokens": 80,
        "clip_reason": "none",
    })
    assert status == "silent_clip"
    assert detail[0].startswith("ratio=")


def test_declared_clip_is_visible():
    status, _ = classify({
        "observed_bytes": 1000,
        "injected_bytes": 120,
        "injected_tokens": 80,
        "clip_reason": "prompt_token_cap",
    })
    assert status == "clipped"


def test_missing_lengths_fail():
    status, missing = classify({"name": "read_file"})
    assert status == "incomplete"
    assert "observed_bytes" in missing

A silent clip is the highest priority failure.
Missing fields are the next failure class.
Declared clips are visible and therefore usable.

Do not fail CI on every declared clip.
Large read_file results will clip by design.
Fail CI when the reason field is missing or none.

Decision table

Symptom	Trace looks like	First check	Do not do
Wrong field from a large file	status=ok, full body stored	injected_bytes versus observed_bytes	Rewrite the system prompt
Binary tool result looks empty	short ASCII replacement text	clip_reason=binary_omitted	Retry the same tool blindly
Intermittent wrong answers	identical observed tool bodies	builder version and cap fields	Swap models immediately
Replay missing expected PII	hashes differ, lengths stay close	clip_reason=redaction	Paste raw traces into chat
Checker exits incomplete	length fields are absent	fix the exporter first	Trust the pretty trace UI

Wrong-port bugs often match row one.
The file was complete in object storage.
The injected slice never included the key.

Where a quiet box fits

You can run this loop on a laptop.
A quiet remote box helps with overnight replays.
Long traces should not compete with your IDE.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.
MonkeyCode is an open-source project for coding agents.
It offers free model access and a free server option.

Those two facts matter for this workflow only.
You need a model to regenerate injected slices.
You need a server if local machines are busy.

Do not treat that option as a benchmark lab.
This article does not publish model names.
It also does not publish quotas or hardware specs.

Capture traces. Run clip_check.py. Then replay.
If the checker is red, skip model comparison.
You would be scoring two different prompts.

If you already store JSONL traces, start there.
The server is optional for the checker itself.

Limitations

This method does not score answer quality.
It only detects undeclared length or hash changes.
A fully injected file can still be the wrong file.

Token counts depend on the provider tokenizer.
Byte ratios are exact. Token ratios are estimates.
Do not alert on token estimates alone.

Streaming tools can update observed bytes twice.
Record final values after the tool span closes.
Partial spans should keep status as incomplete.

Multi-agent graphs need per-child clip fields.
A parent ok can hide a clipped child read.
The checker should walk every tool span record.

Pretty UIs often show the observed blob first.
Engineers then believe the model saw that blob.
Show the injected slice before the observed body.

This loop also ignores semantic irrelevance.
A full injection can still be the wrong excerpt.
You still need evals for that class of error.

Who should not use this

Skip this if your agent has no tools.
A chat wrapper has no result slice to clip.
Prompt-only apps need a different trace shape.

Skip this if traces contain raw secrets.
Hash and preview fields can still leak data.
Redact before JSONL ever leaves the host.

Skip this for hard real-time control loops.
The extra fields add exporter complexity and I/O.
Those systems need different runtime guarantees.

What to watch after you ship this

Once clipping is explicit, add one rate.
Count silent clips per one hundred tool spans.
Watch that rate when prompt builders change.

A second watch is useful after the first.
Track median injected bytes for read_file.
A sudden drop often ships with a new cap.

Keep the original failure in mind here.
The YAML file was on disk and correct.
The model never saw the port key at all.

Record the injected slice on every tool span.
Then green traces stop lying about completed reads.