DEV Community: Riley Wang

The Diff Wasn't the Model: Fingerprint the Environment Before You Replay a Trace

Riley Wang — Mon, 26 Oct 2026 11:31:02 +0000

A replay failed at tool call seven. The run diff was clear and useless. The model had not changed. Our search_docs schema had.

We replayed a two-week-old trace on a Tuesday. Step seven sent {"q": "retry policy"}. The new schema wanted {"query": "retry policy"}. The tool rejected the call, the agent retried twice, then gave up.

Cost of that false alarm: half a day and two engineers. Everyone blamed a model upgrade. The real fix was a fingerprint we never recorded.

Why replay diffs lie

A tool-call diff shows what changed. It cannot show why. Four different causes look identical in a trace:

Model behavior changed.
A tool schema or tool implementation changed.
A prompt template or dependency lock changed.
Non-determinism: timeouts, retries, ordering, clock.

Only cause 1 is a model problem. Causes 2 and 3 live in your repo. Cause 4 lives in your harness. You cannot separate them without environment data.

The fix is cheap. Write one manifest line at the top of each trace. Compare that line before you compare anything else.

Fingerprint what the agent actually sees

Do not fingerprint your intent. Fingerprint the inputs the agent receives.

Tool schemas: canonical JSON of every schema file, hashed by tool name.
Prompt templates: every prompt file, content-hashed with its path.
Model identity and params: requested model, returned model, temperature, max tokens, tool timeout.
Code and deps: lockfile hash plus interpreter version.

Add one field that most harnesses skip: model_returned. Copy it from the response metadata, not from your config. If an endpoint can serve different models behind one name, this field is what catches it.

The artifact: `fingerprint.py`

Two files, no dependencies, Python 3.9 or newer.

# fingerprint.py
import hashlib
import json
import platform
from pathlib import Path

TOOL_DIR = Path("tools/schema")
PROMPT_DIR = Path("prompts")
LOCKFILE = Path("requirements.lock")


def canon(obj) -> str:
    return json.dumps(obj, sort_keys=True, separators=(",", ":"), ensure_ascii=False)


def dir_hash(root: Path, pattern: str) -> str:
    h = hashlib.sha256()
    for f in sorted(root.rglob(pattern)):
        h.update(f.relative_to(root).as_posix().encode() + b"\0")
        h.update(f.read_bytes() + b"\0")
    return h.hexdigest()


def tool_schema_hash() -> str:
    h = hashlib.sha256()
    for f in sorted(TOOL_DIR.glob("*.json")):
        h.update(f.stem.encode() + b"\0")
        h.update(canon(json.loads(f.read_text())).encode() + b"\0")
    return h.hexdigest()


def build_manifest(model_requested, model_returned, params, tool_timeout_s) -> dict:
    m = {
        "model_requested": model_requested,
        "model_returned": model_returned,
        "model_params": params,
        "tool_timeout_s": tool_timeout_s,
        "tool_schemas": tool_schema_hash(),
        "prompts": dir_hash(PROMPT_DIR, "*.txt"),
        "lockfile": hashlib.sha256(LOCKFILE.read_bytes()).hexdigest(),
        "python": platform.python_version(),
    }
    m["manifest_id"] = hashlib.sha256(canon(m).encode()).hexdigest()[:16]
    return m


def open_trace(path: str, manifest: dict):
    f = open(path, "w", encoding="utf-8")
    line = {"span_type": "run_manifest", "schema": 1, **manifest}
    f.write(json.dumps(line, sort_keys=True) + "\n")
    f.flush()
    return f

Call it once per run, before the first span. The snippet below is illustrative; the exact response field depends on your client.

manifest = build_manifest(
    model_requested="<your-model-name>",
    model_returned=response.model,       # from response metadata
    params={"temperature": 0.0, "max_tokens": 1024},
    tool_timeout_s=30,
)
trace = open_trace(f"traces/run-{run_id}.jsonl", manifest)

Note the schema field. Bump it when you change the manifest shape. Old traces then fail loudly instead of silently comparing the wrong keys.

Classify a mismatch in one command

# attribute.py
import json
import sys


MODEL_KEYS = {"model_requested", "model_returned", "model_params"}
IGNORED = {"manifest_id", "schema"}


def load_manifest(path: str) -> dict:
    with open(path, encoding="utf-8") as f:
        return json.loads(f.readline())


def classify(recorded: dict, current: dict):
    keys = set(recorded) | set(current)
    diffs = sorted(
        k for k in keys
        if k not in IGNORED and recorded.get(k) != current.get(k)
    )
    if not diffs:
        return "SAME_ENV", diffs
    if MODEL_KEYS & set(diffs):
        return "MODEL_DRIFT", diffs
    if "tool_schemas" in diffs:
        return "SCHEMA_DRIFT", diffs
    if {"prompts", "lockfile"} & set(diffs):
        return "CODE_DRIFT", diffs
    return "ENV_DRIFT", diffs


if __name__ == "__main__":
    recorded = load_manifest(sys.argv[1])
    with open(sys.argv[2], encoding="utf-8") as f:
        current = json.load(f)
    verdict, diffs = classify(recorded, current)
    print(verdict, diffs)
    sys.exit(0 if verdict == "SAME_ENV" else 3)

Emit the current manifest with the same builder your runner uses. One small script is enough:

# manifest_now.py
import json
from fingerprint import build_manifest

print(json.dumps(build_manifest(
    model_requested="<your-model-name>",
    model_returned="<value from a probe call>",
    params={"temperature": 0.0, "max_tokens": 1024},
    tool_timeout_s=30,
)))

Then wire it in front of your replay step.

python manifest_now.py > current.json
python attribute.py traces/run-abc123.jsonl current.json || exit 3
python replay.py traces/run-abc123.jsonl > replay.jsonl
python trace_diff.py traces/run-abc123.jsonl replay.jsonl

Exit code 3 means "do not trust this diff yet". In CI that is a guard. In a terminal it is a prompt to look at the environment first.

A debug loop you can reuse

Replay under the recorded manifest. Pin model, prompts, and schemas to the trace values. If you cannot pin them, stop here.
Run the classifier. SAME_ENV means the diff is real behavior. Anything else means fix the environment first.
Diff the first divergence only. Ignore everything after it. Later differences are usually cascade noise.
Record the fix as a new trace. Do not edit old traces. Traces are evidence, not scratch paper.
Turn the failing span pair into a test. Two spans plus one assertion equals one regression test.

Steps 2 and 3 are the ones teams skip. The classifier costs one command. It cancels the "is it the model?" meeting before it starts.

Decision table

Verdict	First check	Usual action
`MODEL_DRIFT`	`model_returned` vs recorded value	Accept as new behavior, re-record baseline
`SCHEMA_DRIFT`	`git log tools/schema`	Fix the tool or migrate the prompt
`CODE_DRIFT`	Prompt file or lockfile diff	Revert, or accept and re-record
`ENV_DRIFT`	Timeout, interpreter, clock	Pin the environment, then replay again
`SAME_ENV`	The first differing span	Debug agent logic with confidence

One rule survives all five rows: never debug step N+1 before you explain step N.

Where a free server fits

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

A replay loop is mostly idle work. Nightly replay of ten saved traces is a cron job, not a cluster. The steady load is a small number of model calls plus a lot of file hashing.

That shape of work is exactly what a free server option is good for. The MonkeyCode operator states that the project offers free model access and a free server option, including a free tier with 10 million tokens. Treat that as a claim to verify on the current pricing page, not as a permanent guarantee. Free tiers change, and quotas move.

The second caveat matters more for this article. If model identity behind an endpoint can shift, model_returned is not optional. Record it every call, or your replay diffs will send you hunting for a model bug that never existed.

If you already have a machine, run this locally. The workflow does not depend on any vendor.

Limitations and who should skip this

Fingerprints detect drift, they do not explain it. You still read git log to find the cause.
Hashes cover files, not runtime state. Feature flags, env vars, and database rows are invisible unless you hash them too.
Non-determinism survives. Timeouts and retries still vary between replays. Pin them or accept the noise.
Short-lived traces do not need this. If you replay within the hour, drift is unlikely to be the cause.
Fully pinned containers may make it redundant. Check your existing pinning before adding a layer.
The code above is illustrative. I have not published benchmark numbers for it, and I have not measured how much time it saves across a team.

Skip this entirely if you cannot replay runs at all. A manifest tells you nothing when there is no second run.

What to record next

Two small additions to your span schema buy most of the value:

manifest_id on every span, not just the first line. Then you can group failures by environment.
model_returned on every model call. Then a provider-side change is visible in one column.

With those two fields, one query answers the question that costs teams the most: which failing replays even had a stable environment?

Start with the manifest. The diff can wait until you trust it.

Record Once, Replay Forever: Cassette Tests for Flaky Agent Tool Calls

Riley Wang — Tue, 20 Oct 2026 09:36:05 +0000

An agent fails a task on Tuesday morning. You rerun the identical task on Wednesday and it passes. No commit landed in between.

That is not a fix. It is a new experiment with different inputs. Retrieval changed, rate limits reset, and sampling took another path.

Cassette testing closes that gap. You record the tool boundary once, then replay it a hundred times. The idea comes from HTTP recording, and it transplants to agents with little friction.

Live reruns destroy the evidence you need

Four things change between two live runs of the same task:

Retrieval returns a different top document.
A rate limit turns a 200 into a 429.
Sampling picks a different tool on step four.
The prompt embeds today's date or a fresh run id.

Each one rewrites the trace. A failure you cannot reproduce is an anecdote, not a bug report.

Freeze the tool boundary, not the model

Model output is stochastic on purpose. Tool arguments and responses are plain data. Data can be frozen, diffed, and replayed.

So record the dispatcher. Keep the model live if you prefer, or stub it. The interesting failure usually sits downstream of a tool result.

Cassette format: one JSONL file per run

Each line is one tool call. Keep both the readable arguments and a digest.

{"seq":1,"tool":"search_docs","args":{"q":"retry budget","top_k":5},"args_digest":"sha256:9f2c41ab","response":{"hits":3,"top":"retry-budget.md"}}
{"seq":2,"tool":"read_file","args":{"path":"docs/retry-budget.md"},"args_digest":"sha256:41ab07de","response":{"bytes":812,"sha":"sha256:0d1e..."}}

seq keeps ordering honest. args_digest is the lookup key. args exists so humans can read the file during review.

A replay dispatcher with zero dependencies

Standard library only, so it runs on any small box.

# cassette.py — stdlib only, Python 3.11
import hashlib, json
from pathlib import Path

def digest(args):
    raw = json.dumps(args, sort_keys=True, separators=(',', ':')).encode()
    return 'sha256:' + hashlib.sha256(raw).hexdigest()[:16]

class Cassette:
    def __init__(self, path):
        self.path = Path(path)
        lines = self.path.read_text().splitlines()
        self.entries = [json.loads(l) for l in lines if l.strip()]
        self.index = {(e['tool'], e['args_digest']): e for e in self.entries}
        self.hits, self.misses = {}, []

    def call(self, tool, args):
        key = (tool, digest(args))
        if key not in self.index:
            self.misses.append({'tool': tool, 'args_digest': key[1]})
            raise KeyError(f'cassette miss: {tool} {key[1]}')
        self.hits[key] = self.hits.get(key, 0) + 1
        return self.index[key]['response']

    def report(self):
        repeats = {f'{t}:{d}': n for (t, d), n in self.hits.items() if n > 1}
        return {'entries': len(self.entries), 'used': len(self.hits),
                'repeats': repeats, 'misses': self.misses}

The record side is one function. Wire it into your tool dispatcher and never touch it again.

def record(sink, seq, tool, args, response):
    row = {'seq': seq, 'tool': tool, 'args': args,
           'args_digest': digest(args), 'response': response}
    sink.write(json.dumps(row, sort_keys=True) + '\n')

Misses raise on purpose. A silent fallback to live calls would hide exactly the drift you are hunting. Treat these snippets as a starting point and run them against your own dispatcher first.

Three mismatch signatures, three diagnoses

Symptom	Likely cause	First action
Miss on arguments	Timestamps, uuids, or absolute paths in args	Normalize args before digesting
Same key hit many times	Retry storm or agent loop	Compare counts against recorded `seq`
Replay passes, live fails	Recorded response is stale	Re-record, then diff both cassettes

The third row matters most. It tells you the world moved, not your code.

The debug loop, end to end

Record the failing run once, with network access.
Replay it twenty times with networking disabled.
Confirm the pass rate is zero or one, never fifty percent.
Diff the cassettes from a failing and a passing run.
Freeze the offending key as a fixture.
Re-record after the fix and keep both files in review.

# 1. record one failing run
python agent.py --task retry-budget --cassette runs/fail.jsonl

# 2. replay it 20 times, no network
for i in $(seq 20); do python agent.py --task retry-budget --replay runs/fail.jsonl; done

# 3. re-record after a fix and diff the two
diff <(jq -S 'del(.recorded_at)' runs/fail.jsonl) \
     <(jq -S 'del(.recorded_at)' runs/fix.jsonl)

Step three is the gate. If replay is still flaky, your non-tool layers are nondeterministic. Fix those before blaming retrieval.

Normalize before you digest

Sort keys and strip transport-only fields.
Drop timestamps, uuids, and request ids.
Relativize file paths to the repo root.
Keep semantic arguments intact, including filters and limits.

Over-normalize and unrelated calls collide. Under-normalize and every run misses. Start strict, then loosen one rule at a time.

What this method cannot do

It does not capture model nondeterminism. Sampling stays live unless you stub it.
It does not preserve timing or load. Failures that need concurrency will not appear.
It can leak secrets. Redact responses before the cassette enters a repository.
It struggles with side effects. Writes, payments, and deploys need a dry-run mode.

Skip this approach if your failures depend on live ranking quality or on wall-clock behavior. It also fits poorly when tools are non-idempotent and cannot be stubbed safely.

Run the loop on the cheapest box you have

Replay is CPU work. Only recording touches a model. Split the two and most debugging stops competing for a budget.

MonkeyCode, an open-source project per the operator, offers free model access and a free server option. That split matches this loop: record once on the hosted server, then replay the cassette anywhere, including CI.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Quota figures move, so treat any number as a snapshot. The operator cites a 10M-token allowance; confirm the current figure on the project page before planning a large record batch.

If you want to test the idea without provisioning anything, record one cassette on the free server and check whether twenty replays give you the same failure.

Ship checklist

A cassette file per failing run, committed with the trace.
A replay command that works with networking disabled.
One fixture pinned to the historical bug.
A redaction pass before every commit.
A note in the PR that replay is deterministic, live is not.

The cassette is the artifact. The model stays a variable, and that is fine.

Same Call, Same Result: A No-Progress Detector for Agent Tool Loops

Riley Wang — Mon, 19 Oct 2026 08:19:43 +0000

An agent sat on step 7 for forty minutes. It was not hung. It was repeating.

Every turn it re-ran the same search. Every turn the tool returned the same twelve lines. Then it re-planned the same fix.

The trace above is illustrative. Step counts and timings are placeholders, not measurements.

A diff between two runs will not catch this. A loop lives inside a single run.

Why run-level diffing misses a loop

Diffing answers "what changed between attempt A and attempt B". A stuck agent answers a different question: "what did not change inside attempt A".

The diff shows two identical branches. That looks like stability, not failure.
Token and latency metrics rise smoothly. No spike marks the turn where progress stopped.
The final answer often looks plausible. The loop burned budget, then produced something.

You need a second signal: progress per tool call. Progress is measurable without understanding the task.

Give every tool call a signature

A tool call is a tuple: tool name, normalised arguments, result digest, status. Hash the tuple and you get a signature you can count.

Normalise before you hash

Raw arguments are full of noise. Drop volatile fields or every call looks unique.

Remove timestamps, request IDs, span IDs, nonces, and temp paths.
Sort object keys so key order cannot create false variety.
Canonicalise paths and whitespace before hashing.
Hash the result too. Same call plus different result is progress, not a loop.

That last rule matters most. A polling call repeats by design and still makes progress.

Build the progress ledger

# progress_ledger.py
import json, hashlib
from dataclasses import dataclass

VOLATILE = {"ts", "timestamp", "request_id", "trace_id", "span_id", "nonce"}

def stable(obj):
    """Deterministic JSON: sorted keys, volatile fields removed."""
    if isinstance(obj, dict):
        return {k: stable(v) for k, v in sorted(obj.items()) if k not in VOLATILE}
    if isinstance(obj, list):
        return [stable(v) for v in obj]
    return obj

def digest(obj, n=12):
    blob = json.dumps(stable(obj), sort_keys=True, separators=(",", ":"))
    return hashlib.sha256(blob.encode()).hexdigest()[:n]

@dataclass
class Step:
    seq: int
    tool: str
    args: str
    result: str
    status: str

def load(path):
    steps = []
    with open(path) as f:
        for line in f:
            line = line.strip()
            if not line:
                continue
            ev = json.loads(line)
            if ev.get("type") != "tool_call":
                continue
            steps.append(Step(
                seq=ev["seq"],
                tool=ev["tool"],
                args=digest(ev.get("args", {})),
                result=digest(ev.get("result", {})),
                status=ev.get("status", "ok"),
            ))
    return steps

def ledger(steps):
    seen, rows = {}, []
    for s in steps:
        sig = f"{s.tool}:{s.args}:{s.result}:{s.status}"
        history = seen.setdefault(sig, [])
        history.append(s.seq)
        rows.append({
            "seq": s.seq,
            "tool": s.tool,
            "signature": sig,
            "repeat_of": history[0] if len(history) > 1 else None,
            "count": len(history),
        })
    return rows

def stall_score(rows):
    if not rows:
        return 0.0
    repeats = sum(1 for r in rows if r["repeat_of"] is not None)
    return repeats / len(rows)

The ledger is append-only and cheap. One pass over the trace file, no model calls.

Run it on a trace

Your trace needs three fields per tool event: seq, tool, and args/result. If you already log JSONL spans, you have them.

python - <<'PY'
from progress_ledger import load, ledger, stall_score
rows = ledger(load("traces/run_7f2a.jsonl"))
for r in rows:
    print(r["seq"], r["tool"], r["count"], r["repeat_of"])
print("stall_score", round(stall_score(rows), 3))
PY

Output shape for a loop-shaped trace:

seq	tool	count	repeat_of
5	search	1	None
6	read_file	1	None
7	search	2	5
9	search	3	5
11	search	4	5

stall_score near 0.4 with four repeats of one signature is a loop. A healthy run trends toward 0.0.

Break the loop without killing the run

Detection is postmortem. You also want a live gate. Keep it tiny.

class StallDetected(Exception):
    pass

class NoProgressBreaker:
    def __init__(self, limit=3):
        self.limit, self.last, self.hits = limit, None, 0

    def observe(self, tool, args, result):
        sig = f"{tool}:{digest(args)}"
        cur = (sig, digest(result))
        if cur == self.last:
            self.hits += 1
        else:
            self.last, self.hits = cur, 1
        if self.hits > self.limit:
            raise StallDetected(f"{sig} repeated {self.hits}x with identical result")

Then pick a policy. Do not hard-fail on the first repeat.

Repeats	Result digest	Action
1-2	changed	Continue silently
3	unchanged	Inject a stall report into the context, allow one re-plan
5	unchanged	Break the loop, return partial work plus the ledger
any	changed every time	Not a loop. Cap cost instead.
any	`status=error`	Escalate as a tool failure, not a stall

A stall report is more useful than a stack trace. It names the signature, the repeat count, and the first seq where it appeared. The model often changes strategy when it sees that.

Test the detector with three fixture traces

Write fixtures by hand. Do not wait for a production loop.

import pytest
from progress_ledger import load, ledger, stall_score

@pytest.mark.parametrize("fixture", [
    "fixtures/loop.jsonl",
    "fixtures/progress.jsonl",
    "fixtures/poll_idempotent.jsonl",
])
def test_shapes(fixture):
    score = stall_score(ledger(load(fixture)))
    if fixture.endswith("loop.jsonl"):
        assert score > 0.5
    elif fixture.endswith("progress.jsonl"):
        assert score == 0.0
    else:
        assert score > 0.0  # known false positive, see limitations

Three files, three behaviours. The third one is the important one: it keeps you honest about what the detector cannot do.

Where a free tier fits into this loop

The detector is model-agnostic. It only reads tool calls. That makes it a good fit for testing the same task against a second model without paying twice for the trace.

This is where MonkeyCode is relevant. The open-source project offers free model access and a free server option, and the operator states a 10M-token allowance on the free tier. Terms change, so confirm the current numbers in the repo before you plan capacity around them.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

A practical use: run your stalled task on the free tier, capture JSONL spans, and feed both traces to the same ledger. If the loop signature matches across two models, your prompt or tool schema is the cause. If only one model loops, it is a model-behaviour issue. That distinction saves hours.

If you want a second trace without a second invoice, the free server option is a reasonable starting point.

Limitations and known false positives

This is a narrow detector. It has real failure modes.

Idempotent polling. Cache checks and status polls repeat with unchanged results by design. Allowlist them by tool name.
Nondeterministic results. If the tool returns a fresh UUID or timestamp each call, the result digest never matches. Normalise or exclude that tool.
Aggressive normalisation. Drop too many fields and two genuinely different calls collapse into one signature. You get false loops.
No causal answer. The ledger says progress stopped. It never says why the plan was wrong.
Non-tool reasoning loops. An agent can circle in pure text with zero tool calls. This detector is blind to that.

Set the repeat limit per tool class, not globally. search and deploy deserve different thresholds.

Who should not use this

Skip it if your agent makes one model call per request. There is no loop to detect.

Skip it if you have no structured tool log yet. Fix logging first, then come back.

Skip it if your runs are fully deterministic and short. A repeat count under three is noise, not signal.

The reusable part

Four lines of policy, one hash, one pass over a file. That is the whole mechanism.

The ledger survives model swaps, prompt rewrites, and vendor changes. Tool calls are the stable interface. Count them, hash their results, and stop paying for turns that change nothing.

The Token Leader Is Not the Latency Leader: Build a Span Ledger for Agent Runs

Riley Wang — Sun, 18 Oct 2026 07:01:09 +0000

An agent runs overnight. In the morning the free tier is gone and the log says loop finished. Nobody knows which step spent the budget.

The trace already has the answer. It is just not sorted the way we need. Most trace viewers rank spans by time. Cost is a different axis, and it disagrees.

This post builds a small span ledger. It answers two questions per run: where did the tokens go, and where did the wall clock go? Then it adds a deterministic budget breaker you can test.

Why time-ordered traces mislead you

A trace UI sorts by started_at. That order is correct and useless for cost questions.

A 180-second test run looks like the problem. It may cost zero tokens.
A 4-second model call can hold half the run's tokens.
A retried tool call doubles a line item that looks free.

You need two rankings, not one sorted list. Build both from the same spans.

The minimum span schema

Six fields carry the whole ledger. Anything else is optional.

Field	Type	Why the ledger needs it
`span_id`	string	Deduplicates on replay
`parent_id`	string \| null	Separates sub-agent spend from the root
`seq`	integer	Gives a deterministic iteration order
`kind`	`model` \	`tool` \
`started_at` / `ended_at`	ISO-8601 UTC	Computes wall clock per span
`prompt_tokens` / `completion_tokens`	integer	Records cost; tools must log `0`, not omit
`retry_of`	string \| null	Marks a repeated attempt

If your tracer cannot emit per-span token counts, stop here. Aggregate usage per run is not enough to attribute anything.

Artifact 1: the ledger script

Save the fixture and the script in one directory. The fixture is six spans, one retry, one slow tool.

{"span_id":"s1","parent_id":null,"seq":1,"name":"planner","kind":"model","started_at":"2026-10-18T01:00:00Z","ended_at":"2026-10-18T01:00:06Z","prompt_tokens":900,"completion_tokens":300,"status":"ok","retry_of":null}
{"span_id":"s2","parent_id":"s1","seq":2,"name":"repo.search","kind":"tool","started_at":"2026-10-18T01:00:06Z","ended_at":"2026-10-18T01:00:07Z","prompt_tokens":0,"completion_tokens":0,"status":"ok","retry_of":null}
{"span_id":"s3","parent_id":"s2","seq":3,"name":"reader","kind":"model","started_at":"2026-10-18T01:00:07Z","ended_at":"2026-10-18T01:00:15Z","prompt_tokens":3200,"completion_tokens":800,"status":"ok"}
{"span_id":"s4","parent_id":"s2","seq":4,"name":"repo.search","kind":"tool","started_at":"2026-10-18T01:00:15Z","ended_at":"2026-10-18T01:00:16Z","prompt_tokens":0,"completion_tokens":0,"status":"ok","retry_of":"s2"}
{"span_id":"s5","parent_id":null,"seq":5,"name":"test.run","kind":"tool","started_at":"2026-10-18T01:00:16Z","ended_at":"2026-10-18T01:03:16Z","prompt_tokens":0,"completion_tokens":0,"status":"ok"}
{"span_id":"s6","parent_id":"s3","seq":6,"name":"summarize","kind":"model","started_at":"2026-10-18T01:03:16Z","ended_at":"2026-10-18T01:03:21Z","prompt_tokens":2600,"completion_tokens":400,"status":"ok"}

# ledger.py
import json
from collections import defaultdict
from datetime import datetime
from pathlib import Path


def ts(value: str) -> datetime:
    return datetime.fromisoformat(value.replace("Z", "+00:00"))


def read_spans(path) -> list:
    spans = []
    for lineno, raw in enumerate(Path(path).read_text().splitlines(), 1):
        raw = raw.strip()
        if not raw:
            continue
        try:
            spans.append(json.loads(raw))
        except json.JSONDecodeError as exc:
            raise ValueError(f"{path}:{lineno}: {exc}") from exc
    return sorted(spans, key=lambda s: s["seq"])


def span_row(span) -> dict:
    ms = int((ts(span["ended_at"]) - ts(span["started_at"])).total_seconds() * 1000)
    tokens = span.get("prompt_tokens", 0) + span.get("completion_tokens", 0)
    return {
        "name": span["name"],
        "kind": span["kind"],
        "ms": ms,
        "tokens": tokens,
        "failed": span.get("status", "ok") != "ok",
        "retry": bool(span.get("retry_of")),
    }


def ledger(path):
    rows = [span_row(s) for s in read_spans(path)]
    total_ms = sum(r["ms"] for r in rows) or 1
    total_tokens = sum(r["tokens"] for r in rows) or 1
    groups = defaultdict(
        lambda: {"calls": 0, "ms": 0, "tokens": 0, "retries": 0, "failed": 0}
    )
    for r in rows:
        g = groups[(r["kind"], r["name"])]
        g["calls"] += 1
        g["ms"] += r["ms"]
        g["tokens"] += r["tokens"]
        g["retries"] += int(r["retry"])
        g["failed"] += int(r["failed"])
    out = []
    for (kind, name), g in groups.items():
        out.append({**g, "kind": kind, "name": name,
                    "token_share": round(100 * g["tokens"] / total_tokens, 1),
                    "time_share": round(100 * g["ms"] / total_ms, 1)})
    return sorted(out, key=lambda r: r["tokens"], reverse=True), total_ms, total_tokens


if __name__ == "__main__":
    rows, total_ms, total_tokens = ledger("trace.jsonl")
    print(f"run total: {total_tokens} tokens, {total_ms} ms")
    for r in rows:
        print(
            f"{r['kind']:<5} {r['name']:<12} calls={r['calls']} "
            f"tokens={r['tokens']} ({r['token_share']}%) "
            f"ms={r['ms']} ({r['time_share']}%) retries={r['retries']}"
        )

Expected output for the fixture above (values follow from the six spans; run it to confirm):

run total: 8200 tokens, 201000 ms
model reader       calls=1 tokens=4000 (48.8%) ms=8000   (4.0%)  retries=0
tool  repo.search  calls=2 tokens=0    (0.0%)  ms=2000   (1.0%)  retries=1
model summarize    calls=1 tokens=3000 (36.6%) ms=5000   (2.5%)  retries=0
model planner      calls=1 tokens=1200 (14.6%) ms=6000   (3.0%)  retries=0
tool  test.run     calls=1 tokens=0    (0.0%)  ms=180000 (89.6%) retries=0

The two rankings point at different spans. reader owns 48.8% of tokens. test.run owns 89.6% of wall clock and zero tokens. Cutting the slow tool saves nothing on cost.

Artifact 2: a budget breaker you can test

The ledger is retrospective. Add one forward-looking rule before the next dispatch.

Rule: if committed + estimate > cap, do not dispatch the span. Write the decision and stop the run.

# breaker.py
import json
from pathlib import Path
from ledger import read_spans


def next_span_allowed(committed: int, cap: int, est: int):
    projected = committed + est
    if projected > cap:
        return False, f"projected {projected} > cap {cap}"
    return True, f"projected {projected} <= cap {cap}"


def walk(path, cap: int, out_path):
    committed, last_model = 0, 0
    lines = []
    for span in read_spans(path):
        tokens = span.get("prompt_tokens", 0) + span.get("completion_tokens", 0)
        est = span.get("est_tokens", last_model if span["kind"] == "model" else 0)
        ok, reason = next_span_allowed(committed, cap, est)
        lines.append({"seq": span["seq"], "name": span["name"],
                      "action": "dispatch" if ok else "abort",
                      "committed": committed, "est": est, "reason": reason})
        if not ok:
            break
        committed += tokens
        if span["kind"] == "model":
            last_model = tokens
    Path(out_path).write_text("\n".join(json.dumps(l) for l in lines) + "\n")
    return lines

The estimator is crude on purpose. It uses the last model span, or an explicit est_tokens field when the caller knows better. Crude and deterministic beats clever and untestable.

Artifact 3: the regression set

Five tests keep the ledger honest. They run in under a second and need no network.

# test_ledger.py
import json
import pytest
from ledger import ledger, read_spans
from breaker import walk


def test_totals():
    _, total_ms, total_tokens = ledger("trace.jsonl")
    assert (total_tokens, total_ms) == (8200, 201_000)


def test_rankings_differ():
    rows, _, _ = ledger("trace.jsonl")
    assert rows[0]["name"] == "reader"
    slowest = max(rows, key=lambda r: r["ms"])
    assert slowest["name"] == "test.run"
    assert slowest["tokens"] == 0


def test_retry_is_counted_not_hidden():
    rows, _, _ = ledger("trace.jsonl")
    search = next(r for r in rows if r["name"] == "repo.search")
    assert search["calls"] == 2 and search["retries"] == 1


def test_breaker_aborts_before_dispatch(tmp_path):
    out = tmp_path / "budget.jsonl"
    lines = walk("trace.jsonl", cap=6000, out_path=out)
    assert lines[-1]["action"] == "abort"
    assert lines[-1]["seq"] == 6
    assert all(l["action"] == "dispatch" for l in lines[:-1])
    assert json.loads(out.read_text().splitlines()[-1])["action"] == "abort"


def test_bad_line_names_the_line(tmp_path):
    bad = tmp_path / "bad.jsonl"
    bad.write_text('{"seq": 1}\nnot json\n')
    with pytest.raises(ValueError, match="bad.jsonl:2"):
        read_spans(bad)

At cap=6000 the walk aborts at seq 6 with projected 9200 > cap 6000. The run keeps 5200 committed tokens and skips the 3000-token summary.

The debug loop, five steps

Capture the trace with seq, kind, and per-span tokens.
Run the ledger. Read both rankings before forming a hypothesis.
Name the top token span and the top time span. If they differ, you have two bugs, not one.
Cut one thing: shorten the prompt, cache the retrieval, or drop the retry.
Re-run. Diff the two ledgers. The delta is the evidence.

Step 5 is where most teams stop early. A cheaper run with a different failure mode is not a win. Compare the failed column too.

Where free access fits, and where it does not

The loop above reruns a lot. Every cut means another replay with the same fixture. That is exactly the shape of work where free model access and a free server option reduce friction: the replay runs while you sleep, and the ledger diff is waiting.

MonkeyCode offers free model access and a free server option, per the project's own description. Check the current limits on its site before you size a batch around them; availability and caps are the operator's to state, not a blog post's.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

One more caveat. Do not point the breaker at a paid key with a production budget. Run it in a sandbox first, and keep the decision log next to the trace.

If you want to try the loop without buying tokens first, the free tier is the low-risk place to start.

Limitations and who should skip this

Aggregate-only providers. If your API returns one usage number per request tree, per-span attribution is fiction.
Single-call scripts. One model call has no ranking to compare. Skip the ledger.
Streaming partials. A span that never closes has no ended_at. Emit a sentinel on timeout or the table lies.
Nested sub-agents. Use parent_id to build a tree. Flat sums double-count when a parent includes child tokens.
Sub-second tools. Millisecond rounding will dominate. Track counts, not shares.

The ledger is not a dashboard. It is a five-line answer to "which span do I cut?" That is usually enough.

What to keep

Two rankings, one trace. Token share and time share disagree more often than intuition suggests.

Record seq, kind, and tokens per span. Everything else is arithmetic you can write in forty lines.

Test the breaker on a fixture, not on a live budget. A rule you cannot reproduce is a rule you cannot trust.

Scrub the Trace, Keep the Failure: A Redaction Gate You Can Test

Riley Wang — Sat, 17 Oct 2026 06:05:42 +0000

Every agent team hits the same wall. A run fails in staging, and someone wants a second pair of eyes.

The trace is 4 MB of prompts, tool arguments, and request headers. Sharing it raw is a leak. Sharing a redacted copy that no longer reproduces the failure is worse.

This post builds a small gate between those two outcomes. A failure signature survives the scrub, and the secrets do not. The mechanism is one policy table, two scripts, and two assertions you can run in CI.

The part you cannot lose is the signature

A trace is only useful if it still says where it broke. Call that the signature: the ordered list of failing spans, each reduced to (seq, name, error_kind).

If the signature is identical before and after redaction, the diagnosis path is intact. If it changes, the scrubber destroyed evidence. That is a testable property, so test it rather than eyeballing diffs.

Write the policy as a table, not a regex

Field names are stable across runs. Values are not. Classify each attribute once, then enforce it.

Attribute	Action	Reason
`seq`, `ts`, `type`, `name`	keep	ordering, latency, cost attribution
`span_id`, `parent_id`	keep	rebuild the span tree
`user_id`, `session`, `repo`, `path`	hash	joins survive, identity does not
`prompt`, `message`, `query`	hash	free text is where secrets hide
`tool`, `error_kind`, `exit_code`	keep as enum	this is the signature
`api_key`, `auth_header`, `cookie`, `env`	drop	never needed to debug

Three rules follow from the table. Hash with HMAC so equal inputs stay equal. Drop secrets outright instead of hashing them. Fail on unknown fields so the policy cannot silently rot.

The scrubber is about forty lines

One key, four sets, one function. Adapt the field sets to your own schema before trusting any output.

#!/usr/bin/env python3
"""Field-allowlist scrubber for agent JSONL traces."""
import hashlib, hmac, json, sys

KEY = open("scrub.key", "rb").read()

KEEP = {"seq", "ts", "type", "name", "status", "span_id", "parent_id"}
HASH = {"user_id", "session", "repo", "path", "prompt", "message", "query"}
DROP = {"api_key", "auth_header", "cookie", "env"}
ENUM = {"tool", "error_kind", "exit_code", "http_status", "model"}

def tag(value):
    raw = str(value).encode()
    return "h_" + hmac.new(KEY, raw, hashlib.sha256).hexdigest()[:12]

def scrub(event):
    out = {k: event[k] for k in KEEP if k in event}
    attrs = {}
    for key, value in event.get("attrs", {}).items():
        if key in DROP:
            continue
        if key in HASH:
            attrs[key] = tag(value)
        elif key in ENUM:
            attrs[key] = value
        else:
            attrs[key] = f"<{type(value).__name__}>"
    out["attrs"] = attrs
    return out

for line in sys.stdin:
    if line.strip():
        print(json.dumps(scrub(json.loads(line)), sort_keys=True))

Hashing with a single key keeps cross-run joins alive. Rotate the key and old hashes stop matching. Store the key outside the trace directory and never ship it with the data.

Two assertions on every trace

import json, sys

SENSITIVE = {"user_id", "session", "repo", "path", "prompt", "message",
             "query", "api_key", "auth_header", "cookie", "env"}
ENUMS = {"tool", "error_kind", "exit_code", "http_status", "model"}

def load(path):
    return [json.loads(l) for l in open(path) if l.strip()]

def signature(events):
    sig = []
    for e in events:
        a = e.get("attrs", {})
        failed = e.get("status") == "error" or a.get("exit_code") not in (None, 0)
        if failed:
            sig.append((e.get("seq"), e.get("name"), a.get("error_kind")))
    return sig

raw, clean = load(sys.argv[1]), load(sys.argv[2])
assert signature(raw) == signature(clean), "signature changed"

blob = open(sys.argv[2]).read()
leaks = [v for e in raw for k, v in e.get("attrs", {}).items()
         if k in SENSITIVE and len(str(v)) >= 6 and str(v) in blob]
unknown = {k for e in raw for k in e.get("attrs", {})} - (SENSITIVE | ENUMS)

print(f"signature ok ({len(signature(raw))} failures)")
print(f"leak scan ok ({len(leaks)} hits)")
print(f"unknown attrs: {len(unknown)} {sorted(unknown)}")

Check one is signature equality. Check two is the leak scan: any sensitive raw value of six characters or more appearing in the output fails the job. The third number is the unknown attribute count, which should be zero.

Run it locally first:

python -c "import os;open('scrub.key','wb').write(os.urandom(32))"
chmod 600 scrub.key
python scrub.py < traces/run-0917.jsonl > traces/run-0917.scrubbed.jsonl
python verify.py traces/run-0917.jsonl traces/run-0917.scrubbed.jsonl

A clean run prints 0 hits and unknown attrs: 0. A non-zero leak count means your policy is missing a field. Fix the table, not the test.

The loop you reuse

Reproduce the failure and keep the raw trace on the machine that produced it.
Scrub, then verify. Fail the build if the signature moves.
Hand the scrubbed file to whoever debugs it. They see structure and failure kinds.
Diff two scrubbed runs with any JSONL differ to isolate the changed span.
Fix, rerun, and compare signatures only. The raw text stays where it started.

Step four is where the gate pays off. Hashed IDs stay stable across runs, so a diff aligns on span_id instead of guessing.

Where a free tier actually helps

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The scrub-and-verify job is CPU-only and bursty. It runs when a trace lands, then idles. MonkeyCode's free server option covers that without a card or a provisioning step, which matters for a side project or a solo on-call rotation.

Free model access helps with the tedious part. Paste your attribute name list, never values, and ask for HASH/DROP assignments plus edge cases you missed. Review the output against the table, then add a leak-scan case for each suggestion.

If you want to run the loop before committing infrastructure, this is enough to process a few hundred traces.

What this does not solve

Length and type still leak. <str> plus a character count narrows a token.
Timestamps leak working hours and geography. Coarsen them if that matters.
Nested lists and dicts flatten badly. Extend scrub() or reject the field.
Third-party SDK fields land outside the policy. That is what the unknown counter is for.
error_kind must be set at instrumentation time. Derive it before you scrub, not after.

Hashing is not anonymization. Low-entropy values like a username can be brute-forced from a short hash.

Skip this approach if

Your traces fall under a formal DLP or compliance regime. Use an approved tool and a review process, not a forty-line script.

Also skip it if you need raw text for fine-tuning. This gate is built for debugging, and it is deliberately lossy.

Sample Runs, Not Spans: Shrink Agent Traces Without Losing Failures

Riley Wang — Fri, 16 Oct 2026 05:02:38 +0000

At 09:12 the on-call page fired. A checkout agent returned HTTP 500 for forty minutes.

We opened the trace store and found two weeks of spans. The failing run was not in it.

Our sampler had dropped the interesting spans. It kept the boring ones. That is the classic sampling bug, and it is a retention problem, not a tracing problem.

Why span-level sampling fails

Most teams sample at the span level. A coin flip decides each span independently.

That breaks causality. A tool call survives; its parent LLM call does not. You cannot replay a run with holes in it.

Worse, failures are rare and correlated. A 5% span sample catches roughly 5% of a broken run, and the interesting leaf spans are exactly the ones your status filter already flags.

So you pay 5% of the storage and lose 95% of the debuggability. The incident you care about is the one you stored least.

Change the unit of sampling

Sample runs, not spans. A run is either fully recorded or fully summarized. No partial runs, no orphan spans.

Three storage modes are enough:

Full — every span, every payload, exact bytes, sequence numbers intact.
Summary — one lossy record per span: id, kind, status, duration, token counts, byte counts. No payloads.
Anomaly-pinned — the same as full, but exempt from sampling forever, because the run already proved it can break.

Then add one rule that overrides chance. Any run that touches a known anomaly keeps full detail, regardless of the sample rate.

A decision table you can argue about

Span or run class	Decision	Retention	Why
run has any `error` span	full	30 days	root-cause evidence
any span has `status = retry`	full	30 days	retry storms hide real bugs
span has a truncation flag	full	14 days	the payload was already cut
span exceeds the latency p99 budget	full	14 days	regression evidence
clean run, hash-selected	full	7 days	baseline for diffing
clean run, not selected	summary	90 days	trend lines and cost accounting
health-check span	neither	0	pure noise

Two things stand out. Clean runs are kept longest in summary form, because they are cheap and they answer trend questions. Broken runs are kept shortest in full form, because the fix arrives fast.

The sampler, in about forty lines

This is a proposal, not a benchmarked result in your environment. Run it against archived traces before you put it in front of production traffic.

# sampler.py — proposal; measure it on your own archive first
import hashlib
import os

SALT = os.environ.get("TRACE_SAMPLE_SALT", "change-me")
ANOMALY_STATUSES = {"error", "retry", "truncated", "budget_exceeded"}

def is_anomalous_span(span: dict) -> bool:
    if span.get("status") in ANOMALY_STATUSES:
        return True
    return bool(span.get("truncated"))

def run_is_pinned(run: dict) -> bool:
    return run.get("anomaly_count", 0) > 0

def selected_for_full(run_id: str, success_sample_rate: float = 0.05) -> bool:
    """Deterministic, stable per run_id. Sample whole runs, never spans."""
    digest = hashlib.sha256((SALT + run_id).encode()).hexdigest()
    bucket = int(digest[:8], 16) % 10_000
    return (bucket / 10_000.0) < success_sample_rate

def store_mode(run: dict) -> str:
    if run_is_pinned(run):
        return "full"
    if selected_for_full(run["run_id"]):
        return "full"
    return "summary"

Two design choices matter here.

Hashing on run_id, not a random draw. The same run gets the same verdict in every process, on every retry, and after a restart. Random draws make your retention policy non-reproducible.
A SALT from the environment. Without it, anyone can compute which runs you kept. With it, selection is stable but not guessable.

The one test that matters

A single invariant protects the whole policy: an anomalous run is never summarized away.

# test_retention.py — proposal; wire into CI
def test_anomalous_runs_are_never_summarized():
    for status in ("error", "retry", "truncated", "budget_exceeded"):
        run = {"run_id": "r-1", "anomaly_count": 1, "status": status}
        assert store_mode(run) == "full"

def test_selection_is_stable_across_calls():
    a = [selected_for_full("r-42") for _ in range(100)]
    assert len(set(a)) == 1

def test_selection_is_not_all_or_nothing():
    kept = sum(selected_for_full(f"r-{i}") for i in range(10_000))
    assert 300 < kept < 700  # ~5% with sane variance

The third test is the subtle one. A salt or modulus bug can silently push your sample rate to 0% or 100%, and nobody notices until an incident review.

Where the pipeline runs

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The collector and summarizer are small, periodic, and CPU-bound. That is a good fit for a free server option, because you are not paying for idle capacity between jobs.

MonkeyCode is an open-source project with free model access, and the operator lists a 10M-token free allowance alongside the free server option. Treat both as claims to verify against the current terms before you depend on them.

The model access has a genuine role here. Full payloads are expensive to keep; one-line run digests are not. So generate a digest for every run, not just the sampled ones, and store it permanently next to the summary records. Something like:

Input: 40 summary records for run r-8891
Output: one sentence — what the run attempted, what changed, what failed

That digest is what an on-call engineer reads first at 09:12. It costs a few hundred tokens per run, which is why a free allowance changes what you can afford to keep.

Two cautions. Never put payload contents you cannot share into a third-party model call, and never treat a free tier as a system of record. Check the current limits and retention terms yourself.

Measure before you trust the policy

Do not adopt this on vibes. Compute the ratio on one day of archived traces.

# total bytes and runs
wc -c < traces.jsonl
wc -l < traces.jsonl

# bytes owned by anomalous runs (approximation: runs with an error line)
jq -c 'select(.status == "error") | .run_id' traces.jsonl \
  | sort -u > pinned.txt
rg -F -f pinned.txt traces.jsonl | wc -c

Then report two numbers: pinned_runs / total_runs and kept_bytes / total_bytes.

Acceptance criteria I would use, and you should replace with your own:

pinned_runs / total_runs under 2% in steady state.
kept_bytes / total_bytes under 15% after summarization.
Zero runs missing a stored digest.

If pinned_runs is much higher than your error rate, your anomaly rules are too broad. Fix the rules before you lower the storage bill.

Limitations

Summaries cannot answer unanticipated questions. If you did not record a field, you cannot query it later. Add fields to the summary schema only when you can justify the bytes.
Sampling breaks exact replay for unselected runs. Keep the sample rate high enough that a baseline exists for diffing.
Hash sampling can correlate with your id scheme. If run_id starts with a tenant prefix, some tenants get sampled more than others. Salt alone does not fix that; hash the full id or re-bucket.
Latency budgets need a warm baseline. On a sleeping free server, the first request after idle inflates duration. Exclude cold-start spans or your p99 rule pins everything.
Not for audit or legal hold. A sampling policy is an engineering choice, not a compliance record.

Who should skip this

If you run fewer than a few thousand agent runs per day, storage is cheap and full-fidelity tracing is simpler. Skip the sampler.

If you operate under retention obligations, keep everything and solve cost elsewhere. Skip the sampler.

If your failures are exactly the class you would sample away, for example silent wrong answers with clean statuses, pin on an outcome signal instead of a status code.

Start with one day of traffic

Sample runs, not spans. Pin anomalies unconditionally. Store a model-written digest for every run.

Take one archived day of traces. Run the sampler over it. Report the two ratios and the three invariants.

If the numbers hold, you have shrunk the store without losing the one run you will need at 09:12.

Validate the Trace Before You Trust the Trial: A Conformance Linter for Agent Runs

Riley Wang — Thu, 15 Oct 2026 11:15:28 +0000

The trial that looks like a regression

You run one task three times on the same cheap model. Two runs agree. The third returns a different plan.

You blame the model. You call it flaky. You reach for a bigger, slower one.

Then you open the raw trace. One span has no parent. Another ends before it starts. A tool output was cut at two kilobytes, and nothing recorded the cut.

Nothing wobbled inside the model. The record was broken. You measured a parser bug and called it variance.

Cheaper models make this worse, not better. Free access and cheap compute turn three trials into thirty. Thirty dirty traces give you thirty confident wrong answers.

Model quality keeps climbing. The plumbing around it does not. That gap is where most "the model is flaky" reports live.

Trace trust comes before trial count

A trace is evidence. Evidence needs a shape.

Most agent harnesses emit spans as JSONL. One line is one unit of work. Each unit carries an id, a parent, a name, and time bounds.

Those fields are trivial to write and easy to corrupt. Async workers drop parents. Clocks disagree across hosts. Truncation goes unflagged. A retry reuses an id.

Here is the rule that keeps every later conclusion honest:

Lint the trace before you interpret it.

A clean trace with a bad answer is a model problem. A dirty trace is an instrumentation problem. Those need different fixes. You cannot tell them apart by reading outputs alone.

Eight invariants a runnable trace must satisfy

It parses. Every line is valid JSON. One bad line invalidates the whole run.
Fields are complete. span_id, run_id, parent_id, name, start_ns, end_ns.
Ids are unique. No span_id repeats inside one run.
Time is ordered. end_ns >= start_ns on every span.
Parents resolve. Every non-root parent_id exists in the same run.
Children are contained. A child starts after its parent and ends before it.
Exactly one root. Unless you model fan-out with an explicit join span.
Tools report bytes. Every tool span records stdout_bytes and a truncated flag.

Invariants one through six are structural. Seven and eight are the ones people skip, and they cause the worst conclusions.

The artifact: a small linter

The script below uses only the Python standard library. It reads one run and prints findings. It is not tested against any vendor schema, so change REQUIRED to match your exporter first.

#!/usr/bin/env python3
'''trace_lint.py - conformance checks for one agent run in JSONL.

Not tested against vendor schemas. Adapt REQUIRED to your exporter.
Usage: python3 trace_lint.py path/to/run.jsonl
Exit:  0 clean, 1 findings, 2 unreadable input.
'''
import json
import sys
from collections import defaultdict

REQUIRED = ('span_id', 'run_id', 'parent_id', 'name', 'start_ns', 'end_ns')
TOOL_KINDS = {'tool', 'tool_call', 'function'}


def load(path):
    rows = []
    with open(path, 'r', encoding='utf-8') as fh:
        for lineno, raw in enumerate(fh, 1):
            raw = raw.strip()
            if not raw:
                continue
            try:
                row = json.loads(raw)
            except json.JSONDecodeError as exc:
                row = {'_parse_error': str(exc)}
            rows.append((lineno, row))
    return rows


def lint(rows):
    findings = []
    by_run = defaultdict(list)

    for lineno, row in rows:
        if '_parse_error' in row:
            findings.append(('E_PARSE', lineno, row['_parse_error']))
            continue
        missing = [k for k in REQUIRED if k not in row]
        if missing:
            findings.append(('E_MISSING', lineno, 'missing ' + ','.join(missing)))
            continue
        by_run[row['run_id']].append((lineno, row))

    for run_id, spans in by_run.items():
        index = {row['span_id']: (lineno, row) for lineno, row in spans}
        seen = {}
        roots = 0

        for lineno, row in spans:
            sid = row['span_id']
            if sid in seen:
                findings.append(('E_DUP_ID', lineno, sid + ' first seen line ' + str(seen[sid])))
            seen[sid] = lineno

            if row['end_ns'] < row['start_ns']:
                findings.append(('E_NEG_DUR', lineno, sid + ' end before start'))

            pid = row['parent_id']
            if pid in (None, '', 'root'):
                roots += 1
            elif pid not in index:
                findings.append(('E_ORPHAN', lineno, sid + ' parent ' + str(pid) + ' not in run'))
            else:
                parent = index[pid][1]
                if row['start_ns'] < parent['start_ns']:
                    findings.append(('E_PARENT_TIME', lineno, sid + ' starts before parent'))
                if row['end_ns'] > parent['end_ns']:
                    findings.append(('E_CHILD_OUTSIDE', lineno, sid + ' ends after parent'))

            if row.get('kind') in TOOL_KINDS:
                if 'stdout_bytes' not in row:
                    findings.append(('E_TOOL_BYTES', lineno, sid + ' no stdout_bytes'))
                if 'truncated' not in row:
                    findings.append(('E_TOOL_TRUNC', lineno, sid + ' no truncated flag'))

        if roots != 1:
            findings.append(('E_ROOT_COUNT', 0, 'run ' + run_id + ' has ' + str(roots) + ' roots'))

    return findings


def main(argv):
    if len(argv) != 2:
        print('usage: trace_lint.py <run.jsonl>', file=sys.stderr)
        return 2
    try:
        rows = load(argv[1])
    except OSError as exc:
        print('cannot read ' + argv[1] + ': ' + str(exc), file=sys.stderr)
        return 2

    findings = lint(rows)
    for code, lineno, detail in findings:
        print(code + ' line=' + str(lineno) + ' ' + detail)
    print(str(len(rows)) + ' lines scanned, ' + str(len(findings)) + ' findings')
    return 1 if findings else 0


if __name__ == '__main__':
    raise SystemExit(main(sys.argv))

A fixture that must fail

Save these three lines as fixtures/run-003.bad.jsonl. This trace is deliberately broken in four ways.

{"span_id": "r1", "run_id": "run-003", "parent_id": null, "name": "agent", "kind": "root", "start_ns": 1000, "end_ns": 9000}
{"span_id": "s2", "run_id": "run-003", "parent_id": "r1", "name": "plan", "kind": "llm", "start_ns": 1100, "end_ns": 4000}
{"span_id": "s3", "run_id": "run-003", "parent_id": "gone", "name": "read_file", "kind": "tool", "start_ns": 5000, "end_ns": 4800}

Run it:

python3 trace_lint.py fixtures/run-003.bad.jsonl; echo "exit=$?"

Expected output:

E_ORPHAN line=3 s3 parent gone not in run
E_NEG_DUR line=3 s3 end before start
E_TOOL_BYTES line=3 s3 no stdout_bytes
E_TOOL_TRUNC line=3 s3 no truncated flag
3 lines scanned, 4 findings
exit=1

Keep this fixture in CI. A linter that stops detecting known-bad input is worse than no linter at all.

Lint every trace in the comparison set

Run the linter over the whole directory before you compare any outputs.

dirty=0
for f in traces/*.jsonl; do
  if ! python3 trace_lint.py "$f" > /dev/null; then
    echo "DIRTY $f"
    dirty=$((dirty + 1))
  fi
done
echo "$dirty dirty traces"

If any trace is dirty, the comparison is void. Fix the emitter, regenerate the traces, then compare. Score outputs only from clean runs.

On a free server, this loop is a five-minute cron job. Point it at a traces directory and append failures to a file you actually read.

17 3 * * * cd /srv/agent-lab && bash lint_all.sh >> lint.log 2>&1

A failure taxonomy you can act on

Finding	What it usually means	Where to look first
`E_PARSE`	Interleaved writes from two processes	The log writer, not the agent
`E_MISSING`	Exporter schema drifted	Emitter version and defaults
`E_DUP_ID`	Id from a counter that resets per worker	Id source, not the model
`E_ORPHAN`	Context lost at an async boundary	Worker handoff and trace context
`E_NEG_DUR`	Wall clock mixed with monotonic clock	Clock source per host
`E_CHILD_OUTSIDE`	Parent closed on the first child return	Parent span lifecycle
`E_ROOT_COUNT`	Fan-out with no join span	Orchestrator join logic
`E_TOOL_TRUNC`	Capture layer dropped the flag	Tool output capture

Two rules make these findings trustworthy. Use time.monotonic_ns() on each host. Store host_id and boot_id next to every span. Wall clock is for humans. Monotonic time is for durations.

Where free models and a free server fit

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode offers free model access and a free server option. The account operator supplied those two availability claims. Quotas, model lists, hardware, and limits change over time, so read the project's own page before you plan around them. The project describes itself as open source; check the repository and the license yourself.

What that changes here is quantity, and quantity is exactly what this workflow needs.

You need many trials. Model variance separates from bug variance only across a corpus.
You need a cheap host. A linter plus a nightly replay job wants a small always-on box.
You need room to iterate on the emitter. Instrumentation bugs burn trials before they burn tokens.

A practical ladder looks like this:

Run five trials on the free tier.
Lint every trace. Fix the emitter until all five are clean.
Re-run the same task five more times on the same free models.
Only then compare against a paid model, with clean traces on both sides.

If the free tier survives step four, keep it. Cheap models are not the weak link once your traces are honest. If you want to try that loop without a card, the free tier and the free server option are the cheapest place to start.

Limitations

The linter checks structure, not meaning. A clean trace can still hold a wrong answer.
It assumes one run per file and one clock domain per host. Add host_id and boot_id before trusting cross-host durations.
Missing spans are invisible. If the emitter never wrote a span, no check will find it.
Aggressive sampling breaks containment checks. Sample deliberately, or lint the unsampled stream.
The code above is untested against your schema. Treat it as a starting point, not a product.

Who should not use this

Single-shot scripts with no tool calls. There is nothing to correlate.
Teams without a frozen span schema. Freeze the schema first, then lint.
Anyone who wants statistical significance from three runs. A linter is not a sample size.
Teams that cannot change their emitter. The linter will only report problems you cannot fix.

The five-minute version

Freeze the span schema. Add the linter. Keep a known-bad fixture in CI.

Then run your comparison. If the traces are clean and the answers still diverge, you have a real model finding. That finding is worth acting on. Everything before it was noise.

Hash Three Boundaries Before You Blame the Model

Riley Wang — Tue, 13 Oct 2026 00:29:24 +0000

A night operator opened two logs for the same tool call. The process log held valid JSON with two rows. The next model turn claimed the tool result was unreadable.

This scene is reconstructed for teaching, not a quoted outage. The figures below are fixtures, not measured production rates.

The handoff has three owners

A tool process writes the first byte string. An adapter then parses, wraps, or redacts that string. A prompt builder copies one slice into the next model message.

Any owner can change bytes without changing the tool name. A single stored hash cannot say which owner did it. You need one digest per boundary, plus the length at that boundary.

Fields worth storing

Keep these fields on the same tool-call row. Derive mutation_hop in code, not in an editor. A later reader should recompute that label from hashes.

process_sha256 is the hash of raw process output.
adapter_sha256 is the hash after parse, wrap, and redact.
model_sha256 is the hash of the tool-result slice only.
Store process_len, adapter_len, and model_len as byte lengths.
mutation_hop is the label chosen from the table below.

Hash the tool-result slice, not the full prompt. A full-prompt hash changes when any sibling message changes. That noise hides a stable tool payload from review.

Sample row

This JSON row is an illustrative teaching fixture only. It is not a captured trace from a production agent. Replace the short hash placeholders with real SHA-256 values.

{
  "run_id": "run_7f3a",
  "tool_call_id": "call_91",
  "tool_name": "read_file",
  "process_sha256": "replace-with-real-sha256",
  "adapter_sha256": "replace-with-real-sha256",
  "model_sha256": "replace-with-other-sha256",
  "process_len": 36,
  "adapter_len": 20,
  "model_len": 20,
  "mutation_hop": "adapter_shrink"
}

Read the hop table first

Read this table before you open a provider status page. Compare the three digests first, then use length as a tie break. Route the review to the hop named in the matching row.

Comparison	Length signal	Hop to inspect
Process hash differs, adapter shorter	Shrink	Adapter truncate or parse drop
Process hash differs, adapter longer	Growth	Adapter wrap or injected default
Process equals adapter, model longer	Growth	Prompt builder added a wrapper
Process equals adapter, model shorter	Shrink	Prompt builder dropped fields
All three hashes equal, lengths equal	None	Not a payload mutation
Hashes equal, model length disagrees	Mismatch	You hashed the wrong slice

The last row tests your instrumentation, not the model. Matching hashes with unequal lengths should stay rare. If you see that row, the model hash covered a different byte range.

Hand-counted fixture lengths

These lengths are hand counts of the fixture literals. They are not byte counts from a live agent trace. Confirm every number with len() when you run the file.

The wrap process literal is 20 ASCII bytes long. The wrap model slice literal is 48 ASCII bytes long. The extra 28 bytes are the type field and body wrapper.

The shrink process literal is 36 ASCII bytes long. The shrink adapter literal is 20 ASCII bytes long. Those 16 missing bytes are the inserted note field.

Classifier you can copy

The script below is an unexecuted teaching proposal. It was not executed inside a hosted agent for this article. It uses only the Python standard library for hashing.

#!/usr/bin/env python3
"""Classify which hop mutated a tool result. Unexecuted teaching fixture."""

import hashlib
import json
from typing import Any


def sha256_bytes(data: bytes) -> str:
    return hashlib.sha256(data).hexdigest()


def canonicalize(obj: Any) -> bytes:
    text = json.dumps(
        obj,
        sort_keys=True,
        separators=(",", ":"),
        ensure_ascii=False,
    )
    return text.encode("utf-8")


def classify(process: bytes, adapter: bytes, model_part: bytes) -> dict:
    p = sha256_bytes(process)
    a = sha256_bytes(adapter)
    m = sha256_bytes(model_part)
    if p != a and len(adapter) < len(process):
        hop = "adapter_shrink"
    elif p != a and len(adapter) > len(process):
        hop = "adapter_grow"
    elif p == a and a != m and len(model_part) > len(adapter):
        hop = "prompt_builder_wrap"
    elif p == a and a != m and len(model_part) < len(adapter):
        hop = "prompt_builder_drop"
    elif p == a and a == m and len(process) == len(adapter) == len(model_part):
        hop = "no_payload_mutation"
    elif p == a and a == m:
        hop = "hash_slice_mismatch"
    else:
        hop = "needs_manual_review"
    return {
        "process_sha256": p,
        "adapter_sha256": a,
        "model_sha256": m,
        "process_len": len(process),
        "adapter_len": len(adapter),
        "model_len": len(model_part),
        "mutation_hop": hop,
    }


def fixture_wrap() -> dict:
    process = b'{"ok":true,"rows":2}'
    adapter = canonicalize({"ok": True, "rows": 2})
    model_part = b'{"type":"tool_result","body":{"ok":true,"rows":2}}'
    return classify(process, adapter, model_part)


def fixture_shrink() -> dict:
    process = b'{"ok":true,"rows":2,"note":"abcdef"}'
    adapter = b'{"ok":true,"rows":2}'
    model_part = adapter
    return classify(process, adapter, model_part)


if __name__ == "__main__":
    print(json.dumps({"wrap": fixture_wrap(), "shrink": fixture_shrink()}, indent=2))

Run the fixture on a laptop before you trust the labels. Check two expectations in the printed JSON object. The wrap fixture should report the prompt_builder_wrap label.

The shrink fixture should report the adapter_shrink label. If the wrap fixture reports adapter_grow, stop and compare inputs. You hashed canonical adapter bytes against raw process bytes.

Key order and spaces change the hash even when values match. Treat that mismatch as a fixture bug before you blame the adapter. Rerun both fixtures after any change to canonicalize.

python3 three_boundary.py

Keep a raw hash and a canonical hash

Store both a raw adapter hash and a canonical adapter hash. Use the raw hash to detect any byte change. Use the canonical hash to ignore object key order.

Do not replace the raw hash with the canonical hash. A later incident may depend on exact bytes. Exact bytes include spaces the model actually received.

Canonical form here is compact JSON with sorted keys. It is not a universal JSON standard for every payload. Dates, large integers, and duplicate keys need local rules.

Five-step debug loop

Use this loop on one tool_call_id from the failing turn. Do not start a retry before step four finishes. A second run can replace the only clean copy.

Copy the failing tool-call id from the model turn.
Save process stdout bytes under that same id.
Save adapter output bytes under that same id.
Save the tool-result slice from the outbound model request.
Run classify and append the row to the trace log.

A shell sketch for those files looks like the block below. The flags are a proposed interface for a later script. The teaching script still uses in-memory fixtures only.

python3 three_boundary.py \
  --process traces/call_91.process.bin \
  --adapter traces/call_91.adapter.bin \
  --model traces/call_91.model_part.bin \
  >> traces/mutation_hops.jsonl

After the row exists, route the next owner from the label. adapter_shrink goes to the adapter author for review. prompt_builder_wrap goes to the prompt author for review.

no_payload_mutation leaves transport and enters behavior review. Filter the log so that review starts on real mutations.

jq -c 'select(.mutation_hop != "no_payload_mutation")' traces/mutation_hops.jsonl

This filter keeps only rows where a hop changed bytes. Start the review with that shorter list, not the full file.

What equal hashes do not prove

Equal digests do not mean the tool chose the right file. They mean the hashed slices matched each other exactly. A wrong path can be preserved perfectly across every hop.

A model can still misread a faithful tool payload. That case is a behavior question, not a mutation question. Split those questions or the retry will touch the wrong layer.

Do not delete process stdout after a green digest row. The digest is an index, and the bytes are the evidence. Keep both artifacts until the incident review is closed.

Share digest rows, keep payloads local

Raw tool output often contains source code or secrets. A digest row usually does not contain those secrets. You can share the row more safely than the raw file.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode provides free model access and a free server option. Both points are operator-supplied for this draft only.

This piece states no quota, model name, or machine size. It also states no end date for either availability claim. Leave process files on the tool host that created them.

A teammate can group mutation_hop without opening file contents. Use free model access for one controlled replay check. Send the saved tool-result slice, not a fresh tool run.

Ask whether that exact slice still looks unreadable. If the replay fails, the saved slice reproduces the complaint. If the replay succeeds, the live request differed from the slice.

That check is a workflow suggestion, not a quality score. No accuracy rate is reported anywhere in this article. Policy still decides whether metadata may leave the network.

Store a redaction receipt

Upload nothing until local redaction finishes on the host. A hash of a secret can still be sensitive here. A small private corpus can make even a digest identifying.

Prefer hashing the payload only after replacement finishes. Store a receipt next to the digest row in the log.

Record field_path for each replaced value in the payload.
Record rule_id so later policy changes stay visible.
Record original_len counted only on the tool host.
Record the replacement token, never the original characters.

If rule_id changes between two runs, mark policy drift. Do not call that drift a model regression by default. The hop table cannot see a rule change without the receipt.

Who should skip this loop

Skip the loop when the tool returns no response body. An exit-code-only tool needs a different record shape. Three empty hashes will only waste the review hour.

Skip it when you cannot capture the outbound model request. Token totals on a dashboard cannot fill model_sha256. You would be guessing the slice instead of hashing it.

Skip a shared free server when metadata must stay inside. Digests plus tool names can still identify a private file. A small corpus makes that identification risk concrete.

Skip this loop as a model leaderboard or vendor score. The classifier locates a hop inside one recorded call. It does not rank coding models against each other.

Limits to keep on the page

The classifier ignores streaming chunk boundaries in this sketch. It also ignores compression and content-encoding differences. It assumes the process hash is taken on raw bytes.

canonicalize will change hashes on purpose in this sketch. If key order is meaningful, skip that function for comparison. Keep the canonical hash only as a second stored column.

Clock skew stays out of scope for this method. This method never reads timestamps from either host. A slow but correct hop will look healthy here.

No fixture in this article ran on a hosted agent. Treat the printed labels as expected results to confirm. If your runtime adds a byte-order mark, hashes will differ.

Streaming tools

Streaming tools need one extra rule before classification. Hash the concatenated process output after the stream closes. A mid-stream hash will not match the final adapter hash.

That false adapter_grow label sends you to the wrong owner. Wait for stream close, then hash, then classify. Do not classify a partial buffer during the call.

Close the record before any retry

Select one failed tool-call id from the latest trace. Write the three slices to disk under that id. Run the classifier and store mutation_hop before any retry.

If the team needs a shared digest log, a free server can hold the rows. A MonkeyCode free server is one place for those digest rows. Keep raw payloads on the tool host unless policy allows the copy.

Retry Wrote Twice. Store commit_state Before the Next Attempt.

Riley Wang — Sun, 11 Oct 2026 21:14:40 +0000

A checkout agent called a charge tool at 14:02:11 UTC. The handler returned HTTP 200 before the runner woke. The local runner then hit its eight second timeout.

The span was marked failed in the runner log. A retry policy launched attempt two for the same cart. The payment ledger later showed two successful captures.

The agent summary still reported an overall success. This note reconstructs that case as a teaching fixture. The figures are illustrative, not a production measurement.

Two clocks, one missing fact

Recent posts ask when an automatic retry is justified. This rule answers that only for tools that write. Better model scores do not close a blind retry hole.

What the diff missed

Attempt one reached the payment API before timeout. The response body never landed in the local runner. Attempt two opened a fresh span for the same cart.

Nothing linked those spans to one side effect. A later diff of tool names looked entirely clean. Both calls were stored under the name charge_cart.

Why a status code is not enough

Attempt one had no exit code in the trace file. Attempt two stored exit code 0 and a short preview. A green second attempt does not prove the first was empty.

You need a commit fact, not only a status code. Timeout means the runner stopped waiting for bytes. It does not mean the remote handler skipped the write.

Fields to store on every tool span

Keep the trace fields your runner already persists today. Add four fields before any automatic retry is enabled.

Required additions

Store attempt_id as an integer starting at one.
Store idempotency_key for the side effect, not the span.
Store commit_state as unknown, committed, rejected, or not_sent.
Store retry_reason as empty on attempt one only.

Also store parent_run_id so retries group under one turn. Do not reuse the span id across later attempts. A new attempt is a new span with the same key.

How to read commit_state

The value unknown means the runner lost the response. It does not mean the remote tool was safe to repeat. The value not_sent means failure happened before the socket write.

Only not_sent is a clean retry candidate in this policy. The value committed means the side effect already landed. The value rejected means the server refused that write.

Decision table

Use this table before the runner schedules another attempt. The checker script below encodes these same stop rules.

prior commit_state	next attempt	action	reason
not_sent	allowed	retry	no request left the process
rejected	blocked	stop	server refused the write
committed	blocked	stop	side effect already landed
unknown on attempt 1	blocked until read	reconcile	remote may have committed
unknown on attempt 2+	blocked	stop	another try risks a double write
missing	blocked	stop	a retry without this field is blind

Reconcile means a ledger read using the idempotency key. Do not call the charge tool again until that read returns. If that read fails, keep the state unknown and stop.

Worked numbers from the reconstruction

Attempt one lasted 8000 ms and then timed out. The payment API log showed a capture at 8012 ms. The runner never stored a response body for that call.

Attempt two started 1400 ms after the local timeout. It stored exit code 0 plus a truncated preview string. The ledger count for cart c-1844 then became two.

A correct trace would have blocked that second attempt. Those timestamps are a teaching fixture for this note. Replace them with spans exported from your own runner.

Do not treat 8012 ms as a benchmark for any host. Do not treat the cart id as data from a real shop. The point is the missing state, not the sample latency.

Checker script

The following script is a proposal you can copy. It has not been executed against a live agent fleet. Save it locally as check_retries.py before you run it.

It reads JSONL with one tool span on each line. Lines with a blank body are skipped by the loader. A bad state value is reported as a hard failure.

#!/usr/bin/env python3
'''Flag unsafe retries. Proposal code, not a live-fleet result.'''

import json
import sys
from collections import defaultdict

ALLOWED = {'unknown', 'committed', 'rejected', 'not_sent'}

def load(path):
    rows = []
    with open(path, encoding='utf-8') as handle:
        for line_no, line in enumerate(handle, start=1):
            text = line.strip()
            if not text:
                continue
            row = json.loads(text)
            row['_line'] = line_no
            rows.append(row)
    return rows

def check(rows):
    grouped = defaultdict(list)
    errors = []
    for row in rows:
        key = row.get('idempotency_key')
        state = row.get('commit_state')
        attempt = row.get('attempt_id')
        if not key or attempt is None or state not in ALLOWED:
            errors.append(
                'line %s: missing key, attempt_id, or commit_state' % row['_line']
            )
            continue
        grouped[key].append(row)

    for key, spans in grouped.items():
        ordered = sorted(spans, key=lambda item: item['attempt_id'])
        seen = [item['attempt_id'] for item in ordered]
        expected = list(range(1, len(seen) + 1))
        if seen != expected:
            errors.append('%s: attempt_id sequence is %s' % (key, seen))
        for prev, curr in zip(ordered, ordered[1:]):
            prior = prev['commit_state']
            nxt = curr['attempt_id']
            if prior != 'not_sent':
                errors.append('%s: attempt %s follows %s' % (key, nxt, prior))
            reason = curr.get('retry_reason')
            if reason in (None, ''):
                errors.append(
                    '%s: attempt %s has empty retry_reason' % (key, nxt)
                )
    return errors

def main():
    path = sys.argv[1] if len(sys.argv) > 1 else 'spans.jsonl'
    errors = check(load(path))
    if errors:
        print('FAIL %s' % len(errors))
        for item in errors:
            print(item)
        return 1
    print('PASS 0')
    return 0

if __name__ == '__main__':
    raise SystemExit(main())

Fixture and commands

Fixture file

Create spans.jsonl with the two attempts shown below. The first commit_state is unknown on purpose here. The second attempt should fail the checker immediately.

{"parent_run_id":"run-9","attempt_id":1,"idempotency_key":"cart-c-1844","commit_state":"unknown","retry_reason":"","tool":"charge_cart"}
{"parent_run_id":"run-9","attempt_id":2,"idempotency_key":"cart-c-1844","commit_state":"committed","retry_reason":"timeout","tool":"charge_cart"}

Commands and expected results

Run the checker from that same working directory. Pass the fixture path as the first argument. Record both the printed status and the exit code.

python3 check_retries.py spans.jsonl
echo exit=$?

The expected result on this fixture is FAIL. The process exit code should be non-zero as well. The failure text should name the key cart-c-1844.

It should also name the prior state unknown. That pair is the bug this policy is meant to catch. A passing file needs attempt one marked not_sent.

You can also pass a file that has one line. If the shell mangles quotes, write the file by hand. The rule under test stays the same either way.

Debug loop you can reuse

Export one failed user turn as a JSONL file. Group those lines by idempotency key, not by tool name. Run check_retries.py and keep its exit code in the job log.

If the result is FAIL, reconcile the ledger first. Do that work before you spend another model call. If the result is PASS, rerun on a fresh input only.

Do not start this loop by swapping the model. A different model will not repair a missing commit fact. Fix the trace schema, then compare the two runs.

A cheap place to rerun the loop

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode matters here only as a second runner. The operator supplied a claim of free model access.

A free server option is the other supplied claim. An outreach note mentioned a token allotment, but no primary source was attached. This draft therefore quotes no token number at all.

It also states no hardware size and no offer duration. Those details change, so confirm them on the current page. Use that server as a second runner, not as truth.

Keep the JSONL file on your side as the source. Copy the same parent_run_id into both runner logs. Then diff commit fields, not the prose summary text.

If the free server drops a field, fail the check. A missing commit_state is a stop, not a pass. The checker does not call any vendor API itself.

It only reads the trace file path you pass in. A free tier can throttle or sleep without notice here. This article has no evidence about how long access lasts.

Network delay can widen the timeout window further. That delay makes unknown more common, not more rare. Treat unknown as a stop until a ledger read returns.

What is not claimed

No latency number in this note is a host benchmark. No pass from the script proves the payment API. Confirm current terms before you depend on free access.

If you need a second runner, the free server option can host that pass. Keep the JSONL local and let the checker fail closed.

Who should not use this policy

Skip this policy when the tool is a pure read. A catalog lookup can be retried with less risk. Skip it when the handler already returns a prior receipt.

In that case, store the receipt id in the span. Skip it for writes that cannot carry a stable key. A keyless write cannot be reconciled by this script.

Also skip it if you cannot persist JSONL across timeout. The checker is useless on a preview string alone. Previews drop bytes, and this script needs the raw line.

What this does not prove

The script does not prove the payment API is correct. It only shows that the trace forbids a blind retry. It does not hash the tool catalog before replay.

It does not record argv or a process exit code. Those checks still matter, and they belong in other notes. It does not measure model quality in any way.

A careful model can still time out on a slow tool. Clock skew can make recorded durations look simply wrong. Store both runner time and tool time when you have them.

Do not sort retries by wall clock alone. Sort retry spans by attempt_id before you judge them. If the model writes commit_state, do not trust that field.

The runner or the tool adapter should set that field. A model-filled state is a claim, not a commit record.

Close

A retry is a new span with an old side effect. Store attempt_id and idempotency_key before the policy fires. Store commit_state and retry_reason on that same span.

Run the checker and keep the printed exit code. Stop when the prior state is unknown or committed. Also stop when the prior state is rejected.

That loop stays useful without any hosted runner. The file, the key, and the state are the whole method. Ship the checker beside the agent, not after the incident.

Remote 200, Local Exit 1. Store the Turn Id on Both Sides.

Riley Wang — Sat, 10 Oct 2026 17:39:59 +0000

A coding agent writes a three-file patch during a long run. The remote inference log records HTTP 200 for that turn. The local tool span records exit code 1 for the patch.

The agent summary still reports that the edit landed cleanly. No shared identifier connects those three records together. The failure sits in the gap between the two logs.

Recent community posts spend their energy on model choice. This note does not rank models or repeat those arguments. It asks whether both logs can name the same attempt.

Two clocks, two id spaces

Local traces usually name tools, arguments, and exit codes. Remote logs usually name model calls, status, and latency. Each side keeps its own clock and its own identifier space.

A retry writes a second model call under a new request id. The tool span can still point at the first attempt only. A file diff will not show that the join itself broke.

This split appears on long agent runs with several tools. It appears again when the runner and the server sit far apart. Delay makes the timelines look unrelated even when they match.

What you can reuse

This note gives you a join record and a small checker. It also gives you four mismatch classes and a debug loop. The code is a proposed example and was not executed here.

It is not a benchmark of any model or any server. It does not quote a token allowance or a hardware spec. Treat fixture counts as labels for the method, not as measurements.

Fields that must travel together

Store one join row for every model turn and every tool span.

trace_id identifies the whole agent run from start to finish.
turn_id identifies one model request and the matching response.
tool_call_id identifies one tool call, or stays null on model rows.
parent_turn_id names the model turn that requested that tool.
attempt is an integer that starts at 1 for each logical turn.
side is either model or tool, and no third value is valid.
server_request_id stores the remote id, or null when that id is absent.
status is a short code such as ok or error.
local_mono_ns stores local monotonic time, counted in nanoseconds.

Do not store raw prompts or tool output in this join file. Store a content hash when you need an equality check later. Keep secrets out of the file before you export it anywhere.

Validate before you join

A checker that guesses parents will hide emitter bugs. Fail closed when a required field is missing or contradictory. Do not repair a row by matching the nearest timestamp.

# Proposed example. Not executed in this draft.
from dataclasses import dataclass

@dataclass(frozen=True)
class JoinRow:
    trace_id: str
    turn_id: str
    tool_call_id: str | None
    parent_turn_id: str | None
    attempt: int
    side: str
    server_request_id: str | None
    status: str
    local_mono_ns: int

def validate(row: JoinRow) -> list[str]:
    errors: list[str] = []
    if not row.trace_id or not row.turn_id:
        errors.append("missing_ids")
    if row.attempt < 1:
        errors.append("bad_attempt")
    if row.side not in {"model", "tool"}:
        errors.append("bad_side")
    if row.side == "tool" and not row.parent_turn_id:
        errors.append("tool_without_parent_turn")
    if row.side == "tool" and not row.tool_call_id:
        errors.append("tool_without_tool_id")
    if row.side == "model" and row.tool_call_id is not None:
        errors.append("model_row_has_tool_id")
    if row.local_mono_ns < 0:
        errors.append("bad_clock")
    return errors

Run validate on every row before the join step starts. Stop the report when any row returns a schema error. A partial join over dirty input looks precise and is not.

Classify the gap

Index model rows by trace id, turn id, and attempt. Look up each tool span by its parent turn id. Return one class string, and do not attach a score.

# Proposed example. Not executed in this draft.
def index_models(rows: list[JoinRow]) -> dict[tuple[str, str, int], JoinRow]:
    indexed = {}
    for row in rows:
        if row.side != "model":
            continue
        indexed[(row.trace_id, row.turn_id, row.attempt)] = row
    return indexed

def classify(tool: JoinRow, models: dict) -> str:
    if tool.side != "tool":
        return "not_a_tool_row"
    if not tool.parent_turn_id:
        return "orphan_tool"
    key = (tool.trace_id, tool.parent_turn_id, tool.attempt)
    parent = models.get(key)
    if parent is None:
        return "missing_model_row"
    if parent.server_request_id and tool.server_request_id:
        if parent.server_request_id != tool.server_request_id:
            return "request_id_mismatch"
    if parent.status != "ok" and tool.status == "ok":
        return "tool_ok_after_model_error"
    return "joined"

Empty parent ids should die in validation, not in classify. The orphan class remains for checkers that skipped that gate. The checker does not rank models and does not score patches.

It only reports whether the two logs agree on identity. A joined pair can still contain a wrong edit. Agreement is a precondition, not a verdict on the patch.

Read the four classes

Class	Local evidence	Remote evidence	First check
`orphan_tool`	tool span, empty parent	nothing required	validation should already have failed
`missing_model_row`	parent id is set	no matching model row	dropped log, wrong id, or sampling
`request_id_mismatch`	server id A on the tool	server id B on the model	a retry wrote a new call
`tool_ok_after_model_error`	tool status is `ok`	model status is not `ok`	summary trusted the wrong side

Read the table from the class column toward the check column. Fix the emitter before you rewrite the prompt or the tool. A missing remote row is not proof that the call never ran.

Fixture, not a measurement

Use a tiny fixture so the four classes stay reviewable. Label it as synthetic data inside the test file itself. Do not paste these counts into a status report as results.

Build two model rows with attempts 1 and 2 under one trace.
Build three valid tool rows: one mismatch, one missing parent, one join.
Expect request_id_mismatch, missing_model_row, and joined.
Expect zero schema errors, and fail the test if any appear.

# Proposed fixture. Not executed in this draft.
def test_fixture_classes() -> None:
    rows = [
        JoinRow("run_1844", "turn_a", None, None, 1, "model", "req_1", "ok", 10),
        JoinRow("run_1844", "turn_a", None, None, 2, "model", "req_2", "ok", 20),
        JoinRow("run_1844", "tool_row_1", "tool_1", "turn_missing", 1, "tool", "req_x", "error", 30),
        JoinRow("run_1844", "tool_row_2", "tool_2", "turn_a", 1, "tool", "req_9", "ok", 12),
        JoinRow("run_1844", "tool_row_3", "tool_3", "turn_a", 2, "tool", "req_2", "ok", 21),
    ]
    assert all(validate(row) == [] for row in rows)
    models = index_models(rows)
    got = [classify(row, models) for row in rows if row.side == "tool"]
    assert sorted(got) == ["joined", "missing_model_row", "request_id_mismatch"]

The fixture expects those mismatch classes to appear. The gate command below is for a later run, after the emitter fix. Do not point the gate at this fixture and call the failure a product bug.

# Proposed commands. Paths are examples only.
python join_check.py --local runs/trace.jsonl --remote runs/inference.jsonl --out runs/join-report.json
python join_check.py --local runs/trace.jsonl --remote runs/inference.jsonl --expect-zero orphan_tool,request_id_mismatch

The second command is a regression gate for the emitter. It should fail while those two mismatch classes still appear. Keep missing model rows out of that gate until retention is known.

Report shape

Write the checker output as a small JSON object. Mark synthetic runs so nobody quotes them as production data. Include the input file hash so a later replay can detect edits.

{
  "trace_id": "run_1844",
  "fixture": true,
  "schema_errors": 0,
  "input_sha256": "replace-with-real-hash",
  "counts": {
    "joined": 1,
    "orphan_tool": 0,
    "missing_model_row": 1,
    "request_id_mismatch": 1,
    "tool_ok_after_model_error": 0
  }
}

Six steps you can rerun

Export local tool spans to one JSONL file you can hash.
Export remote inference rows that carry the same turn id.
Validate every row, and stop on the first schema error.
Classify each tool span, then write class counts to a new file.
Open only the mismatch classes, and leave joined pairs for later.
Patch the emitter, replay the same inputs, and confirm the drop.

Keep both input files immutable during that replay. Write each report to a new path with the trace hash in the name. That habit stops a later edit from rewriting the evidence.

Free server, same join contract

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The operator describes MonkeyCode as an open-source agent project. The same brief says free model access and a free server option exist. This draft does not state a token cap or a model list.

It also omits machine size and any uptime promise. Those access details change, and stale numbers mislead planning. Confirm the current limits in the project documentation before a long run.

Do not treat a blog post, including this one, as the pricing source. A free server helps this loop in one narrow way. You can place the agent loop on that server instead of the laptop.

You still keep the local join file under your control. That file remains the record you hold when remote retention is short. Free access does not remove the two-log problem.

A remote log can rotate before you export it. A rate limit can create attempt 2 with no matching tool span. Record attempt on both sides or you will join the wrong call.

Echo the server request id into the local row when one is returned. If the server cannot echo the turn id, stop and fix that contract. Use the free option as a second environment, not as an oracle.

Run the same schema in both places and diff the class counts. Diff those class-count reports, not the product pages. A lower count means the emitter improved, not that the model improved.

Limits you should keep visible

If neither side logged the call, this checker cannot recover it.
This checker does not prove the patch contents or the exit result.
Use clocks as diagnostic notes, and never join rows by time.
A short prompt hash can collide across different inputs.
Hash the full canonical input, or skip the hash field entirely.
A null server request id means the value was not captured.
That null does not mean the call stayed on the local machine.
Sampled remote logs create false missing-model-row hits.
Mark sampling in the report header before anyone triages them.

Do not upload raw prompts just to fill an empty join field. Redact secrets before any export to a shared or free server. If redaction is unclear, keep the join file on the runner.

Who should skip this method

Skip it when one process already writes one complete log. Skip it when you cannot change the emitter that mints ids. Skip it when the question is model quality rather than trace integrity.

A fixed task harness answers quality questions better than this checker. This checker answers whether the two logs describe the same attempt. Mixing those questions produces a confident report about the wrong fault.

Also skip the remote half when the server log is heavily sampled. Say that in the report so a reader does not chase empty rows. Local validation can still run alone in that sampled case.

One pass on a failed trace

Pick one failed run that already has a local JSONL trace. Add the turn id and the parent turn id at the emitter. Classify once, and keep that report beside the trace as a baseline.

If you use MonkeyCode's described free server option, copy the request id. Put that id into the same local join row before you export. Check current access limits in the project docs before the second run.

Repeat the six steps in that second environment. Stop when the gated classes hit zero, then reread the patch. The join is ready only when the same attempt is named on both sides.

Name the Span That Spent the Agent Budget

Riley Wang — Thu, 08 Oct 2026 09:33:57 +0000

A reviewer opens a failed agent run at 02:14.
The status line says the model call was rejected.
Tool six never returned a file diff at all.

The log stores one error string and nothing else.
It lacks a split of input tokens and output tokens.
Nobody can name the span that spent the budget.

The hole that wastes the next hour

Failed agent runs often keep the final error string only.
They omit the running token total at each step.
A retry loop then looks like one red step.

A large file read can look like a planner defect.
A planner defect can look like a quota failure.
Those faults need different fixes and different owners.

This note adds one record type to the trace.
Each model span and each tool span writes one row.
You join those rows by parent span id.

The sample below is an unexecuted local example.
Run it on your machine before you trust the totals.
The fixture numbers are not a hosted model benchmark.

What the ledger must hold

Store one ledger row for every span in the run.
Keep the field set small so writers do not skip rows.
Add a run header before the first span row.

Header fields:

run_id: stable id for this attempt
unit: input_plus_output or the vendor unit name
local_soft_limit: your ceiling, not a vendor promise
local_hard_limit: stop line for this process
prompt_storage: hash_only in this design

Row fields:

span_id: unique inside the run
parent_id: caller id, empty at the root
seq: integer order inside the run
kind: model or tool
name: model call label or tool name
input_tokens: integer, or null if unknown
output_tokens: integer, or null if unknown
running_total: sum of known tokens so far
status: ok, error, budget_warn, or budget_stop
ts_mono_ms: monotonic milliseconds inside one process

Do not store raw prompts in this ledger file.
Prompts can hold secrets and entire file bodies.
Keep a prompt hash only if you already hash prompts.

Unknown usage must stay null, not zero.
Zero means the host reported a measured zero.
Null means the host did not report usage at all.

A recorder you can execute locally

This script writes JSONL and never opens a socket.
It is a proposal you should run before adoption.
Replace the fixture limits with your real local ceiling.

#!/usr/bin/env python3
"""Unexecuted example: per-span token ledger. Not a benchmark."""

import json
from pathlib import Path

class TokenLedger:
    def __init__(self, path, run_id, soft_limit, hard_limit):
        self.path = Path(path)
        self.run_id = run_id
        self.soft_limit = soft_limit
        self.hard_limit = hard_limit
        self.total = 0
        self.seq = 0
        self.rows = []
        header = {
            "run_id": run_id,
            "unit": "input_plus_output",
            "local_soft_limit": soft_limit,
            "local_hard_limit": hard_limit,
            "prompt_storage": "hash_only",
        }
        self.path.write_text(
            json.dumps({"header": header}) + "\n",
            encoding="utf-8",
        )

    def add(self, span_id, parent_id, kind, name,
            input_tokens, output_tokens, status):
        self.seq += 1
        known = 0
        for value in (input_tokens, output_tokens):
            if value is not None:
                known += int(value)
        self.total += known
        if self.total >= self.hard_limit and status == "ok":
            status = "budget_stop"
        elif self.total >= self.soft_limit and status == "ok":
            status = "budget_warn"
        row = {
            "run_id": self.run_id,
            "span_id": span_id,
            "parent_id": parent_id,
            "seq": self.seq,
            "kind": kind,
            "name": name,
            "input_tokens": input_tokens,
            "output_tokens": output_tokens,
            "running_total": self.total,
            "status": status,
        }
        self.rows.append(row)
        with self.path.open("a", encoding="utf-8") as handle:
            handle.write(json.dumps(row) + "\n")
        return row

    def by_parent(self):
        grouped = {}
        for row in self.rows:
            key = row["parent_id"] or "ROOT"
            spent = 0
            for value in (row["input_tokens"], row["output_tokens"]):
                if value is not None:
                    spent += int(value)
            grouped[key] = grouped.get(key, 0) + spent
        ranked = sorted(
            grouped.items(),
            key=lambda item: item[1],
            reverse=True,
        )
        return ranked

if __name__ == "__main__":
    ledger = TokenLedger("usage.jsonl", "run-demo", 800, 1000)
    ledger.add("s1", "", "model", "plan", 120, 40, "ok")
    ledger.add("s2", "s1", "tool", "read_file", 0, 0, "ok")
    ledger.add("s3", "s1", "model", "patch", 700, 180, "ok")
    print(json.dumps(ledger.by_parent()))

The demo parent s1 collects the child spend.
The patch call crosses the hard fixture limit.
Its status becomes budget_stop inside the local writer.

Those token counts are fixtures for the assert.
They do not describe any live vendor model.

Checks that must pass before blame

File assert

Run this consistency check after every recorded attempt.
It refuses a file whose running total drifts.

python3 - << 'PY'
import json
total = 0
prev = 0
seen = set()
for line in open("usage.jsonl", encoding="utf-8"):
    obj = json.loads(line)
    if "header" in obj:
        assert obj["header"]["unit"] == "input_plus_output"
        continue
    spent = 0
    for key in ("input_tokens", "output_tokens"):
        if obj[key] is not None:
            spent += obj[key]
    total += spent
    assert obj["running_total"] == total, obj["span_id"]
    assert obj["seq"] == prev + 1
    assert obj["span_id"] not in seen
    seen.add(obj["span_id"])
    prev = obj["seq"]
print("rows", prev, "known_tokens", total)
PY

If an assert fires, stop and fix the writer.
Do not open the model prompt until the file is consistent.
A drifted total will point you at the wrong span.

Join rules

Four join rules keep the file boring and useful.
Apply them before anyone debates model quality.

Each span_id appears once inside the run.
Each non-root parent_id matches an earlier span.
running_total equals prior known spend plus this row.
A budget_stop row is the last row in the file.

A later tool row after budget_stop is a writer bug.
The agent should have halted before that tool started.
Record the bug, then fix the stop path first.

Debug loop for a budget death

Five steps

Use the same five steps whenever a run dies on tokens.
Do not skip ahead to a prompt edit.

Read the last row and note status, name, and total.
Group known spend by parent_id and rank the parents.
Open the top parent and list each child name.
Mark the child with the largest known sum as suspect.
Replay only that child with a smaller input body.

One change per replay

Change only one variable on each replay pass.
Keep the tool set fixed while you shrink the input.
Then keep the input fixed while you cap retries.

If the new total stays under your local hard limit, stop.
You found a spend path, not a model quality verdict.
Quality checks belong in a separate harness and note.

How to read the stop table

Signal	Likely cause	Next action
Last row is `budget_stop` on a model span	That call or its parent loop hit the local cap	Shrink the input or cap the loop
Many model rows share one parent id	Retry storm or planner loop	Set a retry cap before the next run
Tool rows show zero, then one huge model row	Tool output was copied into the next prompt	Truncate tool output before the model call
Total jumps while `seq` has gaps	The writer dropped rows	Repair the writer, then rerun the same task
Vendor error says quota but local total is low	Units or hidden host tokens disagree	Reconcile units before you blame the prompt

Pick one row in that table per incident.
Do not apply every action in a single replay.
Mixed changes hide the span that actually spent the budget.

Shared traces and free model access

Disclosure: This article was prepared as part of MonkeyCode's product outreach.
The operator describes MonkeyCode as an open-source project.
The same briefing cites free model access and a free server option.

This note states no token quota, model list, hardware size, or expiry.
Those details change, and no primary source is attached here.
Check the project page on the day you plan a run.

A free server helps only as shared storage for redacted traces.
Upload the usage file and the run header only.
Leave raw prompts and raw tool bodies on the machine that created them.

Teammates need the span name and the running total.
They do not need secrets that rode along in a file read.
Strip those fields before the file leaves the laptop.

The debug loop itself runs without a hosted model.
Prove the join asserts on fixture rows first.
Point a live call at the ledger only after those asserts pass.

Limits that stay in force

This ledger cannot see billing the host refuses to return.
A null usage field is a gap, not a measured zero.
Say that gap in the incident note so others do not invent a number.

Parent joins fail when the runtime omits parent ids.
Fall back to seq order and stop drawing a tree.
A flat list is weaker, and it is still honest.

Monotonic time is valid inside one process only.
Do not sort rows from two hosts by wall clock.
Clock skew will invent overlaps that never happened.

The soft limit and hard limit in the sample are fixtures.
Replace both with the ceiling you chose for this run.
If the vendor has not published a cap, label your cap as local.

Treat this file as a debug aid only.
It is not an invoice and not a legal usage record.
Do not cite it as billed spend in a customer report.

Who should skip the extra file

Skip the ledger if your runtime already exports billed usage per span.
A second counter will drift and start avoidable arguments.
Use the vendor export and document its unit in the run header.

Skip the upload if you cannot redact tool output first.
A shared server is the wrong store for credentials and private source.
Keep those runs on disk with restricted permissions.

Skip this design if you need finance-grade metering.
Local JSONL can drop rows when a process dies hard.
That gap is acceptable for debugging and unacceptable for billing.

One next run

Add the four join asserts to the last failed attempt.
Write the ledger beside the trace you already store.
If your team has a free shared server, upload the redacted ledger only.

Confirm the live access limits on the project page that same day.
Then replay one suspect span, and leave the rest of the agent untouched.

Tool Previews Lie. Store stdout_bytes and a Truncation Flag.

Riley Wang — Wed, 23 Sep 2026 16:29:07 +0000

A coding agent closed the flaky test ticket.
The JSONL span showed pytest and a short preview.
The next human run failed in the same module.

The wrapper had capped stdout at 8192 bytes.
The assertion text started after that cap.
The model never read the failing lines.

This is not a prompt problem first.
It is missing I/O metadata on the tool span.

The failure mode

Most agent traces store a short text preview.
They also store a loose success flag.
That pair cannot describe a clipped pipe.

Three wrappers hide the real tool result:

Head-cap keeps only the first N bytes.
Tail-cap keeps only the last N bytes.
Pipe-stall drops the stream and stores empty text.

The model then plans from a partial document.
Later debug sessions trust that partial document.
Teams retune temperature while the bytes were absent.

What typical spans already contain

A typical tool span already has a name.
It often includes arguments and a duration.
Some collectors also keep an exit code.

Those fields still miss clipped pipes.
Exit code 0 can sit beside truncated stdout.
Exit code 1 can sit beside truncated stderr.
Both streams can be empty after a full pipe.

Fields to add on every tool span

Record these keys on each tool span:

stdout_bytes: length before any wrapper cap
stderr_bytes: length before any wrapper cap
stdout_cap: wrapper limit in bytes, or null
stderr_cap: wrapper limit in bytes, or null
stdout_truncated: boolean after the cap check
stderr_truncated: boolean after the cap check
stdout_sha256: full-stream hash when unclipped
stderr_sha256: full-stream hash when unclipped
stdout_head_sha256 / stdout_tail_sha256: clipped case
encoding: utf-8 or binary

Keep the stored preview short on purpose.
Store hashes for bytes you refused to keep.
Never treat the preview as the full stream.

Proposed JSONL schema

The next record is a proposed schema only.
It is not sampled from a production fleet.
Use it as a contract for local collectors.

{
  "span_type": "tool",
  "tool_name": "pytest",
  "cwd": "/work/repo",
  "preview_stdout": "===== test session starts =====\n",
  "preview_stderr": "",
  "stdout_bytes": 24118,
  "stderr_bytes": 902,
  "stdout_cap": 8192,
  "stderr_cap": 8192,
  "stdout_truncated": true,
  "stderr_truncated": false,
  "stdout_sha256": null,
  "stderr_sha256": "b3a1...",
  "stdout_head_sha256": "9c21...",
  "stdout_tail_sha256": "44de...",
  "encoding": "utf-8",
  "ok_preview": true
}

ok_preview is what the agent saw.
It is not a claim about the full process.
Lint that distinction before you debug policy.

A local subprocess wrapper

The wrapper below is a labeled local example.
Run it against one command on your machine.
It is not a production sandbox or jail.

#!/usr/bin/env python3
"""Record truncation metadata around one tool command."""
from __future__ import annotations

import hashlib
import json
import subprocess
import sys
from pathlib import Path

CAP = 8192  # example cap; set from your collector
PREVIEW = 512


def sha256_bytes(data: bytes) -> str:
    return hashlib.sha256(data).hexdigest()


def clip_meta(data: bytes, cap: int) -> dict:
    truncated = len(data) > cap
    kept = data[:cap] if truncated else data
    meta = {
        "bytes": len(data),
        "cap": cap,
        "truncated": truncated,
        "sha256": None if truncated else sha256_bytes(data),
        "head_sha256": sha256_bytes(data[: min(len(data), 256)]),
        "tail_sha256": sha256_bytes(data[-min(len(data), 256) :]) if data else None,
        "preview": kept[:PREVIEW].decode("utf-8", "replace"),
        "encoding": "utf-8",
    }
    return meta


def run_tool(argv: list[str], cwd: str) -> dict:
    proc = subprocess.run(
        argv,
        cwd=cwd,
        stdout=subprocess.PIPE,
        stderr=subprocess.PIPE,
        check=False,
    )
    out = clip_meta(proc.stdout, CAP)
    err = clip_meta(proc.stderr, CAP)
    return {
        "span_type": "tool",
        "argv": argv,
        "cwd": str(Path(cwd).resolve()),
        "returncode": proc.returncode,
        "stdout_bytes": out["bytes"],
        "stderr_bytes": err["bytes"],
        "stdout_cap": out["cap"],
        "stderr_cap": err["cap"],
        "stdout_truncated": out["truncated"],
        "stderr_truncated": err["truncated"],
        "stdout_sha256": out["sha256"],
        "stderr_sha256": err["sha256"],
        "stdout_head_sha256": out["head_sha256"],
        "stdout_tail_sha256": out["tail_sha256"],
        "preview_stdout": out["preview"],
        "preview_stderr": err["preview"],
        "encoding": "utf-8",
        "ok_preview": proc.returncode == 0 and not out["truncated"] and not err["truncated"],
    }


if __name__ == "__main__":
    if len(sys.argv) < 3:
        print("usage: wrap_tool.py CWD CMD [ARGS...]", file=sys.stderr)
        sys.exit(2)
    record = run_tool(sys.argv[2:], sys.argv[1])
    print(json.dumps(record, ensure_ascii=False))

Example command:

python3 wrap_tool.py "$PWD" pytest -q tests/test_billing.py

Redirect that JSON line into run.jsonl.
Keep one file per agent run, not per week.
Do not append unrelated jobs into the same file.

A truncation linter

Feed traces into this checker next.
It fails when flags disagree with byte counts.
It also fails when a preview claims success after a clip.

#!/usr/bin/env python3
"""Lint tool spans for truncation contract breaks."""
from __future__ import annotations

import json
import sys
from pathlib import Path

REQUIRED = (
    "stdout_bytes",
    "stderr_bytes",
    "stdout_cap",
    "stderr_cap",
    "stdout_truncated",
    "stderr_truncated",
)


def problems(span: dict, line_no: int) -> list[str]:
    issues: list[str] = []
    if span.get("span_type") != "tool":
        return issues
    missing = [key for key in REQUIRED if key not in span]
    if missing:
        issues.append(f"L{line_no}: missing {missing}")
        return issues
    for stream in ("stdout", "stderr"):
        n = span[f"{stream}_bytes"]
        cap = span[f"{stream}_cap"]
        flag = span[f"{stream}_truncated"]
        if cap is not None and n > cap and flag is not True:
            issues.append(f"L{line_no}: {stream} clipped without flag")
        if cap is not None and n <= cap and flag is True:
            issues.append(f"L{line_no}: {stream} flag set under cap")
        if flag and span.get(f"{stream}_sha256"):
            issues.append(f"L{line_no}: full {stream} hash on clipped stream")
    if span.get("ok_preview") is True and (
        span["stdout_truncated"] or span["stderr_truncated"]
    ):
        issues.append(f"L{line_no}: ok_preview true while truncated")
    return issues


def lint_path(path: Path) -> int:
    found = 0
    with path.open() as handle:
        for line_no, raw in enumerate(handle, start=1):
            raw = raw.strip()
            if not raw:
                continue
            span = json.loads(raw)
            for item in problems(span, line_no):
                print(item)
                found += 1
    return found


if __name__ == "__main__":
    if len(sys.argv) != 2:
        print("usage: lint_truncation.py run.jsonl", file=sys.stderr)
        sys.exit(2)
    count = lint_path(Path(sys.argv[1]))
    print(f"issues={count}")
    sys.exit(1 if count else 0)

Run it:

python3 lint_truncation.py run.jsonl

A non-zero exit means stop model debugging.
Re-run the clipped tool without a cap first.
Diff that full stderr against the stored preview.

Decision table

Observation	First action	Not the first action
`stderr_truncated=true`, returncode 0	Re-run tool with cap removed	Raise temperature
`stdout_truncated=true` on test output	Fetch tail hash, re-run pytest	Swap the coding model
both byte counts zero, returncode 0	Check pipe stall and ulimit	Add more tool descriptions
preview shows pass, linter fails	Treat span as incomplete	Close the incident
binary `encoding`, hashes only	Skip UTF-8 preview diffs	Parse preview as logs

Use the table during incident review.
Do not turn it into a model rubric.
The artifact is the span contract, not a score.

A debug loop you can reuse

Use this loop on the next opaque agent miss:

Export JSONL for that single run only.
Run lint_truncation.py run.jsonl.
Stop if any span sets a truncation flag.
Re-run that one argv with CAP removed.
Diff new stderr against preview_stderr.
Inspect prompts only after the pipe is honest.

Do not start with a new system prompt.
Do not start with a larger model.
Fix the stream, then re-evaluate the policy.

Where a scratch model and server fit

Batch linting JSONL does not need a GPU.
A small VM can parse traces and re-run tools.
That is the only capacity this method needs.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode provides free model access and a free server option.
Those two are enough to host the linter and cheap re-runs.
The checker does not depend on that product to work.
Remove the product and the span contract still stands.

Limitations

Hashes cannot rebuild bytes you already dropped.
Stdout may contain tokens, paths, and secrets.
Nested shells can apply a second silent cap.
Binary compilers break naive UTF-8 previews.
duration_ms still says nothing about clip point.
Preview diffs fail when logs contain timestamps.

Store redaction rules beside the collector.
Hash first if policy forbids raw command output.
Do not ship full pytest logs to a shared trace bucket.

Who should skip this

Skip this workflow if agents never call tools.
Skip it if your collector already stores full blobs.
Skip it if policy forbids retaining command output.
Skip it for interactive TTY tools with cursor codes.

Close

The model cannot repair bytes it never received.
Record truncation flags before you blame the policy.
Run lint_truncation.py on one failed JSONL file this week.

DEV Community: Riley Wang

The Diff Wasn't the Model: Fingerprint the Environment Before You Replay a Trace

Why replay diffs lie

Fingerprint what the agent actually sees

The artifact: fingerprint.py

Classify a mismatch in one command

A debug loop you can reuse

Decision table

Where a free server fits

Limitations and who should skip this

What to record next

Record Once, Replay Forever: Cassette Tests for Flaky Agent Tool Calls

Live reruns destroy the evidence you need

Freeze the tool boundary, not the model

Cassette format: one JSONL file per run

A replay dispatcher with zero dependencies

Three mismatch signatures, three diagnoses

The debug loop, end to end

Normalize before you digest

What this method cannot do

Run the loop on the cheapest box you have

Ship checklist

Same Call, Same Result: A No-Progress Detector for Agent Tool Loops

Why run-level diffing misses a loop

Give every tool call a signature

Normalise before you hash

Build the progress ledger

Run it on a trace

Break the loop without killing the run

Test the detector with three fixture traces

Where a free tier fits into this loop

Limitations and known false positives

Who should not use this

The reusable part

The Token Leader Is Not the Latency Leader: Build a Span Ledger for Agent Runs

Why time-ordered traces mislead you

The minimum span schema

Artifact 1: the ledger script

Artifact 2: a budget breaker you can test

Artifact 3: the regression set

The debug loop, five steps

Where free access fits, and where it does not

Limitations and who should skip this

What to keep

Scrub the Trace, Keep the Failure: A Redaction Gate You Can Test

The part you cannot lose is the signature

Write the policy as a table, not a regex

The scrubber is about forty lines

Two assertions on every trace

The loop you reuse

Where a free tier actually helps

What this does not solve

Skip this approach if

Sample Runs, Not Spans: Shrink Agent Traces Without Losing Failures

Why span-level sampling fails

Change the unit of sampling

A decision table you can argue about

The sampler, in about forty lines

The one test that matters

Where the pipeline runs

Measure before you trust the policy

Limitations

Who should skip this

Start with one day of traffic

Validate the Trace Before You Trust the Trial: A Conformance Linter for Agent Runs

The trial that looks like a regression

Trace trust comes before trial count

Eight invariants a runnable trace must satisfy

The artifact: a small linter

A fixture that must fail

Lint every trace in the comparison set

A failure taxonomy you can act on

Where free models and a free server fit

Limitations

Who should not use this

The five-minute version

Hash Three Boundaries Before You Blame the Model

The handoff has three owners

Fields worth storing

Sample row

The artifact: `fingerprint.py`