Retries Look Green. Count Attempts on Every Tool Span

The checkout agent returned HTTP 200 to the storefront. Finance later found two captures on one order.

Support exported one successful tool result from logs. The payment processor listed two authorized captures instead.

The span looked healthy because the last attempt won. Earlier attempts still spent tokens and side effects.

Last Success Is Not One Attempt

Many agent runtimes keep only the final tool payload. Retry wrappers often swallow the failed calls underneath.

Logs then show one green span and one result. Reality contains extra network calls and extra token bursts.

This is not an orphan result or missing pairing bug. The final result exists, yet attempts were never counted.

Green status answers a different question than attempt count. Status asks whether the last try worked. Count asks how much work hid beneath that win.

Three Retry Layers, One Log Line

Duplicate charges rarely come from one wrapper. Three layers retry without sharing a counter.

1. Transport libraries retry on reset and timeout.
2. Tool executors retry when JSON arguments fail validation.
3. The agent retries the whole step after a watchdog timeout.

Each layer can emit success on the last try. None of them must emit the earlier tries.

If exporters flatten those layers into one event, storms vanish. Billing systems still see every side effect.

What To Record Per Attempt

Store one event per try, not per finished span. Keep a stable span_id across those numbered tries.

Use these fields on every retry-aware tool event:

• run_id: one agent invocation
• span_id: one logical tool call
• attempt: integer starting at 1
• tool_name: stable tool identifier
• args_hash: SHA-256 of canonical JSON arguments
• idem_hash: SHA-256 of the provider idempotency key
• status: ok, error, or timeout
• t_start and t_end: unix milliseconds
• token_in and token_out: integers or null
• side_effect: none, unknown, or external

Skip raw argument bodies in the shared index. Hash them and keep payloads in a local file.

Why hash arguments

Payment tools often carry account numbers and names. Full payloads do not belong on a shared disk.

A hash still joins retries of the same call. You can group storms without leaking customer PII.

Hash the idempotency key the same way. The raw key still goes to the provider. The trace only needs a joinable digest.

Example Timeline For One Span

The next block is a labeled example, not production evidence. Three attempts share span_19 and one argument hash.

{"event":"tool_attempt","run_id":"run_7f3","span_id":"span_19","attempt":1,"tool_name":"charge_card","args_hash":"a91c9e0c","idem_hash":"e2b1aa44","status":"timeout","t_start":1757385600120,"t_end":1757385604120,"token_in":812,"token_out":0,"side_effect":"unknown"}
{"event":"tool_attempt","run_id":"run_7f3","span_id":"span_19","attempt":2,"tool_name":"charge_card","args_hash":"a91c9e0c","idem_hash":"e2b1aa44","status":"error","t_start":1757385604200,"t_end":1757385606100,"token_in":790,"token_out":48,"side_effect":"unknown"}
{"event":"tool_attempt","run_id":"run_7f3","span_id":"span_19","attempt":3,"tool_name":"charge_card","args_hash":"a91c9e0c","idem_hash":"e2b1aa44","status":"ok","t_start":1757385606200,"t_end":1757385607800,"token_in":788,"token_out":36,"side_effect":"external"}

Attempt three is the only ok row. A last-write exporter would keep that row alone.

The first two rows still consumed tokens and network time. If the provider ignored idem_hash, two captures can exist.

Score Completeness Before Debugging

Do not start a root-cause hunt on thin traces. Score the file first, then inspect storms.

Required keys for this workflow:

1. run_id
2. span_id
3. attempt
4. tool_name
5. args_hash
6. status
7. t_start
8. t_end

A run with missing attempt scores as incomplete. Treat incomplete runs as untrusted for cost analysis.

Completeness is a gate, not a verdict. A full record can still describe a wrong tool.

Artifact: Retry Storm Scanner

The script below is a local example. It reads JSONL from stdin. It does not open a network socket.

It groups events by span_id. It flags spans with more than one attempt. It also flags status flips inside one span.

#!/usr/bin/env python3
"""Retry storm scanner for agent tool traces.

Example only. Not production telemetry.
"""
from __future__ import annotations

import hashlib
import json
import sys
from collections import defaultdict
from typing import Any, Iterator

REQUIRED = (
    "run_id",
    "span_id",
    "attempt",
    "tool_name",
    "args_hash",
    "status",
    "t_start",
    "t_end",
)


def iter_events(stream) -> Iterator[dict[str, Any]]:
    for line_no, raw in enumerate(stream, 1):
        raw = raw.strip()
        if not raw:
            continue
        try:
            event = json.loads(raw)
        except json.JSONDecodeError as exc:
            print(f"skip line {line_no}: {exc}", file=sys.stderr)
            continue
        yield event


def completeness(event: dict[str, Any]) -> float:
    present = sum(
        1 for key in REQUIRED if key in event and event[key] not in (None, "")
    )
    return present / len(REQUIRED)


def canonical_args_hash(args: Any) -> str:
    blob = json.dumps(args, sort_keys=True, separators=(",", ":")).encode()
    return hashlib.sha256(blob).hexdigest()


def scan(events: Iterator[dict[str, Any]], storm_threshold: int = 2) -> dict[str, Any]:
    by_span: dict[str, list[dict[str, Any]]] = defaultdict(list)
    incomplete = 0
    total = 0
    for event in events:
        if event.get("event") != "tool_attempt":
            continue
        total += 1
        if completeness(event) < 1.0:
            incomplete += 1
            continue
        by_span[str(event["span_id"])].append(event)

    storms = []
    flips = []
    for span_id, attempts in by_span.items():
        attempts.sort(key=lambda row: int(row["attempt"]))
        n = len(attempts)
        statuses = [row["status"] for row in attempts]
        hashes = {row["args_hash"] for row in attempts}
        first = attempts[0]
        last = attempts[-1]
        if n >= storm_threshold:
            storms.append(
                {
                    "span_id": span_id,
                    "run_id": first["run_id"],
                    "tool_name": first["tool_name"],
                    "attempts": n,
                    "statuses": statuses,
                    "unique_args": len(hashes),
                    "ms": int(last["t_end"]) - int(first["t_start"]),
                    "idem_hash": first.get("idem_hash"),
                }
            )
        if len(set(statuses)) > 1:
            flips.append(span_id)

    storms.sort(key=lambda row: row["attempts"], reverse=True)
    return {
        "tool_events": total,
        "incomplete_events": incomplete,
        "spans": len(by_span),
        "storm_spans": len(storms),
        "status_flip_spans": len(flips),
        "top_storms": storms[:20],
    }


def main() -> None:
    report = scan(iter_events(sys.stdin))
    json.dump(report, sys.stdout, indent=2)
    sys.stdout.write("\n")


if __name__ == "__main__":
    main()

Dry-run commands

Save the three example rows as attempts.jsonl. Then run the scanner from the project directory.

python3 -m py_compile retry_storm.py
python3 retry_storm.py < attempts.jsonl

Expected report shape for that fixture, not a live benchmark:

{
  "tool_events": 3,
  "incomplete_events": 0,
  "spans": 1,
  "storm_spans": 1,
  "status_flip_spans": 1
}

If incomplete_events is not zero, stop. Fix the exporter before reading top_storms.

Emit Attempts At The Tool Boundary

The wrapper below is proposed example code. It is not a measured production client.

import time
from typing import Any, Callable


def call_with_attempts(
    emit,
    run_id: str,
    span_id: str,
    tool_name: str,
    args: dict[str, Any],
    fn: Callable[[dict[str, Any]], Any],
    *,
    attempts: int = 3,
    idem_hash: str,
    args_hash: str,
    side_effect: str = "unknown",
) -> Any:
    last_exc: Exception | None = None
    for attempt in range(1, attempts + 1):
        t_start = int(time.time() * 1000)
        status = "ok"
        result = None
        try:
            result = fn(args)
        except TimeoutError:
            status = "timeout"
            last_exc = TimeoutError(tool_name)
        except Exception as exc:  # example only; narrow this in real code
            status = "error"
            last_exc = exc
        t_end = int(time.time() * 1000)
        emit(
            {
                "event": "tool_attempt",
                "run_id": run_id,
                "span_id": span_id,
                "attempt": attempt,
                "tool_name": tool_name,
                "args_hash": args_hash,
                "idem_hash": idem_hash,
                "status": status,
                "t_start": t_start,
                "t_end": t_end,
                "token_in": None,
                "token_out": None,
                "side_effect": side_effect if status == "ok" else "unknown",
            }
        )
        if status == "ok":
            return result
    raise last_exc if last_exc else RuntimeError("tool failed")

Pass one span_id into every retry of that logical call. Do not mint a new span for attempt two.

Leave token_in null when the runtime hides usage. Null is better than a fabricated zero.

Classify A Storm Before You Patch

When the scanner prints a storm, classify it first. The table is a decision aid, not a proof.

Observation	Likely cause	First check
unique_args = 1, mixed error/ok	transient dependency	timeout budget and idempotency key
unique_args > 1	agent changed arguments	schema, validator, prompt
many attempts, all ok	retry after success	success predicate
high incomplete_events	exporter bug	required field list
same idem_hash, two provider captures	provider ignored the key	key header and provider docs

Do not start with a model swap. Most storms are counters, predicates, or keys.

Sampling Without Dropping Storms

Full traces grow faster than useful signal. Sample clean spans. Keep every storm span.

A simple local policy looks like this:

• keep 100% of spans with attempts >= 2
• keep 100% of spans with status != ok
• keep 5% of single-attempt ok spans
• drop raw payloads after the hash exists

The 5% figure is an example knob. Tune it against disk, not against a marketing quota.

Where Free Model Access Fits

Some storms need a second pass over redacted error strings. A model can cluster those strings after hashes exist.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode is an open-source project with free model access. It also offers a free server option for small trace indexes.

Use the model on redacted messages only. Keep hashes and counts on the server. Keep raw tool bodies on the laptop.

The scanner still works without that product. Stdin and JSONL are enough for the count.

If those free options are already in reach, point them at storm summaries. Do not upload card numbers or raw keys.

Limitations

Truncated hashes can collide, though full SHA-256 is usually enough. Treat short prefixes as debug labels only.

side_effect=unknown is common and weakens billing conclusions. Clock skew across hosts can inflate ms.

The scanner ignores streaming token events by design. Completeness scoring does not prove a tool was correct.

Example thresholds are not capacity plans. They are starting gates for a local debug loop.

Who Should Skip This

Skip this workflow if tools never retry. A counter on single-attempt spans adds noise.

Skip it under an extreme threat model that forbids argument hashes. Some regulated payloads cannot be digested off-box.

Skip it for hard real-time loops that cannot buffer JSONL. This method assumes a file you can score later.

Do not treat the script as a managed APM replacement. It does not trace processes, GC, or kernel waits.

Tonight's Checklist

1. Emit attempt on every tool try.
2. Reuse one span_id across those tries.
3. Hash arguments before any shared disk.
4. Score completeness on ingest.
5. Alert on storm_spans, not only on errors.

Green spans can still charge twice. Count attempts before you trust the trace.