A checkout agent called a charge tool at 14:02:11 UTC. The handler returned HTTP 200 before the runner woke. The local runner then hit its eight second timeout.
The span was marked failed in the runner log. A retry policy launched attempt two for the same cart. The payment ledger later showed two successful captures.
The agent summary still reported an overall success. This note reconstructs that case as a teaching fixture. The figures are illustrative, not a production measurement.
Two clocks, one missing fact
Recent posts ask when an automatic retry is justified. This rule answers that only for tools that write. Better model scores do not close a blind retry hole.
What the diff missed
Attempt one reached the payment API before timeout. The response body never landed in the local runner. Attempt two opened a fresh span for the same cart.
Nothing linked those spans to one side effect. A later diff of tool names looked entirely clean. Both calls were stored under the name charge_cart.
Why a status code is not enough
Attempt one had no exit code in the trace file. Attempt two stored exit code 0 and a short preview. A green second attempt does not prove the first was empty.
You need a commit fact, not only a status code. Timeout means the runner stopped waiting for bytes. It does not mean the remote handler skipped the write.
Fields to store on every tool span
Keep the trace fields your runner already persists today. Add four fields before any automatic retry is enabled.
Required additions
- Store attempt_id as an integer starting at one.
- Store idempotency_key for the side effect, not the span.
- Store commit_state as unknown, committed, rejected, or not_sent.
- Store retry_reason as empty on attempt one only.
Also store parent_run_id so retries group under one turn. Do not reuse the span id across later attempts. A new attempt is a new span with the same key.
How to read commit_state
The value unknown means the runner lost the response. It does not mean the remote tool was safe to repeat. The value not_sent means failure happened before the socket write.
Only not_sent is a clean retry candidate in this policy. The value committed means the side effect already landed. The value rejected means the server refused that write.
Decision table
Use this table before the runner schedules another attempt. The checker script below encodes these same stop rules.
| prior commit_state | next attempt | action | reason |
|---|---|---|---|
| not_sent | allowed | retry | no request left the process |
| rejected | blocked | stop | server refused the write |
| committed | blocked | stop | side effect already landed |
| unknown on attempt 1 | blocked until read | reconcile | remote may have committed |
| unknown on attempt 2+ | blocked | stop | another try risks a double write |
| missing | blocked | stop | a retry without this field is blind |
Reconcile means a ledger read using the idempotency key. Do not call the charge tool again until that read returns. If that read fails, keep the state unknown and stop.
Worked numbers from the reconstruction
Attempt one lasted 8000 ms and then timed out. The payment API log showed a capture at 8012 ms. The runner never stored a response body for that call.
Attempt two started 1400 ms after the local timeout. It stored exit code 0 plus a truncated preview string. The ledger count for cart c-1844 then became two.
A correct trace would have blocked that second attempt. Those timestamps are a teaching fixture for this note. Replace them with spans exported from your own runner.
Do not treat 8012 ms as a benchmark for any host. Do not treat the cart id as data from a real shop. The point is the missing state, not the sample latency.
Checker script
The following script is a proposal you can copy. It has not been executed against a live agent fleet. Save it locally as check_retries.py before you run it.
It reads JSONL with one tool span on each line. Lines with a blank body are skipped by the loader. A bad state value is reported as a hard failure.
#!/usr/bin/env python3
'''Flag unsafe retries. Proposal code, not a live-fleet result.'''
import json
import sys
from collections import defaultdict
ALLOWED = {'unknown', 'committed', 'rejected', 'not_sent'}
def load(path):
rows = []
with open(path, encoding='utf-8') as handle:
for line_no, line in enumerate(handle, start=1):
text = line.strip()
if not text:
continue
row = json.loads(text)
row['_line'] = line_no
rows.append(row)
return rows
def check(rows):
grouped = defaultdict(list)
errors = []
for row in rows:
key = row.get('idempotency_key')
state = row.get('commit_state')
attempt = row.get('attempt_id')
if not key or attempt is None or state not in ALLOWED:
errors.append(
'line %s: missing key, attempt_id, or commit_state' % row['_line']
)
continue
grouped[key].append(row)
for key, spans in grouped.items():
ordered = sorted(spans, key=lambda item: item['attempt_id'])
seen = [item['attempt_id'] for item in ordered]
expected = list(range(1, len(seen) + 1))
if seen != expected:
errors.append('%s: attempt_id sequence is %s' % (key, seen))
for prev, curr in zip(ordered, ordered[1:]):
prior = prev['commit_state']
nxt = curr['attempt_id']
if prior != 'not_sent':
errors.append('%s: attempt %s follows %s' % (key, nxt, prior))
reason = curr.get('retry_reason')
if reason in (None, ''):
errors.append(
'%s: attempt %s has empty retry_reason' % (key, nxt)
)
return errors
def main():
path = sys.argv[1] if len(sys.argv) > 1 else 'spans.jsonl'
errors = check(load(path))
if errors:
print('FAIL %s' % len(errors))
for item in errors:
print(item)
return 1
print('PASS 0')
return 0
if __name__ == '__main__':
raise SystemExit(main())
Fixture and commands
Fixture file
Create spans.jsonl with the two attempts shown below. The first commit_state is unknown on purpose here. The second attempt should fail the checker immediately.
{"parent_run_id":"run-9","attempt_id":1,"idempotency_key":"cart-c-1844","commit_state":"unknown","retry_reason":"","tool":"charge_cart"}
{"parent_run_id":"run-9","attempt_id":2,"idempotency_key":"cart-c-1844","commit_state":"committed","retry_reason":"timeout","tool":"charge_cart"}
Commands and expected results
Run the checker from that same working directory. Pass the fixture path as the first argument. Record both the printed status and the exit code.
python3 check_retries.py spans.jsonl
echo exit=$?
The expected result on this fixture is FAIL. The process exit code should be non-zero as well. The failure text should name the key cart-c-1844.
It should also name the prior state unknown. That pair is the bug this policy is meant to catch. A passing file needs attempt one marked not_sent.
You can also pass a file that has one line. If the shell mangles quotes, write the file by hand. The rule under test stays the same either way.
Debug loop you can reuse
Export one failed user turn as a JSONL file. Group those lines by idempotency key, not by tool name. Run check_retries.py and keep its exit code in the job log.
If the result is FAIL, reconcile the ledger first. Do that work before you spend another model call. If the result is PASS, rerun on a fresh input only.
Do not start this loop by swapping the model. A different model will not repair a missing commit fact. Fix the trace schema, then compare the two runs.
A cheap place to rerun the loop
Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode matters here only as a second runner. The operator supplied a claim of free model access.
A free server option is the other supplied claim. An outreach note mentioned a token allotment, but no primary source was attached. This draft therefore quotes no token number at all.
It also states no hardware size and no offer duration. Those details change, so confirm them on the current page. Use that server as a second runner, not as truth.
Keep the JSONL file on your side as the source. Copy the same parent_run_id into both runner logs. Then diff commit fields, not the prose summary text.
If the free server drops a field, fail the check. A missing commit_state is a stop, not a pass. The checker does not call any vendor API itself.
It only reads the trace file path you pass in. A free tier can throttle or sleep without notice here. This article has no evidence about how long access lasts.
Network delay can widen the timeout window further. That delay makes unknown more common, not more rare. Treat unknown as a stop until a ledger read returns.
What is not claimed
No latency number in this note is a host benchmark. No pass from the script proves the payment API. Confirm current terms before you depend on free access.
If you need a second runner, the free server option can host that pass. Keep the JSONL local and let the checker fail closed.
Who should not use this policy
Skip this policy when the tool is a pure read. A catalog lookup can be retried with less risk. Skip it when the handler already returns a prior receipt.
In that case, store the receipt id in the span. Skip it for writes that cannot carry a stable key. A keyless write cannot be reconciled by this script.
Also skip it if you cannot persist JSONL across timeout. The checker is useless on a preview string alone. Previews drop bytes, and this script needs the raw line.
What this does not prove
The script does not prove the payment API is correct. It only shows that the trace forbids a blind retry. It does not hash the tool catalog before replay.
It does not record argv or a process exit code. Those checks still matter, and they belong in other notes. It does not measure model quality in any way.
A careful model can still time out on a slow tool. Clock skew can make recorded durations look simply wrong. Store both runner time and tool time when you have them.
Do not sort retries by wall clock alone. Sort retry spans by attempt_id before you judge them. If the model writes commit_state, do not trust that field.
The runner or the tool adapter should set that field. A model-filled state is a claim, not a commit record.
Close
A retry is a new span with an old side effect. Store attempt_id and idempotency_key before the policy fires. Store commit_state and retry_reason on that same span.
Run the checker and keep the printed exit code. Stop when the prior state is unknown or committed. Also stop when the prior state is rejected.
That loop stays useful without any hosted runner. The file, the key, and the state are the whole method. Ship the checker beside the agent, not after the incident.
Top comments (0)