The worker log ended on a comma. Not a stack trace, not a timeout banner, just a trailing comma and then the same INSERT running again. I was on hour three of a borrowed coding box, watching a webhook worker “fix itself” by doing the side effect twice.
The setup was ordinary. A small HTTP handler accepted a delivery, wrote a row, and enqueued a follow-up job. An agent loop was supposed to patch a parser bug. It had a file-write tool, a shell tool, and a test command. The free remote server was useful because I did not want that loop chewing through a billed workstation while I traced a flake.
Disclosure: This article was prepared as part of MonkeyCode's product outreach. I parked the notebook on MonkeyCode’s free server option and used its free model access — the project currently describes a complimentary 10 million token allowance — so the experiment could sit overnight without attaching it to a paid cluster. That is the only product detail this write-up depends on. The failure mode itself is older than any one assistant.
Hour 0–8: the queue looked guilty
The first chart I trusted was the wrong one. Two jobs appeared for one delivery_id. The broker UI showed two publishes a few hundred milliseconds apart. I blamed at-least-once delivery, because that is the story every webhook tutorial trains you to expect.
I added a unique constraint and watched the second write die on UNIQUE. The HTTP layer still returned 201 for both attempts. The agent, reading a truncated tool result from the first write, treated the missing closing brace as “the patch did not land” and issued the mutating call again.
That is the analogy I kept returning to. A cashier prints half a receipt. The customer cannot see the order number, so they pay again. The register did nothing wrong after the first commit. The paper was cut too early.
Hour 8–20: prove the cut, not the feeling
I stopped arguing with the queue and recorded the tool payload as raw bytes. The command was boring on purpose.
python - <<'PY'
from pathlib import Path
p = Path("/tmp/tool_result.json")
data = p.read_bytes()
print(len(data), data[-40:])
PY
wc -c /tmp/tool_result.json
tail -c 80 /tmp/tool_result.json | od -An -tx1
The file ended in the middle of a string. json.loads raised JSONDecodeError. The agent log still said write_file: ok because the transport had streamed a partial object and a watcher had marked the call finished when the byte window went idle.
I rebuilt the handler as a fixture so the flake was not “the internet.” The following is a reconstructed lab script, not a production service.
# receipt_lab.py — reconstructed fixture
import json
from dataclasses import dataclass, field
class IncompleteReceipt(Exception):
pass
@dataclass
class Store:
rows: dict = field(default_factory=dict)
jobs: list = field(default_factory=list)
def insert_delivery(self, delivery_id: str, body: dict) -> None:
if delivery_id in self.rows:
raise ValueError("duplicate delivery_id")
self.rows[delivery_id] = body
self.jobs.append({"delivery_id": delivery_id, "action": "fanout"})
def parse_receipt(raw: str) -> dict:
raw = raw.strip()
if not raw:
raise IncompleteReceipt("empty")
try:
obj = json.loads(raw)
except json.JSONDecodeError as exc:
raise IncompleteReceipt(str(exc)) from exc
if not isinstance(obj, dict) or "ack" not in obj:
raise IncompleteReceipt("missing ack")
return obj
def apply_mutator(store: Store, raw_receipt: str, delivery_id: str, body: dict) -> str:
# Refuse to retry a write unless the previous receipt is a complete document.
receipt = parse_receipt(raw_receipt)
if receipt.get("ack") != "written":
raise IncompleteReceipt("ack not written")
store.insert_delivery(delivery_id, body)
return "ok"
The important line is not the insert. It is the gate in front of the insert. If the receipt cannot be parsed as a finished object with an explicit ack, the mutator does not run. Streaming systems fail closed. Idle timeouts do not count as success.
Hour 20–32: a test that fails on half-objects
I wanted a check I could rerun after every agent patch, including patches that claimed they “only touched comments.” The test below is meant to be copied. It does not talk to a network.
# test_receipt_completeness.py
import json
import pytest
from receipt_lab import Store, apply_mutator, IncompleteReceipt
COMPLETE = json.dumps({"ack": "written", "bytes": 412})
CUT = '{"ack": "writ'
def test_complete_receipt_writes_once():
store = Store()
apply_mutator(store, COMPLETE, "d-9", {"n": 1})
assert list(store.rows) == ["d-9"]
assert len(store.jobs) == 1
def test_truncated_receipt_does_not_write():
store = Store()
with pytest.raises(IncompleteReceipt):
apply_mutator(store, CUT, "d-9", {"n": 1})
assert store.rows == {}
assert store.jobs == []
def test_agent_retry_after_truncation_is_still_single_write():
store = Store()
with pytest.raises(IncompleteReceipt):
apply_mutator(store, CUT, "d-9", {"n": 1})
apply_mutator(store, COMPLETE, "d-9", {"n": 1})
with pytest.raises(ValueError, match="duplicate"):
apply_mutator(store, COMPLETE, "d-9", {"n": 1})
assert len(store.jobs) == 1
Run it the same way on a laptop or on the free box:
python -m pytest -q test_receipt_completeness.py
The third test is the field note. Truncation plus retry is the common agent loop. Uniqueness in the store is the backstop when the loop ignores the gate. You want both. A unique index without a parse gate still burns the first attempt’s downstream webhook. A parse gate without a unique index still doubles the row if two complete receipts arrive.
What actually broke
Three separate layers told a success story about incomplete work.
The streamer treated silence as end-of-message. That is reasonable for a chat token stream and unreasonable for a tool result that must be a document. A chat answer can be “good enough” at 80 tokens. A JSON receipt cannot.
The agent treated JSONDecodeError as “the file write never happened.” It did not distinguish transport failure from application rejection. Those two states need different next actions. Transport failure may retry the same call. Application rejection needs a new plan, or a stop.
My first unique constraint was on the wrong column. I keyed jobs on payload_hash. The agent’s second write changed a comment and the hash. The database accepted the cousin row. Idempotency has to follow the business event, not the bytes of the patch.
A short rule I would keep on a sticky note: if a tool can create a row, send money, drop a mail, or close a ticket, the observation must include a stable event_id and a parseable ack. Anything less is a shrug, not a receipt.
A workflow I would repeat
I now run mutating agent loops behind a local proxy that only forwards a tool result after json.loads succeeds and ack is present. The proxy is about forty lines. It is also a reconstructed example.
# receipt_proxy.py — reconstructed example, not a network service
import json
import sys
raw = sys.stdin.read()
try:
obj = json.loads(raw)
except json.JSONDecodeError:
sys.stderr.write("incomplete receipt; refusing to mark tool success\n")
sys.exit(2)
if obj.get("ack") not in {"written", "noop", "duplicate"}:
sys.stderr.write("receipt missing ack; refusing\n")
sys.exit(2)
sys.stdout.write(json.dumps(obj, separators=(",", ":")))
Wire it in front of the tool observer:
cat /tmp/tool_result.json | python receipt_proxy.py >/tmp/tool_result.ok.json
echo $?
Exit code 2 means the loop must not retry a mutator. It may retry a read. That split is the whole workflow. Reads are cheap to repeat. Writes are not.
On the free server I kept three files in one directory: the fixture, the test, and the proxy. After each agent patch I ran the same two commands. No extra dashboard. If the test failed, the patch did not land, regardless of the commit message the model proposed.
python -m pytest -q test_receipt_completeness.py
git diff --check
git diff --check caught a second, smaller mess: the agent had rewritten a JSON fixture with a UTF-8 ellipsis that looked fine in a pager and exploded in the worker. Truncation is not the only way a document stops being a document.
Limitations, without the brochure voice
This gate does not give you exactly-once delivery. It gives you “do not treat a chopped observation as permission to mutate.” Brokers, mail APIs, and payment providers still need their own idempotency keys. If those keys are missing, a perfect JSON receipt will still double-charge.
It also does not help if the tool lies with a complete document. {"ack":"written"} on a failed disk write is a different bug. You still need the data store to reject duplicates.
A shared free server is a noisy neighbor. Clock jumps, leftover processes, and disk pressure can truncate files for reasons that have nothing to do with model output. That is why the test never talks to the network and why the unique constraint stays in the store. The box is a scratchpad, not a replica of production.
Do not use this approach as the control plane for money movement, medical records, or anything that already has a dedicated transactional outbox. Do not point an unsupervised mutating loop at a database you cannot restore. Do not skip the parse gate because a chat transcript “looked done.”
People who should skip it: anyone hoping a free remote box will match production latency; anyone who cannot add a unique key; anyone whose tool protocol is raw prose with no ack field and no plan to add one.
What I would repeat in the next 48 hours
I would still start with the last forty bytes of the tool result, not with the queue graph. I would still refuse to retry a write on JSONDecodeError. I would still keep the fixture offline so a flake on the box cannot impersonate a product incident.
The agent-loop fashion this week is to talk about whether models already outcode a median developer. That debate does not change the receipt problem. A stronger model that retries faster will double-write faster. Completeness is a protocol property. It does not emerge from fluency.
If you want a throwaway place to rerun the same two commands while a loop chews on a parser, the free server option I used for this notebook is an easy parking spot. Keep the unique key. Keep the ack. Let the comma at the end of the file stay what it is: an unfinished sentence, not a success.
Top comments (0)