48-Hour Field Notes: The Second POST After 200 OK

The mock charged the card at 09:12:03. It wrote {"ok": true, "charge_id": "ch_1041"} and closed the socket. Fourteen seconds later the same JSON body arrived again, same path, no idempotency key. The handler minted ch_1042 because it was an honest POST.

Nothing threw. The client logged two successes. The agent loop treated each 200 as progress.

A mutating tool call is not a read. It is a door that stays open if you knock twice. Tutorials still spend their ink on “the model picks a function.” They spend less on what happens when the function already happened.

This note is a 48-hour protocol for catching that second knock before it leaves a sandbox. It is not a production postmortem and not a benchmark. The numbers below come from a local fixture, not from a live processor.

Hour 0: a charge that cannot defend itself

I wanted one endpoint that made a side effect, and one client that looked like a tool. The server is deliberately small. If the handler is clever, the bug hides in the cleverness.

# charge_mock.py — labeled fixture, not a payment integration
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
import json, threading, time, hashlib
from pathlib import Path

LEDGER = Path("call_ledger.jsonl")
CHARGES = {}

class ChargeHandler(BaseHTTPRequestHandler):
    def log_message(self, fmt, *args):
        return

    def _read(self):
        n = int(self.headers.get("Content-Length", "0") or 0)
        return self.rfile.read(n) if n else b"{}"

    def do_POST(self):
        raw = self._read()
        body = json.loads(raw.decode() or "{}")
        key = self.headers.get("Idempotency-Key", "")
        rec = {
            "t": time.time(),
            "path": self.path,
            "len": len(raw),
            "sha": hashlib.sha256(raw).hexdigest()[:16],
            "idem": key or None,
        }
        LEDGER.parent.mkdir(parents=True, exist_ok=True)
        with LEDGER.open("a") as f:
            f.write(json.dumps(rec) + "\n")

        if self.path != "/v1/charges":
            self.send_response(404); self.end_headers(); return

        if key and key in CHARGES:
            payload = CHARGES[key]
        else:
            payload = {"ok": True, "charge_id": f"ch_{len(CHARGES)+1:04d}"}
            CHARGES[key or f"anon-{time.time_ns()}"] = payload

        blob = json.dumps(payload).encode()
        self.send_response(200)
        self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length", str(len(blob)))
        self.end_headers()
        self.wfile.write(blob)

def serve(port=8765):
    httpd = ThreadingHTTPServer(("127.0.0.1", port), ChargeHandler)
    threading.Thread(target=httpd.serve_forever, daemon=True).start()
    return httpd

Start it, then knock once with curl. The first call is supposed to work. That is not the interesting part.

python - <<'PY'
from charge_mock import serve
import time
serve(8765)
open("call_ledger.jsonl", "w").close()
time.sleep(0.2)
print("ready")
while True:
    time.sleep(1)
PY

curl -sS -X POST http://127.0.0.1:8765/v1/charges \
  -H 'Content-Type: application/json' \
  -d '{"amount": 1900, "currency": "usd", "order_id": "ord_9"}'

You get a charge id. The ledger has one line. The system still looks healthy, the way a kitchen looks healthy before the second kettle boils dry.

Hour 8: the client that retries a success

Agent tool loops fail in a boring way. The model emits a call. The runtime hits a timeout, a truncated JSON body, or a cancelled stream. The server may already have committed. The next step still looks like “the tool has not returned, call it again.”

I did not need a full agent to reproduce that. A twelve-line client is enough if it treats a short read as “try POST again.”

# flaky_tool.py — labeled reproduction, not a recommended client
import json, socket, time

def post_charge(payload, timeout=0.05, retries=3):
    body = json.dumps(payload).encode()
    last_err = None
    for attempt in range(retries):
        s = socket.create_connection(("127.0.0.1", 8765), 2)
        s.settimeout(timeout)
        req = (
            b"POST /v1/charges HTTP/1.1\r\n"
            b"Host: 127.0.0.1\r\n"
            b"Content-Type: application/json\r\n"
            + f"Content-Length: {len(body)}\r\n\r\n".encode()
            + body
        )
        try:
            s.sendall(req)
            buf = b""
            while True:
                chunk = s.recv(64)  # short reads on purpose
                if not chunk:
                    break
                buf += chunk
            s.close()
            if b"\r\n\r\n" in buf and b"charge_id" in buf:
                return buf
            last_err = RuntimeError("incomplete tool result")
        except (socket.timeout, OSError) as e:
            last_err = e
            try:
                s.close()
            except OSError:
                pass
            time.sleep(0.02)
    raise last_err

Run it against the mock with a tight timeout. Watch the ledger, not the exception. The exception is a story the client tells itself. The ledger is what the server heard.

python - <<'PY'
from flaky_tool import post_charge
try:
    print(post_charge({"amount": 1900, "currency": "usd", "order_id": "ord_9"}))
except Exception as e:
    print("client_error", type(e).__name__, e)
PY
wc -l call_ledger.jsonl
cat call_ledger.jsonl

On this machine the client often raised timeout after the mock had already appended a line. A second attempt appended a second line with the same SHA of the body and a different anonymous key. Two charges. One order id. Status code 200 both times, if the second read completed.

That pattern is older than language models. Agents make it louder because the retry policy sits in prose (“if the tool failed, try again”) instead of in a library that knows POST is not GET.

Hour 24: assert on the ledger, not on the smile

Green tests were the wrong instrument. A unit test that mocks requests.post and returns a fixture never sees the second wire call. The check has to sit on the server’s memory of the conversation, the way a night clerk counts keys instead of trusting the guest book.

# test_ledger.py
import json, time
from pathlib import Path
import charge_mock
from flaky_tool import post_charge

def load_ledger():
    p = Path("call_ledger.jsonl")
    if not p.exists():
        return []
    return [json.loads(line) for line in p.read_text().splitlines() if line.strip()]

def test_mutating_tool_does_not_double_post():
    Path("call_ledger.jsonl").write_text("")
    charge_mock.CHARGES.clear()
    charge_mock.serve(8765)
    time.sleep(0.15)
    try:
        post_charge({"amount": 1900, "currency": "usd", "order_id": "ord_9"})
    except Exception:
        pass
    rows = load_ledger()
    posts = [r for r in rows if r["path"] == "/v1/charges"]
    assert len(posts) <= 1, posts

pytest -q test_ledger.py

The first time I ran that file it failed. That was the point. A passing HTTP client and a failing ledger is a better 48-hour outcome than the reverse.

I then added one header on the retry path and made the mock key off it. Same body, same key, same charge_id. The test still failed until the client stopped sending the body twice without the header. Idempotency is not a comment in the tool description. It is a header the runtime must attach before the model gets another turn.

# the stop rule I kept
MUTATING = {"POST", "PUT", "PATCH", "DELETE"}

def allow_retry(method, last_status, idem_key, body_sha, seen):
    if method not in MUTATING:
        return True  # reads may retry
    if not idem_key:
        return False
    prior = seen.get(idem_key)
    if prior is None:
        return True
    return prior == body_sha and last_status in {0, 408, 409, 429, 500, 502, 503, 504}

Read that as a door policy. GET can wander. POST needs a named ticket, and the ticket is invalid if the body changed.

Where a free model and a free server actually sit

The protocol above does not need a GPU. It needs a process that can emit a tool call, a mock that keeps a ledger, and a machine that stays up long enough to fail the test twice. That is the narrow place a hosted sandbox helps: not as a judge of model quality, but as a place the mock and the client can run without pointing at a live card network.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode is an open-source project that, as supplied for this write-up, offers free model access and a free server option. I am not attaching model names, token ceilings, or hardware claims here; those change and I did not re-measure them for this note. The useful property for this protocol is dull: you can keep the mock, the ledger file, and the pytest command on a box that is not your laptop, and you can let a model propose the client while the ledger remains the source of truth.

If you try that path, pin the mock to loopback inside the sandbox and never hand the agent a real secret. A free server that can reach the public internet is still a server. The ledger only protects you if the only listener is the one you wrote.

What broke in the 48 hours

The first break was timeout-after-commit. A 50ms socket timeout on a local handler sounds like a joke until the handler writes the ledger before it writes the body. The client is then correct to say it saw no result. The server is also correct to say it charged. Both logs are true. The test that only checks the client is the liar.

The second break was body hashing without canonical JSON. {"amount": 1900, "currency": "usd"} and {"currency": "usd", "amount": 1900} are one charge in the business and two hashes in a naive ledger. I sorted keys before hashing. The duplicate detector started matching.

The third break was HTTP 200 with a truncated body. The model, in a replayed transcript I kept as a fixture, called the tool again because the previous tool message ended mid-object. That is the same family of failure as a receipt that ends mid-brace, except the damage is a second side effect instead of a parse error. Filling the tool result is not optional if the call already mutated state. Better to surface a hard error than to invite another POST.

The fourth break was classifying GET and POST with the same retry budget. A tool named get_charge can retry. A tool named create_charge cannot, unless the runtime owns the idempotency key. Names that start with create_, charge_, refund_, send_, or delete_ got a mutating flag in the harness. The flag is a heuristic. Heuristics leak. The ledger still has the last word.

What I would repeat

I would keep the ledger as JSONL, one line per request, written before the handler decides the status. If the process dies, the line is still there. I would hash the canonical body and store the idempotency key as a first-class field, not as a comment in a prompt.

I would run the failing pytest as a gate on any agent-authored HTTP client, including clients that only exist for one afternoon. The command is short. The signal is sharper than a coverage percentage.

python -c "from charge_mock import serve; serve(8765); import time; time.sleep(3600)" &
pytest -q test_ledger.py --maxfail=1

I would not let the model invent the base URL. The fixture server binds 127.0.0.1. If a generated client targets any other host, the protocol fails closed. That single string compare caught more “creative” tool calls than the retry policy did.

Limitations, and who should not use this

This harness does not prove an API is correct. It proves a client did not POST twice to this mock with this body. Real processors have more states: captured, refunded, disputed, delayed webhooks. If you need those, contract-test against a recorded catalog from the vendor, not against ch_0001.

It also does not measure latency, cost, or model quality. A free server is the wrong tool for load testing someone else’s API, and a free model path is the wrong tool for certifying a payments integration. Do not point this loop at a live endpoint. Do not store card numbers in the ledger. Do not treat a passing len(posts) <= 1 as PCI, SOX, or anything with an acronym.

Skip the protocol if you do not own a mock of the side-effecting API. Skip it if your tools are all reads. Skip it if the runtime already attaches idempotency keys and you already audit wire traffic. In those shops the interesting bugs live elsewhere.

The 48 hours end when the ledger and the client disagree in public. After that, the work is ordinary: fix the retry policy, keep POST behind a key, and stop asking a language model to decide whether a charge exists. The model can draft the client. The ledger decides whether the draft may ship.

If you want a sandbox that already advertises free model access and a free server option, MonkeyCode is one place to park this mock overnight. Read the project files first, keep the listener on loopback, and let pytest fail before any second charge_id becomes a story you have to tell a human.