New series. Arc Rector answers what do I build on. Arc Ops answers the other half: how do I run an agent that touches real systems without getting paged? Nine levels - L0 Idempotency, L1 Retry budgets, L2 Cost caps, L3 Approval gates, L4 Determinism and replay, L5 Sandboxing, L6 Audit trails, L7 Queues and concurrency, L8 Incident response. Every level ships a measured demonstration and a control: the same code with the guard removed, run over the same input, so the guard is proved load-bearing rather than assumed.
Repo: https://github.com/dev48v/arc-ops - PUBLIC, MIT, dependencies = [], 113 tests, no network and no model. Level 0 computes in your browser on a page that is genuinely self-contained - no CDN, no font host, not one asset fetched off the page: https://dev48.infy.uk/arcops/level0-idempotency.html
Exactly-once delivery does not exist. What exists is at-least-once delivery plus an effect that only lands once, and that second half is the whole level. One fixed loss plan turns 8 invoices into 14 attempts, and five handlers see identical traffic.
| handler | attempts | effects | duplicates | missing | charged |
|---|---|---|---|---|---|
| no guard (control) | 14 | 14 | 6 | 0 | 94,000c |
| intent key | 14 | 8 | 0 | 0 | 48,500c |
| payload hash, stable bodies | 14 | 6 | 0 | 2 | 40,100c |
payload hash + stamped client_request_id
|
14 | 14 | 6 | 0 | 94,000c |
The guard is worth exactly 45,500c on this workload - the difference between two ledgers, not a claim about one.
The headline is the fourth row
A payload hash is what people reach for, and a client that stamps a per-attempt request id is ordinary hygiene. Together they reproduce the control's ledger byte for byte: 14 effects, 94,000c, every retry charged again. And with stable bodies the same strategy silently loses two charges while reporting zero duplicates - so a dashboard that counts duplicates and nothing else scores it as clean. Two error columns, always. Duplicates is the loud failure; missing is the quiet one, an intent that produced no effect at all because a different intent hashed to its key and answered on its behalf.
Fourteen hand-labelled pairs of wire requests, scored in both directions:
| strategy | correct | missed dedup | wrong dedup |
|---|---|---|---|
| hash the body | 4/14 | 7 | 3 |
| hash the sorted body | 6/14 | 5 | 3 |
| intent-shaped, untyped encoding | 12/14 | 0 | 2 |
| intent key | 14/14 | 0 | 0 |
The three wrong-dedup pairs have byte-identical bodies and different meanings, because the invoice, the account and the actor are not in the body. A body hash merges them, and the second call gets a 200 carrying somebody else's receipt.
Count is not order, and a key does not fix order
set_status is naturally idempotent: replay it three times and you get three writes and one state. The trap is reading that as replay-safe. It bounds the count, not the order. Land a stale v2 paid after v3 shipped and last-write-wins reverts the record to paid with no duplicate anywhere to show for it; compare-and-set on a version rejects the stale write and holds at shipped. A key stops a duplicate append and says nothing about ordering. A version stops a stale assignment and does nothing at all to an append. Neither substitutes for the other.
Two workers, one key, deterministically
A lease expiring is indistinguishable from a worker that died, so a broker will hand the same message to two consumers. A threading.Barrier forces every worker to read before any worker writes, which makes the failure assertable rather than flaky. Check-then-act admits N of N at 2, 3, 4, 8 and 16 workers. An atomic claim and a sqlite INSERT OR IGNORE admit exactly 1 at every count. Both stores are interface-identical and behave identically on one thread, which is precisely why the broken one ships: nothing in the calling code changes and the tests pass.
Nine levels, one at a time: https://dev48.infy.uk/arcops.php
Top comments (0)