The "request that woke up late" is the exact failure that breaks naive idempotency — the first attempt wasn't dead, just slow, and it commits after you've already given up on it. Fencing is the right primitive. The piece I'd add: a timeout against that stalled request should move the operation to an explicit UNKNOWN state, never an automatic FAILED, because "I stopped waiting" and "it didn't happen" are different claims and only reconciliation can tell them apart. I went deep on fencing-plus-UNKNOWN for the agent case here: dev.to/rudratosh/your-ai-agent-ret...
Thanks Rudratosh. Agreed that the first attempt wasn't dead, and that fencing has to come with an explicit unknown. Your UNKNOWN framing is the part I'd adopt. My version flags the trip for reconciliation, but the flag sits beside the trip rather than being the trip's own state.
The entry-id-keyed marker is the crux, and it's the thing "just use an idempotency key" advice gets wrong — key on the request and a re-planned retry with fresh arguments slips past; key on the plan-minted entry id and the late request hits the same fence no matter how the retry is shaped. That's the whole game.
On adopting UNKNOWN: moving from "flag beside the trip" to "state the trip carries" is small in code but changes what's possible to get wrong. Once it's the operation's own state, two things fall out for free — every reader sees the uncertainty without knowing to check a side field, and the state machine itself can reject the illegal move (a fenced trip can't reach settled except through reconciliation). The flag-beside version relies on everyone remembering to read the flag; the state version makes forgetting impossible.
Going to read the fence code — curious whether the marker write and the charge dispatch share a transaction, or you're fencing across the same two-systems gap (your DB fence, the provider's charge). That's where "persist the fence before the side effect" starts to carry real weight, since a crash between the two is exactly the window the fence has to survive.
They don't share a transaction. The debit here is our own wallet, not a provider charge, but the trip, the marker and the wallet are still separate writes. So it's the same gap you describe, just inside our own store.
A crash between the two is the case the fence is built to survive. If an attempt burns the marker and dies before the debit, a resume finds the marker taken and no debit on record. It can't prove the first attempt is gone, so it doesn't charge. It flags the trip and stops. That's your UNKNOWN in practice, and Arun's point upthread is the cost: correct, but someone has to work the queue.
The next step closes most of that window: make the marker a condition on the debit write itself, so burning the fence and moving the money happen in one conditional write instead of two.
That last step changes the category, and it's worth being loud about why: because the wallet is your own store, you have the option I don't have in the provider case — collapse the fence and the debit into one conditional write (debit WHERE the marker is claimable, both or neither), and the crash-between-two-writes window closes because there's no "between" anymore. That's the whole advantage of staying inside one transactional boundary: atomicity is on the table, so UNKNOWN stops being a steady state and becomes just the recovery path for infra failures, not for your own ordering.
The reason my piece leans so hard on UNKNOWN is that the second write is usually a provider — a system that doesn't share your transaction, so you genuinely can't make burn-fence-and-move-money atomic, and the queue Arun mentioned is unavoidable. Your version is the strictly better position: when you own both sides, make it one conditional write and you've designed the UNKNOWN out rather than reconciled it away. The rule that falls out of both: atomic where the writes are yours; fence-plus-explicit-UNKNOWN only where they cross a boundary you don't control. You've drawn that line exactly.
One edge on the single conditional write: it closes the gap between fence and debit, but the moment anything downstream observes the result — a receipt, a notification, a balance another service reads — that observation becomes the new second write, and UNKNOWN reappears one hop out. So the conditional write is the floor, not the ceiling, as soon as a third party reads the outcome.
Agreed on the rule: atomic where the writes are yours, fence plus explicit UNKNOWN where they cross a boundary you don't own. That's a cleaner statement of it than anything in the post.
And fair point on the edge. The conditional write closes the gap inside the store, but a receipt or a notification is the next hop, and the same reasoning applies there.
Thanks for the back and forth, Rudratosh. This thread made the post better.
CTO & Co-Founder, Atoa. Building open banking payments for the UK. Half the cost of cards. Instant settlement. FCA-authorised. I hire for intent, not resumes. I write about payments, AI, and the messy
The fence is the right shape. The part I would put a number on is the branch you wrote last.
create_once and debit_if_sufficient are two writes. An attempt that burns the marker and then dies leaves a trip with a plan, no debit, and no marker left to burn. Resume reads landed False, fails to create the marker, and the trip goes to flag_for_reconcile. That is correct. It is also a trip that now needs a person, while the customer is still looking at a payment that never completed.
So the fence has not removed the failure. It has converted a double debit into a queue, which is the trade you want. Whether it stays a good trade depends entirely on whether anyone answers the queue. Payments has the same state and the same queue: an authorisation with no capture and no reversal. The reason card schemes put a time limit on that state is that otherwise nobody clears it.
Two things I would measure rather than assume. The rate into flag_for_reconcile per thousand charges, because that is the real price of your GRACE value and the only way to tell whether GRACE is too short. And the age in hours of the oldest unreconciled trip, not the count. A count normalises, and a flat four hundred reads the same every week. The oldest item maps to one named customer with no trip and no charge.
The version that needs no queue is one write: the uniqueness record as a column on the ledger entry under the pre-minted id, rather than a separate markers row. Whether that is available to you depends on whether the wallet, the ledger and the markers sit in the same store, and the post reads like they may not.
Thanks Arun, this is a sharp read, and you're right on the main point. The fence doesn't remove the failure. It changes what the failure costs: a double debit becomes a trip waiting for a person. That's the trade I wanted, but only if someone works the queue.
Both measurements are better than what I wrote. Reconcile rate per thousand charges is the honest price of GRACE, and I hadn't framed it that way. Oldest unreconciled age is the right alert too. A count would hide one customer stuck for days. I'm adding both.
On the one-write version: that's where the next step in the post is going, putting the fence inside the debit write's own condition so burning the marker and moving the money can't come apart. You read the layout correctly. The wallet, ledger and markers don't share a transaction today, which is why it's a staged change rather than a one-liner.
The authorization-with-no-capture comparison is a good one. Card schemes put a clock on that state for the same reason, and a reconcile queue needs one too.
Stamping the plan with a pre-minted ledger id before any money moves is a neat answer to "who is allowed to finish it". Whoever finds the trip later can see exactly how far the first attempt got, instead of inferring it from the wallet balance.
We dealt with a non-money version of the same shape at Ascendra Ventures, on a candidate enrichment pipeline for a talent platform. Slow steps (LinkedIn enrichment, OpenAI batch parsing, embeddings, PostgreSQL writes) kept leaving people in unclear states. We moved the processor into Temporal: one workflow claims the person, urgent profiles run immediately and the rest are batched, the batch path waits durably and falls back to polling when webhooks go quiet, and every person ends in a final status so failed work is visible. The rule we held to is the one your post describes: every item either finishes or visibly fails, and nothing sits in the middle.
In your setup, what decides that a claim in "charging" is stale enough to fence: a fixed timeout, or something stored in the plan itself?
Hosted cron job service. Schedule HTTP requests on a cron, after a delay or right away. Every run logged, failed jobs retried, alerts by email or Slack.
What I like most is that once the fence is in, GRACE stops being a correctness setting and becomes a performance knob. Set it too short and you get more reconcile flags and wasted attempts, not double debits.
That makes it easy to tune from data: put it a bit above the p99.9 of claim-to-debit time, then count how many takeovers end in the "marker already taken" branch. If that number isn't close to zero, your real stalls are longer than the grace period, and the people working the reconcile queue are the ones paying for it.
We're a place where coders share, stay up-to-date and grow their careers.
The "request that woke up late" is the exact failure that breaks naive idempotency — the first attempt wasn't dead, just slow, and it commits after you've already given up on it. Fencing is the right primitive. The piece I'd add: a timeout against that stalled request should move the operation to an explicit UNKNOWN state, never an automatic FAILED, because "I stopped waiting" and "it didn't happen" are different claims and only reconciliation can tell them apart. I went deep on fencing-plus-UNKNOWN for the agent case here: dev.to/rudratosh/your-ai-agent-ret...
Thanks Rudratosh. Agreed that the first attempt wasn't dead, and that fencing has to come with an explicit unknown. Your UNKNOWN framing is the part I'd adopt. My version flags the trip for reconciliation, but the flag sits beside the trip rather than being the trip's own state.
For context, the fence and the code are in the original post: developer.ericgitangu.com/blog/sta...
The marker is keyed on the entry id minted with the plan, so a retry with a new request ID still hits the same marker.
The entry-id-keyed marker is the crux, and it's the thing "just use an idempotency key" advice gets wrong — key on the request and a re-planned retry with fresh arguments slips past; key on the plan-minted entry id and the late request hits the same fence no matter how the retry is shaped. That's the whole game.
On adopting UNKNOWN: moving from "flag beside the trip" to "state the trip carries" is small in code but changes what's possible to get wrong. Once it's the operation's own state, two things fall out for free — every reader sees the uncertainty without knowing to check a side field, and the state machine itself can reject the illegal move (a fenced trip can't reach settled except through reconciliation). The flag-beside version relies on everyone remembering to read the flag; the state version makes forgetting impossible.
Going to read the fence code — curious whether the marker write and the charge dispatch share a transaction, or you're fencing across the same two-systems gap (your DB fence, the provider's charge). That's where "persist the fence before the side effect" starts to carry real weight, since a crash between the two is exactly the window the fence has to survive.
They don't share a transaction. The debit here is our own wallet, not a provider charge, but the trip, the marker and the wallet are still separate writes. So it's the same gap you describe, just inside our own store.
A crash between the two is the case the fence is built to survive. If an attempt burns the marker and dies before the debit, a resume finds the marker taken and no debit on record. It can't prove the first attempt is gone, so it doesn't charge. It flags the trip and stops. That's your UNKNOWN in practice, and Arun's point upthread is the cost: correct, but someone has to work the queue.
The next step closes most of that window: make the marker a condition on the debit write itself, so burning the fence and moving the money happen in one conditional write instead of two.
That last step changes the category, and it's worth being loud about why: because the wallet is your own store, you have the option I don't have in the provider case — collapse the fence and the debit into one conditional write (debit WHERE the marker is claimable, both or neither), and the crash-between-two-writes window closes because there's no "between" anymore. That's the whole advantage of staying inside one transactional boundary: atomicity is on the table, so UNKNOWN stops being a steady state and becomes just the recovery path for infra failures, not for your own ordering.
The reason my piece leans so hard on UNKNOWN is that the second write is usually a provider — a system that doesn't share your transaction, so you genuinely can't make burn-fence-and-move-money atomic, and the queue Arun mentioned is unavoidable. Your version is the strictly better position: when you own both sides, make it one conditional write and you've designed the UNKNOWN out rather than reconciled it away. The rule that falls out of both: atomic where the writes are yours; fence-plus-explicit-UNKNOWN only where they cross a boundary you don't control. You've drawn that line exactly.
One edge on the single conditional write: it closes the gap between fence and debit, but the moment anything downstream observes the result — a receipt, a notification, a balance another service reads — that observation becomes the new second write, and UNKNOWN reappears one hop out. So the conditional write is the floor, not the ceiling, as soon as a third party reads the outcome.
Agreed on the rule: atomic where the writes are yours, fence plus explicit UNKNOWN where they cross a boundary you don't own. That's a cleaner statement of it than anything in the post.
And fair point on the edge. The conditional write closes the gap inside the store, but a receipt or a notification is the next hop, and the same reasoning applies there.
Thanks for the back and forth, Rudratosh. This thread made the post better.
The fence is the right shape. The part I would put a number on is the branch you wrote last.
create_onceanddebit_if_sufficientare two writes. An attempt that burns the marker and then dies leaves a trip with a plan, no debit, and no marker left to burn. Resume readslandedFalse, fails to create the marker, and the trip goes toflag_for_reconcile. That is correct. It is also a trip that now needs a person, while the customer is still looking at a payment that never completed.So the fence has not removed the failure. It has converted a double debit into a queue, which is the trade you want. Whether it stays a good trade depends entirely on whether anyone answers the queue. Payments has the same state and the same queue: an authorisation with no capture and no reversal. The reason card schemes put a time limit on that state is that otherwise nobody clears it.
Two things I would measure rather than assume. The rate into
flag_for_reconcileper thousand charges, because that is the real price of your GRACE value and the only way to tell whether GRACE is too short. And the age in hours of the oldest unreconciled trip, not the count. A count normalises, and a flat four hundred reads the same every week. The oldest item maps to one named customer with no trip and no charge.The version that needs no queue is one write: the uniqueness record as a column on the ledger entry under the pre-minted id, rather than a separate markers row. Whether that is available to you depends on whether the wallet, the ledger and the markers sit in the same store, and the post reads like they may not.
Thanks Arun, this is a sharp read, and you're right on the main point. The fence doesn't remove the failure. It changes what the failure costs: a double debit becomes a trip waiting for a person. That's the trade I wanted, but only if someone works the queue.
Both measurements are better than what I wrote. Reconcile rate per thousand charges is the honest price of GRACE, and I hadn't framed it that way. Oldest unreconciled age is the right alert too. A count would hide one customer stuck for days. I'm adding both.
On the one-write version: that's where the next step in the post is going, putting the fence inside the debit write's own condition so burning the marker and moving the money can't come apart. You read the layout correctly. The wallet, ledger and markers don't share a transaction today, which is why it's a staged change rather than a one-liner.
The authorization-with-no-capture comparison is a good one. Card schemes put a clock on that state for the same reason, and a reconcile queue needs one too.
Stamping the plan with a pre-minted ledger id before any money moves is a neat answer to "who is allowed to finish it". Whoever finds the trip later can see exactly how far the first attempt got, instead of inferring it from the wallet balance.
We dealt with a non-money version of the same shape at Ascendra Ventures, on a candidate enrichment pipeline for a talent platform. Slow steps (LinkedIn enrichment, OpenAI batch parsing, embeddings, PostgreSQL writes) kept leaving people in unclear states. We moved the processor into Temporal: one workflow claims the person, urgent profiles run immediately and the rest are batched, the batch path waits durably and falls back to polling when webhooks go quiet, and every person ends in a final status so failed work is visible. The rule we held to is the one your post describes: every item either finishes or visibly fails, and nothing sits in the middle.
In your setup, what decides that a claim in "charging" is stale enough to fence: a fixed timeout, or something stored in the plan itself?
What I like most is that once the fence is in, GRACE stops being a correctness setting and becomes a performance knob. Set it too short and you get more reconcile flags and wasted attempts, not double debits.
That makes it easy to tune from data: put it a bit above the p99.9 of claim-to-debit time, then count how many takeovers end in the "marker already taken" branch. If that number isn't close to zero, your real stalls are longer than the grace period, and the people working the reconcile queue are the ones paying for it.