DEV Community

Your payment webhook has a 24-hour cliff. Here's the reconciliation job that catches it.

arun rajkumar on October 28, 2026

Every webhook tutorial covers the happy path: verify the signature, return 200, done. Almost none cover what happens when your endpoint is down fo...
Collapse
 
crtdrops profile image
Israel Negrete Lepe •

Good to see the sweep framed as the part that makes the cliff irrelevant. One thing I'd add to the alerting: besides lastDeliveredAt, watch the age of the oldest payment that is still non-final. lastDeliveredAt tells you the webhook path stopped; it doesn't tell you the sweep is running but getting nothing useful back, for example when the same incident takes out the status endpoint, or a payment keeps coming back PENDING. The oldest-pending age catches both paths failing at once, and it gives you a clean rule for when a payment stops being the sweep's problem and becomes a person's: past a deadline, it goes to a review queue instead of being polled forever.

Collapse
 
mickyarun profile image
arun rajkumar •

Right, and the distinction is worth naming. lastDeliveredAt is a liveness signal about my delivery path. Oldest non-final age is a signal about the work. The first can be green while the second is terrible, and the case you give is the one that does it: a single incident takes out the webhook and the status endpoint, the sweep runs on schedule and learns nothing, and every dashboard says the system is up.

It's also the only one of these signals that can't be made green by restarting something. It's a function of payment state rather than of my infrastructure, which is a property worth having in an alert.

Your deadline rule is where this meets @ilragioniere's per-payment attempt cap further up the thread. The cap tells the sweep when to stop asking. The deadline tells it where to put the row. A cap on its own just drops the payment quietly, which is the cliff again at a smaller scale.

One concession about my own metric: the oldest-non-final threshold has a floor set by the slowest legitimate payment, so it's a number somebody tunes. lastDeliveredAt isn't. That doesn't make it worse, but it's the one that will drift.

Collapse
 
crtdrops profile image
Israel Negrete Lepe •

That floor is why I'd set it per provider rather than as one number. In mobile top-ups I've seen one brand answer a status query in seconds and another only settle through a daily batch file. A single threshold is either deaf for the first or always firing for the second.

Thread Thread
 
mickyarun profile image
arun rajkumar •

Per provider is right and I would go one further: per provider and per method. The batch-file case you give is the one that breaks my metric worst, because for most of its range the oldest-non-final age is not measuring a problem at all. It is measuring the provider's schedule. A threshold tuned for that is deaf by design, and you are back to a number somebody picked.

The version with nothing to tune puts the policy on the row. Each payment carries an expected-by, written when it is created, from its own provider and method. The alert is then a count of payments past their own expected-by. Nothing global, nothing to set an amber band on, and the batch provider and the seconds provider live in the same query.

That is the same move @ilragioniere and I ended up at on the health-check thread by a completely different route: when a number needs tuning, stop tuning it and make it attributable per row. The cost is honest and it is the mirror of the one you named. One global threshold can be wrong once. An expected-by can be wrong per payment, and it is a field somebody has to populate, so it will be wrong quietly on whichever provider got added last.

Thread Thread
 
ilragioniere profile image
Riccardo Vitali •

One hole in the expected-by version, and it is the one you already named: the provider added last.

The alert counts payments past their own expected-by. A payment created before anyone wrote a schedule for that provider has no expected-by, so it is not past it, so it does not count. The new provider is invisible until someone notices it is invisible. Same failure as the global threshold, smaller.

The cheap fix is a constraint plus a second count. NOT NULL at creation, so a missing schedule is a rejected write rather than a silent row; and the alert prints rows waiting on a schedule beside rows past one. The denominator is the provider roster again, but at least this time it fails loudly.

What it still does not settle: that the expected-by was right. It settles that this payment took longer than the schedule assigned to it. Worth printing exactly that sentence.

Thread Thread
 
crtdrops profile image
Israel Negrete Lepe •

@mickyarun Thank you for carrying that metric into the other branches, and for crediting it each time. That's a generous way to run a thread.

Read end to end, the pieces fit now: @_firelinks' outbox gives fulfilment its own row, your expected-by puts the policy on that row, and @ilragioniere's NOT NULL makes a missing schedule fail loudly instead of hiding. I'd keep Riccardo's last sentence as the alert text itself: this payment took longer than the schedule assigned to it.

Collapse
 
_firelinks profile image
Mike Dabydeen •

The read-then-upsert in apply() lets the sweep and the webhook worker fulfil the same order twice. Picture a two-hour outage ending: Atoa's retried COMPLETED events land in the queue while the first cron run picks up the same rows. Both paths read PENDING, both pass the rank check, and both call fulfilOrder. A second COMPLETED is only a no-op when the two arrive one after the other.

I'd let the database decide. Do the transition as one conditional update, UPDATE payments SET status = 'COMPLETED' WHERE id = $1 AND status = 'PENDING' RETURNING id, and fulfil only when it returns a row. Whichever path loses the race gets zero rows and stops. If fulfilOrder lives in another service, a unique constraint on orderId in its table covers the same gap from the other side.

Collapse
 
mickyarun profile image
arun rajkumar •

You're right, and the window is wider than the outage case. Any two deliveries of the same event landing in two workers hits it; the outage just makes it likely enough to notice. Read, decide, write, with the decision made from a value that can change between the read and the write.

Two things I'd change from your version. The hard-coded status = 'PENDING' is the right move taken one step short: once the guard lives in the WHERE clause, the allowed predecessors have to live there too, so it becomes AND status = ANY($2). And the row may not exist yet, which my upsert was quietly handling, so the real shape is an INSERT with ON CONFLICT DO UPDATE and the condition on the update. RETURNING still only fires when the update actually ran.

The constraint on orderId is the half I'd have missed, and it's the more important one. The conditional update makes the transition atomic. It does nothing about fulfilOrder running twice from two services that each legitimately won their own race. I collapsed a state-transition problem and a side-effect-idempotency problem into one function and only guarded the first.

@ilragioniere's reply below is the other end of the same code being wrong.

Collapse
 
_firelinks profile image
Mike Dabydeen •

Once the transition is atomic there's one more gap, between the commit and the fulfilOrder call. If the worker that won the update dies there, the row says COMPLETED, retried webhooks get zero rows, and the sweep skips it because it now reads only non-final rows. The order is paid, never fulfilled, and nothing in the design will look at it again.

I'd record the intent in the same transaction as the transition: insert a fulfilment row keyed on orderId alongside the conditional update, and have a separate worker drain those rows and call fulfilOrder until it succeeds. That's the transactional outbox. The status change and the job then commit together or not at all, and the orderId constraint downstream makes the worker's retries harmless.

Thread Thread
 
mickyarun profile image
arun rajkumar •

The outbox is right, and the reason it matters here is the defect you have not seen yet. My sweep filters status: 'PENDING'. Yours is the better description of what it should do: read only non-final rows. Either way a row that says COMPLETED with no fulfilment is outside the query, so the case you describe is not just unhandled, it is unreachable. The outbox fixes that by giving the unfulfilled work its own row in its own table with its own non-final state, which means there are two sweeps and the second one's subject is fulfilment rather than payment.

Where I would stop short of calling it solved: the outbox moves the atomic boundary, it does not remove it. The drain worker can call fulfilOrder and die before marking its row done. That is the same unknown state one layer out. It is survivable only because of the orderId constraint you named first, which is doing the actual work. So the outbox buys the guarantee that I know I owe the fulfilment. It does not make the side effect idempotent. If fulfilOrder is a third party with no idempotency key, I still have no way to retry it safely, and the outbox has converted a silent loss into a visible one. That is a real improvement and it is a smaller one than it reads as.

The second sweep needs its own oldest-non-final age too, which is @crtdrops' metric from the other branch, applied to a queue instead of a payment.

Thread Thread
 
_firelinks profile image
Mike Dabydeen •

With no idempotency key on fulfilOrder, the drain worker has to pick which failure it would rather have, and the outbox row is the place to write that choice down. Commit the row as dispatched before the call, and a crash leaves it sitting in dispatched where nothing retries it. That's at most once, and every stuck row goes to a person. Mark it done only after the call returns, and a crash means a retry. That's at least once, and sometimes two shipments. For fulfilment I'd usually take the duplicate, since a second parcel is easier to recover than an order that never ships, but it should be a decision made per side effect rather than whatever order the code happened to write.

Either way, I'd stamp an attempt time on the row before calling. Rows with an attempt older than the call's own timeout and no result are the only ones where you don't know what happened, and that count gives the second sweep a sharper alert than oldest non-final age. On a normal day it should be zero.

Thread Thread
 
mickyarun profile image
arun rajkumar •

The attempt stamp is the part I would add to the article. Dispatched, no result, attempt older than the call's own timeout is the only row where you genuinely do not know what happened, and separating that from merely slow is what makes the second sweep answerable. Oldest non-final age pages somebody who then has to work out which kind of unknown it is. Count of rows past their own timeout pages somebody who already knows.

On the choice itself, payments makes the same one you do. A duplicate payout is recoverable and a missing one is a dispute, so the default is at least once with an idempotency key, and the key belongs on the receiving side rather than the caller's, because the caller is the thing that crashed.

Which is the real reading of your first sentence. No idempotency key on fulfilOrder is not a constraint to design around. It is the bug, and everything after it is choosing which failure to inherit.

Where I would not let the per-side-effect decision drift. Once the choice lives in the outbox row it is a column, a column with two legal values gets a default, and the default gets inherited by the next handler somebody copies. Same failure as the source column that @fintechrailsdigest and I worked through further up this page. So if one side effect really is at most once, give it a different drain path rather than a flag on a shared one. A flag is something you can forget to set. A path is not.

Collapse
 
fintechrailsdigest profile image
Fintech Rails Digest •

the doorbell framing is the part most teams skip: the webhook is a hint, the pull is the truth, and the sweep is what makes the pull schedulable. one thing the reconciliation job should carry is a verified_at on every row it touches, not just the event id, because after the endpoint comes back the replays arrive out of order and the sweep is the only thing that can tell which state is current. the bad friday deploy is survivable; trusting arrival time instead of verification time is what turns it into a monday fire.

Collapse
 
mickyarun profile image
arun rajkumar •

verified_at is the right column and the reason is that nothing except the sweep writes it. The provider cannot set it, a replay cannot set it, so it answers when did I last prove this rather than when was I last told something. Same property as a timestamp that the check writes about itself: it is true because nobody maintains it.

One thing it still cannot do. verified_at records the last proof, not which proof. If two replays and one pull land inside a minute, the row reads verified a minute ago and cannot say the pull is what set the status. Your sentence — the sweep is the only thing that can tell which state is current — only holds if the status write is restricted to the sweep's path, or if the row carries the source alongside the timestamp. Without that, verified_at is trusting arrival time again, one layer in, with a better name on it.

And the comparison worth alerting on is verified_at against the provider's own event timestamp, not against now. Against now you learn the sweep ran. Against the event you learn whether the thing you proved is the latest thing that happened.

Collapse
 
fintechrailsdigest profile image
Fintech Rails Digest •

fair hit: verified_at answers when, not which, so either the sweep is the only writer or the row carries the source next to the timestamp. carrying the source is the cheaper of the two since it survives the day someone adds a manual verify path. and agreed on the alert comparison: against now you only learn the sweep ran, against the provider's event timestamp you learn whether the thing you proved is stale relative to the latest thing that happened. that second one is the actual reconciliation signal, the first is just job health.

Thread Thread
 
mickyarun profile image
arun rajkumar •

Agreed on both, and the source column has one failure the restricted-writer version does not have.

A source field is only as honest as the paths that remember to set it. The manual verify path is the one you added it for, and it is also the one most likely to write status and skip source. Then the row reads verified_at set, source NULL, which is neither proof nor absence, and any query that filters on source quietly drops it from both alerts. So source NOT NULL with no default, same as the status it describes. Otherwise the cheap option is cheap because it is optional, and optional is how it goes empty.

The split you drew at the end is worth keeping as two alerts rather than two fields. Sweep ran is job health and it pages whoever owns the job. Verified state is behind the provider's latest event is reconciliation and it pages whoever owns the money. They fail for different reasons, and only one of them is ever worth waking someone for.

Thread Thread
 
fintechrailsdigest profile image
Fintech Rails Digest •

the NOT NULL point is the sharper half: a nullable source column is a permission slip for exactly the path you named, and the manual verify flow is where discipline erodes first because it is the one a tired human drives at 11pm. forcing the value at write time turns the missing case into an exception the operator reads instead of a NULL the queries ignore. on the split, agreed, and keeping them as two alerts keeps each one answerable by whoever gets it: job health goes to the person who owns the cron, verification lag goes to the person who owns the money, and neither page has to explain the other's failure mode.

Thread Thread
 
mickyarun profile image
arun rajkumar •

Agreed, with one gap NOT NULL leaves open. It forces a value, not a true one. The tired human at 11pm will type manual, because they know what they did. The path that lies is the one written six months later by someone who copies the sweep's helper and inherits source = sweep without reading it. That row is populated, passes the constraint, and claims a proof that never ran.

So the write has to be constrained by value, not only required. Only the sweep's own path may write sweep, and everything else writes whatever it actually is. A check constraint cannot do that. The sweep setting its own source through a path nothing else can call is the version that holds, which is your restricted-writer option arriving by the back door after the cheap one failed.

On the split, one thing to wire before they ship as two independent pages. They are not independent. If the sweep is down, verification lag climbs for a reason that has nothing to do with the money, and the person who owns the money gets paged for a cron. So gate the reconciliation alert on sweep health: fire it only when the sweep ran and the proof is still behind the provider's latest event. Otherwise the second alert is the first one with a longer delay and a worse on-call.

Collapse
 
ilragioniere profile image
Riccardo Vitali •

The rank guard fails open, and the payload you quote already tells you why.

The payload carries five statuses: COMPLETED, PENDING, FAILED, plus AUTHORIZED and CANCELLED for card. RANK maps three of them. So for a card event, RANK[event.status] <= RANK[row.status] compares against undefined, and in JS neither undefined <= 2 nor 2 <= undefined is true. The guard returns false and the write goes through. A stale PENDING walks a row backwards out of AUTHORIZED; a duplicate COMPLETED lands on top of a CANCELLED row. Failing open is the wrong direction for every one of those.

Second, smaller, same map: FAILED and COMPLETED share rung 2, and the guard is <=. Whichever arrives first wins and the other is dropped as a duplicate. That is only harmless as long as a payment can never move between those two states, which is an assumption about the provider's state machine that the code neither states nor checks. Give COMPLETED its own rung and the intent holds without the assumption.

So: rank all five statuses explicitly, and make an unknown status an error rather than a pass. The <= guard is fine; it is only as honest as the table behind it.

The sweep filter compounds it. where({ status: 'PENDING' }) never re-reads a row sitting at FAILED or at AUTHORIZED, so the path that repairs a missed webhook is blind to exactly the rows where the webhook got a status the map can't rank. Select every row that isn't COMPLETED and is older than N minutes, with a per-payment attempt cap.

One from your own quote of the docs: "any response other than HTTP 200 is a failure" means a 202 Accepted, the ordinary async ack, counts as a failed delivery and starts the back-off. Your handler returns 200 explicitly, which is right; keep it explicit if anyone ever moves the ack off the request path.

Collapse
 
mickyarun profile image
arun rajkumar •

All four land, and the first two are the same bug wearing two hats. RANK is a total order over a state machine that isn't one. PENDING to AUTHORIZED to COMPLETED is a line. PENDING to CANCELLED and PENDING to FAILED branch off it, and COMPLETED and CANCELLED are both terminal with neither superseding the other. One integer per status cannot express that, so whatever integer I pick is wrong somewhere. FAILED and COMPLETED sharing rung 2 isn't a typo in the table. It's the table being the wrong shape. What that code wanted was an allowed-transitions set keyed on from and to, with everything not in it rejected. That also kills the undefined case for free, because an unknown status is not in the set.

On making unknown an error: yes, with one more turn. The event is already on a queue by then, so a bare throw retries a status nobody will ever add to the map, forever. It has to dead-letter and alert. Otherwise fail-closed becomes a quiet backlog, which is the cliff again with better manners.

The sweep filter is the one that stings. The article sells the sweep as the thing that makes the cliff irrelevant and then filters it to PENDING, so it is blind in exactly the place the rank table already failed. Not-final, not PENDING, with the per-payment cap you mention.

And the 202 is my own quoted sentence contradicting my own code sample, which I did not notice. If any response other than 200 is a failure, the ordinary async ack starts the back-off. I wrote ack fast as a latency argument. The actual argument is that the literal 200 is load-bearing.

Collapse
 
challan116ux profile image
challan116-ux •

The webhook-as-hint framing is the part most teams learn only after an outage. A retry window that ends is not a provider flaw — it is a documented boundary — but any event that falls past it disappears silently unless a second path exists. The reconciliation job is that path, and its real value is that it does not depend on delivery at all: it compares what you recorded against what the source of truth says, on a schedule, whether or not a doorbell ever rang. Every integration that treats delivery as the record will eventually lose an event at exactly the moment the system was already under stress.

Collapse
 
mickyarun profile image
arun rajkumar •

Agreed on the boundary being documented rather than a flaw — that distinction is the whole reason the second path has to exist rather than being a nice-to-have.

One hidden precondition in the part you put the weight on. The reconciliation job does not depend on delivery, but it does depend on the status endpoint answering. @crtdrops found the hole in this thread: one incident takes out the webhook and the status endpoint together, the sweep runs exactly on schedule, learns nothing, and every dashboard stays green. Comparing what you recorded against what the source of truth says has a reachable source of truth buried in it, and when that fails the sweep's silence is indistinguishable from health. Which is why the alert cannot be on the sweep running. It has to be on the age of the oldest thing the sweep has not resolved.

Collapse
 
delayrun profile image
delay.run •

If the endpoint returns 200 but your workers are stuck, the provider thinks delivery succeeded — so the 24h cliff email never fires. Alerting on PENDING older than N (same query the sweep uses) covers that path. And sharing one apply() between the webhook and the sweep matters more than people think; otherwise you end up with two definitions of "paid."

Collapse
 
mickyarun profile image
arun rajkumar •

The 200 is a receipt for the HTTP hop. It says the bytes arrived, and the provider has no other question it could ask. So every retry guarantee you buy from a provider is a guarantee about delivery and not about work, and the cliff email only ever fires for the failures the provider can see. You are right that my alert inherits the provider's blind spot.

On PENDING older than N: right backstop, and it is the one place I would not use a global N. @fintechrailsdigest and I went through this further up the page. A single threshold has to be loose enough for the slowest provider, which makes it useless for the fastest one, and nobody chose the number. Expected-by written per payment from the provider's own stated window, then alert on rows past their own schedule. Same query, and the alert then says this payment took longer than the schedule assigned to it, which somebody can argue with.

One apply() shared between the webhook and the sweep, yes, and I would be precise about which half. The state transition, certainly, or you end up with two definitions of paid. The side-effect dispatch is the part I would keep separate, for the reason worked out in the thread above this one: a shared handler carries its default into the next thing somebody copies.

Collapse
 
fintechrailsdigest profile image
Fintech Rails Digest •

expected-by written per payment from the provider's own window is the only version of N that survives an argument at 2am. and splitting apply() that way matches where the drift actually lives: the webhook and the sweep are allowed to fail differently, they just can't disagree about what paid means. the delivery-versus-work distinction is worth stating out loud in every webhook design doc, it's the assumption nobody writes down.

Collapse
 
krusenas profile image
Karolis •

Returning 200 quickly only helps if the proxy stores the request first. Otherwise it's gone. An LB 503 during a deploy can start the 24-hour retry window and leave the payment stuck in processing. I built Webhook Relay to store the request before forwarding it.

Collapse
 
mickyarun profile image
arun rajkumar •

The load balancer case is the one the post gets wrong, and it is wrong in the direction that matters. Return 200 fast is advice to the application, and a 503 at the load balancer during a deploy never reaches the application, so none of that advice runs. The provider's retry clock starts anyway. Thanks for naming it.

Store before forward is the right shape, and I would state its boundary so nobody buys it as the whole fix. It moves the durable point one hop earlier, to whatever is in front of you. It does not remove the hop in front of that. If the relay acks and then loses it, the provider has a receipt and you have nothing, which is the same failure one layer out. That is not an argument against it. It is the reason the reconciliation job still has to exist: the sweep is the only thing covering the window where somebody sent and nobody stored, and that window gets smaller with a relay in front of it and never closes.

The thing I would pin on the deploy case: the 24-hour window is the provider's, and your deploy is not in it. A ten-minute rolling restart that drops forty webhooks has already spent the first retry attempt before anyone looks at a dashboard.

Collapse
 
fintechrailsdigest profile image
Fintech Rails Digest •

@mickyarun your ticket-plus-actor-reference split is the right boundary, and i'll concede the nonce point. a render-time token is off-label, it dies on reload, and the second attempt walks in clean, which is exactly why the insert-level constraint stays as the backstop even after the button does its job. and on the honest end, agreed: once the second refund is genuinely intended, the system's job is to surface it precisely rather than pretend it was a bug. your you-have-already-paid-this prompt is the right model for that.