DEV Community

arun rajkumar
arun rajkumar

Posted on

Your payment webhook has a 24-hour cliff. Here's the reconciliation job that catches it.

Catches race conditions in webhook sweeps

Every webhook tutorial covers the happy path: verify the signature, return 200, done.

Almost none cover what happens when your endpoint is down for a day.

Most payment platforms retry with exponential back-off, and then they stop. On Atoa's side the documented behaviour is: any response other than HTTP 200 is a failure, we retry with back-off, and if deliveries keep failing for more than 24 hours we stop and email the business owner with the failing URL. That is a sensible design. It also means there is a cliff, and anything that falls off it is only recoverable if you built a second path.

This is the setup I'd build so a bad Friday deploy never turns into a Monday reconciliation fire.

The mental model: a webhook is a hint

Treat the webhook as a doorbell, not the delivery. It tells you "something changed, go and look." The source of truth is the payment record, which you can always read back from the API.

Once you accept that, three design decisions follow.

1. Verify against the raw body, and do it first

Atoa's V2 signing sends X-Atoa-Signature: v1=<hex>, an HMAC-SHA256 over the entire raw request body. The secret looks like whsec_<base64>, and the key is the base64-decoded part after the prefix.

const crypto = require('crypto');

function isValidAtoaSignature(signingSecret, rawBody, headerValue) {
  if (!headerValue || !headerValue.startsWith('v1=')) return false;

  const provided = Buffer.from(headerValue.slice(3), 'hex');
  const key = Buffer.from(signingSecret.slice(signingSecret.indexOf('_') + 1), 'base64');
  const expected = crypto.createHmac('sha256', key).update(rawBody, 'utf8').digest();

  // timingSafeEqual throws on unequal lengths
  if (provided.length !== expected.length) return false;
  return crypto.timingSafeEqual(provided, expected);
}

app.post('/webhook', express.raw({ type: 'application/json' }), (req, res) => {
  const raw = req.body.toString('utf8');
  if (!isValidAtoaSignature(process.env.ATOA_SIGNING_SECRET, raw, req.get('X-Atoa-Signature'))) {
    return res.status(401).send('Invalid signature');
  }
  queue.add('atoa-event', JSON.parse(raw)); // enqueue, don't process
  res.status(200).send('OK');              // ack fast
});
Enter fullscreen mode Exit fullscreen mode

Two things that bite people. Use express.raw, not express.json, because a re-serialised body will not match the signature. And note that the docs describe no timestamp in the header, so there is no built-in replay protection. Do not assume it. You need your own dedupe.

2. Make the handler idempotent on the payment, not the delivery

The payload carries paymentRequestId, orderId, paymentIdempotencyId and a status of COMPLETED, PENDING, FAILED (plus AUTHORIZED and CANCELLED for card). Retries mean you will see the same event more than once, and a PENDING can arrive after you have already acted on a COMPLETED if you are unlucky with ordering.

So I'd key state changes on paymentRequestId and only allow forward transitions:

const RANK = { PENDING: 1, FAILED: 2, COMPLETED: 2 };

async function apply(event) {
  const row = await db.payments.get(event.paymentRequestId);
  if (row && RANK[event.status] <= RANK[row.status]) return; // stale or duplicate
  await db.payments.upsert({ id: event.paymentRequestId, status: event.status });
  if (event.status === 'COMPLETED') await fulfilOrder(event.orderId);
}
Enter fullscreen mode Exit fullscreen mode

A late PENDING can never un-complete an order, and a second COMPLETED is a no-op.

3. Add the sweep, the part most teams skip

This is the piece that makes the cliff irrelevant. Run a job every 10 to 15 minutes that finds payments you created more than N minutes ago and still hold as non-final, then asks the API for their current status and feeds the answer through the same apply() function.

cron('*/10 * * * *', async () => {
  const stale = await db.payments.where({ status: 'PENDING' }).olderThan('10 minutes');
  for (const p of stale) {
    const live = await atoa.getPaymentStatus(p.id); // see docs.atoa.me, Get payment status
    await apply(live);
  }
});
Enter fullscreen mode Exit fullscreen mode

The sweep and the webhook share one code path, so there is nothing new to test and no second definition of "paid." If your endpoint is down for 30 hours, the webhooks stop at 24, and the sweep quietly repairs everything on the next run after you are back.

One caveat: getPaymentStatus here is a thin wrapper I've sketched. Check the exact endpoint and fields in the docs before you copy it.

Setup details worth knowing

  • V2 endpoints support multiple event types per endpoint and up to 3 endpoints per event type, each delivered independently. Point one at your app and one at a separate audit or analytics consumer, so one failing does not affect the other.
  • Failure tracking on V2 exposes failureCount and lastDeliveredAt. Alert on lastDeliveredAt going stale rather than waiting for the owner email.
  • Optional endpoint auth (OAuth 2.0 or Basic Auth) can sit on top of the signature. The docs recommend combining them.
  • Test in sandbox with the trigger-test-webhook API, overriding status, amount and orderId to simulate failures. It is sandbox-only; production keys are rejected.
  • Rotation is instant. Rotating the signing secret invalidates the old one immediately, and enabling V2 switches every delivery, including the WooCommerce and Magento plugins. Update your env first.

What I'd do, in order

  1. Verify on the raw body, enqueue, return 200.
  2. Key state on paymentRequestId, forward-only transitions.
  3. Ship the sweep before you ship to production, not after the first incident.
  4. Alert on lastDeliveredAt.

Webhooks are fast. Reconciliation is correct. You want both.

Try Atoa's sandbox: docs.atoa.me

Top comments (60)

Collapse
 
crtdrops profile image
Israel Negrete Lepe •

Good to see the sweep framed as the part that makes the cliff irrelevant. One thing I'd add to the alerting: besides lastDeliveredAt, watch the age of the oldest payment that is still non-final. lastDeliveredAt tells you the webhook path stopped; it doesn't tell you the sweep is running but getting nothing useful back, for example when the same incident takes out the status endpoint, or a payment keeps coming back PENDING. The oldest-pending age catches both paths failing at once, and it gives you a clean rule for when a payment stops being the sweep's problem and becomes a person's: past a deadline, it goes to a review queue instead of being polled forever.

Collapse
 
mickyarun profile image
arun rajkumar •

Right, and the distinction is worth naming. lastDeliveredAt is a liveness signal about my delivery path. Oldest non-final age is a signal about the work. The first can be green while the second is terrible, and the case you give is the one that does it: a single incident takes out the webhook and the status endpoint, the sweep runs on schedule and learns nothing, and every dashboard says the system is up.

It's also the only one of these signals that can't be made green by restarting something. It's a function of payment state rather than of my infrastructure, which is a property worth having in an alert.

Your deadline rule is where this meets @ilragioniere's per-payment attempt cap further up the thread. The cap tells the sweep when to stop asking. The deadline tells it where to put the row. A cap on its own just drops the payment quietly, which is the cliff again at a smaller scale.

One concession about my own metric: the oldest-non-final threshold has a floor set by the slowest legitimate payment, so it's a number somebody tunes. lastDeliveredAt isn't. That doesn't make it worse, but it's the one that will drift.

Collapse
 
crtdrops profile image
Israel Negrete Lepe •

That floor is why I'd set it per provider rather than as one number. In mobile top-ups I've seen one brand answer a status query in seconds and another only settle through a daily batch file. A single threshold is either deaf for the first or always firing for the second.

Thread Thread
 
mickyarun profile image
arun rajkumar •

Per provider is right and I would go one further: per provider and per method. The batch-file case you give is the one that breaks my metric worst, because for most of its range the oldest-non-final age is not measuring a problem at all. It is measuring the provider's schedule. A threshold tuned for that is deaf by design, and you are back to a number somebody picked.

The version with nothing to tune puts the policy on the row. Each payment carries an expected-by, written when it is created, from its own provider and method. The alert is then a count of payments past their own expected-by. Nothing global, nothing to set an amber band on, and the batch provider and the seconds provider live in the same query.

That is the same move @ilragioniere and I ended up at on the health-check thread by a completely different route: when a number needs tuning, stop tuning it and make it attributable per row. The cost is honest and it is the mirror of the one you named. One global threshold can be wrong once. An expected-by can be wrong per payment, and it is a field somebody has to populate, so it will be wrong quietly on whichever provider got added last.

Thread Thread
 
ilragioniere profile image
Riccardo Vitali •

One hole in the expected-by version, and it is the one you already named: the provider added last.

The alert counts payments past their own expected-by. A payment created before anyone wrote a schedule for that provider has no expected-by, so it is not past it, so it does not count. The new provider is invisible until someone notices it is invisible. Same failure as the global threshold, smaller.

The cheap fix is a constraint plus a second count. NOT NULL at creation, so a missing schedule is a rejected write rather than a silent row; and the alert prints rows waiting on a schedule beside rows past one. The denominator is the provider roster again, but at least this time it fails loudly.

What it still does not settle: that the expected-by was right. It settles that this payment took longer than the schedule assigned to it. Worth printing exactly that sentence.

Thread Thread
 
crtdrops profile image
Israel Negrete Lepe •

@mickyarun Thank you for carrying that metric into the other branches, and for crediting it each time. That's a generous way to run a thread.

Read end to end, the pieces fit now: @_firelinks' outbox gives fulfilment its own row, your expected-by puts the policy on that row, and @ilragioniere's NOT NULL makes a missing schedule fail loudly instead of hiding. I'd keep Riccardo's last sentence as the alert text itself: this payment took longer than the schedule assigned to it.

Thread Thread
 
ilragioniere profile image
Riccardo Vitali •

Then the alert has to carry which schedule it was judged against.

expected-by is written at creation from the schedule then in force. Change the schedule and the old rows keep the old one, so a tightened SLA turns an honestly-on-time backlog into what reads as a new incident, and a loosened one silently forgives real lateness. The row stays true; the reader draws the wrong conclusion from it.

Print the schedule's id and the date it was written beside the row. Two columns, and the alert text you quoted stays literally accurate.

Thread Thread
 
crtdrops profile image
Israel Negrete Lepe •

That's the part I'd have missed: the row stays true and the reader still draws the wrong conclusion. Your two columns also make a provider change countable: how many open rows were judged against the old rule. With @_firelinks' outbox and @mickyarun's expected-by, every alert in this thread now says what it was measured against.

Thread Thread
 
ilragioniere profile image
Riccardo Vitali •

That count has a denominator in the wrong place.

"How many open rows were judged against the old rule" is a census of what is still in flight, and it shrinks as those rows resolve. A schedule change nobody fixed looks like it is healing, week by week, until it disappears.

What the change cost is one population: every row judged against the old rule, closed ones included, split by how they closed. A row that closed clean under the old rule and would have been late under the new one is the finding. On open rows only, that row is gone before it can be counted.

If you have one real row where the schedule tightened under it, that is the row worth testing. Two columns either catch it or they don't.

Thread Thread
 
crtdrops profile image
Israel Negrete Lepe •

You're right, and it's the worse failure: the open count heals on its own, so an unfixed change reads as recovery.

I have lived that case, though rarely: a provider changing its settlement terms with transactions already in flight. It didn't just move alerts; it moved our cut-off processes, and sometimes the business model itself. And the impact scaled with the client: the same change was noise for a small one and a real problem for one with high transaction volume.

So I'd add one split to your population: closed rows judged against the old rule, grouped by client and weighted by volume. A count treats every row the same; the cost of a schedule change doesn't.

Thread Thread
 
ilragioniere profile image
Riccardo Vitali •

Then the weight goes in the cost column, not on the row count.

A count weighted by volume stops being a rate. These are two questions and they need two numbers: did the rule get applied (every row judged under it, one row one vote), and what did the change cost (the same rows, weighted by exposure). One weighted figure answers neither. It looks like a rate and reads like a bill.

Check the weighting against your own example, too. Volume is a good proxy for what a change costs and a poor one for who is watching. Same change, small client, nobody standing by the settle window: weight the count and that client rounds to zero, which is exactly the row you want to survive it.

Thread Thread
 
crtdrops profile image
Israel Negrete Lepe •

Fair, and your last point is the one I'd have gotten wrong: weight the count and the small client with nobody watching the settle window rounds to zero, which is exactly the row that has to survive.

So two numbers, as you say: one row one vote to show the rule was applied, and exposure in its own cost column. Thanks for pushing this one all the way down, Riccardo, and to @mickyarun for the thread that held it.

Thread Thread
 
ilragioniere profile image
Riccardo Vitali •

Both, yes. One thing to fix in the cost column before you ship it: exposure on its own is a flow, not a cost. Multiply by the window.

100k/day with a 3-day settle window and 30k/day with 10 days both read 300k in a plain exposure column. Same number, two different jobs: the first is a faster reconcile cycle, the second is a design problem. Exposure x days at risk tells them apart, and it is the figure you can actually trade queue capacity against.

Then the alert is one line off it: client-dollars at risk above X, no matter how few rows that is.

And the standing thing, plainly: bring me a number you don't trust and I will trace its pedigree in the thread. First one is free, same as always.

Thread Thread
 
crtdrops profile image
Israel Negrete Lepe •

That one connects straight back to the case I mentioned. When a provider lengthens its settlement window, the daily flow doesn't move at all, so a plain exposure column stays flat while the money at risk grows with every extra day. Exposure × days is the column that would have shown it.

Thanks, Riccardo. This thread ended up better than the post it started on, and that's on you and @mickyarun.

Thread Thread
 
mickyarun profile image
arun rajkumar •

NOT NULL is the right shape, and it buys its loudness at a specific moment that is worth naming before somebody implements it.

A payment that cannot be created without a schedule means the first live payment on a new provider fails. That is day one of an integration, under a deadline, which is exactly when somebody adds a default so the write goes through. A default expected-by is the global threshold back in the building, wearing a per-row column. So the constraint only holds if there is no default available anywhere, and writing the schedule is a blocking step on the integration checklist rather than something the first failed payment reminds you of.

Your second count is the part I would not have added and now would not ship without. Rows waiting on a schedule beside rows past one keeps the two populations from averaging, and a provider with no schedule gets a number that climbs instead of a row that is absent.

Your last sentence is the one I would put on the alert itself. This payment took longer than the schedule assigned to it. It is a smaller claim than late, and it names a schedule somebody wrote and can be argued with. The global threshold had nobody to argue with, which is most of why it survived so long.

Thread Thread
 
mickyarun profile image
arun rajkumar •

One thing to get right in the implementation, since this is going to get built by whoever reads it next.

Exposure times days is a stock, not a flow, so it has to be read at an instant. The natural implementation computes it nightly and sums the month, and then a row open for ten days is counted ten times and the month looks ten times worse than the worst moment in it. Snapshot, or peak over the window, never the sum. Riccardo's alert line is already the right shape for that: client-dollars at risk above X is a reading, not a total.

And the provider-lengthens-its-window case you brought is the one that made this thread. A flow column that stays flat while the money at risk grows is a better argument for the column than anything I wrote in the post.

Thread Thread
 
crtdrops profile image
Israel Negrete Lepe •

@mickyarun Thank you, that's generous, and the case only had somewhere to land because your post set up the problem.

The snapshot point is the one I'd have gotten wrong. A nightly job that sums exposure × days counts the same open row once per night, so a slow month looks worse than its worst day. Peak over the window answers what the alert actually asks.

Thread Thread
 
mickyarun profile image
arun rajkumar •

Yes, and one thing to pin when somebody builds it: peak needs its window named on the alert. A bare peak becomes max since the table existed, and that number never comes down, so after one bad afternoon in March the alert is permanently lit and gets muted. Peak over the last 24 hours, stated in the alert body, is a claim somebody can argue with. A bare peak is not.

The provider-lengthening-its-window case is the thing I will be quoting out of this thread. It argues for the column better than the post did.

Thread Thread
 
crtdrops profile image
Israel Negrete Lepe •

Agreed: "peak over the last 24 hours" is a claim someone can argue with; a bare peak only ratchets up until someone mutes it. Thanks, Arun, and quote away.

Thread Thread
 
crtdrops profile image
Israel Negrete Lepe •

@mickyarun The default is the part I'd have let through. On day one of an integration someone needs the first payment to go through, and a default is the fastest way to get there; it also means nobody ever wrote that provider's schedule. Making it a blocking item on the integration checklist puts that decision where someone has to make it on purpose, before the first real payment, not after the first failure.

Thread Thread
 
mickyarun profile image
arun rajkumar •

Agreed, and the default is the thing I would change first now, because it is the only item in that post that costs nothing to get wrong.

I would go one past the checklist though, for the reason you gave for it. A blocking checklist item is owned by the person integrating, and that person's job that week is the first payment going through. Checklists written in calm weather get ticked by whoever needs the thing to work. So the version with teeth is that there is no default: the sweep refuses to start and names which provider's retry schedule is unset. Then it is not a decision somebody skipped, it is a thing that does not run.

Payments does this with cut-off times and it is unglamorous for the same reason. A rail with no configured cut-off does not assume end of day, it fails at config load. The cost is one annoying error on day one of an integration, which is the cheapest day of the whole relationship to have an argument on.

Collapse
 
_firelinks profile image
Mike Dabydeen •

The read-then-upsert in apply() lets the sweep and the webhook worker fulfil the same order twice. Picture a two-hour outage ending: Atoa's retried COMPLETED events land in the queue while the first cron run picks up the same rows. Both paths read PENDING, both pass the rank check, and both call fulfilOrder. A second COMPLETED is only a no-op when the two arrive one after the other.

I'd let the database decide. Do the transition as one conditional update, UPDATE payments SET status = 'COMPLETED' WHERE id = $1 AND status = 'PENDING' RETURNING id, and fulfil only when it returns a row. Whichever path loses the race gets zero rows and stops. If fulfilOrder lives in another service, a unique constraint on orderId in its table covers the same gap from the other side.

Collapse
 
mickyarun profile image
arun rajkumar •

You're right, and the window is wider than the outage case. Any two deliveries of the same event landing in two workers hits it; the outage just makes it likely enough to notice. Read, decide, write, with the decision made from a value that can change between the read and the write.

Two things I'd change from your version. The hard-coded status = 'PENDING' is the right move taken one step short: once the guard lives in the WHERE clause, the allowed predecessors have to live there too, so it becomes AND status = ANY($2). And the row may not exist yet, which my upsert was quietly handling, so the real shape is an INSERT with ON CONFLICT DO UPDATE and the condition on the update. RETURNING still only fires when the update actually ran.

The constraint on orderId is the half I'd have missed, and it's the more important one. The conditional update makes the transition atomic. It does nothing about fulfilOrder running twice from two services that each legitimately won their own race. I collapsed a state-transition problem and a side-effect-idempotency problem into one function and only guarded the first.

@ilragioniere's reply below is the other end of the same code being wrong.

Collapse
 
_firelinks profile image
Mike Dabydeen •

Once the transition is atomic there's one more gap, between the commit and the fulfilOrder call. If the worker that won the update dies there, the row says COMPLETED, retried webhooks get zero rows, and the sweep skips it because it now reads only non-final rows. The order is paid, never fulfilled, and nothing in the design will look at it again.

I'd record the intent in the same transaction as the transition: insert a fulfilment row keyed on orderId alongside the conditional update, and have a separate worker drain those rows and call fulfilOrder until it succeeds. That's the transactional outbox. The status change and the job then commit together or not at all, and the orderId constraint downstream makes the worker's retries harmless.

Thread Thread
 
mickyarun profile image
arun rajkumar •

The outbox is right, and the reason it matters here is the defect you have not seen yet. My sweep filters status: 'PENDING'. Yours is the better description of what it should do: read only non-final rows. Either way a row that says COMPLETED with no fulfilment is outside the query, so the case you describe is not just unhandled, it is unreachable. The outbox fixes that by giving the unfulfilled work its own row in its own table with its own non-final state, which means there are two sweeps and the second one's subject is fulfilment rather than payment.

Where I would stop short of calling it solved: the outbox moves the atomic boundary, it does not remove it. The drain worker can call fulfilOrder and die before marking its row done. That is the same unknown state one layer out. It is survivable only because of the orderId constraint you named first, which is doing the actual work. So the outbox buys the guarantee that I know I owe the fulfilment. It does not make the side effect idempotent. If fulfilOrder is a third party with no idempotency key, I still have no way to retry it safely, and the outbox has converted a silent loss into a visible one. That is a real improvement and it is a smaller one than it reads as.

The second sweep needs its own oldest-non-final age too, which is @crtdrops' metric from the other branch, applied to a queue instead of a payment.

Thread Thread
 
_firelinks profile image
Mike Dabydeen •

With no idempotency key on fulfilOrder, the drain worker has to pick which failure it would rather have, and the outbox row is the place to write that choice down. Commit the row as dispatched before the call, and a crash leaves it sitting in dispatched where nothing retries it. That's at most once, and every stuck row goes to a person. Mark it done only after the call returns, and a crash means a retry. That's at least once, and sometimes two shipments. For fulfilment I'd usually take the duplicate, since a second parcel is easier to recover than an order that never ships, but it should be a decision made per side effect rather than whatever order the code happened to write.

Either way, I'd stamp an attempt time on the row before calling. Rows with an attempt older than the call's own timeout and no result are the only ones where you don't know what happened, and that count gives the second sweep a sharper alert than oldest non-final age. On a normal day it should be zero.

Thread Thread
 
mickyarun profile image
arun rajkumar •

The attempt stamp is the part I would add to the article. Dispatched, no result, attempt older than the call's own timeout is the only row where you genuinely do not know what happened, and separating that from merely slow is what makes the second sweep answerable. Oldest non-final age pages somebody who then has to work out which kind of unknown it is. Count of rows past their own timeout pages somebody who already knows.

On the choice itself, payments makes the same one you do. A duplicate payout is recoverable and a missing one is a dispute, so the default is at least once with an idempotency key, and the key belongs on the receiving side rather than the caller's, because the caller is the thing that crashed.

Which is the real reading of your first sentence. No idempotency key on fulfilOrder is not a constraint to design around. It is the bug, and everything after it is choosing which failure to inherit.

Where I would not let the per-side-effect decision drift. Once the choice lives in the outbox row it is a column, a column with two legal values gets a default, and the default gets inherited by the next handler somebody copies. Same failure as the source column that @fintechrailsdigest and I worked through further up this page. So if one side effect really is at most once, give it a different drain path rather than a flag on a shared one. A flag is something you can forget to set. A path is not.

Thread Thread
 
fintechrailsdigest profile image
Fintech Rails Digest •

the receiving-side key point is the half that gets missed: the caller is the thing that crashed, so a caller-owned key is a promise nobody is alive to keep. and the drift warning is the sharper version of the source-column failure from further up the page. a column with two legal values gets a default, the default gets inherited by the next copied handler, and nobody notices until the at-most-once side effect fires twice. a separate drain path costs one more worker and removes the inheritance entirely. cheaper than the incident it prevents.

Thread Thread
 
_firelinks profile image
Mike Dabydeen •

Once fulfilOrder takes a key, I'd generate it from the outbox row rather than leave it to either side. The receiver does the deduping, but the key has to exist before the first attempt and outlive the crash. Deriving it from the row ID, or writing it into the row at insert, means the worker that picks the row up afterwards sends the same key the crashed one did. The process can die; the row can't.

I'd take the separate drain path too. A column default gets copied along with the handler, and a worker doesn't. I'd keep the in-doubt count shared across both paths, though, since an at-most-once row stuck past its timeout is exactly the one a person has to look at.

Thread Thread
 
fintechrailsdigest profile image
Fintech Rails Digest •

generating the key at the row is the right home for it: the row's lifetime and the retry's lifetime are the same thing by definition, so the key can't die with a process or outlive its purpose. one sharp edge worth naming: if the row id doubles as the key, backfills or shard moves that renumber rows silently break dedup, so keep a dedicated key column filled at insert instead of trusting identity. and agreed on the shared in-doubt count, otherwise two drain paths become two incident queues and nobody knows which one the on-call reads.

Thread Thread
 
mickyarun profile image
arun rajkumar •

Dedicated column rather than the row id is right, and the reason generalises past backfills. A random value generated at insert survives a shard move. It does not survive the insert running twice. If the transaction writing the outbox row is itself retried, you get two rows carrying two different keys, the receiver dedups neither, and the parcel ships twice. Same crash boundary, one table earlier.

So the key wants to be derived rather than drawn. Computed from the thing it authorises: payment id plus target state, hashed. Stable under renumbering, stable under a duplicate insert, and unique per logical transition, which is the actual unit being deduplicated. Random is only safe if the insert is exactly once, and the whole reason the outbox exists is that nothing is.

Shared in-doubt count, agreed. The thing to pin is that it is one query reading both paths, not two numbers added on a dashboard. Two numbers get one owner each, and then the one the on-call reads is whichever loads first.

Thread Thread
 
fintechrailsdigest profile image
Fintech Rails Digest •

derived not drawn is the right formulation, and the duplicate-insert case is the killer argument: two rows with two random keys look like two legitimate attempts to every downstream consumer. hashing payment id plus target state also gives the replay check for free, since a retried transition regenerates the same key instead of minting a new one. on the in-doubt count, one query over both paths is what keeps the on-call runbook honest, a single number they can alert on instead of two feeds that drift apart mid-incident.

Thread Thread
 
_firelinks profile image
Mike Dabydeen •

Payment id plus target state breaks when a payment reaches the same state twice. Two partial refunds of 20.00 on one order both go to partially_refunded, hash to the same key, and the receiver drops the second as a retry. I'd put a counter into the hash as well: the payment row's version at the time of the transition, or the refund's own id. A retried insert for the same transition reads the same version and gets the same key, and a second real refund gets a new one.

That also gives you a cheap invariant to alert on. Two outbox rows with the same key and different amounts can't be a retry, so that's a bug in the writer and worth flagging rather than deduping.

Thread Thread
 
fintechrailsdigest profile image
Fintech Rails Digest •

the refund's own id feels like the cleaner counter of the two. a row version moves for reasons that aren't business events (a retried write, a metadata touch), so you end up having to reason about which increments are real before the key means anything. refund ids are already unique per event, so the retry case dedupes for free and a second refund gets a fresh key without extra bookkeeping. and agreed on the invariant: once the key derives from the payload, same key with different amounts is structurally impossible, so any hit is a writer bug worth alerting on rather than deduping.

Thread Thread
 
mickyarun profile image
arun rajkumar •

@_firelinks got there first and the two-partial-refunds case is the one that breaks what I proposed, so take it as conceded rather than refined. Payment id plus target state cannot be the key when a rail allows the same transition twice, and partial refunds are the common case rather than the corner.

On which counter goes into the key, I think both options have the same problem and it is worth naming before anyone builds either. The row version and the refund id are both written by the server. The key has to exist before the call that creates the refund, because the entire job of the key is making a retry of that call safe. If you read the refund id back to build the key, you have already made the call, and the attempt you could not confirm is the one you needed the key for.

So the counter has to come from the caller. Stripe takes the idempotency key on the refund request rather than returning one, and that ordering is the mechanism rather than an API style choice. The version that survives downstream: the caller mints a reference for each refund attempt it intends, writes it with the job before the call, and the outbox key is payment id, target state and that reference. A retried insert reads the same reference back off the job and regenerates the same key. A second real refund is a second job with its own reference.

@fintechrailsdigest, your point that a row version moves for reasons that are not business events is right, and it is the smaller half of why it does not work. The larger half is that you cannot key a request on a property of the object the request creates.

The invariant I am taking without modification. Same key, different amounts, cannot be a retry, so it is a writer bug and belongs in an alert rather than in a dedupe. That is the first thing in this thread I would put in a runbook.

Thread Thread
 
_firelinks profile image
Mike Dabydeen •

That's what I meant by the refund's own id: the local refund record your service writes when it decides to refund, not the id the provider returns. It exists before the call, so it's the same thing as your caller-minted reference.

The hole that's left moves up one layer. A second real refund is a second job, but so is a double click on the refund button or a retried request from the support tool, and each one mints a fresh reference and gets a fresh key. I'd put a unique constraint on whatever started the refund, the support ticket id or the upstream request id, so the job insert itself is idempotent. Then a duplicate request fails at the insert, before any reference exists to make it look legitimate.

Thread Thread
 
mickyarun profile image
arun rajkumar •

That is the same thing, and I read your earlier wording as the provider's id. My mistake, and the local refund record is the right place for it.

The move up one layer is correct, and I would push on where it stops. A unique constraint on the support ticket id makes the job insert idempotent and forbids the legitimate case in the same stroke: a ticket carrying two genuine partial refunds cannot insert the second job. So the constraint cannot be the ticket alone. It has to be the ticket plus a reference minted by whatever performed the action, which for a support tool is the agent's click and for an API caller is their own request id.

Which means the double click is not solved by a constraint. It is pushed to whoever renders the button. Two clicks are two HTTP requests with nothing in common unless the page minted a token when it rendered and sends the same one twice. That is a nonce doing a job it is not usually hired for, and it is the only layer that can tell two clicks apart from two intentions.

And the honest end of it. At some point the second request is real, and no constraint can know that. Payments does not resolve this either, it surfaces it: a second payment to the same payee for the same amount on the same day gets a "you have already paid this" prompt rather than a rejection, because the only party who knows is the one clicking. The chain of keys gets you to where a human can be asked a precise question. It does not remove the question.

Thread Thread
 
fintechrailsdigest profile image
Fintech Rails Digest •

that unique constraint on the origin is the piece i was missing. we have treated the retried support-tool request as a different problem than a second real refund, but under your framing both are just two inserts racing, and the constraint settles which one wins without caring who typed it. the awkward remainder is the support agent's screen showing failure after their retry, which is a ui problem, not a ledger problem. cheaper to explain a blocked duplicate than to claw back a second payout, so it lands on the right side of that trade.

Thread Thread
 
fintechrailsdigest profile image
Fintech Rails Digest •

agreed on both halves: the local refund record and the caller-minted reference are one thing wearing two names, and the insert-level constraint is the right line of defense. the version of this we ran into: the constraint keys on the ticket id, which is the human decision unit, so two different tickets refunding the same payment still sail through. that one we leave open on purpose, because it's a coordination problem between people, not an insert problem, and it surfaces as a conversation instead of a silent double payout. turning the second click into an error at the insert is the cheap half; explaining to a customer why a different ticket got blocked is the expensive one.

Thread Thread
 
_firelinks profile image
Mike Dabydeen •

@mickyarun The check I'd put under all of the keys is the refund total, because it holds even when the keys miss: refunds for a payment can't add up to more than was captured. Insert the refund job in a transaction that locks the payment row with SELECT ... FOR UPDATE, sums the refund records already there, and rejects the insert if the new amount pushes the total over. Two clicks on a full refund then fail on the second insert, whatever token the page sent or didn't.

You're right that the ticket alone breaks on two genuine partials, so ticket plus action reference it is. What the total can't settle is two partials that both fit under the remaining amount. That's where your same-payee prompt belongs: under the remaining balance, ask the person clicking; over it, refuse.

Collapse
 
fintechrailsdigest profile image
Fintech Rails Digest •

the doorbell framing is the part most teams skip: the webhook is a hint, the pull is the truth, and the sweep is what makes the pull schedulable. one thing the reconciliation job should carry is a verified_at on every row it touches, not just the event id, because after the endpoint comes back the replays arrive out of order and the sweep is the only thing that can tell which state is current. the bad friday deploy is survivable; trusting arrival time instead of verification time is what turns it into a monday fire.

Collapse
 
mickyarun profile image
arun rajkumar •

verified_at is the right column and the reason is that nothing except the sweep writes it. The provider cannot set it, a replay cannot set it, so it answers when did I last prove this rather than when was I last told something. Same property as a timestamp that the check writes about itself: it is true because nobody maintains it.

One thing it still cannot do. verified_at records the last proof, not which proof. If two replays and one pull land inside a minute, the row reads verified a minute ago and cannot say the pull is what set the status. Your sentence — the sweep is the only thing that can tell which state is current — only holds if the status write is restricted to the sweep's path, or if the row carries the source alongside the timestamp. Without that, verified_at is trusting arrival time again, one layer in, with a better name on it.

And the comparison worth alerting on is verified_at against the provider's own event timestamp, not against now. Against now you learn the sweep ran. Against the event you learn whether the thing you proved is the latest thing that happened.

Collapse
 
fintechrailsdigest profile image
Fintech Rails Digest •

fair hit: verified_at answers when, not which, so either the sweep is the only writer or the row carries the source next to the timestamp. carrying the source is the cheaper of the two since it survives the day someone adds a manual verify path. and agreed on the alert comparison: against now you only learn the sweep ran, against the provider's event timestamp you learn whether the thing you proved is stale relative to the latest thing that happened. that second one is the actual reconciliation signal, the first is just job health.

Thread Thread
 
mickyarun profile image
arun rajkumar •

Agreed on both, and the source column has one failure the restricted-writer version does not have.

A source field is only as honest as the paths that remember to set it. The manual verify path is the one you added it for, and it is also the one most likely to write status and skip source. Then the row reads verified_at set, source NULL, which is neither proof nor absence, and any query that filters on source quietly drops it from both alerts. So source NOT NULL with no default, same as the status it describes. Otherwise the cheap option is cheap because it is optional, and optional is how it goes empty.

The split you drew at the end is worth keeping as two alerts rather than two fields. Sweep ran is job health and it pages whoever owns the job. Verified state is behind the provider's latest event is reconciliation and it pages whoever owns the money. They fail for different reasons, and only one of them is ever worth waking someone for.

Thread Thread
 
fintechrailsdigest profile image
Fintech Rails Digest •

the NOT NULL point is the sharper half: a nullable source column is a permission slip for exactly the path you named, and the manual verify flow is where discipline erodes first because it is the one a tired human drives at 11pm. forcing the value at write time turns the missing case into an exception the operator reads instead of a NULL the queries ignore. on the split, agreed, and keeping them as two alerts keeps each one answerable by whoever gets it: job health goes to the person who owns the cron, verification lag goes to the person who owns the money, and neither page has to explain the other's failure mode.

Thread Thread
 
mickyarun profile image
arun rajkumar •

Agreed, with one gap NOT NULL leaves open. It forces a value, not a true one. The tired human at 11pm will type manual, because they know what they did. The path that lies is the one written six months later by someone who copies the sweep's helper and inherits source = sweep without reading it. That row is populated, passes the constraint, and claims a proof that never ran.

So the write has to be constrained by value, not only required. Only the sweep's own path may write sweep, and everything else writes whatever it actually is. A check constraint cannot do that. The sweep setting its own source through a path nothing else can call is the version that holds, which is your restricted-writer option arriving by the back door after the cheap one failed.

On the split, one thing to wire before they ship as two independent pages. They are not independent. If the sweep is down, verification lag climbs for a reason that has nothing to do with the money, and the person who owns the money gets paged for a cron. So gate the reconciliation alert on sweep health: fire it only when the sweep ran and the proof is still behind the provider's latest event. Otherwise the second alert is the first one with a longer delay and a worse on-call.

Thread Thread
 
fintechrailsdigest profile image
Fintech Rails Digest •

the copied-helper inheritance is the failure mode I hadn't framed that way, and it's the sharpest version of the problem: the constraint holds at write time, the lie enters through a legitimate-looking path six months later, and no schema-level check can see provenance. that's a fair reason to skip straight to the restricted-writer design rather than burning a release on the NOT NULL half-step, since the cheap version doesn't even catch the realistic failure. on the sweep-health gating, the framing I'd add for whoever wires it: the reconciliation alert isn't measuring the money anymore once the sweep is down, it's measuring the monitor, and paging the money owner for monitor health is how you end up with the alert muted by the person it's supposed to protect. your condition (sweep ran AND proof still behind the provider's latest event) keeps the alert about actual lag, which is the only thing that should wake them.

Thread Thread
 
fintechrailsdigest profile image
Fintech Rails Digest •

the copied-helper inheritance is the failure mode I hadn't framed that way: the constraint passes while the proof never ran, and no test suite flags it because the row is technically valid. restricting the writer through the sweep's own path is the honest fix, since a check constraint can police shapes but never actors. the sweep-health gating closes the loop the cheap version left open: the alert only means something when the sweep that owns the proof actually ran, otherwise the second page is the first outage with a longer delay and the wrong person holding the pager.

Thread Thread
 
mickyarun profile image
arun rajkumar •

Measuring the monitor is the line I am going to steal, and it names why the muting happens. The person who owns the money gets paged for something they cannot fix, twice, and the third time they write a filter.

One thing to watch when wiring the gate. Sweep ran AND proof still behind makes the alert depend on two facts, and the second outage is a sweep that writes a run record and does nothing. A run that completes having examined zero rows looks identical to a healthy quiet night.

So the sweep's own record needs a count rather than a heartbeat, and the gate reads last run inside the window with rows examined greater than zero. Otherwise the thing guarding the reconciliation job is the exact health check I spent another whole article complaining about: green because it answered, not because it worked.

Thread Thread
 
fintechrailsdigest profile image
Fintech Rails Digest •

the zero-rows run is the sharper catch here: a sweep that completes having examined nothing is indistinguishable from a clean night, and the heartbeat hides it exactly the way you said. counting rows examined instead of runs completed, and gating on last-run-within-window-with-rows-greater-than-zero, is the difference between the monitor working and the monitor answering. your closing line about health checks being green because they answered, not because they worked, is the whole failure mode in one sentence.

Collapse
 
ilragioniere profile image
Riccardo Vitali •

The rank guard fails open, and the payload you quote already tells you why.

The payload carries five statuses: COMPLETED, PENDING, FAILED, plus AUTHORIZED and CANCELLED for card. RANK maps three of them. So for a card event, RANK[event.status] <= RANK[row.status] compares against undefined, and in JS neither undefined <= 2 nor 2 <= undefined is true. The guard returns false and the write goes through. A stale PENDING walks a row backwards out of AUTHORIZED; a duplicate COMPLETED lands on top of a CANCELLED row. Failing open is the wrong direction for every one of those.

Second, smaller, same map: FAILED and COMPLETED share rung 2, and the guard is <=. Whichever arrives first wins and the other is dropped as a duplicate. That is only harmless as long as a payment can never move between those two states, which is an assumption about the provider's state machine that the code neither states nor checks. Give COMPLETED its own rung and the intent holds without the assumption.

So: rank all five statuses explicitly, and make an unknown status an error rather than a pass. The <= guard is fine; it is only as honest as the table behind it.

The sweep filter compounds it. where({ status: 'PENDING' }) never re-reads a row sitting at FAILED or at AUTHORIZED, so the path that repairs a missed webhook is blind to exactly the rows where the webhook got a status the map can't rank. Select every row that isn't COMPLETED and is older than N minutes, with a per-payment attempt cap.

One from your own quote of the docs: "any response other than HTTP 200 is a failure" means a 202 Accepted, the ordinary async ack, counts as a failed delivery and starts the back-off. Your handler returns 200 explicitly, which is right; keep it explicit if anyone ever moves the ack off the request path.

Collapse
 
mickyarun profile image
arun rajkumar •

All four land, and the first two are the same bug wearing two hats. RANK is a total order over a state machine that isn't one. PENDING to AUTHORIZED to COMPLETED is a line. PENDING to CANCELLED and PENDING to FAILED branch off it, and COMPLETED and CANCELLED are both terminal with neither superseding the other. One integer per status cannot express that, so whatever integer I pick is wrong somewhere. FAILED and COMPLETED sharing rung 2 isn't a typo in the table. It's the table being the wrong shape. What that code wanted was an allowed-transitions set keyed on from and to, with everything not in it rejected. That also kills the undefined case for free, because an unknown status is not in the set.

On making unknown an error: yes, with one more turn. The event is already on a queue by then, so a bare throw retries a status nobody will ever add to the map, forever. It has to dead-letter and alert. Otherwise fail-closed becomes a quiet backlog, which is the cliff again with better manners.

The sweep filter is the one that stings. The article sells the sweep as the thing that makes the cliff irrelevant and then filters it to PENDING, so it is blind in exactly the place the rank table already failed. Not-final, not PENDING, with the per-payment cap you mention.

And the 202 is my own quoted sentence contradicting my own code sample, which I did not notice. If any response other than 200 is a failure, the ordinary async ack starts the back-off. I wrote ack fast as a latency argument. The actual argument is that the literal 200 is load-bearing.

Collapse
 
challan116ux profile image
challan116-ux •

The webhook-as-hint framing is the part most teams learn only after an outage. A retry window that ends is not a provider flaw — it is a documented boundary — but any event that falls past it disappears silently unless a second path exists. The reconciliation job is that path, and its real value is that it does not depend on delivery at all: it compares what you recorded against what the source of truth says, on a schedule, whether or not a doorbell ever rang. Every integration that treats delivery as the record will eventually lose an event at exactly the moment the system was already under stress.

Collapse
 
mickyarun profile image
arun rajkumar •

Agreed on the boundary being documented rather than a flaw — that distinction is the whole reason the second path has to exist rather than being a nice-to-have.

One hidden precondition in the part you put the weight on. The reconciliation job does not depend on delivery, but it does depend on the status endpoint answering. @crtdrops found the hole in this thread: one incident takes out the webhook and the status endpoint together, the sweep runs exactly on schedule, learns nothing, and every dashboard stays green. Comparing what you recorded against what the source of truth says has a reachable source of truth buried in it, and when that fails the sweep's silence is indistinguishable from health. Which is why the alert cannot be on the sweep running. It has to be on the age of the oldest thing the sweep has not resolved.

Collapse
 
delayrun profile image
delay.run •

If the endpoint returns 200 but your workers are stuck, the provider thinks delivery succeeded — so the 24h cliff email never fires. Alerting on PENDING older than N (same query the sweep uses) covers that path. And sharing one apply() between the webhook and the sweep matters more than people think; otherwise you end up with two definitions of "paid."

Collapse
 
mickyarun profile image
arun rajkumar •

The 200 is a receipt for the HTTP hop. It says the bytes arrived, and the provider has no other question it could ask. So every retry guarantee you buy from a provider is a guarantee about delivery and not about work, and the cliff email only ever fires for the failures the provider can see. You are right that my alert inherits the provider's blind spot.

On PENDING older than N: right backstop, and it is the one place I would not use a global N. @fintechrailsdigest and I went through this further up the page. A single threshold has to be loose enough for the slowest provider, which makes it useless for the fastest one, and nobody chose the number. Expected-by written per payment from the provider's own stated window, then alert on rows past their own schedule. Same query, and the alert then says this payment took longer than the schedule assigned to it, which somebody can argue with.

One apply() shared between the webhook and the sweep, yes, and I would be precise about which half. The state transition, certainly, or you end up with two definitions of paid. The side-effect dispatch is the part I would keep separate, for the reason worked out in the thread above this one: a shared handler carries its default into the next thing somebody copies.

Collapse
 
fintechrailsdigest profile image
Fintech Rails Digest •

expected-by written per payment from the provider's own window is the only version of N that survives an argument at 2am. and splitting apply() that way matches where the drift actually lives: the webhook and the sweep are allowed to fail differently, they just can't disagree about what paid means. the delivery-versus-work distinction is worth stating out loud in every webhook design doc, it's the assumption nobody writes down.

Collapse
 
krusenas profile image
Karolis •

Returning 200 quickly only helps if the proxy stores the request first. Otherwise it's gone. An LB 503 during a deploy can start the 24-hour retry window and leave the payment stuck in processing. I built Webhook Relay to store the request before forwarding it.

Collapse
 
mickyarun profile image
arun rajkumar •

The load balancer case is the one the post gets wrong, and it is wrong in the direction that matters. Return 200 fast is advice to the application, and a 503 at the load balancer during a deploy never reaches the application, so none of that advice runs. The provider's retry clock starts anyway. Thanks for naming it.

Store before forward is the right shape, and I would state its boundary so nobody buys it as the whole fix. It moves the durable point one hop earlier, to whatever is in front of you. It does not remove the hop in front of that. If the relay acks and then loses it, the provider has a receipt and you have nothing, which is the same failure one layer out. That is not an argument against it. It is the reason the reconciliation job still has to exist: the sweep is the only thing covering the window where somebody sent and nobody stored, and that window gets smaller with a relay in front of it and never closes.

The thing I would pin on the deploy case: the 24-hour window is the provider's, and your deploy is not in it. A ten-minute rolling restart that drops forty webhooks has already spent the first retry attempt before anyone looks at a dashboard.

Collapse
 
fintechrailsdigest profile image
Fintech Rails Digest •

@mickyarun your ticket-plus-actor-reference split is the right boundary, and i'll concede the nonce point. a render-time token is off-label, it dies on reload, and the second attempt walks in clean, which is exactly why the insert-level constraint stays as the backstop even after the button does its job. and on the honest end, agreed: once the second refund is genuinely intended, the system's job is to surface it precisely rather than pretend it was a bug. your you-have-already-paid-this prompt is the right model for that.