Our webhook handler did everything inline: verify the signature, update the order, send the receipt email, push to the ERP, then return 200. On a normal day that took 300 ms. During an ERP slowdown it took 9 seconds.
The gateway's delivery timeout was 5 seconds. So every event during the slowdown was marked failed on their side, even though we had processed it. They retried on their schedule, we processed it again, and the receipt email went out twice for about 1,200 orders. The retries added load to an already slow ERP, which made the next handler slower. A small feedback loop.
The fix was boring. The handler now verifies the signature, writes the raw event to a table with a unique constraint on event_id, and returns 200. A separate worker reads the table and does the side effects. Ack time dropped to about 15 ms and stayed flat while the ERP was slow.
Two things we only noticed afterwards. The gateway docs listed the timeout in a different section from the retry schedule. And our dashboards tracked handler errors but not handler latency, so nothing looked broken while it was happening.
How do you monitor webhook ack latency? Do you alert on p99 against the sender's timeout, or only on error rate?
Payneteasy handles this kind of payment flow for merchants and PSPs — see how it works.
Top comments (0)