We shipped a routine release on a Tuesday. Rolling restart, about 90 seconds where the webhook service returned 502 from the load balancer. Nobody noticed, because health checks were green again right after.
Two days later, support asked why a batch of orders was stuck in "pending" even though the customers had been charged.
The cause: the gateway retries failed deliveries on a schedule, and after enough consecutive failures it marks the endpoint as degraded and stops sending new events until someone re-enables it. Our 90-second window hit a burst of events, each one burned through its first retries, and the endpoint tripped. Everything after that point went nowhere. No alert fired on our side, because from our side there was simply no traffic.
What we changed:
- Alert on the absence of webhooks, not only on errors. If zero events arrive for 15 minutes during business hours, page someone.
- Keep a daily job that pulls recent transactions through the API and compares them with what the webhooks delivered. The webhook is a hint, the API is the source of truth.
- Drain the webhook service last in the rollout, and never return 5xx while shutting down. Accept, queue, return 200.
Do you monitor for silence, or only for failures? I suspect most setups only cover the second one.
Payneteasy handles this kind of payment flow for merchants and PSPs — see how it works.
Top comments (0)