DEV Community

GitHub doesn't retry webhooks. So I gave my tunnel a database.

Danish ali siddiqui on August 09, 2026

Built solo over a weekend for the WeMakeDevs × Zerops challenge. There's a live gateway you can poke at, and a section near the end about what it c...
Collapse
 
eleftheriabatsou profile image
Eleftheria Batsou

Thanks for participating in the hackathon!

I'm Elef, Dev Advocate at Zerops :)

Collapse
 
bigachiever profile image
Danish ali siddiqui

Thank you, Elef! It was my pleasure! Appreciate the opportunity :)

Collapse
 
ofri-peretz profile image
Ofri Peretz

The Shopify detail is the one that would wake me up at night — not "event dropped" but "subscription silently deleted." That's a quiet mutation to shared state that your monitoring almost certainly doesn't cover, and by the time you notice events have stopped arriving you're debugging the wrong layer entirely. The 404 for unreserved names is a small but sharp design call too: without it, the persistence layer is a write-anywhere endpoint, and you're one viral post away from a disk exhaustion incident with no clear owner.

Collapse
 
bigachiever profile image
Danish ali siddiqui

A durable queue only protects events that reach the gateway. Provider-side subscription health needs a separate check, otherwise silence looks healthy. The reservation requirement keeps persistence from becoming anonymous storage.

Collapse
 
ofri-peretz profile image
Ofri Peretz

Persistence covers events already in the queue — it doesn't tell you whether GitHub is still sending them in the first place. I've had a subscription silently drop after a repeated 5xx during a GitHub incident: webhook deliveries stopped, my tunnel had nothing to retry because nothing arrived, and the dashboard looked green because "queue empty" and "queue caught up" render identically. Fixed it by polling the deliveries API for the subscription's last-delivery timestamp and alerting on staleness, separate from queue depth. You're right that the reservation constraint is really an admission check in disguise — it just checks the wrong side of the pipe.

Collapse
 
ida_zhang_cc496f24f8c03ef profile image
Ida Zhang

Implementing a database to handle GitHub's lack of webhook retries is a clever workaround. Wish I'd thought of that before I spent hours troubleshooting missed notifications.

Collapse
 
bigachiever profile image
Danish ali siddiqui

Thank you!

Collapse
 
mudassirworks profile image
Mudassir Khan

'a tunnel is a pipe, not a mailbox' is the cleanest way i've seen this explained. we lost a stripe event at 3am and spent half a day debugging what turned out to be a simple timing gap — no retry, no trace.

the 202 and replay approach is the right call. one thing i'm curious about: how do you handle ordering guarantees when held events flush? if i have 3 events queued and my handler crashes partway through, does the gateway know which ones got through?

Collapse
 
bigachiever profile image
Danish ali siddiqui

Thank you !

Ordering is strict per tunnel: oldest first, one at a time, and a drain stops rather than skips if it loses a row.

Handler never answered? Stays held, redelivers on reconnect. Answered 500 then died? Counts as delivered — recorded and replayable, not auto-retried. That's the gap. Dedupe on X-Doorbell-Id

Collapse
 
jarrodcabarubio profile image
Jarrod Cabarubio

Slick move on the db. I can dig it 🤙

Collapse
 
bigachiever profile image
Danish ali siddiqui

Thanks, Jarrod! Glad you dig it 😀

Some comments have been hidden by the post's author - find out more