DEV Community

Danish ali siddiqui
Danish ali siddiqui

Posted on

GitHub doesn't retry webhooks. So I gave my tunnel a database.

Built solo over a weekend for the WeMakeDevs × Zerops challenge. There's a live gateway you can poke at, and a section near the end about what it can't do.

I was testing a GitHub integration with a tunnel open to my laptop. I shut the lid to go get coffee. When I came back, the push event was gone.

Not delayed. Not sitting in a queue somewhere. Gone.

I figured I'd misconfigured something. I hadn't. From GitHub's own docs:

"GitHub does not automatically redeliver failed deliveries."
docs.github.com

One failed delivery and it's your problem, unless you open the UI and redeliver by hand.

I checked other providers, expecting GitHub to be unusual. It isn't:

Provider What happens when your tunnel is down
GitHub No automatic retries at all
Stripe (test mode) 3 attempts over a few hours
Shopify 8 attempts over 4 hours, then deletes your webhook subscription

Read the last one again. Shopify doesn't just give up on the event. It unsubscribes your app.

The part nobody tells you about tunnels

Here's what I hadn't thought about properly. A tunnel turns your laptop into production infrastructure for somebody else's system.

Senders don't wait for you. A customer checks out at 2am. CI finishes after you've left for lunch. Your wifi drops for thirty seconds on a train.

When that happens the sender gets a 502. It burns one of the three or four attempts it was ever going to make, and then the event is gone.

Someone is about to say "ngrok already has replay." It does, and it's good. But ngrok can only replay a request it already saw, which means you were connected when it arrived. The ones that hurt are the ones that arrive when you aren't.

That's structural, not a missing feature:

A tunnel is a pipe, not a mailbox. If nothing is listening at your end, the request has nowhere to go.

So I built the mailbox. It's called Doorbell, and it's a tunnel with a database in the path.

You can break it right now

You have to take my word for everything else in this post. This part you can check yourself.

I'm running a public gateway — the always-on server that holds tunnels open. It has a tunnel named shop with nothing connected to it, which is the same state as your laptop with the lid shut.

On a phone? Just open the live page and watch the held right now counter. At a keyboard, send it a webhook:

curl -i -X POST https://gw-2ad0-3000.prg1.zerops.app/t/shop/hook -d '{"n":1}'
Enter fullscreen mode Exit fullscreen mode
HTTP/2 202
Enter fullscreen mode Exit fullscreen mode

202 Accepted, where you'd expect a 502.

There is no laptop on the other end. Your request is now a row in Postgres, and it gets delivered the moment someone connects. Reload that live page and the counter is one higher. That's yours.

Now try a name nobody reserved:

curl -i -X POST https://gw-2ad0-3000.prg1.zerops.app/t/zzrandom99/hook -d '{"n":1}'
Enter fullscreen mode Exit fullscreen mode
HTTP/2 404
Enter fullscreen mode Exit fullscreen mode

That 404 matters as much as the 202. Only reserved names get held. If any name worked, anyone could fill my database by inventing URLs, and "we store everything" would just mean "free disk for strangers."

And here's what you get back when you reconnect:

$ doorbell -name demo 3000
  https://gw-2ad0-3000.prg1.zerops.app/t/demo/
  → forwarding to 127.0.0.1:3000

  ▲ requests held while you were away are arriving now
  ✓ 11:11:23 POST   /hooks/github    200  held 4s
  ✓ 11:11:23 POST   /hooks/github    200  held 2s
  ✓ 11:11:24 POST   /hooks/github    200  held <1s
Enter fullscreen mode Exit fullscreen mode

Oldest first, each showing how long it waited.

Before you get excited: is this for you?

One thing you should know 30 seconds in rather than 10 minutes in.

A held webhook will fail a signature check. GitHub and Stripe sign each delivery so your app can prove it really came from them. Doorbell strips anything that looks like a signing header before writing the row, so the signature is gone by the time the request reaches you.

That's a deliberate trade. I'd rather hand you a request whose signature no longer verifies than keep someone's live signing secret sitting in my database. Held requests carry an X-Doorbell-Replay header, so you can skip verification when you see it.

If your handler verifies signatures and you can't branch on that header, this tool will annoy you. Better you know now.

Two more things, and then the good part:

  • You run this yourself. The gateway above is mine, for trying it. For real use you deploy your own from one YAML file, and nobody else's servers sit in the path of your traffic.
  • It's a development tunnel. Don't put it in front of production traffic.

What 202 actually means

This is where a lot of "reliable webhook" tooling quietly lies to you.

202 Accepted does not mean your app handled it. It means: I have taken responsibility for this request. The row is on disk before the sender gets any answer at all.

Two things make that promise real.

Ordering. Each tunnel's queue drains oldest first, one at a time. That's the only order your handler can make sense of. If the delete arrives before the create, your code does the wrong thing.

No duplicates. If you reconnect twice quickly, two drains could race for the same stored request and send the same webhook twice. A duplicate payment notification is somebody's real problem.

So the database decides, not the application. Claiming a request and marking it taken happen in a single SQL statement, so only one drain can ever win it. There's a test that runs eight of them at once to prove it:

if won != 1 {
    t.Fatalf("%d of %d claimers won the same row; every extra winner is a webhook delivered twice", won, racers)
}
Enter fullscreen mode Exit fullscreen mode

The whole thing rests on one function

When you run doorbell 3000, the CLI opens a single outbound connection to the gateway and holds it open. Nothing listens on your machine. Nothing gets opened on your router.

It's like calling a support line. They can't call you. But while you're on the line, they can talk to you.

Each incoming request becomes its own stream on that one connection, using yamux to run many conversations down a single socket. The proxying itself is just Go's standard library, with one substitution:

Transport: &http.Transport{
    // The whole trick: instead of dialling a network address, open a
    // new multiplexed stream on the socket the laptop already holds open.
    DialContext: func(_ context.Context, _, _ string) (net.Conn, error) {
        return session.Open()
    },
},
Enter fullscreen mode Exit fullscreen mode

Swap dial an address for open a stream and you're done. Chunked bodies, keep-alive and WebSocket upgrades all still work, because httputil.ReverseProxy is doing the actual HTTP.

The bug I shipped, and how I found it

Doorbell has a dashboard. It shows the request and response bodies it captured, and it can re-send any stored request to whoever's laptop is connected. Both were behind a token. Good.

The problem: that same token also guarded the port the CLI dials to open a tunnel. One check was answering two completely different questions. May you open a tunnel? and may you read everyone's payloads? ran through the same line of code.

So anyone I let tunnel through my gateway could also read every captured body on it, and re-send any of them.

The file's own doc comment said the opposite:

"Tunnels themselves are not gated by this — only the surfaces that expose captured request and response bodies."

The comment described the intent. The code did something else. Nobody catches that in review, because both call sites look correct on their own.

The fix was to split them:

// tokenOK gates the control port: may this client open a tunnel at all?
func (g *gateway) tokenOK(presented string) bool {
    if g.cfg.clientToken == "" {
        return true
    }
    return subtle.ConstantTimeCompare(...) == 1
}

// adminTokenOK gates the dashboard. Kept separate on purpose.
func (g *gateway) adminTokenOK(presented string) bool {
    return subtle.ConstantTimeCompare(...) == 1
}
Enter fullscreen mode Exit fullscreen mode

Then I nearly made it worse.

The dashboard also accepts ?token= in the URL, and that path called tokenOK too. Look at what tokenOK does when no client token is set: it returns true for anything. My public gateway has no client token, on purpose, so anyone can tunnel. Which means ?token=literally-anything would have opened the dashboard to the first person who tried it.

I caught it before shipping. There's now a test whose only job is to fail if it comes back:

func TestGuessedQueryTokenCannotUnlockTheDashboard(t *testing.T) {
    g := gatewayWithTokens("", "operator-secret")
    if got, _ := reachedOperator(g, req("/dashboard?token=anything")); got {
        t.Fatal("?token=anything unlocked the dashboard on a public gateway")
    }
}
Enter fullscreen mode Exit fullscreen mode

Two privileges, two tokens. Opening a tunnel is a small thing to be allowed to do. Reading what went through it isn't.

Why this can't run on most of the internet

Holding that connection open gives you two requirements you can't negotiate away:

  1. A raw TCP port on the gateway. Plain TCP, not HTTP, because the client speaks its own protocol on it.
  2. A process that never sleeps. It sits there holding the socket for hours.

Those two rule out every serverless platform, by definition. A function stops existing the moment it returns a response, so nobody is left holding the line. And a platform that handles HTTP for you can't give you a port that isn't HTTP.

I'll be straight with you, because a Fly.io user is already typing: Fly.io gives you a raw TCP port too. So does Railway. Zerops isn't magic here.

But the port was never the hard part. The hard part is the database. It's storing captured webhook bodies, so it must never face the internet, and I didn't want to spend a hackathon weekend wiring up a private network by hand to make that true.

Paste one YAML file into Zerops and about ninety seconds later you have a gateway, a Postgres, a Valkey and a private network between them, in your own account. (Valkey is a Redis fork. It tracks which gateway container currently owns which tunnel, so this still works when there's more than one.)

One assumption sat underneath all of it: that a raw TCP port on Zerops is genuinely reachable from the public internet. The docs said ports 10–65435 were available. But documented isn't measured, and if that line was wrong the whole project was dead. So before writing any of the gateway, I deployed a thirty-line echo server on port 7000 and dialled it from my laptop. It answered. That throwaway test is still sitting in my account next to the real project.

One honest note: that YAML can't open the raw port for you. Zerops' import format has no field for it, so it's still one click in the dashboard afterwards.

What still doesn't work

The signature thing above is the big one. Two more:

The CLI needs IPv6 to reach my public gateway. Webhooks don't — every curl in this post runs over ordinary IPv4, because that's HTTP going through Zerops' shared load balancer. A raw TCP port can't use that path, and Zerops won't publish raw ports on a shared IPv4. That leaves IPv6 or a paid dedicated IPv4.

Custom domains are written and unit-tested but have never run end to end. Proving that needs a real domain and a wildcard certificate, which I didn't have.

Things you're about to ask

How long are requests held? Until you reconnect. The cap is 200 per tunnel and 1 MB per body. Past 200 the oldest undelivered ones get dropped, because without a cap the table becomes a way to fill someone's disk.

Is the ordering per tunnel or global? Per tunnel. Each queue drains on its own.

Doesn't that eight-way race contradict the ordering? No. That's the test, hammering one row on purpose to prove only one claimer wins. The real drain loop is sequential.

What if two people connect with the same tunnel name? A reservation belongs to whoever made it. The second one gets the name "shop" is reserved by someone else.

Try it

The two curls at the top take about thirty seconds and need nothing installed. For your own tunnel:

go install github.com/BigAchiever/doorbell/cmd/doorbell@latest
doorbell -name pick-a-name 3000
Enter fullscreen mode Exit fullscreen mode

CI fails the build if the Postgres and Valkey test suites skip, because a green run that quietly tested nothing is worse than a red one. That's the check I'm proudest of.

If you break it, I'd genuinely like to know how.

Top comments (0)

Some comments have been hidden by the post's author - find out more