DEV Community: Danish ali siddiqui

GitHub doesn't retry webhooks. So I gave my tunnel a database.

Danish ali siddiqui — Sun, 09 Aug 2026 19:42:34 +0000

Built solo over a weekend for the WeMakeDevs × Zerops challenge. There's a live gateway you can poke at, and a section near the end about what it can't do.

I was testing a GitHub integration with a tunnel open to my laptop. I shut the lid to go get coffee. When I came back, the push event was gone.

Not delayed. Not sitting in a queue somewhere. Gone.

I figured I'd misconfigured something. I hadn't. From GitHub's own docs:

"GitHub does not automatically redeliver failed deliveries."
— docs.github.com

One failed delivery and it's your problem, unless you open the UI and redeliver by hand.

I checked other providers, expecting GitHub to be unusual. It isn't:

Provider	What happens when your tunnel is down
GitHub	No automatic retries at all
Stripe (test mode)	3 attempts over a few hours
Shopify	8 attempts over 4 hours, then deletes your webhook subscription

Read the last one again. Shopify doesn't just give up on the event. It unsubscribes your app.

The part nobody tells you about tunnels

Here's what I hadn't thought about properly. A tunnel turns your laptop into production infrastructure for somebody else's system.

Senders don't wait for you. A customer checks out at 2am. CI finishes after you've left for lunch. Your wifi drops for thirty seconds on a train.

When that happens the sender gets a 502. It burns one of the three or four attempts it was ever going to make, and then the event is gone.

Someone is about to say "ngrok already has replay." It does, and it's good. But ngrok can only replay a request it already saw, which means you were connected when it arrived. The ones that hurt are the ones that arrive when you aren't.

That's structural, not a missing feature:

A tunnel is a pipe, not a mailbox. If nothing is listening at your end, the request has nowhere to go.

So I built the mailbox. It's called Doorbell, and it's a tunnel with a database in the path.

You can break it right now

You have to take my word for everything else in this post. This part you can check yourself.

I'm running a public gateway — the always-on server that holds tunnels open. It has a tunnel named shop with nothing connected to it, which is the same state as your laptop with the lid shut.

On a phone? Just open the live page and watch the held right now counter. At a keyboard, send it a webhook:

curl -i -X POST https://gw-2ad0-3000.prg1.zerops.app/t/shop/hook -d '{"n":1}'

HTTP/2 202

202 Accepted, where you'd expect a 502.

There is no laptop on the other end. Your request is now a row in Postgres, and it gets delivered the moment someone connects. Reload that live page and the counter is one higher. That's yours.

Now try a name nobody reserved:

curl -i -X POST https://gw-2ad0-3000.prg1.zerops.app/t/zzrandom99/hook -d '{"n":1}'

HTTP/2 404

That 404 matters as much as the 202. Only reserved names get held. If any name worked, anyone could fill my database by inventing URLs, and "we store everything" would just mean "free disk for strangers."

And here's what you get back when you reconnect:

$ doorbell -name demo 3000
  https://gw-2ad0-3000.prg1.zerops.app/t/demo/
  → forwarding to 127.0.0.1:3000

  ▲ requests held while you were away are arriving now
  ✓ 11:11:23 POST   /hooks/github    200  held 4s
  ✓ 11:11:23 POST   /hooks/github    200  held 2s
  ✓ 11:11:24 POST   /hooks/github    200  held <1s

Oldest first, each showing how long it waited.

Before you get excited: is this for you?

One thing you should know 30 seconds in rather than 10 minutes in.

A held webhook will fail a signature check. GitHub and Stripe sign each delivery so your app can prove it really came from them. Doorbell strips anything that looks like a signing header before writing the row, so the signature is gone by the time the request reaches you.

That's a deliberate trade. I'd rather hand you a request whose signature no longer verifies than keep someone's live signing secret sitting in my database. Held requests carry an X-Doorbell-Replay header, so you can skip verification when you see it.

If your handler verifies signatures and you can't branch on that header, this tool will annoy you. Better you know now.

Two more things, and then the good part:

You run this yourself. The gateway above is mine, for trying it. For real use you deploy your own from one YAML file, and nobody else's servers sit in the path of your traffic.
It's a development tunnel. Don't put it in front of production traffic.

What `202` actually means

This is where a lot of "reliable webhook" tooling quietly lies to you.

202 Accepted does not mean your app handled it. It means: I have taken responsibility for this request. The row is on disk before the sender gets any answer at all.

Two things make that promise real.

Ordering. Each tunnel's queue drains oldest first, one at a time. That's the only order your handler can make sense of. If the delete arrives before the create, your code does the wrong thing.

No duplicates. If you reconnect twice quickly, two drains could race for the same stored request and send the same webhook twice. A duplicate payment notification is somebody's real problem.

So the database decides, not the application. Claiming a request and marking it taken happen in a single SQL statement, so only one drain can ever win it. There's a test that runs eight of them at once to prove it:

if won != 1 {
    t.Fatalf("%d of %d claimers won the same row; every extra winner is a webhook delivered twice", won, racers)
}

The whole thing rests on one function

When you run doorbell 3000, the CLI opens a single outbound connection to the gateway and holds it open. Nothing listens on your machine. Nothing gets opened on your router.

It's like calling a support line. They can't call you. But while you're on the line, they can talk to you.

Each incoming request becomes its own stream on that one connection, using yamux to run many conversations down a single socket. The proxying itself is just Go's standard library, with one substitution:

Transport: &http.Transport{
    // The whole trick: instead of dialling a network address, open a
    // new multiplexed stream on the socket the laptop already holds open.
    DialContext: func(_ context.Context, _, _ string) (net.Conn, error) {
        return session.Open()
    },
},

Swap dial an address for open a stream and you're done. Chunked bodies, keep-alive and WebSocket upgrades all still work, because httputil.ReverseProxy is doing the actual HTTP.

The bug I shipped, and how I found it

Doorbell has a dashboard. It shows the request and response bodies it captured, and it can re-send any stored request to whoever's laptop is connected. Both were behind a token. Good.

The problem: that same token also guarded the port the CLI dials to open a tunnel. One check was answering two completely different questions. May you open a tunnel? and may you read everyone's payloads? ran through the same line of code.

So anyone I let tunnel through my gateway could also read every captured body on it, and re-send any of them.

The file's own doc comment said the opposite:

"Tunnels themselves are not gated by this — only the surfaces that expose captured request and response bodies."

The comment described the intent. The code did something else. Nobody catches that in review, because both call sites look correct on their own.

The fix was to split them:

// tokenOK gates the control port: may this client open a tunnel at all?
func (g *gateway) tokenOK(presented string) bool {
    if g.cfg.clientToken == "" {
        return true
    }
    return subtle.ConstantTimeCompare(...) == 1
}

// adminTokenOK gates the dashboard. Kept separate on purpose.
func (g *gateway) adminTokenOK(presented string) bool {
    return subtle.ConstantTimeCompare(...) == 1
}

Then I nearly made it worse.

The dashboard also accepts ?token= in the URL, and that path called tokenOK too. Look at what tokenOK does when no client token is set: it returns true for anything. My public gateway has no client token, on purpose, so anyone can tunnel. Which means ?token=literally-anything would have opened the dashboard to the first person who tried it.

I caught it before shipping. There's now a test whose only job is to fail if it comes back:

func TestGuessedQueryTokenCannotUnlockTheDashboard(t *testing.T) {
    g := gatewayWithTokens("", "operator-secret")
    if got, _ := reachedOperator(g, req("/dashboard?token=anything")); got {
        t.Fatal("?token=anything unlocked the dashboard on a public gateway")
    }
}

Two privileges, two tokens. Opening a tunnel is a small thing to be allowed to do. Reading what went through it isn't.

Why this can't run on most of the internet

Holding that connection open gives you two requirements you can't negotiate away:

A raw TCP port on the gateway. Plain TCP, not HTTP, because the client speaks its own protocol on it.
A process that never sleeps. It sits there holding the socket for hours.

Those two rule out every serverless platform, by definition. A function stops existing the moment it returns a response, so nobody is left holding the line. And a platform that handles HTTP for you can't give you a port that isn't HTTP.

I'll be straight with you, because a Fly.io user is already typing: Fly.io gives you a raw TCP port too. So does Railway. Zerops isn't magic here.

But the port was never the hard part. The hard part is the database. It's storing captured webhook bodies, so it must never face the internet, and I didn't want to spend a hackathon weekend wiring up a private network by hand to make that true.

Paste one YAML file into Zerops and about ninety seconds later you have a gateway, a Postgres, a Valkey and a private network between them, in your own account. (Valkey is a Redis fork. It tracks which gateway container currently owns which tunnel, so this still works when there's more than one.)

One assumption sat underneath all of it: that a raw TCP port on Zerops is genuinely reachable from the public internet. The docs said ports 10–65435 were available. But documented isn't measured, and if that line was wrong the whole project was dead. So before writing any of the gateway, I deployed a thirty-line echo server on port 7000 and dialled it from my laptop. It answered. That throwaway test is still sitting in my account next to the real project.

One honest note: that YAML can't open the raw port for you. Zerops' import format has no field for it, so it's still one click in the dashboard afterwards.

What still doesn't work

The signature thing above is the big one. Two more:

The CLI needs IPv6 to reach my public gateway. Webhooks don't — every curl in this post runs over ordinary IPv4, because that's HTTP going through Zerops' shared load balancer. A raw TCP port can't use that path, and Zerops won't publish raw ports on a shared IPv4. That leaves IPv6 or a paid dedicated IPv4.

Custom domains are written and unit-tested but have never run end to end. Proving that needs a real domain and a wildcard certificate, which I didn't have.

Things you're about to ask

How long are requests held? Until you reconnect. The cap is 200 per tunnel and 1 MB per body. Past 200 the oldest undelivered ones get dropped, because without a cap the table becomes a way to fill someone's disk.

Is the ordering per tunnel or global? Per tunnel. Each queue drains on its own.

Doesn't that eight-way race contradict the ordering? No. That's the test, hammering one row on purpose to prove only one claimer wins. The real drain loop is sequential.

What if two people connect with the same tunnel name? A reservation belongs to whoever made it. The second one gets the name "shop" is reserved by someone else.

Try it

The two curls at the top take about thirty seconds and need nothing installed. For your own tunnel:

go install github.com/BigAchiever/doorbell/cmd/doorbell@latest
doorbell -name pick-a-name 3000

🎥 4-minute demo
🌐 Live gateway
💻 Source (MIT)

CI fails the build if the Postgres and Valkey test suites skip, because a green run that quietly tested nothing is worse than a red one. That's the check I'm proudest of.

If you break it, I'd genuinely like to know how.

WhatsApp Knowledge Miner: Transforming Group Chats into a Searchable Knowledge Base

Danish ali siddiqui — Sun, 07 Sep 2025 10:12:36 +0000

In today's fast-paced digital communication landscape, WhatsApp has become a central hub for group discussions, brainstorming sessions, and knowledge sharing. However, the transient nature of these conversations often leads to valuable insights being lost amidst the chat clutter. Enter the WhatsApp Knowledge Miner, a tool designed to extract, organize, and transform your WhatsApp group chats into a structured, searchable knowledge base.

Project Overview

The WhatsApp Knowledge Miner is an open-source application that bridges the gap between unstructured WhatsApp group chats and structured knowledge management. By leveraging the WhatsApp Business API and advanced natural language processing (NLP) techniques, the application captures messages, analyzes their content, and converts them into question-answer pairs that can be easily searched and referenced.

🧩 Core Components
1. WhatsApp Bridge

The WhatsApp Bridge serves as the interface between your WhatsApp account and the application. It connects to your WhatsApp account and securely stores messages in a local SQLite database.

Setup:

cd whatsapp-bridge
go run main.go

Upon running, scan the QR code displayed to log in (similar to WhatsApp Web). The application will then begin capturing messages from your WhatsApp groups.
**

Message Capture and Storage**

Captured messages are stored in a local SQLite database, ensuring that all data remains secure and accessible. This local storage approach facilitates easy retrieval and analysis of messages without relying on external servers.

*3. Message Processing Server
*
The message processing server analyzes the captured messages to identify key information, such as questions and answers. Using NLP techniques, it extracts relevant content and organizes it into structured data.

*4. Streamlit Web Interface
*
The Streamlit web interface provides a user-friendly platform to interact with the processed data. Users can search for specific questions and view the corresponding answers, making it easy to access valuable information from past discussions.

🔧 Technical Highlights

Go (Golang): Used for building the WhatsApp Bridge, ensuring efficient and concurrent message capturing.

SQLite: Provides a lightweight and reliable local database for storing messages.

Natural Language Processing: Employed to analyze and extract meaningful information from unstructured messages.

Streamlit: Facilitates the creation of an interactive web interface for users to search and view Q&A pairs.

🧪 Installation and Usage

Clone the Repository:

git clone https://github.com/BigAchiever/whatsapp-knowledge-miner.git
cd whatsapp-knowledge-miner

Set Up the WhatsApp Bridge:

cd whatsapp-bridge
go run main.go

Scan the QR code to log in to your WhatsApp account.

Run the Message Processing Server:

Navigate to the whatsapp-mcp-server directory and start the server to begin processing captured messages.

Launch the Streamlit Web Interface:

In the streamlit-app directory, run the following command to start the web interface:

streamlit run app.py

Access the interface through your web browser to search and view Q&A pairs.

🌟 Key Features

Seamless Integration: Effortlessly connects to your WhatsApp account to capture messages.

Automated Q&A Extraction: Utilizes NLP to automatically generate question-answer pairs from conversations.

Local Data Storage: Ensures data privacy by storing messages in a local SQLite database.

Interactive Search Interface: Provides a user-friendly web interface for searching and viewing Q&A pairs.

Open Source: Freely available for modification and enhancement.

💡 Use Cases

Team Collaboration: Easily reference past discussions and decisions made in group chats.

Knowledge Management: Organize and access valuable information shared within WhatsApp groups.

Educational Purposes: Create a repository of questions and answers for study groups or tutoring sessions.

Customer Support: Analyze customer inquiries and responses to improve support services.

🔗 Explore the Project

To delve deeper into the WhatsApp Knowledge Miner, explore the Github Repository
. Here, you'll find comprehensive documentation, installation instructions, and the complete source code.
For more checkout my LinkedIn

🧠 Conclusion

The WhatsApp Knowledge Miner exemplifies the power of combining messaging platforms with advanced data processing techniques. By transforming unstructured group chats into a searchable knowledge base, it enhances collaboration, preserves valuable information, and streamlines access to past discussions. Whether for personal use, team collaboration, or educational purposes, this tool offers a practical solution to manage and utilize the wealth of information shared within WhatsApp groups.