DEV Community

gokul
gokul

Posted on

Why We Rewrote Our Tunnel Stack in Rust

#rust #webdev #devops #opensource

Testing GitHub Webhooks Locally: A 5-Step Workflow That Actually Holds Up

Setting up a GitHub webhook takes two minutes. You paste a URL into your repository settings, pick your events, and GitHub starts POSTing JSON.

Testing that webhook properly is where everyone gets stuck — your development server runs on localhost, and GitHub's servers cannot reach it. Worse, the failure modes that cause real production incidents (retries, duplicates, out-of-order deliveries) never show up in a happy-path test.

After wiring GitHub webhooks more times than I can count, here's the workflow that actually holds up. Five steps, each addressing a failure I've seen take down a real integration.

Step 1: Get a Stable Public URL

Any tunnel puts your localhost on the internet, but what actually matters is whether the URL survives a restart. GitHub stores your webhook URL in the repository settings — if your tunnel hands you a fresh random address every time the agent restarts, you're editing webhook config every morning, and every mysteriously silent integration is just a stale URL.

Use a named subdomain instead:

mytunnel http 3000 --subdomain gh-hooks
Enter fullscreen mode Exit fullscreen mode

This gives you a permanent https://gh-hooks.21tunnel.com pointing at your local endpoint (e.g. /webhooks/github). Paste that into your repo's webhook settings once, and it keeps working across restarts, reboots, and weekends.

ngrok and Cloudflare Tunnel solve the same problem with different tradeoffs — the requirement is the stable address, not the specific tool.

Step 2: Verify Signatures Over the Raw Request Body

GitHub signs every delivery with HMAC-SHA256, using the secret you set in the webhook configuration, and sends it in the X-Hub-Signature-256 header. This signature is your only proof that an event came from GitHub and not from someone who found your endpoint URL.

The trap that burns afternoons: if your web framework parses the JSON body before your verification code runs, the bytes change. Re-serialized JSON is not byte-identical to what GitHub sent — the signature no longer matches, every event fails verification, and the error messages point everywhere except the real cause.

The fix is the same in every framework: read the raw body first, verify the signature against those exact bytes, and only then parse the JSON.

  • Express — use express.raw on the webhook route, registered before express.json
  • Django — read request.body before anything touches the parsed form data

One line of middleware ordering. Get it wrong and nothing works; get it right and you never think about it again.

Step 3: Deduplicate on X-GitHub-Delivery

GitHub promises at-least-once delivery. If your endpoint times out, returns an error, or responds too slowly, the delivery comes back — sometimes even when nothing visibly failed.

Every delivery carries a unique ID in the X-GitHub-Delivery header:

  1. Store it with a unique constraint in your database
  2. Do the actual work in the same transaction as the insert
  3. If the insert fails because the ID already exists → the event was already processed → return 200 and move on

Handlers built this way can be retried forever without corrupting state. Handlers built on hope eventually double-create an issue, double-trigger a deploy, or double-charge someone.

Step 4: Replay Deliveries from GitHub's UI

Most developers never notice the Recent Deliveries tab in the webhook settings page. It shows every delivery GitHub attempted — the full request and response — plus a Redeliver button.

Combined with your tunnel's request inspector, this is the fastest debug loop available:

  1. Pick any past delivery
  2. Hit Redeliver
  3. Watch it arrive in the inspector
  4. See exactly how your handler responded

No fake commits, no dummy pull requests, no waiting for real events to trigger the code path you're testing.

Step 5: Test the Failure Paths on Purpose

Happy-path testing tells you the integration works on a good day. Before calling it done:

  • Return a 500 and watch GitHub's retry behavior
  • Kill your server mid-delivery
  • Redeliver the same event twice and confirm the second attempt is a harmless no-op
  • Disconnect the tunnel, reconnect, and confirm nothing needs manual repair

Each of these takes two minutes. Together they replace the 2 AM incident where you learn all of it at once.

Summary

The complete setup:

Step What it solves
1. Stable public URL Webhook config doesn't break on restart
2. Raw-body signature verification Confirms events actually came from GitHub
3. Idempotent handlers (dedupe on delivery ID) Prevents double-processing from retries
4. Replay-driven debugging Fast iteration without fake test data
5. Deliberate chaos testing Surfaces failure modes before production does

Do these five and moving to production is boring — same handler, same verification, just a new URL and secret in the settings.

Full guide with working verification code in Node, Python, and Rust, plus the retry semantics nobody documents: 21tunnel-blog

Top comments (0)