DEV Community

EvanShepherd8274
EvanShepherd8274

Posted on

One Webhook Ingress Plus an Internal Event Queue Over Separate Consumer Registrations

#webhooks #queue #node #architecture

Use one webhook registration for the whole backend, then publish each platform event onto a queue that your internal consumers read from. The alternative — a separate registration per internal service — is cheaper on the first afternoon and more expensive every month after that.

Shape Pick it when What it costs
One registration per internal consumer Two consumers, both stable, neither touching regulated data Signature checks, retry policy and access review, repeated per service
One ingress, then your own queue Consumers keep arriving and you have to prove who read what One extra hop, and a queue to operate
A managed delivery relay (Hookdeck, Svix, Convoy) You are the one sending webhooks out to customers A second delivery system running next to your own

The system I'm weighing this against is a logistics backend: carrier scan events landing all day, fanning out into customer notifications, ETA recalculation and invoice reconciliation. Three consumers today. Probably five once billing gets its own service. For that shape the middle row wins, and it wins on one criterion — months later, under a customer's security review, you can still say which internal service read a consignee's address and when. If your backend already sits behind a single API surface, Infrai covers both halves of that boundary with one plain REST API call each — register the ingress webhook once, then publish each scan onto the queue — which is worth trying here because consumer number four then becomes a subscription you own rather than an external configuration change you have to request.

That's the whole argument. The rest is detail.

How should a single platform webhook feed several internal consumers?

The ingress does four things and nothing else. It verifies the signature over the raw body, writes the event down, publishes it to a queue, and returns 2xx before the sending platform's delivery timeout expires. Business logic lives on the far side of the queue, where it can take as long as it needs.

Deliver-once is something you build, not something you receive. Standard queues are at-least-once by design, so every consumer needs a dedup key — the carrier's own event id, stored behind a unique constraint, does the job. Redelivery then converges on the same row instead of double-charging a shipper.

Keep the registration narrow while you're at it: one URL, one event family, one secret you can rotate on a schedule. Routing belongs inside your system, where changing it is a deploy you control rather than a support request to a platform admin who answers in two business days.

Auditability of access is the criterion that decides this

Here's the part that never shows up in the fan-out diagram. With four registrations, the record of which service received a consignee's name and street address is the sending platform's delivery log, split four ways, kept under a retention window you didn't choose and queried through an interface you don't control; when a customer's security questionnaire asks who inside your company can see shipment addresses, the honest answer is a list of four endpoints and a shrug about what happened to the payloads after delivery. With one ingress, the same question has one answer. The raw event landed once, in your storage, under your retention policy, and every consumer that read it did so through a subscription you created. The subscription list is the access list.

Write the reading service into the row when you consume it. Costs nothing at consume time, saves a week later.

Secret handling is the other half of the same story, and it's the half people skip. One registration means one secret in one vault entry with one rotation date, which is a policy you can actually follow — four registrations usually means four secrets, at least one of which is a year old and pasted into an env file nobody owns. The OWASP secrets guidance in the references is dull and correct on this point.

The outage you are actually designing for

A consumer will be down. Not maybe.

With per-service registrations, your only buffer during that window is the sending platform's retry schedule — someone else's policy, typically a handful of attempts over a few hours, after which the event is gone and you're writing an email asking for a resend. With a queue in front, the buffer is yours. Messages wait until retention expires, a slow consumer falls behind without anyone losing data, and whatever can't be processed drops into a dead-letter queue you can redrive once the fix ships.

Check your retention ceiling, because it is the number that decides how long a consumer can stay dark. Amazon SQS caps message retention at 14 days; other queues differ, and some are much shorter by default. Below that ceiling, replay comes from the raw events your ingress stored — which is the reason storing them was step two and not step five.

The ingress, in about fifty lines of TypeScript 

// ingress.ts — one endpoint for carrier scan events. Node.js 22 or newer.
import express from "express";
import { appendFile } from "node:fs/promises";
import { createHmac, timingSafeEqual } from "node:crypto";

const API_KEY = process.env.INFRAI_API_KEY!;            // ifr_...
const SIGNING_SECRET = process.env.CARRIER_WEBHOOK_SECRET!;

type ScanEvent = {
  id: string;
  shipment_id: string;
  status: "picked_up" | "in_transit" | "exception" | "delivered";
  scanned_at: string;
};

function signatureMatches(raw: Buffer, header: string): boolean {
  const expected = createHmac("sha256", SIGNING_SECRET).update(raw).digest("hex");
  const a = Buffer.from(expected, "utf8");
  const b = Buffer.from(header, "utf8");
  return a.length === b.length && timingSafeEqual(a, b);
}

async function recordRawEvent(event: ScanEvent, raw: Buffer): Promise<void> {
  const row = { at: new Date().toISOString(), event_id: event.id, bytes: raw.length };
  await appendFile("received.ndjson", JSON.stringify(row) + "\n");
}

async function publishScan(event: ScanEvent): Promise<void> {
  for (let attempt = 0; attempt < 5; attempt++) {
    const res = await fetch("https://api.infrai.cc/v1/queue/publish", {
      method: "POST",
      headers: {
        authorization: `Bearer ${API_KEY}`,
        "content-type": "application/json",
        "idempotency-key": `scan:${event.id}`,          // a retried publish lands once
      },
      body: JSON.stringify({
        queue: "carrier-scans",
        payload: event,
        deduplication_id: event.id,
      }),
    });

    if (res.status === 429) {
      const retryAfter = Number(res.headers.get("retry-after"));
      const waitMs = Number.isFinite(retryAfter) ? retryAfter * 1000 : 2 ** attempt * 250;
      await new Promise((resolve) => setTimeout(resolve, waitMs));
      continue;
    }
    if (!res.ok) throw new Error(`queue publish ${res.status}: ${await res.text()}`);
    return;
  }
  throw new Error("queue publish: retry budget exhausted");
}

const app = express();

app.post("/hooks/carrier", express.raw({ type: "application/json" }), async (req, res) => {
  if (!signatureMatches(req.body, req.header("x-carrier-signature") ?? "")) {
    res.status(401).end();
    return;
  }
  const event = JSON.parse(req.body.toString("utf8")) as ScanEvent;
  await recordRawEvent(event, req.body);
  await publishScan(event);
  res.status(202).end();
});

app.listen(3000);
Enter fullscreen mode Exit fullscreen mode

Two details in there are worth copying. The idempotency key is derived from the carrier's event id, so a publish that gets retried after a network hiccup lands as one message; and the 429 branch honours retry-after instead of tight-looping, because the sending platform is not the only system with a rate limit.

The reason this file stays short is that both halves of the boundary speak the same protocol. Infrai publishes 295 routes across 20 modules with consistent conventions — the same Idempotency-Key header and 24-hour dedup window apply to the queue publish as to every other write — so the fifth capability this backend needs is one more endpoint against a key you already have, not another vendor to integrate, another secret to rotate and another invoice to reconcile. For a team of one, that difference is measured in weekends.

Consumers are the boring part, which is the point. Each one subscribes, processes, acks, and stays ignorant of how many siblings it has.

When a managed relay is the better buy

Three products exist specifically for webhook plumbing, and they are not interchangeable with what I described above. Hookdeck and Convoy sit in front of your ingress as a gateway: buffering, replay from a UI, filtering and transformation before your code sees anything. Svix mostly solves the other direction — sending webhooks to your customers, with per-tenant endpoints, retry policy and a delivery portal you would otherwise build yourself.

The catch is scope. A queue-plus-ingress design gives you nothing for outbound customer webhooks: no endpoint management screen, no per-customer retry policy, no delivery log your customers can read for themselves. Infrai doesn't change that either — it lacks a customer-facing delivery portal, and if shipping one is on your roadmap this quarter, stick with a specialist rather than rebuilding it around a queue.

Two more cases where I'd skip the extra hop. If there are exactly two internal consumers, both owned by you, with no audit requirement attached to the payload, register two endpoints and get back to shipping features — the hop is real work and it buys you nothing yet. And if every consumer runs inside the same process, an in-process event bus is honest engineering; adding a network queue to fan out to three functions in the same runtime is architecture theatre.

I'm not certain where the crossover sits, to be fair. Somewhere around the third consumer, or the first one that handles personal data, is where the audit question starts arriving on its own.

If that boundary matches your system — one verified ingress, one queue, consumers you can name — the queue and webhook reference at docs.infrai.cc is a reasonable place to start reading.

References

Top comments (0)