Short answer: use one authenticated webhook receiver that durably records each event, then fan it out from a queue with a separate acknowledgement for every internal consumer. In logistics, this is the least complex design that still lets you prove who received a shipment update during an outage.
| Choice | Audit trail | Outage behavior | Glue work |
|---|---|---|---|
| Direct webhook to every service | Scattered | Sender retries each target | High |
| One receiver, synchronous fan-out | Central intake, weak delivery history | One slow consumer blocks intake | Medium |
| One receiver, durable queue fan-out | Per-event and per-consumer records | Queue absorbs the gap | Medium |
I would pick the third row. It separates the sender's acknowledgement from downstream work, so a carrier status event can be accepted once and replayed without asking the carrier to understand your internal topology. I've built enough CLIs to know that every extra integration callback becomes another config file someone forgets to rotate.
The catch is operational discipline: a queue does not create an audit trail by itself. You need durable event IDs, consumer delivery records, and a policy for poison messages.
The audit boundary is the first design decision
A webhook endpoint should do four things in a tight order: read the raw request, authenticate it, assign or validate an idempotency key, and commit the event to durable storage before returning a success response. JSON parsing belongs after authentication. Logging the parsed object alone is not enough; signature checks generally cover the exact bytes sent by the producer.
For a shipment event, store the producer event ID, received timestamp, signature key ID, payload hash, and schema version. Keep the raw payload in restricted storage when policy allows it, and keep secrets out of that record. OWASP recommends a secrets-management lifecycle that includes inventory, rotation, access control, and audit logging; those controls apply to the signing secret and the queue credentials too.
I once treated the HTTP 202 as the audit record. That was a mistake. A process restart after the response but before the database commit created a delivery that looked accepted and then vanished. The fix was boring: commit first, acknowledge second.
A short acknowledgement is a feature.
Measure it.
No shortcuts.
How should one webhook registration fan out through a queue acknowledgement flow?
Think in two acknowledgement domains. The ingress domain acknowledges the external sender only after the event is durable. The consumer domain acknowledges a queue message only after that consumer's side effect and delivery record are durable. They are related, but they are not the same transaction.
The following TypeScript sketch uses generic interfaces so the queue can be backed by a managed service or a self-hosted broker. The endpoint path is intentionally application-owned; the important contract is the ordering and the idempotency key.
type ShipmentEvent = {
id: string;
type: "shipment.updated" | "shipment.delayed";
shipmentId: string;
occurredAt: string;
version: number;
};
type QueueMessage<T> = {
messageId: string;
body: T;
receipt: string;
};
interface EventStore {
insertIfAbsent(event: ShipmentEvent, rawHash: string): Promise<boolean>;
markDelivered(eventId: string, consumer: string): Promise<boolean>;
}
interface Queue<T> {
publish(message: T, key: string): Promise<void>;
receive(limit: number): Promise<Array<QueueMessage<T>>>;
acknowledge(receipt: string): Promise<void>;
reject(receipt: string, reason: string): Promise<void>;
}
async function receiveWebhook(
request: Request,
store: EventStore,
queue: Queue<ShipmentEvent>,
): Promise<Response> {
const raw = new Uint8Array(await request.arrayBuffer());
const signature = request.headers.get("x-signature") ?? "";
if (!await verifySignature(raw, signature)) {
return new Response("unauthorized", { status: 401 });
}
const event = JSON.parse(new TextDecoder().decode(raw)) as ShipmentEvent;
const inserted = await store.insertIfAbsent(event, await sha256(raw));
if (inserted) {
await queue.publish(event, event.id);
}
return new Response(null, { status: 202 });
}
async function consume(
queue: Queue<ShipmentEvent>,
store: EventStore,
consumer: string,
apply: (event: ShipmentEvent) => Promise<void>,
): Promise<void> {
for (const message of await queue.receive(20)) {
try {
const firstDelivery = await store.markDelivered(message.body.id, consumer);
if (firstDelivery) await apply(message.body);
await queue.acknowledge(message.receipt);
} catch (error) {
await queue.reject(message.receipt, String(error));
}
}
}
insertIfAbsent makes the ingress retry-safe. markDelivered makes each consumer retry-safe. In production, I would make the delivery row carry an attempt count, last error, and lease expiry. A consumer that dies after its side effect but before acknowledgement will see the same message again; the idempotency check turns that duplicate into a no-op.
Do not publish five copies from the HTTP handler just because there are five teams. Publish one event, then let subscriptions or consumer-specific queues provide isolation. Give each consumer its own acknowledgement and retry policy. The tracking service may tolerate a minute of lag; billing probably needs a stricter alert, while a dashboard index can catch up later.
Failure modes that deserve a test, not a slide
Start with a table-driven test for the awkward edges: duplicate webhook delivery, reordered versions, an invalid signature, a queue timeout after a successful publish, and a consumer crash after applying a side effect. For the publish timeout, query the event store by ID before publishing again. For reordered updates, compare the event version per shipment and reject stale state transitions while retaining the original event for audit.
An outage drill should answer concrete questions. Can an operator list every consumer that has not acknowledged event evt_8f2? Can they replay only the tracking consumer without replaying billing? Does the external sender receive a retryable response when the durable store is unavailable?
Use structured logs with the event ID, consumer name, attempt, and correlation ID. Metrics should separate ingress acceptance latency from consumer age. A single queue-depth number hides the important failure: one consumer can be stuck while the others are healthy.
There is a limit. Exactly-once effects are not supplied by HTTP or most queues; they come from idempotent application code and a durable record of what happened. Your mileage may vary with broker semantics, so verify visibility timeouts, redelivery guarantees, and ordering rules in the system you operate.
When is direct delivery the better choice?
Direct delivery is reasonable for a small integration with one consumer, a short outage window, and no requirement to replay by team. It has fewer moving parts and lower operator load. A queue is not automatically more reliable if nobody owns its dead-letter queue or watches consumer lag.
Stick with direct delivery when the producer already provides signed, replayable events and your endpoint can complete the only side effect inside its timeout. Choose a queue when internal consumers have different availability, retry, or retention needs, or when an audit request must show a per-consumer acknowledgement.
Cost is one dimension, not the decision. Queue storage, retention, and duplicate processing add work; so does building your own retry ledger around direct HTTP calls. I am not sure which side wins for your traffic until you measure event rate, payload size, retention days, and the number of independent consumers.
The practical rule is simple: one registration at the platform edge, one durable event record, and one acknowledgement record per consumer. That gives a logistics team a defensible timeline through an outage without coupling carrier retries to every internal service.
Top comments (0)