A customer-support backend cannot lose billing attribution just because one internal consumer is unavailable. That constraint changes the design: acknowledge the platform webhook only after durably accepting its event, then let each consumer confirm its own work independently.
Short answer: use one webhook registration, persist a canonical event envelope in a queue or inbox, fan it out by consumer name, and record a separate acknowledgement for every consumer so one failure never erases another consumer's progress.
This is deliberately boring infrastructure. It outsources undifferentiated delivery mechanics and keeps the weekly shipping loop focused on support features. The important part isn't raw throughput. It is being able to answer, for any invoice line, which platform event arrived, which account owned it, and which billing consumer committed it.
How should one webhook registration fan out through queue acknowledgement to many internal consumers?
Treat receipt, dispatch, and completion as three different facts. The public handler authenticates the request, parses it once, assigns or preserves a stable event ID, writes the untouched payload plus attribution fields to durable storage, and returns success. A dispatcher then creates one delivery record per subscribed consumer. Each worker leases only its own delivery record and acknowledges that record after its side effect has committed.
That last boundary matters. If the billing worker acknowledges before writing its ledger row, an interruption can turn accepted work into missing revenue. If it writes first but has no idempotency key, a retry can charge twice. The smallest useful invariant is therefore: the ledger write and the consumer's completed marker must share one transaction, while the provider-facing acknowledgement stays tied only to durable receipt.
Persist first.
Keep the envelope small and explicit:
type PlatformEvent = {
eventId: string;
accountId: string;
occurredAt: string;
eventType: string;
payload: unknown;
};
type Delivery = {
eventId: string;
consumer: "billing" | "timeline" | "analytics";
state: "ready" | "leased" | "done";
attempts: number;
availableAt: string;
};
accountId belongs in the envelope, not in a late lookup whose answer may change. For a customer-support system, that field is the attribution anchor shared by billing and audit views. Preserve the raw payload as evidence, but don't make every downstream worker reinterpret account ownership. Normalize that once at ingress.
One event can now be done for timeline, retrying for analytics, and untouched for billing.
Good.
A single global processed flag cannot represent those states without lying.
The smallest working implementation
The handler below uses generic ports so the storage and queue products remain replaceable. The route is an application-owned example, not a vendor API. Signature verification is injected because the exact algorithm and header format belong to the webhook provider's contract. Secrets should come from a managed secret lifecycle rather than source code; the OWASP Secrets Management Cheat Sheet covers storage, rotation, auditing, and access controls for that boundary.
type RequestLike = {
rawBody: Uint8Array;
headers: Record<string, string | undefined>;
};
type ResponseLike = { status: number; body: string };
type Inbox = {
insertOnce(event: PlatformEvent): Promise<"inserted" | "duplicate">;
};
type Dispatcher = { publish(eventId: string): Promise<void> };
type VerifyAndDecode = (request: RequestLike) => Promise<PlatformEvent>;
export function createWebhookHandler(
verifyAndDecode: VerifyAndDecode,
inbox: Inbox,
dispatcher: Dispatcher,
) {
return async (request: RequestLike): Promise<ResponseLike> => {
const event = await verifyAndDecode(request);
const result = await inbox.insertOnce(event);
if (result === "inserted") {
await dispatcher.publish(event.eventId);
}
return { status: 202, body: "accepted" };
};
}
There is a subtle deployment choice hidden here: insertOnce and publish need a recoverable handoff. In a compact build, an outbox row written in the same database transaction as the inbox event is enough; a relay publishes pending outbox rows and marks them sent. Publishing directly after the insert, as the interface makes visually simple, is acceptable only when a periodic scanner also republishes inbox events that have no delivery records. Without one of those recovery paths, a process stop between the two awaits strands an accepted event.
The dispatcher expands subscriptions without touching the original event:
type DeliveryStore = {
createOnce(delivery: Delivery): Promise<void>;
};
const consumers: Delivery["consumer"][] = [
"billing",
"timeline",
"analytics",
];
export async function fanOut(
eventId: string,
store: DeliveryStore,
): Promise<void> {
for (const consumer of consumers) {
await store.createOnce({
eventId,
consumer,
state: "ready",
attempts: 0,
availableAt: new Date().toISOString(),
});
}
}
createOnce needs a uniqueness rule on (eventId, consumer). Replaying the outbox then becomes routine: it may try the expansion again, but it doesn't create extra logical deliveries. This is the sort of constraint that earns revenue per engineering hour because it replaces a pile of timing assumptions with one database rule.
For billing, carry the same key into the ledger transaction. The worker may be interrupted after commit and before its queue acknowledgement; the second attempt must observe the existing ledger entry and finish cleanly. Do not generate a fresh idempotency key inside the attempt.
type BillingStore = {
commitEvent(input: {
idempotencyKey: string;
eventId: string;
accountId: string;
}): Promise<void>;
};
export async function consumeBilling(
event: PlatformEvent,
billing: BillingStore,
): Promise<void> {
await billing.commitEvent({
idempotencyKey: `billing:${event.eventId}`,
eventId: event.eventId,
accountId: event.accountId,
});
}
Retries should have a cap, delayed availability, and a terminal review state. I'm not sure which retry intervals fit your traffic without arrival-rate and recovery-time data. Start with an explicit policy, measure queue age and attempt counts, then tune it; don't bury an infinite immediate retry loop in the worker. A malformed event and a temporarily unavailable dependency need different operator decisions even if both initially appear as failed deliveries.
Proving attribution before shipping
Test the state transitions, not the queue library. Feed the same event twice and assert one inbox row, one delivery per consumer, and one billing ledger entry. Stop a worker after the ledger commit but before acknowledgement, lease the delivery again, and assert the account is still charged once. Hold the timeline consumer offline while billing continues, then release it and confirm its backlog drains without replaying completed billing work.
Use event IDs such as evt_01J8K7M2Q4, account IDs such as acct_2048, and attempt numbers in test output. Concrete identifiers make an attribution break visible. A log line saying only processing failed doesn't.
For deployment, add the ingress path first with consumers disabled, inspect stored envelopes, then enable one consumer at a time. The minimum operational view is queue age by consumer, ready versus leased counts, retry attempts, terminal-review count, and the event-to-account-to-ledger correlation. Alert on age rather than raw queue depth alone: ten old billing events are more urgent than a short burst of ten new timeline events.
Keep payload access narrow. Webhook bodies can contain customer-support content, so workers should receive only the fields they need when practical. Rotate signing secrets, restrict who can read them, and log secret-management operations without logging the secret itself. Those controls follow the lifecycle guidance in the OWASP reference below.
What I would change at scale
At higher volume, partition delivery work by a stable key that preserves the ordering your business rule actually needs, usually the account rather than the whole stream. Add lease expiry with fencing so a slow worker cannot complete after a replacement has taken ownership. Archive raw payloads according to a defined retention policy, while keeping the small attribution index available for disputes.
Do not add those mechanisms by reflex. Per-account ordering can leave one noisy account concentrated on a partition, fencing adds state and tests, and long retention expands the security surface. If events are commutative and billing is derived in periodic batches, strict online ordering may be the wrong spend. Ship the simpler invariant first and promote complexity only after observed contention or a stated audit requirement.
The catch is that this pattern is not suitable when every consumer must commit atomically with every other consumer. Independent acknowledgement intentionally permits temporary disagreement between the support timeline and billing projection. If the business requires one all-or-nothing transaction, keep the affected writes inside one transactional boundary instead of using asynchronous fan-out. Likewise, stick with a direct synchronous call when there is one consumer, failure can be returned to the caller, and replay has no business value; a queue would add operational work without buying isolation.
No architecture erases trade-offs. This one buys independent recovery and traceable billing attribution with duplicate-tolerant handlers, extra stored state, and delayed consistency. For a one-person SaaS, that is often a sensible exchange because the machinery is small, testable, and leaves feature work moving every week.
Top comments (0)