TL;DR: accept a marketplace webhook once, persist one immutable ingress record, and create a separate delivery ledger row for every internal consumer before returning success. A shared queue acknowledgement is the wrong boundary when billing attribution matters: it proves that somebody handled a message, while a per-consumer ledger proves which service accepted which tenant-scoped event. Keep the shared-ack design only when every consumer is interchangeable and no audit, replay, or charge decision depends on consumer identity.
This is an incident lesson best understood as a bounded failure exercise, not a production anecdote. Tenant t-1842 rotates a scoped key; the account service emits event evt-7f3a; billing, audit, and entitlement processing must each observe it. Billing finishes first and acknowledges the shared queue item. Audit is unavailable for 40 seconds. The queue is now honest about its own contract, yet the platform cannot answer whether audit ever accepted the revocation, and retrying the original item risks charging or applying the entitlement transition twice.
The invariant is stricter than “the webhook returned 2xx”: for every accepted event and required consumer, the system needs one durable, independently advancing delivery record. That record, rather than a process log or a common queue receipt, is the attribution boundary.
Why can't one queue acknowledgement represent every consumer?
A queue acknowledgement normally settles work for one delivery. Fan-out changes the cardinality. One ingress event produces several obligations, and those obligations can finish, retry, or exhaust their retry budgets independently. Compressing them back into one acknowledgement discards exactly the identity needed to explain a tenant bill.
One receipt cannot prove three outcomes.
It fails quietly. Imagine the receiver has enqueued the event and returned success, then three workers race for the same message. If this is a competing-consumer queue, only one worker is supposed to win. If the workers all need the event, the topology does not describe the requirement. Adding more workers increases capacity but does not create broadcast semantics.
There is a second trap: publishing three messages and considering the source message complete after the first publish. A crash between publish one and publish two leaves a partial fan-out. The receiver still looks healthy, and the missing consumer may not discover the gap until a reconciliation job compares tenant state much later.
No mystery there.
I use this capacity-planning reflex: count obligations, not incoming requests. At 200 accepted events per second and four required consumers, the steady-state write path creates at least 800 delivery obligations per second, before retries. Those numbers are an example planning input, not a benchmark. The useful calculation is ingress rate × required consumer count × retry amplification; it exposes a database or queue limit that request-rate dashboards conceal.
The incident boundary belongs in durable state
The receiver should authenticate and validate the request, derive a stable event identifier, then commit the ingress row and its consumer delivery rows in one database transaction. Only after that commit is success an accurate response. Downstream processing remains asynchronous, so a slow audit service cannot hold the external webhook connection open.
For the marketplace key lifecycle, the event body should carry the tenant identifier, key identifier, operation, and the source event identifier. Do not put the key secret in the event. The delivery ledger needs metadata for attribution: event_id, tenant_id, consumer, status, attempts, and timestamps. If billing derives usage from successful handling, define in advance whether “billable” means enqueued, first accepted attempt, or completed business transition. I prefer completion because it corresponds to an outcome, but the important property is that the rule is explicit and idempotent.
| Design | What acknowledgement proves | Partial failure behavior | Billing attribution | Choose it when |
|---|---|---|---|---|
| One competing-consumer queue | One worker settled one delivery | Other services may never see the event | Ambiguous | Consumers are interchangeable |
| One topic with independent subscriptions | Each subscription can settle separately | Retry state is isolated by subscription | Good if subscription identity is retained | The broker is the durable system of record |
| Database inbox plus consumer ledger | Ingress and obligations are committed together | Missing or stalled obligations are queryable | Explicit per tenant and consumer | Auditability and controlled replay matter |
| Synchronous calls from the receiver | Every completed call is known immediately | One slow dependency extends ingress latency | Explicit but tightly coupled | The consumer set is tiny and latency is bounded |
For this case, choose the inbox plus consumer ledger. It gives the platform a queryable statement of intent before any worker runs, separates acknowledgement from business completion, and keeps the external registration count at one. A topic with independent subscriptions can implement the same logical contract, but then retention, replay, identity mapping, and evidence collection belong to that messaging layer; that is a reasonable choice only if the team is prepared to operate it as part of the billing control plane.
The limitation is operational weight. A consumer ledger adds transactional writes, indexes that grow with every obligation, a lease protocol, retention work, and a reconciliation job; it is not suitable for a fungible worker pool whose only contract is “run this task once somewhere.” A topic with independent durable subscriptions is the better fit when the messaging system already owns replay and the team accepts its subscription identity as billing evidence. Direct synchronous calls are simpler when there are perhaps two stable consumers, both fit comfortably inside the webhook response budget, and partial completion can be repaired without ambiguity. None of those alternatives is universally weaker. They place the evidence and on-call burden in different systems.
That burden is real.
The SLO follows the same boundary. Receiver availability measures durable acceptance, while consumer freshness measures the oldest pending required delivery. A single end-to-end availability percentage hides whether one optional analytics consumer is lagging or a required revocation consumer has stopped. I would page on the latter and ticket the former, with thresholds derived from the marketplace's revocation and billing commitments rather than copied from a generic dashboard.
Make acknowledgement conditional, not optimistic
The following Go sketch shows the preventative path behind a Node.js webhook receiver: the transaction inserts the immutable event and every required obligation, while workers claim one obligation at a time. The language differs from the ingress runtime on purpose; the contract is a storage invariant, not a framework feature. In a Node.js service, the same transaction boundaries must remain intact.
package fanout
import (
"context"
"database/sql"
"encoding/json"
"errors"
"time"
)
type Event struct {
ID string `json:"id"`
TenantID string `json:"tenant_id"`
KeyID string `json:"key_id"`
Operation string `json:"operation"`
Payload json.RawMessage `json:"payload"`
}
var requiredConsumers = []string{"billing", "audit", "entitlements"}
func Accept(ctx context.Context, db *sql.DB, event Event) error {
if event.ID == "" || event.TenantID == "" || event.KeyID == "" {
return errors.New("missing event identity")
}
if event.Operation != "key.issued" && event.Operation != "key.revoked" {
return errors.New("unsupported operation")
}
tx, err := db.BeginTx(ctx, nil)
if err != nil {
return err
}
defer tx.Rollback()
result, err := tx.ExecContext(ctx, `
INSERT INTO webhook_events
(event_id, tenant_id, key_id, operation, payload, accepted_at)
VALUES (?, ?, ?, ?, ?, ?)
ON CONFLICT (event_id) DO NOTHING`,
event.ID, event.TenantID, event.KeyID, event.Operation, event.Payload, time.Now().UTC())
if err != nil {
return err
}
inserted, err := result.RowsAffected()
if err != nil {
return err
}
if inserted == 1 {
for _, consumer := range requiredConsumers {
_, err = tx.ExecContext(ctx, `
INSERT INTO consumer_deliveries
(event_id, tenant_id, consumer, status, attempts)
VALUES (?, ?, ?, 'pending', 0)`,
event.ID, event.TenantID, consumer)
if err != nil {
return err
}
}
}
return tx.Commit()
}
The handler returns success only when Accept commits. A duplicate source event becomes a successful no-op because event_id is unique; it must not create a second set of billable obligations. Each delivery should also have a uniqueness constraint on (event_id, consumer). Those two constraints turn retry behavior into a property of the data model rather than a hope that two processes will not overlap.
Worker completion needs another guard. Claim a pending row with a lease, increment its attempt count, perform an idempotent consumer operation, then mark that exact row completed. A lease expiry permits recovery after a worker dies. Completion updates should require the current lease token so a delayed worker cannot overwrite the result of a newer attempt.
Do not acknowledge early.
There is an unavoidable crash window between a consumer's external side effect and the ledger completion update. Eliminate its billing impact with an idempotency key such as event_id + consumer, and have the consumer store that key with its business transition. Exactly-once delivery is not the useful promise here; one observable business effect per obligation is.
Operate the ledger as a billing control
Start testing below the HTTP layer. Run the same event through acceptance twice and assert that it leaves one ingress row and three delivery rows. Force the second delivery insert to fail and assert that neither the ingress row nor the first delivery remains. Then kill a worker after its consumer commits but before ledger completion, allow the lease to expire, and verify that the idempotency record suppresses a second business effect.
The deployment order matters because consumer membership is data with financial consequences. Add a new consumer as optional, backfill or establish its start cursor, observe its lag, and only then make it required for new events. Removing one is the reverse: stop creating new obligations at a declared boundary, drain existing rows, preserve the historical ledger, and deploy the worker removal last. Editing a hard-coded slice and deploying receiver instances gradually can otherwise give identical events different obligation sets.
For observability, emit counters for accepted events, obligations created, attempts, completions, and terminal failures. Partition them by consumer and operation, but be cautious with raw tenant identifiers in metric labels; high-cardinality identity belongs in traces or queryable records. Logs should carry the event ID, tenant ID, consumer, attempt, and lease token, while excluding credentials and the scoped key material. The OWASP Secrets Management Cheat Sheet treats auditing, rotation, revocation, and least privilege as parts of secret lifecycle management; that is why the event transports identifiers and state changes, not secrets themselves.
A daily reconciliation should compare accepted events with their required delivery set, then compare completed billable deliveries with the billing journal. This is not a substitute for transactional creation. It is the independent detector for schema bugs, accidental consumer-set changes, and operational mistakes. Set a retention period from dispute, audit, and privacy obligations, then test deletion as carefully as insertion.
The build-versus-buy decision is therefore less about queue throughput than ownership. A managed broker can carry independent subscriptions, retries, and retention, but the platform still owns tenant mapping, idempotent business effects, billing definitions, reconciliation, and the SLO. A self-hosted ledger offers direct queries and transaction coupling at the cost of migrations, vacuuming or compaction, lease code, and on-call responsibility. Choose the system whose failure evidence your team can retrieve at 03:00 without reconstructing it from sampled logs.
This advice does not apply when the consumers truly form a fungible worker pool, when only one successful execution is required, and when no downstream charge or audit statement names a particular consumer. It also may be excessive for a noncritical notification where lossy best-effort delivery is an explicit product decision. For tenant key issuance and revocation tied to billing, those conditions rarely describe the job: identity is the point, so acknowledgement must preserve it.
Top comments (0)