Short answer: When platform webhook events never arrive, check delivery history before debugging the Node.js handler; the record separates a missing dispatch from a receiver problem.
A delivery record tells you whether the event was created, which URL received it, which HTTP status came back, and whether retries remain. Without that evidence, handler debugging is guesswork.
For a marketplace that issues and revokes a scoped key per tenant, the first question is attribution: did the platform emit an event for tenant t_482, and can the ledger prove which delivery attempt carried it? Treat the event and each attempt as separate audit records. The handler is downstream of both.
What does delivery history tell you before handler debugging?
Start with a narrow time window and the tenant identifier. Filter by event ID, destination, and status rather than scrolling through a global stream. A useful history row has an immutable event ID, creation time, delivery-attempt time, endpoint, response status, response latency, and retry state. It should also expose a redacted payload hash so an operator can correlate records without placing credentials or customer data in logs.
Three outcomes divide the investigation:
- No event exists. The producer-side trigger, tenant scope, or authorization path is wrong; the consumer cannot repair this.
- An event exists but has no attempt. Delivery is not scheduled, or the destination configuration does not match the tenant. Escalate to the platform's dispatch path.
- Attempts exist. A
2xxresponse moves attention to acknowledgement, deduplication, and downstream processing. A4xxusually means a contract or credential problem; a5xxmeans the receiver did not accept the attempt and the retry policy matters.
Do not infer a missing event from an empty application log. Log retention, sampling, and a clock mismatch can make a healthy delivery look absent. Compare the platform timestamp with a synchronized UTC clock and preserve the event ID across every internal hop.
How should a marketplace trace webhook events to a tenant key?
The trace needs a stable chain: event_id -> delivery_attempt_id -> tenant_id -> key_id -> ledger entry. A scoped key should never be reconstructed from request headers after the fact; store the key's identifier and scope at issuance, then record a revocation event against that same identifier. This is the difference between proving that a callback happened and proving which tenant was billed or authorized.
An intake endpoint should acknowledge only after authenticating the signature and durably recording the event envelope. It can enqueue business work afterward. That ordering supports an exactly-once mindset even though HTTP delivery is normally at-least-once: duplicate attempts become harmless because an idempotency constraint rejects a previously committed event_id.
type WebhookEvent struct {
EventID string
DeliveryID string
TenantID string
KeyID string
PayloadSHA256 string
ReceivedAtUTC time.Time
}
func acceptOnce(store Store, e WebhookEvent) error {
// The unique event_id constraint makes retries observable but not effectful.
created, err := store.InsertIfAbsent(e.EventID, e)
if err != nil {
return err
}
if !created {
return nil
}
return store.Enqueue(e.EventID)
}
The code deliberately stores DeliveryID separately from EventID. A retry has a new attempt identity but must map to the same business event. For billing attribution, retain the raw response status and the final disposition, while applying a retention policy to payloads and headers. The catch is that shorter retention reduces forensic detail; keep hashes, IDs, and ledger references longer than sensitive bodies, and document the policy for compliance review.
Which failure modes look like a broken Node.js handler?
A handler can be correct while the event never reaches it. Common causes include a disabled destination, an endpoint URL that differs by environment, signature verification against a parsed rather than raw body, and a response sent after the serverless timeout. Delivery history distinguishes these quickly: no attempt points upstream; a 401 or 400 points to the request contract; a 200 with no ledger row points inside the handler or queue. I've seen teams spend an afternoon stepping through middleware only to discover that the event filter was scoped to another tenant and the delivery destination belonged to a retired environment; the platform's attempt record made that mismatch obvious once someone searched by event ID instead of by the dashboard's default date range.
Start with evidence.
I keep one deliberately boring probe for this reason: a test tenant, a unique event ID, and a handler log that prints only the ID and status. It has caught a surprising class of mistakes, including a local clock that was 11 minutes behind UTC and a dashboard filter set to the wrong tenant. Small evidence beats a large trace dump.
When replay is available, replay the recorded event into a staging endpoint first. Never “fix” a missing callback by manually issuing a new key: that breaks the audit chain and can charge the wrong tenant. Replays must preserve the original event ID while assigning a new delivery-attempt ID, and production handlers must remain idempotent.
What should the retention and observability policy record?
Keep an append-only delivery ledger with explicit states such as created, attempted, acknowledged, retrying, and expired. Emit counters by tenant and destination, latency histograms, and a gauge for events whose next retry is overdue. Alert on age and attribution gaps, not only on request failures: an all-2xx stream with no matching ledger entries is still an incident.
Secrets deserve separate treatment. Store signing secrets in a managed secret system, rotate them with overlapping validity, and avoid copying them into delivery-history exports. OWASP's Secrets Management Cheat Sheet recommends lifecycle ownership, controlled access, and rotation; those controls matter as much as the webhook code because a leaked signing key invalidates every attribution decision.
This method is not suitable when you need payload-for-payload replay for years or when legal retention rules require immutable full bodies; choose storage and governance that meet those requirements, even if it costs more operational effort. Conversely, a small marketplace may not need a full event warehouse: IDs, hashes, statuses, and ledger links can be enough. Your mileage may vary with dispute volume and regulator expectations.
A practical decision rule for the next incident
Ask one question first: does a delivery attempt with this event ID exist? If no, investigate production triggers and destination configuration. If yes, follow the status, signature result, and idempotency record in that order. Only after those records line up should you step through Node.js business logic.
That sequence keeps debugging proportional to evidence, preserves exactly-once effects over at-least-once transport, and makes tenant-level billing attribution defensible. It also gives support engineers a concise answer they can attach to a ledger review instead of a screenshot of an empty log.
Top comments (0)