For customer-support notifications, a Node.js push subscriber should verify the signed request, durably record an idempotent handoff, and ACK only after that handoff commits. The downstream notification must run from the handoff, with one business idempotency key reused across retries. This is a delivery-guarantee decision, not a choice between two HTTP status codes.
Short answer: authenticate the raw request, persist the delivery claim and outbox record together, then acknowledge the queue; expect duplicate delivery whenever a response can be lost.
That ordering matters because a support ticket update can arrive while the receiving CRM or notification endpoint is slow. If the public endpoint waits for that destination, a timeout leaves two facts unknowable: whether the destination applied the update, and whether the queue will send it again. Treating transport acknowledgement as business completion creates duplicate customer messages and an audit trail that cannot explain them.
Start with the audit record, not the endpoint
The useful contract has four checkpoints: authenticated receipt, durable claim, queue acknowledgement, and outbound confirmation. The public HTTPS endpoint owns the first two. The queue owns recovery until the third. A worker owns the last one. Each checkpoint needs a durable state transition, so an operator can distinguish a redelivered message from an outbound request whose result is unknown.
| Boundary | Failure or uncertainty | Recovery owner | Required behavior |
|---|---|---|---|
| Authentication | Missing or invalid signature | Subscriber | Reject without ACK |
| Durable handoff | Store or transaction failure | Queue | Return non-success and allow retry |
| ACK response | Response lost after commit | Subscriber and queue | Treat the repeated claim as a duplicate |
| Outbound webhook | Timeout or transient server response | Worker | Retry with the same business key |
| Final result | External effect is uncertain | Reconciliation process | Preserve the uncertainty and investigate |
The application should pursue exactly-once business transitions, not exactly-once network delivery. A queue can redeliver after the first transaction committed. A unique delivery identifier prevents a second intake record, while a separate idempotency key lets the outbound recipient recognize the same support action on every HTTP attempt. Those are two keys with two responsibilities.
The catch is that this design is not suitable when the push source offers no stable delivery identifier or the team cannot operate a durable store. A simpler synchronous integration can be valid when the destination documents idempotency and its latency fits the request budget. Stick with that smaller design for a low-volume, tightly controlled integration; choose the durable handoff when delayed webhook tasks must survive restarts and ambiguous timeouts.
How can a Node.js endpoint verify signed webhooks before ACK?
Preserve the raw body if the sender signs request bytes. Parsing and re-serializing JSON can change whitespace, escaping, or field order. Check the HTTP method, impose a body-size limit, validate the delivery identifier, and compare the decoded MAC with a constant-time operation when the protocol uses HMAC. The header names and digest format in this example are application-level placeholders; the real subscriber must follow its queue's documented signature and ACK contract.
package main
import (
"crypto/hmac"
"crypto/sha256"
"encoding/hex"
"errors"
"io"
"net/http"
)
var errAlreadyClaimed = errors.New("delivery already claimed")
type HandoffStore interface {
ClaimAndWriteOutbox(deliveryID string, body []byte) error
}
func validSignature(body []byte, supplied, secret string) bool {
provided, err := hex.DecodeString(supplied)
if err != nil {
return false
}
digest := hmac.New(sha256.New, []byte(secret))
_, _ = digest.Write(body)
return hmac.Equal(provided, digest.Sum(nil))
}
func receiver(store HandoffStore, secret string) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
if r.Method != http.MethodPost {
http.Error(w, "method not allowed", http.StatusMethodNotAllowed)
return
}
body, err := io.ReadAll(http.MaxBytesReader(w, r.Body, 256<<10))
if err != nil || !validSignature(body, r.Header.Get("X-Webhook-Signature"), secret) {
http.Error(w, "unauthorized", http.StatusUnauthorized)
return
}
deliveryID := r.Header.Get("X-Delivery-ID")
if deliveryID == "" {
http.Error(w, "missing delivery id", http.StatusBadRequest)
return
}
err = store.ClaimAndWriteOutbox(deliveryID, body)
if err != nil && !errors.Is(err, errAlreadyClaimed) {
http.Error(w, "retry delivery", http.StatusServiceUnavailable)
return
}
w.WriteHeader(http.StatusNoContent)
}
}
The important operation is ClaimAndWriteOutbox, not the in-memory shape of the handler. In production it should be one database transaction: insert the intake record, insert the outbox record, and enforce uniqueness on the delivery ID. A duplicate-key result means the subscriber already accepted responsibility, so returning a successful ACK is correct. An unknown storage error means the queue still owns recovery, so the handler must return non-success.
The ACK is a boundary.
What should the worker record after the queue ACK?
The worker reads the outbox item, builds the outbound support notification, and records each attempt with its time and response class. It reuses the same business key, such as a ticket event identifier plus action type, for every retry. It must not generate a fresh key per HTTP attempt, because that turns one uncertain external effect into several apparently new instructions.
Retry policy needs classification. A timeout can mean the destination applied the update and the response vanished, so retry only under a documented idempotency contract. A permanent validation response belongs in review or a dead-letter state rather than an endless retry loop. Authentication and authorization responses need credential handling. Backoff is useful only when the failure is plausibly transient.
Consider a ticket event ticket-4821.updated, and follow the state rather than the HTTP request. The receiver first stores the delivery identifier and the raw event reference, then commits the outbox row in the same transaction. It sends a 204, but the response is lost between the subscriber and the queue. The queue sends the event again, so the second transaction checks the unique delivery identifier before it can create another business action. It records the duplicate observation and ACKs it, because the first committed transaction already transferred responsibility to the worker. The worker later sends the notification with the ticket event's stable business key. Its connection times out after the request leaves the process, which means the destination may have applied the update even though no response reached the worker. The worker retries with the original key, records the response class and timestamp, and leaves the result explicitly uncertain if the recipient cannot confirm what happened. Reconciliation can then compare one outbox action, two intake observations, and the recipient's idempotency result. Without those records, a second queue delivery looks like a second customer instruction; with a new key on the retry, even the recipient has no reliable way to tell a repeated attempt from a new update. This sequence does not promise a magical exactly-once network. It preserves enough evidence to reconcile one uncertain effect instead of guessing from access logs.
For support payloads that may contain personal data, the audit trail should retain the delivery ID, state transitions, timestamps, response classes, and a reference or digest rather than copying unrestricted content into every log. Retention and redaction must follow the applicable privacy and compliance policy. Your mileage may vary: payload sensitivity, queue retention, and regulatory duties determine how much evidence is appropriate.
Test the uncertain states before choosing a pattern
The happy path is a weak test for this system. The useful test matrix interrupts the flow after each durable transition: before the claim, after the claim but before the ACK, after the ACK response is written, and after the outbound request leaves the process. For each interruption, restart the subscriber or worker, replay the same delivery ID, and assert that the audit trail contains one business action rather than one action per attempt. A test that checks only a final 200 cannot expose the lost-response case that causes duplicate support notifications.
The queue contract belongs in integration tests, while signature verification belongs in focused unit tests. Use a known raw byte sequence and verify that changing one byte invalidates the MAC; verify that malformed encodings and missing identifiers do not reach the store. Then exercise the real transaction and uniqueness constraint with concurrent deliveries of the same identifier. The expected result is one committed outbox item and one or more duplicate observations, not an assumption that the queue will serialize requests for you.
Observe the same state machine in deployment. Alert on authenticated deliveries that never become claimed, claimed items that never become outbox work, and outbox work whose final result remains unknown beyond the operational review window. The exact window depends on the queue's retry policy and the recipient's contract, so I would document it with the team rather than inventing a universal timeout. Keep a replay tool restricted and auditable; replaying a support notification is a business action, not a harmless debugging command.
The rejected synchronous option and its valid use
The rejected option sends the outbound notification inside the public push handler and ACKs only after the destination responds. It has fewer moving parts, and it can be reasonable for a low-volume integration whose recipient has a strict idempotency contract and a predictable latency budget.
It is a poor default for delayed webhook tasks. A downstream timeout can occur after the recipient applied the update but before the handler received its response; the queue then cannot tell whether to redeliver, while the process has no durable record separating “not sent” from “sent, result unknown.” The asynchronous handoff costs a store, a worker, and reconciliation work. Those costs buy an explicit failure boundary and a reviewable audit trail.
I would approve the synchronous pattern only when those assumptions are written into the destination contract and tested under lost responses. For a customer-support workflow where duplicate notifications are materially harmful, the durable claim plus outbox is the more defensible architecture. It makes the difficult state visible instead of hiding it behind a long request.
Top comments (0)