Short answer: a public HTTPS queue push webhook subscriber in Node.js should verify the signature on the raw request, persist a unique cleanup command, and ack only after that durable write; a bounded worker should own retries. The request should never wait for the marketplace cleanup itself.
That sounds almost too conservative. It is exactly what makes a delayed task boring to operate. A marketplace can tolerate a cleanup running a few minutes late; it can't easily tolerate deleting a listing twice, charging a seller twice, or turning one delivery retry into two irreversible side effects.
Fast ack.
I frame the design around one bounded scenario: an expired marketplace listing is placed on a delayed queue, a push delivery reaches a public endpoint, and a worker removes stale search and reservation data. The endpoint's job is to accept responsibility for that command. It is not the worker, and it is not a second scheduler hiding inside an HTTP request.
What should a public HTTPS queue webhook subscriber verify before ack?
First, verify the request against the exact bytes received. Parse the payload after authentication, not before it, because re-serialization can change the bytes covered by an HMAC. Check the delivery identifier and the message type as well. An authenticated message with an unexpected schema is still not a valid cleanup command.
Next, claim the delivery in durable shared state. The claim needs a uniqueness constraint on the delivery ID, or on a business key such as listing_id plus cleanup generation when the producer can legitimately issue a new delivery for the same listing. A process-local map is not a claim: it disappears on restart and says nothing to the next instance behind the load balancer.
Only then should the handler return the queue's success acknowledgement. If the database is unavailable, the honest response is a non-success response that allows redelivery; silently acknowledging after an in-memory write creates loss that a dashboard may never reveal.
The following Go example uses an application-owned HMAC contract so the ordering is visible without pretending that every queue uses the same header names. StoreClaim must be backed by a shared transactional store and must return false for an already claimed delivery. The Node.js implementation would follow the same sequence even though this repository's editorial convention keeps examples in Go.
Imagine two copies of the endpoint receive the same listing-4821:expiry-7 command within a few milliseconds. Both can verify the signature. Only one can win the durable uniqueness check; the other sees the existing claim and acknowledges without creating another worker job. That race is the test case, not an edge case to be added later. I would run it while one instance is being drained and while the database transaction is close to its timeout, because the interesting question is not whether a single request works; it is whether the system has one answer when the handoff is interrupted. If the insert commits and the response is lost, the queue sends the command again and the second request must be harmless. If the insert does not commit, the retry must be allowed to establish ownership. If a worker starts and then loses its lease, a replacement worker must either resume from a safe checkpoint or repeat an operation whose key has the same idempotency meaning. Those are different states, and collapsing them into a boolean processed flag makes incident analysis unnecessarily speculative.
package main
import (
"crypto/hmac"
"crypto/sha256"
"encoding/hex"
"io"
"net/http"
)
type ClaimStore interface {
StoreClaim(deliveryID string, body []byte) (newClaim bool, err error)
}
func handler(secret []byte, store ClaimStore) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
if r.Method != http.MethodPost {
http.Error(w, "method not allowed", http.StatusMethodNotAllowed)
return
}
body, err := io.ReadAll(http.MaxBytesReader(w, r.Body, 256*1024))
if err != nil {
http.Error(w, "invalid body", http.StatusBadRequest)
return
}
supplied, err := hex.DecodeString(r.Header.Get("X-App-Signature"))
mac := hmac.New(sha256.New, secret)
_, _ = mac.Write(body)
if err != nil || !hmac.Equal(supplied, mac.Sum(nil)) {
http.Error(w, "invalid signature", http.StatusUnauthorized)
return
}
id := r.Header.Get("X-Delivery-ID")
if id == "" {
http.Error(w, "missing delivery id", http.StatusBadRequest)
return
}
_, err = store.StoreClaim(id, body)
if err != nil {
http.Error(w, "cannot durably accept delivery", http.StatusServiceUnavailable)
return
}
w.WriteHeader(http.StatusNoContent)
}
}
The important omission is also deliberate: the handler does not call the search index, payment service, or seller notification system. Those calls belong to a worker that can set a timeout, record an attempt, and retry the same command without holding an external connection open.
How should retry state and idempotency shape delayed cleanup?
Treat delivery, execution, and side effect as separate states. A useful record has a delivery ID, a business key, an attempt count, a next-attempt time, and a status such as accepted, running, succeeded, or dead_lettered. The exact schema is an implementation choice; the invariant is that a retry can locate the same command and inspect what happened before it tries again.
There are two different duplicate problems here. Queue delivery can repeat because acknowledgement and delivery are separate events. The cleanup operation can also repeat because a worker may finish the remote call and crash before recording success. The first is solved at intake with a unique claim. The second requires the side effect itself to be idempotent, or a provider operation that accepts an idempotency key. A transaction around the local job row does not magically make an HTTP call reversible.
For an expired listing, I would pass a stable cleanup key such as listing-4821:expiry-7 through every internal attempt. The worker can safely observe that the search document is already absent and mark the command complete, but it should not treat an arbitrary timeout as proof that the remote operation did not happen. That is where an idempotent downstream API, a status lookup, or a reconciliation pass earns its keep.
Retry only failures that may change. Authentication failures, malformed payloads, and a missing listing identifier need quarantine and an alert, not an endless redelivery loop. Timeouts, temporary connection failures, and a throttling response may be retried with exponential backoff and a cap. Keep the retry budget explicit; otherwise a small backlog can consume all worker capacity while newer cleanup commands wait.
The operational numbers should come from a capacity test, not from a copied default. If the queue receives 20 cleanup commands per second and one worker slot completes 2 per second, ten slots are the theoretical floor before failure and jitter; production needs headroom for retries, deployment drain, and slow downstream calls. Measure acceptance latency separately from completion age. A healthy intake endpoint can coexist with an unhealthy worker backlog, and a single queue-depth graph will hide that distinction.
A failure table for the intake-to-worker boundary
| Failure point | Correct interpretation | Recovery action |
|---|---|---|
| Signature check rejects | The request is not admitted | Record a security metric; do not create work |
| Claim store times out | Ownership is unknown | Return non-success and allow delivery retry |
| Duplicate delivery arrives | The command already has an owner | Acknowledge after confirming the existing claim |
| Worker times out downstream | Completion is unknown | Retry only with an idempotency key or reconciliation path |
| Payload is valid but business key is absent | The command cannot be safely executed | Quarantine it and page the owning team |
| Retry budget is exhausted | Automation needs human or scheduled repair | Move it to a dead-letter workflow with context |
This table is the part I want in a design review, because it forces the team to name who owns each ambiguous state. “The queue will retry” is not a recovery plan for a side effect whose outcome is unknown.
Where does a push endpoint stop being the right boundary?
Push delivery is a good fit when the consumer can expose public HTTPS, keep authentication material available for verification, and durably accept a small command before doing slow work. It reduces the need for a long-lived poller, but it adds an ingress surface, certificate and secret rotation, request limits, and an acknowledgement contract that must be tested.
The catch is that this pattern is not suitable when policy requires private-only network access, when the workload needs a replayable event history with independent consumer groups, or when the command is really a multi-step workflow with joins and human approval. Use a pull consumer for a network boundary that cannot accept inbound traffic. Use a log or workflow system when replay and coordination are first-class requirements. Keep a simple scheduler when all it has to do is enqueue a bounded command; the scheduler should not become the cleanup worker by accident.
A queue also does not remove the need for deployment discipline. Drain workers before a release, make shutdown leave running work recoverable, and test two simultaneous deliveries with the same ID. Test a worker crash after the remote call but before the local success update. Test a slow downstream service while watching the public handler's latency SLO. These tests exercise the real contract instead of proving only that a handler returns 204 on the happy path.
The decision rule I would put in the runbook
Choose the smallest architecture that preserves three facts: an authenticated delivery was durably accepted, a retry can find the same command, and every irreversible effect has a duplicate-safe operation or a reconciliation plan. If any one of those facts is missing, adding more worker replicas will increase activity without increasing correctness.
Your mileage may vary on the storage choice and on the retry interval; the right values depend on listing volume, downstream limits, and the recovery objective. I am less flexible about the order. Verify raw bytes. Claim durably. Acknowledge. Execute asynchronously. Reconcile ambiguity.
That is enough structure for a marketplace cleanup job to survive ordinary retries without turning a delayed webhook into a second source of truth.
Top comments (0)