Short answer: make the public HTTPS endpoint a short, authenticated delivery boundary, and make the durable worker own retries, idempotency, and the audit trail; use Express or Fastify for HTTP plumbing, but do not confuse a successful HTTP response with a completed payment reconciliation.
The concrete case here is a marketplace that runs a nightly reconciliation against its payment provider. A queued job might contain settlement-42, a business date, and references to provider pages. The job may arrive twice, arrive after a timeout, or be accepted by the receiver just before the process disappears. Those are normal delivery conditions, not exceptional stories to hide behind a framework.
Ack last.
What constraint does a public queue receiver impose?
A public push subscription changes the trust boundary. The receiver is an internet-facing command surface, so it must prove who sent a request before treating its body as executable input. TLS protects data in transit; it does not establish that the caller is the queue publisher. Authentication, body limits, replay controls, and careful logging belong at the edge.
The HTTP handler should do a small amount of work in a fixed order:
- Require
POSTand a bounded body. - Authenticate the delivery before acting on its data.
- Derive a stable job identifier from the delivery contract.
- Claim that identifier in durable storage.
- Perform a short, transactional state change or hand off to a durable private worker.
- Acknowledge only after the durable outcome is known.
That order matters. If the handler updates a ledger and writes its idempotency record afterward, a crash between those writes can post the same settlement twice. If it marks the job complete first, a crash can suppress work that never happened. The useful goal is exactly-once effects for a defined business mutation, not the stronger and usually misleading promise of exactly-once delivery.
For a job that may take minutes, the public handler should authenticate and persist an internal work item, then return only when that handoff is durable. A JavaScript promise left in a request handler is not a queue. Process exit, deployment, or a lost connection can remove it without leaving a recoverable state.
How can a Node.js background worker receive queued jobs securely over public HTTPS?
Express and Fastify make different routing and lifecycle choices, but neither changes the protocol contract. The receiver needs a public HTTPS address, a credential or verified signature, a maximum request size, and a durable claim store shared by every replica. The sample below uses Go because the important behavior is easier to inspect without hiding it inside a Node.js package; the same boundary can sit behind an Express or Fastify route.
package main
import (
"crypto/sha256"
"crypto/subtle"
"encoding/hex"
"io"
"log"
"net/http"
"os"
"sync"
)
const maxBodyBytes = 256 * 1024
var claims = struct {
sync.Mutex
seen map[string]struct{}
}{seen: make(map[string]struct{})}
func authorized(r *http.Request, token string) bool {
want := "Bearer " + token
got := r.Header.Get("Authorization")
return len(got) == len(want) &&
subtle.ConstantTimeCompare([]byte(got), []byte(want)) == 1
}
func claimOnce(id string) bool {
claims.Lock()
defer claims.Unlock()
if _, exists := claims.seen[id]; exists {
return false
}
claims.seen[id] = struct{}{}
return true
}
func process(payload []byte) error {
// Replace this with one transaction for the claim and business mutation.
log.Printf("reconciliation_bytes=%d", len(payload))
return nil
}
func main() {
token := os.Getenv("PUSH_TOKEN")
if token == "" {
log.Fatal("PUSH_TOKEN is required")
}
http.HandleFunc("/reconciliation/push", func(w http.ResponseWriter, r *http.Request) {
if r.Method != http.MethodPost {
http.Error(w, "method not allowed", http.StatusMethodNotAllowed)
return
}
if !authorized(r, token) {
http.Error(w, "unauthorized", http.StatusUnauthorized)
return
}
r.Body = http.MaxBytesReader(w, r.Body, maxBodyBytes)
payload, err := io.ReadAll(r.Body)
if err != nil {
http.Error(w, "body too large", http.StatusRequestEntityTooLarge)
return
}
digest := sha256.Sum256(payload)
jobID := hex.EncodeToString(digest[:])
if !claimOnce(jobID) {
w.WriteHeader(http.StatusNoContent)
return
}
if err := process(payload); err != nil {
http.Error(w, "processing failed", http.StatusConflict)
return
}
log.Printf("job_id=%s status=committed", jobID)
w.WriteHeader(http.StatusNoContent)
})
server := &http.Server{
Addr: ":8443",
ReadHeaderTimeout: 5_000_000_000,
ReadTimeout: 15_000_000_000,
WriteTimeout: 15_000_000_000,
IdleTimeout: 60_000_000_000,
}
log.Fatal(server.ListenAndServeTLS(os.Getenv("TLS_CERT_FILE"), os.Getenv("TLS_KEY_FILE")))
}
The map is deliberately a teaching aid, not a production ledger. It disappears on restart and cannot coordinate replicas. Replace it with a durable unique-key insert in the same database transaction that records the reconciliation result, or with an equivalent claim plus transactional outbox design. The identifier should come from the publisher's stable delivery ID; hashing the raw body is only a generic fallback when the contract supplies no such field, and it treats semantically identical JSON with different key order as different jobs.
I'm not sure which signature header or delivery envelope a chosen queue exposes without its current contract, so I would not invent one. If signed deliveries are available, verify the signature over the exact raw body, enforce its timestamp window, and rotate secrets without an authentication gap. If the publisher supplies only a bearer credential, keep that credential out of logs and configuration checked into source control.
When should retries, claims, and acknowledgments change?
Retries are a state machine, not a loop around process. For settlement-42, the application-owned record can move from started to committed or failed, with timestamps, an attempt count, the authenticated source, and a redacted failure class. A concurrent delivery that sees committed returns an acknowledgment without applying another ledger mutation. A transaction that rolls back leaves no committed claim, allowing a later delivery to try again.
The awkward interval is the one operators must be able to explain: the payment provider was updated, but the receiver timed out before acknowledgment. The queue will reasonably retry. The second attempt must find the durable committed state and produce one business effect plus an auditable duplicate decision. That is why a process log is insufficient.
I write this as a five-event test: commit, lose the response, redeliver, claim the existing ID, and compare the ledger before and after. A green HTTP check is not enough.
I treat 401 as a trust-boundary result, not a processing failure: reject oversized bodies before expensive work, and distinguish transient dependency failures from permanent validation failures. A transient failure should remain retryable; a permanent failure should be quarantined with enough metadata for an operator to inspect without exposing payment data. Do not acknowledge an uncommitted mutation merely to quiet a red dashboard.
Observability should follow the state transitions: delivery ID, job ID, attempt, claim result, processing latency, acknowledgment status, and redrive count. Correlate those fields across the public receiver, the private worker, and the reconciliation table. Metrics that count only HTTP 2xx responses can report a healthy edge while the ledger is quietly accumulating unresolved work.
Which queue boundary fits the payment reconciliation workflow?
The comparison comes after the failure model because the right boundary depends on what must remain private and what must be replayable.
| Boundary | Good fit | Trade-off |
|---|---|---|
| Public HTTPS push | A publisher can reach a small authenticated receiver and the job has a stable idempotency key | The receiver is exposed to the internet and must handle authentication, timeouts, replay, and redelivery correctly |
| Private queue consumer | Long work, private network access, or explicit control over concurrency is required | The team owns consumer deployment, broker credentials, scaling, and recovery behavior |
| Scheduled trigger plus private worker | A nightly reconciliation has a clear start time and the actual work should remain private | A trigger is not a durable work record; missed runs, overlap, and manual replay need an application policy |
| Workflow engine | Several durable steps, compensation, timers, or human review must be coordinated | The model carries more operational and conceptual surface than a queue receiving one idempotent job |
The public endpoint is not suitable when inbound internet traffic is prohibited, when the work needs a private database connection that cannot be reached through a controlled worker, or when the delivery contract cannot provide a stable identity. Stick with a private consumer in those cases. Conversely, a nightly trigger alone is not enough when a reconciliation can overlap, because two runs can inspect the same provider window and produce contradictory settlement decisions.
Exactly-once delivery is not the criterion I would put in a design review. Ask instead: which writes are idempotent, which claim is unique, where is the transaction boundary, what happens after acknowledgment is lost, and how can an operator reconcile a stuck record? Those questions survive a change of queue technology.
How should a secure queue subscription be tested and retained?
Test the ugly sequences before production: duplicate delivery of the same ID, two replicas claiming concurrently, a process stop after the ledger commit, a provider timeout, an invalid credential, and a payload just over 256 KB. Run the tests with a staging payment fixture whose expected balances are explicit. One test should prove that retrying a committed settlement-42 changes no balance; another should prove that a rolled-back transaction remains retryable.
Cron is useful for initiating the nightly run, but it should not become a disguised long-running worker. The supplied cron reference describes a 900-second execution ceiling, public http_url targets, no backfill for triggers missed while paused, and possible second-level timing jitter. That makes the trigger a timing mechanism, not the reconciliation archive. Persist the intended business date and run ID before dispatching work, and define what an operator does when the trigger is late or absent.
Retention is a compliance design choice. GDPR Article 17 describes a right to erasure in applicable circumstances, while financial reconciliation may require durable evidence of a transaction. Keep the minimum audit evidence needed to prove the state transition, separate erasable payload content from financial records, and define retention and deletion rules with the applicable legal and accounting requirements. Storing every raw provider response forever is not an audit strategy.
Keep it boring.
Start with one idempotent reconciliation class, a dedicated hostname, credential rotation, bounded request timeouts, and alerts on authentication failures and unresolved claims. Expand only after redelivery, replay, and deletion behavior are documented. Your mileage may vary because the provider's delivery semantics and the marketplace's accounting obligations determine the exact transaction boundary.
Top comments (0)