ThomasMoore157

Posted on Aug 15

Node.js Delayed Public HTTPS Dispatch Explained for 10,000 Long-Running Property Jobs

#scheduling #reliability #go

Short answer: use cron to discover work that is due, a durable queue to hold each cleanup attempt, and a public HTTPS endpoint only to accept or acknowledge work; make the database idempotency key, not the timer, the authority on whether a property-management cleanup has already run.

That split resolves the important trade-off. A cron process is good at asking, "What is due now?" It is a poor place to hide 10,000 independent lease-cleanup timers. A queue is good at absorbing bursts and redelivering unfinished work, but redelivery means the handler must be safe to run again. An HTTPS request is a delivery boundary, not a waiting room for a long-running job.

Consider a bounded incident scenario, not a customer claim: at 02:00, a daily scan finds 10,000 expired document-upload grants across a property portfolio. The first 6,000 cleanups finish, the worker loses its lease on a message, and delivery resumes. If the cleanup operation means "delete every expired grant for property P as of cutoff T," replay is harmless. If it means "decrement the active-grant counter," replay corrupts state. The invariant is blunt: delivery can repeat; the business effect cannot.

How should per-event delayed webhook tasks handle long-running jobs?

Separate the schedule, delivery, and effect. The schedule decides when a cleanup becomes eligible. Delivery moves an attempt to available compute. The effect changes property data. Combining all three behind one public endpoint makes timeout policy, retry policy, and business correctness impossible to reason about independently.

For a per-event task, store a durable row when the event is created. A lease ending on May 31 might create a cleanup intent with a stable key such as lease-4831:revoke-upload-grants:2026-05-31. A periodic dispatcher claims due intents in bounded batches and publishes that key. A worker then acquires or records the key in the same transactional boundary as the mutation. The exact schema will vary, but the state machine should remain small: pending, claimed, completed, or permanently rejected after an explicit policy decision.

Don't let a request wait until the cleanup finishes. The public HTTPS handler should authenticate, validate, persist the intent, and return 202 Accepted with an opaque job identifier. A status resource can report progress if a caller needs it. This keeps ordinary intermediary and client timeouts out of the execution contract; it also prevents a caller retrying a timed-out POST from silently creating a second cleanup, provided the caller supplies an idempotency key and the service enforces uniqueness.

There is a catch. A 202 response only says that processing was accepted; it doesn't prove the work succeeded. The status model, retention period, and terminal failure policy are part of the API contract. I'm not sure there is one retention value that fits every property portfolio: compliance review, tenant-support response times, and database volume should determine it. Measure those constraints instead of choosing 30 days because it sounds tidy.

Short jobs can still use the same boundary. Consistency is useful.

The incident lesson is a state transition, not a timer choice

The illustrative 02:00 burst has two distinct failure windows. First, the application can commit a lease change but fail before publishing its cleanup message. Second, a worker can perform the cleanup but fail before acknowledging delivery. The transactional outbox pattern addresses the first window by recording the business change and an outgoing intent in one database transaction, then letting a relay publish the intent later. Idempotent consumption addresses the second by making replay converge on the same result.

This is where teams often focus on the wrong number. Queue latency is visible, so it gets a dashboard; the unobservable gap between a committed lease change and a missing message is rarer and more damaging. Track both. I would define an SLO for due-to-start latency, such as the proportion of cleanup intents claimed within the allowed window, and a separate correctness indicator for intents that are overdue without a terminal state. Throughput, oldest-ready age, retry count, and dead-letter volume are supporting signals, not substitutes for those two outcomes.

Capacity planning starts with arrival shape rather than the average. Ten thousand jobs due at 02:00 are not 10,000 jobs spread over a day. If one worker safely completes five cleanups per second and the operational objective is to drain the burst within ten minutes, the rough lower bound is four concurrent workers because 10,000 / (5 * 600) is 3.34. That is only a starting estimate: downstream rate limits, database lock time, retry amplification, and tenant-level fairness can lower safe throughput. Test the burst with production-shaped records and inject duplicate delivery. Otherwise the calculation is decoration.

Retries need classification. A network interruption may justify exponential backoff with jitter. Invalid property data should move directly to a reviewable terminal state because repeating the same input wastes capacity. Rate limiting should respect the receiver's signal. Set a maximum attempt count or maximum retry age, then make exhaustion visible to an operator; an infinite retry loop turns one malformed lease into permanent background load.

No magic here.

A preventative Go boundary

The following handler demonstrates the narrow contract. Store.Accept must atomically enforce uniqueness on the caller's key and persist the cleanup intent. Returning the existing job ID for a repeated key makes a retried POST stable. The worker-side transaction still has to guard the business effect because a queue may deliver the accepted job more than once.

package cleanup

import (
    "context"
    "encoding/json"
    "errors"
    "net/http"
    "time"
)

var ErrKeyConflict = errors.New("idempotency key belongs to different input")

type Request struct {
    PropertyID string    `json:"property_id"`
    LeaseID    string    `json:"lease_id"`
    DueAt      time.Time `json:"due_at"`
}

type Job struct {
    ID    string `json:"id"`
    State string `json:"state"`
}

type Store interface {
    // Accept persists the intent and returns the existing job on an exact replay.
    Accept(ctx context.Context, key string, req Request) (Job, error)
}

type Handler struct {
    Store Store
    Now   func() time.Time
}

func (h Handler) Schedule(w http.ResponseWriter, r *http.Request) {
    key := r.Header.Get("Idempotency-Key")
    if key == "" {
        http.Error(w, "missing Idempotency-Key", http.StatusBadRequest)
        return
    }

    var req Request
    dec := json.NewDecoder(http.MaxBytesReader(w, r.Body, 1<<20))
    dec.DisallowUnknownFields()
    if err := dec.Decode(&req); err != nil || req.PropertyID == "" || req.LeaseID == "" {
        http.Error(w, "invalid cleanup request", http.StatusBadRequest)
        return
    }
    if req.DueAt.Before(h.Now()) {
        http.Error(w, "due_at must not be in the past", http.StatusUnprocessableEntity)
        return
    }

    job, err := h.Store.Accept(r.Context(), key, req)
    if errors.Is(err, ErrKeyConflict) {
        http.Error(w, "idempotency key conflict", http.StatusConflict)
        return
    }
    if err != nil {
        http.Error(w, "request could not be accepted", http.StatusServiceUnavailable)
        return
    }

    w.Header().Set("Content-Type", "application/json")
    w.Header().Set("Location", "/cleanup-jobs/"+job.ID)
    w.WriteHeader(http.StatusAccepted)
    _ = json.NewEncoder(w).Encode(job)
}

The example deliberately doesn't enqueue from the handler after a separate database commit. Accept should write both the cleanup intent and its outbox record in one transaction. A relay can claim outbox rows with a bounded lease and publish them; only after successful publication should it mark the row delivered. On the consumer side, use the stable intent ID as a unique processed-message key inside the same transaction that revokes expired grants. If the cleanup calls an external system that cannot join that transaction, prefer an idempotent target operation such as "set access to revoked" and pass a stable request key when the target supports one.

Deployment deserves the same skepticism as code review. Run the old and new workers against compatible job payloads during a rolling release, version payloads when semantics change, and stop claiming new jobs before process termination while allowing an explicit drain interval. A worker killed after its effect and before acknowledgement is a routine replay case. Test it on purpose.

Buy versus build under an on-call budget

The mechanism can be obtained several ways, but labels conceal important limits. Amazon SQS FIFO queues provide ordered message groups and deduplication behavior, while their delivery semantics still require consumers to be designed for retries. RabbitMQ exposes acknowledgements and dead-letter exchanges, which gives operators control but also leaves cluster operation and topology choices with the owning team. PostgreSQL can support an outbox and claimed-job table close to the source transaction, though high queue churn then competes with application workload and demands vacuum, index, and lock attention. Kubernetes CronJob is a natural fit for periodic discovery, while its documentation explicitly notes that scheduling is approximate and jobs should be idempotent.

Those are boundaries, not rankings.

Option	Best fit	Operational ownership	Important boundary
Managed queue	Burst absorption and independent workers	Policies, consumers, observability	Service-specific delay, ordering, and retention constraints
Self-hosted broker	Teams needing protocol or topology control	Broker upgrades, capacity, recovery, clients	On-call owns the data plane
Database outbox plus workers	Strong coupling to an existing transaction	Table growth, polling, locks, cleanup	Queue load shares database capacity
Cron plus direct execution	Small, bounded, naturally periodic batches	Scheduler and job runtime	Weak fit for independent per-event retries

My decision rule would be to keep cron plus direct execution only when the whole scan is cheap, rerunning the batch is safe, and finishing inside one bounded execution window meets the SLO. Choose a durable queue when events have separate due times, burst size can exceed immediate worker capacity, or failures need item-level retry. Choose a database-backed outbox when losing the handoff after a business commit is unacceptable. A managed queue can reduce broker on-call work; a self-hosted broker can be reasonable when the team already operates it well or needs control unavailable in a managed service. Lock-in is not an abstract concern, so isolate publish and consume semantics behind a small internal job envelope, but don't pretend ordering and retry behavior are portable without design changes.

This architecture is not suitable for every cleanup. If a single task must hold a database transaction open for hours, split it into checkpoints or use a workflow engine with durable step state. If all work is a five-second nightly reconciliation over one compact table, stick with one idempotent scheduled batch; adding a broker creates more failure surfaces than it removes. If legal or operational policy requires a human approval between steps, a plain queue is insufficient because approval state, expiry, and audit history need explicit workflow modeling.

Release gates and the final decision

Before enabling the schedule, prove four behaviors in a staging environment with production-shaped volume: duplicate the same message, terminate a worker after the effect but before acknowledgement, pause consumers while the due backlog grows, and replay an outbox row. The pass condition is not "the queue recovered." It is that each cleanup reaches one terminal business outcome, no tenant monopolizes capacity, and overdue work raises an actionable alert.

Make the dashboard reflect the state machine. Count accepted, due, claimed, completed, retried, and permanently rejected intents; graph the age of the oldest due item; attach the stable job ID to logs and traces. Alert on user impact or SLO burn rather than every retry. One retry is expected behavior. A rising oldest-item age while workers report success is a paging signal because the system is falling behind despite looking busy.

The final choice is therefore conditional: cron discovers periodic eligibility, a queue carries independently retryable work, and HTTPS accepts intent without staying open. Keep the effect idempotent, close the commit-to-publish gap with an outbox, and size workers against the burst and downstream limit. When the workload is a small bounded batch, keep the simpler cron design. That limitation is part of the recommendation, not an exception hidden in fine print.

DEV Community