An every-minute cron trigger changes a clinical background queue's security and recovery boundary: the public webhook can admit a drain run, but it must never become a second worker or a source of unbounded replay. At that point, another timer or another replica can make the outcome worse because both can admit work faster than a rate-limited dependency can accept it.
Short answer: use an every-minute cron trigger only to admit one deduplicated drain run through an authenticated public webhook, then let a separately paced worker pool recover the queue under explicit age, rate, and concurrency limits.
The minute boundary is a control-plane event, not permission to empty the queue. That distinction keeps scheduler retries away from clinical work, gives operators one switch for stopping new drain runs, and makes rollback possible without throwing away jobs already accepted. The recovery objective should be written before deployment: preserve durable work, stay below the downstream start-rate allowance, and bring the oldest eligible job back inside its completion SLO.
Stop there first.
Authenticate the public trigger before admitting a drain
The public endpoint is the new control-plane boundary, so its first design artifact should be a threat model rather than a cron expression. Give the scheduler a narrowly scoped credential, compare the presented secret without timing-sensitive string logic, cap the body size, reject unknown fields, and accept only a recent scheduled timestamp. Rotate that credential independently of worker credentials. If admission must be revoked during recovery, disabling the scheduler identity should stop new drain tokens without preventing workers from acknowledging jobs already in flight.
Replay is the important abuse case even when the caller is trusted. A captured request for one minute must not become an unlimited command to wake workers, and a legitimate network retry must not create a second run. The stable minute-and-job-class identifier handles both cases when the durable store enforces uniqueness. Authentication answers who may ask; atomic admission answers how many times the request may take effect.
What must the recovery SLO prove before cron starts?
Before enabling the cron trigger, define what makes the drain unsafe. Queue depth is useful for capacity planning, but it isn't the primary abort signal because a large set of quick jobs can be healthier than a small set of slow or repeatedly throttled jobs. Oldest eligible job age shows whether the recovery is actually catching up. Pair it with the rate of accepted starts, in-flight work, retry ratio, and the downstream response class. In a healthtech pipeline, logs and metric labels should carry a job class and low-cardinality reason code, never a clinical payload.
The arithmetic is deliberately plain. Let R be the permitted starts per second, W the worker slots, S the observed average service time in seconds, and A the fresh arrival rate. The theoretical drain rate is min(R, W/S) - A. If that value is zero or negative, the backlog cannot recover under the present limits. Faster scheduling doesn't change that.
Consider the planning case already on the whiteboard: 2,400 document-classification jobs are waiting, fresh work arrives at 10 jobs per second, the dependency permits 40 starts per second, and the pool can start 50 per second at the observed service time. The idealized net drain is 30 jobs per second, so recovery takes 80 seconds before retries and tail latency. Raise arrivals to 35 per second and the same backlog takes 480 seconds. Those are planning inputs, not benchmark results. I'm not sure the average alone is good enough for a production decision when service-time tails are wide; a representative load test and a queue-age graph are what resolve that uncertainty.
Write the abort conditions next to that model. Pause new drain admission if oldest-job age rises for two consecutive evaluation windows, if starts exceed the written allowance, or if retry growth consumes the planned recovery margin. Do not delete the queue and do not manufacture a new run identifier during rollback. Stop admission, let bounded in-flight work settle, preserve delayed retries, and return worker concurrency to the last verified setting.
This is the uncomfortable capacity-planning reflex: calculate the slope before adding replicas.
How can Node.js cron trigger a queue every minute under a rate limit?
The Node.js process should act as a clock even though the admission and worker examples below are Go, as required by this implementation's code standard. For each UTC minute and job class, derive a stable run identifier. Send that identifier, the scheduled timestamp, and the job class to an authenticated public webhook. The handler validates the small request, atomically records the run and its drain token once, then returns after durable admission. A retry for the same minute receives the same successful admission result without creating more work. This is the answer to the implementation question: Node.js cron can trigger rate-limited queue processing every minute without owning that processing.
Kubernetes makes the reason for this contract explicit: a CronJob can create two Jobs or no Job in some circumstances, so the work should be idempotent. concurrencyPolicy controls overlap among Jobs created by one CronJob, and startingDeadlineSeconds constrains late starts, but neither setting proves that an HTTP attempt reached durable storage exactly once. The webhook therefore needs its own deduplication boundary.
The public request should contain no protected health information. A scheduling window and job class are enough; workers can resolve authorized records inside the trusted processing boundary. Authentication, replay protection, a small request-size ceiling, and a hard handler timeout belong at this edge. The success condition is accepted control work, not completed clinical work.
package admission
import (
"crypto/subtle"
"encoding/json"
"net/http"
"strings"
"time"
)
type Trigger struct {
RunID string `json:"run_id"`
ScheduledAt time.Time `json:"scheduled_at"`
JobClass string `json:"job_class"`
}
type RunStore interface {
// Admit atomically records the run and creates one durable drain token.
Admit(r *http.Request, trigger Trigger) (created bool, err error)
}
type Handler struct {
Token string
Runs RunStore
Now func() time.Time
}
func (h Handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
if r.Method != http.MethodPost {
http.Error(w, "method not allowed", http.StatusMethodNotAllowed)
return
}
supplied := strings.TrimPrefix(r.Header.Get("Authorization"), "Bearer ")
if subtle.ConstantTimeCompare([]byte(supplied), []byte(h.Token)) != 1 {
http.Error(w, "unauthorized", http.StatusUnauthorized)
return
}
r.Body = http.MaxBytesReader(w, r.Body, 4096)
var trigger Trigger
decoder := json.NewDecoder(r.Body)
decoder.DisallowUnknownFields()
if err := decoder.Decode(&trigger); err != nil {
http.Error(w, "invalid trigger", http.StatusBadRequest)
return
}
if trigger.RunID == "" || trigger.JobClass == "" || trigger.ScheduledAt.IsZero() {
http.Error(w, "missing trigger field", http.StatusBadRequest)
return
}
if trigger.ScheduledAt.Before(h.Now().Add(-2 * time.Minute)) {
http.Error(w, "stale trigger", http.StatusConflict)
return
}
created, err := h.Runs.Admit(r, trigger)
if err != nil {
http.Error(w, "admission unavailable", http.StatusServiceUnavailable)
return
}
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(http.StatusAccepted)
_ = json.NewEncoder(w).Encode(map[string]any{
"accepted": true,
"created": created,
"run_id": trigger.RunID,
})
}
Two details carry most of the safety. First, the caller must derive run_id from the UTC scheduled minute and job class rather than generating it for each attempt. Second, storing that identifier and creating the drain token must be atomic. A read-then-enqueue sequence has a race — two callers can observe absence and both enqueue a drain.
The clock and workers need different controls.
Can the rate-limited worker pass failure-injection tests?
A drain token wakes processing; it doesn't grant unlimited starts. Use a shared rate limiter to pace dependency calls and a separate semaphore to cap in-flight work. Rate controls starts over time. Concurrency controls memory, connections, and the accumulation of slow calls. Increasing the semaphore cannot overcome a binding start-rate limit, while increasing the rate without enough worker slots only changes where requests wait.
Start recovery below the calculated ceiling, observe one evaluation window, and increase only when queue age falls, retry ratio stays within its budget, and the downstream start count remains compliant. This staged approach gives each change a reversible boundary. It also prevents a common operator mistake: scaling from a depth graph, seeing throughput rise briefly, and discovering later that retries have converted the apparent gain into more queued work.
The worker contract needs bounded attempts, delayed retry, dead-letter isolation, and idempotency at the final side effect. A worker can finish an external action and lose its acknowledgement, so queue delivery semantics alone cannot protect the clinical record. Authentication or schema failures call for producer or operator correction; retrying them on a timer just spends capacity. A temporary rate-limit response can be delayed according to the dependency's published policy. Don't retry everything.
package drain
import (
"context"
"time"
)
type Job struct {
ID string
Attempts int
}
type Queue interface {
Receive(context.Context) (Job, error)
Ack(context.Context, Job) error
Retry(context.Context, Job, time.Time) error
DeadLetter(context.Context, Job, string) error
}
type Processor interface {
Process(context.Context, Job) error
}
func RunWorker(ctx context.Context, q Queue, p Processor, startsPerSecond, maxAttempts int) error {
pace := time.NewTicker(time.Second / time.Duration(startsPerSecond))
defer pace.Stop()
for {
select {
case <-ctx.Done():
return ctx.Err()
case <-pace.C:
job, err := q.Receive(ctx)
if err != nil {
continue
}
if err := p.Process(ctx, job); err != nil {
if job.Attempts+1 >= maxAttempts {
_ = q.DeadLetter(ctx, job, "attempt limit reached")
continue
}
delay := time.Duration(1<<min(job.Attempts, 6)) * time.Second
_ = q.Retry(ctx, job, time.Now().Add(delay))
continue
}
_ = q.Ack(ctx, job)
}
}
}
This ticker allows steady starts and no intentional burst. A token bucket with capacity greater than one has different boundary behavior. Your mileage may vary because downstream limits may use fixed or rolling windows; published semantics or a controlled staging test must decide the limiter, not the quota number by itself. AWS also warns that a dead-letter queue can break exact ordering, so an order-sensitive workflow needs a sequence-repair design before poison jobs are moved aside.
Small steps win.
Verify rollback before changing operational ownership
Run three checks before calling the recovery complete. Replay the same scheduled minute and confirm only one drain token exists. Terminate a worker after its external side effect but before acknowledgement and confirm the idempotency key prevents a duplicate effect. Finally, pause admission, return concurrency to its previous value, and verify that durable queued and delayed jobs remain available while oldest-job age stops being distorted by fresh drain runs. The dashboard should make accepted runs, deduplicated runs, starts, in-flight work, retries, dead letters, and oldest eligible age distinguishable without exposing clinical data.
Assign the buy-versus-build recovery duties
The buy-versus-build decision follows the rollback drill, because the queue is only one line in the on-call bill.
| Operating model | Suitable when | The catch | Recovery proof |
|---|---|---|---|
| Managed scheduler and queue | A small team needs durable primitives and documented service objectives | Quotas, delivery semantics, residency, and exit options constrain the design | Export metadata and replay it into a second implementation |
| Self-hosted scheduler and queue | The team already operates stateful systems and needs deployment control | Storage recovery, upgrades, and round-the-clock ownership remain internal | Restore from backup, deduplicate a minute, and drain under the rate limit |
| Database-backed job table | Business state and work admission need one transaction at modest scale | Polling load, indexes, leases, and delayed work become application concerns | Expire a lease and prove duplicate execution is harmless |
A managed service is not suitable when its residency or delivery boundary conflicts with the workload. Self-hosting is not suitable when the team cannot staff storage recovery and upgrades. Stick with a database-backed table when transactional coupling matters more than independent queue scaling, but move away from it when polling and lease contention consume the database's operational margin. No option removes the need for idempotency, capacity math, and a rehearsed pause switch.
The cron expression is the easy part. Operational recovery succeeds when the minute signal is deduplicated, the drain rate is measurable, and an operator can reverse a capacity change without losing durable work or violating the dependency's limit.
Top comments (0)