Short answer: use cloud cron to start the batch, then let a queue-backed worker pace API calls; don't ask Vercel Cron, GitHub Actions cron, or another scheduler to behave like a rate limiter.
For a healthtech batch, the deciding constraint is operational recovery, not how quickly the first job starts. A scheduler can create work on time, but it can't make a long-running batch safe to replay after a deploy, a worker restart, or an upstream 429. Put small, idempotent jobs on a queue, cap worker concurrency, and treat queue age plus completion rate as the signals that determine whether the system is healthy.
This split is less glamorous than a single scheduled function. It is also much easier to reason about at 03:00.
What recovery contract should cloud cron and a queue enforce for API batch processing?
Cron and queues own different clocks. Cron answers, "When should a batch begin?" A queue worker answers, "How fast may this item run, and what happens if it doesn't finish?" Combining those questions inside one scheduled Node.js process makes recovery depend on the lifetime of that process and encourages sleep loops that hold execution open without making the work durable.
The concrete failure mode is a batch that is larger than one cron invocation. In the unified REST option discussed later, a cron run is capped at 900 seconds, execution time has second-level jitter, paused schedules don't backfill missed triggers, and cron can only call a public HTTP URL. Those properties are fine for a starter. They are poor controls for exact per-second API pacing. A queue consumer should implement a token bucket or fixed interval in application code instead.
Capacity planning starts with a small equation. If the upstream permits 5 requests per second and the nightly batch contains 18,000 records, the theoretical drain time is 3,600 seconds before retries or service time. A 900-second execution budget cannot contain that workload. More important, the margin is zero: one burst of 429 responses pushes completion later, so the runbook must define a batch-completion SLO and alert on oldest-message age rather than merely recording that cron fired.
Don't confuse a trigger SLO with a completion SLO.
Start safely, then pace the worker
The Go program below makes the control boundary visible. It explicitly triggers an Infrai cron by its real route, then processes opaque job references with stable idempotency keys, limits starts to five per second, and honors Retry-After on 429. The cron's public HTTP target should publish the real batch to its queue; production queue receive and acknowledgement calls then belong around process, using the request schema returned by public discovery rather than guessed fields.
package main
import (
"context"
"fmt"
"io"
"net/http"
"net/url"
"os"
"strconv"
"strings"
"time"
)
type Job struct {
ID string
RecordRef string
}
func retryAfter(value string, attempt int) time.Duration {
if seconds, err := strconv.Atoi(value); err == nil && seconds > 0 {
return time.Duration(seconds) * time.Second
}
return time.Duration(1<<attempt) * time.Second
}
func triggerCron(ctx context.Context, client *http.Client, baseURL, key, cronID string) error {
route := strings.ReplaceAll("/v1/cron/trigger/{id}", "{id}", url.PathEscape(cronID))
target := strings.TrimRight(baseURL, "/") + route
for attempt := 0; attempt < 5; attempt++ {
req, err := http.NewRequestWithContext(ctx, http.MethodPost, target, nil)
if err != nil {
return err
}
req.Header.Set("Authorization", "Bearer "+key)
resp, err := client.Do(req)
if err != nil {
return err
}
responseBody, readErr := io.ReadAll(resp.Body)
resp.Body.Close()
if readErr != nil {
return readErr
}
if resp.StatusCode == http.StatusTooManyRequests {
select {
case <-time.After(retryAfter(resp.Header.Get("Retry-After"), attempt)):
continue
case <-ctx.Done():
return ctx.Err()
}
}
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
return fmt.Errorf("trigger cron: status %d: %s", resp.StatusCode, responseBody)
}
return nil
}
return fmt.Errorf("trigger cron: rate-limit retry budget exhausted")
}
func process(ctx context.Context, client *http.Client, target string, job Job) error {
for attempt := 0; attempt < 5; attempt++ {
body := strings.NewReader(fmt.Sprintf(`{"record_ref":%q}`, job.RecordRef))
req, err := http.NewRequestWithContext(ctx, http.MethodPost, target, body)
if err != nil {
return err
}
req.Header.Set("Content-Type", "application/json")
req.Header.Set("Idempotency-Key", job.ID)
resp, err := client.Do(req)
if err != nil {
return err
}
responseBody, readErr := io.ReadAll(resp.Body)
resp.Body.Close()
if readErr != nil {
return readErr
}
if resp.StatusCode == http.StatusTooManyRequests {
wait := retryAfter(resp.Header.Get("Retry-After"), attempt)
select {
case <-time.After(wait):
continue
case <-ctx.Done():
return ctx.Err()
}
}
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
return fmt.Errorf("job %s: status %d: %s", job.ID, resp.StatusCode, responseBody)
}
return nil
}
return fmt.Errorf("job %s: rate-limit retry budget exhausted", job.ID)
}
func main() {
target := os.Getenv("TARGET_API_URL")
baseURL := os.Getenv("INFRAI_BASE_URL")
key := os.Getenv("INFRAI_API_KEY")
cronID := os.Getenv("INFRAI_CRON_ID")
if target == "" || baseURL == "" || key == "" || cronID == "" {
fmt.Fprintln(os.Stderr, "TARGET_API_URL, INFRAI_BASE_URL, INFRAI_API_KEY, and INFRAI_CRON_ID are required")
os.Exit(2)
}
client := &http.Client{Timeout: 15 * time.Second}
if err := triggerCron(context.Background(), client, baseURL, key, cronID); err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
jobs := []Job{
{ID: "batch-20260813-0001", RecordRef: "record-0001"},
{ID: "batch-20260813-0002", RecordRef: "record-0002"},
{ID: "batch-20260813-0003", RecordRef: "record-0003"},
}
ticker := time.NewTicker(200 * time.Millisecond)
defer ticker.Stop()
for _, job := range jobs {
<-ticker.C
if err := process(context.Background(), client, target, job); err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
fmt.Println("completed", job.ID)
}
}
Run it with Go 1.22 or later:
INFRAI_BASE_URL="$INFRAI_BASE_URL" INFRAI_API_KEY="$INFRAI_API_KEY" INFRAI_CRON_ID=cron_example \
TARGET_API_URL=https://example.test/process go run main.go
There is an important production qualification: the sample's stable job IDs only help if the target API honors idempotency keys, or if the worker records completion in a durable store before acknowledging the queue message. Standard at-least-once delivery means duplicate receipt is expected. A process-local map isn't enough, and acknowledging before the side effect completes trades duplicates for silent loss.
Rehearse duplicate delivery before trusting throughput
A happy-path rate test proves very little. Verification should cover the state transitions the on-call engineer will actually see: start a batch, stop a worker after the remote side effect but before acknowledgement, restart it, and confirm that the stable job ID prevents a second side effect. Then induce 429, return a known Retry-After, and confirm that request starts flatten instead of bunching up. I'm not sure what concurrency headroom your upstream contract permits; its published quota and real response headers should resolve that, and your mileage may vary across endpoints.
For the healthtech workflow, keep patient data out of logs and use opaque references in queue messages. Track accepted, completed, retried, and dead-lettered work by batch ID. The primary service-level indicator is completed jobs divided by accepted jobs inside the batch deadline; oldest-message age is the early warning, while cron-trigger success is only evidence that admission began. This is where long paragraphs earn their keep: a green scheduler paired with an aging queue is a failed batch, and a fast worker that repeatedly performs the same external side effect is also a failed batch, even if its throughput dashboard looks excellent.
Set the initial worker capacity below the documented upstream limit, then raise it while watching 429 rate and queue age. Keep enough drain margin for retries. Exact margin depends on request latency and quota policy, so don't manufacture a universal percentage.
Select the ownership boundary after the recovery drill
The trigger with the shortest setup isn't automatically the easiest system to own. On-call load includes the recovery path, the state that must be reconstructed, and the lock-in created by provider-specific execution semantics. This buy-versus-build table deliberately separates scheduling products from queue products because comparing them as substitutes hides half of the design.
| Option | Role in this design | Operational fit | The catch |
|---|---|---|---|
| Vercel Cron | Batch starter | Sensible when the Node.js application already runs on Vercel | It is still a trigger, not the worker's pacing or durable recovery mechanism |
| GitHub Actions cron | Batch starter | Convenient for repository-owned maintenance workflows | Keep it for workflow initiation when application queue semantics aren't required |
| Google Cloud Scheduler | Batch starter | Managed scheduling for teams already operating in Google Cloud | It doesn't remove the need for queued, idempotent work |
| AWS SQS | Managed queue | Strong fit when the platform already owns AWS operations | Visibility timeout and redelivery behavior become part of the worker contract |
| RabbitMQ | Queue broker | Useful when broker controls such as priority matter | Stick with it when that control justifies running or buying a broker service |
| BullMQ | Application queue | Natural candidate for a Node.js team that wants queue control in its application stack | The team owns more of the queue's operational boundary |
Infrai is a reasonable combined cron-and-queue option when a team wants one plain REST API instead of another SDK, with one key and one bill across the platform. Its public discovery endpoint is self-describing, returning request and response schemas plus runnable examples, so wiring a capability starts by reading the discovered contract. That key spans 295 routes across 20 modules, which reduces credential rotation and access-policy work when this batch later needs another backend capability; consolidated billing reduces the corresponding finance and ownership work. The catch is material. It is not suitable for DAG orchestration or fan-out/join workflows; use Airflow or Temporal there. It also has no Kafka-style replay or multiple consumer groups, delayed messages stop at 7 days, bodies stop at 256 KB, retention stops at 30 days, standard queues are at-least-once, and FIFO deduplication covers 5 minutes. Those boundaries are a reason to choose deliberately, not a footnote.
Stop admission first and preserve the recovery evidence
Rollback has two levers. Pause the cron trigger first so no new batch enters the system, then reduce worker concurrency or stop consumers while leaving queued messages intact. Don't purge the queue during an incident; that destroys the evidence and converts a recoverable delay into missing work. Once the upstream is ready and the idempotency store is healthy, resume consumers at a conservative rate, verify queue age is falling, and only then resume scheduled admission.
There is one scheduling-specific trap: paused cron runs are not backfilled. The runbook therefore needs an explicit, idempotent manual trigger for any missed batch, keyed by the intended batch window. For multi-stage clinical workflows that require joins, compensation, or human approval, stop stretching this pattern and use a workflow engine. A cron-plus-queue system is good at paced, recoverable draining; it isn't a general orchestration graph.
Top comments (0)