Short answer: use a scheduled batch run when the entire eligible set is bounded and one durable cursor can account for it; use a work queue when API rate limits, variable duration, or independent retry make each item an operational obligation of its own.
The least complex design is a clock that starts a bounded sweep. The important qualifier is bounded: the sweep must finish with room to spare, it must tolerate a second invocation, and it must leave durable evidence of which records were considered and which effects were committed. A timer is useful for making work eligible. It is a poor substitute for ownership, retry state, and reconciliation.
For payment-like or ledger-adjacent processing, the distinction is not academic. A downstream call may complete just before a worker exits, while the local success record is still absent. The next attempt cannot infer the truth from a green dashboard. It needs an idempotency key, an attempt record, and a way to reconcile an uncertain outcome.
What should own rate-limited API batch processing: the clock or the queue?
Start by assigning each concern to a component with a narrow promise. A scheduler decides when a logical window is eligible. A selector reads that window and turns it into stable item identifiers. A queue, when needed, holds those identifiers while they wait for a consumer. The consumer obtains rate-limit permission, performs one effect, writes the durable result, and only then acknowledges its item.
That division matters because a scheduled invocation has no inherent per-item lifecycle. If a 10-minute run selects 80,000 records and stops after 20,000, a new run needs application code to distinguish unfinished work from completed work. Once that code contains delayed retry, leases, attempt counts, and a terminal review state, the system has built a queue-shaped protocol around a clock. It may still be the right choice, but the cost should be acknowledged rather than hidden in a handler.
Use a queue once any of these become true:
- Work arrives faster than the permitted API rate for more than a short interval.
- Items have meaningfully different durations or failure behavior.
- A delayed retry must preserve one item's history without delaying unrelated items.
- Operators need to inspect backlog age and ownership per item.
- More than one consumer can process the same logical batch.
A queue does not provide exactly-once business effects. Delivery is ordinarily at least once, so an item can appear again after a consumer loses its lease or fails before acknowledgement. AWS describes the SQS visibility timeout as the period during which other consumers do not receive a message; it is a temporary claim, not proof that the effect happened exactly once. The application must make the effect repeatable through an idempotency key or a uniqueness rule on the business operation.
A lease is not a receipt.
The practical order is durable attempt, remote call, durable result, acknowledgement. There is still an ambiguous interval between the remote call and the local result. Reusing the same idempotency key makes a redelivery converge when the remote API supports it. When it does not, serialize by business key and hold ambiguous attempts for reconciliation instead of replaying them automatically. That policy is slower, and it is often the only defensible one for value-moving work.
The constraint test before choosing the cheapest or easiest path
“Cheapest” needs an operational definition. Invocation charges and runner minutes are visible; recovery labor, audit queries, secret rotation, log retention, and failed replay drills are not. A small nightly export with a deterministic cursor often has a lower total cost as one scheduled process, because adding a queue would introduce dashboards and ownership rules that the workload does not need. An unbounded synchronization job can make the opposite choice: a simple clock appears inexpensive until overlapping runs and partial progress require a custom state machine.
| Processing shape | Smallest reasonable mechanism | Evidence that must remain durable | Reason to move beyond it |
|---|---|---|---|
| Small, bounded sweep | Scheduled handler plus transactional cursor | Logical window, cursor, item result | A run can overlap or exceed its time budget |
| Rate-limited backlog | Scheduler releases identifiers; consumers drain a queue | Attempt, limiter decision, lease, result | Per-item delay and retry need independent ownership |
| High-consequence external effects | Queue or durable work table with controlled consumers | Idempotency key, remote reference, reconciliation state | The remote side cannot deduplicate uncertain attempts |
| Urgent and routine work together | Separate classes only after measuring starvation | Class, age, service target, terminal disposition | Priority policy hides a capacity shortfall |
Priority is a useful example of why category labels are insufficient. RabbitMQ documents priority queues, including their resource implications and the need for consumers to have time to reprioritize messages. A priority field is therefore not a cure for an overloaded dependency. First determine whether the API quota is global, per tenant, per credential, or per endpoint; then record which dimension denied admission. Without that record, “slow” could mean an exhausted quota, an old lease, a poisoned payload, or a consumer that has stopped progressing.
The same classification avoids a misleading comparison among cloud cron facilities, a cloud scheduler, and a queue such as SQS. Vercel Cron and GitHub Actions cron can initiate an action; in a Node.js service, that action may select a bounded window or publish stable identifiers for consumers. They do not turn a trigger into a per-item work ledger. Don't let the deployment location decide the processing contract: retain the cursor and audit trail for a bounded cron run, or introduce queue ownership when the batch needs it.
Rate limiting belongs near the point of dispatch. A limiter kept in each process allows total throughput to rise with replica count, which quietly violates a shared provider quota. A shared admission mechanism can enforce the aggregate limit, but it should also support the dimensions the provider actually exposes. Keep the limiter decision in the audit trail with an item key and attempt number. This gives reconciliation a timeline rather than a guess.
The catch is plain. A work queue is not suitable for a tiny, easily recomputed batch whose effects are idempotent and whose completion window is comfortably below the scheduled interval; retain the clock, cursor, and audit records. A clock-only design is not suitable when backlog age has a service consequence, individual items need different retry delays, or variable work duration makes overlap normal. Compliance constraints can narrow either design further: retention, deletion, access segregation, and residency rules depend on the data classification and jurisdiction. There is no generic retention interval that can be asserted safely; the applicable control framework and counsel must set it.
Make retries observable and idempotent
The worker should not treat every error as a retry. An explicit rate-limit response can be delayed. A malformed business record should reach a durable review state. A cancellation should be checked before a side effect begins. Backoff needs jitter so a common outage or quota boundary does not create a synchronized retry wave, and an attempt cap protects the system from turning bad input into permanent load.
Measure the boundary.
One useful review exercise is to write the timeline for a single item on paper, including the database transaction that records intent, the moment the request leaves the process, the provider's response, the local commit, and the acknowledgement. Then mark every process termination and network partition that could occur between those points. For example, a timeout after the provider has accepted a request is not equivalent to a rejected request, even though both may look like the same transport error to a Go client. The first requires an idempotent replay or remote-state lookup; the second may be retried under the provider's quota policy. If the system cannot tell those cases apart, it should preserve the attempt as uncertain and let reconciliation decide. That decision may require a remote reference, a request fingerprint, or an operator review queue. It should not be hidden inside a generic retry helper, because a helper cannot know whether the business effect moved money, changed access, or merely generated a report.
The following Go sketch leaves queue and storage adapters generic. Its important property is the persisted business key before dispatch; implementation names are deliberately unimportant.
package batch
import (
"context"
"time"
)
type Item struct {
ID string
IdempotencyKey string
AccountKey string
}
type Queue interface {
Receive(context.Context) (Item, error)
Acknowledge(context.Context, Item) error
Delay(context.Context, Item, time.Duration) error
}
type Attempts interface {
Start(context.Context, Item) error
Commit(context.Context, Item, string) error
Review(context.Context, Item, string) error
}
type Limiter interface {
Wait(context.Context, string) error
}
func ProcessOne(ctx context.Context, q Queue, attempts Attempts, limiter Limiter) error {
item, err := q.Receive(ctx)
if err != nil {
return err
}
if err := attempts.Start(ctx, item); err != nil {
return err
}
if err := limiter.Wait(ctx, item.AccountKey); err != nil {
return q.Delay(ctx, item, time.Minute)
}
// Send uses item.IdempotencyKey for every delivery of this business operation.
remoteRef, err := Send(ctx, item)
if err != nil {
return q.Delay(ctx, item, time.Minute)
}
if err := attempts.Commit(ctx, item, remoteRef); err != nil {
return err
}
return q.Acknowledge(ctx, item)
}
The code is intentionally incomplete as an adapter boundary, not as a failure policy. Start must tolerate a redelivery of the same item, Commit must accept an already-recorded remote reference, and Send must use the stable idempotency key. Test the four awkward stops: before sending, after sending, after the result commit, and before acknowledgement. Also test an item whose processing time approaches the queue lease, then prove that another consumer cannot begin the same effect while the original consumer is still legitimately working.
Metrics should describe obligations, not merely process health. Track oldest eligible work, oldest queued work, time spent waiting for the limiter, lease renewals, retries by classification, terminal-review count, and reconciliation mismatches. Throughput can be high while one account remains starved. A single stale cursor can be hidden by successful invocations. Those are different failures and deserve different alerts.
Roll out the change by moving ownership, not by duplicating effects
Before changing transport, make the current selection deterministic. Give every candidate item a stable identifier and derive a logical window that can be recomputed. Persist the expected item set or its digest, then compare it with the set released to consumers. This is where a migration usually earns its keep: it exposes cursor gaps, time-zone assumptions, and filtering changes before they create external effects.
Run the new release path in observation mode first, with consumers disabled, and compare its planned identifiers against the existing path. Next, move a partition for which the external API accepts idempotency keys, while excluding that same partition from the old owner. Expand only after backlog age, reconciliation, and terminal review remain within the agreed operating limits. Do not let two independent paths own the same business key and hope deduplication becomes a migration strategy.
For the bounded case, stop sooner. A scheduled handler with a transactional cursor, explicit overlap protection, and a reconciliation report is easier to explain and easier to retire. For the open-ended case, let the clock release work and let the queue make every item visible until its outcome is known. The architecture follows the obligation, not the brand of timer.
Top comments (0)