Short answer: for rate-limited API batch processing, use cloud cron to discover shipment work, then use a queue and paced workers to deliver the updates. A cron job is a clock; it is a poor place to keep retry state, absorb duplicate delivery, or enforce one API quota across several workers.
That distinction is easy to miss because the first version of the system looks small: wake up, query pending shipments, call a partner API, and exit. The trouble starts when a partner returns 429, the process is terminated halfway through a page, or two scheduler invocations overlap. The design needs explicit invariants before it needs another scheduling product.
What should cloud cron or a queue own in rate-limited batch processing?
For an e-commerce shipment update fan-out, the scheduler should own discovery. It selects a bounded set of shipment events, assigns stable work IDs, and publishes one item per subscriber or per delivery attempt. The worker should own the slow path: one shared rate limiter, bounded retries, response classification, and the final acknowledgement.
The useful invariants are these:
- Every delivery has a stable idempotency key, such as
shipment_id:subscriber_id:version. - A business update is committed atomically with its idempotency record, or the receiver provides an equivalent idempotent operation.
- A message is acknowledged only after the remote effect and local state transition are complete.
- Retryable responses consume a bounded budget and respect
Retry-Afterwhen present. - The rate limit is global for the upstream account, not accidentally multiplied by the number of workers.
Keep the clock small.
The queue is not a magic rate limiter either. If five workers each believe they may send ten requests per second, the partner sees up to fifty requests per second unless the allowance is coordinated. A fixed delay inside each process only works when the quota is deliberately divided among workers and worker count is stable. A shared token bucket or a single pacing service makes the ownership visible.
There is a second boundary: queue delivery is normally at least once. A worker can send the update, lose its connection before acknowledgement, and receive the same item again. That is not a rare corner case to patch later; it is the normal reason the receiver needs an idempotency key.
Decision record: which architecture survives the failure modes?
The options below are not ranked. They move responsibility between a scheduler, a queue, a worker runtime, and the application database.
| Design | Good fit | You still must implement | The limiting trade-off |
|---|---|---|---|
| Cron-only handler | Small reconciliation with bounded work and acceptable missed runs | Cursor recovery, idempotency, pacing, and retry handling in one process | A long wait or process termination can lose the in-memory batch |
| Cron plus managed queue | Shipment fan-out with uneven latency or overlapping runs | Stable work IDs, consumer pacing, acknowledgement policy, and poison-item handling | More moving parts and an operational queue contract |
| Cron plus self-hosted broker | A team that already operates a broker and needs its controls | Broker durability, upgrades, consumers, limits, and alerting | The broker becomes part of the team's reliability budget |
| Workflow engine | Long-lived orchestration with joins, timers, and human or system states | Activity idempotency, upstream pacing, and workflow history limits | It is more machinery than independent delivery items require |
The table's uncomfortable row is the first one. Cron-only can be the easiest choice for a tiny batch, but “easy” stops meaning “fewest components” once retries, partial completion, and reconciliation live inside one handler. A queue adds a boundary that can be inspected and resumed, while also forcing the team to operate a consumer correctly.
Visibility is part of that consumer design. AWS documents a queue visibility timeout as the period during which a received message is hidden from other consumers; if the worker does not finish before that period, the message can become visible again. The worker therefore needs a timeout longer than its expected processing window, extension logic for unusually slow calls, and idempotency even when the timeout is configured carefully. A larger timeout reduces some duplicate deliveries but delays recovery from a genuinely stuck worker.
Priority is a different concern. RabbitMQ's priority queue feature lets a broker prefer higher-priority messages, but priority does not create capacity at the partner API. A high-priority shipment update can still be rejected if every worker shares the same exhausted quota. Keep urgency and pacing as separate policies.
The catch is that a queue is not suitable when the team cannot own consumer deployment, dead-letter review, and quota monitoring. Stick with a bounded cron handler when the batch is deliberately small, the receiver is idempotent, and a missed run can be reconciled later. Choose a workflow-oriented design when the job has real joins or long-lived state. The right choice follows the failure boundary, not the dashboard's setup time.
A Python critical path for duplicate delivery
This example is intentionally plain. It shows where pacing and idempotency belong; the storage and HTTP client are application boundaries, not details to hide in a scheduler callback.
import time
from dataclasses import dataclass, field
@dataclass
class TokenBucket:
requests_per_second: float
capacity: float = 1.0
tokens: float = field(init=False)
updated_at: float = field(init=False)
def __post_init__(self) -> None:
self.tokens = self.capacity
self.updated_at = time.monotonic()
def acquire(self) -> None:
while True:
now = time.monotonic()
elapsed = now - self.updated_at
self.tokens = min(
self.capacity,
self.tokens + elapsed * self.requests_per_second,
)
self.updated_at = now
if self.tokens >= 1.0:
self.tokens -= 1.0
return
time.sleep((1.0 - self.tokens) / self.requests_per_second)
class IdempotencyStore:
def __init__(self) -> None:
self.completed: dict[str, str] = {}
def apply_once(self, key: str, shipment: str) -> str:
if key not in self.completed:
# Production storage must make this check and write atomic.
self.completed[key] = f"sent:{shipment}"
return self.completed[key]
def deliver_batch() -> None:
limiter = TokenBucket(requests_per_second=2.0)
store = IdempotencyStore()
deliveries = [
("shp-104:sub-8:v3", "shp-104"),
("shp-105:sub-8:v2", "shp-105"),
("shp-104:sub-8:v3", "shp-104"),
]
for key, shipment in deliveries:
limiter.acquire()
print(store.apply_once(key, shipment))
if __name__ == "__main__":
deliver_batch()
The in-memory store is only a model of the state transition. In a real consumer, put the idempotency key in a database with a uniqueness constraint, or use the receiver's documented idempotency mechanism. Do not mark the item complete before the remote call succeeds. Do not repeat a non-idempotent request merely because the client timed out; first decide whether the server may have applied it.
Three response classes help keep retries boring: retry a rate limit or transient transport failure after backoff, record a permanent validation failure for review, and treat an already-applied idempotency key as success. A dead-letter queue is useful only if someone owns the review loop. Otherwise it is a quiet archive of customer-visible omissions.
I’m not sure a “cheapest” comparison can be honest without event volume, retention, worker hours, and the team's existing infrastructure. Your mileage may vary. The durable decision is to pay the operational cost where the failure is easiest to observe: discovery in cron, delivery in a queue, and correctness at the receiver.
Rejected design: one cron loop for every subscriber
The rejected design queries all pending shipments and sends every subscriber update in one scheduled process. It has attractive local simplicity, especially when the partner API is fast and the list is short. It fails as a general batch architecture because a timeout, deployment, or rate-limit response interrupts both discovery and delivery; the next run must infer which side effects happened without a durable per-item record.
That design still has a valid use case: a periodic reconciliation whose maximum work fits comfortably inside one invocation, whose outbound operation is idempotent, and whose missed interval is explicitly acceptable. For shipment fan-out with strict quotas, however, the scheduler should enqueue a bounded batch and return. The worker can then retry one item without replaying the entire scan.
The operational checklist is short but not optional: measure queue age and retry count, alert on quota responses and dead-letter growth, cap batch size, make overlapping discovery runs harmless, and run a shutdown test during delivery. Test duplicate messages before testing throughput. Throughput is easy to demonstrate; correctness after a timeout is the part that pays the bills.
Top comments (0)