A failed game cleanup job must finish after its initiating web request is gone, and that changes the retry design: success means proving queue delivery, DLQ isolation, and selective redrive under crashes and duplicates, not merely proving that a scheduler can call an endpoint.
Short answer: use an at-least-once queue, an idempotent worker, a DLQ, and selective redrive for failed jobs; retain a durable attempt ledger in the application database, while keeping manual database polling for workloads whose low volume and relaxed recovery target justify owning the retry machinery.
This isn't a contest over which product has the shortest setup page. The useful comparison asks which failure evidence the on-call engineer gets, which transitions the application must build, and how much replay can hit the game database without breaking its SLO. Start there.
How should a small app compare failed job retry, DLQ redrive, and database polling?
Write a delivery claim that can fail a test. For a periodic cleanup of expired match sessions, a practical claim might be: every eligible match is eventually cleaned, duplicate deliveries create no duplicate side effect, permanently invalid jobs leave the ready path, and an operator can retry a corrected item without replaying unrelated failures. The exact completion target depends on traffic and dependency headroom; I'm not sure a universal latency number would be honest, so measure the cleanup service time and backlog arrival pattern before setting one.
Then map each claim to evidence. A scheduler success event proves only that the trigger ran. A zero-depth queue proves only that no messages are currently visible. Neither proves completed cleanup, because acknowledgement can remove a message and queue retention is finite. The application database therefore needs a durable ledger keyed by a stable job ID, with attempt state and the final business result, if audits matter.
The transport contract is at-least-once. Assume a worker can commit the cleanup and lose its acknowledgement, causing the same job to arrive again. Assume one malformed payload can fail every attempt. Those aren't exotic cases — they are the two tests that distinguish a delivery design from a timer plus hope.
Use this fault-evidence matrix before discussing vendors:
| Failure injected | Evidence required | Queue with DLQ | Manual database poller |
|---|---|---|---|
| Worker exits after commit | One business effect for two deliveries | Worker idempotency plus ack after commit | Transactional lease and idempotency code |
| Payload is permanently invalid | Bad job stops consuming retry capacity | Retry budget and DLQ isolation | Permanent-failure state and query rules |
| Consumers stop during a match surge | Backlog age recovers within the SLO | Queue age, depth, and worker capacity | Due-row age, lease recovery, and poller capacity |
| Operator retries one corrected job | Only the selected job runs again | Selective DLQ redrive | An audited state transition back to ready |
The database design can satisfy every row. The catch is that the team owns leases, lease expiry, exponential backoff, concurrency control, poison-job isolation, and stuck-job visibility. If those mechanisms already exist and the app processes a handful of non-urgent cleanups, keeping them may be sensible. If they don't, the apparently cheap SQL loop is a small queue implementation hiding in application code.
How can the worker make duplicate delivery harmless before choosing a transport?
The side-effect boundary comes first. Give each cleanup a stable business key such as cleanup:match-1842, write the cleanup result and processed receipt in one transaction, and acknowledge only after that transaction commits. A crash before commit permits retry; a crash after commit permits duplicate delivery; the receipt makes the second execution a no-op.
First, make the transport observable from the same runtime that will operate the worker. This runnable Go probe checks Infrai queue statistics through the verified GET /v1/queue/stats/{queue} route. Set INFRAI_BASE_URL to the API base, and set the key and queue name in the environment; the request uses an explicit method, bounds the response body, surfaces non-success responses, and treats 429 as backpressure by honoring Retry-After or applying exponential delay.
package main
import (
"context"
"fmt"
"io"
"net/http"
"net/url"
"os"
"strconv"
"strings"
"time"
)
func required(name string) string {
value := strings.TrimSpace(os.Getenv(name))
if value == "" {
panic(name + " is required")
}
return value
}
func retryDelay(response *http.Response, attempt int) time.Duration {
if seconds, err := strconv.Atoi(response.Header.Get("Retry-After")); err == nil && seconds > 0 {
return time.Duration(seconds) * time.Second
}
return time.Duration(1<<attempt) * time.Second
}
func main() {
baseURL := strings.TrimRight(required("INFRAI_BASE_URL"), "/")
key := required("INFRAI_API_KEY")
queue := url.PathEscape(required("QUEUE_NAME"))
routeTemplate := "/v1/queue/stats/{queue}"
route := strings.ReplaceAll(routeTemplate, "{queue}", queue)
client := &http.Client{Timeout: 15 * time.Second}
for attempt := 0; attempt < 5; attempt++ {
request, err := http.NewRequestWithContext(
context.Background(), http.MethodGet, baseURL+route, nil,
)
if err != nil {
panic(err)
}
request.Header.Set("Authorization", "Bearer "+key)
response, err := client.Do(request)
if err != nil {
panic(err)
}
body, readErr := io.ReadAll(io.LimitReader(response.Body, 1<<20))
response.Body.Close()
if readErr != nil {
panic(readErr)
}
if response.StatusCode == http.StatusTooManyRequests {
time.Sleep(retryDelay(response, attempt))
continue
}
if response.StatusCode < 200 || response.StatusCode >= 300 {
panic(fmt.Sprintf("queue stats returned %s: %s", response.Status, body))
}
fmt.Println(string(body))
return
}
panic("rate-limit retry budget exhausted")
}
Transport visibility doesn't make the side effect safe. The next runnable Go model deliberately sends the same job twice and rejects a poison payload. The mutex represents the atomic database transaction; replace it with a real transaction, but preserve the relationship between the business write and receipt.
package main
import (
"context"
"errors"
"fmt"
"sync"
)
type Job struct {
ID string
MatchID string
}
type Ledger struct {
mu sync.Mutex
processed map[string]bool
deleted map[string]bool
}
func NewLedger() *Ledger {
return &Ledger{
processed: make(map[string]bool),
deleted: make(map[string]bool),
}
}
// Apply models one transaction containing both the cleanup and its receipt.
func (l *Ledger) Apply(ctx context.Context, job Job) (bool, error) {
if err := ctx.Err(); err != nil {
return false, err
}
if job.ID == "" || job.MatchID == "" {
return false, errors.New("permanent: job ID and match ID are required")
}
l.mu.Lock()
defer l.mu.Unlock()
if l.processed[job.ID] {
return false, nil
}
l.deleted[job.MatchID] = true
l.processed[job.ID] = true
return true, nil
}
func main() {
ledger := NewLedger()
jobs := []Job{
{ID: "cleanup:match-1842", MatchID: "match-1842"},
{ID: "cleanup:match-1842", MatchID: "match-1842"},
{ID: "cleanup:missing-match"},
}
for _, job := range jobs {
applied, err := ledger.Apply(context.Background(), job)
switch {
case err != nil:
fmt.Printf("isolate %s: %v\n", job.ID, err)
case applied:
fmt.Printf("cleaned %s\n", job.MatchID)
default:
fmt.Printf("duplicate ignored: %s\n", job.ID)
}
}
}
Run it twice mentally as well as literally. The first valid delivery changes business state, the second does not, and the invalid payload is classified for isolation instead of consuming retries forever. Temporary dependency failures belong on exponential backoff with jitter; permanent validation failures belong in the DLQ. Don't blur the two.
Keep queue messages to identifiers and immutable routing data. A 256KB message ceiling exists, but payload size isn't the main reason: loading current match state by ID lets the worker re-check whether cleanup is still valid. It also avoids treating a retained message as the authoritative copy of mutable or sensitive game data.
Break the recovery path on purpose
Before production, create a staging exercise with three inputs: one valid cleanup ID, the same ID twice, and one payload that cannot pass validation. Stop consumers after the first delivery, restart them, and confirm that the business ledger shows one completed cleanup, one harmless duplicate, and one isolated failure. Correct the source record, redrive only that item, and retain the link to its original stable ID.
Watch oldest-ready-message age, attempt-count distribution, DLQ depth, and completed ledger entries against the cleanup SLO. Depth alone is ambiguous. It falls when work succeeds, but it also falls when a consumer acknowledges too early, so reconcile transport signals with business completion.
Redrive needs a capacity envelope. Suppose a tournament leaves 12,000 expired matches and measured cleanup time is 250 ms at the constrained dependency: one serial worker represents roughly 50 minutes of service demand before retry overhead. More workers reduce elapsed time only until storage rate limits or database connections saturate. Set the redrive rate below measured spare capacity, stop when the same permanent error class returns, and never empty the whole DLQ just to clean a dashboard.
The rollback is short: halt redrive, leave unselected DLQ entries isolated, and keep normal consumption running.
Preserve the evidence.
Infrai fits a small polyglot platform when the integration constraint is “anything that can send HTTP”: it exposes the capability through a plain REST API, so there's no SDK or client-library version to maintain. Its single key across 295 routes in 20 modules is a second operational advantage because the platform team has fewer capability-specific credentials to distribute. This doesn't change at-least-once semantics or remove worker idempotency, and it doesn't make the product suitable for every recovery topology.
Choose the smallest system that meets the guarantee
A buy-versus-build review should charge engineering ownership to the option that creates it. “No new managed service” is not the same as “no new system”; a poller with leases and retry states is a system, and its pager belongs somewhere.
| Option | What it gives the delivery design | Where it stops fitting | Ownership decision |
|---|---|---|---|
| Infrai queue and cron | Queue, DLQ redrive, and a plain REST boundary under one key | No DAG or fan-out/join orchestration; standard queues remain at-least-once | Consider when a small team values a language-neutral API and broad backend integration |
| AWS SQS FIFO | A managed queue with documented FIFO ordering and deduplication behavior | Application audit history and idempotent side effects still belong to the app | Prefer when the workload and operators already live in AWS |
| GitHub Actions schedule | A documented scheduled workflow trigger | It is a scheduler choice, not the durable failed-job queue described here | Keep for repository automation, not the game cleanup delivery path |
| Temporal or Airflow | Workflow orchestration territory for DAGs and multi-step coordination | More machinery than a single periodic cleanup needs | Choose when the job becomes a workflow with joins or dependent steps |
| Inngest or Trigger.dev | An additional managed-workflow shortlist | Validate delivery, isolation, redrive, region, and operational semantics against the same tests | Compare when the team wants workflow-oriented alternatives |
| BullMQ, Sidekiq, or Celery | Familiar worker ecosystems worth testing with the application's runtime | The platform team must evaluate the backing service and on-call boundary | Compare when an existing language ecosystem already shapes operations |
| PostgreSQL polling | Durable application-owned rows with completely custom state | The team builds leases, backoff, isolation, concurrency, and visibility | Keep when volume is tiny, recovery can be slow, and that ownership is deliberate |
There are hard boundaries around the REST queue choice. Delay is capped at 7 days, retention at 30 days, and FIFO deduplication at 5 minutes; acknowledged messages are removed. Standard queues are at-least-once. There is no native debounce or throttle, topic-style one-to-many delivery, Kafka-style replay with multiple consumer groups, or DAG orchestration. Use N queues for a small fixed fan-out only if the extra operational surface is acceptable; use a log when independent replay is the requirement, and use Temporal or Airflow when the cleanup has become a workflow.
Cron is a trigger, not a worker host. Each execution is limited to 900 seconds and calls a public http_url; paused schedules don't backfill missed runs, timing can have second-level jitter, and recorded output retains only the first 4KB. Push targets likewise require public HTTPS. For longer work, let cron enqueue identifiers and return, then let workers process them. A private-only callback or a need for precise catch-up scheduling calls for a different scheduler design.
Release with an abort condition, not optimism
Shadow the new producer first: calculate which cleanup IDs it would enqueue, but keep the current path authoritative and compare candidate IDs with the database ledger. Then enable a small worker pool, deliberately deliver a duplicate, and verify one business effect before raising concurrency. The release gate is business completion within the chosen SLO plus bounded DLQ growth, not merely successful cron invocations.
Define the abort condition before rollout. Pause new redrive when oldest-message age rises despite added workers, when the constrained dependency reaches its safe capacity, or when one permanent error class repeats. Roll back producers to the prior path, allow already accepted valid jobs to drain if capacity permits, and preserve isolated messages for diagnosis. If the old poller and new producer overlap, fence them with the same stable job key; otherwise the migration itself creates an unbounded duplicate source. Record the cutover time, producer identity, accepted count, completed count, and isolated count in one operator note so the next shift can reconcile the ledger without reconstructing intent from dashboards. Purging removes evidence and is a poor incident response.
This is why the queue usually wins for failed-job recovery: acknowledgement, retry flow, and DLQ isolation make the delivery states inspectable, while the durable ledger makes them auditable. Manual polling remains a valid small-app choice when its slower recovery and code ownership are explicit. The decision is defensible only after the duplicate, poison-message, stopped-consumer, and selective-redrive tests pass.
Top comments (0)