PrestonCole1111

Posted on Oct 29

AWS SQS FIFO vs Standard Queues for Healthtech Payment Retry Design

#sqs #fifo #queues #payments

Short answer: Use Standard for independently idempotent nightly snapshot jobs; use FIFO only when per-entity ordering or queue deduplication is a real invariant.

For a nightly healthtech payment reconciliation, choose queue semantics from the failure boundary you need to preserve, not from the retry count. Standard delivery fits work that is independently safe to repeat and does not require ordering. FIFO fits work that must remain ordered within a message group and benefits from queue-supported deduplication.

The invariant is narrower than exactly once: a clinic and settlement window produce one durable reconciliation result, a retry carries the same key, and a duplicate message cannot create a second ledger adjustment. A queue cannot make a remote payment provider call atomic with a local database.

Should FIFO or Standard queues handle retrying failed background jobs?

Consider clinic 1842 and the 2026-10-24 settlement window. The worker fetches provider rows, compares them with the internal ledger, stores the result, and emits an exception for review. A timeout is retryable. A malformed provider row is not.

Boundary	Required behavior	Decision implication
Identity	Clinic and settlement window map to one job	Put a stable key in both message and database
Ordering	Related mutations cannot race	Use a FIFO group only for a real domain sequence
Delivery	A crash leaves work available	Acknowledge only after durable recording
Poison data	Permanent errors stop retrying	Use an operator-visible dead-letter path

A Standard queue provides at-least-once delivery and best-effort ordering. FIFO adds ordered processing within a message group and queue-supported deduplication, subject to the documented limits. The phrase within a message group matters. Making every clinic one group serializes unrelated clinics.

Let the database enforce the business invariant; let the queue transport work.

Standard or FIFO: where does the failure stop?

A worker can receive a message, call the provider, commit locally, and die before acknowledgement. On redelivery, the provider call may happen again. It can also acknowledge early and lose the local result. Neither queue mode closes that two-system gap.

The idempotency key must be visible at each boundary. The following worker skeleton keeps the queue interface out of the domain model:

from dataclasses import dataclass
from datetime import date


@dataclass(frozen=True)
class ReconciliationJob:
    clinic_id: str
    settlement_date: date
    attempt: int

    @property
    def key(self) -> str:
        return f'reconcile:{self.clinic_id}:{self.settlement_date.isoformat()}'


def process(job, ledger, provider, results, clock):
    if results.already_completed(job.key):
        return
    rows = provider.fetch_settlement(
        clinic_id=job.clinic_id,
        settlement_date=job.settlement_date,
        idempotency_key=job.key,
    )
    comparison = ledger.compare(job.clinic_id, rows)
    results.record_once(
        key=job.key,
        status=comparison.status,
        difference_count=comparison.difference_count,
        observed_at=clock.now(),
    )

The acknowledgement follows the commit. In production, record_once needs a unique database constraint and must treat a uniqueness conflict as an already-completed result. A provider operation that changes remote state needs a provider-supported idempotency mechanism too.

The limitation of Standard is clear: it is not suitable when a real per-entity sequence must be preserved by the queue. Its trade-off is that the consumer owns duplicate handling and race prevention. FIFO has the opposite limitation: it is a poor fit when groups are arbitrary, because a failed message can hold up unrelated work. Its trade-off is useful only when ordering or queue-level deduplication is a real requirement. Standard is appropriate when clinic jobs are independent, comparison is repeatable, and the ledger write is idempotent. FIFO earns its complexity when an account has a strict open, adjust, close sequence. A snapshot comparison often has no such sequence requirement.

How should retries handle poison messages?

Retries are a control loop. Classify before requeueing. A timeout, connection reset, or temporary rate response can receive bounded exponential backoff with jitter. A schema error, unknown clinic, or invalid settlement date needs quarantine and an alert. Replaying a permanent error at 02:00, 02:10, and 02:20 only hides the incident.

Keep the original job key, attempt number, first-seen time, error class, and next eligible time. Do not create a new business key for each attempt. That makes queue metrics look healthy while duplicating the ledger operation.

A FIFO failure can block later messages in its group until the failed work is resolved or moved according to configuration. That is useful when later work depends on it. It is harmful when the group was chosen as a convenient partition rather than a domain dependency. Standard gives fewer ordering promises and therefore demands stronger consumer-side idempotency.

For healthtech, the dead-letter path needs the same access-control and retention review as the primary path. Put references needed for diagnosis in the payload, not unnecessary patient data. Keep audit evidence separate when that reduces exposure, and make the operator action explicit: replay, correct the source, or close the exception.

What do other queue models change?

This is a comparison of delivery models, not a product contest. RabbitMQ exposes routing and acknowledgement primitives, Kafka organizes an append-only log around partitions and consumer offsets, and Redis Streams provides stream entries with consumer-group coordination. Their operational shapes differ, so changing systems does not preserve failure behavior automatically.

Use the same questions everywhere: what identifies work, what is ordered, what happens after a worker crash, how long failed work remains available, and who owns deduplication. A queue may provide a useful primitive while leaving business idempotency to the application.

I would test this workflow with an adversarial matrix: duplicate a message; kill the worker after the provider call but before commit; kill it after commit but before acknowledgement; send a permanent schema error five times; and run two identical clinic windows concurrently. The expected result is one durable reconciliation record, bounded retry history, and an operator-visible exception.

Track oldest pending-job age, retry count by error class, dead-letter depth, processing latency, duplicate-key conflicts, and the delay between provider completion and local recording. Queue depth alone will miss several delivery failures.

Which option belongs in the decision record?

The rejected option is FIFO for every task because payment data feels like a reason to demand ordering. It is valid when a real per-entity sequence exists and later events cannot be evaluated before earlier ones. It is no substitute for an idempotent ledger write.

The other rejected option is Standard plus an assumption that duplicates are rare. A rare duplicate at the wrong point in a payment workflow remains a correctness failure.

Record these invariants:

One key identifies a clinic and settlement window.
A remote mutation uses a stable idempotency key.
A message is acknowledged after the local result is durable.
Retryable and permanent failures have different destinations and alerts.
Ordering is scoped to the smallest entity that needs it.

For the nightly snapshot job, start with independent work items, durable idempotency, bounded retries, and a reviewable dead-letter path. Use FIFO only for work whose correctness depends on per-group ordering or queue-level deduplication. The queue choice should state the guarantee; it should not pretend to solve the database-provider boundary.

DEV Community