Understanding the Core Concepts of an SQS Consumer

Introduction

Queues look boring from the outside: push a message in, pop a message out. Almost none of the interesting behavior is in those two calls — it's in the gap between them, in the third
call most tutorials skip: delete. SQS never removes a message just because you received it, and that one design choice is the root of everything a consumer has to get right: retries, duplicates, dead-letter queues, backoff, parallel throughput.
Get the delete contract wrong and every one of those becomes a production incident instead of an expected case you coded for.

Mental Model for Consumer

The flow for a SQS consumer can be categorized into 3 Stage: Receive, Process, and Delete:

Receive: Query a message from the Queue and turn it invisible for other consumes
Process: Work done by the our application
Delete: ACK sent to Queue in order to remove the message from the SQS

This steps show the whole flow, if the process crashes before Delete, the message will be visible again, this give us retries, but introduces the possibility of duplication, so we must have idempotent.

Standard SQS works with the idea of At-least-once delivery, duplication are normal, and with Best-effort ordering.

A brief on FIFO queues, it sets to exactly-once delivery within 5 minutes dedup window. Which decreases the throughput which goes to 300 msg/s or 3k with batching, and the messages needs to carry a MessageGroupId, Standard queues have no effectively unlimited throughput. FIFO matter only when out of order processing is genuinely incorrect, and keep and mind that exactly once is for delivery and not processing, so it still necessary a idempotency. Because the same message outside the 5 minutes window, the same message but with different MessageGroupId, or a redrive can cause the duplication.

1. Visibility timeout — the foundation everything rests on

When a consumer receives a message, SQS starts a visibility timeout: a window during which that message is hidden from every other ReceiveMessage call. The message is "in flight."
Two things can happen:

• You delete it before the window expires → gone for good, usually done on actively on success.
• The window expires first (you crashed, hung, or were too slow) → the message becomes visible again and is redelivered.

This single mechanic is the engine behind retries, DLQs, and duplicate delivery. Understand it and the rest is bookkeeping.

Sizing the timeout

The visibility timeout must comfortably exceed the worst-case time to process a full batch, because a batch is received under a single shared window, the timeout is set at the SQS itself, through terraform or aws console. Rule of thumb:

visibility_timeout  >  P99(per-message work) × messages_in_flight_per_consumer  +  margin

The default is 30s, and the maximum it can be 12 hours. Too short and you get spurious redeliveries, the message reappears while you're still working on it, a second consumer picks it up, and now you're processing it twice concurrently.
Too long and a genuinely crashed message sits invisible for ages before anyone retries it, inflating latency and hiding failures.

The in-flight ceiling

There is a limit cap on in-flight messages (received but not yet deleted) at ~120,000, if your consumers are slow and your timeout is long, you can hit this ceiling and and ReceiveMessage(call to another batch of messages) starts returning empty even though the queue is backed up. FIFO's ceiling is far lower (~20,000), another reason Standard scales better.

Extending the window mid-flight

For long or variable jobs or backoff strategies, we don't need to set one giant timeout. Start modest and call ChangeMessageVisibility to extend the lease as you make progress, a heartbeat. This keeps a crashed consumer's messages returning quickly while still protecting long-running healthy work.

Kotlin example

private fun extendVisibility(message: Message, timeoutSeconds: Int) {
    try {
        sqsClient.changeMessageVisibility(
            ChangeMessageVisibilityRequest.builder()
                .queueUrl(queueUrl)
                .receiptHandle(message.receiptHandle())
                .visibilityTimeout(timeoutSeconds)
                .build(),
        )
    } catch (e: Exception) {
        log.warn("Failed to extend visibility for message {}", message.messageId(), e)
    }
}

private fun process(message: Message) {
    val heartbeat = CoroutineScope(Dispatchers.IO).launch {
        while (isActive) {
            delay(20_000)  // renew before a 30s timeout expires
            runCatching { extendVisibility(message, 30) }
        }
    }
    try {
        handler.handle(message.body())
        deleteMessage(message)
    } catch (e: JacksonException) {
        deleteMessage(message)
    } catch (e: Exception) {
        log.error(...)
    } finally {
        heartbeat.cancel()
    }
}

Don't worry about all the code, pay attention on the function extendVisibility, the async scope that create a delay and call the extention every 20 seconds, and the finally that cancels the assync scope.

Basic flow on handling messages

When processing a message, map scenarios and decide to delete the message, or ignore it to active the redrive. Usually delete message are triggered on success, but we can add it on some erros, like Malformed messages, it is a conscious call because it will prevent debug of this message, but other exception we may choose to ignore the delete to send the message to the DLQ

Kotlin example

private fun process(message: Message) {
    try {
        handler.handle(message.body())
        deleteMessage(message)          // success → consume
    } catch (e: JacksonException) {
        // poison: can never succeed → drop it (see §3, §7)
        log.error("Malformed message {}; deleting it as unprocessable", message.messageId(), e)
        deleteMessage(message)
    } catch (e: Exception) {
        // transient/unknown → DO NOTHING; let it reappear for redrive
        log.error("Failed to handle message {}; leaving it for redrive", message.messageId(), e)
    }
}

Capping retries: maxReceiveCount

Infinite retries are a bug: a message that always fails will loop forever, burning throughput and log volume. The cap is the redrive policy's maxReceiveCount, after a message has been received that many times without being deleted, SQS moves it to a
Dead-Letter Queue. This is the bridge from "retry" to "give up safely."

3. Dead-Letter Queues "where poison goes to be examined"

A Dead-Letter Queue is an ordinary SQS queue that a source queue points at via its redrive policy. Once a message on queue has been received 5 times, default value, without deletion, SQS
delivers it to Dead Letter Queue instead.
The DLQ has no special powers, its value is isolation: failing messages stop clogging the main queue, and you get a quarantined place to inspect, alarm on, and eventually redrive them back once the bug is fixed, it can be done by AWS console, usually it is done manually by console or terminal.

What belongs in a DLQ

• Transient failures that exhausted retries: the dependency was down longer than maxReceiveCount × visibility_timeout. These are candidates for redrive-back later.
• Genuinely un-processable messages you couldn't detect up front: a schema you didn't anticipate, a referenced entity that never materializes.

DLQ with messages is a incident

It is neccessary to put alarm on the DLQ's
ApproximateNumberOfMessagesVisible > 0 (CloudWatch). A silent DLQ is a DLQ nobody reads, which defeats the point. Also set the DLQ's own retention to the maximum (14 days) so you have time to react before evidence expires.

3. Parallel reading — long polling and batch fan-out

Throughput on a Standard queue comes from two multipliers: reading efficiently and processing concurrently.

Long polling vs. short polling

ReceiveMessage with WaitTimeSeconds = 0 is short polling: it samples a subset of the queue's servers and often returns empty even when messages exist, so you burn API calls spinning. Setting WaitTimeSeconds to 1–20 turns on long polling: the call waits up to that long for a message to appear, queries all servers, and returns as soon as anything's available. It cuts empty receives, reduces cost, and lowers latency. opt for long polling.

Batch receive

One ReceiveMessage call returns up to n messages set on (MaxNumberOfMessages). Batching amortizes the round-trip and is the unit of parallelism,receiving n messages doesn't help if you process them one at a time, you must worker/async/parallel the messages

The shared-visibility-window caveat

Every message in a batch is received under one visibility timeout that started at the same instant. If one confirmation blocks for 25 seconds while the others finish in 200ms, all ten are racing the same clock — but the clock only governs messages still in flight. The fast ones are deleted (permanently, off the
queue) well before it expires, so the clock expiring is a no-op for them — deletion isn't "undone" by a later timeout. Only the still-in-flight straggler is at risk: if its own handle() + deleteMessage() don't complete before the shared clock runs out, it becomes visible again and can be redelivered — possibly duplicating work if it finishes and deletes just after.
If per-message latency is highly variable, either shrink maxMessages, or extend visibility per-message with ChangeMessageVisibility.

Scaling out

Beyond one instance's batch fan-out, horizontal scaling is free: run N worker instances all long-polling the same queue. SQS's visibility mechanic guarantees a given message goes to one
consumer at a time, so instances don't coordinate. Batch size × concurrency × instance count is your throughput dial. (This is precisely where Standard's near-unlimited throughput beats FIFO.)

4. Idempotency, the non-negotiable consequence of at-least-once

Because delivery is at-least-once, your handler will be invoked more than once for the same logical message. Be after a visibility-timeout lapse, a redrive, a producer retry, or a
consumer crash between process and delete. An idempotent handler makes every one of those redeliveries a no-op. This is not optional hardening; it's the price of using a Standard queue correctly.

Usual Strategies, roughly in order of preference

1. Conditional state transition (best when you have state anyway).: Guard the write so it only fires from the expected prior state,re-delivery finds the state already advanced and does nothing.

2. Natural idempotency key.: Use a business identifier already in the payload as the dedupkey, no separate bookkeeping.

3. Dedup table / processed-ID set.: Record each processed message/event ID; skip on repeat. Necessary when the operation has no natural guarding state (e.g. "send an email").

Conclusion

Everything above collapses back into the three-word model from the preface: receive → process → delete. Every mechanism this guide covered — retries, DLQs, backoff, parallel fan-out, idempotency — exists only because delete is explicit and the gap between receive and delete is where the magic happens.