miruky

Posted on Sep 4

Prove FIFO Deduplication in Amazon SQS from the AWS Console

#aws #sqs #eventdriven #tutorial

Introduction

Hi, I'm miruky.

An Amazon SQS FIFO queue tracks MessageDeduplicationId values for a five-minute interval. When a producer repeats an ID during that window, SQS acknowledges the later send but does not make another copy available to consumers.

This Console run disables content-based deduplication so the IDs are explicit. It sends two different bodies with dedup-001, adds a control body with dedup-002, and then proves that consumers receive the original body and the control body—but not the duplicate body.

The validation uses one encrypted FIFO queue and a few SQS requests. Current SQS request pricing and free-tier terms are documented on the linked pricing page.

1. Create a FIFO queue with explicit deduplication IDs

The header confirms United States (N. Virginia) while the SQS Console is in English. us-east-1 is the fixed Region for this run.

Open Amazon SQS and search for miruky-fibqomwvgyqpoceu.fifo. The exact-name result should be empty before the queue is created.

The exact filter for miruky-fibqomwvgyqpoceu.fifo returns no queue. That empty result establishes the resource boundary before creation.

Choose Create queue, select FIFO, and enter miruky-fibqomwvgyqpoceu.fifo. Keep content-based deduplication disabled, use the default FIFO throughput settings, set the receive-message wait time to 20 seconds, and retain SQS-managed server-side encryption.

The form shows FIFO, miruky-fibqomwvgyqpoceu.fifo, disabled content-based deduplication, and a receive wait of 20 seconds. Each send must therefore provide a deduplication ID, while each final empty receive uses long polling rather than the default short-poll sample.

Open the new queue and verify that its type is FIFO, content-based deduplication is off, and the receive wait time is 20 seconds. The .fifo suffix is required by SQS and is recorded separately from the generated random name core in the resource ledger.

The queue details retain FIFO, disabled content-based deduplication, a receive wait of 20 seconds, and the SQS-managed encryption setting. The send comparison can now depend on explicit IDs rather than body hashes.

2. Send an original message and its duplicate

Open Send and receive messages. Enter the body accepted-original, group ID group-001, and deduplication ID dedup-001, then send it.

The original send form contains accepted-original, group-001, and dedup-001. These values establish the message and its deduplication identity.

SQS should report the send as successful. Keep the message in the queue and send another message before the five-minute deduplication interval expires.

For the next send, use the different body accepted-duplicate but keep both group-001 and dedup-001. Because content-based deduplication is disabled and the bodies differ, the repeated explicit ID is the matching input in this comparison.

The duplicate-ID form changes the body to accepted-duplicate while retaining group-001 and dedup-001. The repeated explicit ID is the only deduplication match between the two sends.

Choose Send message and observe another success confirmation. The successful acknowledgement means SQS accepted the API request; it does not mean that another deliverable message was added.

The Console reports that the duplicate-ID request was sent and is ready to be received. Its service-generated message ID is excluded because the acknowledgement alone is the evidence needed here.

3. Add a control message with a new ID

Send one more message with body accepted-control, group ID group-001, and deduplication ID dedup-002. This ID has not appeared in the five-minute window, so the control message should be available after the original.

The control form contains accepted-control, group-001, and dedup-002. Its fresh ID provides the message that should remain deliverable after the original.

All three send operations should have returned success. The expected deliverable set still contains only two bodies because the two dedup-001 requests represent one deduplication identity.

4. Receive the deliverable messages

Use 10 as the batch ceiling and start polling. In this validation, both deliverable messages arrived as two rows during one polling session; the session lasted 30 seconds. If only the original appears, delete it and poll again so the next message in the same group becomes available.

The cropped receive view shows accepted-original, group-001, and dedup-001. No service-generated identifier or receipt value is retained.

Inspect only the body, message group ID, and deduplication ID. Do not publish the service-generated message ID, sequence number, receipt handle, timestamps, queue URL, or queue ARN.

Open the other row and confirm that its body is accepted-control. The expected detail values are group-001 and dedup-002.

The next cropped receive view shows accepted-control, group-001, and dedup-002. This is the unique-ID control message expected after the original.

Delete both rows while their current receipt handles are valid. A receipt handle belongs to one receive action, and a later receive returns a new one. If the 30-second visibility timeout expires while you inspect the details, poll again and delete the two rows immediately.

After deletion, run two complete polling sessions. The displayed polling duration is 30 seconds and the maximum message count is 10, while the queue's receive-message wait remains 20 seconds. Two completed sessions are required, both with zero available messages and zero rows. The suppressed body must remain absent.

The final completed polling session shows no available message and no message row. The preceding completed session produced the same zero-message result, and accepted-duplicate never appeared. The queue's 20-second receive wait enables long polling, which queries all SQS servers and reduces the false-empty behavior of short polling.

The received bodies are the proof set: accepted-original and accepted-control. The repeated dedup-001 request was acknowledged during sending but suppressed from delivery.

Wrap-up

The run produced two deliverable bodies from three acknowledged sends. The repeated identifier suppressed the later body, while the fresh identifier admitted the control.

SQS keeps tracking a deduplication ID even after its message is received and deleted. Outside the five-minute window, or when producer acknowledgement handling is uncertain for longer than that window, consumers should still use an application-level idempotency strategy.

Thanks for reading this far.

See you in the next one.

Disclosure: This article was written with AI assistance and independently verified against the linked primary sources and observed results.

DEV Community