Unraveling the Complexities of Amazon SQS Quotas: What You Need to Know

George Yoshida — Wed, 04 Sep 2024 20:39:58 +0000

When it comes to Amazon SQS (Simple Queue Service), understanding the quota limits can be a bit tricky, especially when using FIFO (First-In-First-Out). In this post, we'll dive deep into the differences between these queues, particularly focusing on the often-overlooked aspect of throughput limits.

SQS Message Throughputs

Standard queue
- nearly unlimited calls/sec/API
FIFO high-throughput
- 300 calls/sec/API/message group ID
- 700,000 API/sec/queue # depends on the region
FIFO non-high-throughput
- 300 calls/sec/API/queue

SQS Queue Types: Standard vs. FIFO

Amazon SQS offers two distinct types of queues:

Standard Queue

This queue type offers nearly unlimited throughput and guarantees at-least-once delivery. This means that each message is guaranteed to be delivered at least once and in some cases, messages may be delivered out of order. Due to this, it's essential to implement idempotent workers in your application.

FIFO Queue

Designed to guarantee that messages are delivered exactly once and in the exact order they are sent. While this is critical for many applications, it comes with certain trade-offs, particularly in terms of throughput.

High-Throughput FIFO(Recommended): In this mode, SQS assigns a dedicated partition to each message group ID within the queue. This partitioning allows for parallel processing of messages. Each partition can handle up to 300 messages per second per API action per message group ID. By increasing the number of message groups, you can scale the queue's throughput to meet higher demands. For more details, please refer to the official documentation, specifically the "Partitions and data distribution for high throughput for SQS FIFO queues" section in "Enabling high throughput for FIFO queues in Amazon SQS."

Queue's throughputs vary depending on regions, like 900,000 in N. Virginia and 91,000 in Tokyo.

Non-High-Throughput FIFO: This mode is more restrictive, with a maximum throughput of 300 messages per second per API action per queue. This limit can be a bottleneck in applications requiring high-volume message processing.

Non-High-Throughput FIFO does not support per message group id throughput

It might seem like you can choose between throughput measured per queue or per message group ID when using FIFO queues, but in reality, the non-high-throughput mode is limited to the former, while the high-throughput mode exclusively uses the latter.

In fact, if you try to create a FIFO queue in non-high-throughput mode with per-message group ID throughput, you will encounter errors similar to the following:

InvalidAttributeValue: Invalid value for the parameter FifoThroughputLimit. Reason: To set FifoThroughputLimit to perMessageGroupId, the DeduplicationScope must be messageGroup.