SQS FIFO queues can handle significantly higher throughput than standard queues, but only if you explicitly enable and configure them correctly.
Let’s see SQS FIFO in action. Imagine you have a microservice that needs to process orders strictly in the order they are received. Using a standard SQS queue could lead to out-of-order processing if messages are duplicated or retried. A FIFO queue guarantees order.
Here’s how you’d create a FIFO queue using the AWS CLI:
aws sqs create-queue \
--queue-name my-order-processing.fifo \
--attributes '{"FifoQueue": "true", "ContentBasedDeduplication": "true"}'
The key here is FifoQueue: "true". This flag tells SQS to treat this queue as a FIFO queue, enforcing message order. ContentBasedDeduplication: "true" is also common for FIFO queues; it automatically deduplicates messages based on their content if you don’t provide explicit deduplication IDs.
Now, let’s consider a producer sending messages to this queue. The MessageGroupId is crucial. All messages with the same MessageGroupId will be processed in the order they were sent.
import boto3
sqs = boto3.client('sqs')
response = sqs.send_message(
QueueUrl='YOUR_QUEUE_URL',
MessageBody='{"order_id": "12345", "item": "widget"}',
MessageGroupId='ORDER_GROUP_123'
)
print(response['MessageId'])
If you send another message with the same MessageGroupId but different content, it will be processed after the first. If you send a message with a different MessageGroupId, it can be processed concurrently with messages from ORDER_GROUP_123. This is how FIFO queues achieve higher throughput: by allowing parallel processing of independent message groups.
The internal mechanism for this is that SQS partitions messages based on MessageGroupId. Within each partition, messages are strictly ordered. SQS can process messages from different partitions concurrently, enabling high throughput. The default throughput for a FIFO queue is 300 messages per second (MPS), with a burst of up to 3,000 MPS.
To increase this throughput, you need to explicitly request an increase from AWS Support. This is a manual process. Once approved, you’ll be notified, and your FIFO queue will automatically support the higher limits without any configuration changes on your end. The key is that the request is the configuration change, not a parameter you set in the console or CLI.
When sending messages, you can also specify a MessageDeduplicationId. If ContentBasedDeduplication is true and you also provide MessageDeduplicationId, the MessageDeduplicationId takes precedence. This is useful if your message body might change slightly but you want to treat it as the same logical message for deduplication purposes.
Consider a scenario where you have a large number of independent message groups. If you have 100,000 unique MessageGroupIds, your FIFO queue can achieve up to 10,000 MPS (3,000 MPS per group * ~3.3 groups). This is because SQS can process up to 10 message groups concurrently.
The most surprising thing about SQS FIFO throughput is that the scaling isn’t a knob you turn. You don’t just set a Throughput parameter to 10000. Instead, you must contact AWS Support and request a higher throughput quota for your account or for specific queues. Once granted, SQS automatically adjusts its internal partitioning and processing to meet the new quota. This means the system is designed for massive scale, but the activation of that scale requires an explicit, albeit manual, approval step.
The next challenge you’ll face is handling deduplication errors when using ContentBasedDeduplication and your message bodies are very similar but not identical, or when you have complex retry logic.