Standard queues don’t guarantee order, but FIFO queues do.
Let’s see what that actually looks like. Imagine you have two tasks:
- Task A: Add an item to a shopping cart.
- Task B: Process payment for that shopping cart.
If these tasks arrive at your system in the order A then B, but are processed by SQS in the order B then A, you have a problem. Task B might try to process a payment for an empty cart.
Here’s a quick Python snippet to illustrate sending messages.
import boto3
sqs = boto3.client('sqs', region_name='us-east-1')
# Standard Queue
standard_queue_url = 'YOUR_STANDARD_QUEUE_URL'
sqs.send_message(QueueUrl=standard_queue_url, MessageBody='{"action": "add_item", "cart_id": "123", "item": "widget"}')
sqs.send_message(QueueUrl=standard_queue_url, MessageBody='{"action": "process_payment", "cart_id": "123"}')
# FIFO Queue
fifo_queue_url = 'YOUR_FIFO_QUEUE_URL.fifo' # Note the .fifo suffix
sqs.send_message(
QueueUrl=fifo_queue_url,
MessageBody='{"action": "add_item", "cart_id": "123", "item": "widget"}',
MessageGroupId='cart_123'
)
sqs.send_message(
QueueUrl=fifo_queue_url,
MessageBody='{"action": "process_payment", "cart_id": "123"}',
MessageGroupId='cart_123'
)
Notice the MessageGroupId for the FIFO queue. This is crucial. All messages with the same MessageGroupId are delivered to the consumer in the exact order they were sent. For our shopping cart example, cart_123 ensures all actions for that specific cart are processed sequentially.
SQS offers two main queue types: Standard and FIFO.
Standard Queues: These are the default and offer maximum throughput. They are designed for high availability and scalability. The trade-off is that they don’t guarantee message order or exactly-once processing (you might get a message more than once, though duplicates are rare). SQS uses a distributed system to achieve high availability, and this distribution inherently makes strict ordering across all messages difficult. Messages are typically delivered in the order they are sent, but this is not a strict guarantee. You might see a message sent later processed before an earlier message.
FIFO Queues:
These guarantee that messages are processed exactly in the order they are sent, and that each message is delivered exactly once. This is achieved by using a MessageGroupId. All messages sent with the same MessageGroupId are processed in strict order. Messages with different MessageGroupIds can be processed in parallel. FIFO queues have a higher cost and lower throughput limits compared to standard queues (e.g., 300 transactions per second per API action, versus 3000 for standard queues).
The core problem FIFO queues solve is maintaining the sequence of operations for a specific entity. Think of financial transactions, state updates for a user profile, or, as in our example, steps in a multi-part process like a shopping cart checkout. If the order of operations is critical for correctness, FIFO is your go-to.
When you send a message to a FIFO queue, you must specify a MessageGroupId. This ID segments your messages. SQS ensures that within a single MessageGroupId, messages are processed in the order they were received. Messages in different MessageGroupIds can be processed concurrently. This allows for parallelism while still guaranteeing order for related messages.
Here’s how the internal mechanism works conceptually: SQS partitions FIFO queues by MessageGroupId. Each partition is processed by a single consumer instance at a time. This ensures that within that partition (all messages for a given MessageGroupId), there’s no concurrency, thus preserving order. For standard queues, messages are distributed across many partitions, allowing for massive parallelism but sacrificing strict ordering.
The MessageDeduplicationId is another parameter for FIFO queues. If you provide it, SQS uses it to detect and discard duplicate messages. If you don’t provide it, SQS uses the message body itself as the deduplication ID. This is useful for idempotency – ensuring that if a message is sent multiple times, it’s only processed once.
The fact that standard queues can deliver messages out of order is a feature, not a bug, for many use cases. If you’re just sending notifications or tasks that can be processed independently and in any order (like sending emails, or background image processing), the high throughput and availability of standard queues are far more valuable than strict ordering. The complexity and cost of guaranteeing order would be unnecessary overhead.
When you set up a FIFO queue, you’ll see a .fifo suffix appended to its name, like my-app-requests.fifo. This is a visual cue that you’re dealing with an ordered queue.
The primary mechanism that enables ordering in FIFO queues is the MessageGroupId. Without it, SQS wouldn’t know which messages are related and need to be processed sequentially. It’s the key that unlocks guaranteed order for a specific set of messages.
If you’re using FIFO queues and find messages for the same MessageGroupId are still being reordered, it’s almost always because you’re consuming from multiple instances of your application without properly coordinating which instance handles which MessageGroupId. SQS guarantees order per group, but if multiple consumers are polling the same queue and pick up messages from the same group, they’ll fight over it, and order can be lost. The solution is to ensure only one consumer instance is actively processing messages for a given MessageGroupId at any time, often achieved by using a distributed locking mechanism or by dedicating specific consumer instances to specific MessageGroupIds.