SQS FIFO queues don’t just guarantee order; they guarantee exactly-once processing within a given context, and that context is defined by two seemingly simple, yet powerful, concepts: DeduplicationId and MessageGroupId.
Let’s see this in action. Imagine we have a system processing orders. Each order has a unique order_id. We want to ensure that even if we accidentally send the same order details twice to SQS, only one order gets processed by our backend. We also want to ensure all messages related to a single order, say, order-123, are processed in the order they were sent, before any messages for order-456 start processing.
Here’s how we’d send messages to an SQS FIFO queue:
aws sqs send-message \
--queue-url https://sqs.us-east-1.amazonaws.com/123456789012/my-fifo-queue \
--message-body '{"order_id": "order-123", "event": "created"}' \
--message-group-id "order-123" \
--message-deduplication-id "order-123-created"
aws sqs send-message \
--queue-url https://sqs.us-east-1.amazonaws.com/123456789012/my-fifo-queue \
--message-body '{"order_id": "order-123", "event": "payment_received"}' \
--message-group-id "order-123" \
--message-deduplication-id "order-123-payment_received"
aws sqs send-message \
--queue-url https://sqs.us-east-1.amazonaws.com/123456789012/my-fifo-queue \
--message-body '{"order_id": "order-456", "event": "created"}' \
--message-group-id "order-456" \
--message-deduplication-id "order-456-created"
# Now, let's send a duplicate message for order-123
aws sqs send-message \
--queue-url https://sqs.us-east-1.amazonaws.com/123456789012/my-fifo-queue \
--message-body '{"order_id": "order-123", "event": "created"}' \
--message-group-id "order-123" \
--message-deduplication-id "order-123-created"
When you receive messages from this queue, you’ll get the created message for order-123, then the payment_received message for order-123, and then the created message for order-456. The duplicate created message for order-123 will be silently dropped by SQS because its MessageDeduplicationId ("order-123-created") has already been seen within the last 5 minutes for the MessageGroupId "order-123".
The MessageGroupId is your primary tool for maintaining order. All messages sent with the same MessageGroupId are guaranteed to be processed in the order they were sent. SQS achieves this by ensuring that only one message from a specific MessageGroupId is delivered to a consumer at any given time. If your application has multiple consumers, they will all process messages for a given MessageGroupId sequentially, but different MessageGroupIds can be processed concurrently by different consumers. This is the key to high throughput while preserving order within logical units.
The MessageDeduplicationId is SQS’s mechanism for preventing duplicate messages. When you send a message, SQS checks if a message with the same MessageDeduplicationId has been sent within the deduplication interval (which is 5 minutes for FIFO queues). If it has, the new message is discarded. If you don’t provide a MessageDeduplicationId, SQS automatically generates one based on the message content, but this is generally not recommended for explicit control. You can also configure SQS to use content-based deduplication, where SQS automatically generates the deduplication ID from the message body.
The relationship between these two is crucial: MessageDeduplicationId is scoped by MessageGroupId. This means a MessageDeduplicationId of "event-1" is unique within MessageGroupId "order-123", but another message with the same "event-1" MessageDeduplicationId could exist for MessageGroupId "order-456". This is why for true end-to-end deduplication, your MessageDeduplicationId must be unique across all messages that you never want to be treated as duplicates, regardless of their group.
The "5-minute deduplication interval" is a hard limit. If you send a message, and then send another message with the same MessageDeduplicationId after 6 minutes, SQS will treat it as a new message, even if the content is identical. This is a fundamental constraint to understand for critical idempotency requirements. For longer-term deduplication, you’ll need to implement that logic in your application layer, perhaps by checking a database before processing a message.
The most surprising thing about MessageGroupId is how it doesn’t strictly mean "all messages for X must be processed before any message for Y." It means that within a MessageGroupId, ordering is guaranteed. However, if you have MessageGroupIds "A" and "B", and messages are sent in the sequence A1, B1, A2, B2, SQS can deliver A1, then B1, then A2, then B2. It just ensures that A2 will not be delivered before A1, and B2 will not be delivered before B1. This allows for concurrency across groups while maintaining strict order within each group.
When you delete a message from an SQS FIFO queue, it is immediately removed. There’s no "visibility timeout" for deletion in the same way as standard queues where the message becomes visible again if not deleted. Once deleted, it’s gone. If you use MessageDeduplicationId and don’t manually handle idempotency in your application, a message that was successfully processed and then deleted could be re-processed if it’s sent again after the 5-minute deduplication window has passed.
Understanding these IDs is key to leveraging SQS FIFO for reliable, ordered, and exactly-once processing. The next challenge is handling message visibility timeouts and ensuring your consumers are robust against partial failures during processing.