SQS FIFO Exactly-Once: Message Deduplication Explained (2026)

SQS FIFO queues offer a powerful feature called exactly-once processing, but it hinges entirely on understanding and correctly implementing message deduplication.

Here’s SQS FIFO in action, processing a message exactly once, even if the producer sends it multiple times.

Imagine you’re building a critical workflow, like processing financial transactions or dispatching urgent alerts. If a message gets processed twice, you could double-charge a customer or send out a redundant alert. SQS FIFO’s exactly-once processing aims to prevent this. It guarantees that a message sent to a FIFO queue will be delivered and processed by a consumer at most once within a 5-minute deduplication interval. This means if you send the same message twice within that window, only one of them will be processed.

The magic behind this is message deduplication. When you send a message to a FIFO queue, you can provide a MessageDeduplicationId. If you omit it, SQS automatically generates one based on the message body. This MessageDeduplicationId is the key. SQS uses it to track recently sent messages. If a message arrives with a MessageDeduplicationId that SQS has seen within the last 5 minutes, SQS discards the duplicate without sending it to a consumer.

Let’s see this with an example. Suppose we’re processing an order with order_id: 12345. We want to ensure this order is processed only once.

Our producer code might look something like this:

import boto3

sqs = boto3.client('sqs', region_name='us-east-1')
queue_url = 'https://sqs.us-east-1.amazonaws.com/123456789012/my-fifo-queue.fifo'

# First send (unique deduplication ID)
response1 = sqs.send_message(
    QueueUrl=queue_url,
    MessageGroupId='ORDER_PROCESSING',
    MessageDeduplicationId='order-12345-process-v1',
    MessageBody='{"order_id": "12345", "action": "process"}'
)
print(f"First send successful: {response1['MessageId']}")

# Simulate a retry or duplicate send within 5 minutes
response2 = sqs.send_message(
    QueueUrl=queue_url,
    MessageGroupId='ORDER_PROCESSING',
    MessageDeduplicationId='order-12345-process-v1', # Same deduplication ID
    MessageBody='{"order_id": "12345", "action": "process"}'
)
print(f"Second send result: {response2}") # Note: This might not return a MessageId if deduplicated

If we run this, the first send_message will succeed and return a MessageId. The second send_message, because it uses the identical MessageDeduplicationId within the 5-minute window, will be detected as a duplicate by SQS. The response2 might not contain a MessageId or indicate that the message was not sent, depending on the exact SQS behavior for deduplicated messages. When a consumer polls the queue, it will only receive the message once.

The two primary ways to manage deduplication are:

Content-based deduplication: You enable this on the SQS queue itself. SQS then automatically generates a MessageDeduplicationId by hashing the message body. This is convenient if your message bodies are naturally unique for each logical operation and you don’t want to manage deduplication IDs in your producer. You can also provide a MessageDeduplicationId explicitly when sending, which will override content-based deduplication for that specific message.
- How to enable:
```
aws sqs create-queue --queue-name my-fifo-queue.fifo --attributes '{"FifoQueue": "true", "ContentBasedDeduplication": "true"}'
```
  or for an existing queue:
```
aws sqs set-queue-attributes --queue-url <your-queue-url> --attributes '{"ContentBasedDeduplication": "true"}'
```
- Why it works: SQS calculates a SHA-256 hash of the message body. If a message with the same hash arrives within the 5-minute window, it’s considered a duplicate.
Application-based deduplication: You explicitly provide a MessageDeduplicationId when sending each message. This gives you more control. A common strategy is to use a unique business identifier, like an order ID, a transaction ID, or a UUID, combined with a version number or timestamp.
- How to implement: In your producer code, when calling send_message:
```
response = sqs.send_message(
    QueueUrl=queue_url,
    MessageGroupId='ORDER_PROCESSING',
    MessageDeduplicationId='unique-business-id-v1', # e.g., 'order-12345-v1'
    MessageBody='...'
)
```
- Why it works: SQS directly uses the provided string as the deduplication identifier. If the same identifier is seen again within 5 minutes, the message is discarded. This is ideal when the message body might change slightly (e.g., adding a timestamp to the body for logging) but you still want to treat it as the same logical operation.

The MessageGroupId is also crucial for FIFO queues. It determines which messages are processed in order. All messages with the same MessageGroupId are delivered to consumers in the exact order they are sent. Importantly, deduplication is per MessageGroupId. This means a MessageDeduplicationId only needs to be unique within the context of a specific MessageGroupId and the 5-minute window.

A common pitfall is relying on the default content-based deduplication when your message bodies might have minor, insignificant variations that you still want to process as distinct events. For instance, if your message body includes a timestamp field that is automatically updated on each send, content-based deduplication will treat each send as unique, defeating the purpose. In such cases, application-based deduplication using a stable business identifier is essential.

The 5-minute deduplication interval is hardcoded and cannot be changed. This means if you send a message, and then send the exact same message again with the exact same deduplication ID after 6 minutes, SQS will treat it as a new message and deliver it.

Once you have exactly-once processing working, the next hurdle is often managing the state of your processing, ensuring that even if a consumer crashes after receiving a message but before completing its work, the message is re-delivered to another consumer without being lost or re-processed incorrectly.

More Deep Dives in Sqs