Articles

SQS FIFO Deduplication ID: Prevent Duplicate Processing (2026)

April 6, 2026 6 min read

SQS FIFO queues ensure exactly-once processing by using a deduplication ID, but how that ID is generated and used can lead to missed messages or unexpected duplicates.

Let’s see SQS FIFO in action with a producer and consumer.

Producer:

import boto3

sqs = boto3.client('sqs', region_name='us-east-1')
queue_url = 'YOUR_FIFO_QUEUE_URL'

# Example message
message_body = '{"order_id": "12345", "item": "widget"}'
message_group_id = 'ORDER_PROCESSING'
deduplication_id = 'ORDER_12345_PROCESS' # This is key!

response = sqs.send_message(
    QueueUrl=queue_url,
    MessageBody=message_body,
    MessageGroupId=message_group_id,
    MessageDeduplicationId=deduplication_id
)

print(f"Sent message: {response['MessageId']}")

Consumer:

import boto3

sqs = boto3.client('sqs', region_name='us-east-1')
queue_url = 'YOUR_FIFO_QUEUE_URL'

while True:
    response = sqs.receive_message(
        QueueUrl=queue_url,
        MaxNumberOfMessages=1,
        WaitTimeSeconds=20, # Long polling
        MessageGroupId='ORDER_PROCESSING' # Optional, but good practice
    )

    if 'Messages' in response:
        for message in response['Messages']:
            print(f"Received message: {message['Body']}")
            # Process the message here...
            # Delete the message after successful processing
            sqs.delete_message(
                QueueUrl=queue_url,
                ReceiptHandle=message['ReceiptHandle']
            )
            print(f"Deleted message: {message['MessageId']}")
    else:
        print("No messages received.")

When you send a message to an SQS FIFO queue, you must provide a MessageGroupId and a MessageDeduplicationId. The MessageGroupId is used to maintain order within a group of related messages. All messages with the same MessageGroupId will be delivered in the order they were sent. The MessageDeduplicationId is the critical component for preventing duplicates. SQS uses this ID to track messages that have been sent within a 5-minute deduplication interval. If a message with the same MessageDeduplicationId is sent again within that interval, SQS will discard it.

The core problem people run into is how the MessageDeduplicationId is generated and managed. If your producer logic doesn’t reliably generate a unique ID for each distinct message, you’ll either miss messages (if you generate an ID that matches a previous distinct message) or get duplicates (if you generate a new ID for a message that was already processed, and the producer retried).

Here are the common pitfalls and their fixes:

  1. Producer Retries Without Idempotency: Your application might retry sending a message if it doesn’t receive a success confirmation from SQS. If the initial send did succeed but the confirmation was lost, a retry with the same MessageDeduplicationId will be dropped by SQS, leading to a lost message.

    • Diagnosis: Review your producer’s retry logic. Does it always generate a new MessageDeduplicationId on retry, or does it attempt to reuse the original?
    • Fix: Implement an idempotent send. Before retrying, check if the message has already been successfully sent. If not, generate a new, unique MessageDeduplicationId for the retry. A common pattern is to use a UUID for each attempt to send a distinct message.
    • Why it works: Each logical message gets a consistent ID, but each attempt to send that message can have a fresh, unique ID to ensure SQS doesn’t drop subsequent attempts if the first was successful.

  2. Inconsistent Deduplication ID Generation: If your MessageDeduplicationId generation logic isn’t deterministic or depends on transient state, you might end up with different IDs for what should be the same logical message.

    • Diagnosis: Examine the code generating the MessageDeduplicationId. Does it always produce the same ID for the same set of message content and business context?
    • Fix: Base the MessageDeduplicationId on the content of the message and any unique business identifiers. For example, if you’re sending an order processing message, use f"ORDER-{order_id}-{timestamp_or_sequence_number}" or a SHA-256 hash of the message body. Ensure the identifier used (like order_id) is unique and stable.
    • Why it works: SQS guarantees deduplication within 5 minutes. By using a stable, content-based ID, SQS can correctly identify and discard duplicate messages sent within that window.

  3. Using MessageDeduplicationId as a Simple Counter: Relying on a simple incrementing counter that isn’t persisted or managed across producer instances will lead to duplicate IDs and dropped messages, especially in distributed systems.

    • Diagnosis: Is your MessageDeduplicationId generated by a simple counter (i++) that resets or is unaware of other producer instances?
    • Fix: Use a UUID generated by uuid.uuid4() in Python for each send operation, or use a composite key that includes a unique entity ID and a timestamp/sequence number from a reliable source.
    • Why it works: UUIDs are statistically guaranteed to be unique, preventing accidental collisions. A composite key ensures uniqueness based on specific business entities.

  4. Forgetting to Set MessageDeduplicationId (and relying on SQS to generate it): If you omit MessageDeduplicationId and don’t enable content-based deduplication at the queue level, SQS will generate a UUID for you. This is not idempotent across producer instances or retries.

    • Diagnosis: Check your send_message calls. Is MessageDeduplicationId always provided?
    • Fix: Explicitly set MessageDeduplicationId in your send_message calls. Alternatively, you can enable content-based deduplication on the SQS queue itself. In this case, SQS will generate a deduplication ID based on the message body.
    • Why it works: Explicitly providing the ID gives you control. Content-based deduplication allows SQS to handle it, but requires the message body to be consistent.

  5. Content-Based Deduplication Mismatch: If you enable content-based deduplication on the queue, SQS will hash the message body to create the deduplication ID. Any change to the message body, however small, will result in a new ID.

    • Diagnosis: If using content-based deduplication, are there subtle, unintended changes to your message bodies across sends (e.g., whitespace, order of JSON keys, metadata)?
    • Fix: Ensure your message serialization is consistent. Canonicalize JSON (sort keys, consistent spacing) before sending. If you need to include dynamic data, either make it part of a separate field that doesn’t affect the core hash, or use explicit MessageDeduplicationId generation as described above.
    • Why it works: Consistent message bodies produce consistent hashes, allowing SQS to correctly identify and deduplicate.

  6. Deduplication Window Expiration: SQS deduplicates messages sent within a 5-minute interval. If a message is sent, processed, and then a duplicate is sent more than 5 minutes later, SQS will treat it as a new message, leading to potential duplicate processing if your consumer logic isn’t idempotent.

    • Diagnosis: Are your message processing times significantly longer than 5 minutes, or are there long delays between retries?
    • Fix: Implement idempotency in your consumer’s processing logic. This means that processing the same message multiple times should have the same effect as processing it once. Store processing results or state externally (e.g., in a database) and check this state before performing an action.
    • Why it works: Consumer idempotency ensures that even if SQS delivers a duplicate message (due to the 5-minute window or other reasons), your application won’t perform the action more than once.

The next error you’ll hit is Amazon.SQS.Model.AmazonSQSException with the Message "The request was rejected because the specified queue is not a FIFO queue." if you try to use FIFO-specific parameters on a standard queue.

More Deep Dives in Sqs

Want structured learning?

Take the full Sqs course →