SQS Exponential Backoff Retry: Handle Processing Failures (2026)

SQS exponential backoff retries are the unsung hero of robust message processing, preventing your system from drowning in retries when a downstream dependency hiccups.

Let’s see it in action. Imagine a worker that processes messages from an SQS queue. This worker calls an external API, and sometimes that API is slow or returns an error.

import boto3
import time
import json

sqs = boto3.client('sqs', region_name='us-east-1')
queue_url = 'https://sqs.us-east-1.amazonaws.com/123456789012/my-processing-queue'

def process_message(message_body):
    # Simulate an external API call that might fail
    try:
        # In a real scenario, this would be an API call, database operation, etc.
        if "fail_me" in message_body:
            raise Exception("Simulated external API failure")
        print(f"Successfully processed: {message_body}")
        return True
    except Exception as e:
        print(f"Error processing message: {e}")
        return False

while True:
    response = sqs.receive_message(
        QueueUrl=queue_url,
        MaxNumberOfMessages=1,
        WaitTimeSeconds=20 # Long polling
    )

    if 'Messages' in response:
        message = response['Messages'][0]
        message_body = message['Body']
        receipt_handle = message['ReceiptHandle']

        if process_message(message_body):
            sqs.delete_message(
                QueueUrl=queue_url,
                ReceiptHandle=receipt_handle
            )
            print(f"Deleted message: {message_body}")
        else:
            # This is where exponential backoff comes in,
            # but SQS handles it intrinsically with Visibility Timeout.
            # If we don't delete, the message becomes visible again.
            # For explicit backoff logic *within* the worker,
            # we'd implement a delay here before the next receive_message,
            # but the primary mechanism is SQS's Visibility Timeout.
            print(f"Message failed processing, will be retried. Receipt Handle: {receipt_handle}")
            # SQS will automatically re-queue this message after Visibility Timeout expires.
            # To *influence* the retry, we can extend or reduce the Visibility Timeout.
            # For true exponential backoff *outside* SQS's default,
            # you'd typically have a separate dead-letter queue strategy
            # or implement custom retry logic in your application.
    else:
        print("No messages received, waiting...")

The core problem SQS exponential backoff retries solve is handling transient failures in your message processing logic. If your worker tries to process a message and a downstream service (like a database, another API, or a file system) is temporarily unavailable, you don’t want to immediately discard the message. You also don’t want to hammer the failing service with constant retries, potentially overwhelming it further.

Here’s how SQS manages this, primarily through the Visibility Timeout and Dead-Letter Queues (DLQs).

When your worker receives a message, SQS makes that message "invisible" to other consumers for the duration of the Visibility Timeout. If your worker successfully processes the message and deletes it before the timeout expires, it’s gone.

However, if your worker fails to process the message (e.g., an exception occurs) and doesn’t delete it, SQS makes the message visible again after the Visibility Timeout. This is the first "retry."

The "exponential backoff" isn’t a direct setting on the queue itself that automatically increases the Visibility Timeout for every failed message. Instead, it’s a strategy you implement or leverage.

Default Retry Behavior (Visibility Timeout):
- What it is: The Visibility Timeout is set per-queue or per-message. When a message is received, it’s hidden for this duration. If not deleted, it reappears.
- How it works: If your worker crashes or returns an error without deleting, SQS automatically "retries" by making the message visible again after the Visibility Timeout. The default is 30 seconds.
- Why it’s important: This is the fundamental mechanism. Without it, failed messages would be lost or immediately reprocessed by the same failing worker.
- Configuration: Set in the SQS console or via aws sqs set-queue-attributes --queue-url <url> --attributes VisibilityTimeout=60 (e.g., 60 seconds).
Implementing Exponential Backoff (Application-Level or DLQ Strategy):
- What it is: A strategy where the delay between retries increases with each failure. This prevents overwhelming a struggling service. SQS doesn’t automatically do exponential backoff by increasing Visibility Timeout on subsequent automatic retries. It just re-exposes the message after the same Visibility Timeout. True exponential backoff usually involves explicit application logic or a DLQ setup.
- How it works (DLQ Strategy):
  - Configure a Dead-Letter Queue (DLQ) on your primary queue.
  - Set a maxReceiveCount on the primary queue’s redrive policy. This is the number of times a message can be received and not deleted before SQS moves it to the DLQ.
  - Your worker processes messages. If it fails, it doesn’t delete. SQS makes it visible again after the Visibility Timeout.
  - If the worker fails maxReceiveCount times, SQS automatically moves the message to the DLQ.
  - You then have a separate process that monitors the DLQ. This process can implement custom retry logic:
    - It might wait a fixed amount of time, then try to move the message back to the primary queue.
    - It might implement an exponential backoff strategy: wait 1 minute, then 5 minutes, then 15 minutes, etc., before attempting to return it.
- Why it’s important: This prevents infinite retries of messages that are fundamentally unprocessable or point to a persistent issue. It also allows for controlled re-ingestion after a problem is resolved.
- Configuration:
  - Primary Queue: aws sqs create-queue --queue-name my-processing-queue --attributes '{ "RedrivePolicy": "{\\"deadLetterTargetArn\\":\\"arn:aws:sqs:us-east-1:123456789012:my-processing-dlq\\",\\"maxReceiveCount\\":5}" }'
  - DLQ: A separate SQS queue (my-processing-dlq).
  - Application Logic: You’d write a separate script or service to poll the DLQ and implement time.sleep(calculated_backoff_time) before sqs.send_message to move messages back to the original queue.

Custom Application-Level Backoff:

What it is: Your worker code explicitly calculates a delay before attempting to receive_message again after a failure.
How it works:
- If process_message returns False:
  - Increment a retry counter for this message (stored in memory or a persistent store).
  - Calculate a backoff delay (e.g., delay = base_delay * (2 ** retry_count)).
  - Use time.sleep(delay) before calling sqs.receive_message again.
Why it’s important: Gives you the most granular control over retry timing, independent of SQS Visibility Timeout. However, it requires careful state management within your worker and doesn’t inherently solve the problem of messages piling up if the worker is always busy. This is often combined with a DLQ.

Configuration: Purely within your application code. For example:

retry_count = 0
max_retries = 5
base_delay = 5 # seconds

while True:
    response = sqs.receive_message(...)
    if 'Messages' in response:
        message = response['Messages'][0]
        receipt_handle = message['ReceiptHandle']
        if process_message(message['Body']):
            sqs.delete_message(...)
            retry_count = 0 # Reset on success
        else:
            retry_count += 1
            if retry_count > max_retries:
                print(f"Max retries reached for message {message['MessageId']}. Moving to DLQ manually (or log/discard).")
                # In a real app, you'd send to DLQ here or use SQS's redrive policy
                sqs.delete_message(QueueUrl=queue_url, ReceiptHandle=receipt_handle) # Or just don't delete if using DLQ redrive
            else:
                delay = base_delay * (2 ** (retry_count - 1)) # 5, 10, 20, 40, 80...
                print(f"Processing failed. Retrying in {delay} seconds. Retry count: {retry_count}")
                # Crucially, we DON'T delete the message. SQS will re-queue it.
                # To implement explicit backoff, we might wait here,
                # but SQS's Visibility Timeout is the primary driver for *when* it reappears.
                # A common pattern is to extend visibility timeout if processing is taking longer than expected,
                # but for *retries*, the DLQ or application-level polling interval is key.
                # If we just 'continue', the worker will try to receive a *new* message.
                # To retry the *same* message with a delay, the worker would typically
                # release the message (i.e., not delete it) and the SQS Visibility Timeout
                # dictates when it's available. For explicit app-level backoff on *failed* messages,
                # you'd typically implement a separate "retry manager" process.
                # The simplest is relying on Visibility Timeout and DLQ.
                # If process_message returns False, we simply don't delete.
                # SQS visibility timeout will handle the retry.
                # For custom backoff *before* SQS re-exposes it, you'd need to
                # change the visibility timeout:
                sqs.change_message_visibility(
                    QueueUrl=queue_url,
                    ReceiptHandle=receipt_handle,
                    VisibilityTimeout=int(delay) # Set visibility to our calculated delay
                )
                print(f"Changed visibility timeout to {int(delay)}s for message {message['MessageId']}")

    else:
        time.sleep(20) # Long polling wait

The key here is change_message_visibility. If your worker fails, it calls change_message_visibility to set how long the message should be hidden before its next retry. This is how you implement explicit backoff within SQS’s retry mechanism.

The most common and robust pattern is to use SQS’s built-in redrive policy to send messages to a DLQ after a maxReceiveCount, and then have a separate, simpler process that monitors the DLQ. This DLQ monitor can implement sophisticated exponential backoff logic for moving messages back to the main queue.

The next error you’ll hit is a Amazon.SQS.Model.MessageNotInflightException if you try to delete a message that has already been deleted or is no longer visible.

More Deep Dives in Sqs