SQS consumers don’t actually "process" messages; they receive them and then signal completion.
Here’s how a typical SQS consumer loop using boto3 in Python looks and what’s happening under the hood:
import boto3
import json
import time
sqs = boto3.client('sqs', region_name='us-east-1')
queue_url = 'YOUR_QUEUE_URL' # Replace with your actual SQS queue URL
while True:
try:
response = sqs.receive_message(
QueueUrl=queue_url,
MaxNumberOfMessages=10,
WaitTimeSeconds=20, # Long polling
VisibilityTimeout=30 # How long the message is hidden from other consumers
)
messages = response.get('Messages', [])
if not messages:
print("No messages received, waiting...")
continue
for message in messages:
print(f"Received message: {message['MessageId']}")
message_body = json.loads(message['Body'])
# --- Simulate processing ---
print(f"Processing payload: {message_body.get('data')}")
time.sleep(2) # Simulate work
# --- End simulation ---
# Delete the message after successful processing
sqs.delete_message(
QueueUrl=queue_url,
ReceiptHandle=message['ReceiptHandle']
)
print(f"Deleted message: {message['MessageId']}")
except Exception as e:
print(f"An error occurred: {e}")
time.sleep(5) # Backoff before retrying
This script continuously polls an SQS queue for new messages. When messages are available, it retrieves them, simulates processing their content (which is often JSON), and then explicitly deletes them from the queue. The WaitTimeSeconds parameter enables long polling, which significantly reduces the number of empty responses and lowers costs. VisibilityTimeout is crucial: it’s the duration a message becomes invisible to other consumers after being received. If the consumer successfully processes and deletes the message within this timeout, it’s gone. If it fails or the consumer crashes, the message reappears in the queue after the timeout, allowing another consumer to pick it up.
The core problem this pattern solves is decoupling: a producer can send messages to SQS without needing to know about or directly interact with the consumers. Consumers can process messages at their own pace, and if a consumer fails, the message isn’t lost. The queue acts as a buffer and a reliable delivery mechanism.
The receive_message call is the heart of the consumer. MaxNumberOfMessages lets you fetch up to 10 messages at once, which is more efficient than fetching one by one. WaitTimeSeconds (up to 20) tells SQS to hold the connection open and wait for messages if the queue is empty, rather than immediately returning an empty response. This is long polling. VisibilityTimeout (in seconds) is set for each message received. During this period, the message is hidden from other ReceiveMessage calls. If the consumer successfully processes and calls DeleteMessage before the timeout expires, the message is permanently removed. If the timeout expires before DeleteMessage is called, SQS makes the message visible again in the queue for another consumer to pick up.
When you delete a message, you use its ReceiptHandle, a unique identifier for that specific receive operation, not the MessageId. This prevents accidental deletion of a message that might have been re-delivered due to a timeout. If your processing logic fails, you simply don’t call delete_message. The message will then become visible again after its VisibilityTimeout expires.
The most surprising aspect of SQS message processing is that there’s no "commit" or "ack" in the traditional sense for successful processing. The "acknowledgment" is the delete_message call. If you forget to delete a message, SQS will eventually redeliver it. This is how SQS provides at-least-once delivery by default. If you need exactly-once processing, you must implement deduplication logic within your consumer, often by checking a unique ID within the message payload against a persistent store.
The next problem you’ll likely encounter is handling poison pills: messages that consistently fail processing, causing their VisibilityTimeout to repeatedly expire and the message to be redelivered.