SQS long polling isn’t just a performance optimization; it fundamentally changes how your application interacts with queues, turning a potentially expensive chatty protocol into an efficient, on-demand data stream.
Let’s see it in action. Imagine you have a queue my-processing-queue and you want to pull messages.
Without long polling (short polling):
aws sqs receive-message --queue-url https://sqs.us-east-1.amazonaws.com/123456789012/my-processing-queue --max-number-of-messages 1
You’ll get a response almost instantly, even if there are no messages. If there are messages, you get them. This is fine for high-throughput, but if messages are infrequent, you’re just burning CPU cycles and API calls polling an empty queue.
With long polling enabled, you add the --wait-time-seconds parameter. Let’s set it to 20 seconds:
aws sqs receive-message --queue-url https://sqs.us-east-1.amazonaws.com/123456789012/my-processing-queue --max-number-of-messages 1 --wait-time-seconds 20
Now, if the queue is empty, SQS won’t respond immediately. It will hold the request open for up to 20 seconds, waiting for a message to arrive. Only after 20 seconds (or if a message arrives sooner) will it send a response. If no message arrives within the wait time, the response will indicate that no messages were received.
This mechanism dramatically reduces the number of empty ReceiveMessage calls. Instead of a consumer constantly asking "Anything there? Anything there? Anything there?" every few seconds, it asks once and then waits patiently for up to the specified time.
The primary problem long polling solves is the cost and inefficiency of short polling, especially for applications that don’t have a constant, high volume of messages. Short polling can lead to:
- Increased API Costs: Each
ReceiveMessagecall, even an empty one, counts towards your SQS API request charges. For a busy application with infrequent messages, this can add up. - Higher Latency: If a message arrives just after a short poll completes, the consumer has to wait for the next poll cycle to pick it up, introducing unnecessary delay.
- Wasted Compute Resources: Consumers constantly polling an empty queue consume CPU and network resources for no productive work.
Long polling addresses these by:
- Reducing API Calls: By waiting for messages, consumers make significantly fewer
ReceiveMessagecalls when the queue is idle. - Lowering Latency: When a message arrives, the consumer is already waiting, ready to process it as soon as it’s available.
- Conserving Resources: Consumers spend less time actively polling and more time waiting efficiently.
The WaitTimeSeconds parameter can be set between 0 and 20 seconds. A value of 0 is equivalent to short polling. Setting it to a value greater than 0 enables long polling. The optimal value depends on your application’s message arrival patterns and your tolerance for latency. A common starting point is 10-20 seconds.
You configure this at the queue level, not per-request, though you can override it per-request. To set it for the queue my-processing-queue via the AWS CLI:
aws sqs set-queue-attributes --queue-url https://sqs.us-east-1.amazonaws.com/123456789012/my-processing-queue --attributes '{"ReceiveMessageWaitTimeSeconds": "15"}'
This change is immediate and affects all subsequent ReceiveMessage calls that don’t explicitly override the wait time.
When you’re using long polling, the ReceiveMessage API call will either return messages (if they arrived within the WaitTimeSeconds) or it will return an empty response body with a 200 OK status code, indicating that no messages were available within the polling interval. You must check for the presence of the Messages array in the JSON response to determine if any messages were actually received.
The internal mechanism involves SQS holding open the connection for your ReceiveMessage request. When a message is sent to the queue, SQS checks if there are any waiting ReceiveMessage requests. If so, it immediately delivers the message to one of the waiting consumers. If not, the message is simply queued as usual. If the WaitTimeSeconds elapses without a message arriving, SQS closes the connection and returns an empty response.
The most surprising part about long polling is that it doesn’t actually increase the cost of messages processed, but it significantly reduces the cost of polling for those messages. You pay for ReceiveMessage calls, and with short polling, you make many calls to get one message. With long polling, you make one call that might wait, but you’re still paying for that one call. The savings come from the drastically reduced number of API calls when the queue is empty.
Once you’ve optimized your consumers to use long polling, the next logical step is to consider how to scale those consumers dynamically based on queue depth.