Publishing a single message to an AWS SNS topic and having it trigger multiple SQS queues, each processed independently, is a fundamental pattern for decoupling services.
Let’s see it in action. Imagine an OrderProcessing service that needs to notify several downstream systems about a new order: inventory, billing, and email notifications. Instead of the OrderProcessing service directly calling each of these, it publishes a single message to an SNS topic.
{
"order_id": "ORD12345",
"customer_email": "customer@example.com",
"items": [
{"sku": "SKU001", "quantity": 2},
{"sku": "SKU005", "quantity": 1}
],
"total_amount": 150.75
}
This message is sent to an SNS topic named OrderEvents.
Now, let’s say we have three SQS queues: InventoryQueue, BillingQueue, and EmailNotificationQueue. Each of these queues is subscribed to the OrderEvents SNS topic.
When the OrderEvents topic receives the JSON message above, SNS automatically creates a copy of that message and sends it to each subscribed SQS queue.
Here’s what the InventoryQueue might look like after receiving the message:
{
"Type": "Notification",
"MessageId": "...",
"Timestamp": "2023-10-27T10:00:00.000Z",
"SNS": {
"Type": "Notification",
"MessageId": "...",
"Timestamp": "2023-10-27T10:00:00.000Z",
"Subject": "New Order",
"Message": "{\"order_id\": \"ORD12345\", \"customer_email\": \"customer@example.com\", \"items\": [{\"sku\": \"SKU001\", \"quantity\": 2}, {\"sku\": \"SKU005\", \"quantity\": 1}], \"total_amount\": 150.75}",
"TopicArn": "arn:aws:sns:us-east-1:123456789012:OrderEvents",
"MessageAttributes": {}
}
}
Notice how the original message is nested within the SNS field of the SQS message. The InventoryService would poll InventoryQueue, retrieve this message, parse the Message field, and then process the order for inventory.
Similarly, the BillingService would poll BillingQueue and the EmailNotificationService would poll EmailNotificationQueue. Each service works independently, consuming its copy of the message and performing its specific task without any knowledge of the other services. This is the essence of fan-out: one event, multiple consumers, all decoupled.
The problem this pattern solves is the N+1 problem of direct service calls. Without fan-out, if OrderProcessing needed to notify 10 other services, it would have to make 10 separate API calls. If any one of those calls failed, OrderProcessing would need complex retry logic for each. With SNS/SQS, OrderProcessing only needs to publish one message to SNS. SNS handles the delivery to all subscribers. If an SQS queue is temporarily unavailable, SNS will retry delivery to that queue. The consuming services remain completely unaware of each other, and the publisher is shielded from their individual availability issues.
The core components are:
- SNS Topic: The central hub that receives the original message.
- SQS Queues: The durable message stores for each consumer.
- SNS Subscriptions: The links that connect the SNS topic to each SQS queue.
When you create an SQS subscription for an SNS topic, AWS automatically sets up the necessary permissions for SNS to send messages to the SQS queue.
The MessageAttributes field in SNS is a powerful, yet often underutilized, feature for routing and filtering messages. You can attach key-value metadata to your SNS messages, and then configure SQS subscriptions to only receive messages that match specific attribute values. For example, you could add an attribute like "order_type": "standard" and have a specific SQS queue only subscribe to "standard" orders. This allows for more granular control over message delivery without requiring complex logic in the publisher or consumers.
Consider the MessageGroupId and MessageDeduplicationId when using FIFO (First-In-First-Out) SQS queues subscribed to an SNS topic. If your SNS topic is configured for FIFO, the MessageGroupId from the SNS message will be propagated to the SQS FIFO queue, ensuring ordered processing within that group. If the SNS message doesn’t have a MessageGroupId, the SQS subscription will use the MessageGroupId from the SQS message itself, which is derived from the SNS message ID. For deduplication, if the SNS message has a MessageDeduplicationId, it’s used. Otherwise, SQS will generate one based on the message content.
The immediate next step after mastering fan-out is implementing dead-letter queues (DLQs) for your SQS queues.