SQS and Kinesis are both AWS services for handling data, but they serve fundamentally different purposes, and the most surprising thing is how often people try to use SQS for streaming or Kinesis for simple message queuing, and how painful that usually is.
Let’s see SQS in action. Imagine a web application that needs to send emails. Instead of the web server directly sending emails (which is slow and can block requests), it just drops a message onto an SQS queue.
{
"MessageId": "12345678-abcd-1234-abcd-1234567890ab",
"ReceiptHandle": "AQEBwJ...example...",
"MD5OfBody": "a1b2c3d4e5f67890abcdef1234567890",
"Body": "{\"email_address\": \"user@example.com\", \"subject\": \"Welcome!\", \"body\": \"Thanks for signing up!\"}"
}
A separate worker process, running independently, polls the SQS queue for messages. When it finds one, it processes it – in this case, sending the email.
aws sqs receive-message --queue-url https://sqs.us-east-1.amazonaws.com/123456789012/my-email-queue --max-number-of-messages 1
Once the email is sent, the worker deletes the message from the queue. If it fails, the message becomes visible again after a visibility timeout and can be retried. This is classic message queuing: reliable, decoupled task execution.
Now, let’s look at Kinesis. Imagine you have a high-throughput application, like a gaming platform, generating millions of events per second. You need to ingest all these events in order and process them in near real-time. Kinesis Data Streams is built for this.
A producer (your game server) sends records to a Kinesis stream.
{
"Data": "eyAidXNlcl9pZCI6IDEyMywgImV2ZW50IjogImxvZ2luIiwgInRpbWVzdGFtcCI6IDE2NzgwNjA4MDB9",
"PartitionKey": "user_123",
"SequenceNumber": "495451152433239538525189012345678901234567890123456789012"
}
The PartitionKey is crucial. It determines which shard a record goes to, ensuring that all events for a given user_id land in the same shard and are processed in order. Consumers (like a real-time analytics service) read from these shards.
aws kinesis get-records --shard-iterator <some-iterator-from-get-shard-iterator-call>
Kinesis Data Streams is designed for ordered, high-volume, real-time data ingestion and processing. It’s about streaming: a continuous flow of data where order within a partition matters.
The core problem SQS solves is decoupling producers and consumers for asynchronous task processing. A producer sends a command or data, and a consumer executes it later. The exact order of unrelated messages doesn’t matter, only that each message eventually gets processed. SQS guarantees at-least-once delivery and provides features like dead-letter queues for failed messages. It’s pull-based; consumers poll for work.
Kinesis Data Streams solves the problem of ingesting and processing high-volume, ordered streams of data in near real-time. The order of records within a shard is guaranteed, which is vital for many analytics and operational monitoring use cases. It’s push-based for producers and pull-based for consumers, but the stream itself is an immutable log. Consumers read sequentially from shards.
The key difference lies in ordering and throughput. SQS is designed for individual messages, where order between unrelated messages is generally not a concern. It scales by adding more queues or workers. Kinesis is designed for ordered streams within partitions, scaling by adding more shards to a stream. Trying to build a streaming system with SQS involves complex workarounds to maintain order, and trying to use Kinesis as a simple task queue is often overkill and more expensive.
Here’s a counterintuitive point: while Kinesis guarantees order within a shard, it does not guarantee global ordering across shards. If you need absolute, system-wide ordering of all events, you’d need to design your partitioning strategy very carefully, perhaps using a single shard if your throughput allows, or implementing custom logic in your consumers to reconcile events from different shards. This is a common pitfall when expecting Kinesis to provide a globally ordered log.
The next step in understanding these services is often exploring how Kinesis integrates with other AWS services like Lambda for event-driven processing, or Kinesis Data Analytics for real-time SQL queries.