Valkey’s publish/subscribe mechanism can broadcast messages to an arbitrary number of subscribers, but it’s not inherently designed for efficient fan-out where a single publisher needs to reach thousands or millions of subscribers simultaneously.
Let’s see Valkey Pub/Sub in action. Imagine a simple chat application. A redis-cli instance acts as a publisher, sending messages to a channel named chat:general. Other redis-cli instances subscribe to this channel and receive the messages.
# Publisher
redis-cli PUBLISH chat:general "Hello everyone!"
# Subscriber 1
redis-cli SUBSCRIBE chat:general
# Output:
# Reading messages... (press Ctrl-C to quit)
# 1) "subscribe"
# 2) "chat:general"
# 3) (integer) 1
# 1) "message"
# 2) "chat:general"
# 3) "Hello everyone!"
# Subscriber 2
redis-cli SUBSCRIBE chat:general
# Output:
# Reading messages... (press Ctrl-C to quit)
# 1) "subscribe"
# 2) "chat:general"
# 3) (integer) 1
# 1) "message"
# 2) "chat:general"
# 3) "Hello everyone!"
This demonstrates the basic pub/sub flow: a message published to a channel is immediately delivered to all clients currently subscribed to that channel.
The core problem Valkey Pub/Sub solves is decoupling message producers from message consumers. A publisher doesn’t need to know who is listening, and subscribers don’t need to know who is sending. They only need to agree on a channel name. This is incredibly powerful for building event-driven architectures, real-time notifications, and distributed systems where components need to communicate asynchronously.
Internally, Valkey maintains a list of subscribers for each channel. When a PUBLISH command arrives, Valkey iterates through the subscriber list for that channel and sends the message to each connected client. This is a straightforward, in-memory operation. The key levers you control are the channel names for publishing and subscribing, and the number of clients connected to Valkey.
The most surprising thing about Valkey Pub/Sub is how it handles message delivery after a subscriber has disconnected and reconnected. If a client subscribes to a channel, then disconnects, and then reconnects later, it will not receive any messages that were published to that channel while it was offline. Valkey does not buffer messages for offline subscribers in its standard Pub/Sub implementation. This is a crucial point often missed when designing systems that require guaranteed delivery or message history.
When considering fan-out for a large number of subscribers, the simple PUBLISH operation on a single Valkey instance can become a bottleneck. Each message requires processing for every single subscriber. If you have 100,000 subscribers on a single channel, publishing a message means Valkey needs to send that message 100,000 times. This can saturate the network bandwidth of the Valkey server and the clients, and also consume significant CPU on the server.
For true fan-out and event broadcasting at scale, you typically need to augment Valkey’s Pub/Sub with other patterns or infrastructure. One common approach is using multiple Valkey instances or a cluster, and having subscribers connect to different instances. The publisher would then publish to the same channel on all instances. However, this introduces complexity in managing multiple Valkey connections and ensuring consistency.
A more robust solution for high-throughput fan-out involves using Valkey as a message broker but with a different strategy. Instead of direct PUBLISH, you might use a list or a stream. A publisher pushes events into a list or stream (e.g., LPUSH my-event-stream event_data). Subscribers then poll this list or stream. For fan-out, you’d need a separate consumer process that reads from the list/stream and then publishes to individual Valkey Pub/Sub channels for each subscriber group or uses a more sophisticated fan-out mechanism.
Another popular pattern for event broadcasting at scale is to use Valkey Pub/Sub in conjunction with a message queueing system like Kafka or RabbitMQ. Valkey can act as a lightweight notification service, informing downstream systems that a new event is available in the message queue. Consumers then pull the actual event data from the queue.
For scenarios where you absolutely need to broadcast to a massive number of subscribers efficiently without overwhelming a single Valkey instance, you’d architect a system where the publisher’s message is first ingested into a scalable message bus (like Kafka). Then, a dedicated fan-out service would consume from this bus and perform the actual distribution, potentially by publishing to multiple Valkey instances, or by managing WebSocket connections directly.
The next problem you’ll likely encounter is handling message persistence and guaranteed delivery, as standard Valkey Pub/Sub offers at-most-once delivery and no historical message storage.