Diagnose ZeroMQ Dropped Messages: HWM and Backpressure (2026)

ZeroMQ’s "dropped messages" problem isn’t about messages vanishing into the ether; it’s about a sender overwhelming a receiver’s capacity, forcing ZeroMQ to discard messages at the socket level to prevent the receiver’s process from grinding to a halt.

Let’s look at a common scenario: a fast publisher sending messages to a slower subscriber.

The Setup:

Imagine a simple publisher/subscriber pair. The publisher is blasting data out as fast as it can, and the subscriber is trying to keep up, but it’s doing some heavy lifting per message.

Publisher Side:

import zmq
import time

context = zmq.Context()
socket = context.socket(zmq.PUB)
socket.bind("tcp://*:5555")

message_count = 0
while True:
    message = f"Message {message_count}".encode('utf-8')
    socket.send(message)
    print(f"Sent: {message_count}")
    message_count += 1
    time.sleep(0.001) # Simulate some delay, but still fast

Subscriber Side:

import zmq
import time

context = zmq.Context()
socket = context.socket(zmq.SUB)
socket.connect("tcp://localhost:5555")
socket.setsockopt_string(zmq.SUBSCRIBE, "") # Subscribe to all

processed_count = 0
while True:
    message = socket.recv()
    print(f"Received: {message.decode('utf-8')}")
    # Simulate heavy processing
    time.sleep(0.1)
    processed_count += 1

If you run this, you’ll quickly see the publisher sending messages much faster than the subscriber can process them. The publisher will report sending 1000 messages, while the subscriber might only report receiving 100. The missing 900? Dropped.

Why This Happens: High Water Mark (HWM)

ZeroMQ sockets have a "High Water Mark" (HWM). This is a buffer size limit. When the buffer for a socket reaches its HWM, ZeroMQ starts dropping messages on the floor to prevent the application from running out of memory or becoming unresponsive.

Diagnosis: The primary indicator of HWM issues is a discrepancy between the number of messages sent by a publisher (or sender) and the number of messages received by a subscriber (or receiver). You’ll often see logs on the sending side showing a high volume of sends, while the receiving side shows a significantly lower volume of receives, or even receive errors if the buffer fills completely before a recv() call.
Fix: Increase the HWM on the receiver’s socket. This gives the receiver more buffer space to hold incoming messages while it processes them.

On the subscriber side, before connecting:
```
socket.setsockopt(zmq.HWM, 10000) # Set a higher HWM, e.g., 10,000 messages
```
This allows the subscriber socket to buffer up to 10,000 messages. The exact value depends on your message size and processing speed. Start high and tune down if memory usage becomes a concern.
Why it works: By increasing the HWM, you’re giving the subscriber more breathing room. The incoming messages fill this larger buffer instead of being immediately dropped. The subscriber can then process messages from this buffer at its own pace.

The Catch: Backpressure

Increasing HWM is a band-aid. If the subscriber consistently cannot keep up, even with a large HWM, you’re still dropping messages eventually, and your application might become a memory hog. This is where backpressure comes in.

ZeroMQ doesn’t inherently provide explicit "backpressure" signals in the way a TCP flow control mechanism does. Instead, it’s an emergent property of the system. When a socket’s HWM is reached, and messages are dropped, it’s a sign of backpressure. However, we can use patterns to simulate or react to this.

Common Causes and Fixes for Dropped Messages:

Receiver is too slow:
- Diagnosis: As described above, a significant mismatch between sent and received message counts. Check application logs on the receiver for processing times per message.
- Fix:
  - Increase HWM on receiver: socket.setsockopt(zmq.HWM, 10000) (as shown above).
  - Optimize receiver processing: Profile and speed up the code on the receiving end. This is the most robust solution.
  - Distribute load: If one receiver can’t keep up, add more subscribers and distribute the load (e.g., using a fan-out/fan-in pattern with a router/dealer).
- Why it works: More buffer, faster processing, or parallel processing allows the receiver to drain the incoming queue faster than it fills.
Network Congestion/Packet Loss:
- Diagnosis: Symptoms are similar to a slow receiver, but you might also see TCP retransmissions or high latency in network monitoring tools (like ping or mtr). If ZeroMQ’s internal buffers are full and the underlying network is dropping packets, you’ll see severe message loss.
- Fix:
  - Ensure reliable network: Investigate and fix network issues.
  - Use TCP with larger HWM: socket.setsockopt(zmq.HWM, 10000) on both sender and receiver. While TCP has its own flow control, ZeroMQ’s HWM acts as an application-level buffer.
  - Consider inproc for same-process communication: If sender and receiver are in the same process, zmq.Context.socket(zmq.PAIR) with inproc://... is orders of magnitude faster and bypasses network issues.
- Why it works: A more robust network or bypassing the network entirely reduces the chance of messages being lost before they even reach ZeroMQ’s buffers.
Sender’s HWM is too low (less common for dropping, but can cause sender-side issues):
- Diagnosis: The sender itself might block or report errors if its own outgoing buffer fills up. This is less about dropping and more about the sender being unable to send.
- Fix: Increase HWM on the sender’s socket.
```
socket.setsockopt(zmq.HWM, 10000) # Set a higher HWM on the sender
```
- Why it works: Gives the sender more room to queue messages before they are handed off to the network, preventing the sender’s send() operation from blocking or failing due to its own buffer being full.
Message Size Exceeds Available Memory/Buffer Limits:
- Diagnosis: If you’re sending extremely large messages, even a high HWM might not be enough if the total memory consumed by the buffer exceeds system limits. You might see MemoryError or system instability.
- Fix:
  - Reduce message size: Serialize data more efficiently.
  - Increase system memory: If feasible.
  - Use multipart messages: Send large data in chunks as separate ZeroMQ messages within a single logical transaction.
- Why it works: Breaking down large data reduces the memory footprint of individual buffered items.
ZeroMQ Version/Configuration Issues:
- Diagnosis: In rare cases, bugs in specific ZeroMQ versions or incorrect socket options can lead to unexpected behavior.
- Fix:
  - Update ZeroMQ: Ensure you’re using a recent, stable version.
  - Review socket options: Double-check all setsockopt calls for correctness.
- Why it works: Correcting underlying library issues or configuration errors resolves the problem.

The Next Problem You’ll Hit:

After you’ve solved the dropped message issue by increasing HWM, your next challenge will likely be managing the latency introduced by the larger buffer. Messages that are buffered are not yet processed, meaning your application’s state will lag behind real-time.

More Deep Dives in Zeromq