ZeroMQ sockets aren’t just simple conduits; they’re sophisticated network endpoints with internal buffering, threading, and state machines that can dramatically impact performance if not tuned.
Let’s see ZeroMQ in action with a simple PUB/SUB scenario.
Publisher (pub.c)
#include <zmq.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
int main(void) {
void *context = zmq_ctx_new();
void *publisher = zmq_socket(context, ZMQ_PUB);
int linger = 0; // Don't wait for unsent messages on close
zmq_setsockopt(publisher, ZMQ_LINGER, &linger, sizeof(linger));
int hwm = 1000; // High Water Mark for outgoing messages
zmq_setsockopt(publisher, ZMQ_SNDHWM, &hwm, sizeof(hwm));
int sndbuf = 1024 * 1024; // Send buffer size in bytes
zmq_setsockopt(publisher, ZMQ_SNDBUF, &sndbuf, sizeof(sndbuf));
if (zmq_bind(publisher, "tcp://*:5556") == -1) {
perror("zmq_bind failed");
return 1;
}
char message[256];
int i = 0;
while (1) {
snprintf(message, sizeof(message), "Message %d", i++);
if (zmq_send(publisher, message, strlen(message), 0) == -1) {
perror("zmq_send failed");
// In a real app, you'd handle this more gracefully
}
usleep(1000); // Send roughly 1000 messages per second
}
zmq_close(publisher);
zmq_ctx_destroy(context);
return 0;
}
Subscriber (sub.c)
#include <zmq.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
int main(void) {
void *context = zmq_ctx_new();
void *subscriber = zmq_socket(context, ZMQ_SUB);
int linger = 0; // Don't wait for unsent messages on close
zmq_setsockopt(subscriber, ZMQ_LINGER, &linger, sizeof(linger));
int hwm = 1000; // High Water Mark for incoming messages
zmq_setsockopt(subscriber, ZMQ_RCVHWM, &hwm, sizeof(hwm));
int rcvbuf = 1024 * 1024; // Receive buffer size in bytes
zmq_setsockopt(subscriber, ZMQ_RCVBUF, &rcvbuf, sizeof(rcvbuf));
// Subscribe to all messages
const char *filter = "";
if (zmq_setsockopt(subscriber, ZMQ_SUBSCRIBE, filter, strlen(filter)) == -1) {
perror("zmq_setsockopt subscribe failed");
return 1;
}
if (zmq_connect(subscriber, "tcp://localhost:5556") == -1) {
perror("zmq_connect failed");
return 1;
}
char message[256];
while (1) {
long bytes_received = zmq_recv(subscriber, message, sizeof(message) - 1, 0);
if (bytes_received == -1) {
perror("zmq_recv failed");
// In a real app, you'd handle this more gracefully
continue;
}
message[bytes_received] = '\0'; // Null-terminate the received data
printf("Received: %s\n", message);
}
zmq_close(subscriber);
zmq_ctx_destroy(context);
return 0;
}
Compile and run these. You’ll see messages flowing. Now, let’s dive into what makes this (or breaks it).
ZeroMQ’s core performance levers are its socket options, which control internal behavior like buffering, message dropping, and connection handling. Understanding these allows you to tailor ZeroMQ to specific network conditions and application needs.
The High Water Mark (HWM), ZMQ_SNDHWM and ZMQ_RCVHWM, is arguably the most critical. It defines the maximum number of messages ZeroMQ will queue internally for sending or receiving before it starts dropping messages or blocking. If your sender is faster than your receiver, the sender’s outgoing queue will fill up. Once it hits the ZMQ_SNDHWM, zmq_send will start returning -1 with EAGAIN if non-blocking, or block if blocking, until space is available. Similarly, the receiver’s ZMQ_RCVHWM dictates how many messages can be buffered before they are dropped if the application isn’t consuming them fast enough. The default HWM is often quite low (e.g., 1000 messages), which can be a bottleneck for high-throughput applications.
The Socket Buffer Sizes, ZMQ_SNDBUF and ZMQ_RCVBUF, control the underlying TCP socket’s send and receive buffer sizes. These are OS-level settings. Setting them too low can limit the amount of data that can be in flight over the network, even if the HWM is high. Setting them too high can lead to increased memory usage and potentially longer latency if packets are lost, as more data needs to be retransmitted. The default values are often conservative.
The Linger Period, ZMQ_LINGER, determines how long ZeroMQ will wait to send any outstanding messages when a socket is closed or the context is terminated. A value of 0 means it will discard any unsent messages immediately. A positive value (in milliseconds) means it will attempt to send them. For high-performance, fire-and-forget scenarios, 0 is usually preferred to avoid blocking your shutdown. For applications that must ensure delivery of in-flight messages upon shutdown, a non-zero linger period is necessary, but it can complicate application shutdown logic.
Message Size and Serialization: While not a socket option, the size of your messages and the efficiency of your serialization format have a massive impact. Sending large, complex objects serialized in verbose formats (like XML) will saturate your network and CPU much faster than small, binary-encoded messages. ZeroMQ itself doesn’t dictate your serialization; it’s up to your application. For performance, consider efficient binary formats like Protocol Buffers, FlatBuffers, or MessagePack.
The ZMQ_RCVTIMEO and ZMQ_SNDTIMEO options set timeouts for receiving and sending operations, respectively. If a zmq_recv or zmq_send operation would block indefinitely, it will return -1 with EAGAIN after this timeout. This is crucial for applications that need to remain responsive or perform other tasks while waiting for messages, preventing deadlocks or long waits. The timeout is specified in milliseconds; a value of -1 (the default) means infinite.
When you set ZMQ_RCVHWM or ZMQ_SNDHWM to a very high value, like 1000000, ZeroMQ will attempt to buffer up to a million messages internally. If your application can’t keep up with consuming messages from the receive buffer, or if the sender can’t keep up with sending to the send buffer, these internal buffers will grow. This can consume significant memory. If the HWM is reached, zmq_send will block (or return EAGAIN), and incoming messages will be dropped by the OS or ZeroMQ’s internal queue if the underlying TCP buffer is full.
After you’ve tuned these socket options for optimal performance, the next challenge you’ll likely encounter is managing message ordering and deduplication in distributed systems.