The ZeroMQ IPC transport uses memory-mapped files to achieve near-zero-copy communication between processes on the same machine.
Let’s see it in action. Imagine two Python scripts: a publisher and a subscriber, both using ZeroMQ’s IPC transport.
Publisher (pub.py):
import zmq
import time
context = zmq.Context()
socket = context.socket(zmq.PUB)
socket.bind("ipc:///tmp/my_ipc_socket")
print("Publisher started, sending messages...")
for i in range(1000000):
message = f"Message {i}"
socket.send_string(message)
if i % 100000 == 0:
print(f"Sent {i} messages")
time.sleep(0.0001) # Small delay to simulate real-world work
print("Publisher finished.")
socket.close()
context.term()
Subscriber (sub.py):
import zmq
context = zmq.Context()
socket = context.socket(zmq.SUB)
socket.connect("ipc:///tmp/my_ipc_socket")
socket.setsockopt_string(zmq.SUBSCRIBE, "") # Subscribe to all messages
print("Subscriber started, waiting for messages...")
received_count = 0
while True:
message = socket.recv_string()
received_count += 1
if received_count % 100000 == 0:
print(f"Received {received_count} messages")
if received_count == 1000000: # Stop after receiving all messages
break
print("Subscriber finished.")
socket.close()
context.term()
To run this, save the code as pub.py and sub.py in the same directory. Open two terminal windows. In the first, run python sub.py. In the second, run python pub.py. You’ll see the subscriber rapidly receiving messages as the publisher sends them. The ipc:// prefix tells ZeroMQ to use the IPC transport, and /tmp/my_ipc_socket is the file path used for memory mapping.
The core problem ZeroMQ IPC solves is the overhead of traditional inter-process communication (IPC) mechanisms. When processes talk over sockets (like TCP/IP, even locally), data often needs to be copied multiple times: from the sending application’s buffer to the kernel’s network buffer, then from the kernel to the receiving application’s buffer. This copying, especially for large amounts of data or very frequent messages, becomes a significant bottleneck.
IPC bypasses much of this. When process A sends a message to process B via ipc://, ZeroMQ creates a shared memory region (backed by a file, often in /tmp or a similar temporary directory). Process A writes its message directly into this shared memory. Process B, which has also mapped this same memory region, can then read the message directly from there. This is often called "zero-copy" because the data doesn’t need to be copied between kernel and user space, or between different application buffers. The sending and receiving processes simply access the same piece of physical RAM.
The ipc:// URL scheme is key. It points to a file path on the filesystem. ZeroMQ uses this path to establish the shared memory segment. The bind operation on the publisher side creates this shared memory segment and makes it available. The connect operation on the subscriber side maps this existing segment into its own address space.
The most surprising thing about ZeroMQ IPC is that it’s not just fast, it’s also remarkably robust due to its underlying file-backed nature. While it’s primarily for on-machine communication, the use of a file path means that even if one process crashes, the shared memory segment persists until explicitly cleaned up or the system reboots (depending on the OS and file system). This persistence, coupled with the speed, makes it ideal for scenarios where you need high-throughput, low-latency communication between local services, such as microservices on a single server or components of a complex data processing pipeline.
The exact mechanics of memory mapping and shared memory are handled by the operating system, but ZeroMQ abstracts this complexity. It manages the lifecycle of the shared memory segment, ensuring it’s created, mapped, and eventually cleaned up. The messages themselves are just raw bytes in this shared buffer. ZeroMQ’s framing and serialization (like send_string and recv_string) add a thin layer of protocol on top of this raw memory access.
The performance gains from IPC come from avoiding context switches and data copies. When process A sends a message, it writes to shared memory. When process B receives, it reads from the same shared memory. The kernel is largely out of the loop for the data transfer itself, only being involved in the initial setup of the shared memory segment. This drastically reduces latency and increases throughput, making it a go-to for high-performance local IPC.
The next concept you’ll want to explore is how ZeroMQ handles message ordering and reliability over IPC, especially when using patterns beyond simple PUB/SUB.