Syslog over UDP is a surprisingly fragile protocol for shipping logs across a network, despite its ubiquity.
Let’s watch a log message travel. Imagine a web server, web-01, needs to send its access logs to a central syslog server, log-aggregator. We’ve configured web-01 to send its logs via UDP port 514 to log-aggregator’s IP address, 192.168.1.100.
On web-01, the rsyslogd process (or syslog-ng, depending on your setup) is configured to forward certain messages. A snippet from /etc/rsyslog.conf might look like this:
# Forward all web server logs to the aggregator
*.* @192.168.1.100:514
The @ symbol here signifies UDP. If it were @@, it would mean TCP.
On log-aggregator, rsyslogd is listening for incoming UDP messages on port 514. Its configuration in /etc/rsyslog.conf would include:
# Receive logs from anywhere via UDP
module(load="imudp")
input(type="imudp" port="514")
Now, when web-01 generates a log line, say a GET request, rsyslogd on web-01 packages this message into a UDP datagram. This datagram is then sent out on the network interface towards 192.168.1.100:514.
The UDP packet is a simple payload. There’s no handshake, no guarantee of delivery, no acknowledgment. It’s like sending a postcard. If the postcard gets lost in the mail, you never know.
The log-aggregator’s rsyslogd process, listening on port 514, receives this UDP datagram. It then processes the message according to its own configuration, often writing it to a file like /var/log/remote/web-01/access.log.
The primary problem this solves is centralizing logs from distributed systems without the overhead of a persistent connection. You can have dozens or hundreds of machines sending logs to one or a few aggregators without each machine needing to manage complex connection states or buffers for each destination. This is crucial for monitoring, security auditing, and debugging across a fleet.
The internal mechanism is straightforward:
- Sender: A syslog daemon (like
rsyslogd) formats a log message. - Packetization: The message is encapsulated in a UDP datagram.
- Transmission: The datagram is sent to the destination IP address and port.
- Receiver: Another syslog daemon on the destination machine listens on the specified UDP port.
- Reception: Incoming datagrams are read, their contents extracted, and processed (e.g., written to a file).
The exact levers you control are primarily the destination address and port in the sending configuration, and the listening port and input module in the receiving configuration. You also control which messages are sent (e.g., *.* for all, or specific facilities/severities).
Here’s a detail that trips many people up: UDP checksums. By default, on Linux systems, if the kernel determines it can’t compute a UDP checksum (e.g., for outgoing packets on certain network interfaces that don’t support hardware checksumming, or due to specific network configurations), it will zero out the checksum field. This zero checksum is valid according to the UDP specification, but it means the packet is technically not protected against accidental modification in transit. Some older or very strict network devices might drop packets with zero checksums, interpreting them as malformed, even though they are technically valid.
The next concept to explore is ensuring log delivery reliably, which often leads to Syslog over TCP.