The tcpdump tool, when used to capture ICMP packets, often reveals the underlying network issues that manifest as ping failures or cryptic error messages. The core problem is that ICMP, while crucial for network diagnostics, operates at a lower level than most application traffic, meaning its signals can be easily obscured or misinterpreted by higher-level network components.
Here’s how to trace ping and error messages using tcpdump and what to look for:
Common Causes of ICMP Capture Issues
-
Firewall Dropping ICMP:
- Diagnosis: On the source machine, run
sudo tcpdump -i any icmp -nn. On the destination machine, run the same command. If you see thepingrequest on the source but not on the destination, and no reply on the source, a firewall is likely blocking it. - Fix: On the firewall (or the source/destination host’s firewall if it’s local), allow ICMP echo-request and echo-reply. For
iptableson Linux, this would be:sudo iptables -A INPUT -p icmp --icmp-type echo-request -j ACCEPT sudo iptables -A OUTPUT -p icmp --icmp-type echo-reply -j ACCEPT - Why it works: This explicitly permits the ICMP packets that
pinguses to traverse the network, bypassing the firewall’s default deny rules for this protocol.
- Diagnosis: On the source machine, run
-
ICMP Rate Limiting:
- Diagnosis: Run
sudo tcpdump -i any icmp -nnon the destination machine. If you see some echo-requests but then they stop, or you see a mix of echo-requests and "Destination Unreachable (Communication administratively prohibited)" messages, the destination might be rate-limiting ICMP. - Fix: On the destination host, you can adjust the
net.ipv4.icmp_ratelimitsysctl parameter. For example, to increase the rate limit (allowing more ICMP packets per second):
To make this permanent, editsudo sysctl -w net.ipv4.icmp_ratelimit=1000/etc/sysctl.confand add or modify the line:net.ipv4.icmp_ratelimit = 1000Then runsudo sysctl -p. - Why it works: By increasing the allowed ICMP packet rate, you prevent the operating system from dropping legitimate diagnostic traffic due to perceived overuse.
- Diagnosis: Run
-
Asymmetric Routing:
- Diagnosis: Run
sudo tcpdump -i any icmp -nnon both the source and destination, and also on intermediate routers if possible. If you see thepingrequest leaving the source, arriving at the destination, but the reply doesn’t return to the source (or vice-versa), you might have asymmetric routing. Thetcpdumpoutput will show packets flowing in one direction but not the other on specific interfaces. - Fix: This is a network infrastructure issue. The ideal fix is to configure routers to ensure traffic takes the same path in both directions. If that’s not immediately feasible, you might need to adjust firewall rules on devices that are seeing only one direction of the traffic to allow it, or explore advanced routing configurations.
- Why it works: Ensures that the return path for ICMP replies is available and not blocked by a firewall or routing policy that only saw the initial request.
- Diagnosis: Run
-
MTU Mismatches (Especially with ICMP Fragmentation Needed):
- Diagnosis: If
pingwith large packet sizes fails, andtcpdumpshows "Destination Unreachable (Fragmentation needed and DF set)" messages, there’s an MTU issue. Runsudo tcpdump -i any icmp -nn. Look for ICMP Type 3, Code 4. - Fix: On the source machine, reduce the packet size used by
pingor configure path MTU discovery (PMTUD) to work correctly. Forping, you can specify the size:
To fix PMTUD, ensure that intermediate firewalls are not blocking ICMP Type 3, Code 4 (Fragmentation Needed) and Type 3, Code 5 (Source Route Failed) packets.ping -s 1472 google.com # For a common 1500 MTU - Why it works: Large packets might be fragmented by routers. If intermediate routers or firewalls block ICMP "Fragmentation Needed" messages, the source never learns to send smaller packets, causing the larger ones to be dropped.
- Diagnosis: If
-
ICMP Tunneling/Obfuscation:
- Diagnosis: In complex network environments, ICMP traffic might be tunneled or encapsulated. If
tcpdumpshows unexpected packet structures or protocols carrying ICMP, or if standard ICMP packets are not appearing where expected, this could be the cause. This is less common for basicpingbut relevant for VPNs or custom network solutions. - Fix: Understand the tunneling mechanism. If ICMP is being encapsulated within another protocol (e.g., GRE, IPsec, or even custom UDP/TCP wrappers), you’ll need to
tcpdumpthe outer protocol or configure the tunneling endpoint to allow/pass ICMP. For example, if ICMP is inside GRE:sudo tcpdump -i any 'proto 47' -nn # Capture GRE - Why it works: Allows you to see the ICMP traffic once it’s decapsulated or by examining the outer encapsulation.
- Diagnosis: In complex network environments, ICMP traffic might be tunneled or encapsulated. If
-
Incorrect Network Configuration (IP Addresses, Subnets):
- Diagnosis: Use
sudo tcpdump -i <interface> icmp -nnon the source. If you see echo-requests leaving with the correct source IP but never see echo-replies, double-check the destination’s IP address and subnet mask. Also, check the source machine’s routing table (ip route show) to ensure it has a valid route to the destination. - Fix: Correct the IP address, subnet mask, or gateway configuration on the source or destination host. Ensure the routing table on the source correctly points to the destination network.
- Why it works: Ensures that the IP packets are addressed correctly and that the operating system knows how to forward them to the intended network.
- Diagnosis: Use
After fixing the immediate ICMP capture issue, the next error you might encounter is related to TCP connection timeouts or resets if the ICMP problem was masking a more fundamental network reachability or firewall issue affecting application-level protocols.