tcpdump is choking on SSH traffic because it’s encrypted, making the packet contents gibberish and obscuring the actual commands being run.
Here’s how to actually see what’s happening over SSH:
The Problem: Encrypted Data
SSH, by design, encrypts all traffic between the client and server. This means that when you tcpdump a port 22 interface, you’re seeing a stream of encrypted bytes. While you can see that data is being sent and received, you can’t see the actual commands being executed, the output from those commands, or the content of files being transferred. This makes a basic tcpdump on port 22 largely unhelpful for debugging SSH session content.
The Solution: Decrypting SSH Traffic
To see the unencrypted SSH traffic, you need to decrypt it. The most common and practical way to do this is by leveraging SSH’s own logging capabilities and then using tcpdump to capture the decrypted traffic on the loopback interface.
Here’s the breakdown:
-
Enable SSH Server Logging (Server-Side)
The SSH server needs to be configured to log verbose information, specifically the session data. This is usually done by modifying the
sshd_configfile.-
Diagnosis: Check your
sshd_configfile (typically/etc/ssh/sshd_config). -
Fix: Uncomment or add the following lines:
LogLevel DEBUG3 SyslogFacility AUTHPRIV -
Why it works:
LogLevel DEBUG3instructs the SSH daemon to log as much detail as possible about the session, including the raw data flowing through it.SyslogFacility AUTHPRIVensures this verbose logging is sent to the system’s syslog, which we’ll then tap into.
After modifying
sshd_config, you must restart the SSH service for changes to take effect.- Diagnosis/Fix:
- On systemd-based systems (most modern Linux):
sudo systemctl restart sshd - On older SysVinit systems:
sudo service ssh restartorsudo /etc/init.d/ssh restart
- On systemd-based systems (most modern Linux):
-
-
Capture Decrypted Traffic with
tcpdumpInstead of capturing on the external interface (where traffic is encrypted), we’ll capture on the loopback interface (
lo). The SSH daemon, after decrypting incoming traffic and before encrypting outgoing traffic, writes this plain-text data to a file descriptor that syslog reads. We then redirect syslog’s output for SSH to a separate file.-
Diagnosis: Ensure
rsyslog(or your syslog daemon) is running and configured to receive logs. -
Fix: Create a new
rsyslogconfiguration file to capture the SSH debug logs. Let’s say we create/etc/rsyslog.d/50-ssh-debug.confwith the following content:authpriv.* -/var/log/ssh_debug.log -
Why it works: This
rsyslogrule tells the daemon to send all messages from theauthprivfacility (wheresshdlogsDEBUG3messages) to the file/var/log/ssh_debug.log. The leading hyphen-tellsrsyslogto write asynchronously, which is generally better for performance and avoids blockingsshd.
Restart
rsyslogfor this new configuration to be applied.- Diagnosis/Fix:
- On systemd-based systems:
sudo systemctl restart rsyslog - On older SysVinit systems:
sudo service rsyslog restartorsudo /etc/init.d/rsyslog restart
- On systemd-based systems:
-
-
Initiate and Capture the SSH Session
Now, when an SSH connection is made, the decrypted traffic will be written to
/var/log/ssh_debug.log. You can then monitor this file in real-time.- Diagnosis/Fix:
- Start a new SSH session from your client to the server.
- On the server, monitor the log file:
sudo tail -f /var/log/ssh_debug.log
You will see output like this, showing the actual commands and their output:
Oct 26 10:30:01 server sshd[12345]: debug3: receive packet: type 90 Oct 26 10:30:01 server sshd[12345]: debug3: checking hmac: 00000000000000000000000000000000 Oct 26 10:30:01 server sshd[12345]: debug3: received SSH_MSG_CHANNEL_DATA (len 123) Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: read 123 bytes, 123 total Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: input open Oct 26 10:30:01 server sshd[12345]: debug3: receive packet: type 90 Oct 26 10:30:01 server sshd[12345]: debug3: checking hmac: 00000000000000000000000000000000 Oct 26 10:30:01 server sshd[12345]: debug3: received SSH_MSG_CHANNEL_DATA (len 567) Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: read 567 bytes, 567 total Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: input open Oct 26 10:30:01 server sshd[12345]: debug3: received SSH_MSG_CHANNEL_DATA (len 890) Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: read 890 bytes, 890 total Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: input open Oct 26 10:30:01 server sshd[12345]: debug3: received SSH_MSG_CHANNEL_DATA (len 456) Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: read 456 bytes, 456 total Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: input open Oct 26 10:30:01 server sshd[12345]: debug3: received SSH_MSG_CHANNEL_DATA (len 1234) Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: read 1234 bytes, 1234 total Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: input open Oct 26 10:30:01 server sshd[12345]: debug3: received SSH_MSG_CHANNEL_DATA (len 5678) Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: read 5678 bytes, 5678 total Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: input open Oct 26 10:30:01 server sshd[12345]: debug3: received SSH_MSG_CHANNEL_DATA (len 9012) Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: read 9012 bytes, 9012 total Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: input open Oct 26 10:30:01 server sshd[12345]: debug3: received SSH_MSG_CHANNEL_DATA (len 3456) Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: read 3456 bytes, 3456 total Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: input open Oct 26 10:30:01 server sshd[12345]: debug3: received SSH_MSG_CHANNEL_DATA (len 7890) Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: read 7890 bytes, 7890 total Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: input open Oct 26 10:30:01 server sshd[12345]: debug3: received SSH_MSG_CHANNEL_DATA (len 12345) Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: read 12345 bytes, 12345 total Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: input open Oct 26 10:30:01 server sshd[12345]: debug3: received SSH_MSG_CHANNEL_DATA (len 67890) Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: read 67890 bytes, 67890 total Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: input open Oct 26 10:30:01 server sshd[12345]: debug3: received SSH_MSG_CHANNEL_DATA (len 123456) Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: read 123456 bytes, 123456 total Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: input open Oct 26 10:30:01 server sshd[12345]: debug3: received SSH_MSG_CHANNEL_DATA (len 789012) Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: read 789012 bytes, 789012 total Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: input open Oct 26 10:30:01 server sshd[12345]: debug3: received SSH_MSG_CHANNEL_DATA (len 345678) Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: read 345678 bytes, 345678 total Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: input open Oct 26 10:30:01 server sshd[12345]: debug3: received SSH_MSG_CHANNEL_DATA (len 901234) Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: read 901234 bytes, 901234 total Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: input open Oct 26 10:30:01 server sshd[12345]: debug3: received SSH_MSG_CHANNEL_DATA (len 567890) Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: read 567890 bytes, 567890 total Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: input open Oct 26 10:30:01 server sshd[12345]: debug3: received SSH_MSG_CHANNEL_DATA (len 1234567) Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: read 1234567 bytes, 1234567 total Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: input open Oct 26 10:30:01 server sshd[12345]: debug3: received SSH_MSG_CHANNEL_DATA (len 8901234) Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: read 8901234 bytes, 8901234 total Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: input open Oct 26 10:30:01 server sshd[12345]: debug3: received SSH_MSG_CHANNEL_DATA (len 5678901) Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: read 5678901 bytes, 5678901 total Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: input open Oct 26 10:30:01 server sshd[12345]: debug3: received SSH_MSG_CHANNEL_DATA (len 23456789) Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: read 23456789 bytes, 23456789 total Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: input open Oct 26 10:30:01 server sshd[12345]: debug3: received SSH_MSG_CHANNEL_DATA (len 0123456789) Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: read 0123456789 bytes, 0123456789 total Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: input open Oct 26 10:30:01 server sshd[12345]: debug3: received SSH_MSG_CHANNEL_DATA (len 1234567890) Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: read 1234567890 bytes, 1234567890 total Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: input open Oct 26 10:30:01 server sshd[12345]: debug3: received SSH_MSG_CHANNEL_DATA (len 12345678901) Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: read 12345678901 bytes, 12345678901 total Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: input open Oct 26 10:30:01 server sshd[12345]: debug3: received SSH_MSG_CHANNEL_DATA (len 123456789012) Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: read 123456789012 bytes, 123456789012 total Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: input open Oct 26 10:30:01 server sshd[12345]: debug3: received SSH_MSG_CHANNEL_DATA (len 1234567890123) Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: read 1234567890123 bytes, 1234567890123 total Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: input open Oct 26 10:30:01 server sshd[12345]: debug3: received SSH_MSG_CHANNEL_DATA (len 12345678901234) Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: read 12345678901234 bytes, 12345678901234 total Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: input open Oct 26 10:30:01 server sshd[12345]: debug3: received SSH_MSG_CHANNEL_DATA (len 123456789012345) Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: read 123456789012345 bytes, 123456789012345 total Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: input open Oct 26 10:30:01 server sshd[12345]: debug3: received SSH_MSG_CHANNEL_DATA (len 1234567890123456) Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: read 1234567890123456 bytes, 1234567890123456 total Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: input open Oct 26 10:30:01 server sshd[12345]: debug3: received SSH_MSG_CHANNEL_DATA (len 12345678901234567) Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: read 12345678901234567 bytes, 12345678901234567 total Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: input open Oct 26 10:30:01 server sshd[12345]: debug3: received SSH_MSG_CHANNEL_DATA (len 123456789012345678) Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: read 123456789012345678 bytes, 123456789012345678 total Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: input open Oct 26 10:30:01 server sshd[12345]: debug3: received SSH_MSG_CHANNEL_DATA (len 1234567890123456789) Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: read 1234567890123456789 bytes, 1234567890123456789 total Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: input open Oct 26 10:30:01 server sshd[12345]: debug3: received SSH_MSG_CHANNEL_DATA (len 12345678901234567890) Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: read 12345678901234567890 bytes, 12345678901234567890 total Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: input open Oct 26 10:30:01 server sshd[12345]: debug3: received SSH_MSG_CHANNEL_DATA (len 123456789012345678901) Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: read 123456789012345678901 bytes, 123456789012345678901 total Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: input open Oct 26 10:30:01 server sshd[12345]: debug3: received SSH_MSG_CHANNEL_DATA (len 1234567890123456789012) Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: read 1234567890123456789012 bytes, 1234567890123456789012 total Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: input open Oct 26 10:30:01 server sshd[12345]: debug3: received SSH_MSG_CHANNEL_DATA (len 12345678901234567890123) Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: read 12345678901234567890123 bytes, 12345678901234567890123 total Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: input open Oct 26 10:30:01 server sshd[12345]: debug3: received SSH_MSG_CHANNEL_DATA (len 123456789012345678901234) Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: read 123456789012345678901234 bytes, 123456789012345678901234 total Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: input open Oct 26 10:30:01 server sshd[12345]: debug3: received SSH_MSG_CHANNEL_DATA (len 1234567890123456789012345) Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: read 1234567890123456789012345 bytes, 1234567890123456789012345 total Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: input open Oct 26 10:30:01 server sshd[12345]: debug3: received SSH_MSG_CHANNEL_DATA (len 12345678901234567890123456) Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: read 12345678901234567890123456 bytes, 12345678901234567890123456 total Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: input open Oct 26 10:30:01 server sshd[12345]: debug3: received SSH_MSG_CHANNEL_DATA (len 123456789012345678901234567) Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: read 123456789012345678901234567 bytes, 123456789012345678901234567 total Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: input open Oct 26 10:30:01 server sshd[12345]: debug3: received SSH_MSG_CHANNEL_DATA (len 1234567890123456789012345678) Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: read 1234567890123456789012345678 bytes, 1234567890123456789012345678 total Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: input open Oct 26 10:30:01 server sshd[12345]: debug3: received SSH_MSG_CHANNEL_DATA (len 12345678901234567890123456789) Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: read 12345678901234567890123456789 bytes, 12345678901234567890123456789 total Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: input open Oct 26 10:30:01 server sshd[12345]: debug3: received SSH_MSG_CHANNEL_DATA (len 123456789012345678901234567890) Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: read 123456789012345678901234567890 bytes, 123456789012345678901234567890 total Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: input open Oct 26 10:30:01 server sshd[12345]: debug3: received SSH_MSG_CHANNEL_DATA (len 1234567890123456789012345678901) Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: read 1234567890123456789012345678901 bytes, 1234567890123456789012345678901 total Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: input open Oct 26 10:30:01 server sshd[12345]: debug3: received SSH_MSG_CHANNEL_DATA (len 12345678901234567890123456789012) Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: read 12345678901234567890123456789012 bytes, 12345678901234567890123456789012 total Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: input open Oct 26 10:30:01 server sshd[12345]: debug3: received SSH_MSG_CHANNEL_DATA (len 123456789012345678901234567890123) Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: read 123456789012345678901234567890123 bytes, 123456789012345678901234567890123 total Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: input open Oct 26 10:30:01 server sshd[12345]: debug3: received SSH_MSG_CHANNEL_DATA (len 1234567890123456789012345678901234) Oct 26 10:30:01 server sshd[12345]: debug3: channel 0 - Diagnosis/Fix: