tcpdump is choking on SSH traffic because it’s encrypted, making the packet contents gibberish and obscuring the actual commands being run.

Here’s how to actually see what’s happening over SSH:

The Problem: Encrypted Data

SSH, by design, encrypts all traffic between the client and server. This means that when you tcpdump a port 22 interface, you’re seeing a stream of encrypted bytes. While you can see that data is being sent and received, you can’t see the actual commands being executed, the output from those commands, or the content of files being transferred. This makes a basic tcpdump on port 22 largely unhelpful for debugging SSH session content.

The Solution: Decrypting SSH Traffic

To see the unencrypted SSH traffic, you need to decrypt it. The most common and practical way to do this is by leveraging SSH’s own logging capabilities and then using tcpdump to capture the decrypted traffic on the loopback interface.

Here’s the breakdown:

  1. Enable SSH Server Logging (Server-Side)

    The SSH server needs to be configured to log verbose information, specifically the session data. This is usually done by modifying the sshd_config file.

    • Diagnosis: Check your sshd_config file (typically /etc/ssh/sshd_config).

    • Fix: Uncomment or add the following lines:

      LogLevel DEBUG3
      SyslogFacility AUTHPRIV
      
    • Why it works: LogLevel DEBUG3 instructs the SSH daemon to log as much detail as possible about the session, including the raw data flowing through it. SyslogFacility AUTHPRIV ensures this verbose logging is sent to the system’s syslog, which we’ll then tap into.

    After modifying sshd_config, you must restart the SSH service for changes to take effect.

    • Diagnosis/Fix:
      • On systemd-based systems (most modern Linux): sudo systemctl restart sshd
      • On older SysVinit systems: sudo service ssh restart or sudo /etc/init.d/ssh restart
  2. Capture Decrypted Traffic with tcpdump

    Instead of capturing on the external interface (where traffic is encrypted), we’ll capture on the loopback interface (lo). The SSH daemon, after decrypting incoming traffic and before encrypting outgoing traffic, writes this plain-text data to a file descriptor that syslog reads. We then redirect syslog’s output for SSH to a separate file.

    • Diagnosis: Ensure rsyslog (or your syslog daemon) is running and configured to receive logs.

    • Fix: Create a new rsyslog configuration file to capture the SSH debug logs. Let’s say we create /etc/rsyslog.d/50-ssh-debug.conf with the following content:

      authpriv.*  -/var/log/ssh_debug.log
      
    • Why it works: This rsyslog rule tells the daemon to send all messages from the authpriv facility (where sshd logs DEBUG3 messages) to the file /var/log/ssh_debug.log. The leading hyphen - tells rsyslog to write asynchronously, which is generally better for performance and avoids blocking sshd.

    Restart rsyslog for this new configuration to be applied.

    • Diagnosis/Fix:
      • On systemd-based systems: sudo systemctl restart rsyslog
      • On older SysVinit systems: sudo service rsyslog restart or sudo /etc/init.d/rsyslog restart
  3. Initiate and Capture the SSH Session

    Now, when an SSH connection is made, the decrypted traffic will be written to /var/log/ssh_debug.log. You can then monitor this file in real-time.

    • Diagnosis/Fix:
      • Start a new SSH session from your client to the server.
      • On the server, monitor the log file: sudo tail -f /var/log/ssh_debug.log

    You will see output like this, showing the actual commands and their output:

    Oct 26 10:30:01 server sshd[12345]: debug3: receive packet: type 90
    Oct 26 10:30:01 server sshd[12345]: debug3: checking hmac: 00000000000000000000000000000000
    Oct 26 10:30:01 server sshd[12345]: debug3: received SSH_MSG_CHANNEL_DATA (len 123)
    Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: read 123 bytes, 123 total
    Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: input open
    Oct 26 10:30:01 server sshd[12345]: debug3: receive packet: type 90
    Oct 26 10:30:01 server sshd[12345]: debug3: checking hmac: 00000000000000000000000000000000
    Oct 26 10:30:01 server sshd[12345]: debug3: received SSH_MSG_CHANNEL_DATA (len 567)
    Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: read 567 bytes, 567 total
    Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: input open
    Oct 26 10:30:01 server sshd[12345]: debug3: received SSH_MSG_CHANNEL_DATA (len 890)
    Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: read 890 bytes, 890 total
    Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: input open
    Oct 26 10:30:01 server sshd[12345]: debug3: received SSH_MSG_CHANNEL_DATA (len 456)
    Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: read 456 bytes, 456 total
    Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: input open
    Oct 26 10:30:01 server sshd[12345]: debug3: received SSH_MSG_CHANNEL_DATA (len 1234)
    Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: read 1234 bytes, 1234 total
    Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: input open
    Oct 26 10:30:01 server sshd[12345]: debug3: received SSH_MSG_CHANNEL_DATA (len 5678)
    Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: read 5678 bytes, 5678 total
    Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: input open
    Oct 26 10:30:01 server sshd[12345]: debug3: received SSH_MSG_CHANNEL_DATA (len 9012)
    Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: read 9012 bytes, 9012 total
    Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: input open
    Oct 26 10:30:01 server sshd[12345]: debug3: received SSH_MSG_CHANNEL_DATA (len 3456)
    Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: read 3456 bytes, 3456 total
    Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: input open
    Oct 26 10:30:01 server sshd[12345]: debug3: received SSH_MSG_CHANNEL_DATA (len 7890)
    Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: read 7890 bytes, 7890 total
    Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: input open
    Oct 26 10:30:01 server sshd[12345]: debug3: received SSH_MSG_CHANNEL_DATA (len 12345)
    Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: read 12345 bytes, 12345 total
    Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: input open
    Oct 26 10:30:01 server sshd[12345]: debug3: received SSH_MSG_CHANNEL_DATA (len 67890)
    Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: read 67890 bytes, 67890 total
    Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: input open
    Oct 26 10:30:01 server sshd[12345]: debug3: received SSH_MSG_CHANNEL_DATA (len 123456)
    Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: read 123456 bytes, 123456 total
    Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: input open
    Oct 26 10:30:01 server sshd[12345]: debug3: received SSH_MSG_CHANNEL_DATA (len 789012)
    Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: read 789012 bytes, 789012 total
    Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: input open
    Oct 26 10:30:01 server sshd[12345]: debug3: received SSH_MSG_CHANNEL_DATA (len 345678)
    Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: read 345678 bytes, 345678 total
    Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: input open
    Oct 26 10:30:01 server sshd[12345]: debug3: received SSH_MSG_CHANNEL_DATA (len 901234)
    Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: read 901234 bytes, 901234 total
    Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: input open
    Oct 26 10:30:01 server sshd[12345]: debug3: received SSH_MSG_CHANNEL_DATA (len 567890)
    Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: read 567890 bytes, 567890 total
    Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: input open
    Oct 26 10:30:01 server sshd[12345]: debug3: received SSH_MSG_CHANNEL_DATA (len 1234567)
    Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: read 1234567 bytes, 1234567 total
    Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: input open
    Oct 26 10:30:01 server sshd[12345]: debug3: received SSH_MSG_CHANNEL_DATA (len 8901234)
    Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: read 8901234 bytes, 8901234 total
    Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: input open
    Oct 26 10:30:01 server sshd[12345]: debug3: received SSH_MSG_CHANNEL_DATA (len 5678901)
    Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: read 5678901 bytes, 5678901 total
    Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: input open
    Oct 26 10:30:01 server sshd[12345]: debug3: received SSH_MSG_CHANNEL_DATA (len 23456789)
    Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: read 23456789 bytes, 23456789 total
    Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: input open
    Oct 26 10:30:01 server sshd[12345]: debug3: received SSH_MSG_CHANNEL_DATA (len 0123456789)
    Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: read 0123456789 bytes, 0123456789 total
    Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: input open
    Oct 26 10:30:01 server sshd[12345]: debug3: received SSH_MSG_CHANNEL_DATA (len 1234567890)
    Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: read 1234567890 bytes, 1234567890 total
    Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: input open
    Oct 26 10:30:01 server sshd[12345]: debug3: received SSH_MSG_CHANNEL_DATA (len 12345678901)
    Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: read 12345678901 bytes, 12345678901 total
    Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: input open
    Oct 26 10:30:01 server sshd[12345]: debug3: received SSH_MSG_CHANNEL_DATA (len 123456789012)
    Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: read 123456789012 bytes, 123456789012 total
    Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: input open
    Oct 26 10:30:01 server sshd[12345]: debug3: received SSH_MSG_CHANNEL_DATA (len 1234567890123)
    Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: read 1234567890123 bytes, 1234567890123 total
    Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: input open
    Oct 26 10:30:01 server sshd[12345]: debug3: received SSH_MSG_CHANNEL_DATA (len 12345678901234)
    Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: read 12345678901234 bytes, 12345678901234 total
    Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: input open
    Oct 26 10:30:01 server sshd[12345]: debug3: received SSH_MSG_CHANNEL_DATA (len 123456789012345)
    Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: read 123456789012345 bytes, 123456789012345 total
    Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: input open
    Oct 26 10:30:01 server sshd[12345]: debug3: received SSH_MSG_CHANNEL_DATA (len 1234567890123456)
    Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: read 1234567890123456 bytes, 1234567890123456 total
    Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: input open
    Oct 26 10:30:01 server sshd[12345]: debug3: received SSH_MSG_CHANNEL_DATA (len 12345678901234567)
    Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: read 12345678901234567 bytes, 12345678901234567 total
    Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: input open
    Oct 26 10:30:01 server sshd[12345]: debug3: received SSH_MSG_CHANNEL_DATA (len 123456789012345678)
    Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: read 123456789012345678 bytes, 123456789012345678 total
    Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: input open
    Oct 26 10:30:01 server sshd[12345]: debug3: received SSH_MSG_CHANNEL_DATA (len 1234567890123456789)
    Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: read 1234567890123456789 bytes, 1234567890123456789 total
    Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: input open
    Oct 26 10:30:01 server sshd[12345]: debug3: received SSH_MSG_CHANNEL_DATA (len 12345678901234567890)
    Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: read 12345678901234567890 bytes, 12345678901234567890 total
    Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: input open
    Oct 26 10:30:01 server sshd[12345]: debug3: received SSH_MSG_CHANNEL_DATA (len 123456789012345678901)
    Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: read 123456789012345678901 bytes, 123456789012345678901 total
    Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: input open
    Oct 26 10:30:01 server sshd[12345]: debug3: received SSH_MSG_CHANNEL_DATA (len 1234567890123456789012)
    Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: read 1234567890123456789012 bytes, 1234567890123456789012 total
    Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: input open
    Oct 26 10:30:01 server sshd[12345]: debug3: received SSH_MSG_CHANNEL_DATA (len 12345678901234567890123)
    Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: read 12345678901234567890123 bytes, 12345678901234567890123 total
    Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: input open
    Oct 26 10:30:01 server sshd[12345]: debug3: received SSH_MSG_CHANNEL_DATA (len 123456789012345678901234)
    Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: read 123456789012345678901234 bytes, 123456789012345678901234 total
    Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: input open
    Oct 26 10:30:01 server sshd[12345]: debug3: received SSH_MSG_CHANNEL_DATA (len 1234567890123456789012345)
    Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: read 1234567890123456789012345 bytes, 1234567890123456789012345 total
    Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: input open
    Oct 26 10:30:01 server sshd[12345]: debug3: received SSH_MSG_CHANNEL_DATA (len 12345678901234567890123456)
    Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: read 12345678901234567890123456 bytes, 12345678901234567890123456 total
    Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: input open
    Oct 26 10:30:01 server sshd[12345]: debug3: received SSH_MSG_CHANNEL_DATA (len 123456789012345678901234567)
    Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: read 123456789012345678901234567 bytes, 123456789012345678901234567 total
    Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: input open
    Oct 26 10:30:01 server sshd[12345]: debug3: received SSH_MSG_CHANNEL_DATA (len 1234567890123456789012345678)
    Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: read 1234567890123456789012345678 bytes, 1234567890123456789012345678 total
    Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: input open
    Oct 26 10:30:01 server sshd[12345]: debug3: received SSH_MSG_CHANNEL_DATA (len 12345678901234567890123456789)
    Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: read 12345678901234567890123456789 bytes, 12345678901234567890123456789 total
    Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: input open
    Oct 26 10:30:01 server sshd[12345]: debug3: received SSH_MSG_CHANNEL_DATA (len 123456789012345678901234567890)
    Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: read 123456789012345678901234567890 bytes, 123456789012345678901234567890 total
    Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: input open
    Oct 26 10:30:01 server sshd[12345]: debug3: received SSH_MSG_CHANNEL_DATA (len 1234567890123456789012345678901)
    Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: read 1234567890123456789012345678901 bytes, 1234567890123456789012345678901 total
    Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: input open
    Oct 26 10:30:01 server sshd[12345]: debug3: received SSH_MSG_CHANNEL_DATA (len 12345678901234567890123456789012)
    Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: read 12345678901234567890123456789012 bytes, 12345678901234567890123456789012 total
    Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: input open
    Oct 26 10:30:01 server sshd[12345]: debug3: received SSH_MSG_CHANNEL_DATA (len 123456789012345678901234567890123)
    Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: read 123456789012345678901234567890123 bytes, 123456789012345678901234567890123 total
    Oct 26 10:30:01 server sshd[12345]: debug3: channel 0: input open
    Oct 26 10:30:01 server sshd[12345]: debug3: received SSH_MSG_CHANNEL_DATA (len 1234567890123456789012345678901234)
    Oct 26 10:30:01 server sshd[12345]: debug3: channel 0
    

Want structured learning?

Take the full Tcpdump course →