SSH’s -vvv flag is your best friend when things go sideways, and it’s not just about seeing more logs; it’s about seeing the exact dance between client and server during authentication and connection setup.
Let’s see it in action. Imagine you’re trying to connect to a server user@remote.example.com and it’s failing. You’d run:
ssh -vvv user@remote.example.com
You’ll see output like this, which is crucial for understanding where the breakdown occurs:
debug1: Reading configuration data /home/user/.ssh/config
debug1: Applying options:
debug1: from /home/user/.ssh/config line 10: ForwardAgent yes
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Applying options:
debug1: from /etc/ssh/ssh_config line 19: ForwardAgent no
debug1: ssh_config_apply_delta: Applying override for ForwardAgent: yes
debug1: Executing proxy command: exec ssh-proxy remote.example.com
debug1: Connecting to 192.0.2.1 port 22.
debug1: Connection established.
debug1: identity file /home/user/.ssh/id_rsa type 1
debug1: Checking offer: publickey
debug1: Sending SSH2_MSG_USERAUTH_REQUEST
debug1: no_auth_methods_available: publickey,password
debug1: Authentications that can continue: publickey,password
debug1: Next authentication method: publickey
debug1: Trying private key: /home/user/.ssh/id_rsa
debug1: read PEM private key: /home/user/.ssh/id_rsa
debug1: private key file: # of keys = 1
debug1: trying secret key file "/home/user/.ssh/id_rsa"
debug1: SSH2_MSG_USERAUTH_FAILURE received: publickey
debug1: Authentications that can continue: password
debug1: Next authentication method: password
debug1: Trying password:
debug1: read PEM private key: /home/user/.ssh/id_rsa
debug1: authentication failed for user user: 10
debug1: ssh_dispatch_run_fatal: Connection to 192.0.2.1 port 22: authentication failed
The core problem SSH solves is secure, encrypted communication over an untrusted network. What -vvv reveals is the detailed, multi-step handshake and authentication process that achieves this. It’s not a single password check; it’s a negotiation. The client offers credentials (like public keys), the server verifies them, and if successful, a secure channel is established. -vvv shows you every single packet and decision made in this negotiation.
The most surprising truth about SSH verbose debugging is how much it exposes about the server’s configuration and willingness to accept certain authentication methods, not just the client’s.
Let’s break down the common failure points -vvv illuminates:
-
Client-side Configuration Issues:
- Diagnosis: Look for lines like
debug1: Reading configuration data /home/user/.ssh/configanddebug1: from /home/user/.ssh/config line 10: ForwardAgent yes. If the path is wrong, or a directive likeIdentityFilepoints to a non-existent or incorrectly permissioned key, the client won’t even try the right credentials. - Fix: Ensure your
~/.ssh/configis correctly pointing to valid files and directives. For example, ifIdentityFile ~/.ssh/my_keyis present, ensure~/.ssh/my_keyexists and has permissions600. - Why it works: The client reads its configuration to know how and with what to connect. Incorrect config means it tries the wrong things or doesn’t try the right things at all.
- Diagnosis: Look for lines like
-
Incorrect or Missing Identity Files:
- Diagnosis: See
debug1: identity file /home/user/.ssh/id_rsa type 1followed bydebug1: Trying private key: /home/user/.ssh/id_rsaand thendebug1: authentication failed for user user: 10. If the path listed is not the key you expect, or if no identity files are listed, the client isn’t offering the correct key. - Fix: Generate a new key pair (
ssh-keygen -t rsa -b 4096 -f ~/.ssh/new_key) and add its public part to the server’s~/.ssh/authorized_keys. Then, either addIdentityFile ~/.ssh/new_keyto your~/.ssh/configor ensure~/.ssh/new_keyis the default (id_rsa,id_ecdsa, etc.) or explicitly specified. - Why it works: SSH uses public-key cryptography for authentication. The client proves it owns the private key corresponding to a public key authorized on the server. If the client offers the wrong key, or no key at all, authentication fails.
- Diagnosis: See
-
Incorrect File Permissions on Client:
- Diagnosis: If you see
debug1: read PEM private key: /home/user/.ssh/id_rsabut it fails later, check permissions.sshis notoriously picky. If~/.sshis world-writable (e.g.,777) or your private key file is group-writable (e.g.,660with others in your group), SSH will refuse to use it. - Fix: Set permissions:
chmod 700 ~/.sshandchmod 600 ~/.ssh/id_rsa. - Why it works: SSH enforces strict permissions to prevent unauthorized access to your private keys, which would compromise your identity.
- Diagnosis: If you see
-
Server-Side
sshd_configRestrictions:- Diagnosis: Look for lines indicating the server is refusing the connection after the client has offered keys. For example, if the server explicitly disables public key authentication, you’ll see
debug1: Authentications that can continue: passwordafter the client triespublickey. - Fix: On the server, edit
/etc/ssh/sshd_config. EnsurePubkeyAuthentication yesis present and uncommented. If you’re using specific key types, ensure they are allowed (e.g.,HostKeyAlgorithms). Restartsshdwithsudo systemctl restart sshd. - Why it works: The server’s
sshd_configdictates which authentication methods are permitted. If public key auth is disabled, the server will reject it.
- Diagnosis: Look for lines indicating the server is refusing the connection after the client has offered keys. For example, if the server explicitly disables public key authentication, you’ll see
-
User Not Authorized on Server:
- Diagnosis: You might see
debug1: SSH2_MSG_USERAUTH_FAILURE received: publickeyanddebug1: Authentications that can continue: password, but the server logs (often found in/var/log/auth.logor/var/log/secureon the server) will show "user@remote.example.com: invalid user" or similar, even if the username is correct. The client’s verbose output might not explicitly state the user is invalid, but the server’s rejection of all authentication methods for that user is the clue. - Fix: Ensure the user account exists on the server and that your public key is correctly added to
~/.ssh/authorized_keysfor that user. Double-check the permissions on the server’s~/.sshdirectory (700) and~/.ssh/authorized_keysfile (600). - Why it works: The server must have a valid user account and the corresponding public key must be listed in that user’s
authorized_keysfile for public-key authentication to succeed.
- Diagnosis: You might see
-
SSH Protocol Mismatch or Version Issues:
- Diagnosis: Less common with modern SSH, but you might see messages about protocol versions or supported algorithms. For example, if the client only supports SSHv1 and the server only SSHv2, or vice-versa.
- Fix: Ensure both client and server are using compatible SSH protocols. Typically, SSHv2 is the standard. On the server,
Protocol 2insshd_configforces v2. On the client, you can sometimes specify-2if needed, though it’s usually the default. - Why it works: Different SSH protocol versions have different message formats and security mechanisms. A mismatch prevents handshake completion.
-
Network Firewalls or Intrusion Detection Systems (IDS):
- Diagnosis: The connection might get reset abruptly after the initial TCP handshake, or you might see no response at all after
debug1: Connecting to 192.0.2.1 port 22.. If the connection is established but then immediately torn down, it could be an IDS. - Fix: Work with your network administrator to ensure port 22 (or your custom SSH port) is open and that SSH traffic isn’t being flagged and blocked by firewalls or IDS.
- Why it works: Network intermediaries can block or terminate connections that appear suspicious or violate policy, even if the SSH client and server are configured correctly.
- Diagnosis: The connection might get reset abruptly after the initial TCP handshake, or you might see no response at all after
After fixing these, the next error you’ll likely encounter if you haven’t fully addressed the server’s configuration is a Permission denied (publickey,password). error, indicating the server is still not accepting your authentication attempt, but now it’s for a subtler reason than a simple missing key.