The TLS handshake is failing because the private key presented by the server doesn’t match the public key embedded in the certificate it’s offering. This means the client can’t cryptographically verify the server’s identity, hence the connection is dropped.

Here are the most common reasons this happens and how to fix them:

  1. Using the Wrong Private Key File: You might have multiple private keys on your server, and the web server is configured to use a key that doesn’t correspond to the certificate.

    • Diagnosis: On your web server (e.g., Nginx, Apache), locate the TLS configuration file. Find the ssl_certificate_key (Nginx) or SSLCertificateKeyFile (Apache) directive. Note the path to the private key file specified. Then, use openssl to inspect the certificate and the private key:
      openssl x509 -noout -modulus -in /path/to/your/certificate.crt | openssl md5
      openssl rsa -noout -modulus -in /path/to/your/private.key | openssl md5
      
      The MD5 hashes for both commands must match exactly. If they don’t, you’re using the wrong key.
    • Fix: Update the ssl_certificate_key or SSLCertificateKeyFile directive in your web server’s configuration to point to the correct private key file that was generated alongside your certificate. For example, in Nginx:
      ssl_certificate /etc/ssl/certs/your_domain.crt;
      ssl_certificate_key /etc/ssl/private/your_domain.key; # Ensure this is the correct key
      
      Then reload your web server: sudo systemctl reload nginx or sudo systemctl reload apache2.
    • Why it works: The modulus is a fundamental component of the public/private key pair. If the moduli match, the key and certificate are a valid pair.
  2. Private Key File is Corrupted or Incomplete: The private key file might have been truncated during transfer, or its contents might be corrupted.

    • Diagnosis: Run the openssl rsa -noout -modulus -in /path/to/your/private.key | openssl md5 command. If it fails with an error like "bad decrypt," "PEM_read_bio_RSAPrivateKey failed," or if the output is gibberish, the key file is likely corrupt. Also, check the file size; a private key for a modern certificate (e.g., RSA 2048-bit) should be at least a few hundred bytes.
    • Fix: Obtain a fresh copy of the private key file from your Certificate Authority (CA) or your internal key management system. Ensure the transfer is complete and the file is intact. Replace the existing corrupt file with the new, correct one.
    • Why it works: A complete and uncorrupted private key file allows openssl to correctly extract the modulus for comparison.
  3. Certificate Was Generated with a Different Key: You might have generated a new private key after obtaining a certificate, or vice-versa, without regenerating the other.

    • Diagnosis: This is essentially the same check as point 1. The openssl x509 -noout -modulus -in ... and openssl rsa -noout -modulus -in ... commands will produce different MD5 hashes if the certificate and private key were not generated as a pair.
    • Fix: If you generated a new private key, you must re-issue your certificate from the CA using that new private key. If you generated a new certificate, ensure your web server is configured to use the private key that matches that specific certificate.
    • Why it works: Certificates are bound to the public key derived from a specific private key at the time of issuance. Mismatched pairs are fundamentally incompatible.
  4. Incorrect File Permissions on the Private Key: The web server process might not have read permissions for the private key file. While this often results in a "permission denied" error when the server starts, it can sometimes manifest as a handshake failure if the server attempts to read it later and fails silently or with an obscure error.

    • Diagnosis: Check the file permissions of the private key file:
      ls -l /path/to/your/private.key
      
      The web server user (e.g., www-data for Apache/Nginx on Debian/Ubuntu, nginx for Nginx on CentOS/RHEL) needs read access. Private keys should also typically be restricted to root and the web server user only.
    • Fix: Set appropriate permissions. A common secure setting is:
      sudo chown root:www-data /etc/ssl/private/your_domain.key
      sudo chmod 640 /etc/ssl/private/your_domain.key
      
      (Adjust root:www-data and /etc/ssl/private/your_domain.key to your specific user/group and path). Then reload your web server.
    • Why it works: The web server process must be able to read the private key file to perform the TLS handshake. Restricting permissions prevents other users from reading it.
  5. Using a Certificate Chain File Instead of the Private Key: In some configurations, people accidentally point the private key directive to a certificate chain file (which contains multiple certificates) instead of the actual private key.

    • Diagnosis: Examine the contents of the file specified by ssl_certificate_key or SSLCertificateKeyFile. If it contains multiple -----BEGIN CERTIFICATE----- blocks, it’s a chain file, not a private key. A private key file will contain a -----BEGIN PRIVATE KEY----- or -----BEGIN RSA PRIVATE KEY----- block.
    • Fix: Locate the correct private key file and update the web server configuration to point to it.
    • Why it works: The web server needs the actual private key to decrypt the session key negotiated with the client, not a chain of intermediate certificates.
  6. Intermediate Certificate Included in the Private Key File: Some tools or processes might incorrectly concatenate the intermediate certificate bundle into the private key file.

    • Diagnosis: Inspect the private key file. If it contains -----BEGIN CERTIFICATE----- blocks in addition to or instead of -----BEGIN PRIVATE KEY----- blocks, it’s malformed.
    • Fix: Remove any -----BEGIN CERTIFICATE----- blocks from the private key file. Ensure the ssl_certificate or SSLCertificateFile directive points to the correct full chain file (which should contain your server certificate followed by intermediate certificates). Reload the web server.
    • Why it works: The private key file must contain only the private key material for the server to use.

The next error you’ll likely encounter if you fix this is a "certificate not trusted" error if the intermediate certificates are missing from your certificate chain file.

Want structured learning?

Take the full Tls-ssl course →