SSH high availability isn’t about making your SSH server itself redundant, it’s about making sure you can reach it reliably, even if one of your access points goes down. This usually means having multiple jump hosts and managing your SSH keys so they work across all of them seamlessly.
Let’s see this in action. Imagine you have two jump hosts, jump1.example.com and jump2.example.com, and you want to SSH into a backend server app.internal.net.
First, ensure your SSH public key is authorized on the backend server for the user you’ll be connecting as. This is standard practice.
# On your local machine, assuming your key is ~/.ssh/id_rsa.pub
ssh user@app.internal.net "mkdir -p ~/.ssh && echo 'ssh-rsa AAAAB3NzaC1...your_public_key... user@local' >> ~/.ssh/authorized_keys"
Now, for the jump hosts. You’ll want your private key available on your local machine, and your corresponding public key needs to be on both jump hosts, authorized for the user you’ll use to log into them.
On your local machine, you might have an SSH config file (~/.ssh/config) that looks like this:
Host jump1
Hostname jump1.example.com
User jumpuser
IdentityFile ~/.ssh/id_rsa
Host jump2
Hostname jump2.example.com
User jumpuser
IdentityFile ~/.ssh/id_rsa
Host app
Hostname app.internal.net
User user
ProxyJump jump1
With this config, you can connect to app.internal.net via jump1 like this:
ssh app
To add redundancy, you’d modify your ~/.ssh/config to use ProxyJump to a list of hosts. This feature, available in OpenSSH 7.3 and later, tells SSH to try the hosts in order until one succeeds.
Host app
Hostname app.internal.net
User user
ProxyJump jump1,jump2 # Comma-separated list
IdentityFile ~/.ssh/id_rsa # Your local private key
Now, when you run ssh app, your SSH client will first attempt to connect to jump1.example.com using ~/.ssh/id_rsa. If jump1 is unreachable or fails, it will automatically try jump2.example.com using the same identity file.
The problem this solves is the single point of failure in your access path. If your primary jump host becomes unavailable (network issue, server reboot, firewall change), your SSH connection can still be established through the secondary jump host. The key management aspect is ensuring that the same SSH key pair is used for authentication to all jump hosts and the final destination, and that the public key is present in the authorized_keys file on each jump host for the jumpuser.
The system isn’t really about making the jump hosts themselves highly available (though that’s a good practice for the infrastructure). It’s about your client’s ability to find a path. The ProxyJump directive is the magic here. When it sees a comma-separated list, it iterates through them. If the first jump fails, the connection attempt doesn’t just die; the client rewinds and tries the next host in the list, re-establishing the proxy tunnel. This is all handled client-side, meaning your jump hosts don’t need to coordinate or know about each other.
What most people don’t realize is that the ProxyJump directive supports an ordered list for failover, and that the client handles the retries and state management internally. It’s not just a simple ProxyCommand where you’d have to script failover yourself; ProxyJump is a built-in, stateful mechanism designed precisely for this. You can even chain them, like ProxyJump jump1,jump2,jump3:port22,jump4.
The next concept you’ll run into is managing these keys at scale, especially when dealing with many users and many jump/backend servers, leading to discussions around SSH certificate authorities or bastion host orchestration.