SSH Hardening: Securing Your Linux Servers

The order matters. Keys go in first, while password auth is still there to catch you if something goes wrong. The daemon config closes that door afterwards. Everything past that point — 2FA, host-based trust for automation, fail2ban and log monitoring — assumes the first two already work.

Keys first

Password authentication can be compromised through brute-force, keyloggers, or credential stuffing. Keys eliminate all of that.

Generate the pair on your local machine — not the server:

# ED25519 recommended
ssh-keygen -t ed25519 -C "your_email@example.com" -f ~/.ssh/id_prod_server

# RSA fallback for older systems
ssh-keygen -t rsa -b 4096 -C "your_email@example.com" -f ~/.ssh/id_prod_server

ED25519 is faster, more secure, and uses shorter keys than RSA. I’ve switched all my infrastructure to it.

Deploy the public half:

ssh-copy-id -i ~/.ssh/id_prod_server.pub username@server_ip

# If ssh-copy-id isn't available
cat ~/.ssh/id_prod_server.pub | ssh username@server_ip "mkdir -p ~/.ssh && cat >> ~/.ssh/authorized_keys"

Open a second terminal and log in with the key, while the first session stays connected:

ssh -i ~/.ssh/id_prod_server username@server_ip

A shell with no password prompt is the only acceptable result. Everything below assumes it.

If you’re still asked for a password, permissions are the usual cause — SSH ignores keys on files it considers too readable, and says nothing about why:

chmod 700 ~/.ssh
chmod 600 ~/.ssh/authorized_keys
chmod 600 ~/.ssh/id_prod_server

The daemon config

This is where the hardening actually happens. Rather than editing /etc/ssh/sshd_config itself, put your settings in a drop-in — /etc/ssh/sshd_config.d/10-hardening.conf. Debian 11+, Ubuntu 20.04+ and RHEL 9 all include that directory at the top of the main file (on RHEL 8, edit the main file instead), and for most keywords sshd uses the first value it reads, so the drop-in wins and package updates never have to merge your edits into theirs:

# Network
Port 2222
AddressFamily inet

# Authentication
PermitRootLogin no
PubkeyAuthentication yes
PasswordAuthentication no
PermitEmptyPasswords no
KbdInteractiveAuthentication no
UsePAM yes
PubkeyAcceptedAlgorithms ssh-ed25519,rsa-sha2-512,rsa-sha2-256

# Who may log in at all
AllowUsers deployer sysadmin
# AllowGroups ssh-users

# Sessions
MaxAuthTries 3
MaxSessions 2
LoginGraceTime 30
ClientAliveInterval 300
ClientAliveCountMax 2

# Features you don't use
X11Forwarding no
PermitUserEnvironment no
AllowAgentForwarding no
AllowTcpForwarding no
PermitTunnel no

# Logging
LogLevel VERBOSE

# Cryptography
KexAlgorithms sntrup761x25519-sha512@openssh.com,curve25519-sha256,curve25519-sha256@libssh.org
Ciphers chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com
MACs hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com

# Trust between machines: off until you need it
HostbasedAuthentication no
IgnoreRhosts yes

Two notes on the cryptography. sntrup761x25519-sha512 is the post-quantum hybrid key exchange OpenSSH has defaulted to since 9.0, and a list without it quietly downgrades every connection to classical curve25519. From OpenSSH 9.9, put mlkem768x25519-sha256 first as well — it’s the default since 10.0. Older versions refuse to start on an algorithm name they don’t know, which sshd -t below catches.

The lines that will lock you out

Two of them, if you paste the block unchanged.

AllowUsers deployer sysadmin is an allowlist, and everyone not on it is refused — including you. Put your own username there before saving.

Port 2222 only works if everything between you and the daemon agrees. On RHEL, SELinux won’t let sshd bind a port that isn’t labelled for it, and sshd fails to start. On Ubuntu 22.10 and newer, sshd is socket-activated and the port lives in ssh.socket. The firewall has to allow it, and so do three other places: the fail2ban jail, your ~/.ssh/config, and any monitoring that connects over SSH.

Three more are correct, but have consequences better met now than at 2am:

• AllowTcpForwarding no kills ssh -L, ssh -D and ProxyJump through this host. If this box is your bastion, the answer isn’t to turn it back on globally — scope it with a Match User block to the account that needs it.
• ClientAliveInterval 300 with ClientAliveCountMax 2 drops connections whose client has stopped answering, after about ten minutes. It does not end idle sessions: a live client answers the keepalive however long it sits there.
• MaxSessions 2 caps channels per connection, and automation tends to open more of them than you’d expect.

Two directives in that config are deliberately off now and come back on later. KbdInteractiveAuthentication no is correct until 2FA exists — turning it on before there’s a second factor just adds a prompt. HostbasedAuthentication no stays off unless you reach the machine-to-machine section; it’s a feature you enable on purpose, not a default worth having.

Validate, open the port, restart

Check the syntax while you still have a session that works. sshd -t prints nothing when the config is valid:

sudo sshd -t

Then, in this order: let the new port through, restart the daemon, and only after a new session has logged in on 2222, close 22.

sudo ufw allow 2222/tcp

# Ubuntu 22.10+ generates ssh.socket from sshd_config's Port
sudo systemctl daemon-reload
sudo systemctl restart ssh.socket ssh

# after a new session on 2222 works:
sudo ufw delete allow 22/tcp

sudo semanage port -a -t ssh_port_t -p tcp 2222
sudo firewall-cmd --permanent --add-port=2222/tcp
sudo firewall-cmd --reload
sudo systemctl restart sshd

# after a new session on 2222 works:
sudo firewall-cmd --permanent --remove-service=ssh
sudo firewall-cmd --reload

Two-factor for human accounts

Even if someone steals your private key, 2FA means they still can’t get in without the second factor.

sudo apt install libpam-google-authenticator

sudo dnf install epel-release
sudo dnf install google-authenticator

Run the enrolment as the user who will actually log in — not under sudo, or the secret ends up in root’s home instead:

google-authenticator

Prompts to answer: time-based tokens → Yes, update ~/.google_authenticator → Yes, disallow multiple uses → Yes, increase time window → No (unless you have time sync issues), enable rate-limiting → Yes.

Scan the QR code with Google Authenticator, Authy, or any TOTP app.

Wire it into PAM, in /etc/pam.d/sshd. Two edits: add the TOTP module, and take out the password stack. The key is already the first factor — left in, the password stack makes every login ask for a password after the code, and fail outright for accounts that don’t have one.

# Debian / Ubuntu: comment out
# @include common-auth
# RHEL / Rocky: comment out
# auth       substack     password-auth

auth required pam_google_authenticator.so nullok

nullok lets users without 2FA configured still log in. Remove it once all users have it set up — and verify they have, because removing it locks out every account that never ran the enrolment.

Then back to the drop-in, where keyboard-interactive now earns the yes it was denied earlier:

KbdInteractiveAuthentication yes
AuthenticationMethods publickey,keyboard-interactive

sudo sshd -t && sudo systemctl restart ssh    # sshd on RHEL

Connections now require both your SSH key and the 2FA code.

Every connection, which is the part that bites. scp, rsync, Ansible, backup jobs — anything non-interactive now stops at a prompt no script can answer. Scope 2FA to the accounts humans use and leave automation on keys alone: AuthenticationMethods publickey inside a Match User block for the automation accounts. Machine-to-machine trust is the next section’s job.

Host-based auth for machines

Host-based authentication lets one server authenticate to another based on the client machine’s host key rather than user keys. I use it for automated backup systems, Ansible/Puppet, monitoring that executes remote commands, database replication, and CI/CD pipelines.

Prerequisites: Root access on both machines, DNS or /etc/hosts entries for hostname resolution.

On the server

Enable it for the accounts that need it, reversing the HostbasedAuthentication no from earlier only inside a Match block — at the very end of the drop-in, since everything after a Match line belongs to it:

HostbasedUsesNameFromPacketOnly yes
HostbasedAcceptedAlgorithms ssh-ed25519,rsa-sha2-512,rsa-sha2-256

Match User backup,monitor,jenkins
    HostbasedAuthentication yes

IgnoreRhosts yes can stay: it covers the per-user ~/.rhosts and ~/.shosts files, while the system-wide /etc/ssh/shosts.equiv is read regardless. Set it to no only if you really want users to grant trust from their own home directories.

Then declare which hosts are trusted, in /etc/ssh/shosts.equiv:

# Format: hostname [username]
backup-server.example.com backup
monitoring.example.com monitor
ci-runner-01.example.com jenkins

sudo chmod 600 /etc/ssh/shosts.equiv
sudo chown root:root /etc/ssh/shosts.equiv

On the client

Edit /etc/ssh/ssh_config:

HostbasedAuthentication yes
EnableSSHKeysign yes
PreferredAuthentications hostbased,publickey

ssh-keysign must be setuid root to read the host keys:

sudo chmod 4711 /usr/lib/openssh/ssh-keysign       # Debian / Ubuntu
sudo chmod 4711 /usr/libexec/openssh/ssh-keysign   # RHEL / Rocky

Exchange the host keys

On the client, get the host public key:

sudo cat /etc/ssh/ssh_host_ed25519_key.pub

Add it to /etc/ssh/ssh_known_hosts on the server:

# Format: hostname,ip key-type public-key
backup-server.example.com,192.168.1.10 ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAILo...

sudo chmod 644 /etc/ssh/ssh_known_hosts
sudo chown root:root /etc/ssh/ssh_known_hosts

Then test. Authentication succeeded (hostbased) in the verbose output is what you’re looking for:

ssh -v backup@production-server.example.com

Beyond two or three clients, collecting those keys by hand stops being reasonable. Save this on the server as /usr/local/bin/distribute-host-keys.sh:

#!/bin/bash
KNOWN_HOSTS="/etc/ssh/ssh_known_hosts"
TEMP_KEYS=$(mktemp)
trap 'rm -f "$TEMP_KEYS"' EXIT

CLIENTS=(
    "backup-server.example.com"
    "monitoring.example.com"
    "ci-runner-01.example.com"
)

for client in "${CLIENTS[@]}"; do
    IP=$(dig +short "$client" | tail -1)
    KEY=$(ssh-keyscan -t ed25519 "$client" 2>/dev/null)

    if [ -n "$KEY" ]; then
        echo "$client,$IP $(echo "$KEY" | awk '{print $2, $3}')" >> "$TEMP_KEYS"
    else
        echo "Failed to get key from $client"
    fi
done

[ -f "$KNOWN_HOSTS" ] && cp "$KNOWN_HOSTS" "${KNOWN_HOSTS}.backup.$(date +%F)"

cat "$TEMP_KEYS" >> "$KNOWN_HOSTS"
sort -u "$KNOWN_HOSTS" -o "$KNOWN_HOSTS"
chmod 644 "$KNOWN_HOSTS"

ssh-keyscan trusts whatever answers at that name. Run it from a network you trust, and compare the fingerprints against the clients themselves the first time.

sudo chmod +x /usr/local/bin/distribute-host-keys.sh
sudo /usr/local/bin/distribute-host-keys.sh

What it looks like in practice

A backup server pulling from production. On the production servers, the Match User backup block above plus the trust line in shosts.equiv:

backup-server.example.com backup

On the backup server, in /etc/ssh/ssh_config:

Host prod-*
    HostbasedAuthentication yes
    PreferredAuthentications hostbased
    User backup

Now the backup server can pull automatically, with no key and no password anywhere in the job:

rsync -avz prod-web-01:/var/www/ /backup/web-01/

Combining with user key auth is the safest configuration, because a compromised client host alone is then not enough:

AuthenticationMethods publickey,hostbased

Review /etc/ssh/shosts.equiv monthly, keep LogLevel VERBOSE so host-based authentications actually appear in the logs, and restrict SSH by firewall to the trusted client IPs.

Revoking access is two removals and a restart — miss either one and the trust survives:

# Remove the line from shosts.equiv
sudo sed -i '/^hostname\.example\.com /d' /etc/ssh/shosts.equiv

# Remove the host key
sudo ssh-keygen -R hostname.example.com -f /etc/ssh/ssh_known_hosts

sudo systemctl restart ssh    # sshd on RHEL

Watching it

Fail2ban monitors logs and blocks IPs that show malicious behavior.

sudo apt install fail2ban

sudo dnf install epel-release
sudo dnf install fail2ban

Create /etc/fail2ban/jail.local. The port has to match the one in your sshd config, or the jail bans on a port nothing is listening on. backend = systemd reads sshd’s entries from the journal: Debian 12 and newer don’t write /var/log/auth.log unless rsyslog is installed, and a jail pointed at a missing file stops fail2ban from starting.

[DEFAULT]
bantime = 3600
findtime = 600
maxretry = 3
destemail = your_email@example.com
sendername = Fail2Ban
action = %(action_mwl)s

[sshd]
enabled = true
port = 2222
backend = systemd

action_mwl sends mail, so it needs a working MTA on the host; without one, use the default %(action_)s.

sudo systemctl enable --now fail2ban

# Status and management
sudo fail2ban-client status sshd
sudo fail2ban-client get sshd banned
sudo fail2ban-client set sshd unbanip 192.168.1.100

Your own SSH config

On your local machine, ~/.ssh/config saves typing the port and key on every connection:

Host production-server
    HostName server_ip
    Port 2222
    User deployer
    IdentityFile ~/.ssh/id_prod_server
    ServerAliveInterval 60
    ServerAliveCountMax 3

Host staging-server
    HostName staging_ip
    Port 2222
    User deployer
    IdentityFile ~/.ssh/id_staging_server
    ProxyJump bastion-host

ssh production-server

Reading the logs

Directly, when you want to know what happened. With password authentication off, failed attempts show up as invalid users and closed pre-auth connections, not as “Failed password”:

# Real-time
sudo journalctl -u ssh -f            # -u sshd on RHEL

# Probing and failures
sudo journalctl -u ssh --since today | grep -E "Invalid user|Connection closed by authenticating user" | tail -20

# Successful logins
sudo journalctl -u ssh --since today | grep "Accepted publickey" | tail -20

And a report worth having in your inbox rather than one you have to remember to run — /usr/local/bin/ssh-monitor.sh:

#!/bin/bash
UNIT=ssh    # sshd on RHEL

echo "SSH Security Report - $(date)"
echo "================================"
echo

echo "Invalid users, last 24h (count, user, source):"
journalctl -u "$UNIT" --since "24 hours ago" --no-pager \
  | grep -oE "Invalid user \S+ from \S+" | sort | uniq -c | sort -nr | head -20
echo

echo "Successful logins, last 24h:"
journalctl -u "$UNIT" --since "24 hours ago" --no-pager | grep "Accepted" | tail -20
echo

echo "Active SSH sessions:"
who
echo

echo "Current Fail2Ban bans:"
fail2ban-client status sshd 2>/dev/null

sudo chmod +x /usr/local/bin/ssh-monitor.sh

# Daily at 09:00, by mail
echo "0 9 * * * root /usr/local/bin/ssh-monitor.sh | mail -s 'SSH Security Report' your_email@example.com" \
  | sudo tee /etc/cron.d/ssh-monitor

A file in /etc/cron.d, not crontab -: piping into crontab - replaces root’s entire crontab with that one line.

When it breaks

Can’t connect at all after a config change. Check the daemon is running, the firewall allows the new port, and the config parses:

sudo systemctl status ssh            # sshd on RHEL
sudo ufw status                      # or: firewall-cmd --list-all
sudo sshd -t
sudo journalctl -u ssh -n 50

On RHEL, sshd failing to start after a port change with a bind error is SELinux: the semanage port line above is missing.

Permission denied (publickey). Almost always file permissions — the same ones from the first section. ~/.ssh must be 700, authorized_keys 600, private keys 600:

ls -la ~/.ssh
chmod 700 ~/.ssh
chmod 600 ~/.ssh/authorized_keys

Too many authentication failures. Your agent is offering every key it holds and the server cuts you off before it reaches the right one:

ssh-add -D
ssh-add ~/.ssh/id_prod_server

# Or force a specific key
ssh -o IdentitiesOnly=yes -i ~/.ssh/id_prod_server user@server

The 2FA code is rejected. TOTP is clock-based, so the server drifting is enough:

timedatectl status
sudo systemctl restart chrony   # chronyd on RHEL

Host-based auth silently falls back to another method. The usual causes, in the order worth checking:

# Server logs
sudo journalctl -u ssh -n 50 | grep -i hostbased

# The client's name must match shosts.equiv
hostname -f

# ssh-keysign must be setuid root: -rws--x--x
ls -l /usr/lib/openssh/ssh-keysign

# The client's host key must be on the server
sudo grep "$(hostname -f)" /etc/ssh/ssh_known_hosts

# Full debug from the client
ssh -vvv -o PreferredAuthentications=hostbased user@server

Keeping it that way

Once a month:

# Review authorized_keys
cat ~/.ssh/authorized_keys

# Host key types and sizes — no DSA, no RSA under 3072 bits
for key in /etc/ssh/ssh_host_*_key.pub; do ssh-keygen -lf "$key"; done

# Users with empty passwords
sudo awk -F: '($2 == "") {print $1}' /etc/shadow

I rotate SSH keys annually: generate the new pair, deploy it everywhere, test, then remove the old public key. The removal is the step people skip, and a key you forgot to revoke is indistinguishable from one you never had.

The auth log that started this still fills up. The difference is that 50,000 attempts against key-only auth on a non-standard port is a graph, not an incident.