SSH Hardening - Securing Your Linux Servers

The default SSH configuration on most distributions is functional but not production-safe. After managing Linux infrastructure for several years — and finding over 50,000 failed login attempts in a single day’s auth log early in my career — I apply the same hardening steps to every server I manage.

The order matters. Keys go in first, while password auth is still there to catch you if something goes wrong. The daemon config closes that door afterwards. Everything past that point — 2FA, host-based trust for automation, fail2ban and log monitoring — assumes the first two already work.

Keys first

Password authentication can be compromised through brute-force, keyloggers, or credential stuffing. Keys eliminate all of that.

Generate the pair on your local machine — not the server:

# ED25519 recommended
ssh-keygen -t ed25519 -C "your_email@example.com" -f ~/.ssh/id_prod_server

# RSA fallback for older systems
ssh-keygen -t rsa -b 4096 -C "your_email@example.com" -f ~/.ssh/id_prod_server

ED25519 is faster, more secure, and uses shorter keys than RSA. I’ve switched all my infrastructure to it.

Deploy the public half:

ssh-copy-id -i ~/.ssh/id_prod_server.pub username@server_ip

# If ssh-copy-id isn't available
cat ~/.ssh/id_prod_server.pub | ssh username@server_ip "mkdir -p ~/.ssh && cat >> ~/.ssh/authorized_keys"

Open a second terminal and log in with the key, while the first session stays connected:

ssh -i ~/.ssh/id_prod_server username@server_ip

A shell with no password prompt is the only acceptable result. Everything below assumes it.

If you’re still asked for a password, permissions are the usual cause — SSH ignores keys on files it considers too readable, and says nothing about why:

chmod 700 ~/.ssh
chmod 600 ~/.ssh/authorized_keys
chmod 600 ~/.ssh/id_ed25519

The daemon config

This is where the hardening actually happens. Back the file up before you touch it — you’ll want the original the moment something stops connecting, and you’ll be editing it again in two of the sections below:

sudo cp /etc/ssh/sshd_config /etc/ssh/sshd_config.backup.$(date +%F)
sudo nano /etc/ssh/sshd_config

# Network Settings
Port 2222
AddressFamily inet
ListenAddress 0.0.0.0

# Authentication
PermitRootLogin no
PubkeyAuthentication yes
PasswordAuthentication no
PermitEmptyPasswords no
ChallengeResponseAuthentication no
UsePAM yes

# Key Types (ED25519 preferred)
PubkeyAcceptedKeyTypes ssh-ed25519,rsa-sha2-512,rsa-sha2-256

# Limit user access
AllowUsers deployer sysadmin
# AllowGroups ssh-users

# Session Settings
MaxAuthTries 3
MaxSessions 2
LoginGraceTime 30
ClientAliveInterval 300
ClientAliveCountMax 2

# Disable Dangerous Features
X11Forwarding no
PermitUserEnvironment no
AllowAgentForwarding no
AllowTcpForwarding no
PermitTunnel no

# Logging
SyslogFacility AUTH
LogLevel VERBOSE

# Modern Cryptography (2025)
KexAlgorithms curve25519-sha256,curve25519-sha256@libssh.org
Ciphers chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com
MACs hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com

# Security
HostbasedAuthentication no
IgnoreRhosts yes

The lines that will lock you out

Two of them, if you paste the block unchanged.

AllowUsers deployer sysadmin is a whitelist, and everyone not on it is refused — including you. Put your own username there before saving.

Port 2222 only works if the firewall agrees, which is the last step in this section. Once you change it, three other places have to agree too: the fail2ban jail, your ~/.ssh/config, and any monitoring that connects over SSH.

Three more are correct, but have consequences better met now than at 2am:

• AllowTcpForwarding no kills ssh -L, ssh -D and ProxyJump through this host. If this box is your bastion, the answer isn’t to turn it back on globally — scope it to the account that needs it.

• ClientAliveInterval 300 with ClientAliveCountMax 2 drops idle sessions after about ten minutes.

• MaxSessions 2 caps channels per connection, and automation tends to open more of them than you’d expect.

Two directives in that config are deliberately off now and come back on later, so don’t be surprised when a later section contradicts this one. ChallengeResponseAuthentication no is correct until 2FA exists — turning it on before there’s a second factor to answer with just adds a prompt. HostbasedAuthentication no stays off unless you actually reach the machine-to-machine section; it’s a feature you enable on purpose, not a default worth having.

Validate, restart, then the firewall

Check the syntax before restarting, while you still have a session that works. sshd -t prints nothing when the file is valid:

sudo sshd -t
sudo systemctl restart sshd
sudo systemctl status sshd

Open the new port before closing the old one, in that order and in one sitting — the gap between the two is where people lock themselves out:

# UFW (Ubuntu/Debian)
sudo ufw allow 2222/tcp
sudo ufw delete allow 22/tcp
sudo ufw reload

# firewalld (RHEL/CentOS)
sudo firewall-cmd --permanent --add-port=2222/tcp
sudo firewall-cmd --permanent --remove-service=ssh
sudo firewall-cmd --reload

Two-factor for human accounts

Even if someone steals your private key, 2FA means they still can’t get in without the second factor.

# Ubuntu/Debian
sudo apt install libpam-google-authenticator

# RHEL/CentOS
sudo yum install google-authenticator

Run the enrolment as the user who will actually log in — not under sudo, or the secret ends up in root’s home instead:

google-authenticator

Prompts to answer: time-based tokens → Yes, update ~/.google_authenticator → Yes, disallow multiple uses → Yes, increase time window → No (unless you have time sync issues), enable rate-limiting → Yes.

Scan the QR code with Google Authenticator, Authy, or any TOTP app.

Wire it into PAM:

sudo nano /etc/pam.d/sshd

Add at the top:

auth required pam_google_authenticator.so nullok

nullok lets users without 2FA configured still log in. Remove it once all users have it set up — and verify they have, because removing it locks out every account that never ran the enrolment.

Then back to sshd_config, where ChallengeResponseAuthentication now earns the yes it was denied earlier:

ChallengeResponseAuthentication yes
AuthenticationMethods publickey,keyboard-interactive

sudo systemctl restart sshd

Connections now require both your SSH key and the 2FA code.

Every connection, which is the part that bites. scp, rsync, Ansible, backup jobs — anything non-interactive now stops at a prompt no script can answer. Scope 2FA to the accounts humans use and leave automation on keys alone. Machine-to-machine trust is the next section’s job.

Host-based auth for machines

Host-based authentication lets one server authenticate to another based on the client machine’s host key rather than user keys. I use it for automated backup systems, Ansible/Puppet, monitoring that executes remote commands, database replication, and CI/CD pipelines.

Prerequisites: Root access on both machines, DNS or /etc/hosts entries for hostname resolution.

On the server

Enable it, reversing the HostbasedAuthentication no set earlier:

HostbasedAuthentication yes
HostbasedUsesNameFromPacketOnly yes
HostbasedAcceptedKeyTypes ssh-ed25519,rsa-sha2-512,rsa-sha2-256
IgnoreRhosts no
IgnoreUserKnownHosts no

sudo systemctl restart sshd

Then declare which hosts are trusted, in /etc/ssh/shosts.equiv:

# Format: hostname [username]
backup-server.example.com deployer
monitoring.example.com monitor
ci-runner-01.example.com jenkins

sudo chmod 600 /etc/ssh/shosts.equiv
sudo chown root:root /etc/ssh/shosts.equiv

For per-user trust, use ~/.shosts with the same format.

On the client

Edit /etc/ssh/ssh_config:

HostbasedAuthentication yes
EnableSSHKeysign yes
PreferredAuthentications hostbased,publickey,password

ssh-keysign must be setuid root to access host keys:

sudo chmod 4711 /usr/lib/openssh/ssh-keysign
# or
sudo chmod 4711 /usr/libexec/openssh/ssh-keysign

Exchange the host keys

On the client, get the host public key:

sudo cat /etc/ssh/ssh_host_ed25519_key.pub

Add it to /etc/ssh/ssh_known_hosts on the server:

# Format: hostname,ip key-type public-key
backup-server.example.com,192.168.1.10 ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAILo...

sudo chmod 644 /etc/ssh/ssh_known_hosts
sudo chown root:root /etc/ssh/ssh_known_hosts

Then test. Authentication succeeded (hostbased) in the verbose output is what you’re looking for:

ssh -v deployer@production-server.example.com
# Look for: "Authentication succeeded (hostbased)"

Beyond two or three clients, collecting those keys by hand stops being reasonable. Save this on the server:

#!/bin/bash
KNOWN_HOSTS="/etc/ssh/ssh_known_hosts"
TEMP_KEYS="/tmp/host_keys_collection.txt"

CLIENTS=(
    "backup-server.example.com"
    "monitoring.example.com"
    "ci-runner-01.example.com"
)

> $TEMP_KEYS

for client in "${CLIENTS[@]}"; do
    IP=$(dig +short $client | tail -1)
    KEY=$(ssh-keyscan -t ed25519 $client 2>/dev/null)

    if [ -n "$KEY" ]; then
        echo "$client,$IP $(echo $KEY | awk '{print $2, $3}')" >> $TEMP_KEYS
    else
        echo "Failed to get key from $client"
    fi
done

[ -f $KNOWN_HOSTS ] && cp $KNOWN_HOSTS ${KNOWN_HOSTS}.backup.$(date +%F)

cat $TEMP_KEYS >> $KNOWN_HOSTS
sort -u $KNOWN_HOSTS -o $KNOWN_HOSTS
chmod 644 $KNOWN_HOSTS

sudo chmod +x /usr/local/bin/distribute-host-keys.sh
sudo /usr/local/bin/distribute-host-keys.sh

What it looks like in practice

A backup server pulling from production. On the production servers:

HostbasedAuthentication yes
Match User backup
    HostbasedAuthentication yes
    PasswordAuthentication no

backup-server.example.com backup

On the backup server:

Host prod-*
    HostbasedAuthentication yes
    PreferredAuthentications hostbased
    User backup

Now the backup server can pull automatically, with no key and no password anywhere in the job:

rsync -avz prod-web-01:/var/www/ /backup/web-01/

Combining with user key auth is the safest configuration, because a compromised client host alone is then not enough:

AuthenticationMethods publickey,hostbased

Review /etc/ssh/shosts.equiv monthly, keep LogLevel VERBOSE so host-based authentications actually appear in the logs, and restrict SSH by firewall to the trusted client IPs.

Revoking access is two removals and a restart — miss either one and the trust survives:

# Remove from shosts.equiv
sudo nano /etc/ssh/shosts.equiv

# Remove from known_hosts
sudo ssh-keygen -R hostname.example.com -f /etc/ssh/ssh_known_hosts

sudo systemctl restart sshd

Watching it

Fail2ban monitors logs and blocks IPs that show malicious behavior.

# Ubuntu/Debian
sudo apt install fail2ban

# RHEL/CentOS
sudo yum install epel-release && sudo yum install fail2ban

Create /etc/fail2ban/jail.local. The port here has to match the one you set in sshd_config, or the jail watches a port nothing is listening on:

1
[DEFAULT]
2
bantime = 3600
3
findtime = 600
4
maxretry = 3
5
destemail = your_email@example.com
6
sendername = Fail2Ban
7
action = %(action_mwl)s
8

9
[sshd]
10
enabled = true
11
port = 2222
12
filter = sshd
13
logpath = /var/log/auth.log
14
# logpath = /var/log/secure  # RHEL/CentOS
15
maxretry = 3
16
bantime = 3600

sudo systemctl enable fail2ban
sudo systemctl start fail2ban

# Status and management
sudo fail2ban-client status sshd
sudo fail2ban-client get sshd banned
sudo fail2ban-client set sshd unbanip 192.168.1.100

Your own SSH config

On your local machine, ~/.ssh/config saves typing the port and key on every connection:

Host production-server
    HostName server_ip
    Port 2222
    User deployer
    IdentityFile ~/.ssh/id_prod_server
    ServerAliveInterval 60
    ServerAliveCountMax 3

Host staging-server
    HostName staging_ip
    Port 2222
    User deployer
    IdentityFile ~/.ssh/id_staging_server
    ProxyJump bastion-host

ssh production-server

Reading the logs

Directly, when you want to know what happened:

# Real-time
sudo tail -f /var/log/auth.log       # Ubuntu/Debian
sudo tail -f /var/log/secure         # RHEL/CentOS

# Failed attempts
sudo grep "Failed password" /var/log/auth.log | tail -20

# Successful logins
sudo grep "Accepted publickey" /var/log/auth.log | tail -20

And a report worth having in your inbox rather than one you have to remember to run:

#!/bin/bash
LOG_FILE="/var/log/auth.log"
REPORT_FILE="/var/log/ssh-security-report.txt"

echo "SSH Security Report - $(date)" > $REPORT_FILE
echo "================================" >> $REPORT_FILE
echo "" >> $REPORT_FILE

echo "Failed Login Attempts:" >> $REPORT_FILE
grep "Failed password" $LOG_FILE | awk '{print $1, $2, $3, $11}' | sort | uniq -c | sort -nr | head -20 >> $REPORT_FILE
echo "" >> $REPORT_FILE

echo "Successful Logins:" >> $REPORT_FILE
grep "Accepted publickey" $LOG_FILE | awk '{print $1, $2, $3, $9, $11}' | tail -20 >> $REPORT_FILE
echo "" >> $REPORT_FILE

echo "Active SSH Sessions:" >> $REPORT_FILE
who >> $REPORT_FILE
echo "" >> $REPORT_FILE

echo "Current Fail2Ban Bans:" >> $REPORT_FILE
fail2ban-client status sshd 2>/dev/null >> $REPORT_FILE

cat $REPORT_FILE

sudo chmod +x /usr/local/bin/ssh-monitor.sh

# Daily email report
echo "0 9 * * * /usr/local/bin/ssh-monitor.sh | mail -s 'SSH Security Report' your_email@example.com" | sudo crontab -

When it breaks

Can’t connect at all after a config change. Check the daemon is running, the firewall allows the new port, and the config parses:

sudo systemctl status sshd
sudo ufw status           # or: firewall-cmd --list-all
sudo sshd -t
sudo journalctl -u sshd -n 50

Permission denied (publickey). Almost always file permissions — the same ones from the first section. ~/.ssh must be 700, authorized_keys 600, private keys 600:

ls -la ~/.ssh
# .ssh: 700 | authorized_keys: 600 | private keys: 600

chmod 700 ~/.ssh
chmod 600 ~/.ssh/authorized_keys

Too many authentication failures. Your agent is offering every key it holds and the server cuts you off before it reaches the right one:

ssh-add -D
ssh-add ~/.ssh/id_prod_server

# Or force a specific key
ssh -o IdentitiesOnly=yes -i ~/.ssh/id_prod_server user@server

The 2FA code is rejected. TOTP is clock-based, so the server drifting is enough:

timedatectl status
sudo systemctl restart chrony   # or ntpd

Host-based auth silently falls back to a password prompt. Three usual causes, in the order worth checking:

# Server logs
sudo journalctl -u sshd -n 50 | grep hostbased

# Verify hostname resolution
hostname -f  # must match what's in shosts.equiv

# Check ssh-keysign permissions
ls -l /usr/lib/openssh/ssh-keysign
# Should be: -rws--x--x (4711)

# Verify host key on server
sudo grep "$(hostname)" /etc/ssh/ssh_known_hosts

# Full debug from client
ssh -vvv -o PreferredAuthentications=hostbased user@server

Keeping it that way

Once a month:

# Review authorized_keys
cat ~/.ssh/authorized_keys

# Check for weak host keys
for key in /etc/ssh/ssh_host_*_key.pub; do ssh-keygen -lf $key; done

# Anomalies in auth logs
sudo grep -i "POSSIBLE BREAK-IN" /var/log/auth.log

# Users with empty passwords
sudo awk -F: '($2 == "") {print $1}' /etc/shadow

I rotate SSH keys annually: generate the new pair, deploy it everywhere, test, then remove the old public key. The removal is the step people skip, and a key you forgot to revoke is indistinguishable from one you never had.

The auth log that started this still fills up. The difference is that 50,000 attempts against key-only auth on a non-standard port is a graph, not an incident.