Safeguarding My Critical Data

Data loss is catastrophic whether you’re running a homelab or managing production infrastructure. This is my personal disaster recovery strategy — what I run, how it’s automated, and how I’d recover from each failure scenario.

Common Backup Mistakes

The ones I see most often: never testing restores, storing backups on the same system as the data, no encryption, no versioning (so ransomware encrypts your backups too), and no offsite copy.

Mitigations I use: PBS snapshot protection (note — users with sufficient permissions can still remove this; see Proxmox Forum), at least one air-gapped or offline copy, MFA on backup systems, and alerting on backup job failures.

Infrastructure Stack

Components

• Primary storage: Synology DS223, 2×2TB in RAID1

• Backup server: Proxmox Backup Server, 4× Intel D3-S4510 SSDs in RAIDz2

• Offsite: Hetzner VPS + StorageBox

• Key management: KeePass on iOS/macOS, storing all encryption keys

3-2-1 in Practice

• 3 copies: original + PBS + Hetzner
• 2 storage types: local NAS and cloud
• 1 offsite: Hetzner StorageBox

Backup Schedule

All three jobs run on Saturday:

1. 02:00 — VM/LXC → PBS: encrypted backup of all virtual machines and containers
2. 06:00 — PBS → Synology: rsync via crontab, RAID1 protection against single drive failure
3. 08:00 — Synology → Hetzner: offsite copy for geographic redundancy

Scripts

PBS to Synology Sync

0 6 * * 6 rsync -av --delete --progress /mnt/datastore/ /mnt/hyperbackup/ >> /var/log/rsync_backup.log 2>&1

Backup to Cloud PBS

#!/bin/bash

# 1) Export token secret as "PBS_PASSWORD"
export PBS_PASSWORD='token-secret-from-PBS'

# 2) Define user@pbs + token
export PBS_USER_STRING='token-id-from-PBS'

# 3) PBS IP/hostname
export PBS_SERVER='PBS-IP'

# 4) Datastore name
export PBS_DATASTORE='DATASTORE_PBS'

# 5) Build complete repository
export PBS_REPOSITORY="${PBS_USER_STRING}@${PBS_SERVER}:${PBS_DATASTORE}"

# 6) Get local server shortname
export PBS_HOSTNAME="$(hostname -s)"

# 7) ENCRYPTION KEY
export PBS_KEYFILE='/root/pbscloud_key.json'

echo "Run pbs backup for $PBS_HOSTNAME ..."

proxmox-backup-client backup \
  srv.pxar:/srv \
  volumes.pxar:/var/lib/docker/volumes \
  netw.pxar:/var/lib/docker/network \
  etc.pxar:/etc \
  scripts.pxar:/usr/local/bin \
  --keyfile /root/pbscloud_key.json \
  --skip-lost-and-found \
  --repository "$PBS_REPOSITORY"

# List existing backups
proxmox-backup-client list --repository "${PBS_REPOSITORY}"

echo "Done."

Restore from PBS

#!/bin/bash

# Global configs
export PBS_PASSWORD='token-secret-from-PBS'
export PBS_USER_STRING='token-id-from-PBS'
export PBS_SERVER='PBS_IP'
export PBS_DATASTORE='DATASTORE_FROM_PBS'
export PBS_KEYFILE='/root/pbscloud_key.json'
export PBS_REPOSITORY="${PBS_USER_STRING}@${PBS_SERVER}:${PBS_DATASTORE}"

# Input parameters
SNAPSHOT_PATH="$1"
ARCHIVE_NAME="$2"
RESTORE_DEST="$3"

# Parameter validation
if [[ -z "$SNAPSHOT_PATH" || -z "$ARCHIVE_NAME" || -z "$RESTORE_DEST" ]]; then
  echo "Usage: $0 <snapshot_path> <archive_name> <destination>"
  echo "Example: $0 \"host/cloud/2025-01-22T15:19:17Z\" srv.pxar /root/restore-srv"
  exit 1
fi

# Create destination if needed
mkdir -p "$RESTORE_DEST"

# Summary display
echo "=== PBS Restore ==="
echo "Snapshot:      $SNAPSHOT_PATH"
echo "Archive:       $ARCHIVE_NAME"
echo "Destination:   $RESTORE_DEST"
echo "Repository:    $PBS_REPOSITORY"
echo "Encryption key $PBS_KEYFILE"
echo "====================="

# Run restore
proxmox-backup-client restore \
  "$SNAPSHOT_PATH" \
  "$ARCHIVE_NAME" \
  "$RESTORE_DEST" \
  --repository "$PBS_REPOSITORY" \
  --keyfile "$PBS_KEYFILE"

EXIT_CODE=$?

if [[ $EXIT_CODE -eq 0 ]]; then
  echo "=== Restore completed successfully! ==="
else
  echo "Restore error (code $EXIT_CODE)."
fi

exit $EXIT_CODE

Disaster Recovery Scenarios

Scenario 1: Synology NAS Failure

Data remains in two places — PBS (RAIDz2) and Hetzner StorageBox. Recovery steps:

1. Replace failed hardware
2. Reconfigure RAID1 on the new NAS
3. Restore HyperBackup schedule (Saturday 08:00)
4. Verify first backup cycle completes successfully

Scenario 2: Hetzner VPS/StorageBox Failure

1. Provision a new VPS
2. Install proxmox-backup-client:
   • Ubuntu: community guide
   • Debian: standard package installation
3. Retrieve /root/pbscloud_key.json from KeePass and place it on the new VPS
4. Deploy backup-pbs.sh and backup-pbs-restore.sh
5. Test backup and restore
6. Restore crontab:

0 2 * * * /usr/local/bin/backup-pbs.sh >> /var/log/backup-cloud.log 2>&1

Scenario 3: PBS Server Failure

1. Install the latest PBS ISO
2. Restore datastore config:

datastore: raidz2
        comment
        gc-schedule sat 03:30
        notification-mode notification-system
        path /mnt/datastore

3. Verify /etc/fstab mount point:

#raidz2
/dev/sdb /mnt/datastore ext4 defaults 0 2

4. Confirm the datastore structure is intact: .chunks, vm, .gc-status, ct, host

5. Data can be pulled from any of three sources:

   • Original RAIDz2 array (if drives survived)
   • Hetzner StorageBox at /mnt/storagebox/Storage_1
   • Synology at /volume1/Backup/Proxmox/hyperbackup

6. Import VM/LXC encryption key from KeePass into the new PVE environment

What to Keep Stored Securely

Always have these accessible from KeePass (iOS/macOS):

• /root/pbscloud_key.json
• PBS VM/LXC encryption key
• PBS config: /etc/proxmox-backup/datastore.cfg
• Backup locations:
  • PBS: /mnt/datastore
  • Synology: /volume1/Backup/Proxmox/hyperbackup
  • Hetzner: /mnt/storagebox/Storage_1

Ongoing Maintenance

Monthly integrity checks on random files, checksum validation for bit rot, and log review for failures. Quarterly test restores with documented timing. Real-time alerting on backup process failures and storage capacity trends.

A backup you’ve never restored from is a backup you don’t actually have.