Restoring from Longhorn Backups

About 30 minutes end to end.

You’ll need: kubectl access, Longhorn installed and connected to your S3 backend, and the Longhorn UI reachable.

Step 1 — Look before you touch

#

kubectl get deployments -A
kubectl get statefulsets -A
kubectl get pvc -A -o wide
kubectl get storageclass

You want the exact PVC names before you delete any of them. StatefulSet PVCs in particular are generated, not authored, and prometheus-kube-prometheus-stack-db-prometheus-kube-prometheus-stack-0 is not a name you’ll reconstruct from memory.

Step 2 — Scale everything down

#

Two controllers will undo a manual scale if you let them. Flux reconciles the apps back to their manifests, so suspend their HelmReleases for the duration of the restore:

flux suspend helmrelease jellyfin qbittorrent sonarr -n default
flux suspend helmrelease grafana loki kube-prometheus-stack -n observability

And prometheus-operator owns the Prometheus and Alertmanager StatefulSets — scaling those directly gets reverted within seconds. Set the replicas on their custom resources instead:

kubectl scale deployment jellyfin qbittorrent sonarr --replicas=0 -n default
kubectl scale deployment grafana --replicas=0 -n observability
kubectl scale statefulset loki --replicas=0 -n observability

kubectl get prometheus,alertmanager -n observability   # the resource names
kubectl patch prometheus <name> -n observability --type merge -p '{"spec":{"replicas":0}}'
kubectl patch alertmanager <name> -n observability --type merge -p '{"spec":{"replicas":0}}'

Wait for the pods to actually terminate. --replicas=0 returns immediately; the pods do not.

Step 3 — Delete the empty PVCs

#

They hold nothing, and they’re what’s standing between the restored volumes and the names the apps expect:

kubectl delete pvc jellyfin -n default
kubectl delete pvc qbittorrent -n default
kubectl delete pvc sonarr -n default

kubectl delete pvc grafana -n observability
kubectl delete pvc storage-loki-0 -n observability
kubectl delete pvc prometheus-kube-prometheus-stack-db-prometheus-kube-prometheus-stack-0 -n observability

Step 4 — Restore the volumes

#

Longhorn UI → Backup tab → the ⟲ restore button on each backup:

• Name: <app>-restored, e.g. jellyfin-restored
• Storage Class: longhorn
• Access Mode: ReadWriteOnce

The name matters — it becomes the volumeHandle in the next step, and there’s no rename afterwards.

Step 5 — A PV per restored volume

#

Adjust storage and volumeHandle for each app:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

# Repeat for each application
apiVersion: v1
kind: PersistentVolume
metadata:
  name: jellyfin-restored-pv
spec:
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: longhorn
  csi:
    driver: driver.longhorn.io
    fsType: ext4
    volumeAttributes:
      numberOfReplicas: "3"
      staleReplicaTimeout: "30"
    volumeHandle: jellyfin-restored

persistentVolumeReclaimPolicy: Retain is deliberate. On Delete, removing the PVC again during the same session takes the restored data with it.

Step 6 — A PVC bound to it by name

#

volumeName is what stops the scheduler from helpfully provisioning a fresh empty volume instead:

# Repeat for each application
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: jellyfin
  namespace: default
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
  storageClassName: longhorn
  volumeName: jellyfin-restored-pv

The PVC name has to be the original one the app asks for, not jellyfin-restored.

Step 7 — Check the binding, then scale up

#

kubectl get pvc -n default | grep -E "(jellyfin|qbittorrent|sonarr)"
kubectl get pvc -n observability | grep -E "(grafana|storage-loki|prometheus)"
kubectl get volumes.longhorn.io -n longhorn-system | grep restored

Every PVC Bound, every Longhorn volume attached and healthy. Anything still Pending is a name mismatch between the PVC and the PV, and scaling up on top of it will only bury the error under a CrashLoopBackOff:

Resuming the HelmReleases scales the apps back to what their manifests say, and restores the Prometheus and Alertmanager replica counts with them:

flux resume helmrelease jellyfin qbittorrent sonarr -n default
flux resume helmrelease grafana loki kube-prometheus-stack -n observability

kubectl get pods -A | grep -v Running | grep -v Completed
kubectl get pods -n default -o wide
kubectl get pods -n observability -o wide

Without the UI

#

The same restore as a CRD, if the UI is down or you’re scripting it:

apiVersion: longhorn.io/v1beta2
kind: Volume
metadata:
  name: jellyfin-restored
  namespace: longhorn-system
spec:
  size: "10737418240"
  numberOfReplicas: 3
  frontend: blockdev
  fromBackup: "s3://longhorn@us-east-1/?backup=backup-<id>&volume=jellyfin"

The fromBackup URL is the backup target plus the backup and volume names; the Longhorn UI shows it on the backup’s detail page.

The seven steps above are what the automation replaced. Writing them out by hand once is still worth it — when the automated restore fails, this is the layer you drop back to, and half an hour of kubectl beats reconstructing the order under pressure.