Restoring from Longhorn Backups

I redeployed my K8s cluster with Flux and got back a perfectly healthy cluster with nothing in it. Every app running, every volume empty — no configs, no dashboards, no media metadata. This is what it took to get six applications back from Longhorn backups.

About 30 minutes end to end.

You’ll need: kubectl access, Longhorn installed and connected to your S3 backend, and the Longhorn UI reachable.

Step 1 — Look before you touch

kubectl get deployments -A
kubectl get statefulsets -A
kubectl get pvc -A -o wide
kubectl get storageclass

You want the exact PVC names before you delete any of them. StatefulSet PVCs in particular are generated, not authored, and prometheus-kube-prometheus-stack-db-prometheus-kube-prometheus-stack-0 is not a name you’ll reconstruct from memory.

Step 2 — Scale everything down

kubectl scale deployment jellyfin --replicas=0 -n default
kubectl scale deployment qbittorrent --replicas=0 -n default
kubectl scale deployment sonarr --replicas=0 -n default
kubectl scale deployment grafana --replicas=0 -n observability

kubectl scale statefulset loki --replicas=0 -n observability
kubectl scale statefulset prometheus-kube-prometheus-stack --replicas=0 -n observability
kubectl scale statefulset alertmanager-kube-prometheus-stack --replicas=0 -n observability

Wait for the pods to actually terminate. --replicas=0 returns immediately; the pods do not.

Step 3 — Delete the empty PVCs

They hold nothing, and they’re what’s standing between the restored volumes and the names the apps expect:

kubectl delete pvc jellyfin -n default
kubectl delete pvc qbittorrent -n default
kubectl delete pvc sonarr -n default

kubectl delete pvc grafana -n observability
kubectl delete pvc storage-loki-0 -n observability
kubectl delete pvc prometheus-kube-prometheus-stack-db-prometheus-kube-prometheus-stack-0 -n observability

Step 4 — Restore the volumes

Longhorn UI → Backup tab → the ⟲ restore button on each backup:

• Name: <app>-restored, e.g. jellyfin-restored
• Storage Class: longhorn
• Access Mode: ReadWriteOnce

The name matters — it becomes the volumeHandle in the next step, and there’s no rename afterwards.

Step 5 — A PV per restored volume

Adjust storage and volumeHandle for each app:

1
# Repeat for each application
2
apiVersion: v1
3
kind: PersistentVolume
4
metadata:
5
  name: jellyfin-restored-pv
6
spec:
7
  capacity:
8
    storage: 10Gi
9
  accessModes:
10
    - ReadWriteOnce
11
  persistentVolumeReclaimPolicy: Retain
12
  storageClassName: longhorn
13
  csi:
14
    driver: driver.longhorn.io
15
    fsType: ext4
16
    volumeAttributes:
17
      numberOfReplicas: "3"
18
      staleReplicaTimeout: "30"
19
    volumeHandle: jellyfin-restored

persistentVolumeReclaimPolicy: Retain is deliberate. On Delete, removing the PVC again during the same session takes the restored data with it.

Step 6 — A PVC bound to it by name

volumeName is what stops the scheduler from helpfully provisioning a fresh empty volume instead:

1
# Repeat for each application
2
apiVersion: v1
3
kind: PersistentVolumeClaim
4
metadata:
5
  name: jellyfin
6
  namespace: default
7
spec:
8
  accessModes:
9
    - ReadWriteOnce
10
  resources:
11
    requests:
12
      storage: 10Gi
13
  storageClassName: longhorn
14
  volumeName: jellyfin-restored-pv

The PVC name has to be the original one the app asks for, not jellyfin-restored.

Step 7 — Check the binding, then scale up

kubectl get pvc -n default | grep -E "(jellyfin|qbittorrent|sonarr)"
kubectl get pvc -n observability | grep -E "(grafana|storage-loki|prometheus)"
kubectl get volumes -n longhorn-system | grep "restored"

Every PVC Bound, every Longhorn volume attached and healthy. Anything still Pending is a name mismatch between the PVC and the PV, and scaling up on top of it will only bury the error under a CrashLoopBackOff:

kubectl scale deployment jellyfin --replicas=1 -n default
kubectl scale deployment qbittorrent --replicas=1 -n default
kubectl scale deployment sonarr --replicas=1 -n default
kubectl scale deployment grafana --replicas=1 -n observability

kubectl scale statefulset loki --replicas=1 -n observability
kubectl scale statefulset prometheus-kube-prometheus-stack --replicas=1 -n observability
kubectl scale statefulset alertmanager-kube-prometheus-stack --replicas=1 -n observability

kubectl get pods -A | grep -v Running | grep -v Completed
kubectl get pods -n default -o wide
kubectl get pods -n observability -o wide

Without the UI

The same restore as a CRD, if the UI is down or you’re scripting it:

1
apiVersion: longhorn.io/v1beta1
2
kind: Volume
3
metadata:
4
  name: jellyfin-restored
5
  namespace: longhorn-system
6
spec:
7
  size: "10737418240"
8
  restoreVolumeRecurringJob: false
9
  fromBackup: "s3://your-minio-bucket/backups/backup-name"

The seven steps above are what the automation replaced. Writing them out by hand once is still worth it — when the automated restore fails, this is the layer you drop back to, and half an hour of kubectl beats reconstructing the order under pressure.