↓ Skip to main content
merox

Migrating Longhorn Backup from MinIO to Garage

·5 mins
Table of Contents
MinIO stopped making sense for homelab use: Docker images dropped, source-only distribution, licensing drift. What that leaves behind is a lot of self-hosted installs frozen on an old version with known CVEs and no clean upgrade path. Mine was one of them, holding every Longhorn backup I have.

I replaced it with Garage — lightweight, actively maintained, S3-compatible, and with none of that baggage. Longhorn only ever sees an S3 endpoint, so the swap is a bucket, a secret and one Helm value.

You’ll need: an Ubuntu host with Docker and Compose, and a cluster with Longhorn already installed. Check the Garage releases for the current version before you pin one.

Setting up Garage
#

Two secrets, generated up front — the RPC secret authenticates nodes to each other, the admin token guards the admin API the WebUI talks to:

sudo mkdir -p /srv/docker/garage/{meta,data}
cd /srv/docker/garage

# Generate RPC secret
openssl rand -hex 32

# Generate admin token
openssl rand -hex 32
garage.toml
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

metadata_dir = "/var/lib/garage/meta"
data_dir = "/var/lib/garage/data"
db_engine = "lmdb"

replication_factor = 1

rpc_bind_addr = "0.0.0.0:3901"
rpc_public_addr = "127.0.0.1:3901"
rpc_secret = "YOUR_RPC_SECRET_HERE"

[s3_api]
s3_region = "us-east-1"
api_bind_addr = "0.0.0.0:3900"
root_domain = ".s3.garage"

[admin]
api_bind_addr = "0.0.0.0:3903"
admin_token = "YOUR_ADMIN_TOKEN_HERE"

replication_factor = 1 is a single copy on a single node. That’s correct here — this bucket is one leg of a backup chain, not the backup itself. If Garage is the only place your data exists, use 3 and more than one node.

Compose
#

The WebUI is optional, but it’s the fastest way to see whether a backup actually landed:

compose.yaml
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

services:
  garage:
    image: dxflrs/garage:v2.1.0
    container_name: garage
    network_mode: "host"
    restart: unless-stopped
    volumes:
      - ./garage.toml:/etc/garage.toml
      - ./meta:/var/lib/garage/meta
      - ./data:/var/lib/garage/data

  webui:
    image: khairul169/garage-webui:latest
    container_name: garage-webui
    restart: unless-stopped
    volumes:
      - ./garage.toml:/etc/garage.toml:ro
    environment:
      API_BASE_URL: "http://127.0.0.1:3903"
      S3_ENDPOINT_URL: "http://127.0.0.1:3900"
    network_mode: "host"
docker compose up -d

Layout, bucket, key
#

A fresh Garage node holds no data until you assign it capacity and apply a layout. Skip layout apply and every write fails with an error that doesn’t mention layouts:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

alias garage="docker exec -ti garage /garage"

garage node id

# Assign storage capacity
garage layout assign <node-id> -z default -c 100G

garage layout show
garage layout apply --version 1

# Create bucket and key
garage bucket create longhorn
garage key create longhorn-key
garage bucket allow longhorn --read --write --owner --key longhorn-key

# Note the Key ID and Secret key
garage key info longhorn-key --show-secret

key create already printed the secret; key info --show-secret shows it again whenever you need it. It goes into the Kubernetes secret next.

Pointing Longhorn at it
#

The secret keeps the name minio-secret. Renaming it means editing the HelmRelease too, and there’s nothing to gain from making both changes at the same time:

minio-secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: minio-secret
  namespace: longhorn-system
type: Opaque
stringData:
  AWS_ACCESS_KEY_ID: <Key-ID-from-garage>
  AWS_SECRET_ACCESS_KEY: <Secret-Key-from-garage>
  AWS_ENDPOINTS: http://<GARAGE_SERVER_IP>:3900
  AWS_REGION: us-east-1
kubectl apply -f minio-secret.yaml

Then the backup target itself:

helmrelease.yaml
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

---
apiVersion: source.toolkit.fluxcd.io/v1
kind: HelmRepository
metadata:
  name: longhorn
  namespace: longhorn-system
spec:
  interval: 1h
  url: https://charts.longhorn.io
---
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: longhorn
  namespace: longhorn-system
spec:
  interval: 1h
  chart:
    spec:
      chart: longhorn
      version: 1.10.0
      sourceRef:
        kind: HelmRepository
        name: longhorn
        namespace: longhorn-system
  values:
    defaultBackupStore:
      backupTarget: "s3://longhorn@us-east-1/"
      backupTargetCredentialSecret: "minio-secret"
      pollInterval: "300"

The target reads s3://<bucket>@<region>/, and the region has to match s3_region in garage.toml.

Warning

Since Longhorn 1.8 the backup target lives under defaultBackupStore. The older defaultSettings.backupTarget, backupTargetCredentialSecret and backupstorePollInterval keys are silently ignored: Helm accepts the values, and Longhorn either shows no backup target or keeps whatever target it had before the upgrade.

Verify
#

Take a real backup — Longhorn UI → Volume → Create Backup — and go looking for it on the Garage side. A backup target that reports healthy while the bucket stays empty is the failure mode worth catching now rather than during a restore:

garage bucket list
garage bucket info longhorn

The WebUI at http://<SERVER_IP>:3909 shows the same thing with object counts.

Restoring works exactly as it did before — the procedure is in Restoring from Longhorn Backups, and nothing in it depends on which S3 implementation is underneath.

Troubleshooting
#

WebUI returns “Unknown Error”. The [admin] section is missing from garage.toml, or the container started before you added it:

docker compose restart
docker logs garage

Port 3903 connection refused. The admin API isn’t bound:

ss -tlnp | grep 3903

Longhorn can’t reach Garage. Test from inside the cluster, not from your laptop — the endpoint has to resolve and route from the pods:

kubectl run -it --rm debug --image=amazon/aws-cli --restart=Never -- \
  s3 ls s3://longhorn --endpoint-url http://<GARAGE_IP>:3900

References
#