Migrating Longhorn Backup from MinIO to Garage

MinIO stopped making sense for homelab use: Docker images dropped, source-only distribution, licensing drift. What that leaves behind is a lot of self-hosted installs frozen on an old version with known CVEs and no clean upgrade path. Mine was one of them, holding every Longhorn backup I have.

I replaced it with Garage — lightweight, actively maintained, S3-compatible, and with none of that baggage. Longhorn only ever sees an S3 endpoint, so the swap is a bucket, a secret and one Helm value.

You’ll need: an Ubuntu host with Docker and Compose, and a cluster with Longhorn already installed. Check the Garage releases for the current version before you pin one.

Setting up Garage

Two secrets, generated up front — the RPC secret authenticates nodes to each other, the admin token guards the admin API the WebUI talks to:

sudo mkdir -p /srv/docker/garage/{meta,data}
cd /srv/docker/garage

# Generate RPC secret
openssl rand -hex 32

# Generate admin token
openssl rand -hex 32

1
metadata_dir = "/var/lib/garage/meta"
2
data_dir = "/var/lib/garage/data"
3
db_engine = "lmdb"
4

5
replication_factor = 1
6

7
rpc_bind_addr = "0.0.0.0:3901"
8
rpc_public_addr = "127.0.0.1:3901"
9
rpc_secret = "YOUR_RPC_SECRET_HERE"
10

11
[s3_api]
12
s3_region = "us-east-1"
13
api_bind_addr = "0.0.0.0:3900"
14
root_domain = ".s3.garage"
15

16
[admin]
17
api_bind_addr = "0.0.0.0:3903"
18
admin_token = "YOUR_ADMIN_TOKEN_HERE"

replication_factor = 1 is a single copy on a single node. That’s correct here — this bucket is one leg of a backup chain, not the backup itself. If Garage is the only place your data exists, use 3 and more than one node.

Compose

The WebUI is optional, but it’s the fastest way to see whether a backup actually landed:

1
version: "3"
2
services:
3
  garage:
4
    image: dxflrs/garage:v2.1.0
5
    container_name: garage
6
    network_mode: "host"
7
    restart: unless-stopped
8
    volumes:
9
      - ./garage.toml:/etc/garage.toml
10
      - ./meta:/var/lib/garage/meta
11
      - ./data:/var/lib/garage/data
12

13
  webui:
14
    image: khairul169/garage-webui:latest
15
    container_name: garage-webui
16
    restart: unless-stopped
17
    volumes:
18
      - ./garage.toml:/etc/garage.toml:ro
19
    environment:
20
      API_BASE_URL: "http://127.0.0.1:3903"
21
      S3_ENDPOINT_URL: "http://127.0.0.1:3900"
22
    network_mode: "host"

docker-compose up -d

Layout, bucket, key

A fresh Garage node holds no data until you assign it capacity and apply a layout. Skip layout apply and every write fails with an error that doesn’t mention layouts:

alias garage="docker exec -ti garage /garage"

garage node id

# Assign storage capacity
garage layout assign <node-id> -z default -c 100G

garage layout show
garage layout apply --version 1

# Create bucket and key
garage bucket create longhorn
garage key create longhorn-key
garage bucket allow longhorn --read --write --owner --key longhorn-key

# Note the Key ID and Secret key
garage key info longhorn-key --show-secret

--show-secret prints the secret once, at creation. Copy it now.

Pointing Longhorn at it

The secret keeps the name minio-secret. Renaming it means editing the HelmRelease too, and there’s nothing to gain from making both changes at the same time:

1
apiVersion: v1
2
kind: Secret
3
metadata:
4
  name: minio-secret
5
  namespace: longhorn-system
6
type: Opaque
7
stringData:
8
  AWS_ACCESS_KEY_ID: <Key-ID-from-garage>
9
  AWS_SECRET_ACCESS_KEY: <Secret-Key-from-garage>
10
  AWS_ENDPOINTS: http://<GARAGE_SERVER_IP>:3900
11
  AWS_REGION: us-east-1

kubectl apply -f minio-secret.yaml

Then the backup target itself:

1
---
2
apiVersion: source.toolkit.fluxcd.io/v1
3
kind: HelmRepository
4
metadata:
5
  name: longhorn
6
  namespace: longhorn-system
7
spec:
8
  interval: 1h
9
  url: https://charts.longhorn.io
10
---
19 collapsed lines
11
apiVersion: helm.toolkit.fluxcd.io/v2
12
kind: HelmRelease
13
metadata:
14
  name: longhorn
15
spec:
16
  interval: 1h
17
  chart:
18
    spec:
19
      chart: longhorn
20
      version: 1.10.0
21
      sourceRef:
22
        kind: HelmRepository
23
        name: longhorn
24
        namespace: longhorn-system
25
  values:
26
    defaultSettings:
27
      backupTarget: "s3://longhorn@us-east-1/"
28
      backupTargetCredentialSecret: "minio-secret"
29
      backupstorePollInterval: "300"

Verify

Take a real backup — Longhorn UI → Volume → Create Backup — and go looking for it on the Garage side. A backup target that reports healthy while the bucket stays empty is the failure mode worth catching now rather than during a restore:

garage bucket list
garage bucket info longhorn

The WebUI at http://<SERVER_IP>:3909 shows the same thing with object counts.

Restoring works exactly as it did before — the procedure is in Restoring from Longhorn Backups, and nothing in it depends on which S3 implementation is underneath.

Troubleshooting

WebUI returns “Unknown Error”. The [admin] section is missing from garage.toml, or the container started before you added it:

docker-compose restart
docker logs garage

Port 3903 connection refused. The admin API isn’t bound:

netstat -tlnp | grep 3903

Longhorn can’t reach Garage. Test from inside the cluster, not from your laptop — the endpoint has to resolve and route from the pods:

kubectl run -it --rm debug --image=amazon/aws-cli --restart=Never -- \
  s3 ls s3://longhorn --endpoint-url http://<GARAGE_IP>:3900

References

• Garage documentation
• Garage WebUI
• Longhorn documentation
• MinIO Docker discontinuation issue