Migrating Longhorn Backup from MinIO to Garage

When MinIO recently announced they’re discontinuing Docker images and moving to a source-only distribution model, many homelab users were left scrambling for alternatives. With hundreds of thousands of installations potentially running outdated versions with known CVEs, it was time to find a more reliable solution.

Enter Garage - a self-hosted, S3-compatible, distributed object storage service that’s perfect for Kubernetes backup storage.

Why Garage?

After MinIO’s controversial decision to drop Docker support (Issue #21647), I needed a drop-in S3-compatible replacement that:

• Has active Docker image support
• Is lightweight enough for homelab use
• Provides reliable S3 API compatibility
• Works seamlessly with Longhorn backups

Garage checked all these boxes.

Prerequisites

• Ubuntu server with Docker and Docker Compose
• Kubernetes cluster with Longhorn installed
• Flux CD for GitOps (optional, but recommended)

Setting Up Garage

1. Directory Structure

sudo mkdir -p /srv/docker/garage/{meta,data}
cd /srv/docker/garage

2. Generate Secrets

# Generate RPC secret for internal communication
openssl rand -hex 32

# Generate admin token for WebUI
openssl rand -hex 32

3. Create Configuration

Create /srv/docker/garage/garage.toml:

1
metadata_dir = "/var/lib/garage/meta"
2
data_dir = "/var/lib/garage/data"
3
db_engine = "lmdb"
4

5
replication_factor = 1
6

7
rpc_bind_addr = "0.0.0.0:3901"
8
rpc_public_addr = "127.0.0.1:3901"
9
rpc_secret = "YOUR_RPC_SECRET_HERE"
10

11
[s3_api]
12
s3_region = "us-east-1"
13
api_bind_addr = "0.0.0.0:3900"
14
root_domain = ".s3.garage"
15

16
[admin]
17
api_bind_addr = "0.0.0.0:3903"
18
admin_token = "YOUR_ADMIN_TOKEN_HERE"

4. Docker Compose Setup

Create /srv/docker/garage/docker-compose.yml:

1
version: "3"
2
services:
3
  garage:
4
    image: dxflrs/garage:v2.1.0
5
    container_name: garage
6
    network_mode: "host"
7
    restart: unless-stopped
8
    volumes:
9
      - ./garage.toml:/etc/garage.toml
10
      - ./meta:/var/lib/garage/meta
11
      - ./data:/var/lib/garage/data
12

13
  webui:
14
    image: khairul169/garage-webui:latest
15
    container_name: garage-webui
16
    restart: unless-stopped
17
    volumes:
18
      - ./garage.toml:/etc/garage.toml:ro
19
    environment:
20
      API_BASE_URL: "http://127.0.0.1:3903"
21
      S3_ENDPOINT_URL: "http://127.0.0.1:3900"
22
    network_mode: "host"

5. Start Services

docker-compose up -d

Configuring Garage Storage

1. Initialize Node Layout

# Create alias for easier command execution
alias garage="docker exec -ti garage /garage"

# Get node ID
garage node id

# Assign storage capacity (adjust based on your available space)
garage layout assign <node-id> -z default -c 100G

# Apply layout
garage layout show
garage layout apply --version 1

2. Create Bucket and Credentials

# Create bucket for Longhorn backups
garage bucket create longhorn

# Create access key
garage key create longhorn-key

# Grant permissions
garage bucket allow longhorn --read --write --owner --key longhorn-key

# View credentials (note the Key ID and Secret key)
garage key info longhorn-key --show-secret

Integrating with Longhorn

1. Create Kubernetes Secret

Update your SOPS-encrypted secret or create a new one:

1
apiVersion: v1
2
kind: Secret
3
metadata:
4
  name: minio-secret
5
  namespace: longhorn-system
6
type: Opaque
7
stringData:
8
  AWS_ACCESS_KEY_ID: <Key-ID-from-garage>
9
  AWS_SECRET_ACCESS_KEY: <Secret-Key-from-garage>
10
  AWS_ENDPOINTS: http://<GARAGE_SERVER_IP>:3900
11
  AWS_REGION: us-east-1

Apply the secret:

kubectl apply -f minio-secret.yaml

2. Update Longhorn HelmRelease

1
---
2
apiVersion: source.toolkit.fluxcd.io/v1
3
kind: HelmRepository
4
metadata:
5
  name: longhorn
6
  namespace: longhorn-system
7
spec:
8
  interval: 1h
9
  url: https://charts.longhorn.io
10
---
11
apiVersion: helm.toolkit.fluxcd.io/v2
12
kind: HelmRelease
13
metadata:
14
  name: longhorn
15
spec:
16
  interval: 1h
17
  chart:
18
    spec:
19
      chart: longhorn
20
      version: 1.10.0
21
      sourceRef:
22
        kind: HelmRepository
23
        name: longhorn
24
        namespace: longhorn-system
25
  values:
26
    defaultSettings:
27
      backupTarget: "s3://longhorn@us-east-1/"
28
      backupTargetCredentialSecret: "minio-secret"
29
      backupstorePollInterval: "300"
30
      # ... other settings

Testing the Setup

1. Verify Connectivity

Access the Garage WebUI at http://<SERVER_IP>:3909 to verify the node is connected and healthy.

2. Test Backup from Longhorn

1. Navigate to Longhorn UI → Volume
2. Select a volume → Create Backup
3. Monitor the backup progress
4. Verify the backup appears in Garage WebUI under the longhorn bucket

3. CLI Verification

# List buckets
garage bucket list

# View bucket info
garage bucket info longhorn

Troubleshooting

WebUI Shows “Unknown Error”

Ensure the [admin] section is present in garage.toml and the container is restarted:

docker-compose restart
docker logs garage

Connection Refused on Port 3903

Check if the admin API is listening:

netstat -tlnp | grep 3903

Longhorn Can’t Connect to Garage

Verify network connectivity from Kubernetes cluster:

kubectl run -it --rm debug --image=amazon/aws-cli --restart=Never -- \
  s3 ls s3://longhorn --endpoint-url http://<GARAGE_IP>:3900

Benefits of This Migration

• Active maintenance: Garage continues to provide Docker images and updates
• Lightweight: Lower resource footprint compared to MinIO
• S3 compatible: Drop-in replacement for most S3 workloads
• Security: Regular updates without the concern of discontinued support
• Community-driven: Open-source project with transparent development

Conclusion

Migrating from MinIO to Garage was straightforward and took less than an hour. With MinIO’s shift to source-only distribution leaving many installations vulnerable, Garage provides a reliable, actively-maintained alternative that integrates seamlessly with Longhorn.

The WebUI makes management simple, and the S3 compatibility ensures that existing backup workflows continue working without modification. If you’re running MinIO in a homelab environment, now is the time to consider alternatives before the next CVE drops.

Resources

• Garage Documentation
• Garage WebUI Project
• Longhorn Documentation
• MinIO Docker Discontinuation Issue