Built to be rebuilt.

Nothing in the rack answers the internet. One UDP port is forwarded, for Tailscale’s direct WireGuard connections, and it ignores anything without a key. Every public name arrives through a tunnel that dials out.

Standalone hosts, joined through Proxmox Datacenter Manager. One Kubernetes node each, on three different chassis. The R730xd that used to carry most of this is now the vault — off most of the day.

Everything that changes lives on a two-disk Synology: the film library, my files, my photos, and the landing spot of every backup. The cluster keeps only its own app volumes, on Longhorn, three replicas.

Talos has no shell and no package manager — it is configured by API and by nothing else. Flux watches the repository and makes the cluster match it. There is no third way to change anything, which is what makes the rebuild time believable.

kubernetes/apps/
├── default/         jellyfin, the *arr stack, n8n
├── kube-system/     Cilium, CoreDNS, Intel GPU plugin
├── network/         gateway, Cloudflare tunnel, netboot.xyz
├── observability/   Prometheus, Grafana, Loki, Alloy
└── storage/         Longhorn

Media and automation in the cluster, files and photos on the NAS, identity and remote access on the VPS. Everything with data behind it is declared in the repo.

Alertmanager, Flux, Proxmox and TrueNAS go straight to Telegram — critical and warning only, tagged by layer. Every scheduled job also pings healthchecks.io on its way out, so a job that stops running raises its own hand. Nobody has to notice.

A message gets read once, though, and then it scrolls away. So there is one page for looking on purpose: sixteen numbers that sit neutral on a good day and turn amber or red only when one needs me, and below them every console in the house, each with a dot that is its own health check. It answers only on the tailnet.

The first evening its Backups card had real data, it showed a job down: the monthly restore drill had been failing since August — Authentik one month, Joplin the next — while both dumps were intact. The drill was racing Postgres’s own start-up. The red in this picture is just as real: one night’s Longhorn backup failed, and the card stays that colour until a good one replaces it.

One of the four warnings never clears, and it is honest: if pve-1 died, the other two nodes could not hold everything it runs. I know, and I chose three small machines anyway.

Every producer writes the one thing it owns to the NAS. Once a day the vault wakes, pulls the NAS read-only, snapshots it, pushes it to Oracle, and powers itself off. Nothing holds a credential into the vault, and Oracle’s restic server is append-only — its retention runs on the VPS, never from home.

flowchart LR
  lh["Longhorn · Garage"] --> nas["Synology NAS
latest only"]
  pf["pfSense config"] --> nas
  vps["VPS services"] --> nas
  files["Files · Photos"] --> nas
  nas -- "daily, pulled read-only" --> vault["R730xd vault
30 days + 12 months"]
  vault -- "daily, append-only" --> oc["Oracle Cloud
cannot be pruned from home"]

A new producer gets both further copies for free by writing to the NAS. A compromised one can overwrite its own latest copy there and nothing further: the vault keeps the old version beside the new, and a pull that would delete more than 500 files stops before it snapshots anything.

Losing any single copy is a non-event. On the first of every month the vault restores a sample from Oracle and compares checksums, rather than trusting that it could.

cluster restore

~35 min

onto different hardware, from git and object storage

VPS rebuild

~15 min

from nothing

found by the drill

4 + 1

tooling bugs, and one library excluded from every restore

Cluster restored onto different hardware. Drilled on 29 August, then done for real on 1 September — from git and object storage, in about thirty-five minutes. The VPS rebuilds from nothing in about fifteen.

The drill is the point. It found four tooling bugs and one library that had been quietly excluded from every restore for months — none of which a backup report would have shown.

Everything above is in one repository, and inside.merox.dev shows it running.