Built to be rebuilt.
Three Proxmox hosts, one Kubernetes cluster, and nothing that answers the internet. All of it comes back from a git repository and three keys in about thirty-five minutes — which is the only reason it is safe to keep changing.
Overview#
3
Proxmox hosts
Talos
kubernetes
~6 TB
pools
~165 W
draw
0
inbound ports
3
backup copies
Network#
No way in.
Not one port is forwarded. Nothing in the rack answers the internet directly, and the two paths that do exist both dial outward: a mesh VPN, and a tunnel that opens itself.
| pfSense | Firewall and gateway on an XCY X44. Physical segmentation — one NIC per segment, not VLAN tags. |
| Tailscale | A subnet router puts the LAN on the tailnet, so the rack and the cloud VPS share one flat mesh. This is the way in. |
| Cloudflare Tunnel | Outbound-only, three routes. A router that does not exist on a port cannot be reached through it. |
| Unbound | Recursive resolution, no upstream forwarder, with overrides for the internal names. |
Compute#
Three machines. Three jobs.
Standalone hosts, joined through Proxmox Datacenter Manager. One Kubernetes node each, on three different chassis.
Beelink GTi13 Ultra
| CPU | Core i9-13900HK · 20 threads |
| Memory | 64 GB DDR5 |
| Node | kubernetes-1 · 14 cores / 32 GB |
| Why | Iris Xe passed through — every transcode happens here |
Dell PowerEdge R730xd
| CPU | Xeon E5-2630 v4 · 10C/20T |
| Memory | 251 GB DDR4 ECC |
| Node | kubernetes-2 · NFS, Garage S3, Nextcloud |
| Why | Twelve SAS bays, and the backup hub everything drains into |
Dell OptiPlex 3050
| CPU | Core i5-6500T · 4 cores |
| Memory | 32 GB DDR4 |
| Node | kubernetes-3 · one Longhorn replica |
| Why | 960 GB D3-S4510 — power-loss protection, the best etcd disk here |
Storage#
Six terabytes, mostly asleep.
Twelve SAS disks hold the library. They park whenever nothing is reading them — forty to fifty watts that would otherwise be spent spinning platters for nobody, in a room somebody sleeps in.
| Pool | Host | Layout | Used |
|---|---|---|---|
| media | pve-2 | 12 × SAS in two raidz2 vdevs | 1.31 TB of ~4.2 TB |
| rpool | pve-2 | SSD mirror — everything touched daily | 159 GB of 861 GB |
| cluster-storage | pve-1 | Single NVMe, one Longhorn replica | 105 GB of 899 GB |
Waking the disks used to stall etcd’s write-ahead log on the shared HBA. The fix was not more hardware — it was raising the election timeout from one second to five, and parking the pool outside the backup window. The whole story is in Spinning Down SAS Disks.
Kubernetes#
git push is the deploy.
Talos has no shell and no package manager — it is configured by API and by nothing else. Flux watches the repository and makes the cluster match it. There is no third way to change anything, which is what makes the rebuild time believable.
kubernetes/apps/
├── default/ jellyfin, immich, the *arr stack, n8n
├── kube-system/ Cilium, CoreDNS, NFS CSI, Intel GPU plugin
├── network/ gateway, Cloudflare tunnel, netboot.xyz
├── observability/ Prometheus, Grafana, Loki, Alloy
└── storage/ Longhorn
Services#
What it actually runs.
Disk-bound things in the cluster, identity and remote access on the VPS. Everything with data behind it is declared in the repo.
Media
- Jellyfin: 4K, hardware transcoded
- Jellyseerr: requests, the one route that faces out
- Radarr · Sonarr: film and TV automation
- Prowlarr · qBittorrent: indexing and download, VPN-routed
Files & photos
- Immich: photo library, own Postgres
- Nextcloud: files and office, AIO on its own VM
- Collabora: documents edited in the browser
- Joplin: notes sync
Platform
- Talos + Flux: the cluster and what reconciles it
- Longhorn: block storage, three replicas
- Cilium: eBPF CNI, Gateway API, L2
- cert-manager: certificates, renewed without asking
Edge & identity
- Traefik: reverse proxy on the VPS
- Authentik: single sign-on
- Guacamole: remote desktop in a tab
- Pi-hole + Unbound: filtering DNS, own resolver
Watching it
- Prometheus: metrics, hosts and cluster
- Grafana: dashboards
- Loki + Alloy: logs
- Alertmanager: straight to Telegram
Odds
- n8n: one workflow, the daily news digest
- Headlamp: Kubernetes, when a UI is faster
- Portainer: the VPS containers
- netboot.xyz: installers over the network
Alerting#
Silence is the alarm.
Alertmanager, Flux and Proxmox go straight to Telegram — critical and warning only, tagged by layer. Every scheduled job also pings healthchecks.io on its way out, so a job that stops running raises its own hand. Nobody has to notice.
Backup#
Three copies. It can delete none of them.
The machine holding the data can add to both off-site targets and remove from neither. Oracle runs an append-only restic server. The NAS pulls with a read-only key and leaves no credential behind it. Retention runs on each target, not on the source.
flowchart LR lh["Longhorn · cluster"] -- nightly --> hub["R730xd
Garage S3"] vps["Oracle VPS"] -- nightly --> hub pf["pfSense config"] --> hub hub -- "weekly, pulled" --> nas["Synology
asleep between wake windows"] hub -- "nightly, append-only" --> oc["Oracle Cloud
cannot be pruned from home"]
Losing any single copy is a non-event. That is the whole point of the arrangement, and it is checked by restoring a five-percent sample every week rather than by trusting it.
Recovery#
Drilled, then done for real.
~35 min
cluster restore
~15 min
VPS rebuild
4 + 1
found by the drill
Cluster restored onto different hardware. Drilled on 29 August, then done for real on 1 September — from git and object storage, in about thirty-five minutes. The VPS rebuilds from nothing in about fifteen.
The drill is the point. It found four tooling bugs and one library that had been quietly excluded from every restore for months — none of which a backup report would have shown.
Questions#
The ones every rack post gets.
Why no Proxmox cluster?
What does it draw?
Is the R730xd loud?
Tech specs#
All of it, in order.
| Hypervisor | Proxmox VE on three standalone hosts, joined by Proxmox Datacenter Manager. No corosync, no HA — deliberate. |
| Kubernetes | Talos Linux, three control planes, one per physical machine. Flux reconciles from GitHub. |
| Networking | pfSense on an XCY X44. Physical segmentation, one NIC per segment. Cilium for CNI, Gateway API and L2 announcement. |
| Storage | ZFS. media = 12 × SAS in 2 × raidz2, parked when idle. rpool = SSD mirror. cluster-storage = NVMe. NFS to the cluster nodes only, ACL per client IP. |
| Off-site | Oracle Cloud Free Tier, Ampere A1 — 4 vCPU ARM, 24 GB, 200 GB. Traefik, Authentik, Guacamole, Joplin, Pi-hole, the public dashboard and an append-only restic server. |
| Cold copy | Synology DS223+. Pulls weekly over a read-only rrsync key, asleep the rest of the week. |
| Power | CyberPower VP700ELCD, monitored by NUT. ~165 W total, ~110 W of it the R730xd. Roughly 119 kWh a month. |
| Noise | Fans off iDRAC — a daemon holds a 1680 RPM floor, about 18 dB under stock. |
| Secrets | SOPS/age for Kubernetes, Ansible Vault for the VPS. Three keys cannot be lost; nothing else is irreplaceable. |
Everything above is in one repository, and inside.merox.dev shows it running.