Built to be rebuilt.

No way in.

Not one port is forwarded. Nothing in the rack answers the internet directly, and the two paths that do exist both dial outward: a mesh VPN, and a tunnel that opens itself.

pfSense

Firewall and gateway on an XCY X44. Physical segmentation — one NIC per segment, not VLAN tags.

Tailscale

A subnet router puts the LAN on the tailnet, so the rack and the cloud VPS share one flat mesh. This is the way in.

Cloudflare Tunnel

Outbound-only, three routes. A router that does not exist on a port cannot be reached through it.

Unbound

Recursive resolution, no upstream forwarder, with overrides for the internal names.

Three machines.
Three jobs.

Standalone hosts, joined through Proxmox Datacenter Manager and never corosync. One Kubernetes node each — so the three etcd votes sit in three different chassis, and losing a machine costs a workload rather than the cluster.

Six terabytes,
mostly asleep.

Twelve SAS disks hold the library. They park whenever nothing is reading them — forty to fifty watts that would otherwise be spent spinning platters for nobody, in a room somebody sleeps in.

Waking them used to stall etcd's write-ahead log on the shared HBA. The fix was not more hardware — it was raising the election timeout from one second to five, and parking the pool outside the backup window.

git push is the deploy.

Talos has no shell and no package manager — it is configured by API and by nothing else. Flux watches the repository and makes the cluster match it. There is no third way to change anything, which is what makes the rebuild time believable.

kubernetes/apps/
├── default/         jellyfin, immich, the *arr stack, n8n
├── kube-system/     Cilium, CoreDNS, NFS CSI, Intel GPU plugin
├── network/         gateway, Cloudflare tunnel, netboot.xyz
├── observability/   Prometheus, Grafana, Loki, Alloy
└── storage/         Longhorn

What it actually runs.

Disk-bound things in the cluster, identity and remote access on the VPS. Everything with data behind it is declared in the repo.

Silence is the alarm.

Alertmanager, Flux and Proxmox go straight to Telegram — critical and warning only, tagged by layer. Every scheduled job also pings healthchecks.io on its way out, so a job that stops running raises its own hand. Nobody has to notice.

Three copies.
It can delete none of them.

The machine holding the data can add to both off-site targets and remove from neither. Oracle runs an append-only restic server. The NAS pulls with a read-only key and leaves no credential behind it. Retention runs on each target, not on the source.

Longhorn (cluster)  ──nightly──▶  Garage S3 on the R730xd
Oracle VPS          ──nightly──▶  R730xd
pfSense config      ──────────▶  R730xd
                                     │
                     ┌───────────────┴───────────────┐
                     │ weekly, pulled                │ nightly, append-only
                     ▼                               ▼
               Synology                        Oracle Cloud
               asleep between                  cannot be pruned
               wake windows                    from home

Losing any single copy is a non-event. That is the whole point of the arrangement, and it is checked by restoring a five-percent sample every week rather than by trusting it.

71min

Cluster restored onto different hardware. Drilled on 29 August, then done for real on 1 September — from git and object storage, in seventy-one minutes. The VPS rebuilds from nothing in about fifteen.

The drill is the point. It found four tooling bugs and one library that had been quietly excluded from every restore for months — none of which a backup report would have shown.

All of it, in order.

Hypervisor

Proxmox VE on three standalone hosts, joined by Proxmox Datacenter Manager. No corosync, no HA — deliberate.

Kubernetes

Talos Linux, three control planes, one per physical machine. Flux reconciles from GitHub.

Networking

pfSense on an XCY X44. Physical segmentation, one NIC per segment. Cilium for CNI, Gateway API and L2 announcement.

Storage

ZFS. media = 12 × SAS in 2 × raidz2, parked when idle. rpool = SSD mirror. cluster-storage = NVMe. NFS to the cluster nodes only, ACL per client IP.

Off-site

Oracle Cloud Free Tier, Ampere A1 — 4 vCPU ARM, 24 GB, 200 GB. Traefik, Authentik, Guacamole, Joplin, Pi-hole, the public dashboard and an append-only restic server.

Cold copy

Synology DS223+. Pulls weekly over a read-only rrsync key, asleep the rest of the week.

Power

CyberPower VP700ELCD, monitored by NUT. ~165 W total, ~110 W of it the R730xd. Roughly 119 kWh a month.

Noise

Fans off iDRAC — a daemon holds a 1680 RPM floor, about 18 dB under stock.

Secrets

SOPS/age for Kubernetes, Ansible Vault for the VPS. Three keys cannot be lost; nothing else is irreplaceable.