On-Prem & Cloud — My Infrastructure

One Proxmox host, a single-node Talos cluster, a NAS that spends most of the week powered off, and a free Oracle VM in another country holding the last copy of everything. One Tailscale mesh, one Git repo, no inbound ports.

Bottom-up: network, hosts, storage, platform, services, backup, recovery. Every number comes from the running setup, and the places where it bit me are marked as such.

Network

Everything starts here. Which physical port a device is plugged into decides what it is allowed to talk to — no VLAN tags, no software doing the separating.

The split is physical, not tagged — one NIC per segment:

WAN   → Orange ISP (public IP, bridge mode)          igb0
LAN   → 10.57.57.0/24, servers and infrastructure    igb1
OPT1  → 10.57.97.0/24, WiFi + IoT (separate port)    igb2
VLAN  → defined on pfSense, idle — see below

• LAN — flat /24, anything that runs a service: Proxmox hosts, K8s nodes, iDRAC, NAS.
• OPT1 — separate NIC, own subnet, everything wireless: AP, laptops, phones, TVs, IoT.

The VLAN interface exists and does nothing. Proper segmentation needs a managed switch and the CRS326 hasn’t been delivered. Until it lands, physical ports do the isolating.

pfSense

A fanless mini PC (~200€) that has been running pfSense for 3+ years: XCY X44 on AliExpress

• Firewall. LAN→WAN open. OPT1 gets internet and nothing else — named pass rules per device, then a block. Nothing is port-forwarded; every pre-tunnel NAT rule is disabled. Inbound arrives through Cloudflare Tunnel or Tailscale, or not at all. pfBlockerNG on both segments.
• Unbound. Recursive resolver with explicit host overrides — every *.k8s.merox.dev name points at the internal Cilium gateway. No wildcard, no delegation.
• Tailscale subnet router. Advertises the LAN to the tailnet, so Synology, iDRAC and the switch are reachable without a client on them — and the VPS reaches back with no inbound port open.

One flat network everywhere: rack and cloud sit on the same mesh, with no tunnels and no firewall holes between them.

Hosts

One machine does all the work now. The small quiet box that used to watch it is switched off — the last thing keeping it awake turned out to be a USB cable.

Compute

• Dell PowerEdge R730xd (pve) — Xeon E5-2630 v3 (8C/16T), 256GB DDR4 ECC. 4× 960GB SSD mirror (rpool) plus 12× 600GB SAS in 2× RAIDZ2 (media). Carries the K8s node, the NFS server, the backup hub, Ollama and the UPS.
• 3× mini PC — Beelink GTi 13 (i9-13900HK, 64GB DDR5) and 2× Dell OptiPlex Micro 3050 (i5-6500T, 16GB). All powered off. They’re the downsizing plan if the R730xd ever goes — for now it stays, because I like real datacenter hardware.
• Synology DS223+ — ARM RTD1619B, 2GB, 2× 2TB RAID1. Cold backup target, nothing else.

Proxmox

One standalone host, no corosync cluster and now nothing to cluster with. Proxmox Datacenter Manager linked the two hosts while there were two; with the Beelink off, pve’s own web UI is the single pane of glass.

An Nvidia Quadro P2200 is passed through to kubernetes-controlplane-1, replacing the old Beelink/QuickSync setup. nvidia-device-plugin exposes it to K8s; Jellyfin is scheduled on that node.

Power

~136 W for everything on the UPS — server, firewall and switch, with the Synology asleep. The R730xd is 125 W of that, down from 135 W as a three-node cluster.

The disks are the real lever: 179 W all twelve spinning, 135 W all twelve asleep. A 44 W swing, and the reason the Spin-down section below exists.

One CyberPower VP700ELCD carries the R730xd, pfSense and the switch. NUT monitors it over USB; upsmon powers the host down locally at five minutes of remaining runtime, out of a measured twelve at 35 % load. It used to hang off px-0 and reach the server over SSH — the machine holding all the data shouldn’t need a second machine awake to be told to shut down.

Storage

Where the data actually lives. Three ZFS pools with separate jobs — and twelve spinning disks that stay asleep most of the day, because platters nobody is reading were burning a quarter of the rack’s power bill.

Two pools now, each with one job.

rpool went from 451 GB to 248 GB in one afternoon: two K8s VMs deleted, Longhorn from three replicas to one. The third pool, cluster-storage, went off with the Beelink.

The R730xd NFS-exports media to the whole cluster. media/library is deliberately one dataset: the *arr hardlink imports need a same-filesystem rename(), and splitting movies from downloads turns every import into a full copy.

Spin-down

Left spinning, the twelve SAS disks cost 40–50 W — around 400–500 lei (~€90) a year to keep platters turning that nothing is reading. So the pool parks itself:

• A stateless enforcer on a short timer, deciding once for the whole pool: if it moved zero bytes for two consecutive runs, park every disk that’s awake. Nothing is remembered per disk, so a drive woken by a stray smartctl is simply parked again on the next quiet tick.
• A sampler for the graphs, and a nightly drift check that alerts on degradation, not only on outright failure.

Every nightly job that touches the pool is packed into a single window for the same reason: the disks wake once per night, not eight times.

Kubernetes

The part that runs the apps. Nothing here is clicked into place by hand — the cluster’s whole desired state is a Git repository, and a controller called Flux keeps reality matching it.

Talos Linux + FluxCD, starting from onedr0p/cluster-template. Everything below lives in one repo: meroxdotdev/infrastructure.

One control-plane node behind a VIP. No worker nodes — at this size the separation buys nothing.

kubernetes/apps/
├── cert-manager/     # TLS automation
├── default/          # Production workloads
├── flux-system/      # Flux operator + instance + alert routing
├── kube-system/      # Cilium, CoreDNS, NFS CSI, metrics-server
├── network/          # k8s-gateway, Cloudflare tunnel + DNS
├── observability/    # Prometheus, Grafana, Loki
└── storage/          # Longhorn configuration

Networking

Cilium handles CNI, L2 announcement and Gateway API. Two gateways split the cluster by exposure:

• internal — LAN and tailnet only. Everything lands here by default.
• external — behind Cloudflare Tunnel. Three routes are attached to it: Jellyseerr, the n8n webhook endpoint, and the Flux webhook. Everything else, Jellyfin included, is internal-only and reached over Tailscale from outside.

The LoadBalancer pool is the whole LAN range, not a carved-out block — each service pins its address by annotation. Nothing dynamic, no second source of truth to keep in sync with DHCP.

Storage

Longhorn 1.12 is the default StorageClass and backs every PVC, now at one replica. A second nfs-slow class exists via csi-driver-nfs but has no claims — the media stack mounts NFS directly in the pod spec, which keeps the mount options visible in the manifest.

Without replication Longhorn stays for one thing: nightly snapshots to Garage S3, which restic pushes off-site. spegel went — a peer-to-peer image cache needs a peer.

Services

The apps themselves — the things I actually open in a browser. Where each one runs comes down to a single question: does it need to be reachable from outside the house?

On the cluster

Immich runs its own Postgres with VectorChord, and both the library and the database live on SSD — a photo app doing random reads all day would keep the SAS pool awake permanently.

Files, on a VM instead

Documents moved off Filebrowser and onto Nextcloud, which runs on its own VM rather than the cluster. Nextcloud’s all-in-one image manages its own containers through the Docker socket — fragile inside an unprivileged LXC, and beside the point inside Kubernetes. Longhorn was never the answer either: a few hundred gigabytes of household documents is arithmetic I didn’t like even at one replica, and files that stay plain files on a disk are worth more to me than an orchestrator’s opinion about them.

It’s the one thing here reachable from outside without a VPN, so it’s treated that way — a dedicated Cloudflare tunnel with nothing forwarded on the firewall, TOTP on every account, and DOCKER-USER rules that stop its containers from reaching anything else on the flat LAN.

Filebrowser stays running until the migration has been proven — two weeks in parallel and a restore drill that actually restored something, not a file count.

On the VPS

Oracle Cloud Free Tier, Ampere A1 — 4 vCPU ARM, 24GB RAM, 200GB disk. External access, backup destination, DR fallback.

The public dashboard is at inside.merox.dev — same config, no credentials, no widgets that would need them.

• App layer — compose files and Homepage config in the infrastructure repo, deployed by Ansible. The old separate repo silently overwrote other Ansible-templated services on every deploy.
• Split DNS — *.cloud.merox.dev resolves to Pi-hole over the tailnet, no Cloudflare round-trip.
• Authentik — proxy auth (Guacamole), OAuth2 (Portainer), K8s outpost.

Alerting

Machines fail quietly. This is the part that makes them speak up — one Telegram channel, with a local AI model turning raw alerts into a sentence worth reading on a phone.

One Telegram channel, AI-triaged.

• Alertmanager → n8n → Ollama → Telegram. Critical and warning only, tagged by layer. Ollama summarizes; if it’s unreachable the raw message goes through unchanged.
• Same path — Flux (Kustomization and HelmRelease errors), pve-exporter (the host and its guests), VPS events (ZFS, fans, docker health).
• UPS and WAN/Tailscale skip n8n, straight to Telegram — those are exactly the moments it might not be running.
• Dead-man’s switch on healthchecks.io: one check pinged by Alertmanager, one by the R730xd. Both silent means the site is down, and nothing inside it could have told me.

Ollama moved from the Beelink to pve with the same IP, so n8n never noticed. qwen3:4b-instruct does ~8.9 tok/s on the Xeon — slower than the i9, irrelevant for one-sentence summaries.

The same n8n instance runs a Claude Haiku news digest each morning: HN plus a Romanian RSS feed, merged, summarized, single-source claims flagged.

Backup

Nothing here is backed up because it is large — it is backed up because it cannot be rebuilt. Three copies: one in another country, one on a machine that is powered off most of the week.

The rule is narrow on purpose — back up only what can’t be rebuilt from Git:

• Declarative config — in the repo, never backed up separately.
• Observability history, caches — regenerable, excluded.
• Movies and TV — re-downloadable. No second copy anywhere; RAIDZ2 is enough.

Longhorn (K8s)  ──nightly──▶  Garage S3 on R730xd
Oracle VPS      ──nightly──▶  R730xd
pfSense, VM dumps, documents ─▶  R730xd
                                    │
                    ┌───────────────┴───────────────┐
                    │ weekly                        │ nightly
                    ▼                               ▼
              Synology                       Oracle Cloud VPS
        (cold, offline the rest             (restic, open format)
             of the week)

The three legs out:

• Longhorn → Garage (R730xd), nightly. Jellyfin and *arr configs, Immich. Caches and observability excluded.
• R730xd → Synology, weekly. Hardlink-deduplicated, 3 weeks retained. Same LAN, so not truly off-site — but the NAS is powered off outside its wake window, and ransomware can’t reach a machine that isn’t running.
• R730xd → Oracle, nightly. restic over SFTP — restorable with just the binary and the password. restic check nightly, a real restore-and-diff drill monthly.

Three copies. Losing one is a non-event, losing two still recovers. Total loss needs both on-prem devices and a VPS in another country to go at once.

/media/backups/
├── dump/              home-assistant vzdump
├── nextcloud/         borg repo, written by the VM over a forced-command key
├── pfsense/           config.xml.gz, pushed by the firewall itself
├── longhorn-garage/   Garage data + nightly meta copy from SSD
├── synology-home/     the old Filebrowser documents — kept as the rollback,
│                      dropped from both outbound legs so the same 30GB
│                      doesn't get shipped twice
├── immich-postgres/   pg_dump, 30-day retention
├── oracle-vps/        VPS service state, receive-only
└── tools/             vendor binaries needed to rebuild this host (storcli)

Recovery

A backup nobody has ever restored is a guess. Both halves of this setup have a rebuild path that has been run start to finish, and timed.

Cluster

Fully declarative, Flux keeps state in Git — full rebuild ~35 min:

task bootstrap:apps
task longhorn:restore

task longhorn:restore runs the sequence: backup target → volume restore from Garage S3 → PVs with correct claimRefs → Flux reconcile. Observability starts fresh — history isn’t backed up, dashboards come from Git.

VPS

Oracle can terminate Always Free instances without warning, so the VPS has an escape hatch: make dr-full provisions a Hetzner server on demand, runs Ansible and brings up the app stack in ~15 min. Not a standing server; it exists only if Oracle is gone. Tunnel and Tailscale reconnect on their own with vault-stored tokens.

make dr-preflight   # checks vault, age.key, Tailscale key + expiry, tools
make dr-full        # terraform apply + ansible setup + app-stack deploy (~15 min)
make dr-restore     # pulls R730xd backups, restores DBs + service state
./scripts/dr-verify.sh --phase 1   # post-DR verification

Keeping this working is a maintenance job of its own — The Update Routine → covers the cadence that keeps all of it from rotting.