On-Prem & Cloud — My Infrastructure

Two standalone Proxmox hosts (R730xd + Beelink), a 3-node Talos Kubernetes cluster split across them, R730xd’s SAS pool as live storage, Synology as cold backup only, Oracle Cloud VPS as the off-site anchor. One Tailscale mesh, one Git repo.

Hardware

Compute

The two OptiPlex Micro 3050s from the old 3-node cluster are retired — powered off.

Network Gear

An Alcatel-Lucent switch is racked but unconfigured — pending decision on replacing or supplementing the TP-Link.

Network

1
WAN   → Orange ISP (public IP, bridge mode)
2
LAN   → 10.57.57.0/24, main homelab network
3
OPT1  → 10.57.97.0/24, dedicated Kubernetes node subnet
4
VLAN  → WiFi/guest, isolated from LAN except whitelisted apps (e.g. Jellyfin)

pfSense

Fanless mini PC (~200€), running pfSense 3+ years: XCY X44 on AliExpress

• Tailscale Subnet Router — advertises LAN subnets to the tailnet, so every device (Synology, iDRAC, everything) is reachable without a client installed on each one. Also how the VPS reaches back into the LAN without opening any inbound port on the home public IP. Full setup guide →
• Unbound DNS — local recursive resolver, domain override for *.k8s.merox.dev → K8s-Gateway.
• Firewall rules: WiFi→LAN blocks all except whitelisted apps; LAN→WAN allows all; WAN→Internal blocks all except explicitly exposed services.

Tailscale Mesh

One flat network everywhere — homelab rack and Oracle VPS on the same mesh, no tunnels, no firewall holes.

Virtualization

Proxmox

Two standalone hosts, no corosync cluster — pve (R730xd) and px-0 (Beelink), linked via Proxmox Datacenter Manager instead of classic HA. PDM itself runs as a VM on px-0.

Storage (per-host ZFS pools):

R730xd NFS-exports media for the whole cluster: media/library (movies/TV/downloads, one dataset so Sonarr/Radarr/qBittorrent’s hardlink imports work), media/photos (Immich), media/isos, media/backups (Longhorn/Garage/pfSense/Immich-DB backups + the VPS’s own service backups — see Backup).

VMs and containers:

Nvidia Quadro P2200 passed through to kubernetes-controlplane-1 for hardware transcoding (replaced the old Beelink/QuickSync setup). nvidia-device-plugin exposes it to K8s; Jellyfin is scheduled on that node. GPU passthrough guide →

Synology DS223+ — cold storage only

No live services — Photos, Drive, Docker all decommissioned. Asleep except once a week:

• Wakes Sunday 02 (DSM Power Schedule + WoL)
• Receives a weekly, deduplicated, versioned push from R730xd (see Backup) — local-ish recovery copy
• Shuts down shortly after

Used to also relay to Oracle via Hyper Backup — retired, R730xd pushes to Oracle directly now (see Backup).

Movies/TV/downloads: no second copy, re-downloadable, RAIDZ2 alone is enough. Photos/documents: weekly to Synology, nightly to Oracle.

Power Management

Two CyberPower UPS units — 1500VA (Proxmox hosts), 1000VA (network gear). Power failure triggers a cascading shutdown: K8s nodes drain before Proxmox hosts power off.

Kubernetes

Talos Linux + FluxCD (GitOps). Starting point: onedr0p/cluster-template.

Repo: github.com/meroxdotdev/infrastructure

Key customizations:

GitOps structure:

1
kubernetes/apps/
2
├── cert-manager/     # TLS automation
3
├── default/          # Production workloads
4
├── flux-system/      # Flux operator + instance + n8n alerting
5
├── kube-system/      # Cilium, CoreDNS, NFS CSI, metrics-server
6
├── network/          # k8s-gateway, Cloudflare tunnel + DNS
7
├── observability/    # Prometheus, Grafana, Loki
8
└── storage/          # Longhorn configuration

Deployed apps:

Live status: inside.merox.dev. LoadBalancer IPs: Cilium L2 announcement, pool 10.57.57.100–120, no external LB.

Cluster Rebuild & Disaster Recovery

Fully declarative, Flux keeps state in Git — full rebuild ~35 min:

task bootstrap:apps
task longhorn:restore

task longhorn:restore handles the sequence: backup target → volume restore from Garage S3 → PVs with correct claimRefs → Flux reconcile. Observability starts fresh (history isn’t backed up, dashboards come from Git).

Runbook: DR.md (tested end-to-end) · DEPLOY.md (full rebuild) · Restoring from Longhorn Backups →

Cloud

Oracle Cloud Free Tier, Ampere A1 (4 vCPU / 24GB RAM / 200GB disk) — external access, backup destination, DR fallback.

Services

App layer: docker-compose.yml + Homepage config live in infrastructure (vps/roles/app_stack_setup/files/), deployed by Ansible. Used to be a separate repo (cloudlab-merox) — retired, it was silently overwriting other Ansible-templated services on every deploy.

Tailscale Split DNS routes *.cloud.merox.dev to Pi-hole over the tailnet — no Cloudflare round-trip for internal traffic.

Authentik — SSO for everything: proxy auth (Guacamole), OAuth2 (Portainer), K8s outpost. Full setup →

Disaster Recovery

Oracle can terminate Always Free instances without warning.

make dr-full provisions a Hetzner VPS on demand, deploys Ansible, brings up the app stack — ~15 min. Not a standing server, only spun up if Oracle is lost. Tunnel + Tailscale reconnect automatically with vault-stored tokens.

make dr-preflight   # checks vault, age.key, Tailscale key + expiry, tools
make dr-full        # terraform apply + ansible setup + app-stack deploy (~15 min)
make dr-restore     # pulls R730xd backups, restores DBs + service state
./scripts/dr-verify.sh --phase 1   # post-DR verification

Full walkthrough: Oracle Cloud Free Tier: Building a Full DR Plan →

Alerting

One Telegram channel, AI-triaged, replacing two Discord-only paths.

• Alertmanager + Flux → n8n → Ollama → Telegram. Critical/warning severities, tagged by layer (Proxmox, K8s, VPS, GitOps). Ollama summarizes; falls back to the raw message if it’s unreachable.
• Ollama: its own VM on px-0, CPU-only, outside the K8s cluster.
• Proxmox hosts + VMs/LXCs: pve-exporter on both standalone hosts — host down, high CPU/memory, storage gone, VM/LXC restarts.
• VPS (UPS, ZFS, fans, docker health): same n8n path. UPS and WAN/Tailscale go straight to Telegram, bypassing n8n.
• Discord: dead-man’s-switch only, two timers.

New routing immediately surfaced two silently-firing alerts (orphaned PDB, stale failed Job) — invisible under the old critical-only Discord path.

n8n also runs a Claude Haiku news digest each morning: HN + a Romanian RSS, merged, summarized, flagged on single-source claims.

Every backup/restore-drill script still pings its own healthchecks.io check on success, /fail on error.

Backup

Only back up what can’t be rebuilt from Git — and only what’s actually worth protecting. Home Assistant’s VM backup used to ride the offsite leg too; doesn’t anymore, not worth the bandwidth.

1
Longhorn (K8s)  ──nightly──▶  Garage S3 on R730xd
2
Oracle VPS      ──nightly──▶  R730xd (/media/backups/oracle-vps)
3
                                    │
4
                    ┌───────────────┴───────────────┐
5
                    │ weekly (Sunday)                │ nightly
6
                    ▼                                 ▼
7
              Synology                          Oracle Cloud VPS
8
        (local-ish recovery)               (restic, open format)

• Longhorn → Garage (R730xd), nightly. Jellyfin + *arr configs only — observability/cache volumes excluded, regenerable.
• Oracle VPS → R730xd, nightly. Authentik/Joplin dumps, Guacamole, Traefik certs, Pi-hole, Homepage, Portainer — plain SSH rsync, no daemon.
• R730xd → Synology, weekly. Whole /media/backups tree, hardlink-deduplicated snapshots, 3 weeks retained. Same LAN — not a real offsite copy.
• R730xd → Oracle, nightly, direct. Used to hop through Synology’s Hyper Backup, which needed a working DSM instance to restore. Replaced with restic over SFTP: open format, restorable with just the binary + password. Verified nightly (restic check) and proven monthly with an actual restore-and-diff drill. Full writeup →

Three independent copies (R730xd, Synology, Oracle). Losing any one is a non-event; losing two still recovers, worst case from Synology’s weekly copy. Total loss requires all three gone at once: two on-prem devices plus a VPS in a different country.

3-2-1 Backup Strategy →