Five AI Agents Running My Infrastructure

This is the setup it grew into: five specialized agents, a dedicated non-root user, and a command center dashboard at agents.cloud.merox.dev. Everything was templated in the infra repo, so it could be rebuilt from scratch in 15 minutes.

Why split them at all

#

One agent trying to do everything becomes a compromise. The system prompt grows, context gets polluted between domains, and the “personality” that makes an agent useful for one task makes it annoying for another.

Each agent has its own workspace directory, its own AGENTS.md (operating instructions), SOUL.md (personality), and sometimes HEARTBEAT.md (what to do proactively). They send Telegram messages on their own, without being prompted.

Architecture

#

flowchart TB
  phone["phone / laptop"] -- Telegram --> gw["openclaw gateway
tailnet:18789, user openclaw"]
  gw --> cc["Claude Code CLI
Pro OAuth"]
  cc --> news & infra & costs & dash["dashboard"] & orch["orchestrator"]
  news --> n1["web search
/srv/dashboard"]
  infra --> i1["kubectl · flux
talosctl · docker"]
  costs --> c1["S3 backups"]
  dash --> d1["/srv/dashboard
index.html"]
  orch --> o1["agents.json
proposals.json"]

The gateway runs as a systemd user service under a dedicated openclaw user. No root. The agent workspaces describe what each agent manages and how it should behave. Claude figures out the commands to run.

Security model

#

This is where the previous setup was weak: running as root because it was easier. The new setup:

A user with the minimum it needs

#

useradd -m -s /bin/bash openclaw
usermod -aG docker openclaw
loginctl enable-linger openclaw

Sudoers, scoped to exact binaries

#

Two files:

/etc/sudoers.d/openclaw, for infra tooling:

Defaults:openclaw !requiretty
openclaw ALL=(ALL) NOPASSWD: /usr/bin/kubectl
openclaw ALL=(ALL) NOPASSWD: /usr/bin/flux
openclaw ALL=(ALL) NOPASSWD: /usr/bin/talosctl
openclaw ALL=(ALL) NOPASSWD: /bin/systemctl status *
openclaw ALL=(ALL) NOPASSWD: /usr/bin/systemctl status *
openclaw ALL=(ALL) NOPASSWD: /usr/bin/node

/etc/sudoers.d/openclaw-fix-perms, for permissions automation:

openclaw ALL=(root) NOPASSWD: /usr/local/bin/openclaw-fix-perms

Docker group membership handles container access. That is root-equivalent on the host too, since anyone who can start a container can mount /. kubectl and talosctl get dedicated config files copied to /home/openclaw/.kube/ and /home/openclaw/.talos/. The agents are explicitly blocked from reading age.key, *.sops.yaml, and .env files, and that is enforced in AGENTS.md rather than left to the model.

The Claude Code permissions problem

#

The gateway runs as openclaw, but if you interact with workspace files as root (e.g. via Claude Code CLI), any file you touch becomes root:root and the agents can’t write to it. The fix is a script that runs every 5 minutes and corrects ownership:

# /usr/local/bin/openclaw-fix-perms
chown -R openclaw:openclaw /home/openclaw/.openclaw/
chown -R openclaw:openclaw /srv/dashboard/
# Update scripts stay root-owned intentionally
chown root:root /srv/dashboard/update-*.sh
chown -R openclaw:openclaw /srv/merox/src/content/blog/

Root crontab: */5 * * * * /usr/local/bin/openclaw-fix-perms Systemd service: ExecStartPost=/usr/bin/sudo /usr/local/bin/openclaw-fix-perms

This means: even if you edit workspace files as root, they’re back to openclaw ownership within 5 minutes. No manual fixup needed.

Gateway on the tailnet only

#

{
  gateway: {
    bind: "tailnet",
    auth: { mode: "token" }
  }
}

The gateway listens on the Tailscale interface directly: reachable from any Tailscale device without a port forward, but not exposed on the public internet.

Telegram allowlist

#

{
  channels: {
    telegram: {
      botToken: "...",
      allowFrom: ["YOUR_NUMERIC_ID"]
    }
  }
}

Only your Telegram user ID can interact with the bot. Anyone else gets ignored at the gateway level, not the agent level.

Workspace files

#

OpenClaw’s workspace system is the key difference from a SKILL.md single-file approach. Each agent gets a directory with:

workspace-infra/
├── AGENTS.md      # operating instructions — what to check, how to respond
├── SOUL.md        # personality — paranoid SRE tone, factual, no false positives
├── HEARTBEAT.md   # what to do proactively on each scheduled tick
└── TOOLS.md       # local notes — paths, commands, how to update the dashboard

The content matters more than the format. Here’s what actually makes the infra agent useful:

AGENTS.md (excerpt):

## Security check (2x/day via heartbeat)

kubectl get nodes
kubectl get pods -A | grep -v Running | grep -v Completed
docker ps --format "{{.Names}}\t{{.Status}}" | grep -v "Up"
df -h

Report on Telegram ONLY if there is a real problem.
Do not send "everything is ok" every check.

SOUL.md (infra agent):

You are an SRE with healthy paranoia. You don't dramatize, but you don't minimize.

- "Trust but verify" — check live, don't assume
- Silence is golden — no false positives; when you send, it's real
- When you don't know something, say so

HEARTBEAT.md (news agent):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

Triggered daily at 04:00 UTC. Trigger message: MORNING_RUN.

You are running headless — use tools for every step, never output the briefing as text.

1. Read /srv/dashboard/data/news-releases.json (GitHub releases, pre-fetched every 6h)
2. Check memory/ for the last 3 days — no repeats
3. Collect tech & community news: Hacker News API, Reddit r/homelab + r/selfhosted,
   web search (CVEs, AI releases, infra news), GitHub trending
4. Write /srv/dashboard/data/news.json — dashboard data, BEFORE the HTML
5. Write /srv/dashboard/news.html — full HTML briefing
6. Send Telegram via Python urllib (not response text)
7. Update /srv/dashboard/data/agents.json
8. Save to memory/YYYY-MM-DD.md

CVE rules: only alert if it affects the exact installed version on K8s cluster
or Oracle VPS. gRPC/OpenSSL/Go runtime CVEs only if a specific service is
directly affected — not "K8s uses this internally". Scanner noise = skip.

The command center dashboard

#

Each agent writes to /srv/dashboard/data/: JSON files that a static HTML page reads and renders:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

/srv/dashboard/
├── index.html          # command center — all agents status
├── news.html           # latest news briefing (written by news agent)
└── data/
    ├── agents.json     # status + last run per agent
    ├── news.json           # structured news items (written by news agent)
    ├── news-releases.json  # GitHub releases pre-fetch (written by update-news.sh)
    ├── infra.json          # live cluster/VPS metrics (written by update-infra.sh)
    ├── backup.json         # backup verification results
    ├── upgrades.json       # open Renovate PRs
    ├── weather.json        # hyperlocal weather — Open-Meteo, no API key (update-weather.sh)
    ├── network.json        # subnet scan — homelab LAN + WiFi + Tailscale (update-network.sh)
    ├── calendar.json       # iCloud CalDAV events today + next 3 days (update-calendar.sh)
    ├── shared-memory.json  # cross-agent context (what agents flagged recently, suppressions)
    ├── orchestrator.json   # orchestrator run history
    └── proposals.json      # pending improvement proposals

The page auto-refreshes every 60 seconds. Dark mode, card-based layout. No backend; nginx serves static files.

An nginx container handles the serving:

# /srv/docker/agents-dashboard/docker-compose.yml
services:
  agents-dashboard:
    image: nginx:alpine
    volumes:
      - /srv/dashboard:/usr/share/nginx/html:ro
    networks:
      network-cloud-merox:
        ipv4_address: 172.25.10.90
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.agents-dashboard.rule=Host(`agents.cloud.merox.dev`)"
      - "traefik.http.routers.agents-dashboard.entrypoints=https"
      - "traefik.http.routers.agents-dashboard.tls.certresolver=cloudflare"
      - "traefik.http.routers.agents-dashboard.middlewares=middlewares-authentik@file,default-headers@file"

Protected by Authentik, the same SSO as the rest of the homelab stack.

After each run, an agent updates its status in agents.json:

import json
with open('/srv/dashboard/data/agents.json') as f:
    data = json.load(f)
data['infra'] = {
    'lastRun': '2026-05-28T08:00:00Z',
    'status': 'ok',   # ok / warn / error
    'summary': 'All nodes healthy. Disk at 45%.'
}
with open('/srv/dashboard/data/agents.json', 'w') as f:
    json.dump(data, f, indent=2)

OpenClaw config

#

All agents in one openclaw.json:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45

{
  gateway: {
    mode: "local",
    port: 18789,
    bind: "tailnet",
    auth: { mode: "token", token: "GENERATED_BY_ONBOARD_DO_NOT_SET_MANUALLY" }
  },
  agents: {
    defaults: {
      model: {
        primary: "anthropic/claude-sonnet-4-6",
        fallbacks: ["anthropic/claude-opus-4-7", "anthropic/claude-opus-4-6", "google/gemini-2.5-flash"]
      },
      thinkingDefault: "low",
      timeoutSeconds: 1800,
      heartbeat: { every: "0m" },       // heartbeats via cron, not gateway polling
      skipBootstrap: true,
      contextPruning: { mode: "cache-ttl", ttl: "5m" },
      agentRuntime: { id: "claude-cli" }
    },
    list: [
      { id: "news",         default: true, workspace: "/home/openclaw/.openclaw/workspace" },
      { id: "infra",        workspace: "/home/openclaw/.openclaw/workspace-infra" },
      { id: "costs",        workspace: "/home/openclaw/.openclaw/workspace-costs" },
      { id: "dashboard",    workspace: "/home/openclaw/.openclaw/workspace-dashboard" },
      { id: "orchestrator", workspace: "/home/openclaw/.openclaw/workspace-orchestrator" }
    ]
  },
  channels: {
    telegram: {
      botToken: "YOUR_TELEGRAM_BOT_TOKEN",
      allowFrom: ["YOUR_TELEGRAM_USER_ID"],
      dmPolicy: "allowlist"
    }
  },
  session: {
    scope: "per-sender",
    reset: { mode: "daily", atHour: 4, idleMinutes: 10080 }
  },
  tools: {
    profile: "coding",
    fs: { workspaceOnly: false },
    agentToAgent: { enabled: true }
  }
}

OpenClaw uses Claude Code CLI’s OAuth, so there is no separate Anthropic API key if you have Claude Pro. All messages go to the default agent (news), which routes to the others via agentToAgent when needed.

Setup

#

Full install in 15 minutes from scratch:

1. Install OpenClaw

#

curl -fsSL https://deb.nodesource.com/setup_24.x | sudo -E bash -
sudo apt install -y nodejs
sudo npm install -g openclaw@latest
openclaw --version

2. Create the service user

#

sudo useradd -m -s /bin/bash -d /home/openclaw -c "OpenClaw Service" openclaw
sudo usermod -aG docker openclaw
sudo loginctl enable-linger openclaw

# Copy sudoers from infra repo
sudo cp /srv/kubernetes/infrastructure/agent/scripts/sudoers-openclaw /etc/sudoers.d/openclaw
sudo chmod 440 /etc/sudoers.d/openclaw

3. Kubeconfig and talosconfig

#

sudo -u openclaw mkdir -p /home/openclaw/.kube /home/openclaw/.talos
sudo cp /srv/kubernetes/infrastructure/kubeconfig /home/openclaw/.kube/config
sudo cp /srv/kubernetes/infrastructure/talos/clusterconfig/talosconfig /home/openclaw/.talos/config
sudo chown openclaw:openclaw /home/openclaw/.kube/config /home/openclaw/.talos/config
sudo chmod 600 /home/openclaw/.kube/config /home/openclaw/.talos/config

4. Configure and install the workspaces

#

sudo -u openclaw mkdir -p /home/openclaw/.openclaw

# Copy config (fill in Telegram token + your user ID)
sudo cp /srv/kubernetes/infrastructure/agent/openclaw.json.example \
        /home/openclaw/.openclaw/openclaw.json
sudo chown openclaw:openclaw /home/openclaw/.openclaw/openclaw.json
sudo chmod 600 /home/openclaw/.openclaw/openclaw.json

# Install workspaces
AGENT_DIR=/srv/kubernetes/infrastructure/agent/workspaces
for agent in news infra costs dashboard orchestrator; do
  sudo -u openclaw cp -r $AGENT_DIR/$agent \
    /home/openclaw/.openclaw/workspace$([ "$agent" = "news" ] && echo "" || echo "-$agent")
done
sudo -u openclaw mkdir -p /home/openclaw/.openclaw/workspace/memory

5. Authenticate Claude Code, then OpenClaw

#

# 1. Log in to Claude Pro via OAuth (opens browser link)
sudo -u openclaw claude login

# 2. Run OpenClaw onboard to wire up Claude CLI auth and generate the gateway token
#    This adds agentRuntime: { id: "claude-cli" } to openclaw.json — no API key needed
sudo -u openclaw XDG_RUNTIME_DIR=/run/user/$(id -u openclaw) \
  openclaw onboard --non-interactive \
  --mode local \
  --auth-choice anthropic-cli \
  --skip-bootstrap \
  --skip-skills \
  --skip-daemon \
  --accept-risk

This uses your Claude Pro subscription: no per-token billing and no separate API key. The onboard command generates gateway.auth.token and adds agentRuntime: { id: "claude-cli" } for all Claude models. After this, re-add your Telegram credentials to ~/.openclaw/openclaw.json if onboard overwrote them.

6. The dashboard

#

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

sudo mkdir -p /srv/dashboard/data
sudo chown openclaw:openclaw /srv/dashboard /srv/dashboard/data

# Copy operational scripts (cron jobs that collect metrics without AI tokens)
sudo cp /srv/kubernetes/infrastructure/agent/dashboard/scripts/*.sh /srv/dashboard/
sudo chmod 755 /srv/dashboard/*.sh
sudo chmod 750 /srv/dashboard/tg-notify.sh  # restricted: openclaw-only
sudo chown openclaw:openclaw /srv/dashboard/*.sh

# Create /srv/dashboard/.env from the template and fill in secrets:
# TELEGRAM_BOT_TOKEN, TELEGRAM_CHAT_ID, GARAGE_TOKEN, GARAGE_BUCKET_ID,
# APPLE_ID, APPLE_APP_PASSWORD
sudo cp /srv/kubernetes/infrastructure/agent/dashboard/.env.example /srv/dashboard/.env
sudo chmod 600 /srv/dashboard/.env && sudo chown openclaw:openclaw /srv/dashboard/.env
sudo -u openclaw nano /srv/dashboard/.env

# Install Python dep for iCloud calendar (caldav)
sudo bash /srv/dashboard/setup-deps.sh

cd /srv/docker/agents-dashboard && docker compose up -d

7. fix-perms and the crontab

#

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

# Copy fix-perms script from infra repo
sudo cp /srv/kubernetes/infrastructure/agent/scripts/openclaw-fix-perms /usr/local/bin/
sudo chmod 755 /usr/local/bin/openclaw-fix-perms
sudo cp /srv/kubernetes/infrastructure/agent/scripts/sudoers-fix-perms \
        /etc/sudoers.d/openclaw-fix-perms
sudo chmod 440 /etc/sudoers.d/openclaw-fix-perms

# Add to root crontab
(crontab -l 2>/dev/null; echo "*/5 * * * * /usr/local/bin/openclaw-fix-perms") | crontab -

# Install talosctl system-wide (mise installs it in root's home, not accessible by openclaw)
sudo cp ~/.local/share/mise/installs/aqua-siderolabs-talos/TALOS_VERSION/talosctl /usr/local/bin/
sudo chmod 755 /usr/local/bin/talosctl

# Install openclaw user crontab
sudo -u openclaw crontab /srv/kubernetes/infrastructure/agent/scripts/openclaw-crontab

8. Start the gateway

#

sudo -u openclaw mkdir -p /home/openclaw/.config/systemd/user
sudo cp /srv/kubernetes/infrastructure/agent/scripts/openclaw-gateway.service \
        /home/openclaw/.config/systemd/user/
sudo chown openclaw:openclaw /home/openclaw/.config/systemd/user/openclaw-gateway.service

XDG_RUNTIME_DIR=/run/user/$(id -u openclaw) \
  sudo -u openclaw systemctl --user daemon-reload

XDG_RUNTIME_DIR=/run/user/$(id -u openclaw) \
  sudo -u openclaw systemctl --user enable --now openclaw-gateway.service

Verify:

XDG_RUNTIME_DIR=/run/user/$(id -u openclaw) \
  sudo -u openclaw systemctl --user status openclaw-gateway
sudo -u openclaw openclaw doctor

Proactive agents

#

Five agents run without being asked:

infra at 08:00 and 20:00 UTC checks cluster nodes, unhealthy pods, disk space, stopped containers. Sends Telegram only if something is wrong. If you stop getting messages, either everything is fine or the agent itself died (check the service).

news at 04:00 UTC reads pre-fetched GitHub release data, then searches Hacker News, Reddit r/homelab and r/selfhosted, and does targeted web searches for CVEs and AI/infra news. CVEs are filtered to what’s actually installed on the K8s cluster and Oracle VPS, with no generic “this library is used somewhere” noise. Writes the structured JSON for the dashboard, the HTML briefing page, then sends a Telegram summary in Romanian.

costs on Sunday at 09:00 UTC verifies backups: Garage S3 object counts, Longhorn snapshot status, NAS accessibility. Writes backup.json for the dashboard and sends a summary on Telegram if anything is stale or missing.

dashboard at 23:00 UTC audits the command center dashboard nightly: validates JSON data files, checks that all JavaScript references have matching HTML elements, then makes one incremental improvement if the audit passes. Self-tests after each change.

orchestrator at 12:00 UTC audits all other agents: checks they ran on schedule, rotates oversized logs, validates data files. If it finds a pattern worth fixing in an agent’s logic, it sends a Telegram proposal with a concrete description. You approve or reject with /da or /nu. Applied changes are backed up automatically with rollback detection.

The orchestrator can also propose infrastructure-level changes (crontab schedule adjustments, new workspace files, openclaw.json agent additions), all requiring explicit approval before any change is applied. It never modifies security-sensitive config (auth tokens, allowlists, sudoers).

Heartbeats are configured via cron jobs on the openclaw user, not via OpenClaw’s built-in heartbeat polling (which burns tokens every 30 minutes even when there’s nothing to do):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

# As openclaw user: crontab -e

# AI agents — proactive (cron-triggered)
0 4  * * *  /srv/dashboard/news-morning-run.sh           # wrapper: fresh session + MORNING_RUN
0 8  * * *  /usr/bin/openclaw agent --agent infra        --message "HEARTBEAT"
0 20 * * *  /usr/bin/openclaw agent --agent infra        --message "HEARTBEAT"
0 9  * * 0  /usr/bin/openclaw agent --agent costs        --message "HEARTBEAT"
0 23 * * *  /usr/bin/openclaw agent --agent dashboard    --message "HEARTBEAT"
0 12 * * *  /usr/bin/openclaw agent --agent orchestrator --message "HEARTBEAT"

# Zero-AI data collection (bash scripts, no tokens)
*/5  * * * *  /srv/dashboard/update-infra.sh       # K8s nodes/pods/docker/longhorn
*/30 * * * *  /srv/dashboard/update-backup.sh      # NAS/Garage S3/Longhorn backup status
*/30 * * * *  /srv/dashboard/update-weather.sh     # Open-Meteo hyperlocal weather (no API key)
0    * * * *  /srv/dashboard/update-network.sh     # Subnet scan — homelab LAN + WiFi + Tailscale
0  */6 * * *  /srv/dashboard/update-news.sh        # GitHub releases → news-releases.json
*/30 * * * *  /srv/dashboard/update-upgrades.sh    # Renovate PRs from infrastructure repo
*/30 * * * *  /srv/dashboard/update-calendar.sh    # iCloud CalDAV events (app-specific password)

# Watchdogs + utilities
5 8,20 * * *  /srv/dashboard/self-healing.sh       # restart stale agents after infra runs
0 */2  * * *  /srv/dashboard/check-logs.sh         # scan agent logs for errors
15 12  * * *  /srv/dashboard/check-proposals.sh    # notify pending orchestrator proposals
*  *   * * *  /srv/dashboard/process-triggers.sh   # pick up manual run-triggers from dashboard UI

# Calendar + utility reminders
0 4    * * *  /srv/dashboard/calendar-reminder.sh  # daily events summary via Telegram
5 7   22 * *  /srv/dashboard/tg-notify.sh "Reminder: transmite indexul de electricitate"
5 7    5 * *  /srv/dashboard/tg-notify.sh "Reminder: transmite indexul de gaz"

Orchestrator proposals

#

The orchestrator introduces an approval loop for agent improvements. When it detects a pattern worth fixing (say, the news agent has been alerting on releases that Renovate would handle anyway, three weeks in a row), it writes a proposal to /srv/dashboard/data/proposals.json and sends a Telegram message:

Propunere #prop-20260529-news-001

Agent: news
Risc: low

Skip Renovate-covered releases in news alerts
News agent has alerted on 4 simple version bumps this week that Renovate
would have caught on Saturday. Add a filter for releases where no CVE
or breaking change is present.

Răspunde cu /da prop-20260529-news-001 sau /nu prop-20260529-news-001

Reply /da and the next orchestrator run patches the relevant section of that agent’s AGENTS.md, backs up the original, and confirms on Telegram. If the change causes a regression (detected by comparing agents.json status before/after), it auto-rolls back and alerts.

Safe fixes like log rotation, missing JSON keys and stale file cleanup happen automatically without asking.

Disaster recovery

#

Everything except secrets and memory was in the infra repo at agent/. Rebuilding on a new server took ~15 minutes:

1. Install Node.js 24 + OpenClaw (npm install -g openclaw@latest)
2. Create openclaw user + docker group + linger
3. Copy sudoers-openclaw → /etc/sudoers.d/openclaw
4. Copy kubeconfig + talosconfig to /home/openclaw/.kube/ and /home/openclaw/.talos/
5. Install talosctl to /usr/local/bin/ (mise installs it per-user, not system-wide)
6. Copy all 5 workspaces from agent/workspaces/ and create memory/ dirs for all
7. Copy openclaw.json.example → ~/.openclaw/openclaw.json, fill in Telegram token
8. sudo -u openclaw claude login (OAuth, opens a browser)
9. sudo -u openclaw openclaw onboard --auth-choice anthropic-cli --skip-bootstrap
10. Install openclaw-fix-perms to /usr/local/bin/ + root crontab */5 * * * *
11. Install openclaw-crontab as openclaw user’s crontab
12. Copy agent/dashboard/scripts/*.sh → /srv/dashboard/, create /srv/dashboard/.env from template
13. Start systemd user service + docker compose up -d for dashboard

Can’t be auto-recovered:

• openclaw.json secrets (Telegram token + gateway auth token): keep in a password manager
• /srv/dashboard/.env (Telegram + Garage S3 + Apple credentials): keep in a password manager
• workspace*/memory/ files: agents rebuild context over a few days, history is lost

What the agents can’t do without attention on first boot:

• infra-extended.json won’t exist until the infra agent runs (08:00 UTC)
• proposals.json needs to be initialized: echo '{"pending":[],"history":[]}' > /srv/dashboard/data/proposals.json

Everything else is in git or regenerates automatically.

The design decision worth stealing from this, if you take nothing else: the agents only speak when something is wrong, and everything that can be collected by a bash script is collected by a bash script. Built-in heartbeat polling burns tokens every thirty minutes to discover that nothing happened. Cron does the same job for free, and an agent that messages you daily regardless is one you stop reading by the second week.

Config template, workspace files, sudoers and systemd unit, as they were before the teardown: meroxdotdev/infrastructure, agent/.