command-center/agent/managed-skills/fleet-host-process-service-wiring/SKILL.md

name, description

name	description
fleet-host-process-service-wiring	Add a host-process (non-quadlet) systemd USER service/exporter to serverAppConfigs — every file that must change together, with the signal-relay precedent and the Prometheus gateway/LAN reach rules.

A host-process service (runs python3/a binary on the host, NOT a container) is the right shape when the thing must read host-only state (DRM/cgroup sysfs, shells to nvidia-smi, signal-cli) that a container cannot reach without passthrough. Precedent to mirror: scripts/signal-relay.{py,service} + install_signal_relay in deploy-remote.sh. A new one landed 2026-10-20: scripts/gpu-exporter.{py,service} + deploy_gpu_exporter.

The unit is a USER unit, not a quadlet

• Hand-written scripts/<svc>.service: [Service] ExecStart=/usr/bin/python3 /home/dank/serverAppConfigs/scripts/<svc>.py (runs straight from the git checkout — a pull updates it, no copy step), Restart=always, [Install] WantedBy=default.target.
• It CAN be systemctl --user enable-ed (generated quadlet units cannot on this host's podman generator). With Linger=yes for dank, [Install]+enable is the whole boot story.

Every file that MUST change together (a lone change on one side fails silently)

1. scripts/<svc>.service — the unit. Base binds loopback only; the LAN address is added by a deploy drop-in, so the checked-in unit is never a wildcard.
2. scripts/deploy-remote.sh:
   • a deploy_<svc>() function: cp the unit to ~/.config/systemd/user/, write a <svc>.service.d/bind.conf drop-in with the per-host Environment=..._BINDS=127.0.0.1,<lan> (map LAN from $TARGET: dankclaw=192.168.0.143, dankserver=192.168.0.235 — do NOT hardcode one host if it runs on both), daemon-reload, systemctl --user enable+restart, then a loop health gate (for i in $(seq 1 15); curl -fsS -m5 http://127.0.0.1:<port>/metrics && return; sleep 1) — a one-shot curl races the bind and once failed a healthy deploy.
   • a dispatcher branch <svc>) deploy_<svc> ;; in the central case "$service" (near the observability branch).
   • an allowlist entry on EVERY target it runs on: dankclaw:<svc> and/or dankserver:<svc> (the allowlist case around the top of the file).
3. scripts/deploy-plan.py: add "<svc>" to SERVICE_ORDER, to DANKCLAW_SERVICES/DANKSERVER_SERVICES (whichever it runs on), and a scripts/<svc>.* path-map in the parts[0] == "scripts" block: if name in {"<svc>.py","<svc>.service"}: _add(services, "<svc>"). target_services then filters per target.
4. scripts/test-changed.sh: a case routing scripts/<svc>.py|scripts/<svc>.service → its seam + deployment_scoping.bats (place before the generic fallbacks). tests/run.sh runs all *.bats regardless, so the mandatory gate is covered even if this route is missing — but add it for focused runs.
5. shared/prometheus/prometheus.yml: one scrape job per host, using the reach rules (below).
6. Dashboard JSON + the owning page under serverAppConfigs-docs/Infrastructure/Observability.md (one owner per fact).

Prometheus reach rules (the whole cross-file contract)

Prometheus is a bridge container. On this netavark host a bridge container CANNOT dial the host's own LAN IP at a host-bind (only at a published port). Therefore:

• Same host (Dankclaw): scrape host.containers.internal:<port> (netavark delivers it to the host LAN bind). Scraping 192.168.0.143:<port> reads DOWN forever even though the endpoint is healthy.
• Different host (Dankserver): plain LAN 192.168.0.235:<port> reaches it.
• Never target a *.franksdank.cloud vhost — Prometheus cannot solve an Anubis challenge; it is red by construction.
• The exporter must bind BOTH loopback and the host LAN (like node-exporter/cadvisor), so host.containers.internal and the LAN both land.

Firewall

Leave it LAN-open like node-exporter/cadvisor (unauthenticated metrics on the LAN is the repo's exporter posture). Only signal-relay gets an iptables allowlist (loopback+host+10.89.0.0/16+DROP) because it is an unauthenticated message-SENDING webhook — do not copy that lock onto a read-only exporter.

Seam test (tests/.bats) pattern

• Black-box the binary with injectable inputs (fixture dirs / stub commands), assert only consumer-visible output; assert absent means absent (anchor guards to a sample line ^metric{, not the always-emitted # TYPE header — grepping the bare family name matches the header and false-fails).
• Cross-file contracts (grep the other files): the unit's ExecStart/bind posture, the deploy function + allowlist + dispatch + health gate, the planner selection (run deploy-plan.py --target T --paths scripts/<svc>.py, assert output == <svc> per target), the prometheus targets, the dashboard json.load + that its targets[].expr reference the emitted families.

Gotchas hit while building gpu-exporter

• Values are float-normalised (397.40→397.4); a test pinning the literal fails — assert the normalised value. Derive expected byte counts from real units (MiB×1048576, sysfs µW/1e6, temp1_input/1000), don't guess the multiply.
• xdg-open <PR-URL> after opening the PR (repo law) — don't make Frank hunt in Gitea.