Files

T

Frank e7f0d93c86 docs(vault): fleet-wide fact re-verification pass (67 pages)

The docs-fact census wave re-read the live hosts and re-keyed every page
it could: runtime/payload identities, ports, sizes, ledger figures, unit
state, and the retirement banners that drifted as the fleet moved. The
wave was interrupted mid-run, so this commit carries only what passed the
integrity gates: every touched page parses (front matter or H1+status
convention intact, balanced fences, no truncated tables, trailing
newline), the docs_completeness and link-integrity seams are green, and
bats is 384/384.

Numbers that this session proved on live hosts are keyed to the measured
value, not the page's earlier claim.

2026-10-18 13:15:43 -07:00

7.3 KiB

Raw Permalink Blame History

type, description, tags, timestamp

type

description

Alertmanager

Alertmanager receives firing rules from Prometheus, groups and dedupes them, and forwards each group as a webhook to the host-side Signal relay — which sends it through signal-cli to Frank's phone. It is the first delivery path in the observability stack (2026-09-17); before it, a firing alert was visible in Prometheus and on the Grafana dashboard's ALERTS panel (shared/grafana/provisioning/dashboards/llm-fleet.json, the "Firing fleet alerts" panel) and reached nobody.

Host: Dankclaw — Port: 9093 (loopback) — Docs: https://prometheus.io/docs/alerting/latest/alertmanager/

Facts

Item	Value
Image	`docker.io/prom/alertmanager:latest`
Published	`127.0.0.1:9093` → 9093, loopback only — never a wildcard (`alertmanager.container:40`; `ss -ltnp` 2026-10-18 shows `127.0.0.1:9093` and no other bind; `tests/observability_quadlet.bats:63` pins it)
Unit	`dankclaw/observability/quadlet/alertmanager.container` → `alertmanager.service`
Boot persistence	`Restart=always`/`RestartSec=10` + `WantedBy=default.target` (`dankclaw/observability/quadlet/alertmanager.container:49-54`); installed at `~/.config/containers/systemd/alertmanager.container` and reporting `generated` + `active` (`systemctl --user is-enabled
Config	`/opt/alertmanager/config/alertmanager.yml`, installed from `shared/prometheus/alertmanager.yml` on every deploy
Data	`/opt/alertmanager/data` (silences, notification log)
Health	`wget --spider -q http://localhost:9093/-/healthy` (`alertmanager.container:43`; the deployer waits on the same path on the published loopback, `scripts/deploy-remote.sh:532`)
Receiver	one `signal` receiver → webhook `http://host.containers.internal:18093/alertmanager` (`shared/prometheus/alertmanager.yml:27-30`)
Grouping	`group_by: [alertname]`, `group_wait` 30 s, `group_interval` 5 m, `repeat_interval` 4 h (`shared/prometheus/alertmanager.yml:21-25`)

Why the webhook address is `host.containers.internal`

The Signal relay is a host-side process, and the address it must bind is the one detail this design gets wrong twice (2026-09-17). From a bridge container, loopback is the container's own, and netavark DNATs published ports only — so a relay bound to 127.0.0.1 refused the first webhook with dial tcp 169.254.1.2:18093: connection refused. The obvious fix — bind host.containers.internal itself — fails too: on this host netavark runs its bridges in aardvark-dns' netns, so 169.254.1.2 is not a local address in the host's namespace (OSError: [Errno 99] Cannot assign requested address). What works: bind the host's LAN IP (192.168.0.143) — netavark delivers the container's 169.254.1.2 dial to that address (proven with a probe socket: container → 169.254.1.2:18094 → host socket bound to 192.168.0.143:18094, while the LAN IP refused from the container, the usual netavark quirk). Since that puts an unauthenticated endpoint on a host interface, the deployer installs an iptables INPUT set admitting 18093/tcp from loopback, the host itself (its own health probe) and the podman bridge subnet (10.89.0.0/16) only — the rest of the LAN gets a DROP.

The Signal relay

scripts/signal-relay.py is a systemd user unit (scripts/signal-relay.service), not a Quadlet: it runs python3 from the checkout, so a deploy reaches it with no copy step, and a hand-written user unit can be enabled (generated quadlet units cannot on this host). Linger is on for dank, so WantedBy=default.target is the whole boot story.

POST /alertmanager — the Alertmanager webhook handler; formats the group and sends it.
GET /health — liveness for the deployer's health wait.
Dedupes per group for 15 minutes, so a stuck alert cannot out-repeat the 4 h repeat_interval during a relay hiccup.
Sends through /usr/local/bin/signal-cli -a +12062476336 -o json send +12062476336 -m <text> (scripts/signal-relay.py:88-104) — Frank's existing account (state dir 318311, ls ~/.local/share/signal-cli/data 2026-10-18: 318311, 318311.d, accounts.json); the credential is the account state under ~/.local/share/signal-cli, and the account/phone are numbers, not secrets.

Delivery is measured, not assumed. The relay's journal carries successful sends for every rule in the set — journalctl --user -u signal-relay (2026-10-18) ends signal-relay: sent to +12062476336: [CRITICAL] LlmEndpointDown (1 firing), … [WARNING] LlmSpecDecodeOff (1 firing), … [INFO] LlmPrefixCacheCollapse (1 firing), with matching (1 resolved) lines and suppressed duplicate where the 15-minute window collapsed a repeat. Each of those lines is printed only after signal-cli exits 0 (scripts/signal-relay.py:124-126), and ss -ltnp (2026-10-18) shows python3 bound to 192.168.0.143:18093, with curl -s http://192.168.0.143:18093/health → ok. What the phone itself shows cannot be checked from this host. (unverified 2026-10-18: handset receipt)

Deployment

./scripts/deploy.sh dankclaw --service alertmanager

deploy_observability installs the quadlet unit, copy_alertmanager_config installs the config, and install_signal_relay copies the user unit, installs the iptables rules, enables and restarts the relay, then waits on 192.168.0.143:18093/health before the container's own /-/healthy wait. The planner selects alertmanager for its quadlet file, shared/prometheus/alertmanager.yml, and the two relay files; prometheus ships the alerting: block that points at alertmanager:9093.

Gotchas

The webhook URL and the relay's listen port are a cross-file contract: the relay moves ports and the webhook string does not move with it, and every alert is then dropped with no error anywhere. tests/observability_quadlet.bats pins both halves.
The container address is alertmanager:9093, never the published port: Prometheus is in a container and the port is loopback-only.
signal-cli account state is on the host, not in the container — a wiped ~/.local/share/signal-cli silences delivery while every status page stays green.
signal-cli 0.14.7 (signal-cli --version on Dankclaw, 2026-10-18) parses an all-digit -a value as a phone number and never falls back to the accounts.json name: the account's numeric name (318311) dies with Invalid account (phone number), while -a +12062476336 works (scripts/signal-relay.py:36-41). The message goes via -m, not a positional — a positional message is parsed as a second recipient (Invalid phone number '…').
A failed send is not marked in the dedupe map (scripts/signal-relay.py:124-129), so the next Alertmanager repeat retries it — but a valid webhook always gets a 200 back, even when the handler raises (scripts/signal-relay.py:155-161: a relay bug must not 500 Alertmanager; only an unparseable payload gets a 400), so a dead signal-cli shows up only in the relay's journal, not in Alertmanager's.

7.3 KiB Raw Permalink Blame History