dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity
Files
a3bce7d123cd968c1a75bca65c97896281571fca
serverAppConfigs/serverAppConfigs-docs/Infrastructure/Alertmanager.md
T
Frank a5fe045b58 docs(vault): one fact, one owner — repair the doubled pages, single-source the facts 
Structural repair:
- 13 pages were the same document twice (each exactly 2x its HEAD size),
  a second frontmatter block whose closing fence had been glued onto a body
  line by a frontmatter rewrite that re-read its own output. Un-doubled by
  cutting at the second frontmatter and keeping the copy whose body is a
  superset; the orphan census reports 0 lost tokens against HEAD.
- log.md: one entry per date, bare `## YYYY-MM-DD` headings, newest first —
  66 dated headings merged into 34, titles demoted to bold leads, every
  content line verified present. The file had been descending at the head
  and ascending at the tail.

Ownership moves:
- New `LLM/frank-medium.md` is the locator for the retired Dankserver
  endpoint: era windows quoted from each era page's own banner, no numbers
  restated, and the two unresolved date conflicts recorded instead of
  silently resolved. `frank-archon.md` links to it instead of restating the
  pre-rename window.
- The four Pipeline-Parallelism stubs (frontmatter only, no body) are gone;
  `Multi-Node Runtimes.md` owns runtime construction, launch, mixed-vendor
  verdicts and cross-split spec decode, and the four links that pointed at
  the stubs were repointed at it or at the drafter page.
- Authentik's three stub pages fold into `Authentik.md`, with the process
  detail and the outpost reference split out so the parent stays under the
  length hint.

Verified corrections (host census + live probe, 2026-10-19):
- The MXFP4 checkpoint two pages called the "on-disk rollback" is gone:
  /home/dank/models/ holds StillDeadcode, rad and tpl.jinja. Stated once, on
  the era page that owns that payload; the swap page now links to it.
- frank-medium-radiance-mxfp4: the 4.24/7 accept rate belongs to the DFlash2
  arm measured on the FP8 target, not to the MXFP4 era; the 3-family soak run
  census is kept once (155 requests) instead of twice (~120 and 155).
- Fleet shapes re-confirmed against the live endpoints before touching any
  page: 3x200000 fp8 KV, 2x200704, 2x262144.

Index rows are regenerated, not hand-carved. Net -1,849 vault lines.
2026-10-19 05:00:43 -07:00

64 lines
7.2 KiB
Markdown
Raw Blame History

---
type: reference
description: "Receives firing rules from Prometheus, groups and dedupes them, and forwards each group to the host-side Signal relay."
tags:
- infrastructure
timestamp: 2026-09-17
---
# Alertmanager
Alertmanager receives firing rules from Prometheus, groups and dedupes them, and forwards each group as a webhook to the host-side Signal relay — which sends it through signal-cli to Frank's phone. It is the first delivery path in the observability stack (2026-09-17); before it, a firing alert was visible in Prometheus and on the Grafana dashboard's ALERTS panel (`shared/grafana/provisioning/dashboards/llm-fleet.json`, the "Firing fleet alerts" panel) and reached nobody.
**Host:** Dankclaw — **Port:** 9093 (loopback) — **Docs:** https://prometheus.io/docs/alerting/latest/alertmanager/
## Facts
| Item | Value |
|------|-------|
| Image | `docker.io/prom/alertmanager:latest` |
| Published | `127.0.0.1:9093` → 9093, loopback only — never a wildcard (`alertmanager.container:40`; `ss -ltnp` 2026-10-18 shows `127.0.0.1:9093` and no other bind; `tests/observability_quadlet.bats:63` pins it) |
| Unit | `dankclaw/observability/quadlet/alertmanager.container` → `alertmanager.service` |
| Boot persistence | `Restart=always`/`RestartSec=10` + `WantedBy=default.target` (`dankclaw/observability/quadlet/alertmanager.container:49-54`); installed at `~/.config/containers/systemd/alertmanager.container` and reporting `generated` + `active` (`systemctl --user is-enabled|is-active alertmanager` 2026-10-18) |
| Config | `/opt/alertmanager/config/alertmanager.yml`, installed from `shared/prometheus/alertmanager.yml` on every deploy |
| Data | `/opt/alertmanager/data` (silences, notification log) |
| Health | `wget --spider -q http://localhost:9093/-/healthy` (`alertmanager.container:43`; the deployer waits on the same path on the published loopback, `scripts/deploy-remote.sh:532`) |
| Receiver | one `signal` receiver → webhook `http://host.containers.internal:18093/alertmanager` (`shared/prometheus/alertmanager.yml:27-30`) |
| Grouping | `group_by: [alertname]`, `group_wait` 30 s, `group_interval` 5 m, `repeat_interval` 4 h (`shared/prometheus/alertmanager.yml:21-25`) |
## Why the webhook address is `host.containers.internal`
The Signal relay is a host-side process, and the address it must bind is the one detail this design gets wrong twice (2026-09-17). From a bridge container, loopback is the container's own, and netavark DNATs *published* ports only — so a relay bound to 127.0.0.1 refused the first webhook with `dial tcp 169.254.1.2:18093: connection refused`. The obvious fix — bind `host.containers.internal` itself — fails too: on this host netavark runs its bridges in aardvark-dns' netns, so 169.254.1.2 is not a local address in the host's namespace (`OSError: [Errno 99] Cannot assign requested address`). What works: bind the **host's LAN IP** (192.168.0.143) — netavark delivers the container's `169.254.1.2` dial to that address (proven with a probe socket: container → `169.254.1.2:18094` → host socket bound to `192.168.0.143:18094`, while the LAN IP refused from the container, the usual netavark quirk). Since that puts an unauthenticated endpoint on a host interface, the deployer installs an iptables INPUT set admitting 18093/tcp from loopback, the host itself (its own health probe) and the podman bridge subnet (10.89.0.0/16) only — the rest of the LAN gets a DROP.
## The Signal relay
`scripts/signal-relay.py` is a systemd **user** unit (`scripts/signal-relay.service`), not a Quadlet: it runs `python3` from the checkout, so a deploy reaches it with no copy step, and a hand-written user unit can be `enable`d (generated quadlet units cannot on this host). Linger is on for `dank`, so `WantedBy=default.target` is the whole boot story.
- POST `/alertmanager` — the Alertmanager webhook handler; formats the group and sends it.
- GET `/health` — liveness for the deployer's health wait.
- Dedupes per group for 15 minutes, so a stuck alert cannot out-repeat the 4 h `repeat_interval` during a relay hiccup.
- Sends through `/usr/local/bin/signal-cli -a +12062476336 -o json send +12062476336 -m <text>` (`scripts/signal-relay.py:88-104`) — Frank's existing account (state dir `318311`, `ls ~/.local/share/signal-cli/data` 2026-10-18: `318311`, `318311.d`, `accounts.json`); the credential is the account state under `~/.local/share/signal-cli`, and the account/phone are numbers, not secrets.
Delivery is measured, not assumed. The relay's journal carries successful sends for every rule in the set — `journalctl --user -u signal-relay` (2026-10-18) ends `signal-relay: sent to +12062476336: [CRITICAL] LlmEndpointDown (1 firing)`, `… [WARNING] LlmSpecDecodeOff (1 firing)`, `… [INFO] LlmPrefixCacheCollapse (1 firing)`, with matching `(1 resolved)` lines and `suppressed duplicate` where the 15-minute window collapsed a repeat. Each of those lines is printed only after `signal-cli` exits 0 (`scripts/signal-relay.py:124-126`), and `ss -ltnp` (2026-10-18) shows `python3` bound to `192.168.0.143:18093`, with `curl -s http://192.168.0.143:18093/health` → `ok`. What the phone itself shows cannot be checked from this host. *(unverified 2026-10-18: handset receipt)*
## Deployment
```bash
./scripts/deploy.sh dankclaw --service alertmanager
```
`deploy_observability` installs the quadlet unit, `copy_alertmanager_config` installs the config, and `install_signal_relay` copies the user unit, installs the iptables rules, enables and restarts the relay, then waits on `192.168.0.143:18093/health` before the container's own `/-/healthy` wait. The planner selects `alertmanager` for its quadlet file, `shared/prometheus/alertmanager.yml`, and the two relay files; `prometheus` ships the `alerting:` block that points at `alertmanager:9093`.
## Gotchas
- The webhook URL and the relay's listen port are a **cross-file contract**: the relay moves ports and the webhook string does not move with it, and every alert is then dropped with no error anywhere. `tests/observability_quadlet.bats` pins both halves.
- The container address is `alertmanager:9093`, never the published port: Prometheus is in a container and the port is loopback-only.
- signal-cli account state is on the host, not in the container — a wiped `~/.local/share/signal-cli` silences delivery while every status page stays green.
- signal-cli 0.14.7 (`signal-cli --version` on Dankclaw, 2026-10-18) parses an all-digit `-a` value as a **phone number** and never falls back to the `accounts.json` name: the account's numeric name (`318311`) dies with `Invalid account (phone number)`, while `-a +12062476336` works (`scripts/signal-relay.py:36-41`). The message goes via `-m`, not a positional — a positional message is parsed as a second recipient (`Invalid phone number '…'`).
- A failed send is **not** marked in the dedupe map (`scripts/signal-relay.py:124-129`), so the next Alertmanager repeat retries it — but a valid webhook always gets a 200 back, even when the handler raises (`scripts/signal-relay.py:155-161`: a relay bug must not 500 Alertmanager; only an unparseable payload gets a 400), so a dead signal-cli shows up only in the relay's journal, not in Alertmanager's.
## See Also
- [[Prometheus]] — the rule evaluator that ships the alerts
- [[Observability]] — the full stack
Reference in New Issue View Git Blame Copy Permalink