Files

T

Frank a5fe045b58 docs(vault): one fact, one owner — repair the doubled pages, single-source the facts

Structural repair:
- 13 pages were the same document twice (each exactly 2x its HEAD size),
  a second frontmatter block whose closing fence had been glued onto a body
  line by a frontmatter rewrite that re-read its own output. Un-doubled by
  cutting at the second frontmatter and keeping the copy whose body is a
  superset; the orphan census reports 0 lost tokens against HEAD.
- log.md: one entry per date, bare `## YYYY-MM-DD` headings, newest first —
  66 dated headings merged into 34, titles demoted to bold leads, every
  content line verified present. The file had been descending at the head
  and ascending at the tail.

Ownership moves:
- New `LLM/frank-medium.md` is the locator for the retired Dankserver
  endpoint: era windows quoted from each era page's own banner, no numbers
  restated, and the two unresolved date conflicts recorded instead of
  silently resolved. `frank-archon.md` links to it instead of restating the
  pre-rename window.
- The four Pipeline-Parallelism stubs (frontmatter only, no body) are gone;
  `Multi-Node Runtimes.md` owns runtime construction, launch, mixed-vendor
  verdicts and cross-split spec decode, and the four links that pointed at
  the stubs were repointed at it or at the drafter page.
- Authentik's three stub pages fold into `Authentik.md`, with the process
  detail and the outpost reference split out so the parent stays under the
  length hint.

Verified corrections (host census + live probe, 2026-10-19):
- The MXFP4 checkpoint two pages called the "on-disk rollback" is gone:
  /home/dank/models/ holds StillDeadcode, rad and tpl.jinja. Stated once, on
  the era page that owns that payload; the swap page now links to it.
- frank-medium-radiance-mxfp4: the 4.24/7 accept rate belongs to the DFlash2
  arm measured on the FP8 target, not to the MXFP4 era; the 3-family soak run
  census is kept once (155 requests) instead of twice (~120 and 155).
- Fleet shapes re-confirmed against the live endpoints before touching any
  page: 3x200000 fp8 KV, 2x200704, 2x262144.

Index rows are regenerated, not hand-carved. Net -1,849 vault lines.

2026-10-19 05:00:43 -07:00

6.9 KiB

Raw Blame History

type, title, description, tags, timestamp

type

title

description

Observability on Dankclaw

Summary

Six containers and one host-side process: Prometheus (metrics), Grafana (dashboards), Loki (log store), Alloy (log shipper), node-exporter (host metrics) and Alertmanager (alert delivery), with signal-relay on the host carrying every firing group to Signal. All of it is Quadlet-managed, boot-persistent and loopback-only, deployed by scripts/deploy-remote.sh → deploy_observability. The metrics half works end to end; the log half is configured, healthy, and carries no data.

This body was rewritten on 2026-10-18 from live evidence: the previous version was deleted down to its front matter by commit 93d1181 (the "OKF v0.1 compliance pass"), while sibling pages kept pointing at a "state note" on this page that no longer existed. Every statement below carries the command or file that proves it today.

The stack

Service	Container	Published	Unit	Page
Prometheus	`docker.io/prom/prometheus:latest`	`127.0.0.1:9090`	`prometheus.container`	Prometheus
Alertmanager	`docker.io/prom/alertmanager:latest`	`127.0.0.1:9093`	`alertmanager.container`	Alertmanager
Grafana	`docker.io/grafana/grafana:latest`	`127.0.0.1:3002` → 3000	`grafana.container`	Grafana
Loki	`localhost/franksdank/loki:latest`	`127.0.0.1:3100`	`loki.container`	Loki
Alloy	`localhost/franksdank/alloy:latest`	`127.0.0.1:12345`	`alloy.container`	Alloy
node-exporter	`docker.io/prom/node-exporter:latest`	host netns: `127.0.0.1:9100` + `192.168.0.143:9100`	`node-exporter.container`	Node-Exporter
Signal relay	host `python3`, not a container	`192.168.0.143:18093`	`scripts/signal-relay.service` (user unit)	Alertmanager

Every unit lives in dankclaw/observability/quadlet/, carries Restart=always/RestartSec=10 and WantedBy=default.target, is installed at ~/.config/containers/systemd/<name>.container, and reports generated + active (systemctl --user is-enabled|is-active for all six, 2026-10-18; linger is on for dank). signal-relay.service reports enabled + active the same day, and homepage-status.timer — the root unit that probes all of these on loopback — is enabled + active (deploy-remote.sh:319-321 copies the probe registry shared/homepage/services.json to /srv/homepage). Dankserver runs one member of this family, node-exporter, as its own unit (dankserver/observability/quadlet/node-exporter.container, generated + active, binding 127.0.0.1:9100 + 192.168.0.235:9100).

Exposure

The five HTTP surfaces publish on 127.0.0.1 only — ss -ltnp (2026-10-18) shows 127.0.0.1:9090, 127.0.0.1:9093, 127.0.0.1:3002, 127.0.0.1:3100, 127.0.0.1:12345 and no wildcard — because each of them is unauthenticated inside, and several carry lifecycle or query APIs. Browser access goes through Caddy: each vhost (dankclaw/Caddyfile:128-145) proxies to the guard anubis:8923, and the post-auth handlers route by container name on dankclaw_default — loki.franksdank.cloud → loki:3100 (:364-366), grafana.franksdank.cloud → grafana:3000 (:374-376), prometheus.franksdank.cloud → prometheus:9090 (:379-381). Alertmanager has no vhost at all. node-exporter is the exception that must live in the host network namespace, and it binds exactly two addresses rather than the wildcard — see Node-Exporter.

The relay's bind is the one address a bridge container can reach on the host: 192.168.0.143:18093, with iptables INPUT admitting loopback, the host itself and the podman bridge subnet 10.89.0.0/16 before the DROP (scripts/deploy-remote.sh:255-262).

Deployment

./scripts/deploy.sh dankclaw --service <name> runs deploy_observability, which installs the unit, copies that service's config (copy_prometheus_config, copy_alertmanager_config, copy_loki_config, copy_alloy_config, copy_grafana_provisioning, plus install_signal_relay with the alertmanager deploy), then waits on the loopback health endpoint (/prometheus: /-/healthy, /alertmanager: /-/healthy, /grafana: /api/health, /loki: /ready, /alloy: /-/healthy, /node-exporter: /metrics). Startup order follows the data dependencies — grafana after prometheus, alloy after loki (grafana.container:32-33, alloy.container:38-39). See Observability Provisioning for what is provisioned as code and how to verify it through the APIs.

Current state (2026-10-18)

Metrics: working. curl -s http://127.0.0.1:9090/api/v1/targets reports eight up targets; the four downs are structural (the comfyui and services HTML probes, and Gitea's disabled metrics endpoint), not outages. Rule set: 4 alert rules + 2 recording rules (/api/v1/rules, matching shared/prometheus/rules/llm-fleet.yml).
Delivery: working. Alertmanager's single signal receiver webhooks the relay, and journalctl --user -u signal-relay records successful sent to +12062476336: lines for every rule in the set.
Dashboards: working. One provisioned Prometheus datasource (uid prometheus) and the llm-fleet dashboard, both installed from shared/grafana/provisioning/.
Logs: configured, empty. Alloy ships zero lines (journal counter 0, both file globs refused) and Loki therefore holds zero streams — /opt/loki/data is a few hundred KB of WAL/compactor bookkeeping with no chunk files (8 files total by find, 2026-10-18) and GET /loki/api/v1/labels returns no label names. See Alloy → Delivery state and Loki → Contents.
Not configured: no Loki datasource in Grafana, no blackbox exporter (the structural downs have no real uptime signal), no Alertmanager vhost.

Citations

dankclaw/observability/quadlet/{prometheus,alertmanager,grafana,loki,alloy,node-exporter}.container
scripts/deploy-remote.sh:219-300 (copiers + relay), :521-536 (health waits), :477-486 (grafana env seed)
scripts/signal-relay.py:26-49, scripts/signal-relay.service
shared/prometheus/prometheus.yml, shared/prometheus/alertmanager.yml, shared/prometheus/rules/llm-fleet.yml
shared/grafana/provisioning/, shared/loki/loki-config.yaml, shared/alloy/config.alloy
tests/observability_quadlet.bats (boot persistence, no wildcard publish, probe addresses, delivery contract)
Host evidence 2026-10-18: ss -ltnp, podman ps, systemctl --user is-enabled|is-active, curl on 127.0.0.1:9090/api/v1/{targets,rules,alerts}, 127.0.0.1:3100/loki/api/v1/labels, 127.0.0.1:12345/metrics, 127.0.0.1:3002/api/health, 192.168.0.143:18093/health, journalctl --user -u signal-relay

6.9 KiB Raw Blame History