Files

T

Frank 327e78b950 docs(vault): destroy the dead weight — 53 pages and the change diary

The vault was 155 pages / 31,210 lines, most of it measurement ledgers for
payloads that no longer run. A throughput figure for a runtime nobody operates is
not a record: AI moves faster than the vault, so the number is wrong before
anyone reads it, and a wrong number in a runbook gets believed. Per the new
purpose test in `AGENTS.md` → *Documentation Policy*, it is destroyed.

Gone (`.AGENTTRASH/20261019-destruction/`, KILL-LIST.txt is the manifest):
- the `frank-medium` endpoint and its five era pages, `frank-npu`, FastFlowLM,
  the ikllama/kingjones/dealign/frank-redo eras, the ling/swift15/freetoken/ubatch
  campaign pages, qwen3.8 spec-matrix and strix-optimizations, the two
  "Running Qwen3.6 …" pages;
- the whole `Pipeline Parallelism/` folder (no live script passes a pipeline flag)
  and the multi-device/x4-pair/hybrid/offload experiment pages;
- every vLLM page — no vLLM endpoint exists on this fleet;
- `LLM/deployment.md` (`Architecture/Deployment Flow.md` is the test-referenced
  owner) and the external surveys (SGLang, Lemonade, llama.cpp forks/diff,
  self-hosted app census, the TTS survey) that were obsolete by construction;
- `log.md`, 2,380 lines. Git holds the chronology; a causal law belongs to the
  page that owns the thing it governs, never to a diary.

Survivors: 101 pages, 17,542 lines, 0 OKF violations, 0 unresolved links. Eras
tables keep the window, payload, runtime, reason and rollback pointer; the rate
columns are blanked. The live shapes stay verbatim — those numbers are what the
deployer and `models.yml` consume.

2026-10-19 10:56:37 -07:00

6.9 KiB

Raw Blame History

type, title, description, tags, timestamp

type

title

description

Observability on Dankclaw

Summary

Six containers and one host-side process: Prometheus (metrics), Grafana (dashboards), Loki (log store), Alloy (log shipper), node-exporter (host metrics) and Alertmanager (alert delivery), with signal-relay on the host carrying every firing group to Signal. All of it is Quadlet-managed, boot-persistent and loopback-only, deployed by scripts/deploy-remote.sh → deploy_observability. The metrics half works end to end; the log half is configured, healthy, and carries no data.

This body was rewritten on 2026-10-18 from live evidence: the previous version was deleted down to its front matter by commit 93d1181 (the "OKF v0.1 compliance pass"), while sibling pages kept pointing at a "state note" on this page that no longer existed. Every statement below carries the command or file that proves it today.

The stack

Service	Container	Published	Unit	Page
Prometheus	`docker.io/prom/prometheus:latest`	`127.0.0.1:9090`	`prometheus.container`	Prometheus
Alertmanager	`docker.io/prom/alertmanager:latest`	`127.0.0.1:9093`	`alertmanager.container`	Alertmanager
Grafana	`docker.io/grafana/grafana:latest`	`127.0.0.1:3002` → 3000	`grafana.container`	Grafana
Loki	`localhost/franksdank/loki:latest`	`127.0.0.1:3100`	`loki.container`	Loki
Alloy	`localhost/franksdank/alloy:latest`	`127.0.0.1:12345`	`alloy.container`	Alloy
node-exporter	`docker.io/prom/node-exporter:latest`	host netns: `127.0.0.1:9100` + `192.168.0.143:9100`	`node-exporter.container`	Node-Exporter
Signal relay	host `python3`, not a container	`192.168.0.143:18093`	`scripts/signal-relay.service` (user unit)	Alertmanager

Every unit lives in dankclaw/observability/quadlet/, carries Restart=always/RestartSec=10 and WantedBy=default.target, is installed at ~/.config/containers/systemd/<name>.container, and reports generated + active (systemctl --user is-enabled|is-active for all six, 2026-10-18; linger is on for dank). signal-relay.service reports enabled + active the same day, and homepage-status.timer — the root unit that probes all of these on loopback — is enabled + active (deploy-remote.sh:319-321 copies the probe registry shared/homepage/services.json to /srv/homepage). Dankserver runs one member of this family, node-exporter, as its own unit (dankserver/observability/quadlet/node-exporter.container, generated + active, binding 127.0.0.1:9100 + 192.168.0.235:9100).

Exposure

The five HTTP surfaces publish on 127.0.0.1 only — ss -ltnp (2026-10-18) shows 127.0.0.1:9090, 127.0.0.1:9093, 127.0.0.1:3002, 127.0.0.1:3100 127.0.0.1:12345 and no wildcard — because each of them is unauthenticated inside, and several carry lifecycle or query APIs. Browser access goes through Caddy: each vhost (dankclaw/Caddyfile:128-145) proxies to the guard anubis:8923, and the post-auth handlers route by container name on dankclaw_default — loki.franksdank.cloud → loki:3100 (:364-366) grafana.franksdank.cloud → grafana:3000 (:374-376) prometheus.franksdank.cloud → prometheus:9090 (:379-381). Alertmanager has no vhost at all. node-exporter is the exception that must live in the host network namespace, and it binds exactly two addresses rather than the wildcard — see Node-Exporter.

The relay's bind is the one address a bridge container can reach on the host: 192.168.0.143:18093, with iptables INPUT admitting loopback, the host itself and the podman bridge subnet 10.89.0.0/16 before the DROP (scripts/deploy-remote.sh:255-262).

Deployment

./scripts/deploy.sh dankclaw --service <name> runs deploy_observability which installs the unit, copies that service's config (copy_prometheus_config, copy_alertmanager_config, copy_loki_config copy_alloy_config, copy_grafana_provisioning, plus install_signal_relay with the alertmanager deploy), then waits on the loopback health endpoint (/prometheus: /-/healthy, /alertmanager: /-/healthy /grafana: /api/health, /loki: /ready, /alloy: /-/healthy /node-exporter: /metrics). Startup order follows the data dependencies — grafana after prometheus, alloy after loki (grafana.container:32-33, alloy.container:38-39). See Observability Provisioning for what is provisioned as code and how to verify it through the APIs.

Current state (2026-10-18)

Metrics: working. curl -s http://127.0.0.1:9090/api/v1/targets reports eight up targets; the four downs are structural (the comfyui and services HTML probes, and Gitea's disabled metrics endpoint), not outages. Rule set: 4 alert rules + 2 recording rules (/api/v1/rules, matching shared/prometheus/rules/llm-fleet.yml).
Delivery: working. Alertmanager's single signal receiver webhooks the relay, and journalctl --user -u signal-relay records successful sent to +12062476336: lines for every rule in the set.
Dashboards: working. One provisioned Prometheus datasource (uid prometheus) and the llm-fleet dashboard, both installed from shared/grafana/provisioning/.
Logs: configured, empty. Alloy ships zero lines (journal counter 0, both file globs refused) and Loki therefore holds zero streams — /opt/loki/data is a few hundred KB of WAL/compactor bookkeeping with no chunk files (8 files total by find, 2026-10-18) and GET /loki/api/v1/labels returns no label names. See Alloy → Delivery state and Loki → Contents.
Not configured: no Loki datasource in Grafana, no blackbox exporter (the structural downs have no real uptime signal), no Alertmanager vhost.

Citations

dankclaw/observability/quadlet/{prometheus,alertmanager,grafana,loki,alloy,node-exporter}.container
scripts/deploy-remote.sh:219-300 (copiers + relay), :521-536 (health waits), :477-486 (grafana env seed)
scripts/signal-relay.py:26-49, scripts/signal-relay.service
shared/prometheus/prometheus.yml, shared/prometheus/alertmanager.yml, shared/prometheus/rules/llm-fleet.yml
shared/grafana/provisioning/, shared/loki/loki-config.yaml, shared/alloy/config.alloy
tests/observability_quadlet.bats (boot persistence, no wildcard publish, probe addresses, delivery contract)
Host evidence 2026-10-18: ss -ltnp, podman ps, systemctl --user is-enabled|is-active, curl on 127.0.0.1:9090/api/v1/{targets,rules,alerts}, 127.0.0.1:3100/loki/api/v1/labels, 127.0.0.1:12345/metrics, 127.0.0.1:3002/api/health, 192.168.0.143:18093/health, journalctl --user -u signal-relay

6.9 KiB Raw Blame History