dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity
Files
a5fe045b588b1240d3df4d20bce77f8491b19a03
serverAppConfigs/serverAppConfigs-docs/Infrastructure/Observability.md
T
Frank a5fe045b58 docs(vault): one fact, one owner — repair the doubled pages, single-source the facts 
Structural repair:
- 13 pages were the same document twice (each exactly 2x its HEAD size),
  a second frontmatter block whose closing fence had been glued onto a body
  line by a frontmatter rewrite that re-read its own output. Un-doubled by
  cutting at the second frontmatter and keeping the copy whose body is a
  superset; the orphan census reports 0 lost tokens against HEAD.
- log.md: one entry per date, bare `## YYYY-MM-DD` headings, newest first —
  66 dated headings merged into 34, titles demoted to bold leads, every
  content line verified present. The file had been descending at the head
  and ascending at the tail.

Ownership moves:
- New `LLM/frank-medium.md` is the locator for the retired Dankserver
  endpoint: era windows quoted from each era page's own banner, no numbers
  restated, and the two unresolved date conflicts recorded instead of
  silently resolved. `frank-archon.md` links to it instead of restating the
  pre-rename window.
- The four Pipeline-Parallelism stubs (frontmatter only, no body) are gone;
  `Multi-Node Runtimes.md` owns runtime construction, launch, mixed-vendor
  verdicts and cross-split spec decode, and the four links that pointed at
  the stubs were repointed at it or at the drafter page.
- Authentik's three stub pages fold into `Authentik.md`, with the process
  detail and the outpost reference split out so the parent stays under the
  length hint.

Verified corrections (host census + live probe, 2026-10-19):
- The MXFP4 checkpoint two pages called the "on-disk rollback" is gone:
  /home/dank/models/ holds StillDeadcode, rad and tpl.jinja. Stated once, on
  the era page that owns that payload; the swap page now links to it.
- frank-medium-radiance-mxfp4: the 4.24/7 accept rate belongs to the DFlash2
  arm measured on the FP8 target, not to the MXFP4 era; the 3-family soak run
  census is kept once (155 requests) instead of twice (~120 and 155).
- Fleet shapes re-confirmed against the live endpoints before touching any
  page: 3x200000 fp8 KV, 2x200704, 2x262144.

Index rows are regenerated, not hand-carved. Net -1,849 vault lines.
2026-10-19 05:00:43 -07:00

6.9 KiB
Raw Blame History

type, title, description, tags, timestamp
type title description tags timestamp
reference Observability on Dankclaw The observability stack: what runs, how it is deployed, how it is exposed, and what is still unconfigured.
observability
prometheus
grafana
loki
alloy
node-exporter
quadlet
dankclaw
2026-09-11

Observability on Dankclaw

Summary

Six containers and one host-side process: Prometheus (metrics), Grafana (dashboards), Loki (log store), Alloy (log shipper), node-exporter (host metrics) and Alertmanager (alert delivery), with signal-relay on the host carrying every firing group to Signal. All of it is Quadlet-managed, boot-persistent and loopback-only, deployed by scripts/deploy-remote.sh → deploy_observability. The metrics half works end to end; the log half is configured, healthy, and carries no data.

This body was rewritten on 2026-10-18 from live evidence: the previous version was deleted down to its front matter by commit 93d1181 (the "OKF v0.1 compliance pass"), while sibling pages kept pointing at a "state note" on this page that no longer existed. Every statement below carries the command or file that proves it today.

The stack

Service Container Published Unit Page
Prometheus docker.io/prom/prometheus:latest 127.0.0.1:9090 prometheus.container Prometheus
Alertmanager docker.io/prom/alertmanager:latest 127.0.0.1:9093 alertmanager.container Alertmanager
Grafana docker.io/grafana/grafana:latest 127.0.0.1:3002 → 3000 grafana.container Grafana
Loki localhost/franksdank/loki:latest 127.0.0.1:3100 loki.container Loki
Alloy localhost/franksdank/alloy:latest 127.0.0.1:12345 alloy.container Alloy
node-exporter docker.io/prom/node-exporter:latest host netns: 127.0.0.1:9100 + 192.168.0.143:9100 node-exporter.container Node-Exporter
Signal relay host python3, not a container 192.168.0.143:18093 scripts/signal-relay.service (user unit) Alertmanager

Every unit lives in dankclaw/observability/quadlet/, carries Restart=always/RestartSec=10 and WantedBy=default.target, is installed at ~/.config/containers/systemd/<name>.container, and reports generated + active (systemctl --user is-enabled|is-active for all six, 2026-10-18; linger is on for dank). signal-relay.service reports enabled + active the same day, and homepage-status.timer — the root unit that probes all of these on loopback — is enabled + active (deploy-remote.sh:319-321 copies the probe registry shared/homepage/services.json to /srv/homepage). Dankserver runs one member of this family, node-exporter, as its own unit (dankserver/observability/quadlet/node-exporter.container, generated + active, binding 127.0.0.1:9100 + 192.168.0.235:9100).

Exposure

The five HTTP surfaces publish on 127.0.0.1 only — ss -ltnp (2026-10-18) shows 127.0.0.1:9090, 127.0.0.1:9093, 127.0.0.1:3002, 127.0.0.1:3100, 127.0.0.1:12345 and no wildcard — because each of them is unauthenticated inside, and several carry lifecycle or query APIs. Browser access goes through Caddy: each vhost (dankclaw/Caddyfile:128-145) proxies to the guard anubis:8923, and the post-auth handlers route by container name on dankclaw_default — loki.franksdank.cloud → loki:3100 (:364-366), grafana.franksdank.cloud → grafana:3000 (:374-376), prometheus.franksdank.cloud → prometheus:9090 (:379-381). Alertmanager has no vhost at all. node-exporter is the exception that must live in the host network namespace, and it binds exactly two addresses rather than the wildcard — see Node-Exporter.

The relay's bind is the one address a bridge container can reach on the host: 192.168.0.143:18093, with iptables INPUT admitting loopback, the host itself and the podman bridge subnet 10.89.0.0/16 before the DROP (scripts/deploy-remote.sh:255-262).

Deployment

./scripts/deploy.sh dankclaw --service <name> runs deploy_observability, which installs the unit, copies that service's config (copy_prometheus_config, copy_alertmanager_config, copy_loki_config, copy_alloy_config, copy_grafana_provisioning, plus install_signal_relay with the alertmanager deploy), then waits on the loopback health endpoint (/prometheus: /-/healthy, /alertmanager: /-/healthy, /grafana: /api/health, /loki: /ready, /alloy: /-/healthy, /node-exporter: /metrics). Startup order follows the data dependencies — grafana after prometheus, alloy after loki (grafana.container:32-33, alloy.container:38-39). See Observability Provisioning for what is provisioned as code and how to verify it through the APIs.

Current state (2026-10-18)

  • Metrics: working. curl -s http://127.0.0.1:9090/api/v1/targets reports eight up targets; the four downs are structural (the comfyui and services HTML probes, and Gitea's disabled metrics endpoint), not outages. Rule set: 4 alert rules + 2 recording rules (/api/v1/rules, matching shared/prometheus/rules/llm-fleet.yml).
  • Delivery: working. Alertmanager's single signal receiver webhooks the relay, and journalctl --user -u signal-relay records successful sent to +12062476336: lines for every rule in the set.
  • Dashboards: working. One provisioned Prometheus datasource (uid prometheus) and the llm-fleet dashboard, both installed from shared/grafana/provisioning/.
  • Logs: configured, empty. Alloy ships zero lines (journal counter 0, both file globs refused) and Loki therefore holds zero streams — /opt/loki/data is a few hundred KB of WAL/compactor bookkeeping with no chunk files (8 files total by find, 2026-10-18) and GET /loki/api/v1/labels returns no label names. See Alloy → Delivery state and Loki → Contents.
  • Not configured: no Loki datasource in Grafana, no blackbox exporter (the structural downs have no real uptime signal), no Alertmanager vhost.

Citations

  • dankclaw/observability/quadlet/{prometheus,alertmanager,grafana,loki,alloy,node-exporter}.container
  • scripts/deploy-remote.sh:219-300 (copiers + relay), :521-536 (health waits), :477-486 (grafana env seed)
  • scripts/signal-relay.py:26-49, scripts/signal-relay.service
  • shared/prometheus/prometheus.yml, shared/prometheus/alertmanager.yml, shared/prometheus/rules/llm-fleet.yml
  • shared/grafana/provisioning/, shared/loki/loki-config.yaml, shared/alloy/config.alloy
  • tests/observability_quadlet.bats (boot persistence, no wildcard publish, probe addresses, delivery contract)
  • Host evidence 2026-10-18: ss -ltnp, podman ps, systemctl --user is-enabled|is-active, curl on 127.0.0.1:9090/api/v1/{targets,rules,alerts}, 127.0.0.1:3100/loki/api/v1/labels, 127.0.0.1:12345/metrics, 127.0.0.1:3002/api/health, 192.168.0.143:18093/health, journalctl --user -u signal-relay
Reference in New Issue View Git Blame Copy Permalink