Files

T

Frank a5fe045b58 docs(vault): one fact, one owner — repair the doubled pages, single-source the facts

Structural repair:
- 13 pages were the same document twice (each exactly 2x its HEAD size),
  a second frontmatter block whose closing fence had been glued onto a body
  line by a frontmatter rewrite that re-read its own output. Un-doubled by
  cutting at the second frontmatter and keeping the copy whose body is a
  superset; the orphan census reports 0 lost tokens against HEAD.
- log.md: one entry per date, bare `## YYYY-MM-DD` headings, newest first —
  66 dated headings merged into 34, titles demoted to bold leads, every
  content line verified present. The file had been descending at the head
  and ascending at the tail.

Ownership moves:
- New `LLM/frank-medium.md` is the locator for the retired Dankserver
  endpoint: era windows quoted from each era page's own banner, no numbers
  restated, and the two unresolved date conflicts recorded instead of
  silently resolved. `frank-archon.md` links to it instead of restating the
  pre-rename window.
- The four Pipeline-Parallelism stubs (frontmatter only, no body) are gone;
  `Multi-Node Runtimes.md` owns runtime construction, launch, mixed-vendor
  verdicts and cross-split spec decode, and the four links that pointed at
  the stubs were repointed at it or at the drafter page.
- Authentik's three stub pages fold into `Authentik.md`, with the process
  detail and the outpost reference split out so the parent stays under the
  length hint.

Verified corrections (host census + live probe, 2026-10-19):
- The MXFP4 checkpoint two pages called the "on-disk rollback" is gone:
  /home/dank/models/ holds StillDeadcode, rad and tpl.jinja. Stated once, on
  the era page that owns that payload; the swap page now links to it.
- frank-medium-radiance-mxfp4: the 4.24/7 accept rate belongs to the DFlash2
  arm measured on the FP8 target, not to the MXFP4 era; the 3-family soak run
  census is kept once (155 requests) instead of twice (~120 and 155).
- Fleet shapes re-confirmed against the live endpoints before touching any
  page: 3x200000 fp8 KV, 2x200704, 2x262144.

Index rows are regenerated, not hand-carved. Net -1,849 vault lines.

2026-10-19 05:00:43 -07:00

10 KiB

Raw Blame History

type, title, description, tags, timestamp

type

title

description

Observability Provisioning

The monitoring stack is provisioned as code: dashboards, datasources, scrape jobs and alert rules live in the repo and ride the normal deploy flow. No hand-edits inside containers — they are wiped on recreate and never committed.

The config triad

Repo path	What it holds
`shared/prometheus/prometheus.yml`	scrape jobs (global + per-host)
`shared/prometheus/rules/llm-fleet.yml`	alert rules + recording rules
`shared/grafana/provisioning/`	datasources + dashboards (provisioned tree)

Targets and datasources reference container names (grafana:3000, loki:3100 — shared/prometheus/prometheus.yml:52-61), never loopback-published ports; alloy is not scraped at all (no job in the config — its 127.0.0.1:12345 UI is only probed by the homepage timer, shared/homepage/services.json:233). The LLM endpoints split by topology: the two same-host ones (frank-dragoon :7770, frank-zealot :7771) are scraped as host.containers.internal:<port> because a bridge container cannot reach a host-netns bind at the host's LAN IP (prometheus.yml:89-99), and only the endpoint on another machine (192.168.0.235:7770) is addressed by LAN IP (prometheus.yml:128-131). Prometheus 3 refuses HTML responses as metrics — HTML-scraped "probes" are structurally inert and need a blackbox exporter (see Stale Signal Audit).

The deploy path

scripts/deploy-remote.sh carries one copier per config surface, all called from deploy_observability:

copy_prometheus_config → /opt/prometheus/config/prometheus.yml and config/rules/llm-fleet.yml (:219-227)
copy_alertmanager_config → /opt/alertmanager/config/alertmanager.yml (:232-235)
install_signal_relay → the user unit + iptables + the health wait on 192.168.0.143:18093/health (:243-274)
copy_loki_config → /opt/loki/config/loki-config.yaml (:276-279)
copy_alloy_config → /opt/alloy/config/config.alloy (:281-284)
copy_grafana_provisioning → /opt/grafana/provisioning/ (:286-300)

scripts/deploy-plan.py maps shared/prometheus/** to prometheus (plus alertmanager when the changed file is alertmanager.yml, :251-256), shared/loki/** to loki and shared/alloy/** to alloy (:257-260), and shared/grafana/** to grafana (:261-264); changes under dankclaw/observability/ select the touched unit and its dependency consumer (:321-328), and a change to either relay file redeploys alertmanager (:399-402). So deploy.sh dankclaw --service grafana (etc.) restarts only what the change touched. Seam tests in tests/observability_quadlet.bats pin the unit files exist and are boot-persistent/self-healing (:36-62), that no unit publishes on a wildcard (:63-81), that the copiers install each unit (:250), that grafana provisioning is declared in git and installed (:345), and that the provisioned paths match the units' mounts.

Entry points

Add a scrape target — edit shared/prometheus/prometheus.yml. Decide the vantage first: container name if the exporter is a container on dankclaw_default; host.containers.internal:<port> if it is a process in the host's own network namespace on this same machine (a bridge container cannot reach the host's LAN IP at an unpublished port); the host's LAN IP only if it is a process on a different machine; a blackbox module if it is an HTTP probe, not a metrics endpoint.

Add an alert/recording rule — edit shared/prometheus/rules/llm-fleet.yml. Current delivered set: 4 alert rules + 2 recording rules — LlmRequestsQueueing, LlmPrefixCacheCollapse, LlmSpecDecodeOff, LlmEndpointDown, plus job:llm_prefix_cache_reuse:rate5m and job:llm_spec_accept_length:rate5m (shared/prometheus/rules/llm-fleet.yml:32-110), confirmed live: curl -s http://127.0.0.1:9090/api/v1/rules (2026-10-18) reports 4 alerting + 2 recording. Rules reload with the prometheus service.

Add a datasource — drop a YAML file into shared/grafana/provisioning/datasources/. Point it at the container name of the backend (http://prometheus:9090), not loopback.

Add a dashboard — drop the JSON into shared/grafana/provisioning/dashboards/. Provisioned dashboards live in unified storage, not (only) the SQLite table — grep the API, not the DB, when verifying (see the verify section of Fleet Audit and Posture). (unverified 2026-10-18: the storage-backend claim needs an admin password to confirm against grafana.db and the API side by side)

Verify through the APIs, not the filesystem

# prometheus: rules and targets actually loaded
curl -s http://127.0.0.1:9090/api/v1/rules | python3 -m json.tool | head
curl -s "http://127.0.0.1:9090/api/v1/targets?state=active" | \
  python3 -c "import json,sys; [print(t['labels'].get('job'), t['health'], (t['lastError'] or '')[:60]) for t in json.load(sys.stdin)['data']['activeTargets']]"
# grafana: datasource + dashboard provisioned
curl -s -u admin:<pw> http://127.0.0.1:3002/api/datasources | python3 -m json.tool | grep -E '"name"|"url"'

The two Prometheus queries were run on 2026-10-18 and are what every number on this page comes from: 4 alerting + 2 recording rules, and up for prometheus, node-dankclaw, node-dankserver, grafana, loki, frank-zealot, frank-dragoon, frank-archon with the four structural downs listed in Prometheus. The Grafana query needs the local admin password and was not run here; the filesystem side (ls /opt/grafana/provisioning/*) is proof of installation only — see the next paragraph.

Health answers from a stale process and provisioned resources can hide in unified storage — a file existing on disk is not proof the running server loaded it. See Stale Signal Audit for the never-green-target decision table (refused = outage, 404/HTML = config defect).

Current delivered state (delivery chain live 2026-09-17, re-verified 2026-10-18)

Prometheus → Alertmanager (dankclaw/observability/quadlet/alertmanager.container, loopback :9093) → host-side Signal relay (scripts/signal-relay.py, user unit, 192.168.0.143:18093 — the LAN IP, the one bind a container can reach via host.containers.internal) → signal-cli, addressed by number +12062476336 (scripts/signal-relay.py:36-43). Every hop re-checked 2026-10-18: ss -ltnp shows python3 on 192.168.0.143:18093, curl -s http://192.168.0.143:18093/health → ok, and journalctl --user -u signal-relay ends with successful sent to +12062476336: lines for LlmEndpointDown, LlmRequestsQueueing, LlmSpecDecodeOff and LlmPrefixCacheCollapse (firing and resolved). Some targets read down for structural reasons (HTML-scraped "probes", Gitea metrics off); the remedy is the blackbox exporter. See Prometheus → Targets.

A llama.cpp endpoint is invisible unless it opts in. Until 2026-10-18, up{job="frank-zealot"} was 0 for the entire retained history of the metric — the oldest sample, 2026-10-11, is already 0 — while the service itself answered /health 200 the whole time. Cause: llama.cpp only exports /metrics when it is started with the flag, so the scrape was refused at the protocol layer (the flag and its opt-out are on frank-zealot) and LlmEndpointDown (critical) stayed firing: the relay re-sent it on the 4 h repeat_interval for days (journalctl --user -u signal-relay — firing sends at 01:37, 05:42, 09:47, 13:52, 17:57 and 22:02 on 2026-10-12). It finally resolved at 2026-10-18T01:53:21-07:00, after the metrics opt-in landed that day; the target reports up since (curl -s http://127.0.0.1:9090/api/v1/targets 2026-10-18). How many payload generations it spanned is not readable from any host artifact. (unverified 2026-10-18)

Two laws fall out of it. A target can read down beside a healthy service, so read lastError from /api/v1/targets before believing an outage. And rule state lives in Prometheus' process, not on disk — a restart re-stamps activeAt on a long-standing alert, so a "fresh" alert after a reboot is not evidence of a fresh fault. (Alertmanager by contrast does persist: --storage.path=/alertmanager on a host volume keeps silences and the notification log — dankclaw/observability/quadlet/alertmanager.container:39-42, /opt/alertmanager/data.)

Remaining known downs are the structural ones above (HTML-scraped probes, Gitea metrics, ComfyUI), and frank-dragoon exports /metrics natively (frank-dragoon), which is why frank-dragoon has always been green. The active-alert count is a moment-in-time value and should never be documented as a constant: /api/v1/alerts was empty at the 2026-10-18 morning census, held one firing LlmPrefixCacheCollapse (activeAt 11:10:48Z, value 2.6e-4) when sampled minutes later, and was empty again by 11:56Z; the relay logged the matching firing and resolved sends at 04:26:21-07:00 and 04:41:21-07:00.

What that zealot scrape measured on 2026-10-18. As of ~04:30 PDT the production frank-zealot root unit was stopped on purpose for a benchmark window on the 3090 — the repo Benchmarking Policy says stop the production service before benching — and :7771 answered from a manually booted candidate arm (systemctl is-active frank-zealot → failed after stop-sigterm timed out. Killing, which is llama.cpp ignoring SIGTERM past TimeoutStopSec, the known shutdown shape for these engines). So up{job="frank-zealot"} == 1 that day measured the trial, not the production shape in scripts/frank-zealot.env — that shape is frank-zealot's to state, and it is the repo contract. The unit stayed enabled; ./scripts/deploy.sh dankclaw --service frank-zealot reinstalls and starts it, restoring boot persistence and self-healing. Never quote a candidate arm's flags or VRAM as fleet state — the process and its shape changed between two censuses ten minutes apart.

Observability — stack topology and the alert delivery chain
Stale Signal Audit — classifying down targets before "fixing" them
Quadlet Lifecycle — the units that host these services
Deploy State Triage — how the planner selects these services

10 KiB Raw Blame History