docs(vault): fleet-wide fact re-verification pass (67 pages)
The docs-fact census wave re-read the live hosts and re-keyed every page it could: runtime/payload identities, ports, sizes, ledger figures, unit state, and the retirement banners that drifted as the fleet moved. The wave was interrupted mid-run, so this commit carries only what passed the integrity gates: every touched page parses (front matter or H1+status convention intact, balanced fences, no truncated tables, trailing newline), the docs_completeness and link-integrity seams are green, and bats is 384/384. Numbers that this session proved on live hosts are keyed to the measured value, not the page's earlier claim.
This commit is contained in:
Frank committed
1 parent
466fd477d8
commit
e7f0d93c86
67 files changed
+2619
-1152
No files matched your search
@@ -21,37 +21,61 @@ committed.
|
||||
| `shared/prometheus/rules/llm-fleet.yml` | alert rules + recording rules |
|
||||
| `shared/grafana/provisioning/` | datasources + dashboards (provisioned tree) |
|
||||
|
||||
Targets and datasources reference **container names** (`loki:3100`,
|
||||
`alloy:12345`), never loopback-published ports; the LLM endpoints are reached
|
||||
by LAN IP (`192.168.0.143:7770`, `:7771`, `192.168.0.235:7770`) because they
|
||||
are host-netns services, not containers. Prometheus 3 refuses HTML responses
|
||||
as metrics — HTML-scraped "probes" are structurally inert and need a blackbox
|
||||
exporter (see [[Stale Signal Audit]]).
|
||||
Targets and datasources reference **container names** (`grafana:3000`,
|
||||
`loki:3100` — `shared/prometheus/prometheus.yml:52-61`), never
|
||||
loopback-published ports; `alloy` is **not scraped at all** (no job in the
|
||||
config — its `127.0.0.1:12345` UI is only probed by the homepage timer,
|
||||
`shared/homepage/services.json:233`). The LLM endpoints split by topology: the
|
||||
two same-host ones (`frank-dragoon` `:7770`, `frank-zealot` `:7771`) are
|
||||
scraped as `host.containers.internal:<port>` because a bridge container cannot
|
||||
reach a host-netns bind at the host's LAN IP (`prometheus.yml:89-99`), and only
|
||||
the endpoint on another machine (`192.168.0.235:7770`) is addressed by LAN IP
|
||||
(`prometheus.yml:128-131`). Prometheus 3 refuses HTML responses as metrics —
|
||||
HTML-scraped "probes" are structurally inert and need a blackbox exporter (see
|
||||
[[Stale Signal Audit]]).
|
||||
|
||||
## The deploy path
|
||||
|
||||
`scripts/deploy-remote.sh` carries two copier functions:
|
||||
`scripts/deploy-remote.sh` carries one copier per config surface, all called
|
||||
from `deploy_observability`:
|
||||
|
||||
- `copy_prometheus_config` → `/opt/prometheus/config/` on the target
|
||||
- `copy_grafana_provisioning` → `/opt/grafana/provisioning/`
|
||||
- `copy_prometheus_config` → `/opt/prometheus/config/prometheus.yml` **and** `config/rules/llm-fleet.yml` (`:219-227`)
|
||||
- `copy_alertmanager_config` → `/opt/alertmanager/config/alertmanager.yml` (`:232-235`)
|
||||
- `install_signal_relay` → the user unit + iptables + the health wait on `192.168.0.143:18093/health` (`:243-274`)
|
||||
- `copy_loki_config` → `/opt/loki/config/loki-config.yaml` (`:276-279`)
|
||||
- `copy_alloy_config` → `/opt/alloy/config/config.alloy` (`:281-284`)
|
||||
- `copy_grafana_provisioning` → `/opt/grafana/provisioning/` (`:286-300`)
|
||||
|
||||
`scripts/deploy-plan.py` maps changes under `shared/prometheus/**` and
|
||||
`shared/grafana/**` to the `prometheus`/`grafana` services, so
|
||||
`deploy.sh dankclaw --service grafana` (etc.) restarts only what the change
|
||||
touched. Seam tests in `tests/observability_quadlet.bats` pin: unit files
|
||||
exist, the copier functions exist, and the provisioned paths match the units'
|
||||
mounts.
|
||||
`scripts/deploy-plan.py` maps `shared/prometheus/**` to `prometheus` (plus
|
||||
`alertmanager` when the changed file is `alertmanager.yml`, `:251-256`),
|
||||
`shared/loki/**` to `loki` and `shared/alloy/**` to `alloy` (`:257-260`), and
|
||||
`shared/grafana/**` to `grafana` (`:261-264`); changes under
|
||||
`dankclaw/observability/` select the touched unit and its dependency consumer
|
||||
(`:321-328`), and a change to either relay file redeploys `alertmanager`
|
||||
(`:399-402`). So `deploy.sh dankclaw --service grafana` (etc.) restarts only
|
||||
what the change touched. Seam tests in `tests/observability_quadlet.bats` pin
|
||||
the unit files exist and are boot-persistent/self-healing (`:36-62`), that no
|
||||
unit publishes on a wildcard (`:63-81`), that the copiers install each unit
|
||||
(`:250`), that grafana provisioning is declared in git and installed
|
||||
(`:345`), and that the provisioned paths match the units' mounts.
|
||||
|
||||
## Entry points
|
||||
|
||||
**Add a scrape target** — edit `shared/prometheus/prometheus.yml`. Decide the
|
||||
vantage first: container name if the exporter is a container on the same
|
||||
netns; host LAN IP if it is a host process (LLM endpoints); a blackbox module
|
||||
if it is an HTTP probe, not a metrics endpoint.
|
||||
vantage first: container name if the exporter is a container on
|
||||
`dankclaw_default`; `host.containers.internal:<port>` if it is a process in the
|
||||
host's own network namespace **on this same machine** (a bridge container
|
||||
cannot reach the host's LAN IP at an unpublished port); the host's LAN IP only
|
||||
if it is a process on a **different** machine; a blackbox module if it is an
|
||||
HTTP probe, not a metrics endpoint.
|
||||
|
||||
**Add an alert/recording rule** — edit `shared/prometheus/rules/llm-fleet.yml`.
|
||||
Current delivered set: 5 alert rules + 1 recording rule for the LLM fleet.
|
||||
Rules reload with the prometheus service.
|
||||
Current delivered set: **4 alert rules + 2 recording rules** —
|
||||
`LlmRequestsQueueing`, `LlmPrefixCacheCollapse`, `LlmSpecDecodeOff`,
|
||||
`LlmEndpointDown`, plus `job:llm_prefix_cache_reuse:rate5m` and
|
||||
`job:llm_spec_accept_length:rate5m` (`shared/prometheus/rules/llm-fleet.yml:32-110`),
|
||||
confirmed live: `curl -s http://127.0.0.1:9090/api/v1/rules` (2026-10-18)
|
||||
reports 4 alerting + 2 recording. Rules reload with the prometheus service.
|
||||
|
||||
**Add a datasource** — drop a YAML file into
|
||||
`shared/grafana/provisioning/datasources/`. Point it at the container name of
|
||||
@@ -60,7 +84,9 @@ the backend (`http://prometheus:9090`), not loopback.
|
||||
**Add a dashboard** — drop the JSON into
|
||||
`shared/grafana/provisioning/dashboards/`. Provisioned dashboards live in
|
||||
unified storage, not (only) the SQLite table — grep the API, not the DB, when
|
||||
verifying (see the verify section of [[Fleet Audit and Posture]]).
|
||||
verifying (see the verify section of [[Fleet Audit and Posture]]). *(unverified
|
||||
2026-10-18: the storage-backend claim needs an admin password to confirm
|
||||
against `grafana.db` and the API side by side)*
|
||||
|
||||
## Verify through the APIs, not the filesystem
|
||||
|
||||
@@ -73,34 +99,80 @@ curl -s "http://127.0.0.1:9090/api/v1/targets?state=active" | \
|
||||
curl -s -u admin:<pw> http://127.0.0.1:3002/api/datasources | python3 -m json.tool | grep -E '"name"|"url"'
|
||||
```
|
||||
|
||||
The two Prometheus queries were run on 2026-10-18 and are what every number on
|
||||
this page comes from: 4 alerting + 2 recording rules, and `up` for
|
||||
`prometheus`, `node-dankclaw`, `node-dankserver`, `grafana`, `loki`,
|
||||
`frank-zealot`, `frank-dragoon`, `frank-archon` with the four structural downs
|
||||
listed in [[Prometheus]]. The Grafana query needs the local admin password and
|
||||
was not run here; the filesystem side (`ls /opt/grafana/provisioning/*`) is
|
||||
proof of installation only — see the next paragraph.
|
||||
|
||||
Health answers from a stale process and provisioned resources can hide in
|
||||
unified storage — a file existing on disk is not proof the running server
|
||||
loaded it. See [[Stale Signal Audit]] for the never-green-target decision
|
||||
table (refused = outage, 404/HTML = config defect).
|
||||
|
||||
## Current delivered state (2026-09-17)
|
||||
## Current delivered state (delivery chain live 2026-09-17, re-verified 2026-10-18)
|
||||
|
||||
Prometheus → Alertmanager (`dankclaw/observability/quadlet/alertmanager.container`,
|
||||
loopback `:9093`) → host-side Signal relay (`scripts/signal-relay.py`, user
|
||||
unit, `192.168.0.143:18093` — the LAN IP, the one bind a container can reach
|
||||
via `host.containers.internal`) → signal-cli. Some targets read down for
|
||||
via `host.containers.internal`) → signal-cli, addressed by number
|
||||
`+12062476336` (`scripts/signal-relay.py:36-43`). Every hop re-checked
|
||||
2026-10-18: `ss -ltnp` shows `python3` on `192.168.0.143:18093`,
|
||||
`curl -s http://192.168.0.143:18093/health` → `ok`, and
|
||||
`journalctl --user -u signal-relay` ends with successful `sent to +12062476336:`
|
||||
lines for `LlmEndpointDown`, `LlmRequestsQueueing`, `LlmSpecDecodeOff` and
|
||||
`LlmPrefixCacheCollapse` (firing and resolved). Some targets read down for
|
||||
structural reasons (HTML-scraped "probes", Gitea metrics off); the remedy is
|
||||
the blackbox exporter. See [[Observability]].
|
||||
the blackbox exporter. See [[Prometheus]] → Targets.
|
||||
|
||||
**A llama.cpp endpoint is invisible unless it opts in.** `up{job="frank-zealot"}`
|
||||
has been 0 for the entire retained history of the metric — the oldest sample,
|
||||
2026-10-11, is already 0 — while the service itself answered `/health` 200 the
|
||||
whole time. Cause: llama.cpp answers **501 Not Implemented** on `/metrics` unless
|
||||
the server is started with `--metrics`, so the scrape was refused at the protocol
|
||||
layer and `LlmEndpointDown` (critical) sat active across four payload generations.
|
||||
Fixed 2026-10-18 by the `METRICS=1` knob in `scripts/frank-zealot.env`. Two laws
|
||||
fall out of it: a target can read down beside a healthy service, so read
|
||||
`lastError` from `/api/v1/targets` before believing an outage; and Alertmanager
|
||||
runs here without persistence, so every restart re-stamps `startsAt` on a
|
||||
long-standing alert — a "fresh" alert after a reboot is not evidence of a fresh
|
||||
fault. Remaining known downs are the structural ones above (HTML-scraped probes,
|
||||
Gitea metrics, ComfyUI), and halogen serves `/metrics` natively, which is why
|
||||
`frank-dragoon` has always been green.
|
||||
**A llama.cpp endpoint is invisible unless it opts in.** Until 2026-10-18,
|
||||
`up{job="frank-zealot"}` was 0 for the entire retained history of the metric —
|
||||
the oldest sample, 2026-10-11, is already 0 — while the service itself answered
|
||||
`/health` 200 the whole time. Cause: llama.cpp answers **501 Not Implemented**
|
||||
on `/metrics` unless the server is started with `--metrics`, so the scrape was
|
||||
refused at the protocol layer and `LlmEndpointDown` (critical) stayed firing:
|
||||
the relay re-sent it on the 4 h `repeat_interval` for days (`journalctl --user
|
||||
-u signal-relay` — firing sends at 01:37, 05:42, 09:47, 13:52, 17:57 and 22:02
|
||||
on 2026-10-12). It finally resolved at `2026-10-18T01:53:21-07:00`, after the
|
||||
`METRICS=1` knob added to `scripts/frank-zealot.env` that day; the target
|
||||
reports up since (`curl -s http://127.0.0.1:9090/api/v1/targets` 2026-10-18).
|
||||
How many payload generations it spanned is not readable from any host
|
||||
artifact. *(unverified 2026-10-18)*
|
||||
|
||||
Two laws fall out of it. A target can read down beside a healthy service, so
|
||||
read `lastError` from `/api/v1/targets` before believing an outage. And rule
|
||||
state lives in Prometheus' process, not on disk — a restart re-stamps
|
||||
`activeAt` on a long-standing alert, so a "fresh" alert after a reboot is not
|
||||
evidence of a fresh fault. (Alertmanager by contrast **does** persist:
|
||||
`--storage.path=/alertmanager` on a host volume keeps silences and the
|
||||
notification log — `dankclaw/observability/quadlet/alertmanager.container:39-42`,
|
||||
`/opt/alertmanager/data`.)
|
||||
|
||||
Remaining known downs are the structural ones above (HTML-scraped probes, Gitea
|
||||
metrics, ComfyUI), and halogen serves `/metrics` natively, which is why
|
||||
`frank-dragoon` has always been green. The active-alert count is a
|
||||
moment-in-time value and should never be documented as a constant:
|
||||
`/api/v1/alerts` was empty at the 2026-10-18 morning census, held one firing
|
||||
`LlmPrefixCacheCollapse` (`activeAt` 11:10:48Z, value 2.6e-4) when sampled
|
||||
minutes later, and was empty again by 11:56Z; the relay logged the matching
|
||||
firing and resolved sends at `04:26:21-07:00` and `04:41:21-07:00`.
|
||||
|
||||
**What that zealot scrape measured on 2026-10-18.** As of ~04:30 PDT the
|
||||
production `frank-zealot` root unit was stopped **on purpose** for a benchmark
|
||||
window on the 3090 — the repo Benchmarking Policy says stop the production
|
||||
service before benching — and `:7771` answered from a manually booted candidate
|
||||
arm (`systemctl is-active frank-zealot` → `failed` after
|
||||
`stop-sigterm timed out. Killing`, which is llama.cpp ignoring SIGTERM past
|
||||
`TimeoutStopSec`, the known shutdown shape for these engines). So
|
||||
`up{job="frank-zealot"} == 1` that day measured the trial, not the production
|
||||
shape in `scripts/frank-zealot.env` (Ternary-Bonsai-27B-PQ2_0, `CTX_SIZE=401408
|
||||
PARALLEL=2` = 2 × 200,704 per slot, q8_0 K+V), which remains the repo contract.
|
||||
The unit stayed `enabled`; `./scripts/deploy.sh dankclaw --service frank-zealot`
|
||||
reinstalls and starts it, restoring boot persistence and self-healing. Never
|
||||
quote a candidate arm's flags or VRAM as fleet state — the process and its shape
|
||||
changed between two censuses ten minutes apart.
|
||||
|
||||
## Related
|
||||
|
||||
|
||||
Reference in new issue
Block a user