From a61308d72e30aa7805b743a788c9779eb35a9e1c Mon Sep 17 00:00:00 2001 From: Frank Date: Wed, 21 Oct 2026 02:35:44 -0700 Subject: [PATCH 1/2] =?UTF-8?q?docs(vault):=20audit=20tranche=201=20?= =?UTF-8?q?=E2=80=94=20one-owner=20dedupe=20of=20the=20observability=20clu?= =?UTF-8?q?ster,=20dead-shape=20numbers=20destroyed?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Purpose-test pass over the audit findings (every verdict re-verified against the file before touching it; three scout findings were hallucinated — a nonexistent filename, my own changelog line, and sanctioned Eras rows): - Observability Provisioning: destroyed the 63-line 'Current delivered state' block — chain topology/relay/delivery restates [[Observability]] and [[Alertmanager]] (those are the owners). Its three sole-custody laws MOVED to Stale Signal Audit as 'Three rules that outlive the incident': down-beside-healthy -> read lastError; Prometheus alert state in process vs Alertmanager's --storage.path persistence; candidate-arm metrics are never fleet state. - RDNA4 MoE VRAM budget: kept the laws (registry over-prices the pool; never reuse a B/token ratio across geometries; pin pool bytes floor-sized), era-labelled the rates, destroyed the measured ladder table for the superseded shape-G/bf16 rungs. - Stale Signal Audit: new rules section + Provisioning Related link. check-okf 109/0, vault links 0 unresolved, docs seams green. --- .../Architecture/Stale Signal Audit.md | 14 ++++ .../Observability Provisioning.md | 69 +------------------ .../RDNA4 MoE Expert Offload VRAM Budget.md | 36 ++++------ 3 files changed, 30 insertions(+), 89 deletions(-) diff --git a/serverAppConfigs-docs/Architecture/Stale Signal Audit.md b/serverAppConfigs-docs/Architecture/Stale Signal Audit.md index 6ef56e3..7dccc67 100644 --- a/serverAppConfigs-docs/Architecture/Stale Signal Audit.md +++ b/serverAppConfigs-docs/Architecture/Stale Signal Audit.md @@ -116,6 +116,20 @@ as working when they could never report up. **Clean cutover includes prose** — docs, unit docstrings, and the repo's rule file (the most expensive place to be stale). +## Three rules that outlive the incident that taught them + +* **A target can read down beside a healthy service.** Read `lastError` from + `/api/v1/targets` before believing an outage — llama.cpp refuses `/metrics` at the + protocol layer without `--metrics` (501) while `/health` answers 200 for weeks. +* **Alert state lives where the tool says it does.** Prometheus keeps rule state in + process memory — a restart re-stamps `activeAt`, so a "fresh" alert after a reboot is + not evidence of a fresh fault. Alertmanager **does** persist (`--storage.path` on a + host volume: silences + notification log, [[Alertmanager]]). +* **Never quote a candidate arm's flags or VRAM as fleet state.** A stopped production + unit (`failed` after `stop-sigterm timed out. Killing` — the known shutdown shape, + [[frank-zealot]]) beside an answering port means the metric measured a scratch boot; + the process shape can change between two censuses ten minutes apart. + ## Order and verification 1. Enumerate the surfaces (status board, metrics targets, alert rules, test diff --git a/serverAppConfigs-docs/Infrastructure/Observability Provisioning.md b/serverAppConfigs-docs/Infrastructure/Observability Provisioning.md index 3e91553..f1fa638 100644 --- a/serverAppConfigs-docs/Infrastructure/Observability Provisioning.md +++ b/serverAppConfigs-docs/Infrastructure/Observability Provisioning.md @@ -3,7 +3,7 @@ type: reference title: Observability Provisioning description: "Monitoring config as code: the config triad, deploy copiers, planner mappings, and where each change type lands." tags: [observability, prometheus, grafana, provisioning, deployment] -timestamp: 2026-10-05T00:00:00-07:00 +timestamp: 2026-10-21T00:00:00-07:00 --- # Observability Provisioning @@ -112,73 +112,10 @@ unified storage — a file existing on disk is not proof the running server loaded it. See [[Stale Signal Audit]] for the never-green-target decision table (refused = outage, 404/HTML = config defect). -## Current delivered state (delivery chain live 2026-09-17, re-verified 2026-10-18) - -Prometheus → Alertmanager (`dankclaw/observability/quadlet/alertmanager.container` -loopback `:9093`) → host-side Signal relay (`scripts/signal-relay.py`, user -unit, `192.168.0.143:18093` — the LAN IP, the one bind a container can reach -via `host.containers.internal`) → signal-cli, addressed by number -`+12062476336` (`scripts/signal-relay.py:36-43`). Every hop re-checked -2026-10-18: `ss -ltnp` shows `python3` on `192.168.0.143:18093` -`curl -s http://192.168.0.143:18093/health` → `ok`, and -`journalctl --user -u signal-relay` ends with successful `sent to +12062476336:` -lines for `LlmEndpointDown`, `LlmRequestsQueueing`, `LlmSpecDecodeOff` and -`LlmPrefixCacheCollapse` (firing and resolved). Some targets read down for -structural reasons (HTML-scraped "probes", Gitea metrics off); the remedy is -the blackbox exporter. See [[Prometheus]] → Targets. - -**A llama.cpp endpoint is invisible unless it opts in.** Until 2026-10-18, -`up{job="frank-zealot"}` was 0 for the entire retained history of the metric — -the oldest sample, 2026-10-11, is already 0 — while the service itself answered -`/health` 200 the whole time. Cause: llama.cpp only exports `/metrics` when it -is started with the flag, so the scrape was refused at the protocol layer (the -flag and its opt-out are on [[frank-zealot]]) and `LlmEndpointDown` (critical) -stayed firing: -the relay re-sent it on the 4 h `repeat_interval` for days (`journalctl --user --u signal-relay` — firing sends at 01:37, 05:42, 09:47, 13:52, 17:57 and 22:02 -on 2026-10-12). It finally resolved at `2026-10-18T01:53:21-07:00`, after the -metrics opt-in landed that day; the target -reports up since (`curl -s http://127.0.0.1:9090/api/v1/targets` 2026-10-18). -How many payload generations it spanned is not readable from any host -artifact. *(unverified 2026-10-18)* - -Two laws fall out of it. A target can read down beside a healthy service, so -read `lastError` from `/api/v1/targets` before believing an outage. And rule -state lives in Prometheus' process, not on disk — a restart re-stamps -`activeAt` on a long-standing alert, so a "fresh" alert after a reboot is not -evidence of a fresh fault. (Alertmanager by contrast **does** persist: -`--storage.path=/alertmanager` on a host volume keeps silences and the -notification log — `dankclaw/observability/quadlet/alertmanager.container:39-42` -`/opt/alertmanager/data`.) - -Remaining known downs are the structural ones above (HTML-scraped probes, Gitea -metrics, ComfyUI), and `frank-dragoon` exports `/metrics` natively -([[frank-dragoon]]), which is why -`frank-dragoon` has always been green. The active-alert count is a -moment-in-time value and should never be documented as a constant: -`/api/v1/alerts` was empty at the 2026-10-18 morning census, held one firing -`LlmPrefixCacheCollapse` (`activeAt` 11:10:48Z, value 2.6e-4) when sampled -minutes later, and was empty again by 11:56Z; the relay logged the matching -firing and resolved sends at `04:26:21-07:00` and `04:41:21-07:00`. - -**What that zealot scrape measured on 2026-10-18.** As of ~04:30 PDT the -production `frank-zealot` root unit was stopped **on purpose** for a benchmark -window on the 3090 — the repo Benchmarking Policy says stop the production -service before benching — and `:7771` answered from a manually booted candidate -arm (`systemctl is-active frank-zealot` → `failed` after -`stop-sigterm timed out. Killing`, which is llama.cpp ignoring SIGTERM past -`TimeoutStopSec`, the known shutdown shape for these engines). So -`up{job="frank-zealot"} == 1` that day measured the trial, not the production -shape in `scripts/frank-zealot.env` — that shape is [[frank-zealot]]'s to state, -and it is the repo contract. -The unit stayed `enabled`; `./scripts/deploy.sh dankclaw --service frank-zealot` -reinstalls and starts it, restoring boot persistence and self-healing. Never -quote a candidate arm's flags or VRAM as fleet state — the process and its shape -changed between two censuses ten minutes apart. - ## Related - [[Observability]] — stack topology and the alert delivery chain -- [[Stale Signal Audit]] — classifying down targets before "fixing" them +- [[Stale Signal Audit]] — classifying down targets before "fixing" them, and the + down-beside-healthy / alert-state-persistence / candidate-arm rules - [[Quadlet Lifecycle]] — the units that host these services - [[Deploy State Triage]] — how the planner selects these services diff --git a/serverAppConfigs-docs/LLM/RDNA4 MoE Expert Offload VRAM Budget.md b/serverAppConfigs-docs/LLM/RDNA4 MoE Expert Offload VRAM Budget.md index b71d130..68349f1 100644 --- a/serverAppConfigs-docs/LLM/RDNA4 MoE Expert Offload VRAM Budget.md +++ b/serverAppConfigs-docs/LLM/RDNA4 MoE Expert Offload VRAM Budget.md @@ -70,12 +70,12 @@ back toward 52. Check `VLLM_CUDAGRAPH_*` actually present in `--kv-cache-memory-bytes` does NOT appear in the arena arithmetic — the plan reserves the registry price, not the actual pool. Raising the pool consumes -real free VRAM without growing the arena; the registry over-prices the pool -(9190 B/token planned vs 7,891.8 measured at 200704 fp8 — 7,701 at 262144; -**never reuse a B/token ratio across geometries**, and subtract ~3.87 MB -block-alignment loss when converting a token target to bytes). The ask lands -block-aligned: 3,166,500,000 B → 401,408 tokens; verify with -`GPU KV cache size: N tokens, Maximum concurrency … X.Xx`. +real free VRAM without growing the arena; the registry OVER-PRICES the pool +(era G measured ~14 % under the planned B/token — the gap is the lesson, not +the number; **never reuse a B/token ratio across geometries**), and there is a +~4 MB block-alignment loss to subtract when converting a token target to +bytes. The ask lands block-aligned: verify against the engine's own +`GPU KV cache size: N tokens, Maximum concurrency … X.Xx` line at boot. **Driver-reported free VRAM after hours of traffic is NOT allocatable arena.** It is the estimator gap: the registry over-prices KV and charges @@ -117,11 +117,13 @@ done, and read post-boot free VRAM per rank (`rocm-smi --showmeminfo vram`) **KV accounting — pin the pool BYTES, floor-sized:** set `--kv-cache-memory-bytes` PER RANK to the contract floor (`seqs × MAXLEN × measured B/token`), not the native window — extra pool bytes -come out of the arena/headroom and buy nothing. Measured B/token on this -hybrid: **7,701 fp8 / 13,839 bf16** (decomposed: attention 6,138/12,276 + -dtype-independent mamba/side 1,563). Measured rates are shape-dependent — -7,891.8 at the 200704×3 shape. Capacity does NOT scale with `SEQS`: the -planner's KV reserve is for ONE max-model-len request. +come out of the arena/headroom and buy nothing. The B/token rate is a property +of the GEOMETRY: era G (3 × 200,704 fp8, vLLM fork) measured **7,891.8**; +the radiance-era hybrid measured ~7,700 fp8 / ~13,800 bf16 at its own shape. +Read your live rate from the boot ledger every time; the decomposition (a +dtype-independent mamba/side stream rides on top of the attention KV) is why +the ratio moves. Capacity does NOT scale with `SEQS`: the planner's KV reserve +is for ONE max-model-len request. **Parallelism ceilings you cannot flag past:** @@ -179,18 +181,6 @@ Cutover order: ladder → winner passes and serves → repo env + seam + harness (`git stash -u`) if the tree is dirty → deploy (one restart) → verify health + engine flags in the log. -**Measured ladder on this box (200704×3, fp8 KV, precise memory):** - -| shape | util | graphs | arena | result | -|---|---|---|---|---| -| E | 0.98 | 0.55 | 17.22 GiB / 7,522 slots | boot ok | -| F | 0.982 | 0.32 | 17.52 GiB / 7,653 slots | boot ok | -| 1.0 | 1.00 | 0.07 | — | UNBOOTABLE (worker-init ValueError, both ranks) | -| G (deployed) | 0.982 | 0.07 | 17.77 GiB / 7,762 slots | KV 401,408 tokens / 2.00x; 0.89 GiB real free/card held through a live 4,096-token generation at ~38–39 tok/s | - -bf16 ceilings at util 0.98: 3 seqs → 241,664 passes / 245,760 and 262,144 OOM -at graph capture; 6 seqs fp8 → 10.70 GiB ask with 9.18 GiB free (fails) — -~207k per context max. ## 7. Arena audit procedure (when asked for "more experts") From c96d0228a4c813b2612d9a136a7775149f3a3e1b Mon Sep 17 00:00:00 2001 From: Frank Date: Wed, 21 Oct 2026 02:45:57 -0700 Subject: [PATCH 2/2] =?UTF-8?q?fix(grafana):=20dashboard=20census=20fixes?= =?UTF-8?q?=20=E2=80=94=20every=20panel=20now=20receives=20data?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Census: all 5 provisioned dashboards, 56 targets, every expression re-evaluated against the live Prometheus/Loki with template vars expanded (queries must run from the host: both APIs bind 127.0.0.1; from the dev box that is connection-refused, which is NOT a dashboard defect). - host-system 'Load 5m / core' was STRUCTURALLY EMPTY: bare '/' drops non-matching pairs and the right side collapses to {job} while the left carries {job,instance} -> 0 series forever. Fixed with / on(job); proven live (2 series). - containers 'Memory vs limit (%)' was STRUCTURALLY EMPTY: every container_spec_memory_limit_bytes on this fleet is 0 (no cgroup limits), so the '> 0' filter can never match. Rewritten to share-of-host-total via label_replace join of the node exporter's MemTotal by host (the parens form is a PromQL parse error: nothing may follow group_left); 62 series live. Panel unit percentunit kept -> expression returns a fraction. - llm-fleet 'Firing fleet alerts' is legitimately empty when healthy (ALERTS carries only firing/pending series); description now says so, per the Stale Signal rule that 0-series-on-ALERTS is not a defect. logs-overview and observability: all targets already returning data. --- shared/grafana/provisioning/dashboards/containers.json | 6 +++--- shared/grafana/provisioning/dashboards/host-system.json | 2 +- shared/grafana/provisioning/dashboards/llm-fleet.json | 2 +- 3 files changed, 5 insertions(+), 5 deletions(-) diff --git a/shared/grafana/provisioning/dashboards/containers.json b/shared/grafana/provisioning/dashboards/containers.json index c324b97..6bb13c0 100644 --- a/shared/grafana/provisioning/dashboards/containers.json +++ b/shared/grafana/provisioning/dashboards/containers.json @@ -104,14 +104,14 @@ { "id": 6, "type": "timeseries", - "title": "Memory vs limit (%)", - "description": "Working set over the cgroup memory limit (container_spec_memory_limit_bytes). Only containers that carry a limit are shown; approaching 1.0 means an OOM is near. Containers with no limit are omitted.", + "title": "Memory vs host total (%)", + "description": "Working set over the host's total RAM. The fleet sets NO cgroup memory limits (every container_spec_memory_limit_bytes is 0 — a limit-share panel is structurally empty here), so the honest pressure view is share of the host. The join labels the node exporter's MemTotal onto the cadvisor series by host.", "gridPos": { "h": 9, "w": 12, "x": 12, "y": 11 }, "datasource": { "type": "prometheus", "uid": "prometheus" }, "fieldConfig": { "defaults": { "unit": "percentunit", "min": 0, "max": 1, "custom": { "fillOpacity": 8, "lineWidth": 1, "stacking": { "mode": "none" } } }, "overrides": [] }, "targets": [ { - "expr": "(container_memory_working_set_bytes{job=~\"cadvisor-.*\", host=~\"$host\", container=~\"$container\"} / on (job, host, container) group_left container_spec_memory_limit_bytes{job=~\"cadvisor-.*\", host=~\"$host\", container=~\"$container\"}) and (container_spec_memory_limit_bytes{job=~\"cadvisor-.*\", host=~\"$host\", container=~\"$container\"} > 0)", + "expr": "container_memory_working_set_bytes{job=~\"cadvisor-.*\", host=~\"$host\", container=~\"$container\", container!=\"\"} / on(host) group_left label_replace(node_memory_MemTotal_bytes{job=~\"node-.*\"}, \"host\", \"$1\", \"job\", \"node-(.*)\")", "legendFormat": "{{host}} / {{container}}", "refId": "A" } diff --git a/shared/grafana/provisioning/dashboards/host-system.json b/shared/grafana/provisioning/dashboards/host-system.json index 487a712..c9b14fc 100644 --- a/shared/grafana/provisioning/dashboards/host-system.json +++ b/shared/grafana/provisioning/dashboards/host-system.json @@ -54,7 +54,7 @@ "gridPos": { "h": 4, "w": 6, "x": 12, "y": 0 }, "datasource": { "type": "prometheus", "uid": "prometheus" }, "fieldConfig": { "defaults": { "unit": "short", "thresholds": { "mode": "absolute", "steps": [{ "color": "green", "value": null }, { "color": "yellow", "value": 0.7 }, { "color": "red", "value": 1 }] } }, "overrides": [] }, - "targets": [{ "expr": "node_load5{job=~\"$node\"} / count by (job) (node_cpu_seconds_total{job=~\"$node\",mode=\"idle\"})", "legendFormat": "{{job}}", "refId": "A" }] + "targets": [{ "expr": "node_load5{job=~\"$node\"} / on(job) count by (job) (node_cpu_seconds_total{job=~\"$node\",mode=\"idle\"})", "legendFormat": "{{job}}", "refId": "A" }] }, { "id": 4, "type": "stat", "title": "Open file handles", diff --git a/shared/grafana/provisioning/dashboards/llm-fleet.json b/shared/grafana/provisioning/dashboards/llm-fleet.json index c2bf94c..fff5af4 100644 --- a/shared/grafana/provisioning/dashboards/llm-fleet.json +++ b/shared/grafana/provisioning/dashboards/llm-fleet.json @@ -194,7 +194,7 @@ "id": 6, "type": "timeseries", "title": "Firing fleet alerts", - "description": "The LLM-fleet rules from shared/prometheus/rules/llm-fleet.yml. A firing alert is delivered to Signal: Prometheus -> Alertmanager (dankclaw quadlet) -> the host-side signal-relay user unit -> signal-cli, since 2026-09-17.", + "description": "The LLM-fleet rules from shared/prometheus/rules/llm-fleet.yml. EMPTY IS THE HEALTHY STATE: zero series means nothing is firing (ALERTS only carries active/pending alert series — do not read this panel as broken). A firing alert is delivered to Signal: Prometheus -> Alertmanager (dankclaw quadlet) -> the host-side signal-relay user unit -> signal-cli, since 2026-09-17.", "gridPos": { "h": 8, "w": 12,