diff --git a/serverAppConfigs-docs/Architecture/Stale Signal Audit.md b/serverAppConfigs-docs/Architecture/Stale Signal Audit.md index 6ef56e3..7dccc67 100644 --- a/serverAppConfigs-docs/Architecture/Stale Signal Audit.md +++ b/serverAppConfigs-docs/Architecture/Stale Signal Audit.md @@ -116,6 +116,20 @@ as working when they could never report up. **Clean cutover includes prose** — docs, unit docstrings, and the repo's rule file (the most expensive place to be stale). +## Three rules that outlive the incident that taught them + +* **A target can read down beside a healthy service.** Read `lastError` from + `/api/v1/targets` before believing an outage — llama.cpp refuses `/metrics` at the + protocol layer without `--metrics` (501) while `/health` answers 200 for weeks. +* **Alert state lives where the tool says it does.** Prometheus keeps rule state in + process memory — a restart re-stamps `activeAt`, so a "fresh" alert after a reboot is + not evidence of a fresh fault. Alertmanager **does** persist (`--storage.path` on a + host volume: silences + notification log, [[Alertmanager]]). +* **Never quote a candidate arm's flags or VRAM as fleet state.** A stopped production + unit (`failed` after `stop-sigterm timed out. Killing` — the known shutdown shape, + [[frank-zealot]]) beside an answering port means the metric measured a scratch boot; + the process shape can change between two censuses ten minutes apart. + ## Order and verification 1. Enumerate the surfaces (status board, metrics targets, alert rules, test diff --git a/serverAppConfigs-docs/Infrastructure/Observability Provisioning.md b/serverAppConfigs-docs/Infrastructure/Observability Provisioning.md index 3e91553..f1fa638 100644 --- a/serverAppConfigs-docs/Infrastructure/Observability Provisioning.md +++ b/serverAppConfigs-docs/Infrastructure/Observability Provisioning.md @@ -3,7 +3,7 @@ type: reference title: Observability Provisioning description: "Monitoring config as code: the config triad, deploy copiers, planner mappings, and where each change type lands." tags: [observability, prometheus, grafana, provisioning, deployment] -timestamp: 2026-10-05T00:00:00-07:00 +timestamp: 2026-10-21T00:00:00-07:00 --- # Observability Provisioning @@ -112,73 +112,10 @@ unified storage — a file existing on disk is not proof the running server loaded it. See [[Stale Signal Audit]] for the never-green-target decision table (refused = outage, 404/HTML = config defect). -## Current delivered state (delivery chain live 2026-09-17, re-verified 2026-10-18) - -Prometheus → Alertmanager (`dankclaw/observability/quadlet/alertmanager.container` -loopback `:9093`) → host-side Signal relay (`scripts/signal-relay.py`, user -unit, `192.168.0.143:18093` — the LAN IP, the one bind a container can reach -via `host.containers.internal`) → signal-cli, addressed by number -`+12062476336` (`scripts/signal-relay.py:36-43`). Every hop re-checked -2026-10-18: `ss -ltnp` shows `python3` on `192.168.0.143:18093` -`curl -s http://192.168.0.143:18093/health` → `ok`, and -`journalctl --user -u signal-relay` ends with successful `sent to +12062476336:` -lines for `LlmEndpointDown`, `LlmRequestsQueueing`, `LlmSpecDecodeOff` and -`LlmPrefixCacheCollapse` (firing and resolved). Some targets read down for -structural reasons (HTML-scraped "probes", Gitea metrics off); the remedy is -the blackbox exporter. See [[Prometheus]] → Targets. - -**A llama.cpp endpoint is invisible unless it opts in.** Until 2026-10-18, -`up{job="frank-zealot"}` was 0 for the entire retained history of the metric — -the oldest sample, 2026-10-11, is already 0 — while the service itself answered -`/health` 200 the whole time. Cause: llama.cpp only exports `/metrics` when it -is started with the flag, so the scrape was refused at the protocol layer (the -flag and its opt-out are on [[frank-zealot]]) and `LlmEndpointDown` (critical) -stayed firing: -the relay re-sent it on the 4 h `repeat_interval` for days (`journalctl --user --u signal-relay` — firing sends at 01:37, 05:42, 09:47, 13:52, 17:57 and 22:02 -on 2026-10-12). It finally resolved at `2026-10-18T01:53:21-07:00`, after the -metrics opt-in landed that day; the target -reports up since (`curl -s http://127.0.0.1:9090/api/v1/targets` 2026-10-18). -How many payload generations it spanned is not readable from any host -artifact. *(unverified 2026-10-18)* - -Two laws fall out of it. A target can read down beside a healthy service, so -read `lastError` from `/api/v1/targets` before believing an outage. And rule -state lives in Prometheus' process, not on disk — a restart re-stamps -`activeAt` on a long-standing alert, so a "fresh" alert after a reboot is not -evidence of a fresh fault. (Alertmanager by contrast **does** persist: -`--storage.path=/alertmanager` on a host volume keeps silences and the -notification log — `dankclaw/observability/quadlet/alertmanager.container:39-42` -`/opt/alertmanager/data`.) - -Remaining known downs are the structural ones above (HTML-scraped probes, Gitea -metrics, ComfyUI), and `frank-dragoon` exports `/metrics` natively -([[frank-dragoon]]), which is why -`frank-dragoon` has always been green. The active-alert count is a -moment-in-time value and should never be documented as a constant: -`/api/v1/alerts` was empty at the 2026-10-18 morning census, held one firing -`LlmPrefixCacheCollapse` (`activeAt` 11:10:48Z, value 2.6e-4) when sampled -minutes later, and was empty again by 11:56Z; the relay logged the matching -firing and resolved sends at `04:26:21-07:00` and `04:41:21-07:00`. - -**What that zealot scrape measured on 2026-10-18.** As of ~04:30 PDT the -production `frank-zealot` root unit was stopped **on purpose** for a benchmark -window on the 3090 — the repo Benchmarking Policy says stop the production -service before benching — and `:7771` answered from a manually booted candidate -arm (`systemctl is-active frank-zealot` → `failed` after -`stop-sigterm timed out. Killing`, which is llama.cpp ignoring SIGTERM past -`TimeoutStopSec`, the known shutdown shape for these engines). So -`up{job="frank-zealot"} == 1` that day measured the trial, not the production -shape in `scripts/frank-zealot.env` — that shape is [[frank-zealot]]'s to state, -and it is the repo contract. -The unit stayed `enabled`; `./scripts/deploy.sh dankclaw --service frank-zealot` -reinstalls and starts it, restoring boot persistence and self-healing. Never -quote a candidate arm's flags or VRAM as fleet state — the process and its shape -changed between two censuses ten minutes apart. - ## Related - [[Observability]] — stack topology and the alert delivery chain -- [[Stale Signal Audit]] — classifying down targets before "fixing" them +- [[Stale Signal Audit]] — classifying down targets before "fixing" them, and the + down-beside-healthy / alert-state-persistence / candidate-arm rules - [[Quadlet Lifecycle]] — the units that host these services - [[Deploy State Triage]] — how the planner selects these services diff --git a/serverAppConfigs-docs/LLM/RDNA4 MoE Expert Offload VRAM Budget.md b/serverAppConfigs-docs/LLM/RDNA4 MoE Expert Offload VRAM Budget.md index b71d130..68349f1 100644 --- a/serverAppConfigs-docs/LLM/RDNA4 MoE Expert Offload VRAM Budget.md +++ b/serverAppConfigs-docs/LLM/RDNA4 MoE Expert Offload VRAM Budget.md @@ -70,12 +70,12 @@ back toward 52. Check `VLLM_CUDAGRAPH_*` actually present in `--kv-cache-memory-bytes` does NOT appear in the arena arithmetic — the plan reserves the registry price, not the actual pool. Raising the pool consumes -real free VRAM without growing the arena; the registry over-prices the pool -(9190 B/token planned vs 7,891.8 measured at 200704 fp8 — 7,701 at 262144; -**never reuse a B/token ratio across geometries**, and subtract ~3.87 MB -block-alignment loss when converting a token target to bytes). The ask lands -block-aligned: 3,166,500,000 B → 401,408 tokens; verify with -`GPU KV cache size: N tokens, Maximum concurrency … X.Xx`. +real free VRAM without growing the arena; the registry OVER-PRICES the pool +(era G measured ~14 % under the planned B/token — the gap is the lesson, not +the number; **never reuse a B/token ratio across geometries**), and there is a +~4 MB block-alignment loss to subtract when converting a token target to +bytes. The ask lands block-aligned: verify against the engine's own +`GPU KV cache size: N tokens, Maximum concurrency … X.Xx` line at boot. **Driver-reported free VRAM after hours of traffic is NOT allocatable arena.** It is the estimator gap: the registry over-prices KV and charges @@ -117,11 +117,13 @@ done, and read post-boot free VRAM per rank (`rocm-smi --showmeminfo vram`) **KV accounting — pin the pool BYTES, floor-sized:** set `--kv-cache-memory-bytes` PER RANK to the contract floor (`seqs × MAXLEN × measured B/token`), not the native window — extra pool bytes -come out of the arena/headroom and buy nothing. Measured B/token on this -hybrid: **7,701 fp8 / 13,839 bf16** (decomposed: attention 6,138/12,276 + -dtype-independent mamba/side 1,563). Measured rates are shape-dependent — -7,891.8 at the 200704×3 shape. Capacity does NOT scale with `SEQS`: the -planner's KV reserve is for ONE max-model-len request. +come out of the arena/headroom and buy nothing. The B/token rate is a property +of the GEOMETRY: era G (3 × 200,704 fp8, vLLM fork) measured **7,891.8**; +the radiance-era hybrid measured ~7,700 fp8 / ~13,800 bf16 at its own shape. +Read your live rate from the boot ledger every time; the decomposition (a +dtype-independent mamba/side stream rides on top of the attention KV) is why +the ratio moves. Capacity does NOT scale with `SEQS`: the planner's KV reserve +is for ONE max-model-len request. **Parallelism ceilings you cannot flag past:** @@ -179,18 +181,6 @@ Cutover order: ladder → winner passes and serves → repo env + seam + harness (`git stash -u`) if the tree is dirty → deploy (one restart) → verify health + engine flags in the log. -**Measured ladder on this box (200704×3, fp8 KV, precise memory):** - -| shape | util | graphs | arena | result | -|---|---|---|---|---| -| E | 0.98 | 0.55 | 17.22 GiB / 7,522 slots | boot ok | -| F | 0.982 | 0.32 | 17.52 GiB / 7,653 slots | boot ok | -| 1.0 | 1.00 | 0.07 | — | UNBOOTABLE (worker-init ValueError, both ranks) | -| G (deployed) | 0.982 | 0.07 | 17.77 GiB / 7,762 slots | KV 401,408 tokens / 2.00x; 0.89 GiB real free/card held through a live 4,096-token generation at ~38–39 tok/s | - -bf16 ceilings at util 0.98: 3 seqs → 241,664 passes / 245,760 and 262,144 OOM -at graph capture; 6 seqs fp8 → 10.70 GiB ask with 9.18 GiB free (fails) — -~207k per context max. ## 7. Arena audit procedure (when asked for "more experts") diff --git a/shared/grafana/provisioning/dashboards/containers.json b/shared/grafana/provisioning/dashboards/containers.json index c324b97..6bb13c0 100644 --- a/shared/grafana/provisioning/dashboards/containers.json +++ b/shared/grafana/provisioning/dashboards/containers.json @@ -104,14 +104,14 @@ { "id": 6, "type": "timeseries", - "title": "Memory vs limit (%)", - "description": "Working set over the cgroup memory limit (container_spec_memory_limit_bytes). Only containers that carry a limit are shown; approaching 1.0 means an OOM is near. Containers with no limit are omitted.", + "title": "Memory vs host total (%)", + "description": "Working set over the host's total RAM. The fleet sets NO cgroup memory limits (every container_spec_memory_limit_bytes is 0 — a limit-share panel is structurally empty here), so the honest pressure view is share of the host. The join labels the node exporter's MemTotal onto the cadvisor series by host.", "gridPos": { "h": 9, "w": 12, "x": 12, "y": 11 }, "datasource": { "type": "prometheus", "uid": "prometheus" }, "fieldConfig": { "defaults": { "unit": "percentunit", "min": 0, "max": 1, "custom": { "fillOpacity": 8, "lineWidth": 1, "stacking": { "mode": "none" } } }, "overrides": [] }, "targets": [ { - "expr": "(container_memory_working_set_bytes{job=~\"cadvisor-.*\", host=~\"$host\", container=~\"$container\"} / on (job, host, container) group_left container_spec_memory_limit_bytes{job=~\"cadvisor-.*\", host=~\"$host\", container=~\"$container\"}) and (container_spec_memory_limit_bytes{job=~\"cadvisor-.*\", host=~\"$host\", container=~\"$container\"} > 0)", + "expr": "container_memory_working_set_bytes{job=~\"cadvisor-.*\", host=~\"$host\", container=~\"$container\", container!=\"\"} / on(host) group_left label_replace(node_memory_MemTotal_bytes{job=~\"node-.*\"}, \"host\", \"$1\", \"job\", \"node-(.*)\")", "legendFormat": "{{host}} / {{container}}", "refId": "A" } diff --git a/shared/grafana/provisioning/dashboards/host-system.json b/shared/grafana/provisioning/dashboards/host-system.json index 487a712..c9b14fc 100644 --- a/shared/grafana/provisioning/dashboards/host-system.json +++ b/shared/grafana/provisioning/dashboards/host-system.json @@ -54,7 +54,7 @@ "gridPos": { "h": 4, "w": 6, "x": 12, "y": 0 }, "datasource": { "type": "prometheus", "uid": "prometheus" }, "fieldConfig": { "defaults": { "unit": "short", "thresholds": { "mode": "absolute", "steps": [{ "color": "green", "value": null }, { "color": "yellow", "value": 0.7 }, { "color": "red", "value": 1 }] } }, "overrides": [] }, - "targets": [{ "expr": "node_load5{job=~\"$node\"} / count by (job) (node_cpu_seconds_total{job=~\"$node\",mode=\"idle\"})", "legendFormat": "{{job}}", "refId": "A" }] + "targets": [{ "expr": "node_load5{job=~\"$node\"} / on(job) count by (job) (node_cpu_seconds_total{job=~\"$node\",mode=\"idle\"})", "legendFormat": "{{job}}", "refId": "A" }] }, { "id": 4, "type": "stat", "title": "Open file handles", diff --git a/shared/grafana/provisioning/dashboards/llm-fleet.json b/shared/grafana/provisioning/dashboards/llm-fleet.json index c2bf94c..fff5af4 100644 --- a/shared/grafana/provisioning/dashboards/llm-fleet.json +++ b/shared/grafana/provisioning/dashboards/llm-fleet.json @@ -194,7 +194,7 @@ "id": 6, "type": "timeseries", "title": "Firing fleet alerts", - "description": "The LLM-fleet rules from shared/prometheus/rules/llm-fleet.yml. A firing alert is delivered to Signal: Prometheus -> Alertmanager (dankclaw quadlet) -> the host-side signal-relay user unit -> signal-cli, since 2026-09-17.", + "description": "The LLM-fleet rules from shared/prometheus/rules/llm-fleet.yml. EMPTY IS THE HEALTHY STATE: zero series means nothing is firing (ALERTS only carries active/pending alert series — do not read this panel as broken). A firing alert is delivered to Signal: Prometheus -> Alertmanager (dankclaw quadlet) -> the host-side signal-relay user unit -> signal-cli, since 2026-09-17.", "gridPos": { "h": 8, "w": 12,