docs(vault): audit tranche 1 — one-owner dedupe of the observability cluster, dead-shape numbers destroyed
Purpose-test pass over the audit findings (every verdict re-verified against the file before touching it; three scout findings were hallucinated — a nonexistent filename, my own changelog line, and sanctioned Eras rows): - Observability Provisioning: destroyed the 63-line 'Current delivered state' block — chain topology/relay/delivery restates [[Observability]] and [[Alertmanager]] (those are the owners). Its three sole-custody laws MOVED to Stale Signal Audit as 'Three rules that outlive the incident': down-beside-healthy -> read lastError; Prometheus alert state in process vs Alertmanager's --storage.path persistence; candidate-arm metrics are never fleet state. - RDNA4 MoE VRAM budget: kept the laws (registry over-prices the pool; never reuse a B/token ratio across geometries; pin pool bytes floor-sized), era-labelled the rates, destroyed the measured ladder table for the superseded shape-G/bf16 rungs. - Stale Signal Audit: new rules section + Provisioning Related link. check-okf 109/0, vault links 0 unresolved, docs seams green.
This commit is contained in:
1 parent
c6e2887f94
commit
a61308d72e
3 files changed
+30
-89
No files matched your search
@@ -116,6 +116,20 @@ as working when they could never report up. **Clean cutover includes prose** —
|
|||||||
docs, unit docstrings, and the repo's rule file (the most expensive place to
|
docs, unit docstrings, and the repo's rule file (the most expensive place to
|
||||||
be stale).
|
be stale).
|
||||||
|
|
||||||
|
## Three rules that outlive the incident that taught them
|
||||||
|
|
||||||
|
* **A target can read down beside a healthy service.** Read `lastError` from
|
||||||
|
`/api/v1/targets` before believing an outage — llama.cpp refuses `/metrics` at the
|
||||||
|
protocol layer without `--metrics` (501) while `/health` answers 200 for weeks.
|
||||||
|
* **Alert state lives where the tool says it does.** Prometheus keeps rule state in
|
||||||
|
process memory — a restart re-stamps `activeAt`, so a "fresh" alert after a reboot is
|
||||||
|
not evidence of a fresh fault. Alertmanager **does** persist (`--storage.path` on a
|
||||||
|
host volume: silences + notification log, [[Alertmanager]]).
|
||||||
|
* **Never quote a candidate arm's flags or VRAM as fleet state.** A stopped production
|
||||||
|
unit (`failed` after `stop-sigterm timed out. Killing` — the known shutdown shape,
|
||||||
|
[[frank-zealot]]) beside an answering port means the metric measured a scratch boot;
|
||||||
|
the process shape can change between two censuses ten minutes apart.
|
||||||
|
|
||||||
## Order and verification
|
## Order and verification
|
||||||
|
|
||||||
1. Enumerate the surfaces (status board, metrics targets, alert rules, test
|
1. Enumerate the surfaces (status board, metrics targets, alert rules, test
|
||||||
|
|||||||
@@ -3,7 +3,7 @@ type: reference
|
|||||||
title: Observability Provisioning
|
title: Observability Provisioning
|
||||||
description: "Monitoring config as code: the config triad, deploy copiers, planner mappings, and where each change type lands."
|
description: "Monitoring config as code: the config triad, deploy copiers, planner mappings, and where each change type lands."
|
||||||
tags: [observability, prometheus, grafana, provisioning, deployment]
|
tags: [observability, prometheus, grafana, provisioning, deployment]
|
||||||
timestamp: 2026-10-05T00:00:00-07:00
|
timestamp: 2026-10-21T00:00:00-07:00
|
||||||
---
|
---
|
||||||
|
|
||||||
# Observability Provisioning
|
# Observability Provisioning
|
||||||
@@ -112,73 +112,10 @@ unified storage — a file existing on disk is not proof the running server
|
|||||||
loaded it. See [[Stale Signal Audit]] for the never-green-target decision
|
loaded it. See [[Stale Signal Audit]] for the never-green-target decision
|
||||||
table (refused = outage, 404/HTML = config defect).
|
table (refused = outage, 404/HTML = config defect).
|
||||||
|
|
||||||
## Current delivered state (delivery chain live 2026-09-17, re-verified 2026-10-18)
|
|
||||||
|
|
||||||
Prometheus → Alertmanager (`dankclaw/observability/quadlet/alertmanager.container`
|
|
||||||
loopback `:9093`) → host-side Signal relay (`scripts/signal-relay.py`, user
|
|
||||||
unit, `192.168.0.143:18093` — the LAN IP, the one bind a container can reach
|
|
||||||
via `host.containers.internal`) → signal-cli, addressed by number
|
|
||||||
`+12062476336` (`scripts/signal-relay.py:36-43`). Every hop re-checked
|
|
||||||
2026-10-18: `ss -ltnp` shows `python3` on `192.168.0.143:18093`
|
|
||||||
`curl -s http://192.168.0.143:18093/health` → `ok`, and
|
|
||||||
`journalctl --user -u signal-relay` ends with successful `sent to +12062476336:`
|
|
||||||
lines for `LlmEndpointDown`, `LlmRequestsQueueing`, `LlmSpecDecodeOff` and
|
|
||||||
`LlmPrefixCacheCollapse` (firing and resolved). Some targets read down for
|
|
||||||
structural reasons (HTML-scraped "probes", Gitea metrics off); the remedy is
|
|
||||||
the blackbox exporter. See [[Prometheus]] → Targets.
|
|
||||||
|
|
||||||
**A llama.cpp endpoint is invisible unless it opts in.** Until 2026-10-18,
|
|
||||||
`up{job="frank-zealot"}` was 0 for the entire retained history of the metric —
|
|
||||||
the oldest sample, 2026-10-11, is already 0 — while the service itself answered
|
|
||||||
`/health` 200 the whole time. Cause: llama.cpp only exports `/metrics` when it
|
|
||||||
is started with the flag, so the scrape was refused at the protocol layer (the
|
|
||||||
flag and its opt-out are on [[frank-zealot]]) and `LlmEndpointDown` (critical)
|
|
||||||
stayed firing:
|
|
||||||
the relay re-sent it on the 4 h `repeat_interval` for days (`journalctl --user
|
|
||||||
-u signal-relay` — firing sends at 01:37, 05:42, 09:47, 13:52, 17:57 and 22:02
|
|
||||||
on 2026-10-12). It finally resolved at `2026-10-18T01:53:21-07:00`, after the
|
|
||||||
metrics opt-in landed that day; the target
|
|
||||||
reports up since (`curl -s http://127.0.0.1:9090/api/v1/targets` 2026-10-18).
|
|
||||||
How many payload generations it spanned is not readable from any host
|
|
||||||
artifact. *(unverified 2026-10-18)*
|
|
||||||
|
|
||||||
Two laws fall out of it. A target can read down beside a healthy service, so
|
|
||||||
read `lastError` from `/api/v1/targets` before believing an outage. And rule
|
|
||||||
state lives in Prometheus' process, not on disk — a restart re-stamps
|
|
||||||
`activeAt` on a long-standing alert, so a "fresh" alert after a reboot is not
|
|
||||||
evidence of a fresh fault. (Alertmanager by contrast **does** persist:
|
|
||||||
`--storage.path=/alertmanager` on a host volume keeps silences and the
|
|
||||||
notification log — `dankclaw/observability/quadlet/alertmanager.container:39-42`
|
|
||||||
`/opt/alertmanager/data`.)
|
|
||||||
|
|
||||||
Remaining known downs are the structural ones above (HTML-scraped probes, Gitea
|
|
||||||
metrics, ComfyUI), and `frank-dragoon` exports `/metrics` natively
|
|
||||||
([[frank-dragoon]]), which is why
|
|
||||||
`frank-dragoon` has always been green. The active-alert count is a
|
|
||||||
moment-in-time value and should never be documented as a constant:
|
|
||||||
`/api/v1/alerts` was empty at the 2026-10-18 morning census, held one firing
|
|
||||||
`LlmPrefixCacheCollapse` (`activeAt` 11:10:48Z, value 2.6e-4) when sampled
|
|
||||||
minutes later, and was empty again by 11:56Z; the relay logged the matching
|
|
||||||
firing and resolved sends at `04:26:21-07:00` and `04:41:21-07:00`.
|
|
||||||
|
|
||||||
**What that zealot scrape measured on 2026-10-18.** As of ~04:30 PDT the
|
|
||||||
production `frank-zealot` root unit was stopped **on purpose** for a benchmark
|
|
||||||
window on the 3090 — the repo Benchmarking Policy says stop the production
|
|
||||||
service before benching — and `:7771` answered from a manually booted candidate
|
|
||||||
arm (`systemctl is-active frank-zealot` → `failed` after
|
|
||||||
`stop-sigterm timed out. Killing`, which is llama.cpp ignoring SIGTERM past
|
|
||||||
`TimeoutStopSec`, the known shutdown shape for these engines). So
|
|
||||||
`up{job="frank-zealot"} == 1` that day measured the trial, not the production
|
|
||||||
shape in `scripts/frank-zealot.env` — that shape is [[frank-zealot]]'s to state,
|
|
||||||
and it is the repo contract.
|
|
||||||
The unit stayed `enabled`; `./scripts/deploy.sh dankclaw --service frank-zealot`
|
|
||||||
reinstalls and starts it, restoring boot persistence and self-healing. Never
|
|
||||||
quote a candidate arm's flags or VRAM as fleet state — the process and its shape
|
|
||||||
changed between two censuses ten minutes apart.
|
|
||||||
|
|
||||||
## Related
|
## Related
|
||||||
|
|
||||||
- [[Observability]] — stack topology and the alert delivery chain
|
- [[Observability]] — stack topology and the alert delivery chain
|
||||||
- [[Stale Signal Audit]] — classifying down targets before "fixing" them
|
- [[Stale Signal Audit]] — classifying down targets before "fixing" them, and the
|
||||||
|
down-beside-healthy / alert-state-persistence / candidate-arm rules
|
||||||
- [[Quadlet Lifecycle]] — the units that host these services
|
- [[Quadlet Lifecycle]] — the units that host these services
|
||||||
- [[Deploy State Triage]] — how the planner selects these services
|
- [[Deploy State Triage]] — how the planner selects these services
|
||||||
@@ -70,12 +70,12 @@ back toward 52. Check `VLLM_CUDAGRAPH_*` actually present in
|
|||||||
|
|
||||||
`--kv-cache-memory-bytes` does NOT appear in the arena arithmetic — the plan
|
`--kv-cache-memory-bytes` does NOT appear in the arena arithmetic — the plan
|
||||||
reserves the registry price, not the actual pool. Raising the pool consumes
|
reserves the registry price, not the actual pool. Raising the pool consumes
|
||||||
real free VRAM without growing the arena; the registry over-prices the pool
|
real free VRAM without growing the arena; the registry OVER-PRICES the pool
|
||||||
(9190 B/token planned vs 7,891.8 measured at 200704 fp8 — 7,701 at 262144;
|
(era G measured ~14 % under the planned B/token — the gap is the lesson, not
|
||||||
**never reuse a B/token ratio across geometries**, and subtract ~3.87 MB
|
the number; **never reuse a B/token ratio across geometries**), and there is a
|
||||||
block-alignment loss when converting a token target to bytes). The ask lands
|
~4 MB block-alignment loss to subtract when converting a token target to
|
||||||
block-aligned: 3,166,500,000 B → 401,408 tokens; verify with
|
bytes. The ask lands block-aligned: verify against the engine's own
|
||||||
`GPU KV cache size: N tokens, Maximum concurrency … X.Xx`.
|
`GPU KV cache size: N tokens, Maximum concurrency … X.Xx` line at boot.
|
||||||
|
|
||||||
**Driver-reported free VRAM after hours of traffic is NOT allocatable
|
**Driver-reported free VRAM after hours of traffic is NOT allocatable
|
||||||
arena.** It is the estimator gap: the registry over-prices KV and charges
|
arena.** It is the estimator gap: the registry over-prices KV and charges
|
||||||
@@ -117,11 +117,13 @@ done, and read post-boot free VRAM per rank (`rocm-smi --showmeminfo vram`)
|
|||||||
**KV accounting — pin the pool BYTES, floor-sized:** set
|
**KV accounting — pin the pool BYTES, floor-sized:** set
|
||||||
`--kv-cache-memory-bytes` PER RANK to the contract floor
|
`--kv-cache-memory-bytes` PER RANK to the contract floor
|
||||||
(`seqs × MAXLEN × measured B/token`), not the native window — extra pool bytes
|
(`seqs × MAXLEN × measured B/token`), not the native window — extra pool bytes
|
||||||
come out of the arena/headroom and buy nothing. Measured B/token on this
|
come out of the arena/headroom and buy nothing. The B/token rate is a property
|
||||||
hybrid: **7,701 fp8 / 13,839 bf16** (decomposed: attention 6,138/12,276 +
|
of the GEOMETRY: era G (3 × 200,704 fp8, vLLM fork) measured **7,891.8**;
|
||||||
dtype-independent mamba/side 1,563). Measured rates are shape-dependent —
|
the radiance-era hybrid measured ~7,700 fp8 / ~13,800 bf16 at its own shape.
|
||||||
7,891.8 at the 200704×3 shape. Capacity does NOT scale with `SEQS`: the
|
Read your live rate from the boot ledger every time; the decomposition (a
|
||||||
planner's KV reserve is for ONE max-model-len request.
|
dtype-independent mamba/side stream rides on top of the attention KV) is why
|
||||||
|
the ratio moves. Capacity does NOT scale with `SEQS`: the planner's KV reserve
|
||||||
|
is for ONE max-model-len request.
|
||||||
|
|
||||||
**Parallelism ceilings you cannot flag past:**
|
**Parallelism ceilings you cannot flag past:**
|
||||||
|
|
||||||
@@ -179,18 +181,6 @@ Cutover order: ladder → winner passes and serves → repo env + seam + harness
|
|||||||
(`git stash -u`) if the tree is dirty → deploy (one restart) → verify health
|
(`git stash -u`) if the tree is dirty → deploy (one restart) → verify health
|
||||||
+ engine flags in the log.
|
+ engine flags in the log.
|
||||||
|
|
||||||
**Measured ladder on this box (200704×3, fp8 KV, precise memory):**
|
|
||||||
|
|
||||||
| shape | util | graphs | arena | result |
|
|
||||||
|---|---|---|---|---|
|
|
||||||
| E | 0.98 | 0.55 | 17.22 GiB / 7,522 slots | boot ok |
|
|
||||||
| F | 0.982 | 0.32 | 17.52 GiB / 7,653 slots | boot ok |
|
|
||||||
| 1.0 | 1.00 | 0.07 | — | UNBOOTABLE (worker-init ValueError, both ranks) |
|
|
||||||
| G (deployed) | 0.982 | 0.07 | 17.77 GiB / 7,762 slots | KV 401,408 tokens / 2.00x; 0.89 GiB real free/card held through a live 4,096-token generation at ~38–39 tok/s |
|
|
||||||
|
|
||||||
bf16 ceilings at util 0.98: 3 seqs → 241,664 passes / 245,760 and 262,144 OOM
|
|
||||||
at graph capture; 6 seqs fp8 → 10.70 GiB ask with 9.18 GiB free (fails) —
|
|
||||||
~207k per context max.
|
|
||||||
|
|
||||||
## 7. Arena audit procedure (when asked for "more experts")
|
## 7. Arena audit procedure (when asked for "more experts")
|
||||||
|
|
||||||
|
|||||||
Reference in new issue
Block a user