Merge pull request 'docs(vault)+fix(grafana): audit tranche 1 and dashboard data-flow fixes' (#67) from fix/dragoon-context-200k into main
This commit was merged in pull request #67.
This commit is contained in:
@@ -116,6 +116,20 @@ as working when they could never report up. **Clean cutover includes prose** —
|
||||
docs, unit docstrings, and the repo's rule file (the most expensive place to
|
||||
be stale).
|
||||
|
||||
## Three rules that outlive the incident that taught them
|
||||
|
||||
* **A target can read down beside a healthy service.** Read `lastError` from
|
||||
`/api/v1/targets` before believing an outage — llama.cpp refuses `/metrics` at the
|
||||
protocol layer without `--metrics` (501) while `/health` answers 200 for weeks.
|
||||
* **Alert state lives where the tool says it does.** Prometheus keeps rule state in
|
||||
process memory — a restart re-stamps `activeAt`, so a "fresh" alert after a reboot is
|
||||
not evidence of a fresh fault. Alertmanager **does** persist (`--storage.path` on a
|
||||
host volume: silences + notification log, [[Alertmanager]]).
|
||||
* **Never quote a candidate arm's flags or VRAM as fleet state.** A stopped production
|
||||
unit (`failed` after `stop-sigterm timed out. Killing` — the known shutdown shape,
|
||||
[[frank-zealot]]) beside an answering port means the metric measured a scratch boot;
|
||||
the process shape can change between two censuses ten minutes apart.
|
||||
|
||||
## Order and verification
|
||||
|
||||
1. Enumerate the surfaces (status board, metrics targets, alert rules, test
|
||||
|
||||
@@ -3,7 +3,7 @@ type: reference
|
||||
title: Observability Provisioning
|
||||
description: "Monitoring config as code: the config triad, deploy copiers, planner mappings, and where each change type lands."
|
||||
tags: [observability, prometheus, grafana, provisioning, deployment]
|
||||
timestamp: 2026-10-05T00:00:00-07:00
|
||||
timestamp: 2026-10-21T00:00:00-07:00
|
||||
---
|
||||
|
||||
# Observability Provisioning
|
||||
@@ -112,73 +112,10 @@ unified storage — a file existing on disk is not proof the running server
|
||||
loaded it. See [[Stale Signal Audit]] for the never-green-target decision
|
||||
table (refused = outage, 404/HTML = config defect).
|
||||
|
||||
## Current delivered state (delivery chain live 2026-09-17, re-verified 2026-10-18)
|
||||
|
||||
Prometheus → Alertmanager (`dankclaw/observability/quadlet/alertmanager.container`
|
||||
loopback `:9093`) → host-side Signal relay (`scripts/signal-relay.py`, user
|
||||
unit, `192.168.0.143:18093` — the LAN IP, the one bind a container can reach
|
||||
via `host.containers.internal`) → signal-cli, addressed by number
|
||||
`+12062476336` (`scripts/signal-relay.py:36-43`). Every hop re-checked
|
||||
2026-10-18: `ss -ltnp` shows `python3` on `192.168.0.143:18093`
|
||||
`curl -s http://192.168.0.143:18093/health` → `ok`, and
|
||||
`journalctl --user -u signal-relay` ends with successful `sent to +12062476336:`
|
||||
lines for `LlmEndpointDown`, `LlmRequestsQueueing`, `LlmSpecDecodeOff` and
|
||||
`LlmPrefixCacheCollapse` (firing and resolved). Some targets read down for
|
||||
structural reasons (HTML-scraped "probes", Gitea metrics off); the remedy is
|
||||
the blackbox exporter. See [[Prometheus]] → Targets.
|
||||
|
||||
**A llama.cpp endpoint is invisible unless it opts in.** Until 2026-10-18,
|
||||
`up{job="frank-zealot"}` was 0 for the entire retained history of the metric —
|
||||
the oldest sample, 2026-10-11, is already 0 — while the service itself answered
|
||||
`/health` 200 the whole time. Cause: llama.cpp only exports `/metrics` when it
|
||||
is started with the flag, so the scrape was refused at the protocol layer (the
|
||||
flag and its opt-out are on [[frank-zealot]]) and `LlmEndpointDown` (critical)
|
||||
stayed firing:
|
||||
the relay re-sent it on the 4 h `repeat_interval` for days (`journalctl --user
|
||||
-u signal-relay` — firing sends at 01:37, 05:42, 09:47, 13:52, 17:57 and 22:02
|
||||
on 2026-10-12). It finally resolved at `2026-10-18T01:53:21-07:00`, after the
|
||||
metrics opt-in landed that day; the target
|
||||
reports up since (`curl -s http://127.0.0.1:9090/api/v1/targets` 2026-10-18).
|
||||
How many payload generations it spanned is not readable from any host
|
||||
artifact. *(unverified 2026-10-18)*
|
||||
|
||||
Two laws fall out of it. A target can read down beside a healthy service, so
|
||||
read `lastError` from `/api/v1/targets` before believing an outage. And rule
|
||||
state lives in Prometheus' process, not on disk — a restart re-stamps
|
||||
`activeAt` on a long-standing alert, so a "fresh" alert after a reboot is not
|
||||
evidence of a fresh fault. (Alertmanager by contrast **does** persist:
|
||||
`--storage.path=/alertmanager` on a host volume keeps silences and the
|
||||
notification log — `dankclaw/observability/quadlet/alertmanager.container:39-42`
|
||||
`/opt/alertmanager/data`.)
|
||||
|
||||
Remaining known downs are the structural ones above (HTML-scraped probes, Gitea
|
||||
metrics, ComfyUI), and `frank-dragoon` exports `/metrics` natively
|
||||
([[frank-dragoon]]), which is why
|
||||
`frank-dragoon` has always been green. The active-alert count is a
|
||||
moment-in-time value and should never be documented as a constant:
|
||||
`/api/v1/alerts` was empty at the 2026-10-18 morning census, held one firing
|
||||
`LlmPrefixCacheCollapse` (`activeAt` 11:10:48Z, value 2.6e-4) when sampled
|
||||
minutes later, and was empty again by 11:56Z; the relay logged the matching
|
||||
firing and resolved sends at `04:26:21-07:00` and `04:41:21-07:00`.
|
||||
|
||||
**What that zealot scrape measured on 2026-10-18.** As of ~04:30 PDT the
|
||||
production `frank-zealot` root unit was stopped **on purpose** for a benchmark
|
||||
window on the 3090 — the repo Benchmarking Policy says stop the production
|
||||
service before benching — and `:7771` answered from a manually booted candidate
|
||||
arm (`systemctl is-active frank-zealot` → `failed` after
|
||||
`stop-sigterm timed out. Killing`, which is llama.cpp ignoring SIGTERM past
|
||||
`TimeoutStopSec`, the known shutdown shape for these engines). So
|
||||
`up{job="frank-zealot"} == 1` that day measured the trial, not the production
|
||||
shape in `scripts/frank-zealot.env` — that shape is [[frank-zealot]]'s to state,
|
||||
and it is the repo contract.
|
||||
The unit stayed `enabled`; `./scripts/deploy.sh dankclaw --service frank-zealot`
|
||||
reinstalls and starts it, restoring boot persistence and self-healing. Never
|
||||
quote a candidate arm's flags or VRAM as fleet state — the process and its shape
|
||||
changed between two censuses ten minutes apart.
|
||||
|
||||
## Related
|
||||
|
||||
- [[Observability]] — stack topology and the alert delivery chain
|
||||
- [[Stale Signal Audit]] — classifying down targets before "fixing" them
|
||||
- [[Stale Signal Audit]] — classifying down targets before "fixing" them, and the
|
||||
down-beside-healthy / alert-state-persistence / candidate-arm rules
|
||||
- [[Quadlet Lifecycle]] — the units that host these services
|
||||
- [[Deploy State Triage]] — how the planner selects these services
|
||||
|
||||
@@ -70,12 +70,12 @@ back toward 52. Check `VLLM_CUDAGRAPH_*` actually present in
|
||||
|
||||
`--kv-cache-memory-bytes` does NOT appear in the arena arithmetic — the plan
|
||||
reserves the registry price, not the actual pool. Raising the pool consumes
|
||||
real free VRAM without growing the arena; the registry over-prices the pool
|
||||
(9190 B/token planned vs 7,891.8 measured at 200704 fp8 — 7,701 at 262144;
|
||||
**never reuse a B/token ratio across geometries**, and subtract ~3.87 MB
|
||||
block-alignment loss when converting a token target to bytes). The ask lands
|
||||
block-aligned: 3,166,500,000 B → 401,408 tokens; verify with
|
||||
`GPU KV cache size: N tokens, Maximum concurrency … X.Xx`.
|
||||
real free VRAM without growing the arena; the registry OVER-PRICES the pool
|
||||
(era G measured ~14 % under the planned B/token — the gap is the lesson, not
|
||||
the number; **never reuse a B/token ratio across geometries**), and there is a
|
||||
~4 MB block-alignment loss to subtract when converting a token target to
|
||||
bytes. The ask lands block-aligned: verify against the engine's own
|
||||
`GPU KV cache size: N tokens, Maximum concurrency … X.Xx` line at boot.
|
||||
|
||||
**Driver-reported free VRAM after hours of traffic is NOT allocatable
|
||||
arena.** It is the estimator gap: the registry over-prices KV and charges
|
||||
@@ -117,11 +117,13 @@ done, and read post-boot free VRAM per rank (`rocm-smi --showmeminfo vram`)
|
||||
**KV accounting — pin the pool BYTES, floor-sized:** set
|
||||
`--kv-cache-memory-bytes` PER RANK to the contract floor
|
||||
(`seqs × MAXLEN × measured B/token`), not the native window — extra pool bytes
|
||||
come out of the arena/headroom and buy nothing. Measured B/token on this
|
||||
hybrid: **7,701 fp8 / 13,839 bf16** (decomposed: attention 6,138/12,276 +
|
||||
dtype-independent mamba/side 1,563). Measured rates are shape-dependent —
|
||||
7,891.8 at the 200704×3 shape. Capacity does NOT scale with `SEQS`: the
|
||||
planner's KV reserve is for ONE max-model-len request.
|
||||
come out of the arena/headroom and buy nothing. The B/token rate is a property
|
||||
of the GEOMETRY: era G (3 × 200,704 fp8, vLLM fork) measured **7,891.8**;
|
||||
the radiance-era hybrid measured ~7,700 fp8 / ~13,800 bf16 at its own shape.
|
||||
Read your live rate from the boot ledger every time; the decomposition (a
|
||||
dtype-independent mamba/side stream rides on top of the attention KV) is why
|
||||
the ratio moves. Capacity does NOT scale with `SEQS`: the planner's KV reserve
|
||||
is for ONE max-model-len request.
|
||||
|
||||
**Parallelism ceilings you cannot flag past:**
|
||||
|
||||
@@ -179,18 +181,6 @@ Cutover order: ladder → winner passes and serves → repo env + seam + harness
|
||||
(`git stash -u`) if the tree is dirty → deploy (one restart) → verify health
|
||||
+ engine flags in the log.
|
||||
|
||||
**Measured ladder on this box (200704×3, fp8 KV, precise memory):**
|
||||
|
||||
| shape | util | graphs | arena | result |
|
||||
|---|---|---|---|---|
|
||||
| E | 0.98 | 0.55 | 17.22 GiB / 7,522 slots | boot ok |
|
||||
| F | 0.982 | 0.32 | 17.52 GiB / 7,653 slots | boot ok |
|
||||
| 1.0 | 1.00 | 0.07 | — | UNBOOTABLE (worker-init ValueError, both ranks) |
|
||||
| G (deployed) | 0.982 | 0.07 | 17.77 GiB / 7,762 slots | KV 401,408 tokens / 2.00x; 0.89 GiB real free/card held through a live 4,096-token generation at ~38–39 tok/s |
|
||||
|
||||
bf16 ceilings at util 0.98: 3 seqs → 241,664 passes / 245,760 and 262,144 OOM
|
||||
at graph capture; 6 seqs fp8 → 10.70 GiB ask with 9.18 GiB free (fails) —
|
||||
~207k per context max.
|
||||
|
||||
## 7. Arena audit procedure (when asked for "more experts")
|
||||
|
||||
|
||||
@@ -104,14 +104,14 @@
|
||||
{
|
||||
"id": 6,
|
||||
"type": "timeseries",
|
||||
"title": "Memory vs limit (%)",
|
||||
"description": "Working set over the cgroup memory limit (container_spec_memory_limit_bytes). Only containers that carry a limit are shown; approaching 1.0 means an OOM is near. Containers with no limit are omitted.",
|
||||
"title": "Memory vs host total (%)",
|
||||
"description": "Working set over the host's total RAM. The fleet sets NO cgroup memory limits (every container_spec_memory_limit_bytes is 0 — a limit-share panel is structurally empty here), so the honest pressure view is share of the host. The join labels the node exporter's MemTotal onto the cadvisor series by host.",
|
||||
"gridPos": { "h": 9, "w": 12, "x": 12, "y": 11 },
|
||||
"datasource": { "type": "prometheus", "uid": "prometheus" },
|
||||
"fieldConfig": { "defaults": { "unit": "percentunit", "min": 0, "max": 1, "custom": { "fillOpacity": 8, "lineWidth": 1, "stacking": { "mode": "none" } } }, "overrides": [] },
|
||||
"targets": [
|
||||
{
|
||||
"expr": "(container_memory_working_set_bytes{job=~\"cadvisor-.*\", host=~\"$host\", container=~\"$container\"} / on (job, host, container) group_left container_spec_memory_limit_bytes{job=~\"cadvisor-.*\", host=~\"$host\", container=~\"$container\"}) and (container_spec_memory_limit_bytes{job=~\"cadvisor-.*\", host=~\"$host\", container=~\"$container\"} > 0)",
|
||||
"expr": "container_memory_working_set_bytes{job=~\"cadvisor-.*\", host=~\"$host\", container=~\"$container\", container!=\"\"} / on(host) group_left label_replace(node_memory_MemTotal_bytes{job=~\"node-.*\"}, \"host\", \"$1\", \"job\", \"node-(.*)\")",
|
||||
"legendFormat": "{{host}} / {{container}}",
|
||||
"refId": "A"
|
||||
}
|
||||
|
||||
@@ -54,7 +54,7 @@
|
||||
"gridPos": { "h": 4, "w": 6, "x": 12, "y": 0 },
|
||||
"datasource": { "type": "prometheus", "uid": "prometheus" },
|
||||
"fieldConfig": { "defaults": { "unit": "short", "thresholds": { "mode": "absolute", "steps": [{ "color": "green", "value": null }, { "color": "yellow", "value": 0.7 }, { "color": "red", "value": 1 }] } }, "overrides": [] },
|
||||
"targets": [{ "expr": "node_load5{job=~\"$node\"} / count by (job) (node_cpu_seconds_total{job=~\"$node\",mode=\"idle\"})", "legendFormat": "{{job}}", "refId": "A" }]
|
||||
"targets": [{ "expr": "node_load5{job=~\"$node\"} / on(job) count by (job) (node_cpu_seconds_total{job=~\"$node\",mode=\"idle\"})", "legendFormat": "{{job}}", "refId": "A" }]
|
||||
},
|
||||
{
|
||||
"id": 4, "type": "stat", "title": "Open file handles",
|
||||
|
||||
@@ -194,7 +194,7 @@
|
||||
"id": 6,
|
||||
"type": "timeseries",
|
||||
"title": "Firing fleet alerts",
|
||||
"description": "The LLM-fleet rules from shared/prometheus/rules/llm-fleet.yml. A firing alert is delivered to Signal: Prometheus -> Alertmanager (dankclaw quadlet) -> the host-side signal-relay user unit -> signal-cli, since 2026-09-17.",
|
||||
"description": "The LLM-fleet rules from shared/prometheus/rules/llm-fleet.yml. EMPTY IS THE HEALTHY STATE: zero series means nothing is firing (ALERTS only carries active/pending alert series — do not read this panel as broken). A firing alert is delivered to Signal: Prometheus -> Alertmanager (dankclaw quadlet) -> the host-side signal-relay user unit -> signal-cli, since 2026-09-17.",
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
|
||||
Reference in New Issue
Block a user