docs(vault): destroy the dead weight — 53 pages and the change diary
The vault was 155 pages / 31,210 lines, most of it measurement ledgers for payloads that no longer run. A throughput figure for a runtime nobody operates is not a record: AI moves faster than the vault, so the number is wrong before anyone reads it, and a wrong number in a runbook gets believed. Per the new purpose test in `AGENTS.md` → *Documentation Policy*, it is destroyed. Gone (`.AGENTTRASH/20261019-destruction/`, KILL-LIST.txt is the manifest): - the `frank-medium` endpoint and its five era pages, `frank-npu`, FastFlowLM, the ikllama/kingjones/dealign/frank-redo eras, the ling/swift15/freetoken/ubatch campaign pages, qwen3.8 spec-matrix and strix-optimizations, the two "Running Qwen3.6 …" pages; - the whole `Pipeline Parallelism/` folder (no live script passes a pipeline flag) and the multi-device/x4-pair/hybrid/offload experiment pages; - every vLLM page — no vLLM endpoint exists on this fleet; - `LLM/deployment.md` (`Architecture/Deployment Flow.md` is the test-referenced owner) and the external surveys (SGLang, Lemonade, llama.cpp forks/diff, self-hosted app census, the TTS survey) that were obsolete by construction; - `log.md`, 2,380 lines. Git holds the chronology; a causal law belongs to the page that owns the thing it governs, never to a diary. Survivors: 101 pages, 17,542 lines, 0 OKF violations, 0 unresolved links. Eras tables keep the window, payload, runtime, reason and rollback pointer; the rate columns are blanked. The live shapes stay verbatim — those numbers are what the deployer and `models.yml` consume.
This commit is contained in:
1 parent
8775e64ed0
commit
327e78b950
126 files changed
+443
-14157
No files matched your search
@@ -21,7 +21,7 @@ committed.
|
||||
| `shared/prometheus/rules/llm-fleet.yml` | alert rules + recording rules |
|
||||
| `shared/grafana/provisioning/` | datasources + dashboards (provisioned tree) |
|
||||
|
||||
Targets and datasources reference **container names** (`grafana:3000`,
|
||||
Targets and datasources reference **container names** (`grafana:3000`
|
||||
`loki:3100` — `shared/prometheus/prometheus.yml:52-61`), never
|
||||
loopback-published ports; `alloy` is **not scraped at all** (no job in the
|
||||
config — its `127.0.0.1:12345` UI is only probed by the homepage timer,
|
||||
@@ -47,7 +47,7 @@ from `deploy_observability`:
|
||||
- `copy_grafana_provisioning` → `/opt/grafana/provisioning/` (`:286-300`)
|
||||
|
||||
`scripts/deploy-plan.py` maps `shared/prometheus/**` to `prometheus` (plus
|
||||
`alertmanager` when the changed file is `alertmanager.yml`, `:251-256`),
|
||||
`alertmanager` when the changed file is `alertmanager.yml`, `:251-256`)
|
||||
`shared/loki/**` to `loki` and `shared/alloy/**` to `alloy` (`:257-260`), and
|
||||
`shared/grafana/**` to `grafana` (`:261-264`); changes under
|
||||
`dankclaw/observability/` select the touched unit and its dependency consumer
|
||||
@@ -71,9 +71,9 @@ HTTP probe, not a metrics endpoint.
|
||||
|
||||
**Add an alert/recording rule** — edit `shared/prometheus/rules/llm-fleet.yml`.
|
||||
Current delivered set: **4 alert rules + 2 recording rules** —
|
||||
`LlmRequestsQueueing`, `LlmPrefixCacheCollapse`, `LlmSpecDecodeOff`,
|
||||
`LlmRequestsQueueing`, `LlmPrefixCacheCollapse`, `LlmSpecDecodeOff`
|
||||
`LlmEndpointDown`, plus `job:llm_prefix_cache_reuse:rate5m` and
|
||||
`job:llm_spec_accept_length:rate5m` (`shared/prometheus/rules/llm-fleet.yml:32-110`),
|
||||
`job:llm_spec_accept_length:rate5m` (`shared/prometheus/rules/llm-fleet.yml:32-110`)
|
||||
confirmed live: `curl -s http://127.0.0.1:9090/api/v1/rules` (2026-10-18)
|
||||
reports 4 alerting + 2 recording. Rules reload with the prometheus service.
|
||||
|
||||
@@ -101,7 +101,7 @@ curl -s -u admin:<pw> http://127.0.0.1:3002/api/datasources | python3 -m json.to
|
||||
|
||||
The two Prometheus queries were run on 2026-10-18 and are what every number on
|
||||
this page comes from: 4 alerting + 2 recording rules, and `up` for
|
||||
`prometheus`, `node-dankclaw`, `node-dankserver`, `grafana`, `loki`,
|
||||
`prometheus`, `node-dankclaw`, `node-dankserver`, `grafana`, `loki`
|
||||
`frank-zealot`, `frank-dragoon`, `frank-archon` with the four structural downs
|
||||
listed in [[Prometheus]]. The Grafana query needs the local admin password and
|
||||
was not run here; the filesystem side (`ls /opt/grafana/provisioning/*`) is
|
||||
@@ -114,12 +114,12 @@ table (refused = outage, 404/HTML = config defect).
|
||||
|
||||
## Current delivered state (delivery chain live 2026-09-17, re-verified 2026-10-18)
|
||||
|
||||
Prometheus → Alertmanager (`dankclaw/observability/quadlet/alertmanager.container`,
|
||||
Prometheus → Alertmanager (`dankclaw/observability/quadlet/alertmanager.container`
|
||||
loopback `:9093`) → host-side Signal relay (`scripts/signal-relay.py`, user
|
||||
unit, `192.168.0.143:18093` — the LAN IP, the one bind a container can reach
|
||||
via `host.containers.internal`) → signal-cli, addressed by number
|
||||
`+12062476336` (`scripts/signal-relay.py:36-43`). Every hop re-checked
|
||||
2026-10-18: `ss -ltnp` shows `python3` on `192.168.0.143:18093`,
|
||||
2026-10-18: `ss -ltnp` shows `python3` on `192.168.0.143:18093`
|
||||
`curl -s http://192.168.0.143:18093/health` → `ok`, and
|
||||
`journalctl --user -u signal-relay` ends with successful `sent to +12062476336:`
|
||||
lines for `LlmEndpointDown`, `LlmRequestsQueueing`, `LlmSpecDecodeOff` and
|
||||
@@ -148,7 +148,7 @@ state lives in Prometheus' process, not on disk — a restart re-stamps
|
||||
`activeAt` on a long-standing alert, so a "fresh" alert after a reboot is not
|
||||
evidence of a fresh fault. (Alertmanager by contrast **does** persist:
|
||||
`--storage.path=/alertmanager` on a host volume keeps silences and the
|
||||
notification log — `dankclaw/observability/quadlet/alertmanager.container:39-42`,
|
||||
notification log — `dankclaw/observability/quadlet/alertmanager.container:39-42`
|
||||
`/opt/alertmanager/data`.)
|
||||
|
||||
Remaining known downs are the structural ones above (HTML-scraped probes, Gitea
|
||||
|
||||
Reference in new issue
Block a user