docs(vault): audit tranche 1 — one-owner dedupe of the observability cluster, dead-shape numbers destroyed
Purpose-test pass over the audit findings (every verdict re-verified against the file before touching it; three scout findings were hallucinated — a nonexistent filename, my own changelog line, and sanctioned Eras rows): - Observability Provisioning: destroyed the 63-line 'Current delivered state' block — chain topology/relay/delivery restates [[Observability]] and [[Alertmanager]] (those are the owners). Its three sole-custody laws MOVED to Stale Signal Audit as 'Three rules that outlive the incident': down-beside-healthy -> read lastError; Prometheus alert state in process vs Alertmanager's --storage.path persistence; candidate-arm metrics are never fleet state. - RDNA4 MoE VRAM budget: kept the laws (registry over-prices the pool; never reuse a B/token ratio across geometries; pin pool bytes floor-sized), era-labelled the rates, destroyed the measured ladder table for the superseded shape-G/bf16 rungs. - Stale Signal Audit: new rules section + Provisioning Related link. check-okf 109/0, vault links 0 unresolved, docs seams green.
This commit is contained in:
1 parent
c6e2887f94
commit
a61308d72e
3 files changed
+30
-89
No files matched your search
@@ -3,7 +3,7 @@ type: reference
|
||||
title: Observability Provisioning
|
||||
description: "Monitoring config as code: the config triad, deploy copiers, planner mappings, and where each change type lands."
|
||||
tags: [observability, prometheus, grafana, provisioning, deployment]
|
||||
timestamp: 2026-10-05T00:00:00-07:00
|
||||
timestamp: 2026-10-21T00:00:00-07:00
|
||||
---
|
||||
|
||||
# Observability Provisioning
|
||||
@@ -112,73 +112,10 @@ unified storage — a file existing on disk is not proof the running server
|
||||
loaded it. See [[Stale Signal Audit]] for the never-green-target decision
|
||||
table (refused = outage, 404/HTML = config defect).
|
||||
|
||||
## Current delivered state (delivery chain live 2026-09-17, re-verified 2026-10-18)
|
||||
|
||||
Prometheus → Alertmanager (`dankclaw/observability/quadlet/alertmanager.container`
|
||||
loopback `:9093`) → host-side Signal relay (`scripts/signal-relay.py`, user
|
||||
unit, `192.168.0.143:18093` — the LAN IP, the one bind a container can reach
|
||||
via `host.containers.internal`) → signal-cli, addressed by number
|
||||
`+12062476336` (`scripts/signal-relay.py:36-43`). Every hop re-checked
|
||||
2026-10-18: `ss -ltnp` shows `python3` on `192.168.0.143:18093`
|
||||
`curl -s http://192.168.0.143:18093/health` → `ok`, and
|
||||
`journalctl --user -u signal-relay` ends with successful `sent to +12062476336:`
|
||||
lines for `LlmEndpointDown`, `LlmRequestsQueueing`, `LlmSpecDecodeOff` and
|
||||
`LlmPrefixCacheCollapse` (firing and resolved). Some targets read down for
|
||||
structural reasons (HTML-scraped "probes", Gitea metrics off); the remedy is
|
||||
the blackbox exporter. See [[Prometheus]] → Targets.
|
||||
|
||||
**A llama.cpp endpoint is invisible unless it opts in.** Until 2026-10-18,
|
||||
`up{job="frank-zealot"}` was 0 for the entire retained history of the metric —
|
||||
the oldest sample, 2026-10-11, is already 0 — while the service itself answered
|
||||
`/health` 200 the whole time. Cause: llama.cpp only exports `/metrics` when it
|
||||
is started with the flag, so the scrape was refused at the protocol layer (the
|
||||
flag and its opt-out are on [[frank-zealot]]) and `LlmEndpointDown` (critical)
|
||||
stayed firing:
|
||||
the relay re-sent it on the 4 h `repeat_interval` for days (`journalctl --user
|
||||
-u signal-relay` — firing sends at 01:37, 05:42, 09:47, 13:52, 17:57 and 22:02
|
||||
on 2026-10-12). It finally resolved at `2026-10-18T01:53:21-07:00`, after the
|
||||
metrics opt-in landed that day; the target
|
||||
reports up since (`curl -s http://127.0.0.1:9090/api/v1/targets` 2026-10-18).
|
||||
How many payload generations it spanned is not readable from any host
|
||||
artifact. *(unverified 2026-10-18)*
|
||||
|
||||
Two laws fall out of it. A target can read down beside a healthy service, so
|
||||
read `lastError` from `/api/v1/targets` before believing an outage. And rule
|
||||
state lives in Prometheus' process, not on disk — a restart re-stamps
|
||||
`activeAt` on a long-standing alert, so a "fresh" alert after a reboot is not
|
||||
evidence of a fresh fault. (Alertmanager by contrast **does** persist:
|
||||
`--storage.path=/alertmanager` on a host volume keeps silences and the
|
||||
notification log — `dankclaw/observability/quadlet/alertmanager.container:39-42`
|
||||
`/opt/alertmanager/data`.)
|
||||
|
||||
Remaining known downs are the structural ones above (HTML-scraped probes, Gitea
|
||||
metrics, ComfyUI), and `frank-dragoon` exports `/metrics` natively
|
||||
([[frank-dragoon]]), which is why
|
||||
`frank-dragoon` has always been green. The active-alert count is a
|
||||
moment-in-time value and should never be documented as a constant:
|
||||
`/api/v1/alerts` was empty at the 2026-10-18 morning census, held one firing
|
||||
`LlmPrefixCacheCollapse` (`activeAt` 11:10:48Z, value 2.6e-4) when sampled
|
||||
minutes later, and was empty again by 11:56Z; the relay logged the matching
|
||||
firing and resolved sends at `04:26:21-07:00` and `04:41:21-07:00`.
|
||||
|
||||
**What that zealot scrape measured on 2026-10-18.** As of ~04:30 PDT the
|
||||
production `frank-zealot` root unit was stopped **on purpose** for a benchmark
|
||||
window on the 3090 — the repo Benchmarking Policy says stop the production
|
||||
service before benching — and `:7771` answered from a manually booted candidate
|
||||
arm (`systemctl is-active frank-zealot` → `failed` after
|
||||
`stop-sigterm timed out. Killing`, which is llama.cpp ignoring SIGTERM past
|
||||
`TimeoutStopSec`, the known shutdown shape for these engines). So
|
||||
`up{job="frank-zealot"} == 1` that day measured the trial, not the production
|
||||
shape in `scripts/frank-zealot.env` — that shape is [[frank-zealot]]'s to state,
|
||||
and it is the repo contract.
|
||||
The unit stayed `enabled`; `./scripts/deploy.sh dankclaw --service frank-zealot`
|
||||
reinstalls and starts it, restoring boot persistence and self-healing. Never
|
||||
quote a candidate arm's flags or VRAM as fleet state — the process and its shape
|
||||
changed between two censuses ten minutes apart.
|
||||
|
||||
## Related
|
||||
|
||||
- [[Observability]] — stack topology and the alert delivery chain
|
||||
- [[Stale Signal Audit]] — classifying down targets before "fixing" them
|
||||
- [[Stale Signal Audit]] — classifying down targets before "fixing" them, and the
|
||||
down-beside-healthy / alert-state-persistence / candidate-arm rules
|
||||
- [[Quadlet Lifecycle]] — the units that host these services
|
||||
- [[Deploy State Triage]] — how the planner selects these services
|
||||
Reference in new issue
Block a user