1. Vault audit tranche 1 (a61308d) — one-owner dedupe of the observability cluster + dead-shape number destruction (detail in the commit message; scout verdicts were individually re-verified first — three were fabricated and rejected).
2. Grafana dashboard census fixes (c96d022) — all 5 dashboards / 56 panel targets re-evaluated against the live datasources:
Panel
Verdict
Fix
host-system Load 5m / core
structurally empty (bare / drops pairs: left {job,instance} vs right {job})
/ on(job) — 2 live series
containers Memory vs limit (%)
structurally empty (every cgroup limit on this fleet is 0; > 0 can never match)
rewritten to share-of-host via label_replace join — 62 live series
llm-fleet Firing fleet alerts
legitimately empty when healthy (ALERTS only carries firing/pending)
description now states EMPTY IS HEALTHY
logs-overview, observability, all other targets
returning data
none
Census method note (worth knowing for next time): Prometheus and Loki bind 127.0.0.1 on Dankclaw — from the dev box every query is connection-refused; run the evaluation on the host. And group_left may be followed by nothing but a bare selector — parenthesizing the right-hand vector is a parse error.
Verification
./tests/run.sh: 490/490.
Fixed expressions evaluated live with template vars expanded: 62 and 2 series.
Landing this and deploy.sh dankclaw --service grafana re-provisions the dashboards (repo is the owner; deploy copies the JSONs).
## Contents (2 commits)
**1. Vault audit tranche 1 (`a61308d`)** — one-owner dedupe of the observability cluster + dead-shape number destruction (detail in the commit message; scout verdicts were individually re-verified first — three were fabricated and rejected).
**2. Grafana dashboard census fixes (`c96d022`)** — all 5 dashboards / 56 panel targets re-evaluated against the live datasources:
| Panel | Verdict | Fix |
|---|---|---|
| host-system `Load 5m / core` | structurally empty (bare `/` drops pairs: left `{job,instance}` vs right `{job}`) | `/ on(job)` — 2 live series |
| containers `Memory vs limit (%)` | structurally empty (every cgroup limit on this fleet is 0; `> 0` can never match) | rewritten to share-of-host via `label_replace` join — 62 live series |
| llm-fleet `Firing fleet alerts` | legitimately empty when healthy (`ALERTS` only carries firing/pending) | description now states EMPTY IS HEALTHY |
| logs-overview, observability, all other targets | returning data | none |
Census method note (worth knowing for next time): Prometheus and Loki bind `127.0.0.1` on Dankclaw — from the dev box every query is connection-refused; run the evaluation **on the host**. And `group_left` may be followed by nothing but a bare selector — parenthesizing the right-hand vector is a parse error.
## Verification
- `./tests/run.sh`: 490/490.
- Fixed expressions evaluated live with template vars expanded: 62 and 2 series.
- Landing this and `deploy.sh dankclaw --service grafana` re-provisions the dashboards (repo is the owner; deploy copies the JSONs).
---
cwd: /home/blank/.omp/wt/wt-20261020-125416-6d1fc1f · session: 01a15d93-efb8-7108-b94d-4fabb14597ef
Purpose-test pass over the audit findings (every verdict re-verified
against the file before touching it; three scout findings were hallucinated
— a nonexistent filename, my own changelog line, and sanctioned Eras rows):
- Observability Provisioning: destroyed the 63-line 'Current delivered
state' block — chain topology/relay/delivery restates [[Observability]]
and [[Alertmanager]] (those are the owners). Its three sole-custody laws
MOVED to Stale Signal Audit as 'Three rules that outlive the incident':
down-beside-healthy -> read lastError; Prometheus alert state in process
vs Alertmanager's --storage.path persistence; candidate-arm metrics are
never fleet state.
- RDNA4 MoE VRAM budget: kept the laws (registry over-prices the pool;
never reuse a B/token ratio across geometries; pin pool bytes
floor-sized), era-labelled the rates, destroyed the measured ladder
table for the superseded shape-G/bf16 rungs.
- Stale Signal Audit: new rules section + Provisioning Related link.
check-okf 109/0, vault links 0 unresolved, docs seams green.
Census: all 5 provisioned dashboards, 56 targets, every expression
re-evaluated against the live Prometheus/Loki with template vars expanded
(queries must run from the host: both APIs bind 127.0.0.1; from the dev box
that is connection-refused, which is NOT a dashboard defect).
- host-system 'Load 5m / core' was STRUCTURALLY EMPTY: bare '/' drops
non-matching pairs and the right side collapses to {job} while the left
carries {job,instance} -> 0 series forever. Fixed with / on(job); proven
live (2 series).
- containers 'Memory vs limit (%)' was STRUCTURALLY EMPTY: every
container_spec_memory_limit_bytes on this fleet is 0 (no cgroup limits),
so the '> 0' filter can never match. Rewritten to share-of-host-total via
label_replace join of the node exporter's MemTotal by host (the parens
form is a PromQL parse error: nothing may follow group_left); 62 series
live. Panel unit percentunit kept -> expression returns a fraction.
- llm-fleet 'Firing fleet alerts' is legitimately empty when healthy
(ALERTS carries only firing/pending series); description now says so,
per the Stale Signal rule that 0-series-on-ALERTS is not a defect.
logs-overview and observability: all targets already returning data.
dank
merged commit 127621bdd7 into main2026-10-21 09:28:44 -07:00
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Contents (2 commits)
1. Vault audit tranche 1 (
a61308d) — one-owner dedupe of the observability cluster + dead-shape number destruction (detail in the commit message; scout verdicts were individually re-verified first — three were fabricated and rejected).2. Grafana dashboard census fixes (
c96d022) — all 5 dashboards / 56 panel targets re-evaluated against the live datasources:Load 5m / core/drops pairs: left{job,instance}vs right{job})/ on(job)— 2 live seriesMemory vs limit (%)> 0can never match)label_replacejoin — 62 live seriesFiring fleet alertsALERTSonly carries firing/pending)Census method note (worth knowing for next time): Prometheus and Loki bind
127.0.0.1on Dankclaw — from the dev box every query is connection-refused; run the evaluation on the host. Andgroup_leftmay be followed by nothing but a bare selector — parenthesizing the right-hand vector is a parse error.Verification
./tests/run.sh: 490/490.deploy.sh dankclaw --service grafanare-provisions the dashboards (repo is the owner; deploy copies the JSONs).cwd: /home/blank/.omp/wt/wt-20261020-125416-6d1fc1f · session: 01a15d93-efb8-7108-b94d-4fabb14597ef
Census: all 5 provisioned dashboards, 56 targets, every expression re-evaluated against the live Prometheus/Loki with template vars expanded (queries must run from the host: both APIs bind 127.0.0.1; from the dev box that is connection-refused, which is NOT a dashboard defect). - host-system 'Load 5m / core' was STRUCTURALLY EMPTY: bare '/' drops non-matching pairs and the right side collapses to {job} while the left carries {job,instance} -> 0 series forever. Fixed with / on(job); proven live (2 series). - containers 'Memory vs limit (%)' was STRUCTURALLY EMPTY: every container_spec_memory_limit_bytes on this fleet is 0 (no cgroup limits), so the '> 0' filter can never match. Rewritten to share-of-host-total via label_replace join of the node exporter's MemTotal by host (the parens form is a PromQL parse error: nothing may follow group_left); 62 series live. Panel unit percentunit kept -> expression returns a fraction. - llm-fleet 'Firing fleet alerts' is legitimately empty when healthy (ALERTS carries only firing/pending series); description now says so, per the Stale Signal rule that 0-series-on-ALERTS is not a defect. logs-overview and observability: all targets already returning data.