Prometheus/Observability/Provisioning pages: every job up after the
blackbox swap, the copier list learns copy_blackbox_config and the
alloy validate gate, the config table gains alloy + blackbox rows, and
the logs bullet states the stream-label law instead of the retired
'awaiting deploy' note.
copy_grafana_provisioning named llm-fleet.json alone, so a new board in
shared/grafana/provisioning/dashboards/ silently never reached
/opt/grafana/provisioning. Measured 2026-10-21: the containers board was
merged, seam-tested, and documented as working since 2026-10-20, yet was
NEVER provisioned — its panel fixes were undeployable; the other four
boards existed on disk only because someone hand-copied them once (and
host-system.json there had gone stale against the repo). The copier now
globs the directory, and GRAFANA_PROV_ROOT lets the new bats case run
the copier for real into a sandbox (red against the old hand-listed
body, green against the glob). Vault: the copier law + the containers
incident on Observability Provisioning, the corrected dashboards claim
on Observability.
Note: a mktemp sandbox + trap RETURN races bats' trap machinery (the
dir vanishes mid-function); BATS_TEST_TMPDIR is the safe idiom.
Purpose-test pass over the audit findings (every verdict re-verified
against the file before touching it; three scout findings were hallucinated
— a nonexistent filename, my own changelog line, and sanctioned Eras rows):
- Observability Provisioning: destroyed the 63-line 'Current delivered
state' block — chain topology/relay/delivery restates [[Observability]]
and [[Alertmanager]] (those are the owners). Its three sole-custody laws
MOVED to Stale Signal Audit as 'Three rules that outlive the incident':
down-beside-healthy -> read lastError; Prometheus alert state in process
vs Alertmanager's --storage.path persistence; candidate-arm metrics are
never fleet state.
- RDNA4 MoE VRAM budget: kept the laws (registry over-prices the pool;
never reuse a B/token ratio across geometries; pin pool bytes
floor-sized), era-labelled the rates, destroyed the measured ladder
table for the superseded shape-G/bf16 rungs.
- Stale Signal Audit: new rules section + Provisioning Related link.
check-okf 109/0, vault links 0 unresolved, docs seams green.
The vault was 155 pages / 31,210 lines, most of it measurement ledgers for
payloads that no longer run. A throughput figure for a runtime nobody operates is
not a record: AI moves faster than the vault, so the number is wrong before
anyone reads it, and a wrong number in a runbook gets believed. Per the new
purpose test in `AGENTS.md` → *Documentation Policy*, it is destroyed.
Gone (`.AGENTTRASH/20261019-destruction/`, KILL-LIST.txt is the manifest):
- the `frank-medium` endpoint and its five era pages, `frank-npu`, FastFlowLM,
the ikllama/kingjones/dealign/frank-redo eras, the ling/swift15/freetoken/ubatch
campaign pages, qwen3.8 spec-matrix and strix-optimizations, the two
"Running Qwen3.6 …" pages;
- the whole `Pipeline Parallelism/` folder (no live script passes a pipeline flag)
and the multi-device/x4-pair/hybrid/offload experiment pages;
- every vLLM page — no vLLM endpoint exists on this fleet;
- `LLM/deployment.md` (`Architecture/Deployment Flow.md` is the test-referenced
owner) and the external surveys (SGLang, Lemonade, llama.cpp forks/diff,
self-hosted app census, the TTS survey) that were obsolete by construction;
- `log.md`, 2,380 lines. Git holds the chronology; a causal law belongs to the
page that owns the thing it governs, never to a diary.
Survivors: 101 pages, 17,542 lines, 0 OKF violations, 0 unresolved links. Eras
tables keep the window, payload, runtime, reason and rollback pointer; the rate
columns are blanked. The live shapes stay verbatim — those numbers are what the
deployer and `models.yml` consume.
Structural repair:
- 13 pages were the same document twice (each exactly 2x its HEAD size),
a second frontmatter block whose closing fence had been glued onto a body
line by a frontmatter rewrite that re-read its own output. Un-doubled by
cutting at the second frontmatter and keeping the copy whose body is a
superset; the orphan census reports 0 lost tokens against HEAD.
- log.md: one entry per date, bare `## YYYY-MM-DD` headings, newest first —
66 dated headings merged into 34, titles demoted to bold leads, every
content line verified present. The file had been descending at the head
and ascending at the tail.
Ownership moves:
- New `LLM/frank-medium.md` is the locator for the retired Dankserver
endpoint: era windows quoted from each era page's own banner, no numbers
restated, and the two unresolved date conflicts recorded instead of
silently resolved. `frank-archon.md` links to it instead of restating the
pre-rename window.
- The four Pipeline-Parallelism stubs (frontmatter only, no body) are gone;
`Multi-Node Runtimes.md` owns runtime construction, launch, mixed-vendor
verdicts and cross-split spec decode, and the four links that pointed at
the stubs were repointed at it or at the drafter page.
- Authentik's three stub pages fold into `Authentik.md`, with the process
detail and the outpost reference split out so the parent stays under the
length hint.
Verified corrections (host census + live probe, 2026-10-19):
- The MXFP4 checkpoint two pages called the "on-disk rollback" is gone:
/home/dank/models/ holds StillDeadcode, rad and tpl.jinja. Stated once, on
the era page that owns that payload; the swap page now links to it.
- frank-medium-radiance-mxfp4: the 4.24/7 accept rate belongs to the DFlash2
arm measured on the FP8 target, not to the MXFP4 era; the 3-family soak run
census is kept once (155 requests) instead of twice (~120 and 155).
- Fleet shapes re-confirmed against the live endpoints before touching any
page: 3x200000 fp8 KV, 2x200704, 2x262144.
Index rows are regenerated, not hand-carved. Net -1,849 vault lines.
18 pages cut their per-endpoint shape, payload, port and window claims and
link [[frank-archon]] / [[frank-dragoon]] / [[frank-zealot]] instead. Several
of the cut claims had already drifted -- the stack page still described the
kingjones ROCmFP4 era on ROCmFPX, OpenWebUI carried the 27B/3-slot bench
narrative, Hermes named a retired max_model_len and the old service name, and
the OMP harness page's live-window block had drifted off frank-zealot's real
262,144 (it is now labelled a dated worked example whose authority is the
endpoint page).
Two facts gain a single owner instead of two: the up{job=~"frank-.*"} scope
lives in Infrastructure/Prometheus.md, and the blackbox/Anubis probe form in
Architecture/Stale Signal Audit.md, each cited from the other page.
Voice Packs and Frank TTS carried live-state claims that contradicted the
engine page's 2026-10-18 deploy record (nothing listening on 7772/8880, model
dir absent); they now point at the pages that own host state.
The docs-fact census wave re-read the live hosts and re-keyed every page
it could: runtime/payload identities, ports, sizes, ledger figures, unit
state, and the retirement banners that drifted as the fleet moved. The
wave was interrupted mid-run, so this commit carries only what passed the
integrity gates: every touched page parses (front matter or H1+status
convention intact, balanced fences, no truncated tables, trailing
newline), the docs_completeness and link-integrity seams are green, and
bats is 384/384.
Numbers that this session proved on live hosts are keyed to the measured
value, not the page's earlier claim.
The scrape has never worked: lastError is "server returned HTTP status 501 Not
Implemented" on http://host.containers.internal:7771/metrics, and the oldest
retained sample of up{job="frank-zealot"} (2026-10-11) is already 0 -- the target
has been red across four payload generations while /health answered 200 the whole
time, keeping the critical LlmEndpointDown alert permanently active. llama.cpp
gates /metrics behind --metrics; halogen serves it natively, which is why the
asymmetry looked like a 3090-only problem.
METRICS=1 in the shipped env, --metrics reaches the argv through EXTRA_ARGS, and
the seam asserts both directions (default ships the flag, METRICS=0 strips it).
Diagnosed via /api/v1/targets lastError rather than the alert list, which is also
the lesson: Alertmanager here has no persistence, so a reboot re-stamps startsAt
and a weeks-old fault looks fresh.
Frank