feat(monitoring,harness): follow frank-medium onto vllm:* metrics and fix red-by-construction targets

The radiance TP=2 cutover changed the metric namespace frank-medium exports
(llamacpp:* -> vllm:*) and left two scrape targets addressing host processes
by an address a container cannot reach.

- shared/prometheus/rules/llm-fleet.yml: two namespaces side by side. Added
  LlmRequestsWaiting, LlmKvCacheSaturated (vllm-only gauges llama.cpp never
  had), the job:llm_vllm_prefix_cache_reuse:rate5m and
  job:llm_spec_accept_length:rate5m recording rules, and re-cut
  LlmPrefixCacheCollapse / LlmSpecDecodeOff to OR both families with
  per-family traffic floors so an idle endpoint cannot fire them. Every
  expression validated against live Prometheus.
- shared/prometheus/prometheus.yml: frank-small -> host.containers.internal:7771
  and frank-large scraped at host.containers.internal:7770. Both are HOST
  processes on Dankclaw and netavark DNATs published container ports only, so
  the LAN-IP targets were red by construction; frank-large's CIRU llama.cpp
  build does export the llamacpp:* family, contra the old comment.
- tests/observability_quadlet.bats: the two seams now encode the new contract
  (per-endpoint reachability by process type; every named series must be on the
  measured list for its engine; engine-independent rules must name both).
- homepage docs.html/services.json: frank-medium sees pictures, frank-large is
  CIRU Orca with a single seat (was halogen w4b, two seats, 200k).
- docs: radiance runbook is ACTIVE (deployed flags, three-arm battery, verified
  capabilities, rollback, replica-era sections labelled); llama.cpp runbook
  carries the SUPERSEDED banner; Dual-R9700-Cooperation states the measured
  verdict; Observability records the namespace split and the DNAT lesson;
  AGENTS.md topology/frank-medium/Model Config/Benchmarking Policy/runbook
  index flipped; both indexes updated; work-log entry added.

Also repaired agent-pass damage: prose prepended above frontmatter openers,
duplicated runbook bodies, a literal tool-call token pasted into a runbook, and
an index destroyed by diff artifacts (restored from HEAD, flips re-applied).
Vault link check: 128 pages, 0 unresolved. ./tests/run.sh: 325 pass.
This commit is contained in:
Frank committed 2026-09-20 19:35:56 -07:00
1 parent c2116658f8
commit 08f1c7ad09
13 files changed
+687 -286

No files matched your search

+16 -20
View File
@@ -115,26 +115,22 @@ The LLM inference fleet on the two hosts. Full index:
- [[deployment]] — unified deploy and update process for both servers
- [[frank-redo-2026-08-17]] — the 2026-08-17 topology change
- [[frank-medium-llamacpp-dflash2]] — the ACTIVE frank-medium runbook since
2026-09-16 (model swapped 2026-09-19): mainline llama.cpp (gfx1201 HIP build)
serving orcarouter/Qwen3.8-27B-Uncensored-GGUF at Q8_0 with the DFlash2
drafter (z-lab/Qwen3.8-27B-DFlash2-GGUF at Q4_K_M) on Dankserver's 2x Radeon
AI PRO R9700 as one layer-split process behind the :7770 endpoint, balanced
32:33 over two 163,840-token slots — no tensor parallelism: the runtime ships
RCCL-less, and the collective has not been re-probed since the 2026-09-20
board swap cleared the atomics gate
- [[frank-medium-radiance-mxfp4]] — RETIRED 2026-09-16: the vllm-radiance
MXFP4 stack serving just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq on
Dankserver's 2x Radeon AI PRO R9700 as two single-GPU replicas behind a
round-robin proxy; kept as the record of the deployment
[[frank-medium-llamacpp-dflash2]] replaced, plus the PCIe-atomics evidence
that ruled out TP=2 on the previous board (gate cleared 2026-09-20)
- [[Dual-R9700-Cooperation]] — why the two R9700s *could not* run tensor-parallel
together: the PCIe-atomics gate quoted from ROCclr, the measured RCCL matrix
(collectives *and* P2P `send`/`recv` both failed on card #2, which ruled out
pipeline parallelism too), what was closed, and the re-measured 2026-09-20
topology after the ProArt B850-Creator x8/x8 swap — both chains `Routing+`,
both endpoints `ReqEn+`, so TP=2 is unblocked but still unprobed
- [[frank-medium-radiance-mxfp4]] — the ACTIVE frank-medium runbook since
2026-09-20: vllm-radiance MXFP4 as ONE tensor-parallel process (TP=2 over the
R4D one-shot allreduce) serving just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq
with the z-lab DFlash2-FP8 drafter on Dankserver's 2x Radeon AI PRO R9700
behind the :7770 endpoint — deployed flags, the three-arm battery that chose
it, verified capabilities, and the record of the 2026-09-16 two-replica era
- [[frank-medium-llamacpp-dflash2]] — RETIRED 2026-09-16 → 2026-09-20:
mainline llama.cpp (gfx1201 HIP build) serving orcarouter Qwen3.8-27B at
Q8_0 with the DFlash2 drafter as ONE layer-split process (32:33, two
163,840-token slots); holds the batch/ubatch optimum curve and the
per-layer split arithmetic the rollback recipe points back to
- [[Dual-R9700-Cooperation]] — whether the two R9700s can run ONE model: the
RCCL/PCIe-atomics evidence on the old TUF GAMING B650-E (which closed tensor
parallelism too), what the 2026-09-20 ProArt B850-Creator x8/x8 swap changed
— both chains `Routing+`, both endpoints `ReqEn+` — and the measured
three-way verdict: radiance TP=2 won and is the deployed shape
- [[frank-medium-rocmfpx]] — PRIOR frank-medium deployment (llama.cpp era):
charlie12345/ROCmFPX serving kingjones777/
Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF; kept as the record of the runtime it