feat(monitoring,harness): follow frank-medium onto vllm:* metrics and fix red-by-construction targets

The radiance TP=2 cutover changed the metric namespace frank-medium exports
(llamacpp:* -> vllm:*) and left two scrape targets addressing host processes
by an address a container cannot reach.

- shared/prometheus/rules/llm-fleet.yml: two namespaces side by side. Added
  LlmRequestsWaiting, LlmKvCacheSaturated (vllm-only gauges llama.cpp never
  had), the job:llm_vllm_prefix_cache_reuse:rate5m and
  job:llm_spec_accept_length:rate5m recording rules, and re-cut
  LlmPrefixCacheCollapse / LlmSpecDecodeOff to OR both families with
  per-family traffic floors so an idle endpoint cannot fire them. Every
  expression validated against live Prometheus.
- shared/prometheus/prometheus.yml: frank-small -> host.containers.internal:7771
  and frank-large scraped at host.containers.internal:7770. Both are HOST
  processes on Dankclaw and netavark DNATs published container ports only, so
  the LAN-IP targets were red by construction; frank-large's CIRU llama.cpp
  build does export the llamacpp:* family, contra the old comment.
- tests/observability_quadlet.bats: the two seams now encode the new contract
  (per-endpoint reachability by process type; every named series must be on the
  measured list for its engine; engine-independent rules must name both).
- homepage docs.html/services.json: frank-medium sees pictures, frank-large is
  CIRU Orca with a single seat (was halogen w4b, two seats, 200k).
- docs: radiance runbook is ACTIVE (deployed flags, three-arm battery, verified
  capabilities, rollback, replica-era sections labelled); llama.cpp runbook
  carries the SUPERSEDED banner; Dual-R9700-Cooperation states the measured
  verdict; Observability records the namespace split and the DNAT lesson;
  AGENTS.md topology/frank-medium/Model Config/Benchmarking Policy/runbook
  index flipped; both indexes updated; work-log entry added.

Also repaired agent-pass damage: prose prepended above frontmatter openers,
duplicated runbook bodies, a literal tool-call token pasted into a runbook, and
an index destroyed by diff artifacts (restored from HEAD, flips re-applied).
Vault link check: 128 pages, 0 unresolved. ./tests/run.sh: 325 pass.

serverAppConfigs-docs/LLM/Dual-R9700-Cooperation.md

@@ -14,12 +14,11 @@ timestamp: 2026-09-16
 
 # Dual R9700 Cooperation
 
-Dankserver's two R9700s serve `frank-medium` as **one llama.cpp process with a
-layer split** ([[frank-medium-llamacpp-dflash2]]). This page records why true
-tensor parallelism was impossible on the previous board (TUF GAMING B650-E
-WIFI), and what the 2026-09-20 board swap changed. See
-[[frank-medium-radiance-mxfp4]] for the replica-pair deployment that ran under
-the old constraint.
+Dankserver's two R9700s serve `frank-medium` as **one vLLM-radiance process at
+tensor_parallel_size 2** ([[frank-medium-radiance-mxfp4]]). This page records why
+true tensor parallelism was impossible on the previous board (TUF GAMING B650-E
+WIFI), and what the 2026-09-20 board swap changed. The llama.cpp layer split that
+served the endpoint until that swap is [[frank-medium-llamacpp-dflash2]].
 
 ## The short answer
 
@@ -35,18 +34,21 @@ bifurcates the CPU's GPP ports **x8/x8**: the board-swap path this page
 recommended, at B850 rather than X870E money. Both cards now hang off CPU root
 ports (`00:01.1`, `00:01.3`), every bridge on both chains reads
 `AtomicOpsCap: Routing+`, and **both endpoints read `AtomicOpsCtl: ReqEn+`** —
-the exact precondition this page's own [test plan](#test-plan) names.
+the exact precondition this page's own test plan named, and it is now
+[measured](#measured-verdict-2026-09-20) rather than assumed.
 
-What that buys, and what it does not:
+What that bought:
 
-- **Unblocked in principle:** vLLM TP=2, vLLM PP=2, and llama.cpp
-  `--split-mode row`. All three died on the same hostcall gate.
-- **Not yet measured.** Nothing has dispatched a collective on this host since
-  the swap. The 2-rank probe is the remaining gate, and policy is stop-and-mask
-  production first.
-- **Not needed for capability.** The layer split already puts both cards on one
-  model. This is a performance question: an RCCL-backed tensor split, and the
-  single pooled KV that comes with it.
+- **Both cooperation modes now work, and they were both run.** vLLM TP=2 booted
+  and served (its R4D allreduce needs no RCCL collective at all), and llama.cpp
+  `--split-mode tensor` booted and served against an RCCL-backed build — so a
+  collective HAS been dispatched on this host since the swap.
+- **One of them is production.** TP=2 wins on every axis measured, by 2.1x on
+  decode and 3.6x at 4-way concurrency, so it replaced the layer split on the day
+  of measurement. See [[frank-medium-radiance-mxfp4]] for the deployed shape.
+- **The drafter, not the collective, is what llama.cpp lost.** Its tensor mode
+  could not carry the DFlash2 drafter, which is the honest caveat on the
+  comparison below.
 - **The bandwidth story moved too.** Both cards ride Gen5 x8 (~31.5 GB/s each):
   card 0 effectively unchanged (was Gen4 x16), card 1 up ~16x (was Gen4 x1,
   ~1.97 GB/s). The M.2→PCIe riser plan is dead — `PCIEX16_2` holds card #2 now
@@ -211,28 +213,47 @@ landed, at x8/x8, clears the arithmetic with room to spare.
 - Independent corroboration: StillDeadcode/vllm-radiance Codeberg issue #22 —
   different host, same image, kernel 7.0.0-30, same `enqueue.cc` failure.
 
-## Test plan (when the hardware changes)
+## Measured verdict (2026-09-20)
 
-**Precondition met 2026-09-20:** the line below reads `ReqEn+` on both cards, so
-this is now a standing plan rather than a hypothesis.
+Both cooperation modes were run on this host on 2026-09-20 — the same salted
+10-prompt battery (`scripts/bench-conversation.py`), the service stopped and
+masked between arms. Decode/prefill are geometric means over the ten rows;
+burst-4 is four concurrent copies of one prompt, reported as aggregate tokens/s.
 
-The verdict to look for is one line of `lspci`, before touching any model:
+| Shape | Cross-device path | Decode | TTFT | Prefill | burst-4 |
+|---|---|---|---|---|---|
+| **radiance TP=2 + DFlash2 — deployed** | R4D one-shot allreduce | **95.8 t/s** | **0.21 s** | **1493.6 t/s** | **259.4 t/s (65.6/stream)** |
+| llama.cpp layer split 32:33 + DFlash2 — predecessor | none, boundary activation copy | 45.7 t/s | 0.72 s | ~530 t/s | 71.7 t/s |
+| llama.cpp `--split-mode tensor` 1:1 | RCCL allreduce | 30.5 t/s | 0.71 s | n/r | 54.3 t/s |
 
-```bash
-sudo lspci -vv -s <card2> | grep AtomicOpsCtl     # want: ReqEn+
-```
+Row 3 carries a caveat that no flag can remove: llama.cpp's tensor mode could not
+load the DFlash2 drafter (it aborts at `ggml-backend.cpp:941` — a draft weight
+pre-allocated on the non-primary device has no backend once `n_copies == 1`), so
+it decoded **without** speculation while the other two rows decoded with it. The
+comparison that decides anything is row 1 against row 2, and some of that gap is
+the 4-bit MXFP4 checkpoint rather than the parallelism.
 
-If that reads `ReqEn+`, the rest is mechanical: `--tensor-parallel-size 2` with
-`HIP_VISIBLE_DEVICES=0,1`, then the same benchmark set
-(`scripts/bench-turn-latency.py`, and the endpoint's own `timings` block) so the
-two architectures are compared on identical workloads. If it still reads
-`ReqEn-`, the card is still on a non-atomic path — check `lspci -tv` for the
-bridge chain rather than assuming the move worked.
+Two numbers elsewhere on this page are therefore not interchangeable:
+
+- The 272 t/s recorded 2026-09-16 is **prefill**, on a **butterfly** allreduce,
+  with RCCL off, on the old board. It was never a TP decode number.
+- 26.3 t/s from the same table is that butterfly tensor's decode. Row 3's 30.5 is
+  an RCCL tensor decode a day later: same shape family, different collective path.
+
+RCCL needs PCIe atomics on *every* participating GPU — the gate the old board
+failed. vLLM's R4D one-shot allreduce dispatches no collective at all, which is
+why TP=2 runs today rather than merely "being unblocked": the image ships TheRock
+RCCL, and TheRock carries no gfx1201 device code (`invalid device function`), so
+radiance could not have used RCCL on this host even with atomics routed. Ubuntu's
+RCCL 2.27 does carry gfx1201 kernels, and that is what row 3 linked against.
 
 **Rule for any future multi-GPU collective experiment: stop and mask the
-production service first** (`sudo systemctl stop frank-medium && sudo systemctl
-mask frank-medium`, per the benchmarking policy), or run it on a host that is
-not serving users. On 2026-09-16 a bare RCCL P2P probe was run against the live
+production service first**, per the benchmarking policy, or run it on a host that
+is not serving users. Since 2026-09-20 `frank-medium` is a rootless quadlet, so
+that is `systemctl --user stop frank-medium && systemctl --user mask frank-medium`
+on Dankserver.
+
+This is not theoretical: on 2026-09-16 a bare RCCL P2P probe was run against the live
 service; the host went hard down ~20 minutes later (no ARP, no ICMP, required a
 power cycle). The mechanism is not proven, but a hung AQL dispatch wedging the
 GPU driver is a plausible cause and this host has precedent for going