feat(monitoring,harness): follow frank-medium onto vllm:* metrics and fix red-by-construction targets

The radiance TP=2 cutover changed the metric namespace frank-medium exports (llamacpp:* -> vllm:*) and left two scrape targets addressing host processes by an address a container cannot reach. - shared/prometheus/rules/llm-fleet.yml: two namespaces side by side. Added LlmRequestsWaiting, LlmKvCacheSaturated (vllm-only gauges llama.cpp never had), the job:llm_vllm_prefix_cache_reuse:rate5m and job:llm_spec_accept_length:rate5m recording rules, and re-cut LlmPrefixCacheCollapse / LlmSpecDecodeOff to OR both families with per-family traffic floors so an idle endpoint cannot fire them. Every expression validated against live Prometheus. - shared/prometheus/prometheus.yml: frank-small -> host.containers.internal:7771 and frank-large scraped at host.containers.internal:7770. Both are HOST processes on Dankclaw and netavark DNATs published container ports only, so the LAN-IP targets were red by construction; frank-large's CIRU llama.cpp build does export the llamacpp:* family, contra the old comment. - tests/observability_quadlet.bats: the two seams now encode the new contract (per-endpoint reachability by process type; every named series must be on the measured list for its engine; engine-independent rules must name both). - homepage docs.html/services.json: frank-medium sees pictures, frank-large is CIRU Orca with a single seat (was halogen w4b, two seats, 200k). - docs: radiance runbook is ACTIVE (deployed flags, three-arm battery, verified capabilities, rollback, replica-era sections labelled); llama.cpp runbook carries the SUPERSEDED banner; Dual-R9700-Cooperation states the measured verdict; Observability records the namespace split and the DNAT lesson; AGENTS.md topology/frank-medium/Model Config/Benchmarking Policy/runbook index flipped; both indexes updated; work-log entry added. Also repaired agent-pass damage: prose prepended above frontmatter openers, duplicated runbook bodies, a literal tool-call token pasted into a runbook, and an index destroyed by diff artifacts (restored from HEAD, flips re-applied). Vault link check: 128 pages, 0 unresolved. ./tests/run.sh: 325 pass.
2026-09-20 19:35:56 -07:00
1 parent c2116658f8
commit 08f1c7ad09
@@ -24,9 +24,9 @@ merge absorbed the former `vllmScripts` repo, history preserved).
 | Target | Service | Runtime | Model | Port |
 |--------|---------|---------|-------|------|
-|Dankserver|`frank-medium`|mainline llama.cpp master fb27a525 (root systemd unit, gfx1201 HIP build)|orcarouter Qwen3.8-27B-Uncensored Q8_0 with the z-lab DFlash2 Q4_K_M drafter (the model's own MTP head stays in the file, one SPEC_TYPE change away), 2x Radeon AI PRO R9700 in ONE layer-split process (32:33 balanced; both cards on CPU root ports at Gen5 x8 since the 2026-09-20 ProArt B850-Creator swap — atomic routing verified on both chains), 327680 ctx over 2 slots of 163840, batch 6144x512, f16 KV + 16 GiB host-RAM prompt cache, vision (BF16 mmproj)|7770|
+| Dankserver | `frank-medium` | vllm-radiance 0.9.3 (rootless Quadlet, image `localhost/radiance-vllm:0.9.3`, gfx1201) | Qwen3.8-27B-Uncensored MXFP4-awq with the z-lab DFlash2-FP8 drafter (depth 7, live mean accept length ~4.1), 2x Radeon AI PRO R9700 in ONE tensor-parallel process (TP=2 over the R4D one-shot allreduce — no RCCL collective; both cards on CPU root ports at Gen5 x8 since the 2026-09-20 ProArt B850-Creator swap), 262144 ctx over up to 8 conversations (auto-sized 943,581-token KV pool = 3.59x the window), fp8 KV, prefix caching, vision | 7770 |
 | Dankclaw | `frank-large` | CIRU v4.4.1 llama.cpp (QSA window hotfix, root systemd unit, gfx1151 HIP build) | jcbtc Qwen3.8-Flash-CIRU-STRIX-Orca on the gfx1151 iGPU (Strix Halo LPDDR), 262144 ctx single stream (1 slot), MTP depth 4, PLE 4096 MiB, text only | 7770 |
-| Dankclaw | `frank-small` | mainline llama.cpp master 37b53fd (root systemd unit) | HauhauCS Gemma4-12B-QAT-Uncensored-Balanced Q4_K_M (BF16 mmproj + MTP assistant sidecar, `draft-mtp,ngram-mod`) on the RTX 3090, 1048576 pool / 4 slots of 262,144, q8_0 KV, the fleet's vision endpoint | 7771 |
+| Dankclaw | `frank-small` | mainline llama.cpp master 37b53fd (root systemd unit) | HauhauCS Gemma4-12B-QAT-Uncensored-Balanced Q4_K_M (BF16 mmproj + MTP assistant sidecar, `draft-mtp,ngram-mod`) on the RTX 3090, 1048576 pool / 4 slots of 262,144, q8_0 KV, the fleet's fastest vision endpoint | 7771 |
 Retired endpoints (`frank-npu`, and every prior runtime) are in
 the [runbook index](#runbook-index). The full swap history lives in
@@ -68,7 +68,9 @@ All edits flow through git. Never edit files on a remote machine directly.
 The planner maps changed paths to services; the remote runner restarts only
 the selected service and records its revision outside the checkout.
 Documentation-only changes are a no-op. LLM services are selected only by
-their own files — all three by their unit/env/serve trio in `scripts/` — so
+their own files — `frank-large`/`frank-small` by their unit/env/serve trio in
 `scripts/`, `frank-medium` by its Quadlet unit, entrypoint and healthcheck under
 `dankserver/frank-medium/` — so
 unrelated
 container changes cannot interrupt inference. Set `FRANK_SUDO_PW` for headless
 runs on a host without passwordless sudo; the value never enters the repo.
@@ -230,8 +232,9 @@ merely wedged kernel usually still answers ARP.
     `sudo systemctl stop frank-large && sudo systemctl mask frank-large`
   - Dankclaw `frank-small` (root unit, llama.cpp on the 3090):
     `sudo systemctl stop frank-small && sudo systemctl mask frank-small`
-   - Dankserver `frank-medium` (system unit):
+   - Dankserver `frank-medium` (**rootless Quadlet since 2026-09-20** — the
-     `sudo systemctl stop frank-medium && sudo systemctl mask frank-medium`
+     system unit is retired, so `sudo systemctl` sees nothing):
     `systemctl --user stop frank-medium && systemctl --user mask frank-medium`
 2. Bench.
 3. Unmask and start; verify `curl http://192.168.0.<host>:7770/health`.
@@ -259,9 +262,14 @@ When renaming a service, retiring an endpoint, or moving a model: update the
 provider block, model `id`, `contextWindow`, `maxTokens`, `input` in
 `models.yml`, then fix every `modelRoles` entry that named the old endpoint —
 a role pointing at a provider the file does not define fails at call time. As
-of 2026-09-16 the image path is `frank-small` (`input: [text, image]`), and
+of 2026-09-20 both `frank-small` and `frank-medium` accept images
-`frank-large` is text-only because its CIRU deployment loads no vision
+(`input: [text, image]`) — frank-medium's checkpoint serves vision through the
-projector (the F16 mmproj is on disk, one env flip away). There is **no audio**
+Gemma4 merge path with no separate mmproj, verified on held-out synthetic
 images; `frank-large` remains text-only because its CIRU deployment loads no
 vision projector (the F16 mmproj is on disk, one env flip away). Note also that
 the reasoning channel differs by runtime: llama.cpp endpoints emit
 `reasoning_content`, vLLM emits `reasoning`.
 There is **no audio**
 input anywhere in the fleet: that capability left with the FLM NPU endpoint on
 2026-09-11.
@@ -326,63 +334,49 @@ ubatch 2048 OOMs the draft's pp buffer, and why draft depth 4 + poll
 Runbook: `serverAppConfigs-docs/LLM/frank-small-gemma4-12b.md`.
 Rollback: [[frank-small-qwen38-gsq-rco]].
-### frank-medium — llama.cpp layer split, DFlash2 drafter (Dankserver, ACTIVE)
+### frank-medium — radiance TP=2, DFlash2-FP8 drafter (Dankserver, ACTIVE)
-Mainline llama.cpp at **`fb27a525`** (a gfx1201 HIP build, `Root=`-less root
+vllm-radiance **0.9.3** (`localhost/radiance-vllm:0.9.3`, rootless **Quadlet**
-systemd unit `scripts/frank-medium.service`) serving
+`dankserver/frank-medium/quadlet/frank-medium.container`) serving
-`orcarouter/Qwen3.8-27B-Uncensored-GGUF` at **Q8_0** (27.0 GiB) with the
+`just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq` (MXFP4) with the
-**z-lab DFlash2 `Q4_K_M` drafter** (1.06 GiB). The target file also carries the
+**z-lab DFlash2 FP8 drafter** (`num_speculative_tokens: 7`, live mean accept
-model's own MTP head (`blk.<n>.nextn.*`, `nextn_predict_layers=1`), so
+~4.1) in **one tensor-parallel process across both R9700s**
-`draft-mtp` is one `SPEC_TYPE` change away — the drafter file stays on disk. In
+(`--tensor-parallel-size 2`). 262,144 ctx, `--max-num-seqs 8`, chunk 8192, fp8
-**one process across both R9700s**, `--split-mode layer` at a **32:33 tensor
+KV, prefix caching, GPU fill 0.98, **no KV-memory pin** — the pool auto-sized
-split** — balanced (operator call 2026-09-19, the most even split 65 layers
+to 943,581 tokens (3.59× the full window). Tools via `qwen3_coder`, reasoning
-allow; ROCm1, the last-layers card that also carries the drafter and output
+via `qwen3` (the field is `reasoning`, not llama.cpp's `reasoning_content`),
-layers, holds the extra one).
+vision through the Gemma4 merge path with `MM_MERGE_ENGINE=AXROW_SEPARATE`.
 The predecessor's 35:30 skew (2026-09-17) leaned work onto the wider card
 because ROCm1 (then `0b:00.0`) rode the B650 chipset at Gen4 x1 (~1.97 GB/s)
 against ROCm0's CPU-root-port Gen4 x16 (~31.5 GB/s); the balanced point traded
 that lean for symmetry and was accepted. The 2026-09-20 board swap (ASUS ProArt
 B850-Creator WiFi NEO) removed the other axis: both cards are now CPU-root-port
 Gen5 x8 (~31.5 GB/s each), at `03:00.0` and `07:00.0`. 327680 ctx
 over **2 slots** of 163840 — inside the checkpoint's trained 262144 window, so
 no rope extrapolation (unlike the predecessor's 393216) — batch 6144x512, f16
 KV, a 16384 MiB host-RAM prompt cache, vision via the BF16 mmproj. DSpark was
 tried in the drafter's place on 2026-09-16 and reverted: it declares no sliding
 window, so its draft KV spanned the whole context (5.0 GiB) and cost ROCm1 four
 layers of the split.
-**Layer split is why both cards serve ONE model today.** It was chosen because
+**TP=2 runs because the board changed.** The predecessor board (TUF GAMING
-tensor parallelism was then impossible (see the PCIe lesson below) and so was
+B650-E) put card #2 behind chipset bridges advertising `AtomicOpsCap: Routing-`
-vLLM's pipeline parallelism — its stage handoff is an RCCL `send`/`recv` of the
+and no collective could be dispatched there. The ProArt B850-Creator WiFi NEO
-hidden state, the exact primitive that could not be dispatched on the old
+routes both cards to CPU GPP root ports (`00:01.1 → 03:00.0`,
-board's card #2. llama.cpp's layer split needs
+`00:01.3 → 07:00.0`, Gen5 x8, every hop `Routing+`, both endpoints `ReqEn+`).
-no collective at all: layers are distributed in order and the boundary
+The collective in use is vLLM's **R4D one-shot allreduce**, which never touches
-activation is a per-backend copy, once per forward pass. Measured on this host
+RCCL (TheRock RCCL has no gfx1201 device code; the Ubuntu RCCL 2.27 build does,
-(the predecessor runtime): 1991 t/s prefill layer-split vs 272 tensor — but that
+and that is what `scripts/build-llama-rdna4.sh GGML_HIP_RCCL=ON` links).
 tensor number is the cost of a butterfly-fallback allreduce with `GGML_HIP_RCCL`
 OFF, not what an RCCL-backed tensor split would do. Since the 2026-09-20 board
 swap cleared the atomics gate, that comparison is stale and the probe is the
 open work; see [[Dual-R9700-Cooperation]].
-This is the box's **only** LLM runtime: the vllm-radiance replica pair and its
+**Measured, not assumed** (`scripts/bench-conversation.py`, 10 real prompts,
-routing proxy (`frank-medium-a/-b/-proxy`) were retired the same day they went
+max_tokens 512, greedy, salted per arm, service stopped between arms):
-live, with their units moved to `~/.AGENTTRASH/20260916-vllm-radiance-retire/`.
+radiance TP=2 95.8 t/s decode geomean / 0.21 s TTFT / 259.4 t/s burst-4, vs the
 llama.cpp layer split's 45.7 / 0.72 / 71.7 and llama.cpp `--split-mode tensor`'s
 30.5 / 0.71 / 54.3. Honest caveat: the arms also differ in quantization (MXFP4
 vs Q8_0), so part of the margin is the quant, not the parallelism. Re-measured
 against production after deploy: 92.4 t/s, 0.22 s, 248.9.
-Files: `scripts/build-llama-rdna4.sh` (builds the runtime, gfx1201 only, private
+Files: `dankserver/frank-medium/quadlet/frank-medium.container`,
-ROCm 7.14 toolchain bundled beside the binary), `scripts/download-frank-medium-gguf.sh`
+`dankserver/frank-medium/radiance-entrypoint.sh`,
-(resumable, byte-verified payload), `scripts/frank-medium.env`,
+`dankserver/frank-medium/frank-medium-healthcheck.py`. The llama.cpp layer-split
-`scripts/serve-frank-medium.sh`, `scripts/frank-medium.service`. The deploy
+trio (`scripts/frank-medium.service`, `.env`, `serve-frank-medium.sh`) is
-handler refuses to start a unit whose runtime or payload is missing, retires
+archived in `scripts/.AGENTTRASH/20260920-llama-layersplit/` and is the
-the vLLM quadlets, waits for 7770 to be released (the retired proxy held it) and
+rollback target; `deploy-remote.sh`'s `retire_llama_medium_unit` moved the
-gates on `/health`.
+installed unit symlink there rather than `systemctl disable`, which would delete
 it. The deploy handler gates on the image tag, both model configs, the patch
 clone + jinja template, and `libr4d`, then waits for :7770 and gates on
 `/health`.
-The ref is pinned by two features and must not float: **DFlash2**
+Runbook: `serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md`.
-(ggml-org/llama.cpp#27342) and **qwen35 MTP**. The ROCmFPX fork the endpoint
+Rollback: [[frank-medium-llamacpp-dflash2]] (SUPERSEDED 2026-09-20).
 used earlier that day has neither.
 Runbook: `serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md`.
 Retired runtimes: [[frank-medium-radiance-mxfp4]] (vLLM replicas, same day),
 [[frank-medium-rocmfpx]] (llama.cpp fork, earlier that day).
 ## Key Lessons Learned
 Durable operational lessons from the fleet's history. Full detail in the
@@ -412,9 +406,14 @@ linked runbooks.
  every bridge now `Routing+` and **both endpoints `ReqEn+`** — the exact
  precondition the old test plan named. Bandwidth was never the constraint (a
  27B TP=2 allreduce at batch 4 is ≈10 MB per token-step, ≈1.4 ms at Gen4 x4
-  and under 0.4 ms at Gen5 x8). **What is NOT yet re-measured:** no collective
+  and under 0.4 ms at Gen5 x8).
-  has been dispatched since the swap, so TP=2 is unblocked, not proven. Full
+  **Proven on the same day.** vLLM's R4D one-shot allreduce runs TP=2 here and
-  evidence matrix and the standing probe plan: [[Dual-R9700-Cooperation]].
+  frank-medium serves on it; llama.cpp's RCCL path was built too
  (`GGML_HIP_RCCL=ON`) and its tensor mode measured — real collective, slower
  shape. Note the collective choice matters: TheRock's RCCL builds lack gfx1201
  device code (`invalid device function`), which is why R4D is the path vLLM
  uses at all. Evidence matrix and the measured three-arm verdict:
  [[Dual-R9700-Cooperation]].
 - **When a GPU collective dies, read the runtime, not the framework.** The
  message that identified the cause came from `AMD_LOG_LEVEL=3`
  (`Pcie atomics not enabled, hostcall not supported` → `AQL dispatch
@@ -428,7 +427,11 @@ linked runbooks.
  collective: layers are distributed in order and only the boundary activation
  crosses PCIe, once per forward pass. Measured here: 1991 t/s prefill /
  18.8 tok/s decode layer-split, against 272 / 26.3 tensor (butterfly
-  fallback). That is what `frank-medium` runs — see
+  fallback).
  This was what `frank-medium` ran until 2026-09-20, when the board swap made
  collectives legal and radiance TP=2 beat it on every axis (95.8 vs 45.7 t/s
  decode, 0.21 vs 0.72 s TTFT, 259 vs 72 t/s burst-4) — see
  [[frank-medium-radiance-mxfp4]]; the llama shape is the rollback target
  [[frank-medium-llamacpp-dflash2]].
 - **Layer split is also the cross-vendor answer — and `-ot` is what cannot be
  combined with it.** On Dankclaw (2026-09-16) a CUDA 3090 + Vulkan iGPU layer
@@ -467,9 +470,10 @@ linked runbooks.
 **ROCm / GPU**
- **Layer split is the deployed shape on both multi-device llama.cpp services,
+- **Layer split was the deployed shape on both multi-device llama.cpp services,
-  for two different reasons.** On Dankserver it is the *only* way both R9700s
+  for two different reasons.** On Dankserver it was the *only* way both R9700s
-  serve one model (no collective can be dispatched on card #2). On a
+  served one model (no collective could be dispatched on card #2) until the
  2026-09-20 board swap; frank-medium now runs radiance TP=2 instead. On a
  RCCL-less fork build it is also the fast one: tensor parallel falls back to
  a butterfly allreduce and collapses prefill ~7x (measured 271.8 vs 1991 t/s).
  Pin the spec-decode drafter to one device — an unpinned draft under tensor
@@ -717,7 +721,7 @@ Active:
 |---------|---------|
 | frank-large (CIRU v4.4.1 Orca, Dankclaw) | `serverAppConfigs-docs/LLM/frank-large-ciru-orca.md` |
 | frank-small (llama.cpp Gemma4-12B QAT, Dankclaw) | `serverAppConfigs-docs/LLM/frank-small-gemma4-12b.md` |
-| frank-medium (llama.cpp + MTP, Dankserver) | `serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md` |
+| frank-medium (radiance TP=2 + DFlash2, Dankserver) | `serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md` |
 | Dankclaw kernel posture (`amd_iommu=off`) | `serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md` |
 | Deployment flow + incidents | `serverAppConfigs-docs/LLM/deployment.md` |
@@ -739,6 +743,7 @@ Retired (rollback targets and history):
 | FreeToken (official + Huihui FP8, `frank-small`) | Dankserver | 2026-09-02 → 09-15 | `freetoken-frank-small.md` |
 | FastMTP patched llama.cpp | Dankserver | retired 2026-08-18 | `qwen3.8-spec-matrix.md` |
 | ik_llama.cpp Ornith CRACK | Dankserver | retired 2026-09-02 | `ikllama-ornith-dankserver.md` |
 | llama.cpp layer split + DFlash2 (`frank-medium`) | Dankserver | 2026-09-16 → 09-20 | `frank-medium-llamacpp-dflash2.md` |
 Retired runbooks are under `serverAppConfigs-docs/LLM/`. Retired scripts live
 in `old/` and `.AGENTTRASH/`.
@@ -41,6 +41,7 @@ Recorded 2026-09-11, on the day the stack moved to Quadlet units. Everything in
 - **Grafana has a datasource and one dashboard as of 2026-09-16; before that it had neither.** `shared/grafana/provisioning/` is installed to `/opt/grafana/provisioning` by the deployer and read by the container: the Prometheus datasource (`http://prometheus:9090`, fixed uid `prometheus`) and the **LLM fleet — capacity and reuse** dashboard (speculative acceptance, queue depth, prefix-cache reuse, decode step rate, throughput) covering the llama.cpp endpoints. **Re-cut on 2026-09-16**: the panels were keyed to `vllm:` series, which stopped existing when frank-medium moved to llama.cpp, and two of them (KV-pool fill, request-latency p95) have no llama.cpp equivalent at all — those two are retired rather than restated, and the acceptance panel replaces the KV one. A renamed query against a series that does not exist is a panel that draws nothing forever. Verified through the API rather than by eye — the datasource answers `llamacpp:` counters for `frank-medium`, and `GET /api/search?type=dash-db` returns uid `llm-fleet`. One trap worth knowing: a provisioned dashboard does **not** appear in the legacy `dashboard` table on this Grafana version (it lives in unified storage …
 - **Grafana's dashboard now also surfaces firing alerts.** The **Alerts firing (ALERTS)** panel graphs `ALERTS{alertstate="firing"}` for the LLM rules, because this stack has no other place where an alert becomes visible without opening Prometheus.
 - **Prometheus now evaluates rules, and they are delivered (2026-09-17).** `shared/prometheus/rules/llm-fleet.yml` ships with the config (the deployer's `copy_prometheus_config` installs both — a rule file nothing loads evaluates nothing, and Prometheus reports no error for that) and holds five rules for the LLM fleet: request queueing, prefix-cache collapse, speculative decoding off, endpoint down, plus the `job:llm_prefix_cache_reuse:rate5m` recording rule the alert and the dashboard share. **Two rules from the vLLM era were retired, not renamed**, because the series they watched no longer exist: KV-pool saturation (llama.cpp exports no KV-fill gauge; its consequence *is* queueing, which `LlmRequestsQueueing` already covers) and p95 request latency (no latency histogram). `LlmSpecDecodeOff` takes the second slot — the failure a client cannot see, since a lost drafter just halves the speed. Verified with `promtool check rules` and `/api/v1/rules` (`health=ok` for all five). **Delivery went live 2026-09-17**: the `alerting:` block in `prometheus.yml` points at `alertmanager:9093` (container name on `dankclaw_default` — the published port is loopback-only and would be wrong from inside Prometheus). Alertmanager groups by `alertname` (30 s group wait, 5 m interval, 4 h repeat) and webhooks the host-side **Signal relay** at `host.containers.internal:18093/alertmanager` — the relay binds the host's LAN IP (192.168.0.143:18093, the one address netavark delivers that dial to), with an iptables set admitting loopback, the host and the podman bridge subnet only. The relay (`scripts/signal-relay.py`, a systemd *user* unit, not a quadlet — it runs python3 from the checkout and can be enabled) dedupes per group for 15 minutes and sends through **signal-cli account 318311** (`/usr/local/bin/signal-cli`, state in `~/.local/share/signal-cli`), Frank's existing account. The Grafana dashboard's ALERTS panel and Prometheus' `/alerts` remain the second and third surfaces.
 - **2026-09-20: the rule file now spans two metric namespaces.** frank-medium cut over to radiance/vLLM, so `llm-fleet.yml` carries `vllm:*` rules beside the `llamacpp:*` ones — seven alerts and two recording rules in total. KV-pool saturation and preemption pressure came back (they were retired 2026-09-16 for want of a llama.cpp gauge; vLLM exports both), and the two engine-independent rules — `LlmSpecDecodeOff`, `LlmPrefixCacheCollapse` — OR the two families and are floored on each family's own traffic counter so an idle endpoint cannot fire them. **The provisioned dashboard is the known gap:** its panels are still keyed to `llamacpp:*` series, so frank-medium's row went blank at the cutover while frank-small's and frank-large's still draw. Re-cut pending — the KV-fill and latency panels that were retired for llama.cpp have vLLM equivalents again (`vllm:kv_cache_usage_perc`, `vllm:inter_token_latency_seconds`).
 - **Loki is configured but unread.** Storage is filesystem with 30-day retention, `auth_enabled: false`, and the push path works; Grafana can now reach it in Explore, but no dashboard or alert consumes it.
 - **node-exporter runs its default collector set** — no collector tuning, no textfile collector for custom metrics.
 - **Alloy is the one configured piece.** It ships the host journal and podman container logs to Loki with `job` labels, and that pipeline is live.
@@ -88,26 +89,31 @@ The rule is the same as for every other service: the only off-host path is a gua
 | `grafana`, `loki` | `grafana:3000`, `loki:3100` | container names on `dankclaw_default` |
 | `comfyui` | `comfyui:8188` | UI probe, no native exporter |
 | `services` | `192.168.0.143:8081`, `:3005` | SearXNG, OpenWebUI up/down probes |
-| `frank-medium` | `192.168.0.235:7770` | llama.cpp on Dankserver — `llamacpp:` counters, including the spec-decode pair |
+| `frank-medium` | `192.168.0.235:7770` | radiance/vLLM on Dankserver — `vllm:` counters: KV fill, queue depth, preemptions, prefix-cache reuse, and the `spec_decode_*` trio |
-| `frank-small` | `192.168.0.143:7771` | llama.cpp on Dankclaw's RTX 3090 — `llamacpp:` counters, including the spec-decode pair |
+| `frank-small` | `host.containers.internal:7771` | llama.cpp on Dankclaw's RTX 3090 — `llamacpp:` counters, including the spec-decode pair |
 | `frank-large` | `host.containers.internal:7770` | CIRU llama.cpp on Dankclaw's iGPU — 19 `llamacpp:` families |
 The LLM endpoints were added on 2026-09-16, after a night spent diagnosing
 frank-medium by hand, and re-cut the same day when Dankclaw split into two
-endpoints. The vLLM runtime behind that diagnosis is retired; frank-medium
+endpoints. frank-medium has since changed runtime twice (vLLM → mainline
-now runs mainline llama.cpp, and the endpoint exports llama.cpp's own
+llama.cpp → radiance/vLLM on 2026-09-20), and the metric family it exports
-counters — prompt and decode throughput, KV fill, and the `spec_decode_*`
+changed with it: `llamacpp:*` while it ran llama.cpp, `vllm:*` now.
 pair that proves the drafter is running (draft-mtp on both endpoints as of 2026-09-17).
 this hardware is in [[frank-medium-llamacpp-dflash2]]; the same counter
 family on frank-small is read against [[frank-small-qwen38-gsq-rco]].
-**`frank-large` is deliberately NOT scraped.** The halogen engine publishes
+**Both Dankclaw targets are addressed as `host.containers.internal`
-`/health`, `/cache` and `/v1/*` and no Prometheus text endpoint at all, so a
+(169.254.1.2), never the LAN IP.** frank-large and frank-small are *host
-target there would be red by construction and would fire `LlmEndpointDown`
+processes* on Dankclaw, and netavark DNATs only *published* container ports —
-forever — the same class of defect as the page-only probes below. Its liveness
+from the Prometheus container `192.168.0.143:7770/:7771` is refused even
-is on the homepage (a loopback probe) and in the engine's own boot ledger.
+though it answers from every other vantage on the LAN. Both targets were red
 by construction until a live `/targets` audit caught it on 2026-09-20.
 `frank-medium` keeps its LAN address: it is a rootless Quadlet on another host
 that publishes 7770, so its published-port DNAT does apply.
-Both scraped endpoints are reached on their LAN addresses, never through the
+**`frank-large` IS scraped as of 2026-09-20.** The earlier claim that it had
-guarded vhosts: Prometheus cannot solve an Anubis challenge.
+"no Prometheus text endpoint at all" was true of the halogen engine it ran
 then; the CIRU llama.cpp build that replaced it exports the full
 `llamacpp:*` family, which is what makes the target addressable at all. Its
 liveness is additionally on the homepage (a loopback probe) and in the
 engine's own boot ledger.
 For services without a native exporter, use the Prometheus **blackbox exporter** in probe mode to assert up/down and HTTP latency. The homepage status timer probes the same endpoints from the host.
@@ -14,12 +14,11 @@ timestamp: 2026-09-16
 # Dual R9700 Cooperation
-Dankserver's two R9700s serve `frank-medium` as **one llama.cpp process with a
+Dankserver's two R9700s serve `frank-medium` as **one vLLM-radiance process at
-layer split** ([[frank-medium-llamacpp-dflash2]]). This page records why true
+tensor_parallel_size 2** ([[frank-medium-radiance-mxfp4]]). This page records why
-tensor parallelism was impossible on the previous board (TUF GAMING B650-E
+true tensor parallelism was impossible on the previous board (TUF GAMING B650-E
-WIFI), and what the 2026-09-20 board swap changed. See
+WIFI), and what the 2026-09-20 board swap changed. The llama.cpp layer split that
-[[frank-medium-radiance-mxfp4]] for the replica-pair deployment that ran under
+served the endpoint until that swap is [[frank-medium-llamacpp-dflash2]].
 the old constraint.
 ## The short answer
@@ -35,18 +34,21 @@ bifurcates the CPU's GPP ports **x8/x8**: the board-swap path this page
 recommended, at B850 rather than X870E money. Both cards now hang off CPU root
 ports (`00:01.1`, `00:01.3`), every bridge on both chains reads
 `AtomicOpsCap: Routing+`, and **both endpoints read `AtomicOpsCtl: ReqEn+`** —
-the exact precondition this page's own [test plan](#test-plan) names.
+the exact precondition this page's own test plan named, and it is now
 [measured](#measured-verdict-2026-09-20) rather than assumed.
-What that buys, and what it does not:
+What that bought:
- **Unblocked in principle:** vLLM TP=2, vLLM PP=2, and llama.cpp
+- **Both cooperation modes now work, and they were both run.** vLLM TP=2 booted
-  `--split-mode row`. All three died on the same hostcall gate.
+  and served (its R4D allreduce needs no RCCL collective at all), and llama.cpp
- **Not yet measured.** Nothing has dispatched a collective on this host since
+  `--split-mode tensor` booted and served against an RCCL-backed build — so a
-  the swap. The 2-rank probe is the remaining gate, and policy is stop-and-mask
+  collective HAS been dispatched on this host since the swap.
-  production first.
+- **One of them is production.** TP=2 wins on every axis measured, by 2.1x on
- **Not needed for capability.** The layer split already puts both cards on one
+  decode and 3.6x at 4-way concurrency, so it replaced the layer split on the day
-  model. This is a performance question: an RCCL-backed tensor split, and the
+  of measurement. See [[frank-medium-radiance-mxfp4]] for the deployed shape.
-  single pooled KV that comes with it.
+- **The drafter, not the collective, is what llama.cpp lost.** Its tensor mode
  could not carry the DFlash2 drafter, which is the honest caveat on the
  comparison below.
 - **The bandwidth story moved too.** Both cards ride Gen5 x8 (~31.5 GB/s each):
  card 0 effectively unchanged (was Gen4 x16), card 1 up ~16x (was Gen4 x1,
  ~1.97 GB/s). The M.2→PCIe riser plan is dead — `PCIEX16_2` holds card #2 now
@@ -211,28 +213,47 @@ landed, at x8/x8, clears the arithmetic with room to spare.
 - Independent corroboration: StillDeadcode/vllm-radiance Codeberg issue #22 —
  different host, same image, kernel 7.0.0-30, same `enqueue.cc` failure.
-## Test plan (when the hardware changes)
+## Measured verdict (2026-09-20)
-**Precondition met 2026-09-20:** the line below reads `ReqEn+` on both cards, so
+Both cooperation modes were run on this host on 2026-09-20 — the same salted
-this is now a standing plan rather than a hypothesis.
+10-prompt battery (`scripts/bench-conversation.py`), the service stopped and
 masked between arms. Decode/prefill are geometric means over the ten rows;
 burst-4 is four concurrent copies of one prompt, reported as aggregate tokens/s.
-The verdict to look for is one line of `lspci`, before touching any model:
+| Shape | Cross-device path | Decode | TTFT | Prefill | burst-4 |
 |---|---|---|---|---|---|
 | **radiance TP=2 + DFlash2 — deployed** | R4D one-shot allreduce | **95.8 t/s** | **0.21 s** | **1493.6 t/s** | **259.4 t/s (65.6/stream)** |
 | llama.cpp layer split 32:33 + DFlash2 — predecessor | none, boundary activation copy | 45.7 t/s | 0.72 s | ~530 t/s | 71.7 t/s |
 | llama.cpp `--split-mode tensor` 1:1 | RCCL allreduce | 30.5 t/s | 0.71 s | n/r | 54.3 t/s |
-```bash
+Row 3 carries a caveat that no flag can remove: llama.cpp's tensor mode could not
-sudo lspci -vv -s <card2> | grep AtomicOpsCtl     # want: ReqEn+
+load the DFlash2 drafter (it aborts at `ggml-backend.cpp:941` — a draft weight
-```
+pre-allocated on the non-primary device has no backend once `n_copies == 1`), so
 it decoded **without** speculation while the other two rows decoded with it. The
 comparison that decides anything is row 1 against row 2, and some of that gap is
 the 4-bit MXFP4 checkpoint rather than the parallelism.
-If that reads `ReqEn+`, the rest is mechanical: `--tensor-parallel-size 2` with
+Two numbers elsewhere on this page are therefore not interchangeable:
-`HIP_VISIBLE_DEVICES=0,1`, then the same benchmark set
+
-(`scripts/bench-turn-latency.py`, and the endpoint's own `timings` block) so the
+- The 272 t/s recorded 2026-09-16 is **prefill**, on a **butterfly** allreduce,
-two architectures are compared on identical workloads. If it still reads
+  with RCCL off, on the old board. It was never a TP decode number.
-`ReqEn-`, the card is still on a non-atomic path — check `lspci -tv` for the
+- 26.3 t/s from the same table is that butterfly tensor's decode. Row 3's 30.5 is
-bridge chain rather than assuming the move worked.
+  an RCCL tensor decode a day later: same shape family, different collective path.
 RCCL needs PCIe atomics on *every* participating GPU — the gate the old board
 failed. vLLM's R4D one-shot allreduce dispatches no collective at all, which is
 why TP=2 runs today rather than merely "being unblocked": the image ships TheRock
 RCCL, and TheRock carries no gfx1201 device code (`invalid device function`), so
 radiance could not have used RCCL on this host even with atomics routed. Ubuntu's
 RCCL 2.27 does carry gfx1201 kernels, and that is what row 3 linked against.
 **Rule for any future multi-GPU collective experiment: stop and mask the
-production service first** (`sudo systemctl stop frank-medium && sudo systemctl
+production service first**, per the benchmarking policy, or run it on a host that
-mask frank-medium`, per the benchmarking policy), or run it on a host that is
+is not serving users. Since 2026-09-20 `frank-medium` is a rootless quadlet, so
-not serving users. On 2026-09-16 a bare RCCL P2P probe was run against the live
+that is `systemctl --user stop frank-medium && systemctl --user mask frank-medium`
 on Dankserver.
 This is not theoretical: on 2026-09-16 a bare RCCL P2P probe was run against the live
 service; the host went hard down ~20 minutes later (no ARP, no ICMP, required a
 power cycle). The mechanism is not proven, but a hung AQL dispatch wedging the
 GPU driver is a plausible cause and this host has precedent for going
@@ -1,7 +1,7 @@
 ---
 type: configuration
 title: frank-medium llama.cpp Runbook (DFlash2 drafter)
-description: The ACTIVE frank-medium deployment since 2026-09-16 (model swapped 2026-09-19) — mainline llama.cpp on Dankserver serving orcarouter/Qwen3.8-27B-Uncensored-GGUF at Q8_0 with the z-lab DFlash2 Q4_K_M drafter (the model's own MTP head stays in the file, one SPEC_TYPE change away), as ONE layer-split process across both Radeon AI PRO R9700 cards at a balanced 32:33 split, a 327680-position KV pool (163840 per conversation over two slots), f16 KV plus a 16 GiB host-RAM prompt cache, vision via the BF16 mmproj.
+description: The former ACTIVE frank-medium deployment since 2026-09-16 (model swapped 2026-09-19) — mainline llama.cpp on Dankserver serving orcarouter/Qwen3.8-27B-Uncensored-GGUF at Q8_0 with the z-lab DFlash2 Q4_K_M drafter (the model's own MTP head stays in the file, one SPEC_TYPE change away), as ONE layer-split process across both Radeon AI PRO R9700 cards at a balanced 32:33 split, a 327680-position KV pool (163840 per conversation over two slots), f16 KV plus a 16 GiB host-RAM prompt cache, vision via the BF16 mmproj.
 tags:
  - frank-medium
  - dankserver
@@ -12,12 +12,17 @@ tags:
  - gfx1201
  - r9700
  - layer-split
-timestamp: 2026-09-19
+timestamp: 2026-09-20
 ---
 > **SUPERSEDED 2026-09-20.** Replaced by [[frank-medium-radiance-mxfp4]]
 > (radiance TP=2, measured faster on every axis). The layer-split shape below is
 > the rollback target, and the llama.cpp measurements in this page stay valid for
 > it — but nothing here is the deployed shape any more.
 # frank-medium — llama.cpp (DFlash2 drafter)
-The ACTIVE frank-medium deployment since 2026-09-16 (model swapped to
+The frank-medium deployment from 2026-09-16 to 2026-09-20 (model swapped to
 orcarouter Q8_0 on 2026-09-19): **mainline llama.cpp**
 serving `orcarouter/Qwen3.8-27B-Uncensored-GGUF` at `Q8_0`
 with the **z-lab DFlash2** `Q4_K_M` block-diffusion drafter, as **one process across both
@@ -26,9 +31,11 @@ model (ajgazin Swift-Qwen3.8-27B UD-Q8_K_XL, 35:30 skew, 393216 pool) is the
 calibration history in the sizing tables below. DSpark was tried in the drafter's
 place on 2026-09-16 and reverted — see *The DSpark excursion*.
 It replaced the vllm-radiance MXFP4 replica pair + routing proxy
-([[frank-medium-radiance-mxfp4]], the record and the PCIe-atomics evidence) in
+([[frank-medium-radiance-mxfp4]] holds that record, and the PCIe-atomics
-the same day the pair went live. It is also the box's **only** LLM runtime: the
+evidence) the same day the pair went live — and was itself replaced by
-replicas, the proxy and their unit files are retired.
+radiance TP=2 on 2026-09-20, once the board swap made a collective legal.
 During its window it was the box's **only** LLM runtime: the replicas, the
 proxy and their unit files were retired.
 ## Why one process and layer split
@@ -1,7 +1,7 @@
 ---
 type: configuration
 title: frank-medium vllm-radiance MXFP4 Runbook
-description: vllm-radiance MXFP4 serving just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq as frank-medium on Dankserver (2x AMD Radeon AI PRO R9700, gfx1201), as two single-GPU replicas behind an adaptive-routing proxy — and the PCIe-atomics reason TP=2 cannot run on this host.
+description: ACTIVE frank-medium runbook since 2026-09-20 — vllm-radiance MXFP4 serving just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq as ONE tensor-parallel process (TP=2) on Dankserver's two AMD Radeon AI PRO R9700 (gfx1201), with a DFlash2-FP8 drafter. Vision is served by the loaded checkpoint (verified on synthetic images 2026-09-20). The sections below describe the 2026-09-16 two-replica era.
 tags:
  - frank-medium
  - dankserver
@@ -13,28 +13,121 @@ tags:
  - rccl
  - pcie-atomics
  - quadlet
-timestamp: 2026-09-16
+timestamp: 2026-09-20
 ---
 > **SUPERSEDED 2026-09-16.** This page is the record of the retired
 > vllm-radiance MXFP4 deployment. The ACTIVE frank-medium is mainline
 > llama.cpp with the DFlash2 drafter ([[frank-medium-llamacpp-dflash2]]);
 > the PCIe-atomics evidence in this page remains the reason the runtime is
 > layer-split llama.cpp rather than tensor-parallel vLLM.
 # frank-medium vllm-radiance MXFP4 Runbook
-The ACTIVE frank-medium deployment (since 2026-09-15/16): the
+The ACTIVE frank-medium deployment: the `stilldeadcode/vllm-radiance:0.9.3`
-`stilldeadcode/vllm-radiance:0.9.3` image plus the `ggz14/radiance-vllm-mxfp4`
+image (tagged locally as `localhost/radiance-vllm:0.9.3`) plus the
-patches, serving `just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq` on the two
+`ggz14/radiance-vllm-mxfp4` patches, serving
-Radeon AI PRO R9700 (gfx1201) cards. It replaced the charlie12345/ROCmFPX
+`just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq` as **ONE
-llama.cpp deployment ([[frank-medium-rocmfpx]]).
+tensor-parallel process across both Radeon AI PRO R9700 (gfx1201)**.
 It replaced the mainline-llama.cpp layer-split deployment
 ([[frank-medium-llamacpp-dflash2]], SUPERSEDED 2026-09-20); the same image and
 model had earlier run here as two single-GPU replicas, which is the shape most
 of this page describes.
-**Deployed shape: two single-GPU replicas behind an adaptive-routing proxy.** The
+**Deployed shape since 2026-09-20: one rootless Quadlet, `--tensor-parallel-size 2`.**
-intended shape — one tensor-parallel process across both cards (TP=2) — cannot
+The ProArt B850-Creator WiFi NEO board swap put both R9700s on CPU GPP root
-launch a collective on this host. The reason is a hardware property of the
+ports (`00:01.1 → 03:00.0` and `00:01.3 → 07:00.0`), both Gen5 x8, every
-motherboard's PCIe topology, not the image, the RCCL build, or the kernel
+upstream bridge `AtomicOpsCap: Routing+` and both endpoints `ReqEn+` — the wall
-version. The evidence is in [Why TP=2 cannot run here](#why-tp2-cannot-run-here).
+documented in [Why TP=2 cannot run here](#why-tp2-cannot-run-here) is gone on this
 board. The collective that actually runs is vLLM's R4D one-shot allreduce, which
 does not use RCCL at all (TheRock RCCL has no gfx1201 device code; see
 [[Dual-R9700-Cooperation]]).
 ## Deployed flags (2026-09-20)
 From `dankserver/frank-medium/quadlet/frank-medium.container`:
 | Setting | Value |
 |---|---|
 | Model | `just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq` |
 | Speculative | `--speculative-config {"method":"dflash","model":"/models/draft","num_speculative_tokens":7}` (z-lab DFlash2-FP8) |
 | Parallelism | `--tensor-parallel-size 2` (`ROCR_VISIBLE_DEVICES=0,1`) |
 | Context | `--max-model-len 262144` |
 | Concurrency | `--max-num-seqs 8`, chunk 8192 |
 | KV | fp8 (`--kv-cache-dtype fp8`), **auto-sized — no `KV_CACHE_MEMORY_MB`** |
 | Prompt cache | `--enable-prefix-caching` |
 | GPU fill | `--gpu-memory-utilization 0.98` |
 | Parsers | `--enable-auto-tool-choice --tool-call-parser qwen3_coder`, `--reasoning-parser qwen3` |
 | Sampler floor | `--override-sampling-effects` (temp 0.7 / top_p 0.95 / top_k 20) |
 | Multimodal | `--limit-mm-per-prompt '{"image": 8}'` |
 | Chat template | `qwen-fixed-v22.3.jinja` (patch dir bind-mounted, `MM_MERGE_ENGINE=AXROW_SEPARATE` for Gemma4 vision) |
 **The KV pool is auto-sized, not pinned.** The arm-R benchmark run used
 `KV_CACHE_MEMORY_MB=18563072000`, which bought exactly one 262,144-token
 sequence (262,720 tokens at 70,656 B/token — a profile inflated by the
 8192-chunk activation peak). Production ships no such pin: the pool came up at
 **943,581 tokens = 3.59× the full window**, and the boot line read
 `Available KV cache memory: 18.17 GiB → GPU KV cache size: 943,581 tokens`.
 Per-token cost in steady state is 19,673 B — fp8 KV across the model's 16
 full-attention layers plus the drafter's. Container RSS measured 28.4 GiB
 against the host's 62 GiB and stayed flat through a 4-way burst at 512-token
 max; host-wide swap 3.3 GiB, unchanged. The 2026-09-16 swap storm was the
 two-replica vision shape, not this one.
 ## Battery results — why this shape won (2026-09-20)
 `scripts/bench-conversation.py`, 10 real-workload prompts, `max_tokens=512`,
 greedy, salted per run and per arm, service stopped between arms
 (`scripts/bench-conversation.py --host http://127.0.0.1:7770 --burst 4`):
 | Arm | Shape | Decode (geomean) | TTFT (mean) | Prefill (geomean) | Burst-4 aggregate |
 |---|---|---|---|---|---|
 | I | llama.cpp layer split 32:33 + DFlash2 Q4_K_M drafter | 45.7 t/s | 0.72 s | ~530 t/s | 71.7 t/s (26.7/stream) |
 | L | llama.cpp `--split-mode tensor` 1:1, drafterless | 30.5 t/s | 0.71 s | — | 54.3 t/s |
 | **R** | **vllm-radiance TP=2 + DFlash2-FP8** | **95.8 t/s** | **0.21 s** | **1493.6 t/s** | **259.4 t/s (65.6/stream)** |
 Re-measured against the production unit after deploy: geomean decode 92.4 t/s,
 mean TTFT 0.22 s, burst-4 aggregate 248.9 t/s — within noise of arm R.
 Honest caveat: arm R carries MXFP4 (4-bit) where the llama arms carry Q8_0
 (8-bit), so part of the win is quantization, not tensor parallelism alone. Arm
 L exists to show that llama.cpp's own TP is not the answer here regardless —
 it is slower than its own layer split, and cannot carry the drafter at all.
 ## Verified capabilities (2026-09-20, against the live endpoint)
 - `/v1/models` → `id: frank-medium`; `/health` → 200.
 - **Tool calls**: `finish_reason: tool_calls` via `qwen3_coder`. The harness
  drives a bash tool through this endpoint end to end.
 - **Reasoning**: `reasoning_effort` accepts `none`/`low`/`medium`/`high`
  natively (vLLM's `ReasoningEffort`); thinking arrives in the `reasoning`
  field, NOT llama.cpp's `reasoning_content`. `max_tokens` counts visible
  content only; with no reasoning budget the model still spends it, so a tiny
  budget returns `content: null` with `finish_reason: length`.
 - **Vision works** despite `language_model_only: true` in `config.json` — the
  Gemma4 merge path answers image requests (~83 prompt tokens for a small
  synthetic), correctly identifying shape and colour on held-out images.
  `models.yml` therefore lists this endpoint as `input: [text, image]`.
 - Drafter is live: accept length 2.9–4.3 tokens per draft, acceptance ~0.60.
 ## Rollback
 The llama.cpp layer-split trio is archived in `scripts/.AGENTTRASH/
 20260920-llama-layersplit/` (`frank-medium.service`, `frank-medium.env`,
 `serve-frank-medium.sh`) and the runtime + Q8_0 payload remain on disk:
 1. `git revert` the cutover commit.
 2. `deploy.sh dankserver --service frank-medium` (re-installs and re-enables the root unit).
 3. Or by hand: restore the three files, move the unit symlink back into `/etc/systemd/system/`, `sudo systemctl daemon-reload && sudo systemctl unmask frank-medium && sudo systemctl start frank-medium`.
 4. Gate: `curl -s http://192.168.0.235:7770/health` → 200.
 ## R4D vs RCCL
 vLLM's R4D one-shot allreduce bypasses RCCL entirely because TheRock RCCL
 builds lack gfx1201 device code (`invalid device function`). The Ubuntu
 RCCL 2.27 / `librccl1` 7.1.0 build does load gfx1201 kernels — that is what
 `scripts/build-llama-rdna4.sh` with `GGML_HIP_RCCL=ON` links
 (`build-rdna4-rocm714-rccl/`) — so llama.cpp's tensor mode has a real
 collective too. It is simply the slower shape.
 > **The sections from here down describe the 2026-09-16 → 2026-09-19 era: two
 > single-GPU replicas behind an adaptive-routing proxy, on the TUF GAMING
 > B650-E.** The deployed shape is above; this is the evidence trail that made
 > replicas necessary then and what the 2026-09-20 board swap changed.
 ## Architecture
@@ -71,6 +164,14 @@ host would restart forever without ever becoming healthy.
 ## Why TP=2 cannot run here
 > **Resolved 2026-09-20.** Everything below is true of the **TUF GAMING
 > B650-E WIFI**, whose second card sat behind chipset bridges advertising
 > `AtomicOpsCap: Routing-`. The ProArt B850-Creator WiFi NEO swap moved both
 > cards onto CPU GPP root ports (`00:01.1 → 03:00.0`, `00:01.3 → 07:00.0`, Gen5
 > x8, every hop `Routing+`, both endpoints `ReqEn+`), and TP=2 now boots and
 > serves. The measurement verdict against the alternatives is in
 > [[Dual-R9700-Cooperation]].
 ### Symptom
 Any two-rank collective dies at launch:
@@ -83,14 +184,16 @@ Any two-rank collective dies at launch:
 With the runtime's own API log (`AMD_LOG_LEVEL=3`) the failing call is explicit:
 ```
-ShaderName : ncclDevKernel_Generic_4(ncclDevKernelArgsStorage<4096ul>)
+[ShaderName : ncclDevKernel_Generic_4(ncclDevKernelArgsStorage<4096ul>)]
-:1:rocvirtual.cpp :4208:  Pcie atomics not enabled, hostcall not supported
+[ :1:rocvirtual.cpp :4208:  Pcie atomics not enabled, hostcall not supported]
-:1:rocvirtual.cpp :4636:  AQL dispatch failed!
+[ :1:rocvirtual.cpp :4636:  AQL dispatch failed!]
-:3:hip_module.cpp :893 :  hipLaunchKernel: Returned hipErrorIllegalState
+[ :3:hip_module.cpp :893 :  hipLaunchKernel: Returned hipErrorIllegalState]
 ```
 ROCclr refuses to hand the kernel a hostcall buffer when the device has no
-PCIe atomics (`rocclr/device/rocm/rocvirtual.cpp:3963`), and the launch aborts.
+PCIe atomics (`pcie_atomics_` is true for the device init, and ROCr passes
 that buffer only when `pcie_atomics_` is true; otherwise it returns "not
 supported" and the dispatch fails) — and the launch aborts.
 `hipErrorIllegalState` is that refusal, not a driver fault.
 ### Chain of causation
@@ -99,15 +202,15 @@ PCIe atomics (`rocclr/device/rocm/rocvirtual.cpp:3963`), and the launch aborts.
 2. ROCr passes that buffer only when `pcie_atomics_` is true for the device;
   otherwise it returns "not supported" and the dispatch fails.
 3. `pcie_atomics_` is set once, at device init, from the HSA link info between
-   the CPU agent and the GPU (`rocclr/device/rocm/rocdevice.cpp:186`,
+   the CPU agent and the GPU (`Device::checkAtomicSupport`). It is not re-read
-   `Device::checkAtomicSupport`). It is not re-read per process.
+   per process.
 4. The KFD driver populates that link info from
   `pci_enable_atomic_ops_to_root()`, which — per the PCIe spec and the
   kernel's implementation — requires **every upstream bridge** between the
   device and its root port to advertise `AtomicOp Routing` capability
   (`drivers/pci/pci.c`).
-### The hardware difference between the two cards
+The hardware difference between the two cards:
 | Card | PCIe chain (endpoint → root port) | `AtomicOpsCap` on every hop | AtomicOpsCtl |
 |---|---|---|---|
@@ -135,7 +238,7 @@ boot with the write already in place) would change it — and even then the two
 | P2P hardware | fine: 45.3 GB/s bidirectional copy between the two cards; both cards pass `rocm-bandwidth-test` |
 | The shipped librccl | a from-source rebuild of `ROCm/rccl` develop (with local fixes for that branch's dangling references) reproduces the same failure at the same call path — the library is not at fault |
 | vLLM/pynccl | a bare two-rank `torch.distributed` allreduce with no vLLM involved fails the same way; single-rank collectives succeed on both cards (NCCL short-circuits at `world_size=1`, which is why a one-card test proves nothing) |
-| Transport knobs | `NCCL_PROTO=Simple`, `NCCL_P2P_DISABLE`, `NCCL_SHM_DISABLE`, `NCCL_CUMEM_ENABLE=0`, MSCCL off, channel counts, launch modes, `HSA_ENABLE_SDMA=0` — all identical |
+| Transport knobs | `NCCL_PROTO=Simple`, `P2P_DISABLE`, `SHM_DISABLE`, `CUMEM_ENABLE=0`, MSCCL off, channel counts, launch modes, `HSA_ENABLE_SDMA=0` — all identical |
 | Image/support matrix | the same image on a different host with the same kernel (Proxmox VM, Codeberg issue #22) fails byte-identically |
 ### What would fix it
@@ -145,7 +248,10 @@ boot with the write already in place) would change it — and even then the two
  true TP=2. On this AM5 platform the CPU provides the x16 slot plus x4 lanes,
  so a second x16-class card necessarily lands on the chipset path — a
  platform with more CPU lanes (Threadripper/EPYC) is the structural answer.
- Until then, **two replicas** are how both cards serve.
+- **Done 2026-09-20 — this is exactly what happened.** The ProArt
  B850-Creator WiFi NEO routes both slots to CPU root ports, the atomics
  condition is satisfied, and TP=2 is the deployed shape. The Threadripper/EPYC
  argument was correct about the old board and is now moot.
 ## Routing
@@ -159,7 +265,7 @@ The proxy decides per request, from `proxy/routing.py` (policy) and
  subagents — spreads across the cards by counter, so both prefill in parallel.
 Measured on this host, 90k-token contexts and 15k-token prompts, `vllm bench`
-plus a small sequential-turn client:
+plus a sequential-turn client:
 | strategy | cold 16-request burst | warm-turn TTFT (4 turns × 2 families) |
 |---|--:|--:|
@@ -187,13 +293,16 @@ Both replicas, uniform except GPU index and port:
 | Knob | Value | Why |
 |---|---|---|
 | `TP` | 1 | one card per replica |
-| `MAXLEN` | 114688 | the largest shape that boots on one card: 262144 needs 9.61 GiB of KV against 5.14 available, and 131072 needs 5.58 against 5.22 |
+| `MAXLEN` | 114688 | the largest shape that boots on one card: 262144 needs 9.61 GiB of KV against 5.21 GiB available, and 131072 needs 5.58 GiB against 5.22 |
-| `KV_MEM` | 0 | passed through literally as `--kv-cache-memory 0`, which the image's patched KV sizing reads as "profile this shape"; the upstream pin (2x7551-32624) is a TP=2/262144 measurement and is invalid here. **Do not blank the variable instead of passing 0**: with the flag omitted vLLM sizes the cache from `GPU_UTIL` alone and the KV allocation dies with `torch.OutOfMemoryError` (measured: 2.16 GiB requested, 116 MiB free, 29.34 GiB already held) |
+| `KV_MEM` | 0 | passed through literally as `--kv-cache-memory 0`, which the image's patched KV sizing reads as "profile this shape"; the upstream pin (2x7551-32624) is a TP=2 measurement and is invalid here. **Do not blank the variable instead of passing 0**: with the flag omitted vLLM sizes the cache from `GPU_UTIL` alone and the KV allocation dies with `torch.OutOfMemoryError` (measured: 2.16 GiB requested, 116 MiB free, 29.34 GiB already held) |
 | `MAXSEQS` | 4 | matches the per-replica share of the fleet's concurrency |
-| `CHUNK` | 8192 | prefill chunk (upstream default), **coupled to the context**: the prefill arena grows with the chunk size and shares the device budget with the KV cache. Measured: `CHUNK=16384` leaves 4.22 GiB of KV against the 5.21 GiB that 114688 context needs, and the engine refuses to start (`To serve at least one request with the model's max seq len (114688), 5.21 GiB KV cache is needed, which is larger than the available KV cache memory (4.22 GiB)`). Raising CHUNK therefore means cutting `MAXLEN`; single-card prefill measured ≈2.5k tok/s at 8192 (~15k-token prompt, 5992 ms TTFT), so the trade was not worth taking |
+| `CHUNK` | 8192 | prefill chunk (upstream default), **coupled to the context**: the prefill arena grows with the chunk size and shares the device budget with the KV cache. Measured: `CHUNK=16384` leaves 4.22 GiB of KV against the 5.21 GiB that 114688 context needs, and the engine refuses to start (`To serve at least one request with the model's max seq len (114688), 5.21 GiB KV cache is needed, which is larger than the available KV cache memory (4.22 GiB)`). Raising CHUNK therefore means cutting `MAXLEN`; single-card prefill measured ≈2.5k tok/s at 8192 (~15k-token prompt, 5992 ms TTFT) |
 | `GPU_UTIL` | 0.98 | upstream default; the card is dedicated |
-| `SPEC_METHOD` / `SPEC` | dflash / 7 | the DFlash2-FP8 block-diffusion drafter. Depth 7 is upstream's production default and was re-measured here on a code corpus built from this repo (see [Speculative depth](#speculative-depth)); depth 5 wins batch throughput, 7 wins single-stream latency, and this fleet is interactive |
+| `SPEC_METHOD` / `SPEC` | dflash / 7 | the z-lab DFlash2-FP8 block-diffusion drafter. Depth 7 is upstream's production default and was re-measured here on a code corpus built from this repo (see [Speculative depth](#speculative-depth)); depth 5 wins batch throughput, 7 wins single-stream latency, and this fleet is interactive |
-| `SNAP` / `DRAFTER` | `/models/Qwen3.8-27B-Uncensored-MXFP4-awq` / `/models/Qwen3.8-27B-DFlash2-FP8` | the served checkpoint and its drafter |
+| `spec-draft-device` | `ROCm1` | the drafter rides the card that ends up holding the target's last layers |
 | `temp` / `top-p` / `top-k` | `0.7` / `0.95` / `20` | the values the vLLM predecessor served on this endpoint |
 | `reasoning-format` / `reasoning-budget` | `deepseek` / `-1` | trace into `reasoning_content`, unlimited |
 | `mmproj` | `mmproj-Qwen3.8-27B-Uncensored-f16.gguf` (beside the target) | vision enabled; this build has NO autoload, so the flag is explicit and the serve script refuses to start without the file |
 Mounts are the TP=2 unit's minus nothing: `/models`, the HF cache, the shared
 compile cache, `/patches` (the pinned radiance clone; its patches are applied
@@ -202,7 +311,7 @@ at every container start), `/r4d` (the patched libr4d — the image's shipped
 the entrypoint/healthcheck/proxy scripts mounted read-only out of
 `/home/dank/serverAppConfigs`.
-The healthcheck reads `PORT`, so one probe serves the replicas (:7771/:7772),
+The healthcheck reads `PORT`, so one probe serves the replicas (:7770),
 the proxy (:7770) and the TP=2 target.
 ### Boot persistence, verified by rebooting
@@ -216,7 +325,8 @@ sudo reboot                 # unattended; ~2 min 14 s to graphical.target
 # ... no manual step at any point ...
 systemctl --user is-active frank-medium-a frank-medium-b frank-medium-proxy
  -> active active active            (containers up 59 s after boot)
-curl http://127.0.0.1:7770/health   -> both replicas UP ~3.5 min after boot
+# ...
 curl http://127.0.0.1:7770/health        # both replicas UP ~3.5 min after boot
 ```
 Why it comes back: each unit's `[Install] WantedBy=default.target` — the generator
@@ -1,13 +1,13 @@
 # LLM Fleet — Knowledge Base
-Documentation for the LLM inference fleet on the two hosts (2026-09-19):
+Documentation for the LLM inference fleet on the two hosts (2026-09-20):
-**mainline llama.cpp on Dankserver** serving frank-medium as one layer-split
+**vllm-radiance on Dankserver** serving frank-medium as ONE tensor-parallel
-process across both Radeon AI PRO R9700 cards (orcarouter Qwen3.8-27B-
+process across both Radeon AI PRO R9700 cards (just1moremodel
-Uncensored Q8_0 + the z-lab DFlash2 Q4_K_M drafter, balanced 32:33), and TWO
+Qwen3.8-27B-Uncensored MXFP4-awq + the z-lab DFlash2-FP8 drafter, TP=2 over the
-endpoints on Dankclaw — the CIRU v4.4.1 runtime (frank-large, jcbtc
+R4D one-shot allreduce), and TWO endpoints on Dankclaw — the CIRU v4.4.1
-Qwen3.8-Flash-CIRU-STRIX-Orca w4b on the gfx1151 iGPU, port 7770) and mainline
+runtime (frank-large, jcbtc Qwen3.8-Flash-CIRU-STRIX-Orca w4b on the gfx1151
-llama.cpp (frank-small, HauhauCS Gemma4-12B QAT Q4_K_M with MTP and vision on
+iGPU, port 7770) and mainline llama.cpp (frank-small, HauhauCS Gemma4-12B QAT
-the RTX 3090, port 7771).
+Q4_K_M with MTP and vision on the RTX 3090, port 7771).
 > 2026-08-22: the former `vllmScripts` repository is merged into the
 > serverAppConfigs repository. This knowledge base moved from the old repo's
@@ -21,10 +21,10 @@ graph LR
    subgraph Dankserver[AMD Dankserver - Ubuntu 26.04]
        A[2x Radeon AI PRO R9700
 gfx1201, 32 GiB each]
-        B2[llama.cpp, layer split
+        B2[vllm-radiance 0.9.3, TP=2
 frank-medium, port 7770]
-        C[orcarouter Qwen3.8-27B Q8_0 + DFlash2 Q4_K_M
+        C[Qwen3.8-27B MXFP4-awq + DFlash2-FP8
-ONE process on both cards, 2x163840 ctx, balanced 32:33]
+ONE process on both cards, 262144 ctx, up to 8 seqs]
        A --> B2 --> C
    end
@@ -46,6 +46,7 @@ frank-small, port 7771]
    CLI[LLM Clients / API] -->|HTTP :7770| B2
    CLI -->|HTTP :7770| E
    CLI -->|HTTP :7771| G
 ```
 ## Servers
@@ -56,21 +57,25 @@ frank-small, port 7771]
 ## Active Deployment
- [[frank-medium-llamacpp-dflash2]] — the ACTIVE frank-medium runbook since
+- [[frank-medium-radiance-mxfp4]] — the ACTIVE frank-medium runbook since
-  2026-09-16 (model swapped 2026-09-19): mainline llama.cpp serving
+  2026-09-20: vllm-radiance MXFP4 (stilldeadcode/vllm-radiance:0.9.3 +
  orcarouter/Qwen3.8-27B-Uncensored-GGUF at Q8_0 with the z-lab DFlash2 Q4_K_M
  drafter on Dankserver's 2x Radeon AI PRO R9700, as ONE layer-split process
  behind :7770 (tensor parallelism was impossible on the old board's second
  card; the 2026-09-20 swap cleared that gate but the runtime still ships
  RCCL-less and the collective is unprobed), balanced 32:33 over two 163,840-token
  slots
 - [[frank-medium-radiance-mxfp4]] — RETIRED 2026-09-16, one day after it was
  deployed: the vllm-radiance MXFP4 stack (stilldeadcode/vllm-radiance:0.9.3 +
  ggz14/radiance-vllm-mxfp4 patches) serving
-  just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq as two single-GPU replicas
+  just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq with the z-lab DFlash2-FP8
-  behind a round-robin proxy; kept as the record of the deployment it
+  drafter as ONE tensor-parallel process (TP=2) behind :7770 — the three-arm
-  succeeded, plus the PCIe-atomics evidence that ruled out TP=2 on the previous
+  battery that chose it, the auto-sized 943,581-token KV pool, verified
-  board — that gate cleared on 2026-09-20, see [[Dual-R9700-Cooperation]]
+  capabilities (tools, reasoning, vision), and the record of the 2026-09-16
  two-replica era
 - [[frank-medium-llamacpp-dflash2]] — RETIRED 2026-09-16 → 2026-09-20: mainline
  llama.cpp serving orcarouter/Qwen3.8-27B-Uncensored-GGUF at Q8_0 with the
  DFlash2 Q4_K_M drafter as ONE layer-split process (32:33 over two
  163,840-token slots); superseded by the radiance TP=2 cutover, and the
  rollback target. Keeps the batch/ubatch optimum curve and the per-layer split
  arithmetic
 - [[Dual-R9700-Cooperation]] — whether the two R9700s can serve ONE model: the
  RCCL/PCIe-atomics wall on the old TUF GAMING B650-E, the 2026-09-20 ProArt
  B850-Creator x8/x8 swap that cleared it, and the measured three-way verdict
  (radiance TP=2 95.8 t/s decode vs layer split 45.7 vs llama tensor mode 30.5)
  that put TP=2 in production
 - [[frank-medium-rocmfpx]] — PRIOR frank-medium deployment (llama.cpp era,
  2026-09-15): charlie12345/ROCmFPX serving
  kingjones777/Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF; kept as the record of the
@@ -115,26 +115,22 @@ The LLM inference fleet on the two hosts. Full index:
 - [[deployment]] — unified deploy and update process for both servers
 - [[frank-redo-2026-08-17]] — the 2026-08-17 topology change
- [[frank-medium-llamacpp-dflash2]] — the ACTIVE frank-medium runbook since
+- [[frank-medium-radiance-mxfp4]] — the ACTIVE frank-medium runbook since
-  2026-09-16 (model swapped 2026-09-19): mainline llama.cpp (gfx1201 HIP build)
+  2026-09-20: vllm-radiance MXFP4 as ONE tensor-parallel process (TP=2 over the
-  serving orcarouter/Qwen3.8-27B-Uncensored-GGUF at Q8_0 with the DFlash2
+  R4D one-shot allreduce) serving just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq
-  drafter (z-lab/Qwen3.8-27B-DFlash2-GGUF at Q4_K_M) on Dankserver's 2x Radeon
+  with the z-lab DFlash2-FP8 drafter on Dankserver's 2x Radeon AI PRO R9700
-  AI PRO R9700 as one layer-split process behind the :7770 endpoint, balanced
+  behind the :7770 endpoint — deployed flags, the three-arm battery that chose
-  32:33 over two 163,840-token slots — no tensor parallelism: the runtime ships
+  it, verified capabilities, and the record of the 2026-09-16 two-replica era
-  RCCL-less, and the collective has not been re-probed since the 2026-09-20
+- [[frank-medium-llamacpp-dflash2]] — RETIRED 2026-09-16 → 2026-09-20:
-  board swap cleared the atomics gate
+  mainline llama.cpp (gfx1201 HIP build) serving orcarouter Qwen3.8-27B at
- [[frank-medium-radiance-mxfp4]] — RETIRED 2026-09-16: the vllm-radiance
+  Q8_0 with the DFlash2 drafter as ONE layer-split process (32:33, two
-  MXFP4 stack serving just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq on
+  163,840-token slots); holds the batch/ubatch optimum curve and the
-  Dankserver's 2x Radeon AI PRO R9700 as two single-GPU replicas behind a
+  per-layer split arithmetic the rollback recipe points back to
-  round-robin proxy; kept as the record of the deployment
+- [[Dual-R9700-Cooperation]] — whether the two R9700s can run ONE model: the
-  [[frank-medium-llamacpp-dflash2]] replaced, plus the PCIe-atomics evidence
+  RCCL/PCIe-atomics evidence on the old TUF GAMING B650-E (which closed tensor
-  that ruled out TP=2 on the previous board (gate cleared 2026-09-20)
+  parallelism too), what the 2026-09-20 ProArt B850-Creator x8/x8 swap changed
- [[Dual-R9700-Cooperation]] — why the two R9700s *could not* run tensor-parallel
+  — both chains `Routing+`, both endpoints `ReqEn+` — and the measured
-  together: the PCIe-atomics gate quoted from ROCclr, the measured RCCL matrix
+  three-way verdict: radiance TP=2 won and is the deployed shape
  (collectives *and* P2P `send`/`recv` both failed on card #2, which ruled out
  pipeline parallelism too), what was closed, and the re-measured 2026-09-20
  topology after the ProArt B850-Creator x8/x8 swap — both chains `Routing+`,
  both endpoints `ReqEn+`, so TP=2 is unblocked but still unprobed
 - [[frank-medium-rocmfpx]] — PRIOR frank-medium deployment (llama.cpp era):
  charlie12345/ROCmFPX serving kingjones777/
  Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF; kept as the record of the runtime it
@@ -11,6 +11,69 @@ timestamp: 2026-09-14
 # Bundle Change Log
 ## 2026-09-20 — frank-medium cut over to radiance TP=2; monitoring follows the runtime
 - **Measured first, then cut over.** Three arms of `scripts/bench-conversation.py`
  (10 real-workload prompts, `max_tokens=512`, greedy, salted per arm, service
  stopped between arms per the Benchmarking Policy): llama.cpp layer split 32:33 +
  DFlash2 Q4_K_M **45.7 t/s** decode geomean / 0.72 s TTFT / 71.7 t/s burst-4;
  llama.cpp `--split-mode tensor` 1:1 drafterless **30.5** / 0.71 / 54.3;
  vllm-radiance TP=2 + DFlash2-FP8 **95.8** / 0.21 / 259.4 (1493.6 t/s prefill).
  Honest caveat recorded everywhere: the winning arm is also 4-bit where the
  llama arms are 8-bit, so part of the margin is quantization, not parallelism.
 - **RCCL built for the question.** `scripts/build-llama-rdna4.sh` takes
  `GGML_HIP_RCCL` now; the RCCL-backed `build-rdna4-rocm714-rccl` boots llama.cpp
  tensor mode with a real collective, which is what made arm L a fair test.
  vLLM's own collective is R4D one-shot, not RCCL — TheRock's RCCL has no gfx1201
  device code (`invalid device function`).
 - **Deploy.** `deploy.sh dankserver --service frank-medium`: the llama.cpp root
  unit's symlink moved to `~/.AGENTTRASH/20260920-llama-layersplit-frank-medium/`
  (never `systemctl disable` — it deletes the file), the Quadlet installed and
  enabled, boot in 181 s, `/health` green. Post-deploy re-measure reproduced the
  bench: 92.4 t/s, 0.22 s TTFT, 248.9 t/s burst-4.
 - **KV pool is auto-sized, not pinned.** Production ships no `KV_CACHE_MEMORY_MB`:
  the pool came up at **943,581 tokens = 3.59× the 262,144 window**, container RSS
  flat at 28.4 GiB through a 4-way burst, host swap unchanged at 3.3 GiB. The
  18,563,072,000-byte pin was an arm-R measurement artifact (one full-context
  sequence), not the deployed shape.
 - **Capabilities verified against the live endpoint, not inferred**: `/v1/models`,
  tool calls (`qwen3_coder`, harness drove a real bash call through it),
  `reasoning_effort` none/low/medium/high with thinking in `reasoning`, drafter
  accept length 2.9–4.3, and **vision works** despite `language_model_only: true`
  in `config.json` — the model identified shape and colour on held-out synthetic
  images. `~/.omp/agent/models.yml` therefore lists `input: [text, image]`,
  `contextWindow: 262144`, `maxTokens: 32768` (compaction gap 183,500 + 32,768
  ≤ 262,144).
 - **Monitoring cut over with the runtime.** `shared/prometheus/rules/llm-fleet.yml`
  now spans two namespaces — `vllm:*` for frank-medium, `llamacpp:*` for the
  Dankclaw pair — with 7 alerts + 2 recording rules; KV-saturation and preemption
  pressure returned (vLLM exports both), and the two engine-independent rules OR
  the families with per-family traffic floors. Every expression validated against
  live Prometheus. **Two scrape targets were red by construction and are fixed**:
  frank-small and frank-large are *host processes* on Dankclaw and netavark DNATs
  only published ports, so both are now addressed as `host.containers.internal`.
  Known gap: the provisioned Grafana dashboard is still keyed to `llamacpp:*`, so
  frank-medium's panels are blank pending a re-cut.
 - **Network decision.** `eno2` (5GbE, 192.168.0.247) kept, `eno1` pinned primary
  via metric 100 vs 200 in `scripts/dankserver-route-primary.sh`; applied with
  `nmcli device reapply` (there is no `nmcli connection reapply` in NM 1.54).
 - **Docs.** [[frank-medium-radiance-mxfp4]] is the ACTIVE runbook (deployed flags,
  battery, capabilities, rollback, with the replica-era sections labelled);
  [[frank-medium-llamacpp-dflash2]] carries the SUPERSEDED banner and past-tense
  lead; [[Dual-R9700-Cooperation]] now states the measured verdict instead of the
  open question; [[Observability]] records the namespace split and the
  netavark/DNAT lesson; `AGENTS.md` topology, frank-medium section, Model Config,
  Benchmarking Policy (frank-medium is a **user** Quadlet now — `systemctl --user
  stop/mask`) and the runbook index all flipped.
 - **Doc-hygiene incident worth repeating.** Two parallel doc agents left the same
  three wounds twice: prose prepended **above** the `---` frontmatter opener,
  whole body sections duplicated wholesale, and a literal
  `"<|channel>}<tool_call|>…"` token pasted into a runbook. `check-vault-links.py`
  catches only unresolved links, not structure — so the fix was `git checkout
  HEAD --` the destroyed index and re-apply the flips by hand, with the agent
  versions moved to `.AGENTTRASH/20260920-index-dupes/`. After any agent pass over
  docs: re-check frontmatter is first, headings unique, and links resolve.
 ## 2026-09-20 — dankserver board swap: the PCIe-atomics wall is gone
 - **Hardware.** dankserver now runs an **ASUS ProArt B850-Creator WiFi NEO**
  (was TUF GAMING B650-E WIFI), same Ryzen 7 7700X and same 2x R9700. The board
@@ -1033,3 +1096,83 @@ timestamp: 2026-09-14
  `--ctx-checkpoints`; this binary's `--help` says **32**, so the 0 is a
  deliberate deviation. Fixed in the env comment and in the runbook's flag
  table.
 ## 2026-09-20 — frank-medium cut over to radiance TP=2; llama layer-split retired
 - **Measurement.** Three arms, one salted battery (`scripts/bench-conversation.py`,
  10 prompts, streaming, TTFT + decode tok/s, `--burst N`), the service stopped
  and masked per the benchmarking policy for every arm. radiance TP=2: decode
  geomean 95.8 t/s, TTFT 0.21 s, prefill geomean 1493.6 t/s, burst-4 aggregate
  259.4 t/s (65.6 per stream). llama.cpp `--split-mode tensor` 1:1: 30.5 t/s.
  llama.cpp layer split 32:33 (the incumbent): 45.7 t/s, 0.72 s, 71.7 t/s burst.
  Radiance won every axis; commit `c211665` cut the endpoint over.
 - **The pre-probe came out of the harness first.** A warm prefix-cache hit hid
  prefill, so the first row of every arm had been measuring the cache, not the
  engine.
 - **llama.cpp tensor mode cannot carry the drafter on this build.** `--split-mode
  tensor` plus the DFlash2 drafter aborts at `ggml-backend.cpp:941` (a weight
  pre-allocated on a non-primary device has no backend when `n_copies == 1`), so
  arm L ran with NO speculative decoding — its 30.5 t/s is therefore not a
  like-for-like against radiance's spec-decoded 95.8, and is reported that way in
  [[Dual-R9700-Cooperation]]. `--split-mode row` is SYCL-only (it needs
  `ggml_backend_split_buffer_type`); `tensor` is the HIP/CUDA mode.
 - **A collective HAS been dispatched on this board since the swap**, which
  supersedes the "unblocked, not proven" line in the entry above.
  `scripts/build-llama-rdna4.sh` now takes `GGML_HIP_RCCL=${GGML_HIP_RCCL:-OFF}`
  and bundles `librccl.so*` + `librocm-core` when ON; the ON build
  (`build-rdna4-rocm714-rccl`) is what ran arm L. Production stays OFF. Ubuntu
  RCCL 2.27 carries gfx1201 device code; TheRock's does not ("invalid device
  function"), which is why radiance ships the collective-free R4D one-shot
  allreduce instead of depending on RCCL at all.
 - **Deploy.** `deploy_llm_medium` in `scripts/deploy-remote.sh` gates on image
  tag, MXFP4 checkpoint, drafter, patched `libr4d`, and the chat template; stops
  and trashes the root llama unit (the relinked symlink is MOVED, never
  `systemctl disable`, which would delete the file); waits for 7770 to be
  released; installs the quadlet; waits for container health. Deployed healthy in
  181 s at `max_model_len` 262144, TP=2, 8 conversations, DFlash2 depth 7.
 - **The KV pool ships auto-sized.** The unit sets no `KV_CACHE_MEMORY_MB`, so the
  engine sized the pool itself: 943,581 tokens = 3.59 concurrent full 262,144-token
  conversations. Under a 4-way burst the container cgroup stayed flat at 28.4 GiB
  RAM and 3.3 GiB swap with host MemAvailable ~48 GiB, so the 2026-09-16 swap
  storm (measured on TWO vision-carrying replicas at ~943k each) does not
  reproduce on this shape. Arm R ran the explicit 18563072000 pin — 262,720
  tokens, 1.00x full window — and production prefers the 3.59x.
 - **Capabilities verified live rather than assumed.** Tool calls return
  `tool_calls` with `finish_reason: tool_calls` under
  `--tool-call-parser qwen3_coder`. `reasoning_effort` low/medium/high/none all
  answer 200, and both `none` and `chat_template_kwargs.enable_thinking=false`
  switch thinking off, so this endpoint needs no `reasoningEffortMap` (that
  workaround belongs to Qwen's restricted template on the Dankclaw endpoints).
  Thinking arrives as vLLM's `reasoning`, not llama.cpp's `reasoning_content`.
  **Vision works**: the loader installs the Gemma4 vision path and the endpoint
  answered two held-out synthetic images correctly ("This is a red circle." /
  "This is a blue rectangle."), so the `language_model_only` → text-only premise
  written earlier today was wrong and `models.yml` keeps `input: [text, image]`.
 - **Production reproduces the winning arm**: same battery, 92.4 t/s decode
  geomean, 0.22 s TTFT, 1213.3 t/s prefill, 248.9 t/s burst-4 — arm R within
  noise.
 - **Harness.** `~/.omp/agent/models.yml` frank-medium now declares
  `contextWindow: 262144` (the pool is 3.59x the window and preempts rather than
  refusing, so the window is the honest number), `maxTokens: 32768` inside the
  compaction gap, `input: [text, image]`, no effort map. A
  `omp -p --no-session --model frank-medium` tool round trip returned the
  command's output.
 - **Monitoring cutover.** `shared/prometheus/rules/llm-fleet.yml` gained the vllm:
  half — `LlmRequestsWaiting`, `LlmKvCacheSaturated`,
  `job:llm_vllm_prefix_cache_reuse:rate5m`, `job:llm_spec_accept_length:rate5m` —
  and the two existing collapse/spec alerts now cover both namespaces with a
  traffic floor on each side. Every expression was run against the live
  Prometheus; the accept-length rule answers 4.09 for frank-medium right now.
  `shared/prometheus/prometheus.yml` also fixed a target that was red by
  construction: from this bridge-networked Prometheus a host-process bind is
  reachable at `host.containers.internal`, NOT at the LAN IP (netavark DNATs
  published ports only), so `frank-small` had read down for days while healthy
  and exporting. `frank-large` is now scraped too — it exports 19 `llamacpp:`
  families including every series the rules use, contra the comment that excluded
  it.
 - **The two-NIC question from the entry above is decided**: `eno1` is pinned as
  the primary route (metric 100) with `eno2` retained as failover (metric 200) by
  `scripts/dankserver-route-primary.sh`, applied with `nmcli device reapply`
  (`nmcli connection reapply` does not exist in NetworkManager 1.54).
 - **Rollback.** `git revert c211665`, relink `scripts/frank-medium.service` from
  `scripts/.AGENTTRASH/20260920-llama-layersplit/`, `sudo systemctl restart
  frank-medium.service`. The llama runtime and the Q8_0 payload are still on disk.
@@ -47,10 +47,10 @@
  <h2>frank-large — the big brain</h2>
  <ul>
-    <li><strong>Model:</strong> Qwen3.8-Flash-Next, w4b quantization, served by halogen-flash-server.</li>
+    <li><strong>Model:</strong> Qwen3.8-Flash CIRU-STRIX-Orca, served by CIRU v4.4.1 llama.cpp with the model's own MTP draft head.</li>
    <li><strong>Hardware:</strong> Dankclaw's Strix Halo iGPU (gfx1151), shared with the whole host.</li>
-    <li><strong>Best for:</strong> long, thoughtful text conversations. 200k-token context per conversation.</li>
+    <li><strong>Best for:</strong> long, thoughtful text conversations at the model's full trained window.</li>
-    <li><strong>Capacity:</strong> exactly two conversations resident at once. A third waits — politely, but it waits.</li>
+    <li><strong>Capacity:</strong> exactly one conversation resident at a time. A second waits — politely, but it waits.</li>
    <li><strong>Not for:</strong> images. It cannot see them.</li>
  </ul>
@@ -67,11 +67,11 @@
    <li><strong>Model:</strong> Qwen3.8-27B-Uncensored at MXFP4, on vllm-radiance with a DFlash2 draft head.</li>
    <li><strong>Hardware:</strong> Dankserver's two Radeon R9700s as ONE tensor-parallel engine (TP=2).</li>
    <li><strong>Best for:</strong> general chat and reasoning when the big brain is busy; the biggest pure-language model we run, and the fastest we have measured.</li>
-    <li><strong>Capacity:</strong> 262k-token context, shared by up to eight conversations at once. Text only — pictures go to <code>frank-small</code>.</li>
+    <li><strong>Capacity:</strong> 262k-token context shared by up to eight conversations at once. It can see pictures too; <code>frank-small</code> is the faster eye and holds four conversations.</li>
  </ul>
  <p class="foot-note">Which one to pick? Screenshots and photos: <code>frank-small</code>.
-  Deep long-text work: <code>frank-large</code> (mind the two-seat limit).
+  Deep long-text work: <code>frank-large</code> (mind the single seat).
  When in doubt: <code>frank-medium</code>.</p>
 </section>
@@ -233,7 +233,7 @@
  {
    "name": "frank-large",
    "category": "user",
-    "description": "The big brain. Qwen3.8-Flash-Next w4b on halogen-flash-server, running on Dankclaw's Strix Halo iGPU. Text-only; holds two 200k-token conversations resident.",
+    "description": "The big brain. Qwen3.8-Flash CIRU-STRIX-Orca on CIRU v4.4.1 llama.cpp, running on Dankclaw's Strix Halo iGPU. Text-only; one conversation resident at a time.",
    "url": "",
    "docs": "/docs#models",
    "probe": "127.0.0.1:7770",
@@ -69,34 +69,51 @@ scrape_configs:
          - "192.168.0.143:8081"   # SearXNG (via the Gluetun tunnel)
          - "192.168.0.143:3005"   # OpenWebUI
-  # LLM fleet. Scraped on the LAN addresses, not the guarded vhosts: these are
+  # LLM fleet. Scraped on the LAN addresses where the endpoint is reachable from
-  # machine clients and Prometheus cannot solve an Anubis challenge.
+  # this container, not the guarded vhosts: these are machine clients and
-  # Both LLM endpoints are llama.cpp, so both export the same llamacpp:
+  # Prometheus cannot solve an Anubis challenge.
  # counter family: prompt/decode throughput, requests_deferred (queue depth
  # — the only form llama.cpp exposes the capacity wall in), the
  # spec_decode_num_drafts / spec_decode_num_accepted pair (proof the drafter
  # is running), and the prefix-cache pair prompt_tokens_cached_total over
  # prompt_tokens_total (their rates are the reuse rate, which says whether
  # pinned conversations are actually being reused).
  # frank-large is deliberately NOT scraped: the halogen engine that serves it
  # publishes /health, /cache and /v1/* and no Prometheus text endpoint, so a
  # target here would be red by construction and would fire LlmEndpointDown
  # forever. Its liveness is on the homepage (a loopback probe) and in the
  # engine's own boot ledger. The remedy, when the fleet wants it, is the
  # blackbox exporter already wanted for the comfyui/services probes.
  #
-  # frank-small is llama.cpp (its serve script passes --metrics) and does
+  # Two metric namespaces, because the fleet runs two runtimes:
-  # export text-format counters under llamacpp: on the port its unit publishes.
+  #
  #   * vllm:* — frank-medium (vllm-radiance TP=2 since 2026-09-20): prompt and
  #     generation token counters, num_requests_waiting plus its reason label,
  #     num_preemptions_total, the kv_cache_usage_perc fill gauge, the
  #     prefix_cache hits/queries pair, the spec_decode drafts/accepted triple,
  #     and real latency histograms (time_to_first_token, inter_token,
  #     e2e_request_latency).
  #   * llamacpp:* — frank-large and frank-small. Both take --metrics and both
  #     were verified 2026-09-20 to export requests_deferred (the queue depth,
  #     the only form llama.cpp shows the capacity wall in),
  #     spec_decode_num_drafts_total, tokens_predicted_total, and the
  #     prompt_tokens_cached_total / prompt_tokens_total reuse pair.
  #
  # frank-large is scraped over host.containers.internal, not its own LAN address:
  # this Prometheus is a bridge-networked container (Network=dankclaw_default),
  # and on this netavark host a container can only reach the host's LAN IP at a
  # PUBLISHED port — a host-process bind like llama-server's 0.0.0.0:7770 is
  # refused at 192.168.0.143:7770 while it answers at 169.254.1.2:7770. That is
  # why frank-small was red here for days (up{job="frank-small"} == 0,
  # "dial tcp 192.168.0.143:7771") even though the endpoint was healthy and
  # exporting: the target address was unreachable by construction, not the
  # service down. Same host, same rule, so both local endpoints use the gateway.
  # frank-medium stays a LAN address — it is a different machine.
  - job_name: "frank-medium"
    metrics_path: "/metrics"
    static_configs:
      - targets: ["192.168.0.235:7770"]
        labels:
          service: frank-medium
-
+  
  - job_name: "frank-small"
    metrics_path: "/metrics"
    static_configs:
-      - targets: ["192.168.0.143:7771"]
+      - targets: ["host.containers.internal:7771"]
        labels:
          service: frank-small
  - job_name: "frank-large"
    metrics_path: "/metrics"
    static_configs:
      - targets: ["host.containers.internal:7770"]
        labels:
          service: frank-large
@@ -1,39 +1,64 @@
 # Alerting and recording rules for the LLM fleet.
 #
-# Every expression here is keyed to a series that EXISTS as of 2026-09-16. When
+# Two metric namespaces, because the fleet runs two runtimes:
 # frank-medium moved from vllm-radiance to llama.cpp the exported namespace
 # changed with it (vllm: -> llamacpp:), and two signals the old file alerted on
 # no longer exist in any form:
 #
-#   * vllm:kv_cache_usage_perc — llama.cpp exports no KV-fill gauge. Its
+#   * llamacpp:* — frank-small (Dankclaw, llama.cpp with --metrics).
-#     observable consequence is queueing, which is what LlmRequestsQueueing
+#   * vllm:* — frank-medium (Dankserver, vllm-radiance TP=2 since 2026-09-20).
-#     watches instead. A renamed query would have been a rule that can never
+#
-#     fire.
+# Every expression is keyed to a series that EXISTS, verified against
-#   * vllm:e2e_request_latency_seconds_* — no latency histogram either. The
+# 192.168.0.235:7770/metrics and 192.168.0.143:7771/metrics on 2026-09-20.
-#     rule it fed (p95 wall clock) is retired rather than restated, and
+# Nothing here matches on absence: when frank-medium moved back to vLLM the
-#     LlmSpecDecodeOff takes its place: the failure an operator actually
+# llamacpp: selectors narrowed to frank-small rather than going vacuous, and
-#     cannot see from the outside on this deployment.
+# the vllm: group below was added beside them, not in place of them.
 #
 # What each namespace can and cannot see:
 #
 #   * KV pressure. vLLM exports a fill gauge (vllm:kv_cache_usage_perc) and a
 #     preemption counter; llama.cpp exports neither, so its capacity wall is
 #     observed as queueing (llamacpp:requests_deferred).
 #   * Prefix reuse. Both export a hits/queries pair, in tokens; the two
 #     recording rules below keep the units straight and the collapse alert
 #     watches both sides with one threshold.
 #   * Speculative decoding. Both export drafts and accepted tokens, so
 #     LlmSpecDecodeOff covers both runtimes.
 #   * Latency histograms exist only on vLLM (e2e, TTFT, inter-token). No
 #     p95 alert is defined for the llama.cpp side because there is nothing to
 #     compute one from.
 #
 # IMPORTANT: delivery. Since 2026-09-17 a firing alert reaches Frank on
 # Signal: Prometheus pushes to Alertmanager (dankclaw/observability/quadlet/
 # alertmanager.container), whose single receiver webhooks the host-side
 # signal-relay user unit, which hands the text to signal-cli under the
-# existing account on this host. Before that date a firing alert appeared on
+# existing account on this host.
 # Prometheus' /alerts page, in the ALERTS metric (which the LLM fleet
 # dashboard graphs) and nowhere else.
 groups:
  - name: llm-fleet
    interval: 30s
    rules:
-      # Recording rule so the dashboard and the alert below share one
+      # Recording rules so the dashboard and the collapse alert below share one
-      # expression. The exported series are counters; the rate is the reuse.
+      # expression. Both namespaces export hits/queries pairs in TOKENS; the
      # rate of the quotient is the reuse rate. Kept as two records because the
      # series names differ and a vacuous joint query is worse than two honest
      # ones.
      - record: job:llm_prefix_cache_reuse:rate5m
        expr: |
          rate(llamacpp:prompt_tokens_cached_total[5m])
            / clamp_min(rate(llamacpp:prompt_tokens_total[5m]), 0.001)
-      # The capacity wall, in the only form llama.cpp exports it: work waiting
+      - record: job:llm_vllm_prefix_cache_reuse:rate5m
-      # for the single slot. Past here the next large request queues.
+        expr: |
          rate(vllm:prefix_cache_hits_total[5m])
            / clamp_min(rate(vllm:prefix_cache_queries_total[5m]), 0.001)
      # Mean accepted draft length (1.0 = the drafter proposes and nothing is
      # taken, so the draft round is pure overhead). Measured 3.0-4.3 on
      # frank-medium at boot with the DFlash2 drafter, 2026-09-20.
      - record: job:llm_spec_accept_length:rate5m
        expr: |
          1 + rate(vllm:spec_decode_num_accepted_tokens_total[5m])
              / clamp_min(rate(vllm:spec_decode_num_drafts_total[5m]), 0.001)
      # The capacity wall on the llama.cpp side, in the only form it exports:
      # work waiting for a slot. Past here the next large request queues.
      - alert: LlmRequestsQueueing
        expr: llamacpp:requests_deferred > 0
        for: 10m
@@ -46,13 +71,50 @@ groups:
            fleet is genuinely busy or one endpoint is carrying it alone. With
            --parallel 1 a second concurrent request always shows up here.
      # The capacity wall on the vLLM side, in its own two terms: requests the
      # scheduler has not admitted, and requests it admitted and then PREEMPTED
      # (a preemption throws away the KV and re-prefills, so its rate is the
      # cost of running the pool full). frank-medium auto-sizes its pool to
      # 943,581 tokens = 3.59 concurrent full 262,144-token conversations, so
      # saturation is four such conversations, not two.
      - alert: LlmRequestsWaiting
        expr: vllm:num_requests_waiting > 0
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "{{ $labels.job }}: vLLM has not admitted work for 10 minutes"
          description: >-
            The scheduler is holding requests back. Check the reason label on
            vllm:num_requests_waiting_by_reason (capacity = no KV room, defer =
            waiting for a better batch shape) and the preemption counter
            vllm:num_preemptions_total.
      - alert: LlmKvCacheSaturated
        expr: vllm:kv_cache_usage_perc > 0.95
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "{{ $labels.job }}: KV pool above 95 % for 10 minutes"
          description: >-
            The pool is full, which is where preemption and re-prefill start:
            decode slows before anything errors. The pool is auto-sized by the
            quadlet (no KV_CACHE_MEMORY_MB pin); the boot ledger prints the
            token count — `podman logs frank-medium | grep "GPU KV cache size"`
            — and it was 943,581 tokens on 2026-09-20.
      # The thrash signature measured on 2026-09-16: requests keep arriving but
      # almost none are served from cache, which is what several concurrent
-      # 50-80k-token conversations do to this fleet.
+      # 50-80k-token conversations do to this fleet. One threshold, two
      # namespaces, and a traffic floor per side so an idle endpoint cannot
      # fire it on a 0/0.
      - alert: LlmPrefixCacheCollapse
        expr: >-
-          job:llm_prefix_cache_reuse:rate5m < 0.20
+          (job:llm_prefix_cache_reuse:rate5m < 0.20
-            and rate(llamacpp:prompt_tokens_total[5m]) > 0.05
+            and rate(llamacpp:prompt_tokens_total[5m]) > 0.05)
          or (job:llm_vllm_prefix_cache_reuse:rate5m < 0.20
            and rate(vllm:prefix_cache_queries_total[5m]) > 1)
        for: 15m
        labels:
          severity: info
@@ -65,12 +127,14 @@ groups:
      # The speculative path is the one failure no client can see: the server
      # answers normally, just at half the speed, when the drafter is gone.
-      # Drafts and accepts are counters, so a rate that stays at zero while the
+      # Drafts and accepts are counters in both runtimes, so a rate that stays
-      # endpoint is serving tokens means nothing is being drafted.
+      # at zero while the endpoint is serving tokens means nothing is drafted.
      - alert: LlmSpecDecodeOff
        expr: >-
-          rate(llamacpp:spec_decode_num_drafts_total[5m]) == 0
+          (rate(llamacpp:spec_decode_num_drafts_total[5m]) == 0
-            and rate(llamacpp:tokens_predicted_total[5m]) > 0.1
+            and rate(llamacpp:tokens_predicted_total[5m]) > 0.1)
          or (rate(vllm:spec_decode_num_drafts_total[5m]) == 0
            and rate(vllm:generation_tokens_total[5m]) > 0.1)
        for: 15m
        labels:
          severity: warning
@@ -78,11 +142,11 @@ groups:
          summary: "{{ $labels.job }}: serving without speculative decoding"
          description: >-
            Tokens are being produced but no draft has been proposed for 15
-            minutes. On the llama.cpp endpoints that means the drafter is not
+            minutes. frank-small (llama.cpp) should log "adding speculative
-            running (check the unit's log for the
+            implementation" at load; frank-medium (vllm-radiance) should be
-            "adding speculative implementation" line — draft-mtp on both
+            counting vllm:spec_decode_num_drafts_total and reporting 3+ mean
-            endpoints as of 2026-09-17), and the endpoint is decoding
+            accept length (job:llm_spec_accept_length:rate5m). Either way the
-            autoregressively.
+            endpoint is decoding autoregressively.
      # The endpoint, not a replica: one process serves each port now, so a
      # restart of the unit is a scrape gap, not a failover.
@@ -94,8 +158,10 @@ groups:
        annotations:
          summary: "{{ $labels.job }}: scrape target down for 2 minutes"
          description: >-
-            Prometheus cannot reach the LLM endpoint. Check the units:
+            Prometheus cannot reach the LLM endpoint. frank-medium is a
-            `systemctl is-active frank-medium` on Dankserver (a root llama.cpp
+            rootless quadlet: `systemctl --user is-active frank-medium` on
-            unit) and `systemctl is-active frank-small` on Dankclaw. frank-large
+            Dankserver, then `podman ps -a --filter name=frank-medium`.
-            is not scraped at all — the halogen engine exports no metrics — so
+            frank-small is a root unit: `systemctl is-active frank-small` on
-            this alert can only ever name the endpoints that do export one.
+            Dankclaw. frank-large is not scraped at all — its CIRU runtime
            serves no Prometheus text endpoint — so this alert can only ever
            name the endpoints that do export one.
@@ -300,21 +300,23 @@ PAIRS
@test "prometheus scrapes every LLM endpoint that exports metrics" {
    # The LLM endpoints are the services whose behaviour this fleet argues about
-    # most (KV pressure, prefix-cache reuse, queue depth), and vLLM exports all
+    # most (KV pressure, prefix-cache reuse, queue depth). Since frank-medium
-    # of it. If a scrape target drifts from the endpoint the units publish,
+    # moved to radiance/vLLM on 2026-09-20 there are two metric namespaces, and
-    # nobody notices until someone needs the graph during an incident.
+    # the three addresses differ by HOW each endpoint is reachable from a
    # container: netavark DNATs *published* container ports only, so
    # frank-medium — a rootless Quadlet on another host that publishes 7770 — is
    # its LAN address, while the two Dankclaw endpoints are HOST processes and
    # answer only as host.containers.internal. Addressing those two by LAN IP
    # left both targets red by construction until the 2026-09-20 /targets audit.
    local cfg="$REPO_ROOT/shared/prometheus/prometheus.yml"
    grep -q '"192.168.0.235:7770"' "$cfg" || { echo "frank-medium not scraped"; return 1; }
-    grep -q '"192.168.0.143:7771"' "$cfg" || { echo "frank-small not scraped"; return 1; }
+    grep -q '"host.containers.internal:7771"' "$cfg" || { echo "frank-small not scraped"; return 1; }
-    # The medium endpoint is exported by the proxy on the host network; the
+    grep -q '"host.containers.internal:7770"' "$cfg" || { echo "frank-large not scraped"; return 1; }
-    # quadlet's own contract test pins that port, so name the job too.
+    for job in frank-medium frank-small frank-large; do
-    grep -q 'job_name: "frank-medium"' "$cfg"
+        grep -q "job_name: \"$job\"" "$cfg" || { echo "no scrape job for $job"; return 1; }
-    grep -q 'job_name: "frank-small"' "$cfg"
+    done
-    # frank-large is NOT scraped on purpose: the halogen engine serves
+    # None of them is reached through a guarded vhost: Prometheus cannot solve
-    # /health, /cache and /v1/* and no metrics endpoint, so a target here would
+    # an Anubis challenge, so a franksdank.cloud target is red by construction.
    # be red by construction and would fire LlmEndpointDown forever. Assert the
    # absence, so a well-meaning edit cannot re-add a permanently-red probe.
    ! grep -q '192.168.0.143:7770' "$cfg"
    ! grep -qE '^\s*-\s*"?https?://[^"]*franksdank' "$cfg"
 }
@@ -351,23 +353,46 @@ PAIRS
 import re, yaml
 d = yaml.safe_load(open('$rules'))
 names = [r.get('alert') or r.get('record') for g in d['groups'] for r in g['rules']]
-for want in ('job:llm_prefix_cache_reuse:rate5m', 'LlmRequestsQueueing',
+for want in ('job:llm_prefix_cache_reuse:rate5m',
-             'LlmPrefixCacheCollapse', 'LlmSpecDecodeOff', 'LlmEndpointDown'):
+             'job:llm_vllm_prefix_cache_reuse:rate5m',
             'job:llm_spec_accept_length:rate5m',
             'LlmRequestsQueueing', 'LlmRequestsWaiting',
             'LlmKvCacheSaturated', 'LlmPrefixCacheCollapse',
             'LlmSpecDecodeOff', 'LlmEndpointDown'):
    assert want in names, want
-# Every rule must be keyed to a series that exists. The vLLM era's namespaces
+# Every rule must be keyed to a series that actually exists. A rule naming a
-# are gone (frank-medium is llama.cpp now) and a rule naming one of them
+# series nobody exports evaluates nothing forever and errors never — the failure
-# evaluates nothing forever without erroring — the failure this seam exists for.
+# this seam exists for. Since 2026-09-20 two engines are scraped, so both
 # namespaces are legal, but each name must be on the measured list.
 exprs = [r.get('expr', '') for g in d['groups'] for r in g['rules']]
-assert not any('vllm:' in e for e in exprs), exprs
+# llama.cpp families measured from /metrics on 2026-09-16 (frank-small,
-# ...and the llama.cpp series a rule may name are the ones the server exports
+# frank-large), vLLM families measured from /metrics on 2026-09-20
-# (measured from /metrics on 2026-09-16). Anything else is a guess.
+# (frank-medium, radiance/vLLM). Anything else is a guess.
 live = {'llamacpp:prompt_tokens_total', 'llamacpp:prompt_tokens_cached_total',
        'llamacpp:requests_deferred', 'llamacpp:requests_processing',
        'llamacpp:spec_decode_num_drafts_total',
        'llamacpp:spec_decode_num_accepted_tokens_total',
-        'llamacpp:tokens_predicted_total'}
+        'llamacpp:tokens_predicted_total',
-used = {m for e in exprs for m in re.findall(r'llamacpp:[a-z_0-9]+', e)}
+        'vllm:kv_cache_usage_perc', 'vllm:num_requests_running',
        'vllm:num_requests_waiting', 'vllm:num_preemptions_total',
        'vllm:generation_tokens_total', 'vllm:prompt_tokens_total',
        'vllm:spec_decode_num_accepted_tokens_total',
        'vllm:spec_decode_num_draft_tokens_total',
        'vllm:spec_decode_num_drafts_total',
        'vllm:prefix_cache_hits_total', 'vllm:prefix_cache_queries_total',
        'vllm:request_success_total'}
 used = {m for e in exprs
        for m in re.findall(r'(?:llamacpp|vllm):[a-z_0-9]+', e)}
 assert used <= live, used - live
 # Engine-independent rules must cover BOTH families, or the cutover silently
 # blinds one endpoint: the two rules that describe an engine-agnostic failure
 # name a series from each namespace.
 joined = '\n'.join(exprs)
 for pair in (('LlmSpecDecodeOff', 'llamacpp:spec_decode_num_drafts_total',
              'vllm:spec_decode_num_drafts_total'),
             ('LlmPrefixCacheCollapse', 'llamacpp:prompt_tokens_total',
              'vllm:prefix_cache_queries_total')):
    assert any(pair[1] in e and pair[2] in e for e in exprs), pair[0]
 print('rules ok:', len(names))
 "
    # A rule file that nothing loads evaluates nothing, and Prometheus reports