feat(monitoring,harness): follow frank-medium onto vllm:* metrics and fix red-by-construction targets

The radiance TP=2 cutover changed the metric namespace frank-medium exports
(llamacpp:* -> vllm:*) and left two scrape targets addressing host processes
by an address a container cannot reach.

- shared/prometheus/rules/llm-fleet.yml: two namespaces side by side. Added
  LlmRequestsWaiting, LlmKvCacheSaturated (vllm-only gauges llama.cpp never
  had), the job:llm_vllm_prefix_cache_reuse:rate5m and
  job:llm_spec_accept_length:rate5m recording rules, and re-cut
  LlmPrefixCacheCollapse / LlmSpecDecodeOff to OR both families with
  per-family traffic floors so an idle endpoint cannot fire them. Every
  expression validated against live Prometheus.
- shared/prometheus/prometheus.yml: frank-small -> host.containers.internal:7771
  and frank-large scraped at host.containers.internal:7770. Both are HOST
  processes on Dankclaw and netavark DNATs published container ports only, so
  the LAN-IP targets were red by construction; frank-large's CIRU llama.cpp
  build does export the llamacpp:* family, contra the old comment.
- tests/observability_quadlet.bats: the two seams now encode the new contract
  (per-endpoint reachability by process type; every named series must be on the
  measured list for its engine; engine-independent rules must name both).
- homepage docs.html/services.json: frank-medium sees pictures, frank-large is
  CIRU Orca with a single seat (was halogen w4b, two seats, 200k).
- docs: radiance runbook is ACTIVE (deployed flags, three-arm battery, verified
  capabilities, rollback, replica-era sections labelled); llama.cpp runbook
  carries the SUPERSEDED banner; Dual-R9700-Cooperation states the measured
  verdict; Observability records the namespace split and the DNAT lesson;
  AGENTS.md topology/frank-medium/Model Config/Benchmarking Policy/runbook
  index flipped; both indexes updated; work-log entry added.

Also repaired agent-pass damage: prose prepended above frontmatter openers,
duplicated runbook bodies, a literal tool-call token pasted into a runbook, and
an index destroyed by diff artifacts (restored from HEAD, flips re-applied).
Vault link check: 128 pages, 0 unresolved. ./tests/run.sh: 325 pass.

serverAppConfigs-docs/log.md

@@ -11,6 +11,69 @@ timestamp: 2026-09-14
 
 # Bundle Change Log
 
+## 2026-09-20 — frank-medium cut over to radiance TP=2; monitoring follows the runtime
+
+- **Measured first, then cut over.** Three arms of `scripts/bench-conversation.py`
+  (10 real-workload prompts, `max_tokens=512`, greedy, salted per arm, service
+  stopped between arms per the Benchmarking Policy): llama.cpp layer split 32:33 +
+  DFlash2 Q4_K_M **45.7 t/s** decode geomean / 0.72 s TTFT / 71.7 t/s burst-4;
+  llama.cpp `--split-mode tensor` 1:1 drafterless **30.5** / 0.71 / 54.3;
+  vllm-radiance TP=2 + DFlash2-FP8 **95.8** / 0.21 / 259.4 (1493.6 t/s prefill).
+  Honest caveat recorded everywhere: the winning arm is also 4-bit where the
+  llama arms are 8-bit, so part of the margin is quantization, not parallelism.
+- **RCCL built for the question.** `scripts/build-llama-rdna4.sh` takes
+  `GGML_HIP_RCCL` now; the RCCL-backed `build-rdna4-rocm714-rccl` boots llama.cpp
+  tensor mode with a real collective, which is what made arm L a fair test.
+  vLLM's own collective is R4D one-shot, not RCCL — TheRock's RCCL has no gfx1201
+  device code (`invalid device function`).
+- **Deploy.** `deploy.sh dankserver --service frank-medium`: the llama.cpp root
+  unit's symlink moved to `~/.AGENTTRASH/20260920-llama-layersplit-frank-medium/`
+  (never `systemctl disable` — it deletes the file), the Quadlet installed and
+  enabled, boot in 181 s, `/health` green. Post-deploy re-measure reproduced the
+  bench: 92.4 t/s, 0.22 s TTFT, 248.9 t/s burst-4.
+- **KV pool is auto-sized, not pinned.** Production ships no `KV_CACHE_MEMORY_MB`:
+  the pool came up at **943,581 tokens = 3.59× the 262,144 window**, container RSS
+  flat at 28.4 GiB through a 4-way burst, host swap unchanged at 3.3 GiB. The
+  18,563,072,000-byte pin was an arm-R measurement artifact (one full-context
+  sequence), not the deployed shape.
+- **Capabilities verified against the live endpoint, not inferred**: `/v1/models`,
+  tool calls (`qwen3_coder`, harness drove a real bash call through it),
+  `reasoning_effort` none/low/medium/high with thinking in `reasoning`, drafter
+  accept length 2.9–4.3, and **vision works** despite `language_model_only: true`
+  in `config.json` — the model identified shape and colour on held-out synthetic
+  images. `~/.omp/agent/models.yml` therefore lists `input: [text, image]`,
+  `contextWindow: 262144`, `maxTokens: 32768` (compaction gap 183,500 + 32,768
+  ≤ 262,144).
+- **Monitoring cut over with the runtime.** `shared/prometheus/rules/llm-fleet.yml`
+  now spans two namespaces — `vllm:*` for frank-medium, `llamacpp:*` for the
+  Dankclaw pair — with 7 alerts + 2 recording rules; KV-saturation and preemption
+  pressure returned (vLLM exports both), and the two engine-independent rules OR
+  the families with per-family traffic floors. Every expression validated against
+  live Prometheus. **Two scrape targets were red by construction and are fixed**:
+  frank-small and frank-large are *host processes* on Dankclaw and netavark DNATs
+  only published ports, so both are now addressed as `host.containers.internal`.
+  Known gap: the provisioned Grafana dashboard is still keyed to `llamacpp:*`, so
+  frank-medium's panels are blank pending a re-cut.
+- **Network decision.** `eno2` (5GbE, 192.168.0.247) kept, `eno1` pinned primary
+  via metric 100 vs 200 in `scripts/dankserver-route-primary.sh`; applied with
+  `nmcli device reapply` (there is no `nmcli connection reapply` in NM 1.54).
+- **Docs.** [[frank-medium-radiance-mxfp4]] is the ACTIVE runbook (deployed flags,
+  battery, capabilities, rollback, with the replica-era sections labelled);
+  [[frank-medium-llamacpp-dflash2]] carries the SUPERSEDED banner and past-tense
+  lead; [[Dual-R9700-Cooperation]] now states the measured verdict instead of the
+  open question; [[Observability]] records the namespace split and the
+  netavark/DNAT lesson; `AGENTS.md` topology, frank-medium section, Model Config,
+  Benchmarking Policy (frank-medium is a **user** Quadlet now — `systemctl --user
+  stop/mask`) and the runbook index all flipped.
+- **Doc-hygiene incident worth repeating.** Two parallel doc agents left the same
+  three wounds twice: prose prepended **above** the `---` frontmatter opener,
+  whole body sections duplicated wholesale, and a literal
+  `"<|channel>}<tool_call|>…"` token pasted into a runbook. `check-vault-links.py`
+  catches only unresolved links, not structure — so the fix was `git checkout
+  HEAD --` the destroyed index and re-apply the flips by hand, with the agent
+  versions moved to `.AGENTTRASH/20260920-index-dupes/`. After any agent pass over
+  docs: re-check frontmatter is first, headings unique, and links resolve.
+
 ## 2026-09-20 — dankserver board swap: the PCIe-atomics wall is gone
 - **Hardware.** dankserver now runs an **ASUS ProArt B850-Creator WiFi NEO**
   (was TUF GAMING B650-E WIFI), same Ryzen 7 7700X and same 2x R9700. The board
@@ -1033,3 +1096,83 @@ timestamp: 2026-09-14
   `--ctx-checkpoints`; this binary's `--help` says **32**, so the 0 is a
   deliberate deviation. Fixed in the env comment and in the runbook's flag
   table.
+
+## 2026-09-20 — frank-medium cut over to radiance TP=2; llama layer-split retired
+- **Measurement.** Three arms, one salted battery (`scripts/bench-conversation.py`,
+  10 prompts, streaming, TTFT + decode tok/s, `--burst N`), the service stopped
+  and masked per the benchmarking policy for every arm. radiance TP=2: decode
+  geomean 95.8 t/s, TTFT 0.21 s, prefill geomean 1493.6 t/s, burst-4 aggregate
+  259.4 t/s (65.6 per stream). llama.cpp `--split-mode tensor` 1:1: 30.5 t/s.
+  llama.cpp layer split 32:33 (the incumbent): 45.7 t/s, 0.72 s, 71.7 t/s burst.
+  Radiance won every axis; commit `c211665` cut the endpoint over.
+- **The pre-probe came out of the harness first.** A warm prefix-cache hit hid
+  prefill, so the first row of every arm had been measuring the cache, not the
+  engine.
+- **llama.cpp tensor mode cannot carry the drafter on this build.** `--split-mode
+  tensor` plus the DFlash2 drafter aborts at `ggml-backend.cpp:941` (a weight
+  pre-allocated on a non-primary device has no backend when `n_copies == 1`), so
+  arm L ran with NO speculative decoding — its 30.5 t/s is therefore not a
+  like-for-like against radiance's spec-decoded 95.8, and is reported that way in
+  [[Dual-R9700-Cooperation]]. `--split-mode row` is SYCL-only (it needs
+  `ggml_backend_split_buffer_type`); `tensor` is the HIP/CUDA mode.
+- **A collective HAS been dispatched on this board since the swap**, which
+  supersedes the "unblocked, not proven" line in the entry above.
+  `scripts/build-llama-rdna4.sh` now takes `GGML_HIP_RCCL=${GGML_HIP_RCCL:-OFF}`
+  and bundles `librccl.so*` + `librocm-core` when ON; the ON build
+  (`build-rdna4-rocm714-rccl`) is what ran arm L. Production stays OFF. Ubuntu
+  RCCL 2.27 carries gfx1201 device code; TheRock's does not ("invalid device
+  function"), which is why radiance ships the collective-free R4D one-shot
+  allreduce instead of depending on RCCL at all.
+- **Deploy.** `deploy_llm_medium` in `scripts/deploy-remote.sh` gates on image
+  tag, MXFP4 checkpoint, drafter, patched `libr4d`, and the chat template; stops
+  and trashes the root llama unit (the relinked symlink is MOVED, never
+  `systemctl disable`, which would delete the file); waits for 7770 to be
+  released; installs the quadlet; waits for container health. Deployed healthy in
+  181 s at `max_model_len` 262144, TP=2, 8 conversations, DFlash2 depth 7.
+- **The KV pool ships auto-sized.** The unit sets no `KV_CACHE_MEMORY_MB`, so the
+  engine sized the pool itself: 943,581 tokens = 3.59 concurrent full 262,144-token
+  conversations. Under a 4-way burst the container cgroup stayed flat at 28.4 GiB
+  RAM and 3.3 GiB swap with host MemAvailable ~48 GiB, so the 2026-09-16 swap
+  storm (measured on TWO vision-carrying replicas at ~943k each) does not
+  reproduce on this shape. Arm R ran the explicit 18563072000 pin — 262,720
+  tokens, 1.00x full window — and production prefers the 3.59x.
+- **Capabilities verified live rather than assumed.** Tool calls return
+  `tool_calls` with `finish_reason: tool_calls` under
+  `--tool-call-parser qwen3_coder`. `reasoning_effort` low/medium/high/none all
+  answer 200, and both `none` and `chat_template_kwargs.enable_thinking=false`
+  switch thinking off, so this endpoint needs no `reasoningEffortMap` (that
+  workaround belongs to Qwen's restricted template on the Dankclaw endpoints).
+  Thinking arrives as vLLM's `reasoning`, not llama.cpp's `reasoning_content`.
+  **Vision works**: the loader installs the Gemma4 vision path and the endpoint
+  answered two held-out synthetic images correctly ("This is a red circle." /
+  "This is a blue rectangle."), so the `language_model_only` → text-only premise
+  written earlier today was wrong and `models.yml` keeps `input: [text, image]`.
+- **Production reproduces the winning arm**: same battery, 92.4 t/s decode
+  geomean, 0.22 s TTFT, 1213.3 t/s prefill, 248.9 t/s burst-4 — arm R within
+  noise.
+- **Harness.** `~/.omp/agent/models.yml` frank-medium now declares
+  `contextWindow: 262144` (the pool is 3.59x the window and preempts rather than
+  refusing, so the window is the honest number), `maxTokens: 32768` inside the
+  compaction gap, `input: [text, image]`, no effort map. A
+  `omp -p --no-session --model frank-medium` tool round trip returned the
+  command's output.
+- **Monitoring cutover.** `shared/prometheus/rules/llm-fleet.yml` gained the vllm:
+  half — `LlmRequestsWaiting`, `LlmKvCacheSaturated`,
+  `job:llm_vllm_prefix_cache_reuse:rate5m`, `job:llm_spec_accept_length:rate5m` —
+  and the two existing collapse/spec alerts now cover both namespaces with a
+  traffic floor on each side. Every expression was run against the live
+  Prometheus; the accept-length rule answers 4.09 for frank-medium right now.
+  `shared/prometheus/prometheus.yml` also fixed a target that was red by
+  construction: from this bridge-networked Prometheus a host-process bind is
+  reachable at `host.containers.internal`, NOT at the LAN IP (netavark DNATs
+  published ports only), so `frank-small` had read down for days while healthy
+  and exporting. `frank-large` is now scraped too — it exports 19 `llamacpp:`
+  families including every series the rules use, contra the comment that excluded
+  it.
+- **The two-NIC question from the entry above is decided**: `eno1` is pinned as
+  the primary route (metric 100) with `eno2` retained as failover (metric 200) by
+  `scripts/dankserver-route-primary.sh`, applied with `nmcli device reapply`
+  (`nmcli connection reapply` does not exist in NetworkManager 1.54).
+- **Rollback.** `git revert c211665`, relink `scripts/frank-medium.service` from
+  `scripts/.AGENTTRASH/20260920-llama-layersplit/`, `sudo systemctl restart
+  frank-medium.service`. The llama runtime and the Q8_0 payload are still on disk.