frank-medium: serve the whole 262144-token window, and repoint every signal to llama.cpp

Context: 262144 was measured before it was deployed -- 24.72 GB on GPU 0 and
29.14 GB on GPU 1 of 34.21 GB each, ~5 GB of headroom on the busier card, and a
140,069-token prompt prefilled through the single slot in 134 s to prove the
window is real. The 114688 wall was the vLLM predecessor's arithmetic, not this
runtime's.

Throughput, measured through the service: 1.0-1.4k tok/s prefill from 4k to
140k, 43.8 tok/s decode on a 512-token generation, 41-71 % DFlash2 acceptance,
and the prefix cache re-prefilling 4 of 998 tokens on a repeated prompt.

The signals move with the runtime, so the fleet's rules and dashboard are
re-cut rather than renamed: vllm: series do not exist any more, and two of the
old rules watched things llama.cpp does not export at all (KV-pool fill, p95
latency). Those are retired with the reason written down, and LlmSpecDecodeOff
takes one of the slots -- a lost drafter is the failure no client can see. The
observability seam now fails if any rule names a series the endpoint does not
export, which is what would have caught the stale vllm: expressions.

omp models.yml (outside the repo) is updated in the same pass: contextWindow
262144, and the full reasoning-effort map, because Qwen3.8's template answers
500 on 'high' -- verified against the live endpoint, low/medium/xhigh answer
200. Subagents run at high.

AGENTS.md

@@ -24,7 +24,7 @@ merge absorbed the former `vllmScripts` repo, history preserved).
 
 | Target | Service | Runtime | Model | Port |
 |--------|---------|---------|-------|------|
-| Dankserver | `frank-medium` | vllm-radiance 0.9.3 (ggz14/radiance-vllm-mxfp4 patches), 2x single-GPU rootless Quadlets behind an adaptive-routing proxy | just1moremodel Qwen3.8-27B-Uncensored-MXFP4-awq + Qwen3.8-27B-DFlash2-FP8 drafter, 2x Radeon AI PRO R9700 (one card per replica, TP=1), 114688 ctx / 4 seqs each | 7770 |
+| Dankserver | `frank-medium` | mainline llama.cpp master fb27a525 (root systemd unit, gfx1201 HIP build) | ajgazin Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP UD-Q8_K_XL + z-lab Qwen3.8-27B-DFlash2 Q8_0 drafter, 2x Radeon AI PRO R9700 in ONE layer-split process, 262144 ctx / 1 slot, text only | 7770 |
 | Dankclaw | `frank-large` | halogen-flash-server 0.5.8 (rootless Quadlet) | peonist-ai Qwen3.8-Flash-Next w4b `.hgn` on the gfx1151 iGPU (Strix Halo LPDDR), 200192 ctx per conversation over one 400384-position KV pool / 2 slots, text only | 7770 |
 | Dankclaw | `frank-small` | mainline llama.cpp master 37b53fd (root systemd unit) | ISTA-DASLab Qwen3.8-27B-GSQ-RCO IQ3_S (`-mtp` + BF16 mmproj) on the RTX 3090, 131072 ctx / 1 slot, MTP speculative decoding, the fleet's vision endpoint | 7771 |
 
@@ -69,9 +69,9 @@ The planner maps changed paths to services; the remote runner restarts only
 the selected service and records its revision outside the checkout.
 Documentation-only changes are a no-op. LLM services are selected only by
 their own files — frank-large by `dankclaw/frank-large/quadlet/`, frank-small
-by its unit/env/serve trio in `scripts/` — so unrelated container changes
-cannot interrupt inference. Set `FRANK_SUDO_PW` for headless runs on a host
-without passwordless sudo; the value never enters the repo.
+and frank-medium by their unit/env/serve trio in `scripts/` — so unrelated
+container changes cannot interrupt inference. Set `FRANK_SUDO_PW` for headless
+runs on a host without passwordless sudo; the value never enters the repo.
 
 ### Deployment completeness (MANDATORY)
 
@@ -300,34 +300,41 @@ the halogen side refuses images with a named reason; 22,362 MiB of the card's
 
 Runbook: `serverAppConfigs-docs/LLM/frank-small-qwen38-gsq-rco.md`.
 
-### frank-medium — vllm-radiance MXFP4 (Dankserver, ACTIVE)
+### frank-medium — llama.cpp layer split + DFlash2 (Dankserver, ACTIVE)
 
-`stilldeadcode/vllm-radiance:0.9.3` plus the `ggz14/radiance-vllm-mxfp4`
-patches (pinned clone at `/home/dank/GitPackages/radiance-vllm-mxfp4`; its
-patches are applied at every container start), serving
-`just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq` with the
-`Qwen3.8-27B-DFlash2-FP8` block-diffusion drafter (`SPEC=7`).
+Mainline llama.cpp at **`fb27a525`** (a gfx1201 HIP build, `Root=`-less root
+systemd unit `scripts/frank-medium.service`) serving
+`ajgazin/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF` at **UD-Q8_K_XL**
+(29.29 GiB) with the **z-lab DFlash2** `Q8_0` drafter, in **one process across
+both R9700s**, `--split-mode layer`, 262144 ctx / 1 slot, text only.
 
-The deployment is **two single-GPU replicas behind an adaptive-routing proxy** —
-`frank-medium-a` (:7771, R9700 #1), `frank-medium-b` (:7772, R9700 #2) and
-`frank-medium-proxy` (:7770, the fleet endpoint), three rootless Quadlets
-under `dankserver/frank-medium/quadlet/`. Each replica is TP=1 at
-114688 context / 4 sequences with vLLM-profiled KV (`KV_MEM=0`).
+**Layer split is why both cards now serve ONE model.** Tensor parallelism is
+impossible (see the PCIe lesson below), and so is vLLM's pipeline parallelism —
+its stage handoff is an RCCL `send`/`recv` of the hidden state, the exact
+primitive that cannot be dispatched on card #2. llama.cpp's layer split needs
+no collective at all: layers are distributed in order and the boundary
+activation is a per-backend copy, once per forward pass. Measured on this host
+(the predecessor runtime): 1991 t/s prefill layer-split vs 272 tensor.
 
-**TP=2 across both cards cannot run on this host.** RCCL's collective kernels
-carry a hidden hostcall-buffer parameter, and ROCr refuses to dispatch them on
-a device without PCIe atomics; the second R9700 sits behind the chipset
-bridges `05:00.0`/`06:09.0`, which advertise `AtomicOpsCap: Routing-` — a
-read-only capability, so no software can enable atomics on that path. Not the
-kernel version (7.0.0-28/-30/-31 identical), not the image, not the RCCL
-build (a from-source rebuild fails identically). Putting both cards on CPU root
-ports restores TP=2: on this board that means an M.2→PCIe x4 riser into the
-free, CPU-attached M.2_2 slot, or a board whose CPU x16 bifurcates to x8/x8.
-`frank-medium.container` stays in the repo
-as the TP=2 target; the deploy handler does not install it.
+This is the box's **only** LLM runtime: the vllm-radiance replica pair and its
+routing proxy (`frank-medium-a/-b/-proxy`) were retired the same day they went
+live, with their units moved to `~/.AGENTTRASH/20260916-vllm-radiance-retire/`.
 
-Runbook: `serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md`.
-Prior runtime: [[frank-medium-rocmfpx]] (llama.cpp era, same day).
+Files: `scripts/build-llama-rdna4.sh` (builds the runtime, gfx1201 only, private
+ROCm 7.14 toolchain bundled beside the binary), `scripts/download-frank-medium-gguf.sh`
+(resumable, byte-verified payload), `scripts/frank-medium.env`,
+`scripts/serve-frank-medium.sh`, `scripts/frank-medium.service`. The deploy
+handler refuses to start a unit whose runtime or payload is missing, retires
+the vLLM quadlets, waits for 7770 to be released (the retired proxy held it) and
+gates on `/health`.
+
+The ref is pinned by two features and must not float: **DFlash2**
+(ggml-org/llama.cpp#27342) and **qwen35 MTP**. The ROCmFPX fork the endpoint
+used earlier that day has neither.
+
+Runbook: `serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md`.
+Retired runtimes: [[frank-medium-radiance-mxfp4]] (vLLM replicas, same day),
+[[frank-medium-rocmfpx]] (llama.cpp fork, earlier that day).
 
 ## Key Lessons Learned
 Durable operational lessons from the fleet's history. Full detail in the
@@ -364,14 +371,35 @@ linked runbooks.
   (`Pcie atomics not enabled, hostcall not supported` → `AQL dispatch
   failed!`), not from vLLM or RCCL. `AMD_LOG_LEVEL` costs one env var and
   prints the failing HIP call with its return code.
+- **The one software path that lets both cards serve ONE model: llama.cpp's
+  layer split.** vLLM cannot do it here in any of its parallelisms — TP needs
+  an allreduce, PP needs the same p2p primitive, and both die on the card that
+  cannot dispatch an RCCL kernel. `--split-mode layer` in llama.cpp needs no
+  collective: layers are distributed in order and only the boundary activation
+  crosses PCIe, once per forward pass. Measured here: 1991 t/s prefill /
+  18.8 tok/s decode layer-split, against 272 / 26.3 tensor (butterfly
+  fallback). That is what `frank-medium` runs — see
+  [[frank-medium-llamacpp-dflash2]].
 
 
 **ROCm / GPU**
 
-- **Layer split, never tensor, on the RCCL-less ROCmFPX builds.** Tensor
-  parallel falls back to a butterfly allreduce and collapses prefill ~7x
-  (measured 271.8 vs 1991 t/s). Pin the MTP draft head to ROCm1 — an
-  unpinned draft under tensor mode aborts the KV build.
+- **Layer split is the deployed shape on both multi-device llama.cpp services,
+  for two different reasons.** On Dankserver it is the *only* way both R9700s
+  serve one model (no collective can be dispatched on card #2). On a
+  RCCL-less fork build it is also the fast one: tensor parallel falls back to
+  a butterfly allreduce and collapses prefill ~7x (measured 271.8 vs 1991 t/s).
+  Pin the spec-decode drafter to one device — an unpinned draft under tensor
+  mode aborts the KV build, and on a layer split the natural home is the card
+  that ends up holding the target's last layers.
+- **Bundling a ROCm runtime next to a llama.cpp binary: `cp -r SRC DST/` is a
+  trap.** With a missing `DST`, cp creates `DST` as a copy of `SRC` and drops
+  one directory level. rocBLAS and hipBLASLt look for their device code at a
+  fixed depth (`../lib/rocblas/library`, `../lib/hipblaslt/library/<arch>`),
+  and the flattened copy fails at *request* time, not load time: the engine
+  reads the weights, answers `/health` and dies on the first completion with
+  `Cannot read .../TensileLibrary.dat: No such file or directory for GPU arch
+  : gfx1201`. Declare the destination, then copy the contents.
 - **mmap must stay on** for the big-PLE models: `--no-mmap` forces the
   ~50 GiB PLE table into anonymous memory → silent cgroup kill, dmesg is the
   only evidence. mmap pages ARE page cache. Never pin the PLE tensor with
@@ -492,7 +520,7 @@ Active:
 |---------|---------|
 | frank-large (halogen-flash-server, Dankclaw) | `serverAppConfigs-docs/LLM/halogen-flash-server.md` |
 | frank-small (llama.cpp Qwen3.8-27B GSQ-RCO, Dankclaw) | `serverAppConfigs-docs/LLM/frank-small-qwen38-gsq-rco.md` |
-| frank-medium (vllm-radiance MXFP4, Dankserver) | `serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md` |
+| frank-medium (llama.cpp + DFlash2, Dankserver) | `serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md` |
 | Dankclaw kernel posture (`amd_iommu=off`) | `serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md` |
 | Deployment flow + incidents | `serverAppConfigs-docs/LLM/deployment.md` |