docs: frank-medium is llama.cpp with DFlash2 on both cards

The runbook for the new deployment, plus the sweep of everything that still
described the vllm-radiance replica pair: the vault index and its ACTIVE
marker, the superseded banner on the radiance page, the pointer from the
ROCmFPX page, the homepage card, the Grafana dashboard description and the
observability page's scrape table.

The new page carries the two facts a future agent needs and the flags do not
say: llama.cpp's layer split is what lets BOTH cards serve ONE model without
any collective (vLLM's PP handoff is an RCCL send/recv, which is the primitive
this host cannot dispatch), and the rocBLAS Tensile-path trap that makes a
healthy-looking build fail on the first request.

serverAppConfigs-docs/Infrastructure/Observability.md

@@ -35,7 +35,7 @@ graph LR
 
 Recorded 2026-09-11, on the day the stack moved to Quadlet units. Everything in the table above is deployed, healthy and boot-persistent. Almost nothing is wired up beyond the default install:
 
-- **Grafana has a datasource and one dashboard as of 2026-09-16; before that it had neither.** `shared/grafana/provisioning/` is installed to `/opt/grafana/provisioning` by the deployer and read by the container: the Prometheus datasource (`http://prometheus:9090`, fixed uid `prometheus`) and the **LLM fleet — capacity and reuse** dashboard (KV pool fill, queue depth, prefix-cache reuse, latency p95, throughput) covering both LLM endpoints. Verified through the API rather than by eye — the datasource answers `vllm:kv_cache_usage_perc` for `frank-medium`, and `GET /api/search?type=dash-db` returns uid `llm-fleet`. One trap worth knowing: a provisioned dashboard does **not** appear in the legacy `dashboard` table on this Grafana version (it lives in unified storage behind `dashboard.grafana.app`), so the API is the way to check. Still outstanding: no dashboards beyond this one, no Loki queries, and only datasources+dashboards are provisioning-as-code.
+- **Grafana has a datasource and one dashboard as of 2026-09-16; before that it had neither.** `shared/grafana/provisioning/` is installed to `/opt/grafana/provisioning` by the deployer and read by the container: the Prometheus datasource (`http://prometheus:9090`, fixed uid `prometheus`) and the **LLM fleet — capacity and reuse** dashboard (KV pool fill, queue depth, prefix-cache reuse, latency p95, throughput) covering both LLM endpoints. Verified through the API rather than by eye — the datasource answers `llamacpp:` counters for `frank-medium`, and `GET /api/search?type=dash-db` returns uid `llm-fleet`. One trap worth knowing: a provisioned dashboard does **not** appear in the legacy `dashboard` table on this Grafana version (it lives in unified storage behind `dashboard.grafana.app`), so the API is the way to check. Still outstanding: no dashboards beyond this one, no Loki queries, and only datasources+dashboards are provisioning-as-code.
 - **Grafana's dashboard now also surfaces firing alerts.** The **Alerts firing (ALERTS)** panel graphs `ALERTS{alertstate="firing"}` for the LLM rules, because this stack has no other place where an alert becomes visible without opening Prometheus.
 - **Prometheus now evaluates rules; nothing delivers them (2026-09-16).** `shared/prometheus/rules/llm-fleet.yml` ships with the config (the deployer's `copy_prometheus_config` installs both — a rule file nothing loads evaluates nothing, and Prometheus reports no error for that) and holds six rules for the LLM fleet: pool saturation, request queueing, prefix-cache collapse, endpoint down, p95 latency, plus the `job:llm_prefix_cache_reuse:rate5m` recording rule the alert and the dashboard share. Verified with `promtool check rules` (SUCCESS, 6 rules) and `/api/v1/rules` (all six `health=ok`). **There is still no Alertmanager and no contact point anywhere**: a firing alert appears on Prometheus' `/alerts`, in the `ALERTS` metric and on the LLM fleet dashboard's alerts panel — and reaches a human only if one is looking.
 - **Loki is configured but unread.** Storage is filesystem with 30-day retention, `auth_enabled: false`, and the push path works; Grafana can now reach it in Explore, but no dashboard or alert consumes it.
@@ -85,21 +85,17 @@ The rule is the same as for every other service: the only off-host path is a gua
 | `grafana`, `loki` | `grafana:3000`, `loki:3100` | container names on `dankclaw_default` |
 | `comfyui` | `comfyui:8188` | UI probe, no native exporter |
 | `services` | `192.168.0.143:8081`, `:3005` | SearXNG, OpenWebUI up/down probes |
-| `frank-medium` | `192.168.0.235:7770` | vLLM on Dankserver — 418 `vllm:` series |
+| `frank-medium` | `192.168.0.235:7770` | llama.cpp on Dankserver — `llamacpp:` counters, including the spec-decode pair |
 | `frank-small` | `192.168.0.143:7771` | llama.cpp on Dankclaw's RTX 3090 — `llamacpp:` counters, including the spec-decode pair |
 
 The LLM endpoints were added on 2026-09-16, after a night spent diagnosing
 frank-medium by hand, and re-cut the same day when Dankclaw split into two
-endpoints. vLLM exports exactly what that diagnosis needed:
-`vllm:kv_cache_usage_perc` (how full the per-card pool is),
-`vllm:num_requests_waiting` (queue depth), the request-latency histograms, and
-the prefix-cache counters `vllm:prefix_cache_hits_total` /
-`vllm:prefix_cache_queries_total`, whose rates give the reuse rate that says
-whether a pinned conversation is being served warm or re-prefilled cold. What
-those numbers mean on this hardware is in [[frank-medium-radiance-mxfp4]];
-llama.cpp's counters (prompt and decode throughput, KV fill, and the
-`spec_decode_*` pair that proves MTP is running) are read against
-[[frank-small-qwen38-gsq-rco]].
+endpoints. The vLLM runtime behind that diagnosis is retired; frank-medium
+now runs mainline llama.cpp, and the endpoint exports llama.cpp's own
+counters — prompt and decode throughput, KV fill, and the `spec_decode_*`
+pair that proves the DFlash2 drafter is running. What those numbers mean on
+this hardware is in [[frank-medium-llamacpp-dflash2]]; the same counter
+family on frank-small is read against [[frank-small-qwen38-gsq-rco]].
 
 **`frank-large` is deliberately NOT scraped.** The halogen engine publishes
 `/health`, `/cache` and `/v1/*` and no Prometheus text endpoint at all, so a

serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md

@@ -0,0 +1,231 @@
+---
+type: configuration
+title: frank-medium llama.cpp + DFlash2 Runbook
+description: The ACTIVE frank-medium deployment since 2026-09-16 — mainline llama.cpp on Dankserver serving ajgazin/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP at UD-Q8_K_XL with the z-lab DFlash2 Q8_0 drafter, as ONE layer-split process across both Radeon AI PRO R9700 cards.
+tags:
+  - frank-medium
+  - dankserver
+  - llama-cpp
+  - dflash2
+  - qwen35
+  - gfx1201
+  - r9700
+  - layer-split
+timestamp: 2026-09-16
+---
+
+# frank-medium — llama.cpp + DFlash2
+
+The ACTIVE frank-medium deployment since 2026-09-16: **mainline llama.cpp**
+serving `ajgazin/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF` at `UD-Q8_K_XL`
+with the **z-lab DFlash2** block-diffusion drafter, as **one process across
+both Radeon AI PRO R9700 cards**, layer-split.
+
+It replaced the vllm-radiance MXFP4 replica pair + routing proxy
+([[frank-medium-radiance-mxfp4]], the record and the PCIe-atomics evidence) in
+the same day the pair went live. It is also the box's **only** LLM runtime: the
+replicas, the proxy and their unit files are retired.
+
+## Why one process and layer split
+
+The endpoint's whole history is one constraint: **the two cards cannot take
+part in an RCCL collective.** RCCL's kernels declare a hidden hostcall buffer,
+and ROCr refuses to supply it to a device without PCIe atomics; the second
+R9700 hangs off the B650 chipset (`AtomicOpsCap: Routing-`, read-only), so
+every collective that touches it dies with `hipErrorIllegalState`. That rules
+out tensor parallelism — and it rules out **vLLM's pipeline parallelism too**,
+whose stage handoff is a `send`/`recv` of the hidden state on the device
+backend (`vllm/distributed/parallel_state.py`), the very primitive that fails.
+The full matrix is in [[Dual-R9700-Cooperation]].
+
+llama.cpp's `--split-mode layer` needs **no collective at all**: layers are
+distributed across the cards in order and the boundary activation is a plain
+per-backend copy, once per forward pass. Measured on this host with the
+predecessor runtime (llama-bench, pp4096 / tg128, f16 KV, flash-attn on):
+
+| Split | pp4096 | tg128 |
+|---|---:|---:|
+| one R9700 (`-sm none`) | 1160.9 | 19.42 |
+| **two cards, layer (`-sm layer`)** | **1991.2** | **18.76** |
+| two cards, tensor (`-sm tensor`, no RCCL: butterfly fallback) | 271.8 | 26.26 |
+
+Tensor split buys ~40 % decode and loses 7× prefill. The workload here is
+prefill-heavy (agentic, long prompts, reasoning traces), so layer split is the
+shape — prefill nearly doubles and one model serves from both cards.
+
+## Hardware
+
+| Property | Value |
+|---|---|
+| GPUs | 2x AMD Radeon AI PRO R9700 (DID 0x7551), gfx1201 (Navi 48, RDNA4) |
+| VRAM | 34,208,743,424 B (31.86 GiB) each, as `rocm-smi` reports it |
+| Link | PCIe, no XGMI; card #2 behind the chipset, **no atomic routing** |
+| iGPU | Raphael (gfx1036) enumerates as HIP device 2 — hidden via `HIP_VISIBLE_DEVICES=0,1` |
+| CPU / RAM | Ryzen 7 7700X (8c/16t), 60 GiB RAM, 63 GiB swap |
+
+## Runtime
+
+Mainline llama.cpp at **`fb27a525`** (2026-09-16), built for **gfx1201 only** by
+`scripts/build-llama-rdna4.sh` against a private ROCm `7.14.0a20260624`
+toolchain (TheRock nightly) that is **bundled next to the binary** —
+`RPATH=$ORIGIN/../lib`, no `LD_LIBRARY_PATH`, nothing from the host's ROCm 7.1
+packages.
+
+Two features pin that ref, and neither is in the ROCmFPX fork the endpoint used
+earlier the same day (its tip, `c49ebdb`, predates both):
+
+- **DFlash2** (`ggml-org/llama.cpp#27342`): grouped dynamic depthwise
+  convolution plus the candidate selector, detected from the draft GGUF
+  (`selector_top_k > 0`) and served through the ordinary
+  `--spec-type draft-dflash` path.
+- **qwen35 MTP**: the `draft-mtp` implementation covers this architecture's
+  single trained head.
+
+Build with:
+
+```bash
+bash scripts/build-llama-rdna4.sh     # ~25 min on this box, JOBS=$(nproc)
+```
+
+> **Bundling trap, measured.** rocBLAS and hipBLASLt look for their device code
+> at a fixed depth under the binary's lib dir (`../lib/rocblas/library`,
+> `../lib/hipblaslt/library/<arch>`). `cp -r SRC DST/` with a missing `DST`
+> creates `DST` as a copy of `SRC` and **flattens it one level**, and the
+> failure is invisible at startup: the engine loads 29 GiB of weights, answers
+> `/health`, and dies on the **first request** with
+> `rocBLAS error: Cannot read .../lib/rocblas/library/TensileLibrary.dat: No
+> such file or directory for GPU arch : gfx1201`. The build script now declares
+> the destination and copies contents.
+
+## Model and drafter
+
+| File | Bytes | Role |
+|---|---:|---|
+| `Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-UD-Q8_K_XL.gguf` | 31,457,990,112 | target, 29.29 GiB |
+| `Qwen3.8-27B-DFlash2-Q8_0.gguf` | 2,056,414,816 | DFlash2 drafter, 1.92 GiB |
+
+Fetched by `scripts/download-frank-medium-gguf.sh` (resumable, byte-verified —
+a truncated 31 GB GGUF is otherwise a load-time crash).
+
+The target is Unsloth Dynamic 3.0 (per-tensor types from Unsloth's own quant of
+the same size, on Unsloth's importance matrix) of an abliterated Swift
+fine-tune: orcarouter's single refusal direction projected out of 131 tensors,
+vision tower untouched. **Text only here** (`--no-mmproj`); the fleet's image
+endpoint is frank-small.
+
+**The file also carries the model's MTP head** (`blk.64.nextn.*`). With
+`--spec-type draft-dflash` those tensors are deliberately **not** loaded — the
+log says `model has unused tensor blk.64.nextn.eh_proj.weight ... ignoring` —
+because DFlash2 drafts from its own 5-layer drafter. Switching (or chaining)
+`--spec-type draft-mtp,draft-dflash` is a one-line env change that pulls the
+head in; it is not enabled by default because the DFlash2 path is the measured
+one.
+
+## Deployment
+
+| File | Role |
+|---|---|
+| `scripts/frank-medium.env` | every tunable, with the sizing arithmetic in comments |
+| `scripts/serve-frank-medium.sh` | env-driven llama-server launcher; no arguments |
+| `scripts/frank-medium.service` | ROOT systemd unit, `User=dank`, `WantedBy=multi-user.target` |
+
+```bash
+./scripts/deploy.sh dankserver --service frank-medium
+```
+
+The handler verifies the runtime and payload exist (and refuses otherwise),
+retires the three vLLM quadlets into `~/.AGENTTRASH/20260916-vllm-radiance-retire/`,
+waits for **port 7770 to be released** (the retired proxy held it on the host
+network), relinks and restarts the unit, and gates on
+`http://127.0.0.1:7770/health`.
+
+Deployed flags:
+
+| Flag | Value | Why |
+|---|---|---|
+| `--split-mode` / `--device` / `--tensor-split` | `layer` / `ROCm0,ROCm1` / `1,1` | both cards, one model, no collective |
+| `--gpu-layers` / `--fit` | `999` / `off` | everything on the cards; auto-fit must not resize a context we set |
+| `--ctx-size` / `--parallel` | `131072` / `1` | one slot; `--parallel` **divides** the context |
+| `--flash-attn` / cache types | `on` / `f16` K and V | an agentic checkpoint is the wrong place to spend KV precision |
+| `--spec-type` / `--spec-draft-n-max` | `draft-dflash` / `7` | DFlash2 blocks are 8 tokens, so 7 drafts is the ceiling llama.cpp allows |
+| `--spec-draft-device` | `ROCm1` | the drafter rides the card that ends up holding the target's last layers |
+| `--temp` / `--top-p` / `--top-k` | `0.7` / `0.95` / `20` | the values the vLLM predecessor served on this endpoint |
+| `--reasoning-format` / `--reasoning-budget` | `deepseek` / `-1` | trace into `reasoning_content`, unlimited |
+| `--no-mmproj` | — | text only |
+
+## Verified serving behaviour (2026-09-16)
+
+- Load: `model loaded` ~29 s after start with a warm page cache, then
+  `listening on http://0.0.0.0:7770`. The drafter adds ~2.5 s.
+- Slot shape: `n_slots = 1, n_ctx_slot = 131072`; `/props` reports
+  `modalities: {vision: false}`, one slot, and the sampler defaults above.
+- DFlash2 engaged — the init line is the proof, not the flag:
+
+  ```
+  common_speculative_impl_draft_dflash: adding speculative implementation 'draft-dflash'
+  common_speculative_impl_draft_dflash: - n_max=7, n_min=0, p_min=0.00
+  common_speculative_impl_draft_dflash: - block_size=8, mask_token_id=248070, n_extract=5, sample_from_anchor=true
+  ```
+
+  `block_size=8` and `n_extract=5` are DFlash2's own metadata; the selector is
+  what distinguishes it from DFlash 1.
+- Both cards hold the model: at 131072 context, `rocm-smi` reports
+  **17.9 GiB on GPU 0 and 22.0 GiB on GPU 1** (the drafter and the attention
+  layers' KV sit on the second).
+- A real completion (60-token prompt, 43 output tokens incl. reasoning):
+  **28.3 tok/s**, `draft_n = 49`, `draft_n_accepted = 36` → **73 % acceptance**.
+  The reasoning trace arrives in `reasoning_content`.
+
+## Observability
+
+The endpoint exports Prometheus metrics under the **`llamacpp:`** namespace
+(`--metrics`), which is what the fleet's scrape and dashboard read — the
+`vllm:` series the predecessor exported no longer exist:
+
+| Need | Series |
+|---|---|
+| Throughput | `llamacpp:prompt_tokens_seconds`, `llamacpp:predicted_tokens_seconds` |
+| Queue | `llamacpp:requests_processing`, `llamacpp:requests_deferred` |
+| Cache reuse | `llamacpp:prompt_tokens_cached_total` vs `llamacpp:prompt_tokens_total` |
+| **Speculative decoding is live** | `llamacpp:spec_decode_num_drafts_total` and `llamacpp:spec_decode_num_accepted_tokens_total` (their ratio is the acceptance rate) |
+
+`spec_decode_num_drafts_total` staying at 0 is the one-line answer to "is the
+drafter actually running": the flags can all be right while the draft model
+failed to load.
+
+## Client contract
+
+OpenAI-compatible `/v1` on **7770**, alias **`frank-medium`**, streaming and
+`/health`, `/props`, `/metrics` as llama.cpp provides them. Reasoning traces
+are projected into `reasoning_content`. A request that sends no sampler fields
+gets the server defaults above; a client that sends its own wins. Text only.
+
+## Troubleshooting
+
+- **Loads, then dies on the first request with a rocBLAS Tensile error** — the
+  bundled device-library depth (see the bundling trap above); check
+  `build-rdna4-rocm714/lib/rocblas/library/` exists and is populated.
+- **`hipErrorNoDevice` / a card missing** — `HIP_VISIBLE_DEVICES` must be
+  `0,1`; the iGPU enumerates as device 2 and must stay hidden.
+- **Drafting not engaged** — read the journal for
+  `common_speculative_impl_draft_dflash`; a missing drafter file makes the
+  serve script exit before llama-server starts, and a non-DFlash2 drafter (or
+  an older llama.cpp) fails the load instead.
+- **Why is `blk.64.*` "unused"?** That is the model's own MTP head and it is
+  correct for `draft-dflash`. It is not a truncated download:
+  `bash scripts/download-frank-medium-gguf.sh --verify-only` proves the bytes.
+- **A rebuild segfaults at load** — the build is gfx1201-only on purpose; a
+  gfx1200 sibling binary links, loads the model and then segfaults on these
+  cards.
+
+## See Also
+
+- [[frank-medium-radiance-mxfp4]] — the retired vllm-radiance MXFP4 deployment
+  (replicas + proxy) and the PCIe-atomics evidence
+- [[frank-medium-rocmfpx]] — the llama.cpp/ROCmFPX deployment earlier on
+  2026-09-15, whose `--split-mode layer` measurements this service reuses
+- [[Dual-R9700-Cooperation]] — why the two cards cannot take part in a
+  collective, and every software escape that is closed
+- [[frank-small-qwen38-gsq-rco]] — the fleet's other llama.cpp endpoint (RTX
+  3090, MTP, the image path)
+- [[Dankserver System Specs]] — hardware inventory

serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md

@@ -16,6 +16,12 @@ tags:
 timestamp: 2026-09-16
 ---
 
+> **SUPERSEDED 2026-09-16.** This page is the record of the retired
+> vllm-radiance MXFP4 deployment. The ACTIVE frank-medium is mainline
+> llama.cpp with the DFlash2 drafter ([[frank-medium-llamacpp-dflash2]]);
+> the PCIe-atomics evidence in this page remains the reason the runtime is
+> layer-split llama.cpp rather than tensor-parallel vLLM.
+
 # frank-medium vllm-radiance MXFP4 Runbook
 
 The ACTIVE frank-medium deployment (since 2026-09-15/16): the

serverAppConfigs-docs/LLM/frank-medium-rocmfpx.md

@@ -16,7 +16,7 @@ timestamp: 2026-09-15
 > **PRIOR deployment (retired 2026-09-16).** This page documents the
 > llama.cpp era of `frank-medium`, which served the endpoint for part of
 > 2026-09-15 before the vllm-radiance MXFP4 stack replaced it. The ACTIVE
-> deployment is [[frank-medium-radiance-mxfp4]]; this page is kept as the
+> deployment is [[frank-medium-llamacpp-dflash2]]; this page is kept as the
 > record of the runtime it replaced.
 
 # frank-medium ROCmFPX Runbook

serverAppConfigs-docs/index.md

@@ -114,12 +114,19 @@ The LLM inference fleet on the two hosts. Full index:
 
 - [[deployment]] — unified deploy and update process for both servers
 - [[frank-redo-2026-08-17]] — the 2026-08-17 topology change
-- [[frank-medium-radiance-mxfp4]] — the ACTIVE frank-medium runbook: the
-  vllm-radiance MXFP4 stack serving
-  just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq on Dankserver's 2x Radeon
-  AI PRO R9700 as two single-GPU replicas behind a round-robin proxy, plus
-  the PCIe-atomics evidence that rules out TP=2 on this host (since
-  2026-09-15/16)
+- [[frank-medium-llamacpp-dflash2]] — the ACTIVE frank-medium runbook since
+  2026-09-16: mainline llama.cpp (gfx1201 HIP build) serving
+  ajgazin/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF at UD-Q8_K_XL with
+  the DFlash2 drafter (z-lab/Qwen3.8-27B-DFlash2-GGUF at Q8_0) on
+  Dankserver's 2x Radeon AI PRO R9700 as one layer-split process behind the
+  :7770 endpoint — no tensor parallelism, because the second card cannot
+  dispatch an RCCL collective
+- [[frank-medium-radiance-mxfp4]] — RETIRED 2026-09-16: the vllm-radiance
+  MXFP4 stack serving just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq on
+  Dankserver's 2x Radeon AI PRO R9700 as two single-GPU replicas behind a
+  round-robin proxy; kept as the record of the deployment
+  [[frank-medium-llamacpp-dflash2]] replaced, plus the PCIe-atomics evidence
+  that rules out TP=2 on this host (since 2026-09-15/16)
 - [[Dual-R9700-Cooperation]] — why the two R9700s cannot run tensor-parallel
   together: the PCIe-atomics gate quoted from ROCclr, the measured RCCL matrix
   (collectives *and* P2P `send`/`recv` both fail on card #2, which rules out
@@ -157,6 +164,11 @@ The LLM inference fleet on the two hosts. Full index:
 - [[ciru-activefpx-promptforge]] — the RETIRED v3 CIRU PromptForge runtime on Dankclaw (superseded 2026-08-24)
 - [[kingjones-strix-mtp]] — the kingjones Qwen3.8-27B ROCmFP4 STRIX MTP model on Dankclaw (active)
 - [[dealign-ornith-crack]] — the retired Ornith-1.5-35B-A3B CRACK model on Dankserver (retired 2026-09-02)
+- [[frank-small-qwen38-gsq-rco]] — the ACTIVE frank-small runbook since
+  2026-09-16: mainline llama.cpp serving ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-
+  GGUF at IQ3_S (own MTP head + BF16 vision projector) on Dankclaw's
+  RTX 3090, port 7771 — the 3090 freed when halogen took the iGPU back for
+  frank-large
 - [[ikllama-ornith-dankserver]] — the retired ik_llama.cpp runtime for the old frank-small (q8_0 + Hadamard KV cache, split-mode none; 2026-08-26)
 - [[freetoken-frank-small]] — RETIRED 2026-09-15: the FreeToken/CUDA frank-small runbook, retired with the RTX 3090
 - [[kairic-edge]] — the retired Kairic Edge IU4 experiment (2026-08-22)