docs: frank-medium runbook, fleet topology and client cutover

New runbook page serverAppConfigs-docs/LLM/frank-medium-rocmfpx.md records the hardware swap, the self-contained gfx1201 build, the measured layer-vs- tensor decision (llama-bench multi-device syntax gotcha included), the tensor-mode draft abort, served-path numbers, and the post-cleanup host ROCm posture. AGENTS.md, README, docs indexes, Dankserver System Specs, deployment/Hermes/OpenWebUI/Plex/Stack pages follow the rename; freetoken-frank-small is marked RETIRED and kingjones-q8agent SUPERSEDED. Work log entry added.
2026-09-15 19:27:04 -07:00
parent d53962fec5
commit d4024a9180
15 changed files with 426 additions and 80 deletions
@@ -46,7 +46,7 @@ graph LR
 ./scripts/deploy.sh dankserver
 ./scripts/deploy.sh frankbook-pro --service qbittorrent   # podman-machine target
 ./scripts/deploy.sh dankclaw --service searxng
-./scripts/deploy.sh dankserver --service frank-small
+./scripts/deploy.sh dankserver --service frank-medium
 ./scripts/deploy.sh dankclaw --dry-run
 ```

@@ -63,7 +63,7 @@ service on the other host.
 | Dankclaw compose services | `podman-compose -f dankclaw/compose.yaml up -d --no-deps NAME` | Recreates only `NAME`; Caddy is handled separately because it fronts both hosts. |
 | SearXNG, Gluetun, Valkey, OpenWebUI, Pi-hole, Gitea database/server | user Quadlet + `systemctl --user` | Installs the selected unit and restarts only the required dependency-coupled units. Gitea's web proxy is the canonical Caddy compose service. |
 | `frank-large` | systemd + `systemctl` | Relinks and restarts only the large inference service; waits up to 20 minutes for `/health`. |
-| `frank-small` | systemd + `systemctl` | Relinks and restarts only the small inference service; waits up to 20 minutes for `/health`. |
+| `frank-medium` | systemd + `systemctl` | Retires the legacy frank-small unit on first deploy, then relinks and restarts only the inference service; waits up to 20 minutes for `/health`. |
 | Dankserver media stack (RomM trio, Jellyfin, Calibre-Web, Copyparty) | user Quadlet + `systemctl --user` | Installs `dankserver/<svc>/quadlet/*.container`, restarts the generated unit. The first cutover per service is idempotent: it retires the compose-era bare container, rescues any anonymous volume into the data root, and merge-moves `/opt/<svc>` to `/home/dank/Containers/<svc>` (mapped-uid roots move without chown). Steady state is restart-only. |

 The shared chat template maps to the inference service on the selected target.
@@ -17,7 +17,7 @@ stack. Its LLM inference endpoint `frank-large` (`:7770`) is a ROCmFPX
 llama.cpp fork serving the kingjones Qwen3.8-Flash-Next-ROCmFP4-STRIX GGUF on
 the card's pinned runtime; an NPU endpoint (`frank-npu`, `:7771`,
 FLM v1.0.1) serves sight+sound models. The completion endpoint
-`frank-small` (FreeToken, `:7770`) runs on Dankserver (192.168.0.235).
+`frank-medium` (ROCmFPX llama.cpp, `:7770`) runs on Dankserver (192.168.0.235).

 ```mermaid
 graph LR
@@ -50,7 +50,7 @@ graph LR
 | Pi-hole | `pihole.franksdank.cloud` (basicauth edge, not the guard chain) | admin UI `8083`; DNS on `:53` LAN-wide | [[Pi-hole]] |
 | frank-large (host service) | — | `7770` | [[deployment]] |
 | frank-npu (host service) | — | `7771` (`/v1/version` health) | [[frank-npu]] |
-| frank-small (host service, Dankserver) | — | `7770` | [[deployment]] |
+| frank-medium (host service, Dankserver) | — | `7770` | [[frank-medium-rocmfpx]] |

 The LLM rows are bare-metal systemd services, not containers. This
 repo owns their scripts, units, and knowledge base (see
@@ -58,8 +58,8 @@ repo owns their scripts, units, and knowledge base (see
 `frank-large` runs the kingjones Qwen3.8-Flash-Next-ROCmFP4-STRIX GGUF
 on the card's pinned `kingjones30/ROCmFPX` llama.cpp build;
 `frank-npu` runs FLM v1.0.1 on the XDNA2 NPU;
-`frank-small` runs FreeToken v0.1.2 serving
-Huihui-Qwen3.6-35B-A3B-abliterated-FP8 on Dankserver.
+`frank-medium` runs charlie12345/ROCmFPX llama.cpp serving
+kingjones777/Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF on Dankserver.

 ## Boot Persistence

@@ -39,7 +39,7 @@ is not durable. The tracked, sanitized configuration baseline is
 - Hermes has two named local providers: `dankclaw-large` reaches
  `http://192.168.0.143:7770/v1` and exposes `frank-large`;
  `dankserver-small` reaches `http://192.168.0.235:7770/v1` and exposes
-  `frank-small`.
+  `frank-medium` (renamed from `frank-small` on 2026-09-15).
 - The live `config.yaml` contains secrets and user state and is not committed;
  the tracked example is the sanitized backup of its non-secret settings.

@@ -28,7 +28,7 @@ host; `frank-large` moved from `:52625` to `:7770` on 2026-08-22.
 | Endpoint | Model name | Host | Capability |
 |----------|-----------|------|------------|
 | `:7770` (ROCmFPX llama.cpp fork) | `frank-large` | Dankclaw | completion |
-| `:7770` (FreeToken v0.1.2) | `frank-small` | Dankserver | completion, multimodal |
+| `:7770` (ROCmFPX llama.cpp fork) | `frank-medium` | Dankserver | completion |

 The default connection points at the on-host endpoint (`frank-large`).
 To add the other backend, open **Admin → Connections** in the UI and point an
@@ -21,7 +21,7 @@ public traffic can be fronted by Caddy like every other vhost.
 | Identity | fleet rootless model — container root maps to the `dank` user (1000:1000); the deployer chowns the data root on first cutover |
 | Port | `32400/tcp` published on `0.0.0.0` — zero client reconfiguration |
 | Network | `dankserver_appnet`, alias `plex`; Tautulli reaches it at `http://plex:32400` |
-| GPU | none by decision — CPU decode only; the RTX 3090 stays dedicated to `frank-small` |
+| GPU | none by decision — CPU decode only; the two R9700s stay dedicated to `frank-medium` |
 | Deploy | `./scripts/deploy.sh dankserver --service plex` |
 | Docs | https://support.plex.tv |

@@ -1,8 +1,24 @@
+---
+type: configuration
+title: Dankserver System Specs
+description: Dankserver hardware and OS state since the 2026-09-15 GPU swap (2x AMD Radeon AI PRO R9700, gfx1201) serving frank-medium.
+tags:
+  - dankserver
+  - hardware
+  - r9700
+  - gfx1201
+  - rocm
+timestamp: 2026-09-15
+---
+
 # Dankserver System Specs

 ## Server Overview

-Dankserver is an NVIDIA GPU server for vLLM inference. The host runs Ubuntu 26.04 (Resolute Raccoon) with an RTX 3090 and 62 GB of system RAM.
+Dankserver is a media/library server and the `frank-medium` LLM host. The
+host runs Ubuntu 26.04 (Resolute Raccoon). The RTX 3090 that served the
+FreeToken/CUDA `frank-small` era was removed on 2026-09-15 and replaced with
+two AMD Radeon AI PRO R9700 cards.

 ## Hardware Specifications

@@ -13,42 +29,49 @@ Dankserver is an NVIDIA GPU server for vLLM inference. The host runs Ubuntu 26.0
 | Architecture | x86_64 |
 | Vendor | AuthenticAMD |
 | Model | AMD Ryzen 7 7700X |
-| Cores | 8 (1 socket) |
-| NUMA Nodes | 1 (node 0, 63503 MB) |
+| Cores | 8 cores / 8 threads (1 socket, SMT off) |
+| NUMA Nodes | 1 (node 0) |

-### GPU
+### GPUs

 | Property | Value |
 |----------|-------|
-| Model | NVIDIA GeForce RTX 3090 |
-| VRAM | 24576 MiB (24 GB) |
-| Driver | 595.84 |
-| CUDA Version | 13.2 |
-| CUDA Toolkit | 12.0 (nvcc) |
+| Model | 2x AMD Radeon AI PRO R9700 (Navi 48, DID 0x7551, rev c0) |
+| Architecture | gfx1201 (RDNA4) |
+| VRAM | 32,624 MiB each |
+| Link | PCIe, no XGMI; different IOMMU groups |
+| PCI | 03:00.0 and 0b:00.0 |
+| iGPU | Raphael (gfx1036) at 10:00.0 — enumerates as ROCm device 2, hidden from HIP |
+
+The RTX 3090 is gone; stale `nvidia-smi` binaries and the disabled NVIDIA
+apt lists remain on disk but no NVIDIA PCI device exists.

 ### Memory

 | Property | Value |
 |----------|-------|
-| System RAM | 62 GiB |
-| Swap | 127 GiB |
+| System RAM | 60 GiB |
+| Swap | 63 GiB |
 | /dev/shm | 32 GiB |

 ### Storage

 | Filesystem | Size | Used | Avail | Use% |
 |------------|------|------|-------|------|
-| / (nvme0n1p2) | 915G | 729G | 140G | 84% |
+| / (nvme0n1p2) | 915G | ~640G | ~280G | ~70% |
 | /mnt/entropy (dm-0) | 91T | 86T | 910G | 99% |

+Models live under `/home/dank/models`; the ROCmFPX build tree under
+`/home/dank/GitPackages/charlie-ROCmFPX` with its self-contained ROCm 7.14
+toolchain under `~/.local/share/rocm-7.14.0a20260624`.
+
 ### Network

 | Interface | Type | Speed | IP | Notes |
 |-----------|------|-------|----|-------|
-| eno1 | onboard | 2500 Mb/s (2.5 GbE) | 192.168.0.235 (DHCP) | RPC worker endpoint (port 50052). Default route via 192.168.0.1. |
-| enxe8ea6ab7ebca | USB-C class adapter | 2500 Mb/s | 192.168.0.252 | UNPLUGGED 2026-08-17. Was idle (unused by the RPC path). |
+| eno1 | onboard | 2500 Mb/s (2.5 GbE) | 192.168.0.235 (DHCP) | LLM endpoint (port 7770). Default route via 192.168.0.1. |
+| enxe8ea6ab7ebca | USB-C class adapter | 2500 Mb/s | 192.168.0.252 | Unplugged since 2026-08-17. |
 | wlp8s0 | WiFi | - | - | Unused (carrier down). |
-| docker0 | bridge | - | 172.17.0.1 | Down; internal docker traffic only. |

 ### OS and Software

@@ -57,21 +80,35 @@ Dankserver is an NVIDIA GPU server for vLLM inference. The host runs Ubuntu 26.0
 | OS | Ubuntu 26.04 LTS (Resolute Raccoon) |
 | Kernel | 7.0.0-30-generic |
 | Python | 3.14.4 |
+| ROCm userspace | Ubuntu 7.1 packages in /usr (hipcc 7.1.1, rocminfo, rocm-smi 7.1.1) |
+| Serving stack | charlie12345/ROCmFPX llama.cpp with its own ROCm 7.14 toolchain (self-contained, RPATH-bundled) |
+
+The stale AMD-repo ROCm 6.3.3 packages (167, `~24.04`-suffixed) that used to
+hijack `libhsa-runtime64` were purged on 2026-09-15 — see
+[[frank-medium-rocmfpx]].

 ### Active Services

-The active service on Dankserver is `frank-small.service`: the llama.cpp Ornith-1.5-35B-A3B CRACK (Q6_K) on port 7770, with expert tensors selected by `--override-tensor`; every full-attention block plus blocks 35–40 remain on CUDA. The earlier PP worker (`frank-pp-worker.service`) was retired with the 2026-08-17 redo. See [[deployment]].
+The active inference service is `frank-medium.service`: charlie12345/ROCmFPX
+llama-server on port 7770, layer-split across both R9700s with the MTP draft
+head on ROCm1. See [[frank-medium-rocmfpx]]. The former
+`frank-small.service` (FreeToken/CUDA) was retired with the GPU swap; its
+unit is archived in `~/.AGENTTRASH/20260915-frank-small-retire/host-units/`.
+The earlier PP worker (`frank-pp-worker.service`) was retired with the
+2026-08-17 redo. See [[deployment]].

 ## Hardware Relevance

- 24 GB VRAM hosts the Q6_K non-expert tensors and the full-attention/final-layer expert placement. The context pool contains 491520 f16 cells across three 163840-token slots. The service uses ngram-mod only and `ubatch=1024`. Verify actual headroom after each model restart with `nvidia-smi` and the service journal.
- 62 GB system RAM holds the remaining CPU-resident expert tensors with room to spare.
- 127 GB swap is available for CPU KV cache paging if GPU KV offload is disabled.
- Root disk has approximately 140 GB free. Store the GGUF models under `/home/dank/models` (about 32 GB total).
+- 2x 32 GB VRAM: the 26.28 GiB Q8_0_ROCMFPX_AGENT checkpoint plus MTP draft
+  and a 131,072-token f16 KV cache fit comfortably; layer split gives each
+  card about half the layers and its own KV share.
+- 60 GB system RAM is mostly headroom; nothing in the serving path is
+  CPU-resident.
+- The iGPU is deliberately unused: `HIP_VISIBLE_DEVICES=0,1` keeps work off
+  the shared-memory Raphael device.

 ## See Also

- [[VLLM Overview]]
- [[Hybrid CPU GPU Processing Dankserver]]
- [[RAM CPU Offloading in VLLM]]
- [[Running Qwen3.6 35B A3B FP8 on Dankserver]]
+- [[frank-medium-rocmfpx]] — the active serving runbook
+- [[deployment]] — the deploy flow
+- [[Shared Memory Architecture Dankclaw]] — for contrast, Dankclaw's unified memory
@@ -19,7 +19,7 @@ SearXNG, Caddy, or another container service does **not** restart inference.

 | Target | Host | Service | Port |
 |---|---|---|---:|
-| Dankserver | `dank@dankserver` | `frank-small` | 7770 |
+| Dankserver | `dank@dankserver` | `frank-medium` | 7770 |
 | Dankclaw | `dank@dankclaw` | `frank-large` | 7770 |
 | Dankclaw | `dank@dankclaw` | `frank-npu` (FLM on the XDNA2 NPU) | 7771 |

@@ -39,7 +39,7 @@ graph LR
 Run the unified deployer from the repository root:

 ```bash
-./scripts/deploy.sh dankserver --service frank-small
+./scripts/deploy.sh dankserver --service frank-medium
 ./scripts/deploy.sh dankclaw --service frank-large
 ./scripts/deploy.sh dankclaw --dry-run
 ```
@@ -47,8 +47,8 @@ Run the unified deployer from the repository root:
 Without `--service`, `scripts/deploy-plan.py` maps changed files to the service
 on the selected host. The following files select the corresponding service:

- `scripts/frank-small.env`, `frank-small.service`, and
-  `serve-frank-small.sh` select `frank-small` on Dankserver.
+- `scripts/frank-medium.env`, `frank-medium.service`, and
+  `serve-frank-medium.sh` select `frank-medium` on Dankserver.
 - `scripts/frank-large.env`, `frank-large.service`, and
  `serve-frank-large.sh` select `frank-large` on Dankclaw.
 - `scripts/frank-npu.env`, `frank-npu.service`,
@@ -70,7 +70,7 @@ slow. A failed handler does not advance its per-service state.
 Both units point into the stable target checkout:

 ```text
-/home/dank/serverAppConfigs/scripts/frank-small.service
+/home/dank/serverAppConfigs/scripts/frank-medium.service
 /home/dank/serverAppConfigs/scripts/frank-large.service
 ```

@@ -105,14 +105,14 @@ change in that service's inputs.

 If the LLM handler fails after the target pull:

-1. Inspect `journalctl -u frank-small -n 200` or
+1. Inspect `sudo journalctl -u frank-medium -n 200` or
   `journalctl -u frank-large -n 200`.
 2. Confirm the model files, custom runtime binary, and chat template are
   readable by `dank`.
 3. Retry only the affected service:

   ```bash
-   ./scripts/deploy.sh dankserver --service frank-small
+   ./scripts/deploy.sh dankserver --service frank-medium
   # or
   ./scripts/deploy.sh dankclaw --service frank-large
   ```
@@ -0,0 +1,214 @@
+---
+type: configuration
+title: frank-medium ROCmFPX Runbook
+description: charlie12345/ROCmFPX llama.cpp serving kingjones777/Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF as frank-medium on Dankserver (2x AMD Radeon AI PRO R9700, gfx1201).
+tags:
+  - frank-medium
+  - dankserver
+  - rocmfpx
+  - llama-cpp
+  - mtp
+  - gfx1201
+  - r9700
+timestamp: 2026-09-15
+---
+
+# frank-medium ROCmFPX Runbook
+
+`frank-medium` is the Dankserver LLM endpoint (port 7770) since 2026-09-15,
+when the RTX 3090 left the box and two **AMD Radeon AI PRO R9700** cards
+(Navi 48, RDNA4, `gfx1201`, 32,624 MiB each, PCIe 5.0 x16) took its place.
+It replaced the retired FreeToken/CUDA `frank-small`
+([[freetoken-frank-small]]), which cannot run on this hardware at all.
+
+Stack: **charlie12345/ROCmFPX llama.cpp** (`main`, `c49ebdb`, 2026-08-22)
+serving **kingjones777/Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF** with the
+companion MTP draft head. Stock llama.cpp cannot load this GGUF —
+`Q8_0_ROCMFPX_AGENT` is ftype 115 / `GGML_TYPE_Q8_0_ROCMFPX` (type 103) and
+exists only in the fork.
+
+## Hardware
+
+| Property | Value |
+|----------|-------|
+| GPUs | 2x AMD Radeon AI PRO R9700 (DID 0x7551), rev c0 |
+| Architecture | gfx1201 (Navi 48, RDNA4) |
+| VRAM | 32,624 MiB each (rocm-smi) |
+| Link | PCIe, no XGMI; cards sit in different IOMMU groups |
+| iGPU | Raphael (gfx1036) enumerates as ROCm device 2 — hidden from HIP |
+| CPU / RAM | Ryzen 7 7700X (8c/8t), 60 GiB RAM, 63 GiB swap |
+| Kernel driver | in-tree `amdgpu`; no new driver needed |
+
+`HIP_VISIBLE_DEVICES=0,1` hides the iGPU so nothing can drift onto shared
+system RAM. Devices are addressed as `ROCm0,ROCm1` (llama-server) or
+`ROCm0/ROCm1` (llama-bench multi-device syntax — see the bench gotcha below).
+
+## Runtime build (self-contained ROCm 7.14)
+
+The host ROCm install is quarantined rather than repaired: the build uses the
+fork's self-contained toolchain, so the binary carries its own
+`libamdhip64`/`libhsa-runtime64`/`rocblas`/`hipblaslt` with
+`RPATH=$ORIGIN/../lib` and never touches system ROCm.
+
+```bash
+cd /home/dank/GitPackages/charlie-ROCmFPX
+env JOBS=8 ./scripts/build-rocmfp4-rocm714-local.sh
+# -> build-rdna4-rocm714/bin/llama-server
+```
+
+The script downloads the gfx120X-all TheRock ROCm 7.14 nightly tarball into
+`~/.local/share/rocm-7.14.0a20260624` (~17 GiB extracted; tarball kept next to
+the build tree) and builds `gfx1200;gfx1201`.
+
+**gfx1201 is not interchangeable with gfx1200** — the fork's
+`docs/BUILD-AMD-ARCHITECTURES.md` warns a wrong-sibling build links, loads the
+model, then segfaults. `scripts/rocmfpx-hip-arch.sh` auto-detects and refuses.
+
+## Model
+
+| File | Size | Role |
+|------|------|------|
+| `Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf` | 28,214,425,952 B (26.28 GiB, 8.39 bpw) | target |
+| `mtp-Qwen3.8-27B-Q4_0.gguf` | 1,680,271,648 B | MTP draft head |
+| `mmproj-Qwen3.8-27B-BF16.gguf` | 931,145,888 B | vision tower — **unused** (`--no-mmproj`) |
+
+The AGENT routing keeps 13 of the 17 `*.output.weight` tensors at full Q8_0
+(12 more than the plain `Q8_0_ROCMFPX` build), which raises MTP draft
+acceptance (0.953 vs 0.911 on the author's gfx1151 box). Take this variant
+only when serving with MTP. Fetch with `scripts/download-q8agent.sh`
+(resumable, size-verified; run `--verify-only` to re-check).
+
+The model card documents 131,072 context; llama.cpp reports
+`n_ctx_train (262144)` from the GGUF, but the author never tested beyond 131k,
+so the deployment uses 131,072.
+
+## Parallelism: layer split, not tensor (measured)
+
+llama-bench on the host (pp4096 / tg128, same at depth 32768, f16 KV,
+flash-attn on, 3 reps):
+
+| Split | pp4096 | tg128 | pp4096 @d32768 | tg128 @d32768 |
+|-------|-------:|------:|---------------:|--------------:|
+| 1x R9700 (`-sm none`) | 1160.9 | 19.42 | ~512 | ~18.4 |
+| **2x layer (`-sm layer`)** | **1991.2** | 18.76 | **859.1** | 17.64 |
+| 2x tensor (`-sm tensor`) | 271.8 | 26.26 | 228.6 | 25.09 |
+
+Tensor parallel wins decode by ~40 % but collapses prefill **7x**: this build
+ships without RCCL, so llama.cpp logs
+`internal AllReduce init failed (n_devices != 2?); falling back to meta-backend
+butterfly` and every layer pays a slow cross-card reduction. The workload is
+prefill-heavy (agentic, long prompts, reasoning traces), so the operator
+switched to layer split on 2026-09-15. Revisit if the fork ships a working
+RCCL tensor path (`-DGGML_HIP_RCCL=ON`).
+
+Served-path comparison with MTP on (real completions): layer 35.7–46.2 tok/s
+vs tensor 28.7 tok/s — layer wins decode too once the draft head is in play.
+
+### llama-bench multi-device gotcha
+
+`llama-bench` and `llama-server` disagree on device syntax. In llama-bench,
+`-dev ROCm0,ROCm1` means *two separate single-card benches* (the value is
+split on `,` into separate device sets, one model load each); a single split
+bench needs `-dev ROCm0/ROCm1`. The invalid form silently produces
+single-card numbers in a table that looks like a split test.
+
+### Tensor-mode draft abort (if tensor is ever revisited)
+
+Under `--split-mode tensor` the MTP draft head must be pinned to exactly one
+device (`--spec-draft-device ROCm0`); with it empty the draft joins the
+meta-device KV allocation and the server aborts while building the KV cache:
+
+```
+ggml/src/ggml-backend.cpp:119: GGML_ASSERT(buffer) failed
+  <- ggml_backend_meta_alloc_ctx_tensors_from_buft <- llama_kv_cache ctor
+```
+
+## Deployment
+
+| File | Role |
+|------|------|
+| `scripts/frank-medium.env` | all tunables (sampler, split, draft, context) |
+| `scripts/serve-frank-medium.sh` | env-driven llama-server launcher |
+| `scripts/frank-medium.service` | root systemd unit (relinked into /etc/systemd/system) |
+| `scripts/download-q8agent.sh` | model fetch + size verification |
+
+```bash
+./scripts/deploy.sh dankserver --service frank-medium
+```
+
+First deploy (2026-09-15) also ran `retire_root_llm_small()` in
+`scripts/deploy-remote.sh`: the CUDA-era `frank-small` unit was stopped,
+disabled and moved to `~/.AGENTTRASH/20260915-frank-small-retire/host-units`
+(the host's `systemctl disable` has been observed to delete unit files,
+2026-08-26 incident, so both outcomes are tolerated).
+
+Deployed flags (see `scripts/frank-medium.env` for the full set):
+`--split-mode layer --device ROCm0,ROCm1 --gpu-layers 999 --fit off`,
+`--ctx-size 131072 --parallel 1`, f16 K/V, flash-attn on,
+`--spec-type draft-mtp` (n_max 6, p_min 0.6) with the draft pinned to
+`ROCm1` — the card holding the target model's last layers, where the final
+hidden state is produced. Reasoning on, deepseek format, unlimited budget;
+sampler defaults are the checkpoint's instruct table (temp 0.7, top-p 0.8,
+top-k 20, min-p 0, presence 1.5). Text only — vision stays on
+`frank-large` (Dankclaw).
+
+## Verified serving behavior (2026-09-15)
+
+- Health `{"status":"ok"}` ~18 s after start; model load ~17 s (warm page cache).
+- MTP engaged: draft acceptance 60–69 % across prompts
+  (`timings.draft_n` / `draft_n_accepted` in every chat response).
+- Decode 35.7–46.2 tok/s on a 200-token budget; cold prompt 949 tokens
+  prefilled in 2036 ms (~466 tok/s at ubatch 512).
+- **Prefix caching works with the draft head**: the model card's
+  `mtp-prompt-cache-fix.patch` defect (draft head disables prefix caching) is
+  already fixed in `c49ebdb` — the speculative boundary rides inside the
+  context checkpoint (`common_prompt_checkpoint::data_spec`). Measured: turn 2
+  of a conversation reprocessed 22 tokens (`cache_n` 945) instead of 949.
+- Quirk: the request-level `"speculative": {"n_max": 0}` override does **not**
+  disable drafting (still drafts and accepts). To serve without MTP, restart
+  with `SPEC_TYPE=none`.
+
+## Client contract
+
+OpenAI-compatible `/v1` on port 7770, alias `frank-medium`. Reasoning traces
+are projected into `reasoning_content` (deepseek format); requests that send
+no sampler fields get the server's CLI defaults, not greedy. Text only.
+Harness routing after the cutover: blankpad `smol`/`tiny`/`commit`/`default`
+→ `frank-medium`, Hermes `title_generation` (and its other dankserver-small
+routes) → `frank-medium`, vision stays `frank-large`.
+
+## Host ROCm posture (post-cleanup)
+
+Historically the host had a broken mix: AMD-repo ROCm 6.3.3 in
+`/opt/rocm-6.3.3` hijacked `libhsa-runtime64` via
+`/etc/ld.so.conf.d/10-rocm-opencl.conf`, breaking Ubuntu's ROCm 7.1 userspace
+(`undefined reference to 'hsa_amd_memory_get_preferred_copy_engine'`).
+On 2026-09-15 the stale 6.3.3 packages (~24.04-suffixed, 167 pkgs) were
+purged, the hijack conf and the `repo.radeon.com` apt pin moved to
+`~/.AGENTTRASH/20260915-rocm633-cleanup/`, and `ldconfig` now resolves only
+Ubuntu's ROCm 7.1 libs. `rocm-smi` is Ubuntu's own 7.1.1 package. The
+self-contained toolchain in `~/.local/share/rocm-7.14.0a20260624` does not
+depend on any of it.
+
+## Troubleshooting
+
+- **Wrong arch build**: a gfx1200 binary on this card links, loads, then
+  segfaults. Rebuild with the pinned script and check the "Build complete for
+  gfx1200;gfx1201" line plus the emitted code objects.
+- **Crash at load**: read the journal (`sudo journalctl -u frank-medium -n 100`),
+  then run the serve script directly; the bundled libs make the script and the
+  raw binary behave identically. The known abort shape is the tensor-mode
+  draft KV abort above.
+- **MTP "not engaged"**: check `timings.draft_n` in a response; zero means the
+  draft head is not loaded (look for `common_speculative_init: adding
+  speculative implementation 'draft-mtp'` in the journal).
+- **Model files missing/truncated**: `bash scripts/download-q8agent.sh
+  --verify-only`.
+
+## See Also
+
+- [[freetoken-frank-small]] — the retired predecessor (FreeToken/CUDA on the 3090)
+- [[kingjones-q8agent-rocmfpx]] — the gfx1151 investigation of this checkpoint family
+- [[Llama Cpp Runtimes]] — where the ROCmFPX fork sits among the runtimes
+- [[Dankserver System Specs]] — current hardware
+- [[deployment]] — the deploy flow
@@ -1,7 +1,7 @@
 ---
 type: configuration
 title: FreeToken frank-small Runbook
-description: FreeToken v0.1.2 serving batsclamp/Huihui-Qwen3.6-35B-A3B-abliterated-FP8 as frank-small on Dankserver (RTX 3090).
+description: RETIRED 2026-09-15 - FreeToken v0.1.2 serving batsclamp/Huihui-Qwen3.6-35B-A3B-abliterated-FP8 as frank-small on Dankserver (RTX 3090 era).
 tags:
  - freetoken
  - dankserver
@@ -14,6 +14,13 @@ timestamp: 2026-08-30

 # FreeToken frank-small Runbook

+> **RETIRED 2026-09-15.** The RTX 3090 left Dankserver and the successor
+> [[frank-medium-rocmfpx]] (charlie12345/ROCmFPX llama.cpp on 2x Radeon AI
+> PRO R9700) took the :7770 endpoint and the port. This page is kept as the
+> retirement record of the FreeToken/CUDA era; the unit was archived to
+> `~/.AGENTTRASH/20260915-frank-small-retire/host-units/` and the repo files
+> to `scripts/.AGENTTRASH/`.
+
 The `frank-small` service on Dankserver is served by
 **FreeToken v0.1.2** — an edge-native MoE serving engine that keeps
 expert weights in host RAM with an LRU GPU cache. Model history on this
@@ -1,9 +1,10 @@
 # LLM Fleet — Knowledge Base

-Documentation for the LLM inference fleet on two local machines
-(2026-09-11): FreeToken v0.1.2 on Dankserver (frank-small,
-Huihui-Qwen3.6-35B-A3B-abliterated-FP8) and halogen-flash-server 0.5.8 on
-Dankclaw (frank-large, peonist-ai Qwen3.8-Flash-Next w4b).
+Documentation for the LLM inference fleet on the two hosts
+(2026-09-15): charlie12345/ROCmFPX llama.cpp on Dankserver (frank-medium,
+kingjones777 Qwen3.8-27B-ROCmFPX-Q8_0-AGENT on 2x Radeon AI PRO R9700) and
+halogen-flash-server 0.5.8 on Dankclaw (frank-large, peonist-ai
+Qwen3.8-Flash-Next w4b).

 > 2026-08-22: the former `vllmScripts` repository is merged into the
 > serverAppConfigs repository. This knowledge base moved from the old repo's
@@ -14,12 +15,13 @@ Dankclaw (frank-large, peonist-ai Qwen3.8-Flash-Next w4b).

 ```mermaid
 graph LR
-    subgraph Dankserver[NVIDIA Dankserver - Ubuntu 26.04]
-        A[RTX 3090 24GB + 60GB RAM]
-        B2[FreeToken v0.1.2
-frank-small, port 7770]
-        C[Huihui-Qwen3.6-35B-A3B-abliterated-FP8
-block-fp8, 20% MoE residency, 453k KV / 4 streams]
+    subgraph Dankserver[AMD Dankserver - Ubuntu 26.04]
+        A[2x Radeon AI PRO R9700
+gfx1201, 32 GiB each]
+        B2[ROCmFPX llama.cpp
+frank-medium, port 7770]
+        C[Qwen3.8-27B-ROCmFPX-Q8_0-AGENT
+layer split, 131k ctx / 1 slot, MTP]
        A --> B2 --> C
    end

@@ -40,14 +42,18 @@ w4b .hgn + overlay, 262k ctx / 4 slots]

 | Server                                  | GPU                           | VRAM / Memory        | OS           |
 | --------------------------------------- | ----------------------------- | -------------------- | ------------ |
-| [[Dankserver System Specs\|Dankserver]] | NVIDIA RTX 3090               | 24 GB VRAM, 62 GB RAM | Ubuntu 26.04 |
+| [[Dankserver System Specs\|Dankserver]] | 2x AMD Radeon AI PRO R9700 | 2x 32 GB VRAM, 60 GB RAM | Ubuntu 26.04 |
 | [[Dankclaw System Specs\|Dankclaw]]     | AMD Radeon 8060S (Strix Halo) | gfx1151 unified memory | Ubuntu 26.04 |

 ## Active Deployment

- [[freetoken-frank-small]] — the active frank-small
-  runbook: FreeToken v0.1.2 serving batsclamp/Huihui-Qwen3.6-35B-A3B-
-  abliterated-FP8 on Dankserver (swapped 2026-08-30; same format/pools)
+- [[frank-medium-rocmfpx]] — the ACTIVE frank-medium runbook:
+  charlie12345/ROCmFPX llama.cpp serving kingjones777/
+  Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF (MTP draft on) on Dankserver's
+  2x Radeon AI PRO R9700 (since 2026-09-15; replaced frank-small)
+- [[freetoken-frank-small]] — RETIRED 2026-09-15 with the RTX 3090: the
+  FreeToken v0.1.2 frank-small runbook (batsclamp Huihui-Qwen3.6-35B-A3B-
+  abliterated-FP8), kept as the retirement record
 - [[freetoken-benchmark-sweep]] — KV pool vs decode speed sweep
  (bs1/bs3/bs4/bs6, 8k–700k KV) on Dankserver RTX 3090
 - [[deployment]] — deploy and update process for both servers
@@ -80,7 +86,7 @@ w4b .hgn + overlay, 262k ctx / 4 slots]
  on Dankclaw (superseded 2026-08-24 by the kingjones ROCmFP4 model)
 - [[kairic-edge]] — the retired Kairic Edge IU4 experiment
  (deployed and reverted 2026-08-22)
- [[Dankserver System Specs]] — the RTX 3090 box (frank-small)
+- [[Dankserver System Specs]] — the 2x R9700 box (frank-medium)
 - [[Dankclaw System Specs]] — the Strix Halo gfx1151 box (frank-large,
  on halogen-flash-server since 2026-09-11)
 - [[frank-npu]] — RETIRED 2026-09-11: the FLM NPU sight+sound endpoint
@@ -106,8 +112,8 @@ w4b .hgn + overlay, 262k ctx / 4 slots]
 - [[VLLM Overview]] — installation, CLI, API, architecture, quantization, parallelism
 - [[VLLM Index]] — quick reference with commands, endpoints, and parameter table
 - [[Running Qwen3.6 35B A3B FP8 on Dankserver]] — Qwen3.6-35B-A3B hardware
-  notes and sizing (this architecture family is what frank-small
-  serves today in the Huihui abliterated FP8 variant — see
+  notes and sizing (this architecture family is what the retired frank-small
+  served in the Huihui abliterated FP8 variant — see
  [[freetoken-frank-small]]; the Ornith-1.5-35B-A3B CRACK checkpoint
  that preceded it is archived at [[dealign-ornith-crack]])

@@ -136,7 +142,7 @@ w4b .hgn + overlay, 262k ctx / 4 slots]

 - [[Llama Cpp Runtimes]] — mainline vs charlie12345/ROCmFPX vs ik_llama.cpp: positioning, quantization, server fit
 - [[Llama Cpp Code Diff]] — code-level quantization diff of the three runtimes + server recommendations
- [[SGLang]] — the structured-generation serving engine; when it fits Dankserver (3090) vs Dankclaw (Strix Halo)
+- [[SGLang]] — the structured-generation serving engine; when it fits Dankserver (2x R9700) vs Dankclaw (Strix Halo)

 ## Related

@@ -1,6 +1,12 @@
 # frank-large: kingjones Qwen3.8-27B-ROCmFPX-Q8_0-AGENT

-**Status:** Deployed on Dankclaw (since 2026-08-25). Serving is **BLOCKED by
+> **SUPERSEDED 2026-09-15.** The Q8_0_ROCMFPX-AGENT checkpoint (with the
+> separate `draft-mtp` head) now serves Dankserver as **frank-medium** on the
+> fork's fixed build — see [[frank-medium-rocmfpx]]. The failure below was
+> specific to the gfx1151 legacy-read path and the Q6_0 single-file era.
+
+**Status (historical, 2026-09-13):** was to deploy on Dankclaw. Serving is
+**BLOCKED by
 the `set_rows` build bug** — the model loads but crashes in
 `ggml_compute_forward_set_rows` before it can serve. All config goals are met;
 a build fix is required for the model to run.
@@ -114,8 +114,11 @@ The LLM inference fleet on the two hosts. Full index:

 - [[deployment]] — unified deploy and update process for both servers
 - [[frank-redo-2026-08-17]] — the 2026-08-17 topology change
- [[freetoken-frank-small]] — the active frank-small runbook: FreeToken v0.1.2 serving Qwen3.6-35B-A3B-FP8 on Dankserver (2026-09-02)
- [[freetoken-benchmark-sweep]] — KV pool vs decode speed sweep (bs1/bs3/bs4/bs6, 8k–700k KV) on Dankserver RTX 3090
+- [[frank-medium-rocmfpx]] — the ACTIVE frank-medium runbook:
+  charlie12345/ROCmFPX llama.cpp serving kingjones777/
+  Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF on Dankserver's 2x Radeon AI PRO
+  R9700 (since 2026-09-15; replaced the FreeToken frank-small)
+- [[freetoken-benchmark-sweep]] — KV pool vs decode speed sweep (bs1/bs3/bs4/bs6, 8k–700k KV) on the retired Dankserver RTX 3090
 - [[kingjones-strix-mtp]] — the Qwen3.8-27B-Q6_0 ROCmFPX AGENT era with the
  standalone MTP draft (Dankclaw, 2026-08-25 → 2026-08-29; historical)
 - [[agentionai-fast-imatrix]] — the retired frank-large runbook:
@@ -138,10 +141,11 @@ The LLM inference fleet on the two hosts. Full index:
 - [[ciru-activefpx-promptforge]] — the RETIRED v3 CIRU PromptForge runtime on Dankclaw (superseded 2026-08-24)
 - [[kingjones-strix-mtp]] — the kingjones Qwen3.8-27B ROCmFP4 STRIX MTP model on Dankclaw (active)
 - [[dealign-ornith-crack]] — the retired Ornith-1.5-35B-A3B CRACK model on Dankserver (retired 2026-09-02)
- [[ikllama-ornith-dankserver]] — the retired ik_llama.cpp runtime for frank-small (q8_0 + Hadamard KV cache, split-mode none; 2026-08-26)
+- [[ikllama-ornith-dankserver]] — the retired ik_llama.cpp runtime for the old frank-small (q8_0 + Hadamard KV cache, split-mode none; 2026-08-26)
+- [[freetoken-frank-small]] — RETIRED 2026-09-15: the FreeToken/CUDA frank-small runbook, retired with the RTX 3090
 - [[kairic-edge]] — the retired Kairic Edge IU4 experiment (2026-08-22)
 - [[dflash2-large-vulkan]] — the retired DFlash2 Vulkan deployment (2026-08-21)
- [[Dankserver System Specs]] — the RTX 3090 box hardware
+- [[Dankserver System Specs]] — the 2x R9700 box hardware (frank-medium)
 - [[Dankclaw System Specs]] — the Strix Halo gfx1151 box (frank-large)
 - [[Dankclaw Improvements]] — host-level changes on the Strix Halo box
 - [[Swap Configuration]] — the 64 GiB `/swap.img` host swapfile (raised from
@@ -11,6 +11,47 @@ timestamp: 2026-09-14

 # Bundle Change Log

+## 2026-09-15 — frank-medium: Dankserver GPU swap (2x R9700), frank-small retired
+- The RTX 3090 left Dankserver; two AMD Radeon AI PRO R9700 cards (Navi 48,
+  gfx1201, 32,624 MiB each) replaced it. FreeToken/CUDA cannot run there, so
+  `frank-small` is retired and `frank-medium` took the :7770 endpoint.
+- Runtime: charlie12345/ROCmFPX llama.cpp `c49ebdb` built for **gfx1201** with
+  the fork's self-contained ROCm 7.14 toolchain (TheRock nightly, RPATH-bundled
+  libs) — the host's broken mixed ROCm (AMD-repo 6.3.3 hijacking libhsa via
+  `/etc/ld.so.conf.d/10-rocm-opencl.conf`) is fully bypassed. Model:
+  kingjones777/Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF (ftype 115, 26.28 GiB) +
+  Q4_0 MTP draft head; mmproj unused (vision stays on frank-large).
+- **Parallelism decision, measured not argued:** llama-bench with the correct
+  multi-device syntax (`-dev ROCm0/ROCm1` — comma form silently benches each
+  card separately): layer 1991/18.8 t/s (pp4096/tg128) vs tensor 271.8/26.3 vs
+  single card 1160.9/19.4. Tensor's decode win costs a 7x prefill collapse
+  (no RCCL → "meta-backend butterfly" allreduce); served with MTP, layer also
+  wins decode (35.7–46.2 tok/s vs 28.7). Operator picked layer ("pipeline
+  parallelism"); the data agrees on both axes. Draft head pinned to ROCm1
+  (card holding the last target layers); under tensor mode an unpinned draft
+  aborts the KV build (GGML_ASSERT(buffer) in ggml-backend.cpp).
+- Verified live before and after the systemd cutover: health ~18 s, MTP
+  acceptance 60–69 %, cold 949-token prefill in 2036 ms, and **prefix caching
+  works with the draft head** (turn 2 reprocessed 22 tokens, cache_n 945) —
+  the model card's mtp-prompt-cache-fix.patch defect is already fixed in
+  `c49ebdb` (`common_prompt_checkpoint::data_spec`).
+- Deploy: `deploy.sh dankserver --service frank-medium` (new
+  `retire_root_llm_small()` archived the CUDA-era unit to
+  `~/.AGENTTRASH/20260915-frank-small-retire/host-units/`); e2e gate 30/30
+  including 11-admin (the blankpad→192.168.0.112 path issue has cleared).
+- Host cleanup (the "Both" half of the plan): purged 167 stale AMD-repo
+  `~24.04` ROCm 6.3.3 packages, removed the radeon apt pin and the
+  ld.so.conf hijack (staged in `~/.AGENTTRASH/20260915-rocm633-cleanup/`);
+  `libhsa-runtime64` now resolves only to Ubuntu's ROCm 7.1 and `rocm-smi`
+  7.1.1 (Ubuntu package) works. frank-medium unaffected (self-contained).
+- Client cutover: blankpad `models.yml`/`config.yml` renamed the model id to
+  `frank-medium` (ctx 131072, maxTokens 32768); Hermes live config on Dankclaw
+  repointed `dankserver-small` routes to `frank-medium`.
+- New: `/serverAppConfigs-docs/LLM/frank-medium-rocmfpx.md` runbook;
+  updated AGENTS.md, README, docs indexes, Dankserver System Specs,
+  deployment/Hermes/OpenWebUI/Plex/Stack pages; freetoken-frank-small and
+  kingjones-q8agent marked retired/superseded.
+
 ## 2026-09-12 — Marreta content-cleaning proxy deployed on Dankclaw
 - Added `manualdousuario/marreta` (Laravel / `serversideup/php:8.4-fpm-nginx`)
  as a rootless Quadlet pair: `marreta` + `marreta-browser` (Lightpanda),