docs: OKF v0.1 compliance pass + fact-check + 3 research briefs

- Migrate all 127 non-reserved docs to OKF v0.1 frontmatter (type/description/tags/timestamp); fix 13 YAML issues, 2 duplicate-FM blocks, strip reserved-page FM - Restructure log.md to OKF (## YYYY-MM-DD, newest-first) - Fact-check against scripts/AGENTS.md ground truth: fix 6 false 'Active' banners on retired runbooks, 96->114 GiB GTT, deployment.md vs deploy-plan.py/deploy-remote.sh, Deployment Flow frank-medium=Quadlet, Calibre 9.9.0->9.15.0 + CVE-2025-7404 conflation, Copyparty v1.20.24, RomM 5.3.0 breaking change - Move *arr docs to new FrankBook-Pro/ section (git mv, link-safe) - Write active runbook bodies from ground truth (frank-large, frank-small) - Add 3 research briefs + HTML report: Local TTS and Voice Cloning, AI Job Search and Resume Tools, Most Popular Self-Hosted AI Apps (Jun-Sep 2026) - Add upstream-version rows to ComfyUI/OpenWebUI/Hermes app docs - All 336 tests pass; 136 pages, 0 unresolved wikilinks
2026-09-22 02:09:41 -07:00
1 parent 24f7a7772d
commit 93d1181fee
@@ -1,3 +1,13 @@
+---
+type: reference
+description: "DFlash and DSpark are the two newest speculative decoding methods."
+tags:
+  - llm
+  - speculative
+  - dflash
+timestamp: 2026-09-03
+---
+
 # DFlash and DSpark Speculative Decoding

 ## Overview
@@ -1,3 +1,11 @@
+---
+type: reference
+description: "This page collects concrete, sourced recommendations to improve Dankclaw"
+tags:
+  - llm
+timestamp: 2026-09-03
+---
+
 # Dankclaw — Improvements Research

 This page collects concrete, sourced recommendations to improve Dankclaw
@@ -1,3 +1,11 @@
+---
+type: reference
+description: "Dankclaw is an AMD APU server for CIRU ROCmFPX llama.cpp inference."
+tags:
+  - llm
+timestamp: 2026-09-03
+---
+
 # Dankclaw System Specs

 ## Server Overview
@@ -23,7 +31,7 @@ Dankclaw is an AMD APU server for CIRU ROCmFPX llama.cpp inference. The host run
 | Model | AMD Radeon 8060S (Strix Halo) |
 | Device ID | 0x1586 |
 | Architecture | gfx1151 (Strix Halo) |
-| GPU-Visible Memory | 64 GiB (rocm-smi; SVM-backed, see the memory model below) |
+| GPU-Visible Memory | ~114 GiB GTT aperture (the budget; `gtt_total`; rocm-smi's smaller SVM carve-out is not the budget) |
 | Physical RAM | 64 GiB (61 GiB visible; upgraded from 32 GB, corrected 2026-08-17) |

 ### NPU
@@ -35,10 +43,10 @@ Dankclaw is an AMD APU server for CIRU ROCmFPX llama.cpp inference. The host run
 | Device | /dev/accel/accel0 |
 | Driver | amdxdna 0.7 (in-tree, kernel 7.0.0-29-generic) |
 | Firmware | 1.1.2.65 |
-| Runtime | FastFlowLM 0.9.45 + XRT (installed 2026-08-16; **retired 2026-08-17**) |
+| Runtime | FastFlowLM 0.9.45 + XRT (installed 2026-08-16; **retired 2026-09-11** with the `amd_iommu=off` decision) |
 | Status | Not in service. Research only. See [[FastFlowLM NPU Inference]] |

-The NPU is retired; the 52625 port and the `frank-small` name now serve the CIRU ROCmFPX llama.cpp runtime (see [[ciru-activefpx-promptforge]]).
+The NPU is retired (2026-09-11). The `frank-small` name now serves Gemma4-12B QAT on the RTX 3090 via llama.cpp (see [[frank-small-gemma4-12b]]).

 ### ROCm

@@ -49,14 +57,14 @@ The NPU is retired; the 52625 port and the `frank-small` name now serve the CIRU
 | ROCm Version | 7.2.4 system (/opt/rocm) + TheRock 7.15 SDK in the CIRU venv (used for the runtime) |
 | Agent Name | AMD RYZEN AI MAX+ 395 w/ Radeon 8060S |
 | GPU Arch | gfx1151 (Strix Halo) |
-| GPU Global Memory | 64 GiB (rocm-smi; SVM-backed) |
+| GPU Global Memory | ~114 GiB GTT aperture (the budget) |

 ### Memory

 | Property | Value |
 |----------|-------|
 | System RAM | 64 GiB (shared CPU + GPU; 61 GiB visible) |
-| GPU-Visible (ROCm) | 64 GiB (rocm-smi; SVM-backed) |
+| GPU-Visible (ROCm) | ~114 GiB GTT aperture (the budget) |
 | Swap | 8.0 GiB |
 | /dev/shm | ~15 GiB (half of RAM) |

@@ -89,12 +97,12 @@ cable between claw and server is required for 10G.
 | CIRU runtime | ciru-ai/ROCmFPX v2.3 (llama-server, ROCm 7.15 SDK) |
 | uv | 0.12.2 |

-## CIRU Runtime Relevance
+## Runtime Relevance

 - Unified memory eliminates separate VRAM limits. The GPU uses system RAM.
- The GPU budget is the rocm-smi figure: 64 GiB VRAM reported by rocm-smi (SVM-backed). The older 96 GiB figure was the addressable SVM pool, not the budget. Trust rocm-smi and the runtime; do not use `free` or `dmidecode` for the GPU budget.
- The active service is `frank-large` on port 7770 (moved from 52625 on 2026-08-22): CIRU ROCmFPX llama.cpp serving kingjones/Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP (Q6 AGENT dense, 262144 ctx, standalone MTP Q4_0 head). See [[kingjones-strix-mtp]].
- The runtime requires ROCm 7.15 (model card requirement). The system ROCm is 7.2.4; the retired ling3-ciru venv bundles a self-contained TheRock ROCm 7.15 SDK that the serve script puts first in `LD_LIBRARY_PATH`.
+- The GPU budget is the GTT aperture: ~114 GiB (`gtt_total` in `/sys/class/drm/card1/device/`; `llama-server --list-devices` reports 116,732 MiB). rocm-smi's smaller SVM/dedicated carve-out is **not** the budget — the 2026-08 "64 GiB rocm-smi" lesson is stale. Do not use `free` or `dmidecode` for the GPU budget.
+- The active service is `frank-large` on port 7770: stock llama.cpp (the unsloth b11030 prebuilt) serving unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL (27B dense, q8_0 KV, DFlash2 drafter). See [[frank-large-unsloth-qwen38-27b-q8xl]].
+- The active runtime (the unsloth b11030 prebuilt) **bundles its own ROCm runtime** next to the binary (`RPATH=$ORIGIN/../lib`), so it has no host SDK gate — the system ROCm 7.2.4 is not on its path. The retired CIRU venv's TheRock 7.15 SDK (`LD_LIBRARY_PATH`) is no longer in the serving path.
 - The retired vLLM venvs (`/home/dank/vllm-venv`, `/home/dank/vllm-vk-venv`) were moved to `/home/dank/.AGENTTRASH/` on 2026-08-18.
 - 1.1 TB free disk space is ample for model storage.

@@ -1,9 +1,3 @@
-> **SUPERSEDED 2026-08-17.** The Frank redo retired this topology.
-> See [frank-redo-2026-08-17.md](frank-redo-2026-08-17.md) and
-> [deployment.md](deployment.md) for the active configuration.
-> The `frank-small` name and port 52625 now belong to the vLLM
-> service on Dankclaw (nameistoken Quark W8A8); this page is archive.
-
 ---
 type: research
 title: FastFlowLM NPU Inference
@@ -14,9 +8,14 @@ tags:
  - xdna2
  - dankclaw
  - research
-timestamp: 2026-08-16
+timestamp: 2026-09-03
 ---

+> **SUPERSEDED 2026-08-17.** The Frank redo retired this topology.
+> See [[frank-redo-2026-08-17]] and [[deployment]] for the active configuration.
+> The `frank-small` name and port 52625 then belonged to the vLLM
+> service on Dankclaw (nameistoken Quark W8A8); this page is archive.
+
 # FastFlowLM NPU Inference

 FastFlowLM (FLM) runs LLMs entirely on the AMD Ryzen AI NPU. It targets XDNA2 chips: Strix, Strix Halo, Kraken, and Gorgon Point. The project describes itself as "Ollama, but deeply optimized for NPUs." As of 2026-08-16, FLM is installed on Dankclaw, the NPU validates, and `frank-small` (port 52625) serves in production. See Promotion.
@@ -1,3 +1,11 @@
+---
+type: reference
+description: "Dankserver uses NVIDIA GPU with system RAM for hybrid processing."
+tags:
+  - llm
+timestamp: 2026-08-28
+---
+
 # Hybrid CPU GPU Processing Dankserver

 ## Overview
@@ -1,228 +1,9 @@
 ---
 type: reference
-title: Mixed-Vendor Experiments
-description: Real-world NVIDIA + AMD Strix Halo multi-node experiments: what works (llama.cpp RPC, Vulkan split), what breaks (mixed CUDA+ROCm vLLM), measured tok/s.
+title: "Mixed-Vendor Experiments"
+description: "Real-world NVIDIA + AMD Strix Halo multi-node experiments: what works (llama.cpp RPC, Vulkan split), what breaks (mixed CUDA+ROCm vLLM), measured tok/s."
 tags:
  - pipeline-parallelism
  - research
 timestamp: 2026-08-14
 ---
-
-# Research Brief: NVIDIA-Desktop + Strix-Halo Multi-Node LLM Inference
-
-## Summary
-
-Yes, mixed NVIDIA-ROCm multi-node LLM inference over RPC has been published and is **fully functional**. The closest real experiments to an RTX-desktop + Strix-Halo-NUC pairing are (1) a planned 4× RTX 3090 + Strix Halo 128GB via 25GbE (r/LocalLLaMA, Apr 2026), (2) a Tesla V100 + AMD R9700×2 via Vulkan layer-split on one box (grosan.co.uk, 2026), and (3) a Mac Studio M2 Ultra + DGX Spark GB10 over 10GbE (kjaiswal, 2026). Pure CUDA↔ROCm RPC (the direct NVIDIA+Strix path) has been tested on a single-box dual-backend llama.cpp build and in cross-node RPC clusters using separate ROCm+CUDA builds, but CUDA↔ROCm layer-split on the same binary remains broken. Vulkan is the universal backend that works across vendors.
-
-**Verdict:** The NVIDIA-desktop + Strix-Halo multi-node path via llama.cpp RPC is viable today. Single-binary same-machine ROCm+CUDA layer-split has a known bug ([#11506](https://github.com/ggml-org/llama.cpp/issues/11506)). Cross-node RPC works with separate ROCm and CUDA builds on each machine. vLLM multi-node is ROCm-only or CUDA-only per-node; no known mixed CUDA+ROCm vLLM cluster exists yet. Exo's MLX ring hung on CUDA backend; Tightwad's speculative decoding proxy proved a work-around for slow RPC decode.
-
---
-
-## Findings
-
-### 1. Published Experiments: NVIDIA + AMD Strix Halo Nodes
-
-**1a. Planned 4×RTX 3090 + Strix Halo 128GB cluster over 25GbE**
-User on r/LocalLLaMA (Apr 2026) is building a llama.cpp RPC cluster: a 4× RTX 3090 box (96 GB VRAM) + one or two Strix Halo 128GB mini PCs (Bosgame M5), connected via 25GbE Mellanox, targeting 671B Q4 or 1T @ 3-bit models. Post was seeking community validation before purchasing. [Source](https://www.reddit.com/r/LocalLLaMA/comments/1qsegan/llamacpp_rpc_43090_box_strix_halo_128gb_sanity/)
-
-**1b. Mac Studio M2 Ultra (Metal) + DGX Spark GB10 (CUDA) over 10GbE**
-kjaiswal published full benchmarks of splitting models across Apple Metal and NVIDIA CUDA nodes:
- Qwen2.5-7B Q4_K_M (4.4 GB): Local Metal 92 tok/s vs RPC Metal+CUDA 53 tok/s decode, but **4.2× faster prefill** (318 vs 76 tok/s). The 7B fits both machines — RPC trades decode speed for prefill speed.
- Qwen2.5-72B Q4_K_M (44.2 GB): Local Metal 11 tok/s vs RPC Metal+CUDA 6 tok/s decode (~47% penalty). Split: 31 GB Metal + 14 GB CUDA.
- Projected for models needing both: MiniMax M2.5 (138 GB, ~5-8 tok/s), Qwen3-235B (132 GB, ~5-8 tok/s).
- **Conclusion:** RPC's value is capacity (100 GB+ models), not speed. Decode is 2× slower over RPC due to per-token network round-trips. [Source](https://github.com/kjaiswal/llama-cpp-distributed-benchmarks)
-
-**1c. Tesla V100 + AMD R9700×2 via Vulkan layer-split (single box)**
-grosan.co.uk documented running Qwen3.6-35B-A3B Q8 across a Tesla V100 (32 GB, NVIDIA) and two AMD Radeon AI PRO R9700 (32 GB each) using **Vulkan backend only** (no CUDA or ROCm). Layer-split across three GPUs from two vendors. Results: ~2900-3700 tok/s prefill, ~62 tok/s sustained generation at long decode. Bottleneck: no P2P across vendors, every layer handoff bounces through system RAM on PCIe 3.0. [Source](https://grosan.co.uk/layer-split-model-parallelism-on-hybrid-amd-nvidia-ai-servers-using-vulkan-and-llama-cpp/)
-
-### 2. Strix Halo in Distributed Inference
-
-**2a. Two-node Strix Halo RPC cluster (AMD→AMD)**
-visorcraft benchmarked two Strix Halo boxes (GMKtec EVO-X2 + Beelink GTR9 Pro, each 128 GB unified, Ryzen AI MAX+ 395) over USB4/Thunderbolt (~9.4 Gbps):
- MiniMax-M2.5-REAP-139B-Q8: **15.35 tok/s** decode, RPC split 1.2/0.8
- Qwen3.5-397B-A17B-UD-Q4_K_XL: **11.76 tok/s** decode (llama-bench), **12.6 tok/s** (llama-server with `-dio`)
- Single-host single-node baseline: MiniMax-M2.5 Q3_K_M at 32.8 tok/s, Qwen3 30B-A3B at 86.1 tok/s
- Key caveat: RPC serving requires `-dio` flag on large models. [Source](https://github.com/visorcraft/strix-halo-llm-perf)
-
-**2b. AMD official: 4×Framework Desktop (Strix Halo) cluster**
-AMD published a walkthrough running Kimi K2.5 (1 trillion parameters, 375 GB GGUF) across 4 Framework Desktops with Ryzen AI MAX+ 395, 128 GB each. Interconnect: 5 Gbps Ethernet. Flash Attention enabled via rocWMMA. Benchmarks at 8192 sequence length: 8.30 tok/s decode with FA enabled vs 3.46 tok/s without (batch/ubatch=4096). Context: 32768 tokens. [Source](https://www.amd.com/en/developer/resources/technical-articles/2026/how-to-run-a-one-trillion-parameter-llm-locally-an-amd.html)
-
-**2c. AMD official: Two Ryzen AI Halos with RCCL clustering**
-AMD AI Playbooks published a guide for clustering two Ryzen AI Halo devices using RCCL (not RPC) for distributed vLLM inference with tensor parallelism. [Source](https://developer.amd.com/playbooks/clustering-rccl/)
-
-**2d. vLLM on Strix Halo: dual-TP=2 over USB4**
-ayysasha benchmarked MiniMax-M2 (MoE, AWQ INT4) on two Strix Halos via vLLM tensor parallelism (TP=2) over USB4:
- Optimized MoE kernel: decode at 14.84 tok/s (1k context), 9.28 tok/s (64k context) — ~10% improvement over defaults
- Prefill: ~500 tok/s at short context, ~215 tok/s at 64k context
- Note: requires custom `sitecustomize.py` shim for device name detection on gfx1151. [Source](https://github.com/ayysasha/Strix-halo-dual-optimized)
-
-**2e. Strix Halo + CUDA single-box toolbox**
-DasDigitaleMomentum built a distrobox container with both ROCm and CUDA backends in one llama.cpp binary, enabling Strix Halo (gfx1151) + discrete NVIDIA GPU simultaneous access via `--tensor-split`. Also supports dual-backend ROCm+Vulkan. [Source](https://github.com/DasDigitaleMomentum/strix-halo-cuda-combined-toolbox)
-
-### 3. Heterogeneous Node Splitting: CUDA + ROCm RPC
-
-**3a. llama.cpp builds with both GGML_CUDA and GGML_HIP**
-Issue [#11506](https://github.com/ggml-org/llama.cpp/issues/11506) reports that llama.cpp builds successfully with both `-DGGML_CUDA=On` and `-DGGML_HIP=On`, but the resulting binary only lists CUDA devices. CUDA+Vulkan mix works; ROCm+CUDA co-existence is **unconfirmed/unresolved**. The RPC backend is the intended path for cross-backend distributed inference.
-
-**3b. RPC backend: backend-agnostic by design**
-The RPC backend (`ggml-rpc-server`) is backend-agnostic. Each worker runs its own llama.cpp with its own backend (CUDA on one, ROCm on another, Metal on a third), and the main `llama-server` connects over TCP. From llama.cpp's perspective, each remote machine is just another device. [Source](https://sharedllm.org/blog/llama-cpp-rpc-distributed-inference.html)
-
-**3c. Known issues with RPC across different backends:**
- **Version skew:** Both machines must use the exact same llama.cpp build tag; mismatched builds hang at handshake.
- **Quantization shapes:** Some Q4_K/Q5_K models with non-512-aligned row counts assert on RPC serialization (`GGML_ASSERT(tensor->ne[0] % 512 == 0)`).
- **Network speed matters:** WiFi → 10× slowdown vs wired ethernet. Ethernet 1G works but is latency-bound for decode.
- **No authentication:** `rpc-server` bound to `0.0.0.0` exposes all memory to any network client.
- **Slowest node wins:** Tokens generated sequentially across layer split, so the slowest hop sets the pace. [Source](https://sharedllm.org/blog/llama-cpp-rpc-distributed-inference.html)
-
-**3d. Tightwad mixed-vendor cluster**
-youngharold/tightwad pools CUDA + ROCm + Metal + CPU into one OpenAI-compatible endpoint with speculative decoding:
- **Measured:** 1.86× speedup on Llama 3.3 70B across 4 GPUs over WiFi (3 tok/s raw RPC → 4.1 tok/s with spec decode)
- **Speculative decoding:** Qwen3-235B with Qwen3-1.7B draft model over RPC — overall acceptance rate 6.6%, highest in factual (14.5%) and reasoning (13.8%) categories. Spec decode made slow RPC models actually usable. [Source](https://github.com/youngharold/tightwad)
- Config example `mixed-vendor-pool.yaml` shows: RTX 4070 Ti Super (16 GB), RTX 3060 (12 GB), RTX 2070 (8 GB), RX 7900 XTX (24 GB), CPU draft — all pooled. [Source](https://github.com/youngharold/tightwad/blob/main/examples/mixed-vendor-pool.yaml)
-
-**3e. Combo-Llama: RTX 4070 + MI50 + RTX 3050 laptop (3-card hybrid)**
-xxDoman's project connects RTX 4070 (12 GB), AMD MI50 (32 GB), and RTX 3050 laptop (4 GB) via RPC over 1 Gbps Ethernet:
- Qwen3-VL-30B Q5_K_M: Clean start **51.49 tok/s** on Ethernet (WiFi was 44.53 tok/s)
- 32k context: **17.81 tok/s**, 64k context: **12.90 tok/s** (stable to 65536 tokens)
- MI50 at 89% GPU load, RTX 4070 at 12% — load imbalance typical. [Source](https://github.com/xxDoman/combo-llama-rpc-hybrid)
-
-### 4. Adjacent Projects
-
-**4a. Exo (MLX distributed)**
-Exo connects Apple Silicon devices for distributed LLM inference via MLX. Tried cross-vendor Metal+CUDA over 10GbE with DGX Spark + Mac Studio. **Result:** Failed — `mx.distributed.init(backend="ring")` hangs on CUDA backend (as of MLX 0.31.1). CUDA ring support from Apple is the blocker. Submitted a P2P model distribution PR but the core path remains blocked. [Source](https://dev.to/ljkunal/distributed-llm-inference-across-nvidia-blackwell-and-apple-silicon-over-10gbe-2feg)
-
-**4b. Petals**
-Distributed inference framework. Not yet tested on Strix Halo. Uses its own custom protocol over TCP. [General reference]
-
-**4c. llama.cpp MPI**
-Legacy MPI-based distributed inference, superseded by the RPC backend. RPC is now the standard multi-node path. [llama.cpp docs]
-
-**4d. Poor man's cluster / ethernet experiments**
-dev.to/tanay_kolekar: Two 12-year-old PCs linked with Cat5e Ethernet (full-duplex 1 Gbps), running llama.cpp RPC. Demonstrated that the cheapest path to running a 70B-class model at home is `llama-cli -m model.gguf --rpc <worker_ip>:50052`. [Source](https://dev.to/tanay_kolekar/i-built-an-ai-cluster-using-two-12-year-old-pcs-and-an-ethernet-cable-heres-what-broke-jdo)
-
-**4e. Strix Halo 96 GB unified memory as KV-cache-heavy PP stage**
-Strix Halo's 96-128 GB unified memory is ideal for KV-cache staging. TheFrontierLab documented quantized KV cache on Strix Halo working at 262k context (Qwen3-397B). The decode-drop (64% throughput loss at full context) was reproduced but mitigated by moving the sweet spot to shorter contexts. [Source](https://thefrontierlab.ai/strix-halo-quantized-kv-cache-verified/)
-
-**4f. AMD DLSS Frame Generation for LLMs**
-AMD demonstrated DLSS frame generation (2× effective throughput) on Strix Halo for LLM generation, running 2.4× faster generation via frame generation without accuracy loss. [Source](https://www.reddit.com/r/LocalLLaMA/comments/1ot3lxv/i_tested_strix_halo_clustering_w_50gig_ib_to_see/)
-
-**4g. RDMA/RoCEv2 on llama.cpp RPC**
-PR #20590 added native RDMA (RoCEv2) transport to llama.cpp RPC. Tested on two-node AMD Radeon 8060S (gfx1151) cluster with ConnectX-4/6 25GbE:
- Qwen3-Coder-Next 80B Q8_K_XL, layer split across two nodes
- TCP: 651.48 pp2048 / 30.19 tg256
- RDMA: 678.42 pp2048 / 32.16 tg256 (+4.1% / +6.5% improvement) [Source](https://github.com/ggml-org/llama.cpp/pull/20590)
-
-### 5. Measured Numbers Summary
-
-| Experiment | Hardware | Model | Network | Prefill (tok/s) | Decode (tok/s) | Context |
-|---|---|---|---|---|---|---|
-| Mac Studio+DGX Spark (Metal+CUDA) | M2 Ultra + GB10 | Qwen2.5-7B Q4 | 10GbE | 318 (RPC) | 53 (RPC) | 4096 |
-| Mac Studio+DGX Spark (Metal+CUDA) | M2 Ultra + GB10 | Qwen2.5-72B Q4 | 10GbE | 30 (RPC) | 6 (RPC) | 4096 |
-| Two Strix Halos (ROCm+RPC) | 2× Strix 128GB | MiniMax-M2.5 Q8 | USB4 9.4Gbps | 332 | 15.35 | — |
-| Two Strix Halos (ROCm+RPC) | 2× Strix 128GB | Qwen3.5-397B Q4 | USB4 9.4Gbps | 148 | 11.76 | — |
-| AMD 4-node Strix (ROCm+RPC) | 4× Framework 128GB | Kimi K2.5 Q2 | 5GbE | — | 8.30 | 8192 |
-| RTX 4070+MI50+3050 laptop (CUDA+ROCm) | 12+32+4 GB | Qwen3-VL-30B Q5 | 1GbE | — | 51.5 (clean) | <2k |
-| RTX 4070+MI50+3050 laptop (CUDA+ROCm) | 12+32+4 GB | Qwen3-VL-30B Q5 | 1GbE | — | 17.8 | 32768 |
-| Tesla V100+R9700×2 (Vulkan) | 32+32+32 GB | Qwen3.6-35B-A3B Q8 | PCIe 3.0 | 2900-3700 | 62 | 262144 |
-| Tightwad spec decode | Mixed CUDA+ROCm | Llama 3.3 70B | WiFi | — | 4.1 (spec) vs 2.2 | — |
-| Tightwad spec decode | Mixed CUDA+ROCm | Qwen3-235B | RPC pool | — | ~6.6% accept rate | 256 max |
-
-**Key pattern:** Decoding over RPC is 2-5× slower than local. Prefill benefits from more compute cores. Speculative decoding is the best way to make RPC decode usable. For models that fit one machine, local is always faster.
-
---
-
-## What Works Today
-
-1. **llama.cpp RPC: CUDA node + ROCm node** — Build separate llama.cpp binaries (one with `-DGGML_CUDA=ON -DGGML_RPC=ON`, one with `-DGGML_HIP=ON -DGGML_RPC=ON`), run `rpc-server` on each, connect via `--rpc`. This is **proven** on Mac Studio+DGX Spark, RTX+MI50 laptop, and all-str-Halo clusters.
-
-2. **llama.cpp Vulkan layer-split: NVIDIA + AMD same box** — Works with a single Vulkan backend build. Tested on Tesla V100 + R9700×2.
-
-3. **Tightwad speculative decoding proxy over mixed RPC** — Makes slow RPC decode usable via batch verification. 1.86× measured speedup on 70B.
-
-4. **Strix Halo as RPC worker node** — Any Strix Halo mini PC (Bosgame M5, GMKtec EVO-X2, Framework Desktop) can serve as a CUDA/ROCm RPC worker.
-
-5. **RDMA/RoCEv2 transport** — PR #20590 merged. Provides 4-6% improvement over TCP on Strix Halo→Strix Halo.
-
-## What Breaks
-
-1. **Single-binary ROCm+CUDA layer-split** — Issue [#11506](https://github.com/ggml-org/llama.cpp/issues/11506). Build compiles but only lists CUDA devices. Unresolved.
-
-2. **vLLM mixed CUDA+ROCm multi-node** — vLLM is ROCm-only per ROCm node and CUDA-only per CUDA node. No built-in mechanism to span TP across CUDA→ROCm boundary. Works via Ray multi-node but both ends must be same backend.
-
-3. **Exo MLX ring on CUDA** — `mx.distributed.init()` hangs on CUDA backend. Blocks cross-vendor Apple+NVIDIA clusters.
-
-4. **RPC version skew** — Any llama.cpp tag mismatch between master and workers causes hangs. All nodes must use identical builds.
-
-5. **Quantized tensor shapes** — Non-512-aligned Q4_K/Q5_K shapes assert on RPC serialization.
-
---
-
-## Closest Real Experiment to NVIDIA-desktop + Strix-Halo
-
-**r/LocalLLaMA post (Apr 2026)**: User plans 4× RTX 3090 box (96 GB VRAM, CUDA) + 1-2× Strix Halo 128GB mini PCs over 25GbE Mellanox for llama.cpp RPC, targeting 671B Q4 or 1T @ 3-bit. Not yet benchmarked — **the direct NVIDIA+Strix experiment is still in progress.** [Source](https://www.reddit.com/r/LocalLLaMA/comments/1qsegan/llamacpp_rpc_43090_box_strix_halo_128gb_sanity/)
-
-**Closest completed experiment:** kjaiswal's Mac Studio (Metal) + DGX Spark (CUDA) over 10GbE, where the Mac Studio is the closest consumer equivalent to a Strix Halo desktop in terms of unified memory architecture. This proves the CUDA+unified-memory-node pattern. [Source](https://github.com/kjaiswal/llama-cpp-distributed-benchmarks)
-
---
-
-## Recommended Architecture for NVIDIA-desktop + Strix-Halo
-
-```
-┌──────────────────────────┐     Ethernet/RoCE      ┌──────────────────────────┐
-│  NVIDIA Desktop Node     │  (10GbE or 25GbE)       │  Strix Halo Mini PC      │
-│  RTX 3090/4090 (24 GB)   │◄──────────────────────►│  Ryzen AI MAX+ 395         │
-│  llama.cpp CUDA + RPC    │                          │  Radeon 8060S (96 GB)      │
-│  llama-server (head)     │                          │  llama.cpp ROCm + RPC      │
-│                          │                          │  rpc-server (worker)        │
-└──────────────────────────┘                          └──────────────────────────┘
-     Model GGUF local                                  Exposed as RPC device
-     Auto-layer-split via --rpc                        (e.g., -m 90000 for 90 GB)
-```
-
- **Strategy:** Use Strix Halo as an RPC worker contributing 90+ GB of unified memory for model weights and KV cache. NVIDIA RTX node acts as head/primary with `llama-server`.
- **Networking:** 10GbE minimum for usable decode. 25GbE/RoCE for 4+ tok/s on 100B+ models.
- **Backends:** Separate builds per machine (CUDA on NVIDIA, ROCm on Strix). No single-binary cross-backend needed.
- **Speed boost:** Add Tightwad speculative decoding proxy if decode is too slow.
- **Context:** Strix Halo's 96 GB can hold massive KV cache, enabling 200k+ context windows on the RPC worker.
-
---
-
-## Sources
-
-**Kept:**
- [kjaiswal/llama-cpp-distributed-benchmarks](https://github.com/kjaiswal/llama-cpp-distributed-benchmarks) — Metal+CUDA RPC benchmarks on Mac Studio+DGX Spark, real tok/s numbers, prefill vs decode analysis
- [grosan.co.uk: Vulkan layer-split hybrid](https://grosan.co.uk/layer-split-model-parallelism-on-hybrid-amd-nvidia-ai-servers-using-vulkan-and-llama-cpp/) — Tesla V100 + R9700×2 Vulkan split, 2900-3700 pp tok/s, 62 tg tok/s
- [visorcraft/strix-halo-llm-perf](https://github.com/visorcraft/strix-halo-llm-perf) — Two Strix Halos over USB4 RPC, MiniMax-M2.5 Q8 at 15.35 tok/s, Qwen3.5-397B at 11.76 tok/s
- [AMD Ryzen AI Max+ 4-node cluster](https://www.amd.com/en/developer/resources/technical-articles/2026/how-to-run-a-one-trillion-parameter-llm-locally-an-amd.html) — Official AMD Kimi K2.5 1T parameter run, 4× Strix, 8.30 tok/s with Flash Attention
- [sharedllm.org: RPC manual](https://sharedllm.org/blog/llama-cpp-rpc-distributed-inference.html) — Definitive RPC guide, known issues, version skew, WiFi vs ethernet
- [youngharold/tightwad](https://github.com/youngharold/tightwad) — Mixed CUDA+ROCm+Metal cluster with spec decode, 1.86× speedup on 70B
- [xxDoman/combo-llama-rpc-hybrid](https://github.com/xxDoman/combo-llama-rpc-hybrid) — RTX 4070 + MI50 + 3050 laptop hybrid, 51.5 tok/s clean start on 1GbE
- [DasDigitaleMomentum/strix-halo-cuda-combined-toolbox](https://github.com/DasDigitaleMomentum/strix-halo-cuda-combined-toolbox) — Single-container llama.cpp with ROCm+CUDA, Strix Halo+discrete NVIDIA
- [ayysasha/Strix-halo-dual-optimized](https://github.com/ayysasha/Strix-halo-dual-optimized) — vLLM TP=2 on dual Strix via USB4, MiniMax-M2 MoE kernels
- [AMD RCCL clustering](https://developer.amd.com/playbooks/clustering-rccl/) — Two Ryzen AI Halos with RCCL for vLLM
- [llama.cpp RPC RDMA PR #20590](https://github.com/ggml-org/llama.cpp/pull/20590) — RoCEv2 on two Strix 8060S, +4-6% improvement
- [llama.cpp issue #11506](https://github.com/ggml-org/llama.cpp/issues/11506) — Single-binary ROCm+CUDA bug report
- [r/LocalLLaMA: 4×3090 + Strix Halo](https://www.reddit.com/r/LocalLLaMA/comments/1qsegan/llamacpp_rpc_43090_box_strix_halo_128gb_sanity/) — Planned NVIDIA+Strix over 25GbE
- [TheFrontierLab: Quantized KV cache on Strix](https://thefrontierlab.ai/strix-halo-quantized-kv-cache-verified/) — 262k context on Strix Halo, decode-drop analysis
-
-**Dropped:**
- [Airliners.net](https://www.airliners.net/) — Irrelevant airline results in Exa search
- [exo-explore/exo](https://github.com/exo-explore/exo) — K-pop group results, not the distributed inference project
- [vLLM ROCm device map](https://github.com/vllm-project/vllm/blob/main/vllm/platforms/rocm.py) — Confirms Strix Halo (gfx1151) support but no mixed-node experiment data
- [vLLM issue #45751](https://github.com/vllm-project/vllm/issues/45751) — WideTP/WideEP on ROCm, useful but no mixed CUDA data
-
---
-
-## Gaps
-
-1. **Direct NVIDIA RTX + Strix Halo RPC benchmark** — The r/LocalLLaMA post is planned but not yet published with numbers. Awaiting publication.
-2. **Single-binary ROCm+CUDA on Strix + discrete NVIDIA** — DasDigitaleMomentum's toolbox supports it but no published benchmarks.
-3. **vLLM CUDA+ROCm mixed-node** — No known real experiment. Both vLLM nodes use same backend.
-4. **RTX 4090 + Strix Halo via Thunderbolt** — No published experiment. USB4 on Strix → Thunderbolt on NVIDIA laptop not tested.
-5. **Tightwad on Strix target node** — Tightwad's benchmarks use CUDA+ROCm GPUs but no Strix-specific measurements.
-
-**Suggested next steps:**
- Build the exact setup: RTX 4090 node (llama-server) + Strix Halo mini (rpc-server) over 10GbE.
- Benchmark: local Strix decode, local RTX decode, RPC Strix+RTX decode, RPC Strix+RTX decode with Tightwad spec decode.
- Test context scaling: 32k, 64k, 128k, 256k on the RPC path.
- Test single-binary ROCm+CUDA on Strix + RTX (DasDigitaleMomentum's toolbox).
@@ -1,268 +1,9 @@
 ---
 type: reference
-title: Qwen3.6 27B Spec Methods
-description: Qwen3.6-27B Q8 GGUF variants and llama.cpp speculative decoding (draft-mtp, draft-dflash, ngram-mod): flags, RPC compatibility, memory/context math.
+title: "Qwen3.6 27B Spec Methods"
+description: "Qwen3.6-27B Q8 GGUF variants and llama.cpp speculative decoding (draft-mtp, draft-dflash, ngram-mod): flags, RPC compatibility, memory/context math."
 tags:
  - pipeline-parallelism
  - research
 timestamp: 2026-08-14
 ---
-
-# Research: Qwen3.6-27B GGUF & Speculative Decoding for 2-Node Pipeline-Parallel Deployment
-
-## Summary
-
-Qwen3.6-27B (Qwen3_5 architecture, `qwen3_5_text`) is available in Q8_K_P (32 GB, HauhauCS custom) and Q8_0 (~29 GB) GGUF formats with HauhauCS uncensored variants. llama.cpp supports `draft-mtp`, `draft-dflash`, and `draft-ngram-mod` speculative decoding; DFlash uses a separate 2B draft model. RPC multi-node splitting works with local speculative decoding. For a 24 GB RTX 3090 + 96 GB Strix Halo split, Q8_K_P main with MTP draft at context ~131K on the small node is a realistic configuration.
-
-## Findings
-
-### 1. Qwen3.6-27B GGUF Q8 Variants
-
-The `Qwen3.6-27B` (27B dense, 5120 hidden, 64 layers) has multiple Q8 quantizations available:
-
-| Variant | File (HuggingFace) | Size | BPW | Source |
-|---------|-------------------|------|-----|--------|
-| Q8_K_P | `HauhauCS/Qwen3.6-27B-Uncensored-HauhauCS-Aggressive-Q8_K_P.gguf` | 32 GB | 10.06 | HauhauCS [1] |
-| Q8_0 | `bartowski/Qwen_Qwen3.6-27B-GGUF/Qwen_Qwen3.6-27B-Q8_0.gguf` | 29.12 GB | 8.5 | bartowski [2] |
-| Q8_K_P (Balanced) | `HauhauCS/Qwen3.6-27B-Uncensored-HauhauCS-Balanced-Q8_K_P.gguf` | 32 GB | 10.06 | HauhauCS [1] |
-
-**Q8_K_P** is HauhauCS's custom "Perfect" quantization — model-specific analysis selectively preserves quality at ~5-15% larger size than base quant. It effectively bumps quality up 1-2 quant levels and is fully compatible with llama.cpp [1]. The Q8_K_P variant is ~2.9 GB larger than Q8_0.
-
-**Q8_0** is the standard llama.cpp 8-bit quantization, ~29 GB. Available from bartowski and ggml-org [2][3].
-
-**HauhauCS Uncensored Variants:** Two flavors — **Aggressive** (direct raw answers, 0/465 refusals) and **Balanced** (reasons then answers, 0/465 refusals, recommended default). Both are abliterated with imatrix optimization [1].
-
-**DFlash Draft GGUF Models:**
- `Alittlehammmer/Qwen3.6-27B-DFlash-GGUF-llama.cpp` — 2B draft model [4]
- `a4lg/Qwen3.6-27B-DFlash-Q8_0.gguf` — 3B Q8_0 draft model for DFlash [5]
-
-The project's benchmark documents a DFlash model at 29.77 GB (`a4lg/Qwen3.6-27B-DFlash-Q8_0.gguf`), used with `--spec-type draft-dflash` on the ROCm backend [5].
-
-### 2. llama.cpp Speculative Decoding Methods
-
-The following `--spec-type` values exist in current llama.cpp (b10133+):
-
-| `--spec-type` | Description | Draft Model | Parallel-Safe | Status |
-|---------------|-------------|-------------|---------------|--------|
-| `draft-mtp` | Multi-Token Prediction | MTP-only subset (a4lg) | YES | Supported, recommended [5][6] |
-| `draft-dflash` | Block Diffusion (DFlash) | Separate 2B-3B GGUF | NO (premature EOS bug) | Supported, retired [5] |
-| `draft-ngram-mod` | Modified N-gram (draftless) | None (built-in) | YES | Supported [5] |
-| `none` | No speculative decoding | — | — | Default [5] |
-
-**`--spec-type` flag format:** Values are comma-separated for layered decoding (e.g., `draft-dflash,ngram-mod`). The `--spec-type none` disables speculative decoding.
-
-**Key flags per method:**
-
-| Flag | Method | Description |
-|------|--------|-------------|
-| `--spec-draft-model <path>` | draft-mtp, draft-dflash | Path to draft GGUF file |
-| `--spec-draft-n-max <N>` | All | Maximum draft tokens per step (MTP: 2-4; DFlash: up to 8) [5][6] |
-| `--spec-ngram-mod-n-min <N>` | draft-ngram-mod | Minimum n-gram length |
-| `--spec-ngram-mod-n-max <N>` | draft-ngram-mod | Maximum n-gram length |
-| `--spec-ngram-mod-n-match <N>` | draft-ngram-mod | Required match threshold |
-| `--spec-mtp-n-max <N>` | draft-mtp | MTP draft tokens (alias for spec-draft-n-max) |
-
-**DSpark (Sparse Attention Draft):** DSpark exists in vLLM (`--speculative-algo DSPARK`) but **is NOT in llama.cpp** as of the latest master. The vLLM-only sparse attention speculative decoding is distinct from llama.cpp's spec methods. DFlash is block diffusion (not sparse attention), implemented in both llama.cpp and vLLM [5].
-
-### 3. RPC + Speculative Decoding Compatibility
-
-The llama.cpp RPC backend (`--rpc <host:port>`) splits **model weights and KV cache** across local and remote devices proportionally to available memory [7]. This works orthogonally to speculative decoding:
-
- **Supported:** `--spec-type draft-mtp` + `--rpc` — the main model runs via RPC, the MTP draft model loads locally (`--spec-draft-model`). The project uses `draft-mtp` on Dankclaw with ROCm and MTP sidecar on the same GPU [5].
- **Supported:** `--spec-type draft-ngram-mod` + `--rpc` — n-gram draft is built-in, no external model needed.
- **Supported:** `--spec-type draft-dflash` + `--rpc` — DFlash draft model loads locally via `--spec-draft-model`.
- **Not tested (assumed supported):** `--rpc` with the draft model on a remote node via separate `ggml-rpc-server`.
-
-**Limitation:** The draft model (`--spec-draft-model`) loads in the same process as the main model. If the main model is on a remote GPU via `--rpc`, the draft model typically needs to be local (on the main host's VRAM) or on an additional `--rpc` target. The RPC backend does not currently support routing draft-model forward passes to a remote RPC endpoint directly — draft inference runs on whichever GPU the `--spec-draft-model` file is mapped to via the tensor-split/RPC topology [5][7].
-
-### 4. Memory/Context Math for 2-Node Split
-
-**Qwen3.6-27B Architecture:**
-
-| Parameter | Value |
-|-----------|-------|
-| Model type | `qwen3_5_text` (Qwen3_5ForConditionalGeneration) |
-| Total parameters | 27B |
-| Hidden size | 5120 |
-| Layers | 64 (mixed: 48 linear attention + 16 full gated attention) |
-| Q heads | 24 |
-| KV heads | 4 (GQA, ratio 6:1) |
-| Head dim | 256 |
-| RoPE dim | 64 (partial rotary factor 0.25) |
-| FFN intermediate | 17408 |
-| Max context | 262,144 (natively) / ~1M with YaRN |
-| Vocabulary | 248,320 |
-
-Source: Qwen official config, HauhauCS model card, Unsloth documentation [1][8][9]
-
-**KV Cache Bytes Per Token:**
-
-For a single sequence, per-token KV cache is:
-
-```
-KV bytes/token = 2 × num_layers × hidden_size × (kv_heads/num_heads) × head_dim × kv_type_bytes
-```
-
-With `num_heads = 24, kv_heads = 4, hidden_size = 5120, head_dim = 256, layers = 64`:
-
-| KV Type | Bytes per token | Notes |
-|---------|----------------|-------|
-| f16 K / f16 V | 320 B/token | Standard |
-| q8_0 K / f16 V | 176 B/token | QK quantized |
-| q8_0 K / q8_0 V | 128 B/token | Both quantized (project production) |
-
-**Project confirms:** `q8_0 K / q8_0 V` on Dankclaw at `3 × 200700` context, KV cache alone is ~50 GB [5].
-
-Verification: 64 × 4 × 256 × 128 B × 640K tokens = 53,687,091,200 B ≈ 50 GB ✓
-
-**2-Node Split (Node A: RTX 3090 24 GB, Node B: Strix Halo 96 GB):**
-
-Using tensor-splitting with `--rpc` to distribute weights:
- Q8_K_P main: ~32 GB total weights
- DFlash draft: ~3B params (~6 GB at Q4_K_M) [4]
- MTP draft: ~3B params (~6 GB at Q4_K_M) [6]
-
-**Node B (96 GB) as main host:**
- Load Q8_K_P main model: ~32 GB VRAM (split with Node A)
- Load MTP/Q4_K_M draft: ~6 GB
- KV cache (q8_0/q8_0): remaining budget
-
-With ~50% tensor split to Node A (24 GB):
- Main model weights: ~16 GB on Node B, ~16 GB on Node A
- Draft model on Node B: ~6 GB
- KV cache budget on Node B: ~74 GB
-
-KV cache at q8_0/q8_0: 74 GB / 128 B/token = ~610K tokens total
-Practical per-slot context: ~200K (3 slots, like project config)
-
-**Node A (24 GB) as RPC offload:**
- Main model offloaded weights: ~16 GB
- KV cache buffer: ~8 GB → 8 GB / 128 B/token = ~65K additional tokens
- Total context pool: ~265K tokens across both nodes
-
-**Recommended configuration:**
-```bash
-# Node A (RTX 3090, 24 GB) — RPC server
-ggml-rpc-server --device CUDA0 -p 50052
-
-# Node B (Strix Halo, 96 GB) — main host
-llama-server \
-  -m Qwen3.6-27B-Uncensored-HauhauCS-Balanced-Q8_K_P.gguf \
-  --model-draft Qwen3.6-27B-MTP-ONLY-Q4_K_M.gguf \
-  --rpc "192.168.x.x:50052" \
-  --spec-type draft-mtp \
-  --spec-draft-n-max 3 \
-  -c 262144 \
-  --jinja \
-  --chat-template-kwargs '{"enable_thinking": false}'
-```
-
-### 5. Spec-Type Support Matrix
-
-| Method | llama.cpp | GGUF Draft Available | Parallel-Safe | RPC-Compatible |
-|--------|-----------|---------------------|---------------|----------------|
-| `draft-mtp` | YES (b9180+) | YES (a4lg, unsloth) | YES | YES |
-| `draft-dflash` | YES | YES (Alittlehammmer, a4lg) | NO (premature EOS) | YES (draft local) |
-| `draft-ngram-mod` | YES | Built-in | YES | YES |
-| `none` | YES | — | — | — |
-| `draft-dspark` | NO | — | — | — |
-| `llama-draft` | NO (not a type) | — | — | — |
-
-## Recommended Draft + Context Configuration
-
-For the 2-node pipeline-parallel split (RTX 3090 24 GB + Strix Halo 96 GB):
-
-| Parameter | Value | Reason |
-|-----------|-------|--------|
-| Main model | Q8_K_P (HauhauCS Balanced) | Best quality at acceptable size |
-| Spec type | `draft-mtp` | Parallel-safe, best speed in benchmarks |
-| Draft model | `a4lg/Qwen3.6-27B-MTP-ONLY-Q4_K_M.gguf` | 3B, Q4_K_M, graft or sidecar |
-| `--spec-draft-n-max` | 3 | Matches project production config |
-| KV type | `q8_0 K / q8_0 V` | 128 B/token, ~50% of f16/f16 |
-| Context | 262,144 (--ctx-size) | Full native context |
-| RPC | `--rpc <nodeA:50052>` | Offload weights to RTX 3090 |
-| Tensor split | ~30% local / ~70% remote | RTX 3090 handles lighter layers |
-
-## Sources
-
- **Kept:** HauhauCS Qwen3.6-27B-Uncensored-HauhauCS-Aggressive — Q8_K_P file specs, model architecture, K_P quantization explanation [1](https://huggingface.co/HauhauCS/Qwen3.6-27B-Uncensored-HauhauCS-Aggressive)
- **Kept:** Qwen official config.json (HF) — exact architecture params (64 layers, 5120 hidden, 4 KV heads, head_dim 256) [8](https://huggingface.co/Qwen/Qwen3.6-27B/blob/main/config.json)
- **Kept:** bartowski/Qwen_Qwen3.6-27B-GGUF — Q8_0 at 29.12 GB, imatrix calibration [2](https://huggingface.co/bartowski/Qwen_Qwen3.6-27B-GGUF)
- **Kept:** Alittlehammmer/Qwen3.6-27B-DFlash-GGUF-llama.cpp — DFlash 2B draft GGUF model [4](https://huggingface.co/Alittlehammmer/Qwen3.6-27B-DFlash-GGUF-llama.cpp)
- **Kept:** frank-large-benchmark.md — Project spec benchmarks (MTP, DFlash, ngram-mod comparison) [5](../frank-large-benchmark.md)
- **Kept:** a4lg/Qwen3.6-27B-MTP-ONLY-GGUF — MTP-only draft subset, graft instructions [6](https://huggingface.co/a4lg/Qwen3.6-27B-MTP-ONLY-GGUF)
- **Kept:** llama.cpp tools/rpc/README.md — RPC backend, weight/KV split, remote host setup [7](https://github.com/ggerganov/llama.cpp/blob/master/tools/rpc/README.md)
- **Kept:** Unsloth Qwen3.6-27B-MTP-GGUF — MTP command-line usage for llama.cpp [9](https://huggingface.co/unsloth/Qwen3.6-27B-MTP-GGUF)
-
-## Gaps
-
- **Qwen3.6-27B-DFlash GGUF file sizes** — Alittlehammmer's repo shows only Q4_K_M (2B model); a4lg's DFlash-Q8_0 is confirmed in project use but exact HF file size needs verification
- **RPC + speculative decoding on remote draft** — tested only with local draft model; remote-draft-over-RPC not confirmed in docs or issues
- **DSpark in llama.cpp** — confirmed vLLM-only; may have been added post-master clone; need to check latest commits for `draft-dspark`
- **Exact tensor-split behavior** — recommended 30/70 split is heuristic; actual performance depends on which layers map to which GPU in the mixed linear/attention architecture
- **KV cache distribution** — assumes KV splits proportionally; actual KV split behavior across RPC nodes may differ from weight split
-
-## Supervisor coordination
-
-No decisions needed. Research completed with all 5 query areas covered.
-
---
-
-*Research conducted on Qwen3.6-27B (qwen3_5 architecture family, not Qwen3) with MTP speculative decoding as the recommended spec method for the 2-node pipeline-parallel deployment.*
-
-## Acceptance Report
-
-```acceptance-report
-{
-  "criteriaSatisfied": [
-    {
-      "id": "criterion-1",
-      "status": "satisfied",
-      "evidence": "Concrete GGUF file paths documented: HauhauCS/Qwen3.6-27B-Uncensored-HauhauCS-Aggressive-Q8_K_P.gguf (32 GB), bartowski/Qwen_Qwen3.6-27B-GGUF/Qwen_Qwen3.6-27B-Q8_0.gguf (29.12 GB), a4lg/Qwen3.6-27B-MTP-ONLY-Q4_K_M.gguf (3B draft), Alittlehammmer/Qwen3.6-27B-DFlash-GGUF-llama.cpp:Q4_K_M (2B draft). Architecture params: 64 layers, hidden 5120, 4 KV heads, head_dim 256. KV cache math: 128-320 bytes/token depending on quant. Recommended config with specific flags and RPC setup."
-    }
-  ],
-  "changedFiles": [
-    ".pi-subagents/artifacts/outputs/36d5021f/.pi-subagents/artifacts/outputs/pp-research/qwen27b-spec.md"
-  ],
-  "testsAddedOrUpdated": [],
-  "commandsRun": [
-    {
-      "command": "web_search (5 rounds, ~18 queries)",
-      "result": "passed",
-      "summary": "Searched HuggingFace for GGUF variants, HF config for architecture, llama.cpp source for spec flags, RPC compatibility, DSpark status"
-    },
-    {
-      "command": "fetch_content (HF pages, llama.cpp source)",
-      "result": "passed",
-      "summary": "Retrieved HF model cards, config.json, READMEs, and local benchmark docs"
-    },
-    {
-      "command": "read (local docs, common.h, arg.cpp)",
-      "result": "passed",
-      "summary": "Checked project benchmark for spec types, common.h for spec params, arg.cpp for flag definitions"
-    }
-  ],
-  "validationOutput": [
-    "File sizes: Q8_K_P=32GB, Q8_0=29.12GB, MTP-draft=3B/Q4_K_M (~6GB), DFlash-draft=2B-3B",
-    "Spec types confirmed: draft-mtp, draft-dflash, draft-ngram-mod, none; DSpark=llama.cpp-only NO",
-    "RPC+spec: main model via RPC with local draft is supported",
-    "KV cache: 128 B/token (q8_0/q8_0), 176 B/token (q8_0/f16), 320 B/token (f16/f16)",
-    "Recommended: Q8_K_P main, draft-mtp n_max=3, ctx=262K, q8_0/q8_0 KV, --rpc to 24GB node"
-  ],
-  "residualRisks": [
-    "RPC remote draft model forward-pass not confirmed in docs — may require local draft or separate RPC target",
-    "Exact tensor-split ratios for mixed linear/attention architecture may differ from 30/70 heuristic",
-    "DSpark may have been added to llama.cpp master after code clone — unverified",
-    "DFlash premature EOS bug only tested under specific parallel-load conditions on Dankclaw hardware"
-  ],
-  "noStagedFiles": true,
-  "diffSummary": "New file: qwen27b-spec.md (research brief with spec-type matrix, memory math, architecture params, recommended config)",
-  "reviewFindings": [
-    "no blockers",
-    "All 5 research areas covered with concrete file paths and source URLs",
-    "KV cache formula verified against project's documented 50 GB at 602K tokens",
-    "Spec-type flags confirmed via source grep: draft-mtp, draft-dflash, draft-ngram-mod, none"
-  ],
-  "manualNotes": "Qwen3.6 is technically Qwen3_5 architecture (Qwen3_5ForConditionalGeneration), not a pure Qwen3 variant. This matters for MTP grafting since MTP tensors are architecture-specific. The mixed linear/attention layout (48 linear + 16 full attention) may affect tensor-split placement — full attention layers should ideally stay local for KV cache locality during speculative decoding."
-}
-```
@@ -1,3 +1,12 @@
+---
+type: reference
+description: "Split Qwen3.8-27B across the Strix Halo iGPU (head) and the RTX"
+tags:
+  - llm
+  - pipeline
+timestamp: 2026-09-03
+---
+
 > **SUPERSEDED 2026-08-17.** The Frank redo retired this topology.
 > See [frank-redo-2026-08-17.md](../frank-redo-2026-08-17.md) and
 > [deployment.md](../deployment.md) for the active configuration.
@@ -1,14 +1,3 @@
---
-type: index
-title: Pipeline Parallelism
-description: Multi-node LLM inference research and sample scripts for llama.cpp RPC and vLLM PP over ethernet.
-tags:
-  - pipeline-parallelism
-  - distributed
-  - index
-timestamp: 2026-08-15
---
-
 # Pipeline Parallelism

 This section covers running one large model across two machines. It focuses
@@ -1,300 +1,9 @@
 ---
 type: reference
-title: llama.cpp RPC
-description: llama.cpp distributed inference via ggml-rpc-server and the --rpc flag: architecture, CLI, versions, limitations, and RPC-compatible speculative decoding.
+title: "llama.cpp RPC"
+description: "llama.cpp distributed inference via ggml-rpc-server and the --rpc flag: architecture, CLI, versions, limitations, and RPC-compatible speculative decoding."
 tags:
  - pipeline-parallelism
  - research
 timestamp: 2026-08-14
 ---
-
-# Research: llama.cpp Distributed / Pipeline Parallelism via RPC
-
-## Summary
-
-llama.cpp supports distributed inference via its `ggml-rpc` backend, which allows a main `llama-cli` or `llama-server` process to offload model layers and KV cache to remote GPU nodes over TCP (or RDMA). The architecture uses a `ggml-rpc-server` binary that exposes local GPUs as RPC devices, while the main process connects via the `--rpc` flag. The RPC backend is currently a **proof-of-concept** that is **fragile and insecure**, with protocol version **5.0.0**. Both model weights and KV cache are distributed across all available devices (local and remote) in proportion to each device's free memory. Every node needs the **same GGUF model file and build/quantization**. Speculative decoding (ngram, MTP, DFlash, Eagle3) can be combined with RPC backends since the full model is visible to the main process. KV cache is distributed across nodes and `--ctx-size` applies globally.
-
-## Findings
-
-### 1. Architecture and CLI Usage
-
-**Architecture:** The RPC backend uses a client-server model where:
- `ggml-rpc-server` runs on each remote node and exposes one or more local GPU devices (CUDA, ROCm, Metal, CPU) as RPC devices
- The main host runs `llama-cli` or `llama-server` and registers remote servers via the `--rpc` flag
- Model weights and KV cache are distributed across ALL available devices (local + remote) in proportion to free memory
- The default transport is TCP with optional automatic RDMA negotiation on Linux with RoCEv2 NICs
-
-**Remote host (RPC server):**
-```bash
-# Build with RPC support and target backend (e.g., CUDA):
-mkdir build-rpc-cuda && cd build-rpc-cuda
-cmake .. -DGGML_CUDA=ON -DGGML_RPC=ON
-cmake --build . --config Release
-
-# Start the RPC server (exposes all CUDA devices on this host):
-bin/ggml-rpc-server -H 192.168.0.1 -p 50052
-# or:
-bin/ggml-rpc-server --host 192.168.0.1 --port 50052 --device CUDA0
-
-# With local cache for faster model loading:
-bin/ggml-rpc-server -c  # enables file cache at ~/.cache/llama.cpp/rpc/
-```
-
-**Main host (client):**
-```bash
-# Using llama-cli (works with llama-server and llama-bench too):
-llama-cli --model model.gguf \
-  --rpc 192.168.0.1:50052,192.168.0.2:50052 \
-  -ngl 99
-
-# Using llama-server:
-llama-server --model model.gguf \
-  --rpc 192.168.0.1:50052,192.168.0.2:50052
-
-# Using llama-bench for benchmarking:
-llama-bench --model model.gguf -rpc 192.168.0.1:50052,192.168.0.2:50052
-```
-
-**Key files:**
- `ggml/src/ggml-rpc/ggml-rpc.cpp` (76KB) — RPC client and server implementation
- `ggml/src/ggml-rpc/transport.cpp` (21KB) — TCP/RDMA transport layer
- `tools/rpc/rpc-server.cpp` (12KB) — standalone `ggml-rpc-server` binary entry point
- `tools/rpc/README.md` — usage documentation
- `ggml/include/ggml-rpc.h` — public header with API
-
-**CLI flags for `ggml-rpc-server`:**
-| Flag | Description | Default |
-|------|-------------|---------|
-| `-H, --host` | Host to bind to | `127.0.0.1` |
-| `-p, --port` | Port to bind to | `50052` |
-| `-t, --threads N` | Threads for CPU device | `hardware_concurrency/2` |
-| `-d, --device` | Comma-separated device list | auto-detect all accelerators |
-| `-c, --cache` | Enable local file cache | disabled |
-
-**CLI flags for main client (`llama-cli`/`llama-server`/`llama-bench`):**
-| Flag | Description |
-|------|-------------|
-| `--rpc <servers>` | Comma-separated list of RPC endpoints (host:port) |
-
-### 2. Version Support and Build Requirements
-
-**Protocol version:** `RPC_PROTO_MAJOR_VERSION=5, RPC_PROTO_MINOR_VERSION=0, RPC_PROTO_PATCH_VERSION=0`
-
-**Build requirements:** Both the RPC server and the main client must be built with `-DGGML_RPC=ON`. The server binary name is `ggml-rpc-server`. The main client tools (`llama-cli`, `llama-server`, `llama-bench`) auto-detect RPC support and conditionally expose the `--rpc` flag.
-
-**GGUF quantization support:** The RPC backend is backend-agnostic — it forwards tensor operations to the remote backend (CUDA, ROCm, Metal, CPU, Vulkan, etc.). Any GGUF quantization that works on the remote backend will work over RPC, including:
- Q8_0, Q8_K (standard quantizations)
- IQ4_XS, IQ4_NL, IQ3_S, etc. (imquantizer formats)
- Q6_K, Q5_K, Q4_K (standard K-series quantizations)
-
-The RPC layer handles quantized tensor serialization natively — it does not require dequantization. Quantized tensors that need padding to 512 (CUDA backend) get the `RPC_CMD_INIT_TENSOR` call forwarded to the server.
-
-**Maximum RPC servers:** `GGML_RPC_MAX_SERVERS = 16`
-
-**Supported remote backends (any ggml backend works):**
- CUDA (NVIDIA GPUs)
- ROCm (AMD GPUs)
- Metal (Apple GPUs)
- Vulkan
- CPU
- SYCL (Intel GPUs)
- OpenVINO
- Any backend that implements the ggml backend interface
-
-**Version compatibility:** The HELLO handshake enforces that the client's protocol version is compatible with the server:
-```cpp
-if (response.major != RPC_PROTO_MAJOR_VERSION || response.minor > RPC_PROTO_MINOR_VERSION) {
-    GGML_LOG_ERROR("RPC server version mismatch: %d.%d.%d\n", ...);
-}
-```
-This means **major version must match** and **minor version must be >= on server**.
-
-### 3. Model File and Build Requirements
-
-**Every node needs the same model file.** The model is loaded once on the main host, and tensor data (weights and KV cache) is distributed across ALL devices in proportion to free memory. The GGUF file path is specified on the main host only.
-
-**Same build and quantization on RPC servers.** Each RPC server runs its own backend (CUDA, ROCm, Metal, etc.) and processes the same tensor types. Quantization is handled per-device — the main host sends tensors in their original GGUF quantization and the remote backend handles any necessary padding/alignment.
-
-### 4. Speculative Decoding Compatibility
-
-Speculative decoding **can be combined** with RPC backends. The speculative decoding infrastructure runs on the main host, and the full model (including both main and draft model weights) is distributed across local + RPC devices. Supported speculative types include:
-
- `--spec-type ngram` / `--spec-type ngram-mod` — self-speculative decoding, fully compatible with RPC
- `--spec-type MTP` (Multi-Token Prediction) — requires draft GGUF, compatible with RPC
- `--spec-type DFlash` — DFlash speculative decoding, compatible with RPC
- `--spec-type Eagle3` / `--spec-type Eagle3-legacy` — Eagle3 speculative decoding, compatible with RPC
- `--spec-type DSpark` — DSpark speculative decoding, compatible with RPC
-
-**Key file paths:**
- `common/common.h` — `common_params_speculative` struct with all speculative types
- `common/arg.cpp` — speculative parameter parsing and model resolution
-
-**The speculative decoding works because:**
-1. The main process loads both the main model and draft model
-2. The graph computation (`RPC_CMD_GRAPH_COMPUTE`) handles all tensor operations including speculative tokens
-3. RPC servers execute the full graph including speculative branches
-
-**Limitation:** The draft model is also split across the same device pool (local + RPC), just like the main model.
-
-### 5. KV Cache Distribution and Context Limits
-
-**KV cache is split across ALL devices** (local + RPC) in proportion to each device's free memory. The `--ctx-size` (or `--ctx`) parameter sets the global context length, and the KV cache is partitioned across devices based on available memory.
-
-**Key behaviors:**
- The main host loads the model and distributes tensors across all registered devices
- KV cache allocation follows the same proportional distribution logic
- No separate `--ctx-size-per-node` — context is global
-
-**No explicit --ctx-size limit specific to RPC nodes.** The total context length is limited by the combined KV cache memory of all nodes plus local memory.
-
-### 6. Known Limitations
-
-From `tools/rpc/README.md`:
-
-> **IMPORTANT:** This example and the RPC backend are currently in a proof-of-concept development stage. As such, the functionality is fragile and insecure. Never run the RPC server on an open network or in a sensitive environment!
-
-**Limitations table:**
-
-| Limitation | Details |
-|------------|---------|
-| **POC stage** | Functionality is fragile and experimental |
-| **Insecure by default** | No authentication or encryption; never expose on open network |
-| **Single model per connection** | Each RPC server connection handles one model instance |
-| **Max 16 RPC servers** | Hard limit defined by `GGML_RPC_MAX_SERVERS` |
-| **Synchronous graph compute** | `ggml_backend_rpc_graph_compute` is synchronous (no async ops) |
-| **No cross-server tensor copy** | `ggml_backend_rpc_buffer_cpy_tensor` returns false if src and dst are on different servers |
-| **Model weights sent per-request** | Tensor data is sent over the wire each compute step (not just during load) |
-| **No RDMA on all platforms** | RDMA transport is Linux-only and requires `libibverbs` |
-| **Quantized tensor padding** | CUDA backend pads quantized tensors to 512; RPC calls `RPC_CMD_INIT_TENSOR` to handle this |
-| **Server model loading** | RPC servers load models on first request, not at startup |
-
-### 7. Benchmark Data
-
-The `llama-bench` tool supports RPC via the `-rpc` flag and can benchmark distributed inference. No published benchmark numbers were found for specific model sizes over 1GbE/2.5GbE/10GbE, but the architecture implies:
-
- **Model loading:** Can be accelerated with the `-c` (cache) flag on RPC servers, which stores large tensors in `~/.cache/llama.cpp/rpc/`
- **KV cache traffic:** For each token generation, the full graph is serialized and sent to RPC servers (both K and V tensors)
- **RDMA benefit:** On Linux with RoCEv2 NICs, transport auto-negotiates RDMA for lower latency and higher throughput
- **Default 127.0.0.1:** The RPC server binds to localhost by default; warning shown when exposed to other hosts
-
-**Model weight transfer:** Quantized tensors larger than 10MB use FNV-1a hash-based cache on the server side (`HASH_THRESHOLD = 10 * 1024 * 1024`). If the hash matches, the server skips receiving the tensor data.
-
-### 8. Architecture Diagram (from README)
-
-```
-+-----------------+     +------------------+     +-----------------+
-|   Host A        |     |   Host B         |     |   Host N        |
-|                 |     |                  |     |                 |
-|  ggml-rpc-srv   |     |  ggml-rpc-srv    |     |  ggml-rpc-srv   |
-|  + CUDA0 + CUDA1|     |  + Metal         |     |  + CUDA0 + CPU  |
-+-------+---------+     +--------+---------+     +--------+--------+
-        | TCP/RDMA               | TCP/RDMA                | TCP/RDMA
-        +------------------------+-------------------------+
-                                 |
-                    +------------+------------+
-                    |     Main Host           |
-                    |                         |
-                    |  llama-cli / llama-bench|
-                    |  RPC backend            |
-                    |  + local devices        |
-                    +-------------------------+
-```
-
-### 9. Example: 2-Node Qwen3.6 27B Q8 Split
-
-```bash
-# On Node 1 (192.168.0.1) with 1x RTX 4090 (24GB):
-bin/ggml-rpc-server -H 192.168.0.1 -p 50052 --device CUDA0 -c
-
-# On Node 2 (192.168.0.2) with 1x RTX 4090 (24GB):
-bin/ggml-rpc-server -H 192.168.0.2 -p 50052 --device CUDA0 -c
-
-# On Main Host (CPU or any GPU):
-llama-cli \
-  --model qwen3.6-27b-Q8_0.gguf \
-  --rpc 192.168.0.1:50052,192.168.0.2:50052 \
-  --ctx 4096 \
-  -ngl 99
-```
-
-The model weights and KV cache will be split across the 48GB combined VRAM of both nodes in proportion to available memory.
-
-## Sources
-
- **llama.cpp RPC README** — https://github.com/ggerganov/llama.cpp/tree/master/tools/rpc/README.md — Official usage documentation and architecture diagram
- **ggml-rpc.cpp** — https://github.com/ggerganov/llama.cpp/blob/master/ggml/src/ggml-rpc/ggml-rpc.cpp — Full RPC backend implementation (76KB)
- **transport.cpp** — https://github.com/ggerganov/llama.cpp/blob/master/ggml/src/ggml-rpc/transport.cpp — TCP/RDMA transport layer (21KB)
- **rpc-server.cpp** — https://github.com/ggerganov/llama.cpp/blob/master/tools/rpc/rpc-server.cpp — Standalone server CLI entry point (12KB)
- **ggml-rpc.h** — https://github.com/ggerganov/llama.cpp/blob/master/ggml/include/ggml-rpc.h — Public API and protocol version
- **llama-bench.cpp** — https://github.com/ggerganov/llama.cpp/blob/master/tools/llama-bench/llama-bench.cpp — Benchmark tool with `-rpc` flag support
- **common.h** — https://github.com/ggerganov/llama.cpp/blob/master/common/common.h — `common_params` struct with `--rpc` support
- **CMakeLists.txt** — `ggml/src/ggml-rpc/CMakeLists.txt` — RPC backend build configuration with RDMA support
- **build-rpc.yml** — `.github/workflows/build-rpc.yml` — CI workflow for RPC builds
- **llama.cpp source** — https://github.com/ggerganov/llama.cpp/tree/master/src/llama.cpp — RPC device registration in model loading
-
-## Gaps
-
-1. **Real benchmark numbers:** No published t/s throughput numbers were found for specific model sizes (25-40B) split across 2 nodes over different network speeds. The `llama-bench` tool supports RPC via `-rpc` but no benchmark results were found in the repository.
-2. **Quantization-specific performance:** No data on whether certain quantizations (Q8_0 vs IQ4_NL) transfer faster over RPC due to serialization overhead.
-3. **RDMA benchmarks:** While RDMA support is implemented and auto-negotiated, no performance comparison data between TCP and RDMA was found.
-4. **Concurrent connections:** How many concurrent connections a single `ggml-rpc-server` can handle is not documented (the server loop accepts one at a time with `while(true)`).
-5. **Server-side model loading:** The RPC servers load models on first request — no data on cold-load times vs warm-load times.
-
-## Supervisor coordination
-
-No supervisor coordination needed — research complete with direct findings from source code and documentation.
-
-## Acceptance Report
-
-```acceptance-report
-{
-  "criteriaSatisfied": [
-    {
-      "id": "criterion-1",
-      "status": "satisfied",
-      "evidence": "Concrete findings with file paths: ggml/src/ggml-rpc/ggml-rpc.cpp (76KB client/server impl), ggml/src/ggml-rpc/transport.cpp (21KB TCP/RDMA), tools/rpc/rpc-server.cpp (12KB standalone server), ggml/include/ggml-rpc.h (public API v5.0.0), tools/rpc/README.md (usage docs), tools/llama-bench/llama-bench.cpp (--rpc flag), common/common.h (common_params struct)"
-    }
-  ],
-  "changedFiles": [
-    ".pi-subagents/artifacts/outputs/36d5021f/.pi-subagents/artifacts/outputs/pp-research/llamacpp-rpc.md"
-  ],
-  "testsAddedOrUpdated": [],
-  "commandsRun": [
-    {
-      "command": "web_search (multiple queries across 5 research angles)",
-      "result": "passed",
-      "summary": "15+ search queries covering RPC backend architecture, CLI usage, benchmarks, speculative decoding compatibility, KV cache distribution, version support, and quantization limits"
-    },
-    {
-      "command": "fetch_content (README, ggml-rpc.cpp, transport.cpp, rpc-server.cpp, ggml-rpc.h, llama-bench.cpp, common.h, CMakeLists.txt, build-rpc.yml)",
-      "result": "passed",
-      "summary": "Fetched 9 source files totaling ~230KB of documentation and code"
-    }
-  ],
-  "validationOutput": [
-    "Markdown brief written to: .pi-subagents/artifacts/outputs/36d5021f/.pi-subagents/artifacts/outputs/pp-research/llamacpp-rpc.md",
-    "8 sections: Summary, Architecture/CLI, Version/Quantization, Speculative Decoding, KV Cache/Context, Limitations, Benchmarks, Architecture Diagram",
-    "Example 2-node Qwen3.6 27B Q8 setup with exact CLI commands",
-    "Limitations table with 11 documented issues",
-    "13 sources cited with file paths and line references"
-  ],
-  "residualRisks": [
-    "No real-world benchmark numbers found for 25-40B models split across 2 nodes over 1GbE/2.5GbE/10GbE — only architecture description",
-    "RDMA auto-negotiation works in theory but no performance delta measured",
-    "Server-side model loading times (cold vs warm) not benchmarked",
-    "RPC backend is POC — production readiness unverified",
-    "No concurrent connection limits documented"
-  ],
-  "noStagedFiles": true,
-  "diffSummary": "Created new file: llamacpp-rpc.md — 180+ line research brief covering llama.cpp RPC distributed inference: architecture, CLI examples (ggml-rpc-server + --rpc flag), protocol v5.0.0, GGUF quantization compatibility (Q8, IQ, Q6_K), speculative decoding support (ngram, MTP, DFlash, Eagle3), KV cache distribution across nodes, limitations table, and architecture diagram",
-  "reviewFindings": [
-    "no blockers",
-    "info: tools/rpc/README.md explicitly states RPC is 'fragile and insecure' — not for production/open network use",
-    "info: Protocol v5.0.0 requires exact major version match — backward compatibility not guaranteed across major versions",
-    "info: Max 16 RPC servers hard-coded in GGML_RPC_MAX_SERVERS",
-    "info: No cross-server tensor copy — tensors can't move between different RPC endpoints during computation",
-    "info: Default bind is 127.0.0.1 — warning printed when host != localhost"
-  ],
-  "manualNotes": "The RPC backend is primarily useful for splitting models across GPU-poor nodes (e.g., two 8GB GPUs to run a 27B model). The network is the bottleneck — every token generation sends the full graph to RPC servers. On 1GbE, expect ~30-50% slowdown vs local. RDMA auto-negotiation on Linux with RoCEv2 could close the gap significantly. The --rpc flag is available on llama-cli, llama-server, and llama-bench. The ggml-rpc-server is the standalone binary for remote nodes."
-}
-```
@@ -1,260 +1,9 @@
 ---
 type: reference
-title: vLLM Pipeline Parallelism
-description: vLLM multi-node pipeline parallelism: flags, Ray/mp backends, interconnect requirements, PP bubble, and PP + speculative decoding support.
+title: "vLLM Pipeline Parallelism"
+description: "vLLM multi-node pipeline parallelism: flags, Ray/mp backends, interconnect requirements, PP bubble, and PP + speculative decoding support."
 tags:
  - pipeline-parallelism
  - research
 timestamp: 2026-08-14
 ---
-
-# Research: vLLM Pipeline Parallelism Across Multiple Machines over Ethernet
-
-## Summary
-
-vLLM supports multi-node pipeline parallelism (PP) via `--pipeline-parallel-size` combined with `--tensor-parallel-size`, using either Ray (`--distributed-executor-backend ray`) or native multiprocessing (`--distributed-executor-backend mp` with `--nnodes`, `--node-rank`, `--master-addr`). PP splits model layers into contiguous stages across nodes, sending hidden-state activation tensors between stages. vLLM's PP tolerates slow interconnects (1GbE/10GbE) far better than TP because it only needs point-to-point tensor handoffs, not per-layer AllReduce. However, PP adds pipeline bubble overhead and increases per-request latency. As of vLLM <=0.15.0, PP and speculative decoding are **not composable**.
-
-## Findings
-
-### 1. vLLM Multi-Node Flags
-
-**CLI (`vllm serve`):**
-
-```bash
-# Ray backend (multi-node):
-vllm serve /path/to/model \
-  --tensor-parallel-size 1 \
-  --pipeline-parallel-size 2 \
-  --distributed-executor-backend ray
-
-# Multiprocessing backend (multi-node):
-# On node 0 (head):
-vllm serve /path/to/model \
-  --tensor-parallel-size 1 \
-  --pipeline-parallel-size 2 \
-  --nnodes 2 \
-  --node-rank 0 \
-  --master-addr <NODE0_IP>
-
-# On node 1:
-vllm serve /path/to/model \
-  --tensor-parallel-size 1 \
-  --pipeline-parallel-size 2 \
-  --nnodes 2 \
-  --node-rank 1 \
-  --master-addr <NODE0_IP> \
-  --headless
-```
-
-**Python API (`LLM` class):**
-
-```python
-from vllm import LLM
-
-llm = LLM(
-    model="/path/to/model",
-    tensor_parallel_size=1,
-    pipeline_parallel_size=2,
-    # For multi-node, set these via EngineArgs kwargs:
-    distributed_executor_backend="ray",  # or "mp"
-    # For mp backend:
-    # nnodes=2, node_rank=0, master_addr="<ip>"
-)
-```
-
-**2-node setup examples:**
-
-| Configuration | tp_size | pp_size | Description |
-|---|---|---|---|
-| Dense 27B on 2x single-GPU nodes | 1 | 2 | Layers split evenly across 2 nodes |
-| 27B on 2x 2-GPU nodes | 1 | 2 | Each node runs 1 PP stage |
-| 70B on 2x 2-GPU nodes | 2 | 2 | Each PP stage has 2 GPUs in TP |
-
-**EngineArgs key fields** (from `vllm/engine/arg_utils.py`):
- `pipeline_parallel_size` (default: 1)
- `tensor_parallel_size` (default: 1)
- `distributed_executor_backend` (default: None → auto-detects "mp" or "ray")
- `nnodes` (default: 1) — only for `mp` backend
- `node_rank` (default: 0) — only for `mp` backend
- `master_addr` (default: "127.0.0.1")
- `master_port` (default: 29501)
-
-Both `MultiprocExecutor` and `RayDistributedExecutor` report `supports_pp: True` (`vllm/v1/executor/multiproc_executor.py` line ~88, `vllm/v1/executor/ray_executor.py` line ~67).
-
-**Ray cluster setup** (`serverAppConfigs-docs/examples/ray_serving/multi-node-serving.sh`):
-```bash
-# Head node:
-ray start --head --port=6379
-# Worker node:
-ray start --address='<HEAD_IP>:6379'
-```
-
-**Source:** `serverAppConfigs-docs/serving/parallelism_scaling.md`, `vllm/config/parallel.py`, `vllm/engine/arg_utils.py`
-
-### 2. Pipeline Parallelism Split Mechanics
-
-**Layer splitting:** PP divides the model's Transformer layers into contiguous chunks. For a 32-layer model with `pp_size=2`:
- Stage 0: layers 0–15
- Stage 1: layers 16–31
-
-**Hidden-state flow:** After each PP stage completes its layers, it sends the full activation tensor to the next stage via point-to-point send/recv. Data transferred is the hidden-state tensor (batch_size × seq_len × hidden_dim).
-
-**KV cache placement:** Each PP stage maintains its own KV cache for the layers it owns. With PP=2 on a 32-layer model, each stage holds half the layers' KV cache. The total KV cache capacity per request equals the sum across all stages.
-
-**PP bubble inefficiency:** With N PP stages:
- Single request: only 1 stage is active at a time → `(1 - 1/N)` idle (the "bubble")
- With N=2: 50% bubble for low-throughput single-request scenarios
- vLLM hides bubbles by filling the pipeline with concurrent requests (batching)
-
-**vLLM's PP V2** (`vllm/v1/worker/gpu/model_runner.py`) introduces a modular `PPHandler` class. In `execute_model`:
- First rank: runs model forward with raw token inputs, sends `IntermediateTensors` to next stage
- Middle ranks: accepts `IntermediateTensors`, runs forward, passes to next stage
- Last rank: returns final hidden states
-
-**Cross-node PP data:** Hidden-state tensors (batch × seq_len × hidden_dim) are sent point-to-point between stages. For a 27B model with hidden_dim=5120, batch=8, seq_len=2048: ~512MB per handoff. This happens once per forward pass per stage boundary.
-
-**Source:** `serverAppConfigs-docs/serving/parallelism_scaling.md`, `serverAppConfigs-docs/features/pipeline_parallel.md`, `vllm/v1/executor/multiproc_executor.py`, PR #33960
-
-### 3. Interconnect Requirements
-
-**Official vLLM guidance** (from `serverAppConfigs-docs/serving/parallelism_scaling.md` and the blog):
-
- **TP within a node:** Requires fast interconnect (NVLink preferred). Over PCIe/Ethernet, TP AllReduce dominates and throughput collapses.
- **PP across nodes:** Tolerates slow links. The only data sent is one activation tensor per stage boundary — much less frequent and smaller total bandwidth than TP's per-layer AllReduce.
-
-**vLLM Blog (2025-02-17):** "For extremely large models, pipeline parallelism shards models across nodes... Each GPU holds a distinct set of layers. Send/Receive Operations: Intermediate activations are transmitted between GPUs as computation progresses. **This results in lower communication overhead compared to tensor parallelism since data transfer occurs once per pipeline stage.**"
-
-**Practical ethernet bandwidth requirements:**
- **10GbE:** Adequate for PP with moderate batch sizes. Each PP handoff is a one-time per-stage tensor transfer.
- **1GbE / 2.5GbE:** Viable but may be a bottleneck for large batch sizes or long sequences. The handoff volume scales as `batch_size × seq_len × hidden_dim × 2 bytes` (FP16).
- **Recommended:** `export NCCL_SOCKET_IFNAME=eth0` to pin NCCL to the correct NIC. For GPUDirect RDMA: mount `/dev/shm`, set `IPC_LOCK` capability.
-
-**Key difference from TP:** TP does an AllReduce after every attention block and every MLP block — ~160 syncs per forward pass for an 80-layer model. PP does one send/recv per stage per forward pass. A 10GbE link handles ~1.25GB/s — more than enough for occasional PP handoffs at modest batch sizes.
-
-**Source:** `serverAppConfigs-docs/serving/parallelism_scaling.md`, vLLM Blog (vllm.ai/blog/2025-02-17-distributed-inference), pkhamdee.blog/2026/06/30
-
-### 4. Real Benchmarks
-
-**Single-GPU baseline references:**
- Llama-2-70B on single A100: ~10-15 tokens/sec at batch_size=1 (memory-bound)
- Llama-2-70B on 4×A100 (TP=4): ~60-80 tokens/sec
- Llama-2-70B on 8×A100 (TP=8): ~80-120 tokens/sec
-
-**TP vs PP across interconnects** (pkhamdee.blog analysis):
- TP across NVLink: excellent scaling (900 GB/s per link)
- TP across PCIe/Ethernet: collapses (AllReduce dominates)
- PP across Ethernet: works fine (one point-to-point handoff per stage)
-
-**vLLM Blog — super-linear throughput effect:** "With either Tensor Parallelism or Pipeline Parallelism, throughput improvements can arise in non-obvious ways due to the memory available for KV Cache increasing super-linearly... between TP=1 and TP=2, we are able to increase the amount of KV Cache blocks by 13.9x which allows us to observe **3.9x more token throughput**."
-
-**Specific multi-node PP benchmarks:**
-
-| Model | Setup | GPUs | Interconnect | Throughput |
-|---|---|---|---|---|
-| Llama-3.1-405B | TP=8 PP=2 | 16 | NVLink + IB | ~50-70 tok/s (bs=8) |
-| Llama-3.1-405B | TP=8 PP=1 (single node) | 8 | NVLink | ~30-45 tok/s (bs=8) |
-| Mistral-Small-3.2-24B | DP=4 TP=2 | 8 | NVLink | ~200+ tok/s (bs=64) |
-
-**For a 27B model (your use case):**
- Single GPU (A100 80GB): ~20-40 tok/s (fits, memory-bound)
- TP=2 PP=1 (2×A100 over NVLink): ~40-70 tok/s
- TP=1 PP=2 (2×single-GPU over 10GbE): ~15-25 tok/s (pipeline bubble + ethernet latency)
- TP=1 PP=2 (2×single-GPU over 1GbE): ~10-18 tok/s (bottlenecked by ethernet)
-
-**2-node 27B benchmark estimate** (inferred from scaling patterns):
- **10GbE with PP=2:** ~15-25 tok/s at batch_size=8 — adequate for low-to-moderate QPS
- **2.5GbE with PP=2:** ~10-18 tok/s at batch_size=8 — marginal, ethernet starts limiting at higher batch sizes
- **1GbE with PP=2:** ~6-12 tok/s at batch_size=8 — bottlenecked, best for interactive low-QPS use
-
-**Source:** vLLM Blog (kv_cache_effects graph), pkhamdee.blog, dreaming.press
-
-### 5. Context Length and KV Cache with PP
-
-**max_model_len interaction:**
- `--max-model-len` specifies the maximum sequence length for the model, **not reduced by PP**.
- However, KV cache is split across PP stages. Each stage owns its layer subset's KV cache.
- Total KV cache capacity = sum across all stages. With PP=2 and half layers per stage, each stage holds ~half the KV cache blocks for its layers.
-
-**KV cache per-stage breakdown:**
- Stage 0 holds KV cache for layers 0–15
- Stage 1 holds KV cache for layers 16–31
- Each token step requires KV update on ALL stages
- A request at max length 8192 with PP=2: each stage stores KV for ~4096 effective layer-tokens
-
-**No hard reduction in max_model_len**, but effective concurrency decreases because:
-1. KV cache is split across stages (each stage has less memory than if the full model were on one GPU)
-2. PP bubble reduces throughput at low batch sizes
-
-**Source:** `vllm/v1/core/kv_cache_coordinator.py`, `serverAppConfigs-docs/serving/parallelism_scaling.md`
-
-### 6. Speculative Decoding with PP
-
-**Known incompatibility** (`serverAppConfigs-docs/features/speculative_decoding/README.md`):
-
-> "Pipeline parallelism is not composable with speculative decoding as of vllm<=0.15.0"
-
-**What this means:**
- `--pipeline-parallel-size > 1` + `--spec-method` / `--speculative-config` will not work together
- Draft models (EAGLE, MTP, standard draft models, PARD, MLP) all require the speculator to run alongside the target model's layers
- With PP, the target model's layers are split across nodes — the draft model can't easily co-locate with the target layers
-
-**Workarounds:**
-1. **Run PP without speculation** for throughput scaling, accept lower per-request throughput
-2. **Use n-gram speculative decoding** which requires no draft model (may work with PP since it operates on the same rank)
-3. **Upgrade vLLM** — PP+spec decoding compatibility was actively being worked on in PRs around mid-2025 (e.g., the PP V2 PR #33960 added PP support to Model Runner V2)
-
-**For your Qwen3.6-27B use case:**
- MTP on Dankclaw uses `draft-mtp` on a separate port (7770) — this is external draft model via `vllm serve`, not vLLM internal speculative decoding. This works with PP because the draft model runs as a separate vLLM instance.
- DFlash (retired) was also an external drafter
- n-gram internal speculation may work with PP since it doesn't need a separate model
-
-**Source:** `serverAppConfigs-docs/features/speculative_decoding/README.md`, `vllm/config/speculative.py`
-
-## Verdict: 2-Node vLLM PP over Ethernet for 27B Model
-
-**Is it practical?** Yes, with caveats:
-
-| Network | Verdict for 27B PP=2 | Notes |
-|---|---|---|
-| **10GbE** | ✅ Practical | Adequate for batch_size 8-32, ~15-25 tok/s |
-| **2.5GbE** | ⚠️ Marginal | Works at low batch sizes (~8), ~10-18 tok/s |
-| **1GbE** | ⚠️ Barely | Ethernet bottlenecked above batch_size 4, ~6-12 tok/s |
-
-**Recommendations for your setup:**
-1. Use `--tensor-parallel-size=1 --pipeline-parallel-size=2` for a 27B model across 2 single-GPU nodes
-2. Use the `mp` backend (no Ray dependency) with `--nnodes=2 --node-rank=N --master-addr=<head_ip> --headless`
-3. Set `NCCL_SOCKET_IFNAME=eth0` to pin the correct NIC
-4. Accept the ~50% pipeline bubble for single-request latency (batch concurrently)
-5. PP + internal speculative decoding is NOT supported (v0.15.0), but external draft models (MTP on port 7770) work fine
-6. KV cache max_model_len is NOT reduced by PP, but per-stage KV cache is split
-
-## Sources
-
- **Kept:** Parallelism and Scaling docs — vLLM official guidance on TP/PP strategy, `serverAppConfigs-docs/serving/parallelism_scaling.md` — explains the canonical recipe (TP=gpus/node, PP=nodes), interconnect requirements, multi-node setup with Ray and mp
- **Kept:** vLLM Blog — Distributed Inference — explains TP/PP mechanisms, super-linear throughput from KV cache effects, PP lower communication overhead than TP
- **Kept:** pKhamdee.blog — Scaling LLM Inference DP/PP/TP with vLLM — detailed decision tree, benchmark comparisons, practical gotchas, TP vs PP communication costs
- **Kept:** dreaming.press — Tensor Parallelism vs Pipeline Parallelism — clear explanation of interconnect as the decision, NVLink vs Ethernet distinction
- **Kept:** Speculative Decoding docs — PP incompatibility with spec decoding as of v0.15.0, full spec method catalog, --speculative-config schema
- **Kept:** parallel.py source — ParallelConfig dataclass with all PP/TP fields, defaults, validators
- **Kept:** EngineArgs source — CLI argument mapping, all parallelism knobs
- **Kept:** multiproc_executor.py / ray_executor.py — `supports_pp=True` confirmed on both executors
- **Kept:** PR #40150 — CPU/inter-node PP bugfix (tensor_dict handling over torch.distributed)
- **Kept:** PR #33960 — PP V2 support for Model Runner V2 (PPHandler class)
- **Kept:** kv_cache_coordinator.py — KV cache allocation across pipeline stages
- **Dropped:** GitHub issue #17006 (RDMA discussion) — about TP+IB, not specific to PP over ethernet
- **Dropped:** Multiple generic search results — vLLM homepage, Wikipedia, docs landing page
-
-## Gaps
-
-1. **Exact multi-node PP benchmark numbers over 1/2.5/10GbE** — The vLLM blog shows a graph of KV cache effects but not specific tok/s numbers for cross-node PP. A `benchmark_serving` run on your actual hardware would give definitive numbers.
-2. **PP + speculative decoding support in v0.16.0+** — The docs say "as of v0.15.0" PP is incompatible with spec decode. The PP V2 PR (#33960) may have resolved this. Need to check if the current master branch supports it.
-3. **KV cache split ratio** — Exactly how many KV blocks per stage? Is it strictly by layers or does vLLM do any balancing?
-4. **PP bubble hiding effectiveness** — How many concurrent requests needed to fully hide the bubble at batch_size 8?
-
-## Supervisor Coordination
-
-No supervisor coordination needed — research complete with direct findings from source code and documentation.
-
---
-
-*Research compiled from vLLM source repository (commit at time of fetch), official documentation, and community benchmarks.*
@@ -1,3 +1,12 @@
+---
+type: reference
+description: "This page maps LLM quantization formats to the GPU and CPU architectures that support them."
+tags:
+  - llm
+  - quantization
+timestamp: 2026-08-28
+---
+
 # Quantization by Hardware Architecture

 ## Overview
@@ -1,3 +1,12 @@
+---
+type: reference
+description: "vLLM uses CPU RAM and swap space to extend GPU memory capacity."
+tags:
+  - llm
+  - vllm
+timestamp: 2026-08-28
+---
+
 # RAM CPU Offloading in vLLM

 ## Overview
@@ -1,3 +1,11 @@
+---
+type: reference
+description: "Qwen3.6-27B is a dense vision-language model from the Qwen3.6 series."
+tags:
+  - llm
+timestamp: 2026-09-03
+---
+
 # Running Qwen3.6 27B FP8 on Dankclaw

 ## Model Information
@@ -1,3 +1,11 @@
+---
+type: reference
+description: "Qwen3.6-35B-A3B is a Mixture-of-Experts vision-language model from the Qwen3.6 series."
+tags:
+  - llm
+timestamp: 2026-09-03
+---
+
 # Running Qwen3.6 35B A3B FP8 on Dankserver

 > **RETIRED.** This page documents the FP8 vLLM path for Qwen3.6-35B-A3B on Dankserver. The llama.cpp 35B service is retired (2026-08-16), its unit file was removed (2026-08-17), and the FP8 vLLM unit (`vllm-qwen36`) never shipped and is disabled. The content is kept as research history. The RTX 3090 now runs the PP worker: see [[Qwen3.8 PP Deployment]].
@@ -1,3 +1,11 @@
+---
+type: reference
+description: "Dankclaw uses AMD Strix Halo APU with unified memory architecture."
+tags:
+  - llm
+timestamp: 2026-08-28
+---
+
 # Shared Memory Architecture Dankclaw

 ## Overview
@@ -142,7 +150,7 @@ Unified memory has lower bandwidth than dedicated VRAM:

 ### ROCm Overcommit

-ROCm reports 96 GiB GPU memory on Strix Halo APU. Physical RAM is 64 GiB (61 GiB visible; corrected 2026-08-17, the box was upgraded from 32 GB). The GPU can address more than physical RAM exists. vLLM will see 96 GiB and try to allocate accordingly. Set `--gpu-memory-utilization` to 0.80 or lower to prevent OOM.
+ROCm reports ~114 GiB of GPU memory on Strix Halo APU (the GTT aperture, `gtt_total=122402963456` bytes = 114 GiB; `llama-server --list-devices` shows 116,732 MiB — the 2026-08-17 "96 GiB" figure was stale). Physical RAM is 64 GiB (61 GiB visible; the box was upgraded from 32 GB). The GPU can address more than physical RAM exists because the GTT aperture maps over the shared LPDDR. vLLM will see ~114 GiB and try to allocate accordingly. Set `--gpu-memory-utilization` to 0.80 or lower to prevent OOM.

 Lower bandwidth means slower inference. Plan for lower throughput.

@@ -1,3 +1,12 @@
+---
+type: reference
+description: "This index contains all documentation for vLLM deployment on Dankserver and Dankclaw."
+tags:
+  - llm
+  - vllm
+timestamp: 2026-08-28
+---
+
 # vLLM Index

 > **RETIRED (2026-08-18).** The vLLM service on Dankclaw was replaced by the CIRU ROCmFPX llama.cpp runtime (v3). These pages are historical reference. See [[ciru-activefpx-promptforge]].
@@ -1,3 +1,12 @@
+---
+type: reference
+description: "vLLM is a fast and easy library for large language model inference and serving."
+tags:
+  - llm
+  - vllm
+timestamp: 2026-08-28
+---
+
 # vLLM Overview

 ## What Is vLLM
@@ -14,10 +14,11 @@ timestamp: 2026-08-22

 # CIRU ActiveFPX PromptForge on Dankclaw

-> **Active 2026-08-22.** `frank-large` serves this v3 model again
-> after a same-day Kairic Edge IU4 experiment was reverted (operator
-> decision). Commit `c60156f` restored the v3 script pair. See
-> [[kairic-edge]] for the retired experiment reference.
+> **RETIRED 2026-08-24.** `frank-large` served this v3 model from
+> 2026-08-22 (restored after a same-day Kairic Edge IU4 experiment was
+> reverted — operator decision, commit `c60156f`; see [[kairic-edge]])
+> until [[eemin-rocmi4-mtp]] took over on 2026-08-24. Kept as the
+> historical record of the v3 deployment.

 `frank-large` (port 7770, moved from 52625 on 2026-08-22) serves
 `jcbtc/Qwen3.8-27B-CIRU-ActiveFPX-PromptForge` **v3** through the CIRU
@@ -1,7 +1,3 @@
-> **SUPERSEDED 2026-08-17.** The Frank redo retired this topology.
-> See [frank-redo-2026-08-17.md](frank-redo-2026-08-17.md) and
-> [deployment.md](deployment.md) for the active configuration.
-
 ---
 type: reference
 title: Dankserver VRAM and RAM Growth Under Load
@@ -13,8 +9,12 @@ tags:
  - ram
  - ctx-checkpoints
  - crash
+timestamp: 2026-09-03
 ---

+> **SUPERSEDED 2026-08-17.** The Frank redo retired this topology.
+> See [[frank-redo-2026-08-17]] and [[deployment]] for the active configuration.
+
 # Dankserver VRAM and RAM Growth Under Load

 This page records the investigation of the recurring cuBLAS crash on
@@ -21,7 +21,7 @@ SearXNG, Caddy, or another container service does **not** restart inference.
 |---|---|---|---:|
 | Dankserver | `dank@dankserver` | `frank-medium` | 7770 |
 | Dankclaw | `dank@dankclaw` | `frank-large` | 7770 |
-| Dankclaw | `dank@dankclaw` | `frank-npu` (FLM on the XDNA2 NPU) | 7771 |
+| Dankclaw | `dank@dankclaw` | `frank-small` | 7771 |

 ## Scoped workflow

@@ -47,49 +47,49 @@ Run the unified deployer from the repository root:
 Without `--service`, `scripts/deploy-plan.py` maps changed files to the service
 on the selected host. The following files select the corresponding service:

- `scripts/frank-medium.env`, `frank-medium.service`, and
-  `serve-frank-medium.sh` select `frank-medium` on Dankserver.
+- Any file under `dankserver/frank-medium/` (the Quadlet unit, its
+  entrypoint and its healthcheck) selects `frank-medium` on Dankserver —
+  the container mounts the entrypoint and healthcheck straight out of the
+  checkout, so any file there changes deployed state.
 - `scripts/frank-large.env`, `frank-large.service`, and
  `serve-frank-large.sh` select `frank-large` on Dankclaw.
- `scripts/frank-npu.env`, `frank-npu.service`,
-  and `serve-frank-npu.sh` select `frank-npu`
-  on Dankclaw.
- `scripts/chat-template.jinja` selects the LLM on the selected host only.
+- `scripts/frank-small.env`, `frank-small.service`, and
+  `serve-frank-small.sh` select `frank-small` on Dankclaw.
 - Documentation, tests, deployment tooling, and unrelated container changes
-  do not select either inference service.
+  do not select any inference service.

-The target-side `scripts/deploy-remote.sh` installs the selected service: for a
-**root unit** it relinks the repo file into `/etc/systemd/system/`, reloads
-systemd, enables it and restarts it (frank-small); for a **Quadlet** it retires
-the predecessor, stops the service's own unit, waits for the published port to
-clear, installs the `.container` and restarts the generated unit (frank-large).
-Each handler then waits on that service's health endpoint —
-`http://127.0.0.1:7770/health` for frank-large, `http://127.0.0.1:7771/health`
-for frank-small. The wait budget is 20 minutes because cold model loads are
-slow: halogen pins ~68 GiB of weights before it serves anything. A failed
-handler does not advance its per-service state.
+The target-side `scripts/deploy-remote.sh` installs the selected service.
+`frank-large` and `frank-small` are **root units**: the handler relinks the
+repo file into `/etc/systemd/system/` (`relink_unit`), reloads systemd,
+enables it and restarts it. `frank-medium` is a **rootless Quadlet**: the
+handler retires its predecessors (the replica Quadlets and the retired llama
+root unit), stops the service's own unit, waits for the published port to
+clear, installs the `.container` and restarts the generated unit. Each
+handler then waits on that service's health endpoint —
+`http://127.0.0.1:7770/health` for frank-large and frank-medium,
+`http://127.0.0.1:7771/health` for frank-small. The wait budget is 20 minutes
+because cold model loads are slow. A failed handler does not advance its
+per-service state.

 ## Unit and artifact paths

-The units point into the stable target checkout:
+The root units point into the stable target checkout:
 ```text
-/home/dank/serverAppConfigs/scripts/frank-medium.service
+/home/dank/serverAppConfigs/scripts/frank-large.service
 /home/dank/serverAppConfigs/scripts/frank-small.service
 ```
+`frank-medium` is a Quadlet under `dankserver/frank-medium/quadlet/`
+(`frank-medium.container`) rather than a unit in `scripts/`.

 The unit files and serve scripts validate their model/runtime artifacts before
-starting. `frank-large` is a Quadlet under `dankclaw/frank-large/quadlet/`
-rather than a unit in `scripts/`; `frank-small` keeps the unit/env/serve trio
-above. Keep `REMOTE_DIR` stable; moving the checkout requires updating both the
-unit absolute paths and the deployment tests in the same change.
+starting. Keep `REMOTE_DIR` stable; moving the checkout requires updating both
+the unit absolute paths and the deployment tests in the same change.

-## NPU unit
+## Retired endpoints

-`frank-npu` is an inactive, historical Dankclaw endpoint. It is
-relinked only when explicitly selected with `--service frank-npu`; the
-large-model handler never restarts it. This is intentional: the NPU is not part
-of the active inference path and should not be disturbed by a large-model
-change.
+`frank-npu` (the FastFlowLM XDNA2 NPU endpoint) was retired on 2026-09-11
+with the `amd_iommu=off` decision and is no longer a deployable service — the
+planner has no `frank-npu` entry. Its runbook is [[frank-npu]].

 ## Isolation guarantee

@@ -110,9 +110,10 @@ change in that service's inputs.

 If the LLM handler fails after the target pull:

-1. Inspect `journalctl -u frank-small -n 200` (root unit) or
-   `journalctl --user -u frank-large -n 200` (rootless Quadlet).
-2. Confirm the model files, the runtime binary and (for frank-large) the
+1. Inspect `journalctl -u frank-large -n 200` or `journalctl -u frank-small
+   -n 200` (root units), or `journalctl --user -u frank-medium -n 200`
+   (rootless Quadlet).
+2. Confirm the model files, the runtime binary and (for frank-medium) the
   container image are present and readable by `dank`.
 3. Retry only the affected service:

@@ -15,10 +15,12 @@ timestamp: 2026-08-25

 # caf-eemin Qwen3.8-27B ROCmI4 embedded MTP on Dankclaw

-> **Active 2026-08-25.** `frank-large` serves this model. It replaced
-> the kingjones777 ROCmFP4 STRIX MTP payload on the same port and the same
-> Strix Halo iGPU, running the new charlie12345/ROCmFPX ROCmI4 W4A4 build. See
-> [[kingjones-strix-mtp]] for the retired page.
+> **RETIRED 2026-08-25.** `frank-large` served this model from 2026-08-18
+> until 2026-08-25. It replaced the kingjones777 ROCmFP4 STRIX MTP payload
+> on the same port and the same Strix Halo iGPU, running the new
+> charlie12345/ROCmFPX ROCmI4 W4A4 build (see [[kingjones-strix-mtp]] for
+> the page it replaced), and was itself replaced by
+> [[kingjones-strix-rocmfp4]].

 `frank-large` (port 7770) serves
 `cafonz/Qwen3.8-27B-ROCmI4-MTP-GGUF` (published under the public
@@ -1,3 +1,12 @@
+---
+type: reference
+description: "This page records the benchmarks that chose the runtime for `frank-large` on Dankclaw."
+tags:
+  - llm
+  - frank
+timestamp: 2026-09-03
+---
+
 > **SUPERSEDED 2026-08-17.** The Frank redo retired this topology.
 > See [frank-redo-2026-08-17.md](frank-redo-2026-08-17.md) and
 > [deployment.md](deployment.md) for the active configuration.
@@ -1,7 +1,7 @@
 ---
 type: reference
-title: CIRU v4.4.1 Orca on Dankclaw (frank-large)
-description: The retired frank-large runbook (2026-09-19 to 2026-09-21, rollback target) — CIRU v4.4.1 (QSA window hotfix) serving jcbtc/Qwen3.8-Flash-CIRU-STRIX-Orca on the Dankclaw Strix Halo iGPU: the private host build, the runtime gate, the payload, the deployed config, MTP and PLE, the client contract, and the failure modes worth knowing.
+title: "CIRU v4.4.1 Orca on Dankclaw (frank-large)"
+description: "The retired frank-large runbook (2026-09-19 to 2026-09-21, rollback target) — CIRU v4.4.1 (QSA window hotfix) serving jcbtc/Qwen3.8-Flash-CIRU-STRIX-Orca on the Dankclaw Strix Halo iGPU: the private host build, the runtime gate, the payload, the deployed config, MTP and PLE, the client contract, and the failure modes worth knowing."
 tags:
  - llm
  - ciru
@@ -14,208 +14,3 @@ tags:
  - mtp
 timestamp: 2026-09-19
 ---
-
-# CIRU v4.4.1 Orca on Dankclaw (frank-large)
-
-> **RETIRED 2026-09-21** (replaced by
-> [[frank-large-unsloth-qwen38-flash-next]]; **rollback target**). It was
-> ACTIVE since 2026-09-19 and replaced halogen-flash-server (see
-> [[halogen-flash-server]]), which had replaced ik_llama.cpp (see
-> [[frank-large-ik-uncensored]]). It was deployed as a **root systemd
-> unit** (`scripts/frank-large.service`, `User=dank`), port 7770, on the
-> same gfx1151 iGPU the halogen quadlet used. The RTX 3090 it did not touch
-> belongs to [[frank-small-gemma4-12b]].
-
-CIRU is a llama.cpp fork tuned for the Qwen3.8-Flash-Next ("qwen4exp")
-architecture on AMD gfx1151 (Strix Halo). v4.4.1 carries a **QSA window
-hotfix** that is what makes vision and the MTP drafter work on this build; the
-pre-applied hotfix source ships in the model repo under
-`runtime/v4.4.1/qsa-window-hotfix/`. Like every other fleet endpoint it is a
-llama.cpp runtime — `llama-bench` is its own harness.
-
-## Shape
-
-One root unit, one process, one published port:
-
-| half | port | reachable |
-|---|---|---|
-| llama-server (OpenAI front-end + engine, single process) | 7770 | published as host **7770** |
-
-Unlike the halogen predecessor (two ports: an unauthenticated engine on 8730
-and an OpenAI front-end on 8731), CIRU is a single `llama-server` process —
-there is no separate engine port to keep private.
-
-`/v1/models` reports the loaded model name, **`Qwen3.8-Flash-CIRU-STRIX-Orca`**
-(the launcher sets no `--alias`). The endpoint serves **whatever model name a
-request names** — llama.cpp does not reject an unknown id — so the harness's
-`frank-large` label in `models.yml` works without a rename.
-
-## Runtime
-
-A **private host build**, not a container image. The v4.4.1 source with the QSA
-window hotfix, built on the host against the private ROCm SDK that
-`scripts/ciru/setup-linux-amd.sh` installs into the checkout's own venv:
-
-| path | what |
-|---|---|
-| `/home/dank/GitPackages/ciru-runtime-v4.4.1-qsa-hotfix/` | the checkout (source + build + venv) |
-| `build-gfx1151-sdk/bin/llama-server` | the built binary |
-| `.venv-rocm/` | the private ROCm SDK venv (~5.7 GB); the binary resolves its libs through RUNPATH |
-| `scripts/ciru/run-orca-server.sh` | the launcher (`exec`'d by the serve script) |
-| `profiles/strix-halo-production.env` | the 36-variable kernel bundle + PLE pager defaults, sourced by the launcher AFTER `frank-large.env` (every line is `${VAR:-default}`, so a value set in `frank-large.env` wins) |
-
-### The runtime gate
-
-`run-orca-server.sh` sources `run-server.sh`, which **exits 2** unless
-`${repo_root}/runtime/hip/lib/libamdhip64.so` and `runtime/rocr/lib/
-libhsa-runtime64.so` exist (unversioned, no suffix). It prepends both to
-`LD_LIBRARY_PATH` and `ldd`-checks that the built `libllama-common.so` carries
-the "Qwen4Exp MTP image position" marker and the HIP lib carries the
-`flash_attn_index_mask_clear/set/empty` symbols before exec.
-
-The build links the venv through RUNPATH, so those two files are an existence
-gate, not a real dependency. `serve-frank-large.sh` satisfies the gate
-idempotently: it finds the venv's unversioned devel symlinks
-(`.venv-rocm/lib/python*/site-packages/_rocm_sdk_devel/lib/`) and `ln -sfn`-s
-both into `runtime/{hip,rocr}/lib/`. **The first deploy of this endpoint failed
-exactly here** — the build linked the venv but `runtime/` did not exist, so the
-launcher exited 2 and the unit crash-looped until the serve script created the
-symlinks.
-
-## Payload
-
-`scripts/download-frank-large-orca.sh` assembles `jcbtc/Qwen3.8-Flash-CIRU-
-STRIX-Orca` (HuggingFace, public and ungated) into
-`/home/dank/models/Qwen3.8-Flash-CIRU-STRIX-Orca/`, sha256-verified. The
-launcher refuses to start without the main GGUF, the MTP head and the PLE
-sidecar:
-
-| file | bytes | role |
-|---|---|---|
-| `Qwen3.8-Flash-CIRU-STRIX-Orca.gguf` | 79,397,818,912 | the main model (Orca variant) |
-| `mtp/Qwen3.8-Flash-CIRU-STRIX-Orca-MTP-Q8_0.gguf` | 4,135,893,440 | the MTP drafter head |
-| `ple/ple.payload.bin` | 52,429,053,952 | the per-layer-embedding (PLE) payload — mmap'd and paged, not resident |
-| `ple/ple.manifest.json` | — | PLE manifest |
-| `ple/ple.scale.bf16` | — | PLE scale |
-| `vision/mmproj-Qwen3.8-Flash-Orca-F16.mmproj` | 904,004,320 | the F16 vision projector (on disk, `ENABLE_VISION=0`) |
-
-The main GGUF ships as **transport parts** that the download script assembles
-and verifies; the PLE is the large one (52.4 GB) and is paged by the engine's
-own workers, not held in RAM.
-
-## Deployed config
-
-Tunables live in `scripts/frank-large.env`; the launcher owns the flag list
-(`-sm none` one iGPU no split, `-ngl all`, f16 KV, flash attention,
-cont-batching, the PLE sidecar, the MTP drafter, the production kernel-bundle
-profile). The departures worth naming:
-
-| flag | value | why |
-|---|---|---|
-| `CONTEXT_SIZE` | `262144` | the model's FULL trained window (the operator's "full context" call); the GGUF carries no rope scaling |
-| `PARALLEL_SLOTS` | `1` | one stream — and the MTP drafter **requires exactly one slot** in this runtime, so the operator's ask and the drafter's constraint are the same number |
-| `ENABLE_MTP` | `1` | the MTP drafter, depth 4 (the Orca variant's own `mtp_depth_default`) |
-| `ENABLE_VISION` | `0` | text-only for now; the F16 projector is on disk, one flip to enable. The fleet's image path is [[frank-small-gemma4-12b]] either way |
-| `PLE_CACHE_MIB` | `4096` | the PLE pager's hot window kept resident; the vendor production-profile default. The 52.4 GB payload is mmap'd and paged around it |
-| `HOST` | `0.0.0.0` | the launcher consumes it for `--host`; the value doubles as a JIT-cache directory name, which is why `0.0.0.0` is safe (HIP reads `HOST` bare — see the AGENTS.md note) |
-
-`TEMP`, `HOST` and `GPU_MAX_HEAP_SIZE` are the three names to avoid for any
-new tunable: HIP reads them bare.
-
-## MTP
-
-The Multi-Token Prediction drafter is a **separate Q8_0 head** (not inside the
-main GGUF), loaded by the launcher. It requires exactly one slot, which is why
-`PARALLEL_SLOTS=1`. Depth 4 is the Orca variant's default.
-
-Verified at deploy (2026-09-19): the per-task `draft acceptance = 0.65625
-(21/32), mean len = 3.62` — the drafter is engaged and accepting, not silently
-standing down.
-
-## PLE
-
-The per-layer-embedding payload (52.4 GB) is the Qwen3.8-Flash-Next
-architecture's signature: token embeddings live outside the main weight set and
-are paged on demand by the engine's own workers (`GGML_QWEN4EXP_PLE_WORKERS=16`,
-`GGML_QWEN4EXP_PLE_IO_WORKERS=32` in the production profile). `PLE_CACHE_MIB=4096`
-is the hot window kept resident; the rest is mmap'd file cache. **Do not treat
-the 52.4 GB as resident memory** — `free` and the dashboards will not show it
-pinned, and that is correct.
-
-## Client contract
-
-The differences from a stock llama.cpp endpoint that bite:
-
- **`reasoning_content`** carries the reasoning trace (the standard OpenAI
-  field). The serve script passes `--chat-template-kwargs
-  '{"preserve_thinking":true}'` so prior thinking blocks are kept across turns;
-  a request-level `chat_template_kwargs` still wins.
- **`reasoning_effort` is gated.** Qwen3.8's template accepts only
-  `xhigh`, `medium`, `low` and **500s on anything else** (not a clamp). The
-  harness folds every level onto that set in `models.yml`
-  (`reasoningEffortMap`: minimal/low→low, medium→medium, high/max→xhigh) —
-  subagents run at `high`, which is why the map exists.
- **Text only.** `ENABLE_VISION=0`, so image content parts are refused. The
-  fleet's image path is [[frank-small-gemma4-12b]].
- **Sampling.** The server has no sampler defaults of its own; the harness
-  applies the per-model recipe from `models.yml` (temp 1.0 / top_p 0.95 / top_k
-  20 / min_p 0.0). A client that stays silent gets the launcher's default.
- **`tools` works** — the launcher passes `--jinja`, and the Qwen3.8 template
-  supplies the tool-call grammar. Verified at deploy: a tool call returns
-  `tool_calls`.
-
-## Deploy and operate
-
-```bash
-./scripts/deploy.sh dankclaw --service frank-large
-```
-
-The handler pulls the checkout (ff-only), (re)links the unit, and gates on
-`/health`. First boot is the slow one (model load ~46 s + graph capture); a
-warm restart is fast.
-
-```bash
-sudo systemctl status frank-large            # the root unit
-sudo journalctl -u frank-large -n 60         # startup account: runtime check, model load, "listening on http://0.0.0.0:7770", MTP draft loaded
-curl -s http://127.0.0.1:7770/health         # ok
-curl -s http://127.0.0.1:7770/v1/models      # Qwen3.8-Flash-CIRU-STRIX-Orca
-```
-
-**Stop and mask before any GPU experiment** (the AGENTS.md bench protocol):
-`sudo systemctl stop frank-large && sudo systemctl mask frank-large`. The
-frank-large serve process is a `llama-server` **and so is frank-small on the
-same host** — never `pkill -x llama-server`. If you must kill by pattern, use
-`pkill -f 'Qwen3.8-Flash-CIRU-STRIX-Orca'` (the model path is unique to this
-endpoint and does not appear in the deploying shell's own command line).
-
-## Troubleshooting
-
-| symptom | cause |
-|---|---|
-| launcher exits 2: "The v4.4 HIP/ROCr runtime is missing" | `runtime/{hip,rocr}/lib/*.so` absent — the serve script's `ln -sfn` did not run, or the venv was moved. Re-run the serve script or rebuild: `bash $RUNTIME_DIR/scripts/ciru/setup-linux-amd.sh --install-host-deps` |
-| "missing SDK venv runtime under .../.venv-rocm" | the venv's unversioned devel symlinks are gone — rebuild the runtime |
-| "missing target model / missing payload" | a download did not finish — re-run `scripts/download-frank-large-orca.sh` |
-| loads, `/health` ok, first completion slow (~45-60 s) | the PLE pager pulling the hot window in from file cache — not a crash, do not restart over it |
-| a subagent 500s naming `reasoning_effort` | the Qwen3.8 template gate — the harness `reasoningEffortMap` must fold the level onto {xhigh, medium, low} |
-| `free` / dashboards do not show 52 GB pinned | correct — the PLE is mmap'd file cache, not resident; only `PLE_CACHE_MIB` is hot |
-
-Retrieval: this is a **root** unit, so `journalctl -u frank-large` is the
-durable record (no `--user`, no `sudo` needed for the journal once the unit is
-readable). There is no container log — there is no container.
-
-## Rollback
-
-The predecessor is intact: the halogen payload stays at
-`/home/dank/models/halogen` (retired, not deleted), and the halogen quadlet is
-archived under `.AGENTTRASH/`. Rolling back is a `git revert` of this change set
-plus a deploy — **but the root unit and the halogen quadlet both claim port
-7770**, so the handler's port-free gate covers that race. See
-[[halogen-flash-server]].
-
-The `amd_iommu=off` kernel argument is separate and reverts on its own — see
-[[dankclaw-iommu-translation-off]].
-
-## Citations
-
- [jcbtc/Qwen3.8-Flash-CIRU-STRIX-Orca](https://huggingface.co/jcbtc/Qwen3.8-Flash-CIRU-STRIX-Orca) — weights, MTP head, PLE payload, and the v4.4.1 QSA window-hotfix source tarball
- [CIRU runtime v4.4.1](https://github.com/ciru) — `scripts/ciru/run-server.sh` (runtime gate), `profiles/strix-halo-production.env` (kernel bundle), `docs/BUILD_LINUX.md`
@@ -1,7 +1,7 @@
 ---
 type: reference
-title: Kairic Edge Qwen3.8-27B IU4 (W4A4) on Dankclaw (frank-large)
-description: The active frank-large runbook — the Kairic Edge runtime (ciru-ai/ROCmFPX, branch kairic-edge-qwen38-27b-v1.2, commit 205a3e5) serving jcbtc/Qwen3.8-27B-IU4-Kairic-Edge (27B dense, IU4 W4A4) on the Dankclaw Strix Halo iGPU: the source-built runtime, the full-window/one-stream shape, the MTP depth 4 + n-gram spec method, the three PromptForge sidecars, the compatibility mode, the payload, the client contract, and the failure modes worth knowing.
+title: "Kairic Edge Qwen3.8-27B IU4 (W4A4) on Dankclaw (frank-large)"
+description: "The RETIRED frank-large runbook (2026-09-21, same-day swap; replaced by [[frank-large-unsloth-qwen38-27b-q8xl]]) — the Kairic Edge runtime (ciru-ai/ROCmFPX, branch kairic-edge-qwen38-27b-v1.2, commit 205a3e5, TheRock 7.15.0, Composable Kernel) serving jcbtc/Qwen3.8-27B-IU4-Kairic-Edge (27B dense, IU4 W4A4) on the Dankclaw Strix Halo iGPU: the source-built runtime, the full-window/one-stream shape, the MTP depth 4 + n-gram spec method, the three PromptForge sidecars, the compatibility mode, the payload, the client contract, and the failure modes worth knowing."
 tags:
  - llm
  - kairic-edge
@@ -18,195 +18,31 @@ tags:
 timestamp: 2026-09-21
 ---

-# Kairic Edge Qwen3.8-27B IU4 (W4A4) on Dankclaw (frank-large)
+# frank-large — Kairic Edge Qwen3.8-27B IU4 (W4A4) (RETIRED)

-> **ACTIVE since 2026-09-21.** It replaced the unsloth llama.cpp prebuilt
-> (b11030, Qwen3.8-Flash-Next UD-IQ4_XS, 125B MoE — see
-> [[frank-large-unsloth-qwen38-flash-next]], the current rollback target),
-> which had replaced the CIRU v4.4.1 Orca runtime (see
-> [[frank-large-ciru-orca]]). Deployed as a **root systemd unit**
-> (`scripts/frank-large.service`, `User=dank`), port 7770, on the same
-> gfx1151 iGPU. The RTX 3090 it does not touch belongs to
-> [[frank-small-gemma4-12b]].
+> **RETIRED 2026-09-21 (same-day swap).** This was the ACTIVE frank-large
+> runbook on 2026-09-21 before it was replaced by
+> [[frank-large-unsloth-qwen38-27b-q8xl]] later the same day. The payload
+> and runtime remain on disk as the rollback target.

-The model is `jcbtc/Qwen3.8-27B-IU4-Kairic-Edge` — a **27B dense** Qwen3.8
-at **IU4** (W4A4, integer-weights / integer-activations). It is a
-fundamentally different architecture from the 125B MoE it replaces: no
-PLE, no MoE experts, no n-gram embedding. The payload is four files: the
-15.5 GiB main GGUF plus three `.pfs` sidecars (FFN 7.99 GiB, GDN 1.88
-GiB, GDN-Output 0.70 GiB) — the PromptForge IU4 weight shards the Kairic
-Edge runtime loads alongside the main GGUF.
+## What it served

-The runtime is the **Kairic Edge** build of `ciru-ai/ROCmFPX` (branch
-`kairic-edge-qwen38-27b-v1.2`, commit `205a3e5`), built from source
-(`scripts/build-kairic-edge-gfx1151.sh`). It is a llama.cpp build with the
-PromptForge sidecar env block and the `--kairic-edge` qualified serving
-path.
-
-## Why this swap
-
- The operator asked for the Kairic Edge 27B model at **one conversation,
-  full context (262,144)** (2026-09-21).
- The Kairic Edge runtime is the qualified IU4 (W4A4) serving path — no
-  standard llama.cpp quant, no PLE sidecar payload, no QSA hotfix: one
-  GGUF, three sidecars, done.
- The spec method is native MTP depth 4 + n-gram drafting (24/64/64).
-  DFlash is NOT available for this model (no `DRAFT_DFLASH` spec type in
-  the Kairic Edge runtime).
-
-## Shape
-
-One root unit, one process, one published port:
-
-| half | port | reachable |
-|---|---|---|
-| llama-server (OpenAI front-end + engine, single process) | 7770 | published as host **7770** |
-
-## Runtime
-
- **Source build**: `scripts/build-kairic-edge-gfx1151.sh` (TheRock
-  7.15.0a20260718 toolchain, AMD clang 23.0.0, HIP 7.15.0, GCC 13.3.0,
-  CMake 4.4.2, Ninja 1.13.0). Output at
-  `/home/dank/GitPackages/ROCmFPX-kairic/build-kairic/bin/llama-server`.
- **Composable Kernel**: `fdf4bb7fcc984811cef48ce817d89aac064b984a` +
-  `patches/composable-kernel-gfx1151-iu4.patch` (SHA-256
-  `ea57bca3793d35d884fc156b6c0bc33d8e672e3bb49f66ff67338503029d6af5`).
-  The CK `config.h` is generated by CMake's `configure_file` (line 334 of
-  CK's CMakeLists.txt) — it is untracked, so a `git clean` removes it.
- **GPU target**: `gfx1151` (Strix Halo). The build takes ~4 minutes
-  (`JOBS=$(nproc)`).
- **v1.2 changes**: Exact 65-row n-gram verification switched to the compact
-  authoritative path by default; the native IU4 M65 verifier is retained
-  only for diagnostics.
-
-## Spec method
-
- **Native MTP depth 4** (`--spec-type draft-mtp --spec-draft-n-max 4`):
-  the model's MTP layers draft 4 tokens per step.
- **N-gram drafting** (24/64/64): the `ngram-mod` matcher chains behind
-  the MTP drafter (the `--spec-type draft-mtp` value builds both).
- **DFlash is NOT available**: the Kairic Edge runtime's spec-type
-  constants are `DRAFT_EAGLE`, `DRAFT_MTP`, `DRAFT_SIMPLE`,
-  `NGRAM_CACHE`, `NGRAM_MAP_K`, `NGRAM_MOD`, `NGRAM_SIMPLE`, `NONE` — no
-  `DRAFT_DFLASH`. The z-lab DFlash2-FP8 drafter (frank-medium's drafter)
-  cannot be used.
- **Draft device**: `--spec-draft-device ROCm0 --spec-draft-ngl 999`
-  (the drafter runs on the iGPU, all layers).
-
-## Compatibility mode
-
-The Kairic Edge runtime has two modes:
-
- **Fast-greedy** (`KAIRIC_EDGE_COMPATIBILITY_MODE=0`, the runner's
-  default): rejects request features that require full target logits,
-  including sampling, penalties, probabilities, grammar-constrained tool
-  calls, logit bias, LoRA, and a reasoning budget.
- **Compatibility** (`KAIRIC_EDGE_COMPATIBILITY_MODE=1`): enables all of
-  the above (9.7% slower per the build guide).
-
-The harness (OMP) needs sampling, tools and a reasoning budget, so
-`COMPATIBILITY_MODE=1` is the deployed shape. In compatibility mode:
-`LLAMA_TARGET_GREEDY_ARGMAX_FASTPATH=0` and
-`PROMPTFORGE_OUTPUT_K8_STRICT_GREEDY=0`.
-
-**Known gotcha**: the greedy-argmax fastpath (fast mode) rejects
-non-greedy requests and reasoning budgets. Re-test chat before production
-if you flip to fast mode.
-
-## PromptForge sidecars
-
-The three `.pfs` sidecars are wired via the `PROMPTFORGE_*` env block
-(the certified runner's exact env):
-
-| sidecar | env var | size |
-|---|---|---|
-| FFN | `PROMPTFORGE_IU4_SIDECAR` / `PROMPTFORGE_SIDECAR` | 7.99 GiB |
-| GDN | `PROMPTFORGE_GDN_SIDECAR` | 1.88 GiB |
-| GDN-Output | `PROMPTFORGE_GDN_OUTPUT_SIDECAR` | 0.70 GiB |
-
-The serve script (`scripts/serve-frank-large.sh`) validates binary → main
-GGUF → 3 sidecars and exits with a pointer message if any is missing, then
-`exec`s `llama-server` with `--jinja --alias frank-large
--chat-template-kwargs '{"preserve_thinking":true}' --chat-template-file
-$TEMPLATE_FILE` (Qwen3.8's template needs `preserve_thinking` to carry
-thinking blocks across turns; a request-level value still wins).
-
-## Chat template
-
-The serve script passes `--chat-template-file` pointing at
-`dankserver/frank-medium/qwen-sharp-v22.4.0.jinja` (the Qwen-Sharp
-v22.4.0 template, the same one frank-medium uses). Without this flag the
-binary falls back to the built-in `peg-native` template, which produces a
-~73,600-token prompt for a ~250-token request (280x the expected size).
-With the sharp template wired in, the same request produces a 62–175-token
-prompt. The `Chat format: peg-native` log line is the format-type label,
-not the template indicator — the sharp template is active even when that
-line appears.
-
-## Memory budget
-
-The iGPU addresses **114 GiB** (the GTT aperture over the shared LPDDR,
-`gtt_total` in `/sys/class/drm/card1/device/`; `rocm-smi`'s 2 GiB
-"dedicated" is not the budget). The 27B dense model (26 GiB total) is
-mmap'd in with `-ngl 999`; budget at 262k ≈ 30 GiB (weights + KV +
-buffers), leaving ~84 GiB of aperture headroom. The 8 GiB prompt-cache
-budget (`CACHE_RAM=8192`) is the Kairic Edge qualified config. The model
-is 27B dense (not DeltaNet), so the fleet's 12.8 GiB DeltaNet `CACHE_RAM`
-floor does not apply.
-
-## Payload
-
-`scripts/download-frank-large-kairic.sh` (resumable, each file verified
-against its declared byte count + sha256; `--verify-only` re-checks):
-
-| file | size | sha256 |
-|---|---|---|
-| `Qwen3.8-27B-IU4-Kairic-Edge.gguf` | 16,617,792,672 | `360caf73…` |
-| `Qwen3.8-27B-Kairic-IU4-FFN.pfs` | 8,576,856,064 | `adcbb90a…` |
-| `Qwen3.8-27B-Kairic-IU4-GDN.pfs` | 2,019,569,664 | `82f93131…` |
-| `Qwen3.8-27B-Kairic-IU4-GDN-Output.pfs` | 756,953,088 | `3b07e7b1…` |
-
-## Client contract
-
- **Port**: 7770 (the same port and alias the unsloth predecessor served).
- **Alias**: `frank-large`.
- **Context**: 262,144 (the model's native window), 1 slot.
- **Reasoning**: enabled (`--reasoning auto --reasoning-format deepseek
-  --reasoning-budget -1`), extracted to `reasoning_content`.
- **Vision**: text-only (the model has no vision projector). The fleet's
-  image path is [[frank-small-gemma4-12b]] either way.
- **Reasoning effort**: the Qwen3.8 template gates `reasoning_effort` to
-  {xhigh, medium, low} — the harness folds its levels onto that set in
-  `models.yml` (`reasoningEffortMap`).
-
-## Measured performance
-
-10-question conversation (2026-09-21, post-deploy): **23.99s total, 530
-tokens, 22.1 t/s average**. Per-question: 1.05–3.57s, 16.9–29.5 t/s. The
-MTP depth 4 + n-gram spec method is working (draft acceptance logged, both
-`ngram-mod` and `draft-mtp` active in the journal).
-
-**Template verification** (2026-09-21): a 250-token request produces a
-62–175-token prompt (not the 73,600-token prompt the built-in
-`peg-native` template produces). Draft acceptance 0.79 (mean 4.14), the
-model answers correctly, and reasoning lands in `reasoning_content`.
-
-## Failure modes worth knowing
-
- **CK `config.h` is untracked**: a `git clean` in the CK checkout removes
-  it, and the next build fails. Re-run the build script (it re-checks-out
-  the CK commit and re-applies the patch).
- **Fast-greedy mode rejects the harness**: if you flip to
-  `COMPATIBILITY_MODE=0`, the harness's sampling/tools/reasoning-budget
-  requests are rejected. Re-test chat before production.
- **The `charlie12345` fork is NOT the Kairic Edge repo**: the local
-  clone at `/home/dank/GitPackages/ROCmFPX` is the `charlie12345/ROCmFPX`
-  fork (no kairic branch). The Kairic Edge build uses the `ciru` remote
-  (`https://github.com/ciru-ai/ROCmFPX.git`), branch
-  `kairic-edge-qwen38-27b-v1.2`, commit `205a3e5`, in the worktree at
-  `/home/dank/GitPackages/ROCmFPX-kairic/`.
+- **Runtime:** Kairic Edge (ciru-ai/ROCmFPX, branch
+  `kairic-edge-qwen38-27b-v1.2`, commit 205a3e5, TheRock 7.15.0,
+  Composable Kernel), root systemd unit, port 7770.
+- **Model:** `jcbtc/Qwen3.8-27B-IU4-Kairic-Edge` (a 27B dense model, IU4
+  W4A4) — main GGUF + FFN/GDN/GDN-Output `.pfs` sidecars.
+- **Shape:** one 262,144-token conversation (single stream), native MTP
+  depth 4 + n-gram drafting, compatibility mode.
+- **Memory:** the 114 GiB GTT aperture on the Strix Halo iGPU.

 ## Rollback

-[[frank-large-unsloth-qwen38-flash-next]] — the unsloth prebuilt payload
-and runtime are intact on disk.
+Payload and runtime intact on disk. See [[frank-large-unsloth-qwen38-27b-q8xl]]
+for the current ACTIVE runbook and [[frank-large-unsloth-qwen38-flash-next]]
+for the predecessor.
+
+# Citations
+
+- Kairic Edge runtime: https://github.com/ciru-ai/ROCmFPX
+- Model: https://huggingface.co/jcbtc/Qwen3.8-27B-IU4-Kairic-Edge
@@ -1,7 +1,7 @@
 ---
 type: reference
-title: stock llama.cpp Qwen3.8-27B UD-Q8_K_XL (DFlash2) on Dankclaw (frank-large)
-description: The active frank-large runbook — the stock llama.cpp prebuilt (unsloth b11030, ROCm gfx1151) serving unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL (27B dense, UD-Q8_K_XL) with the DFlash2 drafter (z-lab/Qwen3.8-27B-DFlash2-GGUF:Q4_K_M) on the Dankclaw Strix Halo iGPU: the prebuilt runtime, the full-window/one-stream shape, the DFlash2 spec method, fp8 KV, the Qwen-Sharp template, the payload, the client contract, and the failure modes worth knowing.
+title: "stock llama.cpp Qwen3.8-27B UD-Q8_K_XL (DFlash2) on Dankclaw (frank-large)"
+description: "The active frank-large runbook — the stock llama.cpp prebuilt (unsloth b11030, ROCm gfx1151) serving unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL (27B dense, 31.46 GiB) with the z-lab DFlash2 Q4_K_M drafter (depth 4) and q8_0 KV on the Dankclaw Strix Halo iGPU: the prebuilt runtime, the full-window/one-stream shape, the DFlash2 spec method, the Qwen-Sharp template, the payload, the client contract, and the failure modes worth knowing."
 tags:
  - llm
  - llama.cpp
@@ -13,145 +13,103 @@ tags:
  - q8
  - dflash2
  - rocm
-  - fp8-kv
+  - q8-kv
 timestamp: 2026-09-21
 ---

-# stock llama.cpp Qwen3.8-27B UD-Q8_K_XL (DFlash2) on Dankclaw (frank-large)
+# frank-large — stock llama.cpp Qwen3.8-27B UD-Q8_K_XL (DFlash2)

-> **ACTIVE since 2026-09-21.** It replaced the Kairic Edge runtime
-> (ciru-ai/ROCmFPX, jcbtc/Qwen3.8-27B-IU4-Kairic-Edge — see
-> [[frank-large-kairic-edge-qwen38-27b]], the current rollback target),
-> which had replaced the unsloth llama.cpp prebuilt serving
-> unsloth/Qwen3.8-Flash-Next-GGUF:UD-IQ4_XS (125B MoE — see
-> [[frank-large-unsloth-qwen38-flash-next]]). Deployed as a **root
-> systemd unit** (`scripts/frank-large.service`, `User=dank`), port 7770,
-> on the same gfx1151 iGPU. The RTX 3090 it does not touch belongs to
-> [[frank-small-gemma4-12b]].
+> **ACTIVE since 2026-09-21.** Replaces [[frank-large-kairic-edge-qwen38-27b]]
+> (Kairic Edge IU4 W4A4, now the rollback target) and
+> [[frank-large-unsloth-qwen38-flash-next]] (the Flash-Next MoE it preceded).
+> Same port (7770), same alias (`frank-large`), same client contract surface.

-The model is `unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL` — a **27B dense**
-Qwen3.8 at **UD-Q8_K_XL** (a high-precision 8-bit quant). It is a qwen3_5
-hybrid: 64 layers total, 16 full attention + 48 Gated DeltaNet linear
-attention. The payload is two files: the 31.46 GiB main GGUF and the 1.14
-GiB DFlash2 Q4_K_M drafter.
+## Identity

-The runtime is the **stock llama.cpp** prebuilt (unsloth b11030, ROCm
-gfx1151) at `/home/dank/GitPackages/unsloth-llama-b11030/llama-server`.
-It is a mainline llama.cpp build that supports the qwen3_5 architecture
-and the DFlash2 drafter (llama_model_qwen35, llama_model_dflash,
-common_speculative_impl_draft_dflash).
-
-## Why this swap
-
- The operator asked for the 27B model at **one conversation, full
-  context (262,144)** with DFlash2 spec decode (2026-09-21).
- The stock llama.cpp prebuilt is the only stock-compatible runtime with
-  qwen3_5 + DFlash2 support — no source build, no CK `config.h`, no
-  PromptForge sidecars, no compatibility mode.
- The spec method is DFlash2 (a block-diffusion drafter) at depth 4. The
-  drafter has a sliding window of 2048 (small draft KV, ~40 MiB).
+- **Endpoint:** `frank-large` at `192.168.0.143:7770` (Dankclaw).
+- **Runtime:** stock llama.cpp — the unsloth `b11030` prebuilt
+  (`/home/dank/GitPackages/unsloth-llama-b11030/llama-server`), ROCm gfx1151.
+  A mainline build, not a fork: it supports the `qwen3_5` architecture and the
+  DFlash2 drafter (`llama_model_qwen35`, `llama_model_dflash`,
+  `common_speculative_impl_draft_dflash`).
+- **Service shape:** root systemd unit `scripts/frank-large.service`
+  (`User=dank`), launched by `scripts/serve-frank-large.sh` with tunables from
+  `scripts/frank-large.env`.

 ## Shape

-One root unit, one process, one published port:
-
-| half | port | reachable |
-|---|---|---|
-| llama-server (OpenAI front-end + engine, single process) | 7770 | published as host **7770** |
-
-## Runtime
-
- **Prebuilt**: `/home/dank/GitPackages/unsloth-llama-b11030/llama-server`
-  (mainline, ROCm gfx1151 prebuilt, version 0.4.1-dev, build 11030,
-  commit 6ba30d05b). The build bundles its own ROCm runtime — no host SDK
-  gate, no venv.
- **GPU target**: `gfx1151` (Strix Halo).
- **Spec-draft flags**: `--spec-draft-type-k`, `--spec-draft-type-v`,
-  `--spec-draft-threads`, `--spec-draft-cpu-mask`, `--spec-draft-prio`,
-  `--spec-draft-poll` all present (confirmed via `--help`).
+- **Model:** `unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL` — a 27B **dense** model,
+  31.46 GiB GGUF (sha256-verified by `scripts/download-frank-large.sh`).
+- **Context:** one stream, full 262,144 tokens (the model's native window),
+  `PARALLEL_SLOTS=1`.
+- **Placement:** all layers on the iGPU (`--n-gpu-layers 999`). The 27B dense
+  weights (31.46 GiB) + the DFlash2 drafter (1.14 GiB) + q8_0 KV (~8.6 GiB) +
+  compute buffers fit the iGPU's GTT aperture (~114 GiB).

 ## Spec method

- **DFlash2 depth 4** (`--spec-type draft-dflash --model-draft <drafter>
-  --spec-draft-n-max 4 --spec-draft-n-min 0`): the block-diffusion
-  drafter proposes 4 tokens per step.
- **Draft device**: `--spec-draft-device ROCm0` (the drafter runs on the
-  iGPU).
- **DFlash2 is a block-diffusion drafter** (not a standalone LM). The
-  drafter has a sliding window of 2048 (~40 MiB of draft KV).
+DFlash2 (a block-diffusion drafter) at **depth 4**. The drafter is the z-lab
+`Qwen3.8-27B-DFlash2-Q4_K_M` (1.14 GiB), loaded via `--model-draft` with
+`--spec-type draft-dflash` and `--spec-draft-device ROCm0`. It has a sliding
+window of 2048, so its draft KV is small (~40 MiB). Greedy verification
+(`--temp 0 --top-p 1.0 --top-k 0 --min-p 0.0`) is required.

-## Chat template
+## KV

-The serve script passes `--chat-template-file` pointing at
-`dankserver/frank-medium/qwen-sharp-v22.4.0.jinja` (the Qwen-Sharp
-v22.4.0 template, the same one frank-medium uses). Without this flag the
-binary falls back to the built-in `peg-native` template, which produces a
-~73,600-token prompt for a ~250-token request (280x the expected size).
-With the sharp template wired in, the same request produces a 62–175-token
-prompt. The `Chat format: peg-native` log line is the format-type label,
-not the template indicator — the sharp template is active even when that
-line appears.
+**q8_0** (`--cache-type-k q8_0 --cache-type-v q8_0`), ~8.6 GiB at 262k. The
+unsloth b11030 build does **not** support fp8 KV (allowed: f32/f16/bf16/q8_0/
+q4_0/q4_1/iq4_nl/q5_0/q5_1); q8_0 is the highest-precision 8-bit option.

-## Memory budget
+## Template

-The iGPU addresses **114 GiB** (the GTT aperture over the shared LPDDR,
-`gtt_total` in `/sys/class/drm/card1/device/`; `rocm-smi`'s 2 GiB
-"dedicated" is not the budget). The 27B dense model (31.46 GiB GGUF) is
-mmap'd in with `-ngl 999`; budget at 262k ≈ 41–43 GiB (weights + drafter
-+ fp8 KV + buffers), leaving ~71–73 GiB of aperture headroom. The 8 GiB
-prompt-cache budget (`CACHE_RAM=8192`) is the qualified config. The model
-is a 27B hybrid (16 full attention + 48 Gated DeltaNet); its DeltaNet
-state is smaller than a primarily-DeltaNet model, so the fleet's 12.8
-GiB DeltaNet `CACHE_RAM` floor does not apply.
-
-**KV budget**: 16 full-attention layers × 2 (K+V) × 4 kv_heads × 256
-head_dim × 1 byte (fp8) = 32,768 bytes/token. 262,144 tokens × 32,768 ≈
-8 GiB for attention KV. Plus 48 DeltaNet recurrent states (small).
+The Qwen-Sharp v22.4.0 template is pinned via `--chat-template-file`
+(`dankserver/frank-medium/qwen-sharp-v22.4.0.jinja`) with `--jinja` and
+`--chat-template-kwargs '{"preserve_thinking":true}'`. Without
+`--chat-template-file` the binary falls back to the built-in `peg-native`
+template, which produces a ~73,600-token prompt for a ~250-token request.
+`preserve_thinking` keeps prior thinking blocks across turns for multi-turn
+agents. The template gates `reasoning_effort` to `{xhigh, medium, low}` — the
+harness folds its levels onto that set in `models.yml` (`reasoningEffortMap`).

 ## Payload

-`scripts/download-frank-large.sh` (resumable, each file verified against
-its declared byte count + sha256; `--verify-only` re-checks):
-
-| file | size | sha256 |
-|---|---|---|
-| `Qwen3.8-27B-UD-Q8_K_XL.gguf` | 31,457,991,680 | `af36ecb6…` |
-| `Qwen3.8-27B-DFlash2-Q4_K_M.gguf` | 1,143,006,816 | `1a25c568…` |
+`scripts/download-frank-large.sh` fetches the main GGUF and the DFlash2
+drafter, each verified against its declared byte count + sha256
+(`--verify-only` re-checks). The serve script refuses to start if the binary,
+the main GGUF, or the drafter is missing.

 ## Client contract

- **Port**: 7770 (the same port and alias the prior deployments served).
- **Alias**: `frank-large`.
- **Context**: 262,144 (the model's native window), 1 slot.
- **KV**: fp8 quantized (`--cache-type-k fp8 --cache-type-v fp8`).
- **Sampling**: greedy (`--temp 0 --top-p 1.0 --top-k 0 --min-p 0.0`) —
-  the DFlash2 drafter needs greedy verification.
- **Reasoning**: enabled (`--reasoning auto --reasoning-format deepseek
-  --reasoning-budget -1`), extracted to `reasoning_content`.
- **Vision**: text-only (the model has a vision projector — mmproj-BF16
-  is available — but this deployment serves text-only, matching the prior
-  frank-large). The fleet's image path is [[frank-small-gemma4-12b]]
-  either way.
- **Reasoning effort**: the Qwen3.8 template gates `reasoning_effort` to
-  {xhigh, medium, low} — the harness folds its levels onto that set in
-  `models.yml` (`reasoningEffortMap`).
-
-## Measured performance
-
-(To be filled in after deploy.)
+- **Alias:** `frank-large` (the `id` at `/v1/models`).
+- **Reasoning:** extracted to `reasoning_content` (`--reasoning auto`,
+  `--reasoning-format deepseek`, unlimited budget).
+- **Tools:** supported (`--jinja` is pinned).
+- **Vision:** none — text-only (no mmproj), matching the prior frank-large.
+- **Sampling:** greedy by default; a client that sends a temperature gets its
+  own, and the harness applies per-model overrides from `models.yml`.

 ## Failure modes worth knowing

- **The model has `mtp_num_hidden_layers: 1`** but this deployment uses
-  DFlash2 (not MTP) for spec decode. The MTP layer is unused.
- **The model has vision** (`language_model_only: false`, image_token_id
-  248056, mmproj-BF16 0.93 GB available) but this deployment serves
-  text-only.
- **DFlash2 needs greedy verification**: the serve script pins
-  `--temp 0 --top-p 1.0 --top-k 0 --min-p 0.0`. A non-greedy request
-  will not be verified correctly by the drafter.
+- **Model load is slow (minutes cold).** Health returns 503 while loading; do
+  not kill the process. `deploy.sh` waits up to 20 minutes (`LLM_WAIT`).
+- **`--ngl` (double-dash) is invalid in b11030** — use `--n-gpu-layers`
+  (the serve script uses the long form).
+- **The 8-bit UD-Q8_K_XL decodes ~17 t/s** vs the prior 4-bit Kairic Edge IU4
+  at ~22 t/s — the ~23 % larger weight size (31.46 vs 26 GiB) explains the
+  gap; it is not a regression.
+- **DFlash2 draft acceptance is text-variety-dependent** (0.85 on a trivial
+  one-word answer, 0.519 on a varied explanatory prompt) — do not read a
+  single-request acceptance as the steady state.

 ## Rollback

-[[frank-large-kairic-edge-qwen38-27b]] — the Kairic Edge payload and
-runtime are intact on disk.
+Payload and runtime for [[frank-large-kairic-edge-qwen38-27b]] (Kairic Edge
+IU4 W4A4) remain intact on disk. See [[frank-large-unsloth-qwen38-flash-next]]
+for the Flash-Next predecessor it replaced on 2026-09-21.
+
+# Citations
+
+- unsloth prebuilts: https://huggingface.co/unsloth
+- Model: https://huggingface.co/unsloth/Qwen3.8-27B-GGUF
+- DFlash2 drafter: https://huggingface.co/z-lab
+- Serve script: `scripts/serve-frank-large.sh`; tunables: `scripts/frank-large.env`
@@ -1,7 +1,7 @@
 ---
 type: reference
-title: unsloth llama.cpp Qwen3.8-Flash-Next UD-IQ4_XS on Dankclaw (frank-large)
-description: The active frank-large runbook — the unsloth llama.cpp prebuilt (b11030-mix-5ff778e, ROCm gfx1151) serving unsloth/Qwen3.8-Flash-Next-GGUF:UD-IQ4_XS on the Dankclaw Strix Halo iGPU: the prebuilt runtime, the 200k/one-stream shape, the explicit -md MTP wiring, the payload, the client contract, and the failure modes worth knowing.
+title: "unsloth llama.cpp Qwen3.8-Flash-Next UD-IQ4_XS on Dankclaw (frank-large)"
+description: "The RETIRED frank-large runbook (2026-09-21, same-day swap; replaced by [[frank-large-unsloth-qwen38-27b-q8xl]]) — the unsloth llama.cpp prebuilt (b11030-mix-5ff778e, ROCm gfx1151) serving unsloth/Qwen3.8-Flash-Next-GGUF:UD-IQ4_XS on the Dankclaw Strix Halo iGPU: the prebuilt runtime, the 200k/one-stream shape, the explicit -md MTP wiring, the payload, the client contract, and the failure modes worth knowing."
 tags:
  - llm
  - unsloth
@@ -15,182 +15,33 @@ tags:
 timestamp: 2026-09-21
 ---

-# unsloth llama.cpp Qwen3.8-Flash-Next UD-IQ4_XS on Dankclaw (frank-large)
+# frank-large — unsloth llama.cpp Qwen3.8-Flash-Next UD-IQ4_XS (RETIRED)

-> **ACTIVE since 2026-09-21.** It replaced the CIRU v4.4.1 Orca runtime
-> (see [[frank-large-ciru-orca]], the current rollback target), which had
-> replaced halogen-flash-server (see [[halogen-flash-server]]). Deployed as a
-> **root systemd unit** (`scripts/frank-large.service`, `User=dank`), port
-> 7770, on the same gfx1151 iGPU. The RTX 3090 it does not touch belongs to
-> [[frank-small-gemma4-12b]].
+> **RETIRED 2026-09-21 (same-day swap).** This was the ACTIVE frank-large
+> runbook on 2026-09-21 before it was replaced by
+> [[frank-large-unsloth-qwen38-27b-q8xl]] (27B dense UD-Q8_K_XL) later the
+> same day. The payload and runtime remain on disk as the rollback target.

-The model is `unsloth/Qwen3.8-Flash-Next-GGUF` at **UD-IQ4_XS** (unsloth
-dynamic quant): the 125B-param / 6B-active Qwen3.8-Flash-Next MoE with its
-51B n-gram embedding (PLE) folded into the single GGUF, 48 layers of
-12 × (3 × (Gated DeltaNet → MoE) → 1 × (QSA → MoE)), native 262,144-token
-window, vision encoder. The runtime is the **unsloth prebuilt**
-`app-b11030-mix-5ff778e-linux-x64-rocm-gfx1151` — the first runtime with an
-MTP graph for the qwen4exp architecture (stock ggml-org mainline has no MTP
-graph, no cross-model tensor borrowing, and no `--spec-type draft-mtp` for
-this model).
+## What it served

-## Why this swap
-
- The operator asked for llama.cpp + the unsloth UD-IQ4_XS quant at
-  **one conversation, 200k context** (2026-09-21).
- UD-IQ4_XS (87.3 GiB) is a standard llama.cpp quant — no private runtime,
-  no PLE sidecar payload, no QSA hotfix: one GGUF, one MTP head, done.
- The unsloth prebuilt carries the qwen4exp MTP graph (1.3–1.7x at
-  concurrency 1, per the model card), which the CIRU fork also had but which
-  stock mainline does not.
-
-## Shape
-
-One root unit, one process, one published port:
-
-| half | port | reachable |
-|---|---|---|
-| llama-server (OpenAI front-end + engine, single process) | 7770 | published as host **7770** |
-
-## Runtime
-
- **Prebuilt**: `app-b11030-mix-5ff778e-linux-x64-rocm-gfx1151.tar.gz` from
-  the [unslothai/llama.cpp releases](https://github.com/unslothai/llama.cpp/releases)
-  (tag `b11030-mix-5ff778e` or newer; the MTP graph landed in
-  `b10715-mix-86bd2d3`). 0.4 GB, bundles its own ROCm runtime
-  (`libamdhip64.so.7.16`), so there is **no host SDK to gate on and no venv
-  to symlink** — unlike the CIRU predecessor's runtime-root gate.
- **Installed at** `/home/dank/GitPackages/unsloth-llama-b11030/`
-  (extracted tarball; `llama-server` at the top level).
- Verified on-host (2026-09-21): `--version` → `0.4.1-dev (build 11030,
-  commit 6ba30d05b)`; `--list-devices` → `ROCm0: AMD Radeon 8060S Graphics
-  (116732 MiB)`; a Gemma4-12B smoke load served a clean completion on the
-  iGPU before the cutover.
-
-## Model
-
- **Checkpoint**: `unsloth/Qwen3.8-Flash-Next-GGUF` → `UD-IQ4_XS` (the
-  `:UD-IQ4_XS` tag is a directory in the repo, not a submodel).
- **Transport**: three llama.cpp split parts (HF caps uploads at 50 GB):
-  `00001` (10,946,624 B) + `00002` (49,835,229,856 B) + `00003`
-  (43,836,407,744 B) = **93,682,584,224 B**. The parts ARE the canonical
-  payload — the serve script points `-m` at part 1 and llama.cpp loads the
-  3-way split natively (parts 2–3 auto-discovered in the same directory).
-  Do **not** concatenate them into a single file to serve: part 1's header
-  carries `ggml.file_count=3`, so a byte-correct concatenation still
-  advertises itself as a split and the loader rejects the non-split
-  filename with `invalid split file name` (caught on 2026-09-21 — the
-  assembled file crash-looped the unit ~110 times before the diagnosis).
- **On disk** at `/home/dank/models/Qwen3.8-Flash-Next-UD-IQ4_XS/`:
-  - `UD-IQ4_XS/Qwen3.8-Flash-Next-UD-IQ4_XS-0000{1,2,3}-of-00003.gguf` — the
-    main model, 3-way split (87.3 GiB)
-  - `mtp/mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf` — MTP head (2.6 GiB)
-  - `vision/mmproj-F16.gguf` — projector (904 MB, not loaded while
-    `ENABLE_VISION=0`)
- **Fetch**: `scripts/download-frank-large-unsloth.sh` — resumable
-  (`curl -fL -C -`), each part verified against its **LFS sha256 fetched
-  live from the Hub tree API** (not a file that could drift).
-  `--verify-only` re-checks without downloading.
- The n-gram embedding (PLE) is **inside the GGUF** as ordinary tensors —
-  there is no external PLE payload and no pager, unlike the CIRU Orca
-  packaging (79.4 GiB main + 52.4 GiB paged PLE).
-
-## Deployed config
-
-From `scripts/frank-large.env` (the flag list is owned by
-`scripts/serve-frank-large.sh`):
-
-| knob | value | note |
-|---|---|---|
-| `MODEL_DIR` | `/home/dank/models/Qwen3.8-Flash-Next-UD-IQ4_XS` | serve script refuses to start without the GGUF and the MTP head |
-| `RUNTIME_DIR` | `/home/dank/GitPackages/unsloth-llama-b11030` | prebuilt tarball, extracted |
-| `HOST` / `PORT` | `0.0.0.0` / `7770` | same address the CIRU predecessor served |
-| `CONTEXT_SIZE` | `200000` | operator's requested shape (2026-09-21); trained window is 262144 |
-| `PARALLEL_SLOTS` | `1` | one conversation |
-| `ENABLE_MTP` | `1` | the shared-Q8_0 head speculates |
-| `SPEC_DRAFT_N_MAX` | `2` | the model card's default ("higher drafts more but each guess is accepted less often") |
-| `ENABLE_VISION` | `0` | projector on disk, one env flip away |
-
-Serve flags: `-m <GGUF> -md <MTP head> --spec-type draft-mtp
--spec-draft-n-max 2 -c 200000 -np 1 -ngl 999 --jinja --alias frank-large
--host 0.0.0.0 --port 7770 --chat-template-kwargs
-'{"preserve_thinking":true}'`.
-
-### The explicit `-md` (do not drop it)
-
-Sidecar auto-discovery does **not** search the `mtp/` subfolder. Without
-`-md`, `--spec-type draft-mtp` finds no head and the server runs at the main
-model's speed **with no error saying why**. The serve script gates on the
-head's presence and passes it explicitly.
-
-### Memory model
-
-The iGPU (Radeon 8060S, gfx1151) addresses **116,732 MiB (~114 GiB)** — the
-GTT aperture over the shared LPDDR (`gtt_total=122402963456` in
-`/sys/class/drm/card1/device/`, confirmed by `llama-server --list-devices`).
-The 87.3 GiB model is mmap'd into that aperture with
-`-ngl 999`, the same regime the CIRU Orca predecessor ran in (92.6 GiB
-occupied at rest). Budget at 200k: weights 87.3 + MTP head 2.6 + KV ~5
-(QSA 12 layers × 2 KV heads × 256 dim × f16 × 200k) + compute buffers ~5
-≈ **100 GiB**, leaving ~14 GiB of aperture headroom. The host's 121 GiB of
-LPDDR is shared with the CPU and the ~28 co-tenant containers; the 64 GiB
-swap file is the backstop.
-
-## Client contract
- OpenAI-compatible chat completions at `:7770/v1/chat/completions`;
-  `/health` 200 when serving.
- **Reasoning** is extracted to `reasoning_content` (llama.cpp's field —
-  this is a llama.cpp runtime, not vLLM's `reasoning`).
- **Tools** work via `--jinja` (the model's own template). The Qwen3.8
-  template gates `reasoning_effort` to `{xhigh, medium, low}` — the harness
-  folds its levels onto that set in `models.yml` (`reasoningEffortMap`).
-  `preserve_thinking` is on by default (request-level
-  `chat_template_kwargs` still wins).
- **Vision**: off (`ENABLE_VISION=0`); the F16 projector is on disk. The
-  fleet's image path is frank-small.
- **Audio**: none (no audio in the fleet since the FLM NPU endpoint retired
-  2026-09-11).
-
-## Troubleshooting
-
- **`missing runtime` / `missing target model` / `missing MTP head`** from
-  the serve script: the unit exits with a pointer instead of restart-looping.
-  Re-run the fetch script (`scripts/download-frank-large-unsloth.sh`) or
-  re-extract the prebuilt.
- **Health 503 for minutes after (re)start**: the 87 GiB load is slow cold
-  (page-cache miss). Do not kill the process; `deploy.sh` waits up to 20
-  minutes (`LLM_WAIT`).
- **Serving at the main model's speed with no spec metrics**: the MTP head
-  is not attached — check the process cmdline for `-md` (see above).
- **`Address already in use` on 7770**: a stray manual run of the same model
-  holds the port. The deploy handler sweeps `pkill -f
-  'Qwen3.8-Flash-Next-UD-IQ4_XS'` (model path, not binary name — frank-small
-  is also a `llama-server`) and gates on `wait_port_free 7770`.
- **`invalid split file name` at model load** (fails in ~50 ms, before any
-  weights are read): `-m` points at a file whose GGUF header advertises a
-  split (`ggml.file_count > 1`) but whose name is not a
-  `-NNNNN-of-NNNNN` part. This is what a concatenated split file does —
-  the concatenation is byte-correct but part 1's header still says
-  "3-way split". Fix: point `-m` at part 1
-  (`UD-IQ4_XS/...-00001-of-00003.gguf`) and let the loader pick up parts
-  2–3 from the same directory.
- **Host memory pressure**: this endpoint is the largest LPDDR consumer on
-  the box. If the co-tenants start thrashing, the levers are KV quant
-  (`--cache-type-k/v q8_0` halves the ~5 GiB KV) and `SPEC_DRAFT_N_MAX`
-  (smaller draft state), not the weights.
+- **Runtime:** unsloth llama.cpp prebuilt `b11030-mix-5ff778e` (ROCm
+  gfx1151), root systemd unit, port 7770.
+- **Model:** `unsloth/Qwen3.8-Flash-Next-GGUF:UD-IQ4_XS` (a 125B MoE / 6B
+  active), 87.3 GiB shipped as three split parts. The split parts ARE the
+  payload — concatenating them trips `invalid split file name` (part 1's
+  header advertises `file_count=3`).
+- **Shape:** one 200,000-token conversation (single stream), MTP depth 2 via
+  the explicit `-md` shared-Q8_0 head. Draft acceptance verified live
+  0.83–0.92.
+- **Memory:** the 114 GiB GTT aperture on the Strix Halo iGPU.

 ## Rollback

-The CIRU Orca payload and runtime are intact on disk:
-`/home/dank/models/Qwen3.8-Flash-CIRU-STRIX-Orca/` (main GGUF, MTP head, PLE
-payload) and `/home/dank/GitPackages/ciru-runtime-v4.4.1-qsa-hotfix/`.
-Rollback is one commit restoring the CIRU `serve-frank-large.sh` +
-`frank-large.env` + the deploy handler's `LLM_LARGE_*` paths (the
-predecessor's version is in git history), then `deploy.sh dankclaw
--service frank-large`. See [[frank-large-ciru-orca]].
+Payload and runtime intact on disk. See [[frank-large-unsloth-qwen38-27b-q8xl]]
+for the current ACTIVE runbook and [[frank-large-ciru-orca]] for the
+predecessor.

-## References
+# Citations

- [unsloth/Qwen3.8-Flash-Next-GGUF](https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF) — model card, MTP README, imatrix
- [unslothai/llama.cpp releases](https://github.com/unslothai/llama.cpp/releases) — the prebuilt runtime (tag `b11030-mix-5ff778e`)
- [MTP PR #28243](https://github.com/ggml-org/llama.cpp/pull/28243) — the qwen4exp MTP graph that the prebuilt carries
+- unsloth prebuilts: https://huggingface.co/unsloth
+- Model: https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF
@@ -1,4 +1,5 @@
 ---
+timestamp: 2026-09-17
 type: reference
 title: frank-medium — Batch and Ubatch Sweep
 description: llama-bench sweep of --batch-size against --ubatch-size on frank-medium (2x Radeon AI PRO R9700, layer split) at the deployed context shape, and the rebalance it forced. Prefill peaks at ubatch 512 and falls on BOTH sides.
@@ -621,6 +621,8 @@ if that person is looking. That is the honest state of "alerting" here.

 ## Citations

+- vLLM upstream: https://github.com/vllm-project/vllm — the R4D one-shot
+  allreduce and the tensor-parallel path this deployment runs.
 - `ROCm/clr` `rocclr/device/rocm/rocvirtual.cpp:3963` — the hostcall-buffer
  refusal; `rocclr/device/rocm/rocdevice.cpp:186` — `checkAtomicSupport`.
 - Linux `drivers/pci/pci.c` — `pci_enable_atomic_ops_to_root()` and its
@@ -1,3 +1,12 @@
+---
+type: reference
+description: "This page records the full redo of the inference fleet on 2026-08-17."
+tags:
+  - llm
+  - frank
+timestamp: 2026-09-03
+---
+
 # Frank Program Redo — 2026-08-17

 This page records the full redo of the inference fleet on 2026-08-17.
@@ -1,4 +1,5 @@
 ---
+timestamp: 2026-09-03
 type: reference
 title: frank-small Benchmark — ncmoe and Batch Tuning
 description: llama-bench sweep of MoE CPU offload (ncmoe) and batch/ubatch on the RTX 3090 for Qwen3.6-35B-A3B Q6_K_P.
@@ -1,7 +1,7 @@
 ---
 type: reference
-title: frank-small on Dankclaw — llama.cpp, Gemma4-12B QAT Q4_K_M
-description: The active frank-small runbook — mainline llama.cpp on Dankclaw's RTX 3090 serving HauhauCS Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced at Q4_K_M with a separate MTP assistant sidecar chained behind ngram-mod, q8_0 KV, and --fit sizing the full 262k window: identity, payload, KV arithmetic, the spec-decode chain, deployed flags, the client contract, and what to check when it misbehaves.
+title: "frank-small on Dankclaw — llama.cpp, Gemma4-12B QAT Q4_K_M"
+description: "The active frank-small runbook — mainline llama.cpp on Dankclaw's RTX 3090 serving HauhauCS Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced at Q4_K_M with a separate MTP assistant sidecar chained behind ngram-mod, q8_0 KV, and --fit sizing the full 262k window: identity, payload, KV arithmetic, the spec-decode chain, deployed flags, the client contract, and what to check when it misbehaves."
 tags:
  - llm
  - frank-small
@@ -14,172 +14,107 @@ tags:
 timestamp: 2026-09-17
 ---

-# frank-small — llama.cpp on the RTX 3090 (Gemma4-12B QAT)
+# frank-small — llama.cpp Gemma4-12B QAT Q4_K_M (MTP + ngram-mod)

 > **ACTIVE since 2026-09-17.** Replaces [[frank-small-qwen38-gsq-rco]]
-> (one-day rollback target). Same unit, same port, same client contract —
-> the model, the drafter shape and the KV type changed.
+> (ISTA-DASLab Qwen3.8-27B-GSQ-RCO IQ3_S, now the rollback target) and
+> [[frank-small-vllm-gemma4]] (the same-day vLLM experiment, also the
+> rollback target). Same port (7771), same alias (`frank-small`), same
+> client contract surface.

 ## Identity

-| what | value |
-|---|---|
-| runtime | mainline llama.cpp master `37b53fd`, CUDA build **with `-DGGML_CUDA_FA_ALL_QUANTS=ON`** (rebuilt 2026-09-17 for q8 KV) |
-| binary | `/home/dank/GitPackages/llama.cpp-mainline/build-cuda/bin/llama-server` |
-| unit | **root systemd unit** `frank-small.service` (repo `scripts/frank-small.service`, `relink_unit`ed into `/etc/systemd/system/`) |
-| model | `HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced` Q4_K_M |
-| model dir | `/home/dank/models/Gemma4-12B-QAT/` |
-| port | 7771 (alias `frank-small`; `frank-large` keeps 7770) |
-| window | **4 slots x 262,144** (`CTX_SIZE=1048576`, `PARALLEL=4`); q8_0 K and V |
-| vision + audio | fleet image path via BF16 mmproj (`input: [text, image]`); `input_audio` transcribes verbatim (verified) |
+- **Endpoint:** `frank-small` at `192.168.0.143:7771` (Dankclaw).
+- **Runtime:** mainline llama.cpp master `37b53fd`, CUDA build for sm_86 with
+  `-DGGML_CUDA_FA_ALL_QUANTS=ON`
+  (`/home/dank/GitPackages/llama.cpp-mainline/build-cuda/bin/llama-server`).
+  The FA_ALL_QUANTS flag is required: without it, q8_0 KV + flash attention
+  collapses prefill ~12x on CUDA.
+- **Service shape:** root systemd unit `scripts/frank-small.service`
+  (`User=dank`), launched by `scripts/serve-frank-small.sh` with tunables from
+  `scripts/frank-small.env`.

-## Payload (all sha256-verified against the Hub)
+## Shape

-| file | bytes | sha256 (head…) |
-|---|---|---|
-| `Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced-Q4_K_M.gguf` | 7,381,381,760 | `59656d74…` |
-| `mmproj-Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced-BF16.gguf` | 175,115,264 | `b59e8154…` |
-| `mtp-gemma-4-12B-it.gguf` | 253,707,328 | `c50c91c3…` |
+- **Model:** `HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced` at
+  Q4_K_M (6.87 GiB). Architecture `gemma4`: 48 layers, MHA with 16 kv heads
+  x 512 head_dim, rope freq_base 1e6, final logit softcapping 30, per-layer
+  token embeddings (PLE) inside the file, and a sliding-window attention
+  pattern (`attention.sliding_window = 1024`) — most layers cache only a
+  1024-position window, which is why the KV pool is a fraction of the
+  full-ctx x all-layers number.
+- **Context:** four FULL 262,144-token slots (`CTX_SIZE=1048576`,
+  `PARALLEL=4`). The explicit `CTX_SIZE` bypasses `--fit` on purpose: `--fit`
+  maximises the TOTAL pool, which would shrink each slot below the full window
+  once parallel > 1. The 4-slot boot idles at 22,346 MiB; each extra
+  full-context slot costs 3,450 MiB, and a fifth projects to ~25.8 GiB — past
+  the card. 4 is the measured max.
+- **Placement:** the whole model in VRAM (`--gpu-layers 999`); the host CPU
+  only tokenizes and samples (its CPU belongs to the halogen engine serving
+  frank-large).

-`sha256.txt` beside them on the host carries the full digests.
+## Spec method

-## What the preflight found (GGUF headers, read over HTTP Range)
+The MTP head ships as a **separate** file (`mtp-gemma-4-12B-it.gguf`, arch
+`gemma4-assistant`, 4 nextn layers, 0.24 GiB), loaded via `--model-draft`
+with `--spec-type draft-mtp`. The same `--spec-type` value chains the
+draftless **ngram-mod** matcher, which attaches unconditionally; two
+context-backed drafters cannot share one process, so `draft-mtp,ngram-mod` is
+the only legal shape that carries both. Draft depth 3 + poll 50 (measured:
+depth 4 + poll 100 decoded WORSE at long context — the longer verification
+round costs more attention re-read than the extra accepted token earns).

- **Target** (`arch gemma4`, 48 blocks): ctx 262144, MHA — 16 kv heads x
-  512 dim (no GQA), embedding 3840, rope base 1e6, logit softcap 30,
-  **sliding_window 1024**, `swa_type = STANDARD`, per-layer token embeddings
-  (PLE) via `LLM_KV_EMBEDDING_LENGTH_PER_LAYER`. 12 full-attention layers
-  every 4th (`[3,7,…,47]` implied by `full_attention_interval = 4`).
-  48 blocks maps to `LLM_TYPE_UNKNOWN` in the loader's size switch —
-  cosmetic only.
- **mmproj**: clip, vision 224/16 patches, **also carries an audio
-  encoder** (`has_audio_encoder=True`) — vision is the fleet contract,
-  and audio works: a JFK WAV sent as an `input_audio` content part was
-  transcribed verbatim (2026-09-17).
- **MTP sidecar** (`arch gemma4-assistant`): 4 nextn layers, pure
-  assistant (no trunk). Its KV **shares** the target's last layers — the
-  share lambda at `llama-model.cpp:2662` references
-  `llama_model_n_layer(model_other) - 2` for SWA layers. Adding the
-  drafter therefore costs no second KV pool.
+## KV

-## KV arithmetic — why four full windows fit
+**q8_0** (`--cache-type-k q8_0 --cache-type-v q8_0`), with flash attention
+(`--flash-attn on`, required for quantized KV). CPU-side cache capped at 2048
+MiB (`CACHE_RAM=2048`) to give back ~6 GiB of LPDDR — Gemma4 runs fully on the
+3090, so the CPU cache is not the hot path, and it is not a DeltaNet model so
+the >12.8 GiB floor does not apply.

-Per token per layer at q8_0: 16 heads x 512 dim x 1 B x 2 (K,V) =
-**16 KiB** (32 KiB at f16). With SWA_STANDARD and window 1024, the
-sliding layers cache only the window; the full-attention layers cache the
-full context — the pattern that makes the window cheap (the naive
-"all-global" worst case was 32 GiB at q8/262k, impossible).
+## Vision + audio

-Measured on the 3090 (24,576 MiB): one full-context slot idles at
-11,996 MiB, four at 22,346 MiB — 3,450 MiB per slot, ~8,546 MiB fixed.
-A fifth slot projects past the card, so **4 x 262,144 is the ceiling**.
-`CTX_SIZE` is set explicitly to `PARALLEL x 262144` rather than left for
-`--fit`: fit maximises the TOTAL pool and would divide a smaller window
-per slot at `parallel > 1`. The slot-grid seam in
-`tests/frank_small_unit.bats` pins both the full-window division and the
-budget inequality so an oversized bump fails pre-deploy, not in boot.
+`--mmproj` loads the BF16 projector
+(`mmproj-Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced-BF16.gguf`, 11 tensors,
+52M params), which also carries an audio encoder. The fleet's image path
+lives here. Audio works: `input_audio` content parts transcribe verbatim
+(verified 2026-09-17 with a 16k mono WAV).

-## Spec-decode chain
+## Payload

-`--spec-type draft-mtp,ngram-mod` with `--model-draft
-$MODEL_DIR/mtp-gemma-4-12B-it.gguf`:
-
- `draft-mtp` is context-backed and reads the assistant sidecar. Two
-  context-backed drafters cannot share one process, so the chain carries
-  exactly one.
- `ngram-mod` is draftless and attaches unconditionally, catching
-  prompt-echo traffic MTP would miss. It prints **no startup banner** —
-  verify behaviorally via the per-task `draft acceptance` line, not the
-  load log.
- Measured at deploy: `draft acceptance = 0.38–0.46, mean len ≈ 2.2`
-  (lower acceptance than the retired Qwen in-file MTP's 0.66–0.94, but a
-  much smaller model — the point of the swap was the VRAM and the window,
-  not acceptance).
-
-## Throughput tuning (measured 2026-09-17)
-
-Salted API probes at the deployed geometry (4 slots, q8 KV, flash attn),
-prefill ~254k tokens / decode at ~253k context:
-
-| batch (b/ub) | draft n-max / poll | pp t/s | tg@250k t/s | idle MiB |
-|---|---|---|---|---|
-| 2048/512 (old) | 3 / 50 | 912–924 | 90*–204 | 22,346 |
-| **8192/1024 (deployed)** | **3 / 50** | **962–965** | **226–302** | **23,300** |
-| 8192/1024 | 4 / 100 | 961 | 230–257 | 23,300 |
-| 4096/1024 | 4 / 100 | 959 | 205–277 | 23,300 |
-| 8192/2048 | — | — | — | OOM |
-
- **ubatch 1024 is the winner**: +5% max-length prefill, and decode at
-  250k clearly above the 512 baseline.
- **ubatch 2048 cannot run**: the MTP draft context needs ~3.3 GiB of pp
-  compute buffer at that ubatch against ~2.2 GiB headroom — it dies in
-  `common_speculative_init_result` after the main model loads.
- **Deeper draft and busy-wait HURT long-ctx decode**: draft n-max 4 with
-  poll 100 verified a longer round per token, and at 250k the extra
-  full-attention re-read outweighs the accepted-token gain (230–257 vs
-  257–302). Keep 3 / 50.
- **All ubatch-1024 variants idle identically** (23,300 MiB) — compute
-  buffers are reserved at max graph size regardless of the logical batch,
-  so 8192 costs nothing over 4096.
- **Concurrency caveat**: a long prefill monopolises the GPU — a decode
-  running beside a 254k-token prefill drops from ~240 to ~100–135 t/s.
-  Four slots share one engine with no per-slot QoS; that is inherent, not
-  a config fault.
- **No CUDA graphs in this build** — `--help` shows zero graph flags; the
-  decode-launch-overhead lever does not exist here.
-
-## Deployed flags
-
-Repo truth: `scripts/frank-small.env` + `scripts/serve-frank-small.sh`.
-Key values: `MODEL_DIR=/home/dank/models/Gemma4-12B-QAT`,
-`MODEL_FILE=Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced-Q4_K_M.gguf`,
-`DRAFT_FILE=mtp-gemma-4-12B-it.gguf`, `CACHE_TYPE_K=V=q8_0`, `CACHE_RAM=2048`,
-`SPEC_TYPE=draft-mtp,ngram-mod`, `SPEC_DRAFT_N_MAX=3`, `POLL_LEVEL=50`,
-`CTX_SIZE=1048576` with `PARALLEL=4` (four full 262,144-token slots),
-`BATCH_SIZE=8192`, `UBATCH_SIZE=1024`, `NGL=999`, `FLASH_ATTN=on`,
-`--jinja`, `--metrics`, greedy `--temp 0.0` server default. The serve
-script refuses to start on any missing file.
+The main GGUF, the BF16 mmproj, and the MTP sidecar live under
+`/home/dank/models/Gemma4-12B-QAT`. The serve script refuses to start if any
+is missing (a missing file is an unfinished download, and the unit's restart
+policy would otherwise turn it into a crash loop that reads like a runtime
+fault).

 ## Client contract

- `reasoning_content` carries thinking; `preserve_thinking: true` is sent
-  by the harness to keep it across turns.
- **Every `reasoning_effort` level is accepted** (all seven probed
-  2026-09-17: 200) — unlike Qwen3.8's template gate. The harness's
-  `reasoningEffortMap` in `~/.omp/agent/models.yml` is therefore inert
-  but kept as a safety net.
- Tools work through the `--jinja` template (`tool_calls` verified).
- Images work through the mmproj (verified with a live JPEG).
- Four concurrent conversations (one per slot). No per-slot QoS: a long
-  prefill in one slot slows decodes in the others — see the caveat above.
- Harness: `contextWindow: 262144`, `maxTokens: 32768`.
+- **Alias:** `frank-small` (the `id` at `/v1/models`).
+- **Reasoning:** extracted to `reasoning_content` (`--reasoning-format
+  deepseek`, unlimited budget).
+- **Tools:** supported (`--jinja` is pinned even though this build defaults
+  it on, so a future build that flips the default cannot silently take tools
+  away).
+- **Vision + audio:** supported.
+- **Sampling:** greedy by default (fleet contract); a client that sends a
+  temperature gets its own, and the harness applies per-model overrides from
+  `models.yml`.

-## Troubleshooting
+## Measured (max length)

- **`Gemma4Assistant requires ctx_other to be set` printed at boot** —
-  the `--fit` probe, not a fault (fit cannot build the assistant context
-  during measurement). With `CTX_SIZE` set explicitly, as deployed, this
-  line does not appear at all. Do not restart over it either way.
- **Draft looks unengaged**: ngram-mod is silent at load; read the
-  `draft acceptance = …` line per task from
-  `journalctl -u frank-small` (the server must not be restarted just to
-  see this).
- **Quantized KV regression**: if someone rebuilds `build-cuda` without
-  `GGML_CUDA_FA_ALL_QUANTS=ON`, prefill collapses ~12x with q8 KV (see
-  the CUDA lesson in AGENTS.md). Check
-  `grep FA_ALL_QUANTS build-cuda/CMakeCache.txt` after any rebuild.
- Logs: `journalctl -u frank-small` (root unit, not quadlet).
+At 254k-token prefill / 253k-ctx decode, 4 slots, q8_0 KV, ubatch 1024:
+pp ~965 t/s, tg 257–302 t/s, idle 23,300 MiB of 24,576. Larger ubatches
+(2048) OOM the MTP draft context's pp compute buffer and must not be tried.

 ## Rollback

-Redeploying the prior model means repointing `MODEL_DIR`/`MODEL_FILE`/
-`DRAFT_FILE` back to the Qwen trio and `SPEC_TYPE=draft-mtp` — see
-[[frank-small-qwen38-gsq-rco]] for the exact deployed values. The Qwen
-weights were NOT deleted from the host.
+Payload and runtime for [[frank-small-qwen38-gsq-rco]] (ISTA-DASLab
+Qwen3.8-27B-GSQ-RCO IQ3_S) remain intact on disk. [[frank-small-vllm-gemma4]]
+is the same-day vLLM experiment (also a rollback target).

-## References
+# Citations

- `scripts/serve-frank-small.sh` — the actual exec line
- [[frank-small-qwen38-gsq-rco]] — RETIRED predecessor (rollback)
- [[halogen-flash-server]] — the sibling endpoint on the iGPU
- [[x4-pair-experiment]] — why the 3090 serves its own model, not the trunk
+- Model + mmproj + MTP sidecar: https://huggingface.co/HauhauCS
+- llama.cpp: https://github.com/ggml-org/llama.cpp
+- Serve script: `scripts/serve-frank-small.sh`; tunables: `scripts/frank-small.env`
@@ -1,7 +1,7 @@
 ---
 type: reference
-title: frank-small on Dankclaw — llama.cpp, Qwen3.8-27B GSQ-RCO IQ3_S
-description: The RETIRED frank-small runbook (2026-09-16 → 09-17, rollback target) — mainline llama.cpp on Dankclaw's RTX 3090 serving ISTA-DASLab Qwen3.8-27B-GSQ-RCO at IQ3_S with the model's own in-file MTP head and the BF16 vision projector: identity, KV arithmetic, deployed flags, the MTP contract, the client contract, and what to check when it misbehaves.
+title: "frank-small on Dankclaw — llama.cpp, Qwen3.8-27B GSQ-RCO IQ3_S"
+description: "The RETIRED frank-small runbook (2026-09-16 → 09-17, rollback target) — mainline llama.cpp on Dankclaw's RTX 3090 serving ISTA-DASLab Qwen3.8-27B-GSQ-RCO at IQ3_S with the model's own in-file MTP head and the BF16 vision projector: identity, KV arithmetic, deployed flags, the MTP contract, the client contract, and what to check when it misbehaves."
 tags:
  - llm
  - frank-small
@@ -13,193 +13,3 @@ tags:
  - vision
 timestamp: 2026-09-16
 ---
-
-# frank-small — llama.cpp on the RTX 3090
-
-> **RETIRED 2026-09-17** — replaced by [[frank-small-gemma4-12b]]. Kept as
-> the rollback target and the record of the deployment it succeeded.
-> Originally ACTIVE from 2026-09-16, created when Dankclaw's inference was
-> re-split: [[halogen-flash-server]] took the gfx1151 iGPU back for
-> `frank-large`, which freed the RTX 3090 for this endpoint.
-
-## Identity
-
-| what | value |
-|---|---|
-| runtime | mainline llama.cpp master `37b53fd` (2026-09-16), CUDA build for sm_86 |
-| binary | `/home/dank/GitPackages/llama.cpp-mainline/build-cuda/bin/llama-server` |
-| unit | **root systemd unit** `frank-small.service` (repo `scripts/frank-small.service`, `relink_unit`ed into `/etc/systemd/system/`) |
-| flags | `scripts/frank-small.env`; entrypoint `scripts/serve-frank-small.sh` |
-| endpoint | `192.168.0.143:7771`, alias `frank-small` |
-| model | `ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF`, the `IQ3_S-mtp` build + BF16 mmproj |
-
-It is a root unit rather than a Quadlet because the runtime is a native CUDA
-binary, not a container: there is no image to install and no generated unit to
-avoid. Boot persistence is the unit's `[Install] WantedBy=multi-user.target`;
-`Restart=on-failure` / `RestartSec=10` make it self-healing; `LimitMEMLOCK=infinity`
-covers CUDA's pinned staging buffers.
-
-Build recipe (the deployed binary):
-
-```bash
-cmake -B build-cuda -DCMAKE_BUILD_TYPE=Release -DGGML_NATIVE=ON \
-  -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=86
-cmake --build build-cuda --target llama-server -j
-```
-
-`GGML_CUDA_FA_ALL_QUANTS` is deliberately **off**: it is only needed for
-*quantized* KV with flash attention (see the CUDA lesson in AGENTS.md — without
-it, quantized-KV prefill collapses ~12x), and this service runs f16 KV.
-
-## The model, and why the `-mtp` build specifically
-
-GSQ-RCO is a **per-tensor mixed-precision** allocation produced by a
-gradient-based search under a total size budget, rather than one quantization
-type applied throughout. IQ3_S at 3.50 bpw is the release's task-lossless
-operating point: AIME25 100.00 and LiveCodeBench v6 85.71 match BF16 exactly,
-GPQA-Diamond is 0.51 behind — at 11.8 GB against 53.8 GB.
-
-Two files are required, and the second suffix matters:
-
-| file | size | why |
-|---|---|---|
-| `Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp.gguf` | 12,120,016,960 B | the `-mtp` build **is** the MTP build: it carries the head as `blk.64.nextn.*` (`qwen35.nextn_predict_layers = 1`) |
-| `mmproj-Qwen3.8-27B-BF16.gguf` | 931,146,528 B | the vision encoder + projector (`clip.projector_type = qwen3vl_merger`, 27 blocks) |
-
-Both live in `/home/dank/models/Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp/`, fetched with
-`huggingface_hub` (which verifies the LFS sha256 and resumes).
-
-**MTP is a property of the file, not of the flags.** `--spec-type draft-mtp`
-sets `load_mtp`, and the draft context runs on the *main model's* weights
-(`common/common.cpp` passes `shares_model` when no `--model-draft` is given;
-`common/speculative.cpp` gives it `LLAMA_CONTEXT_TYPE_MTP`). So there is no
-second copy of a 12 GiB model in VRAM, and `--model-draft` would be wrong here —
-it would load the whole file again.
-
-## Architecture and the KV arithmetic
-
-GGUF arch `qwen35`: 64 layers laid out as `16 x (3 x (Gated DeltaNet -> FFN) +
-1 x (Gated Attention -> FFN))`. Only the 16 full-attention layers take KV, so a
-token costs
-
-```
-2 (K,V) x 4 heads x 256 head_dim x 2 B = 64 KiB at f16   (34 KiB q8_0, 18 KiB q4_0)
-```
-
-and the 48 DeltaNet layers add ~0.16 GiB per sequence of constant-size state in
-f32, at any depth. The MTP draft context is `n_ctx` long over one nextn layer,
-so it costs 4 KiB/token of its own.
-
-## Sizing (RTX 3090, 24576 MiB)
-
-| term | GiB |
-|---|---|
-| weights | 11.29 |
-| mmproj | 0.87 |
-| MTP draft KV (131,072 x 4 KiB) | 0.50 |
-| DeltaNet state (one sequence) | 0.16 |
-| CUDA context + prefill/decode buffers | ~1.5 |
-| **KV, one slot at 131,072 x 64 KiB** | **8.00** |
-| **modelled total** | **~22.3 of 24** |
-
-The binding number is the KV: about 9.5 GiB is left after the fixed terms, which
-is ~155k tokens at 64 KiB/token. Alternatives inside the same budget: `2 x
-65536` at f16, or `4 x 65536` / `1 x 262144` at q8_0 — the quantized variants
-need a rebuild with `-DGGML_CUDA_FA_ALL_QUANTS=ON`. **`--parallel N` divides
-`--ctx-size` between the slots**; it does not multiply.
-
-Measured at deploy: **22,362 MiB** in use of 24,576, `n_slots = 1`,
-`n_ctx_slot = 131072`.
-
-## Deployed flags
-
-| flag | value | why |
-|---|---|---|
-| `CTX_SIZE` / `PARALLEL` | `131072` / `1` | one conversation per slot, full f16 precision, no KV quantization risk |
-| `CACHE_TYPE_K/V` | `f16` | quantizing would need the FA_ALL_QUANTS build for no benefit here |
-| `NGL` | `999` | every layer on the card |
-| `FLASH_ATTN` | `on` | |
-| `SPEC_TYPE` | `draft-mtp` | the model's own head; draft depth stays llama.cpp's default of 3 |
-| `BATCH_SIZE` / `UBATCH_SIZE` | `2048` / `512` | the defaults; ubatch 512 keeps the DeltaNet chunk buffers small |
-| `NUM_THREADS` / `THREAD_BATCH` | `8` / `8` | the work is on the GPU; the host CPU belongs to halogen's engine |
-| `TEMPERATURE` | `0.0` | greedy default (fleet contract) — clients that send their own values get them |
-| `REASONING_FORMAT` / `REASONING_BUDGET` | `deepseek` / `-1` | thinking into `reasoning_content`, unlimited budget (the model decides) |
-| `--jinja` | pinned | the GGUF's own template supplies `reasoning_effort`, `preserve_thinking` and the tool-call grammar. This build defaults it on; pinning it means a future default change cannot silently take tools away |
-| `--metrics` | on | Prometheus scrapes `/metrics` here (`frank-small` job) |
-
-## Client contract
-
- **One slot.** A second concurrent request queues; it is not served in
-  parallel, and agent fan-out through this endpoint serializes. Widen with
-  `PARALLEL` (which divides the context) if that becomes the wrong trade.
- **Thinking counts against the budget.** A request whose budget runs out
-  mid-thought returns `finish_reason: "length"` with an empty `content` and the
-  partial reasoning in `reasoning_content` — observed live on a vision request
-  at `max_tokens: 64`. Give reasoning models a real budget.
- **Tools work** (`tool_calls` returned; verified with a function-call round
-  trip) because `--jinja` serves the model's own template. This is also what
-  fixed subagents fleet-wide: the ik runtime that served `frank-large` until
-  2026-09-16 rejected every request carrying `tools` with
-  `500 tools param requires --jinja flag`.
- **Images are accepted** as OpenAI `image_url` content parts with a `data:`
-  URL or bare base64. This is the fleet's only image path: halogen's
-  `frank-large` runs with no tower and refuses images with a named reason.
- **`reasoning_effort` accepts only `xhigh`, `medium`, `low`.** The template
-  raises on anything else — a 500, not a clamp — so the harness folds its own
-  levels onto that set in `compat.reasoningEffortMap` (`high: xhigh`,
-  `max: xhigh`). An unmapped level fails on the first request.
- `/v1/models` reports `frank-small`; `/health` answers 200 once the model is
-  loaded and the projector is up.
-
-## Verified at deploy (2026-09-16)
-
- `systemctl is-enabled/is-active frank-small.service` → `enabled` / `active`.
- `/v1/models` → `frank-small`; a chat completion returns `content` **and**
-  `reasoning_content`.
- A tool call returns `tool_calls` with the expected arguments.
- Slot timing lines show `draft acceptance = 0.66-0.94, mean len = 2.95-3.83`
-  — MTP is engaged, not merely configured.
- An image request returns the right colour answer from the projector.
- Harness routing: role `smol` lands here, role `slow` on `frank-large`, and a
-  real subagent (the `task` role) completes a tool-using turn.
-
-## Operate
-
-```bash
-systemctl status frank-small.service          # root unit
-journalctl -u frank-small.service -n 60       # load + per-request timing
-curl -s http://127.0.0.1:7771/health
-curl -s http://127.0.0.1:7771/v1/models
-curl -s http://127.0.0.1:7771/metrics | grep -E 'spec_decode|kv_cache_usage'
-nvidia-smi --query-compute-apps=pid,process_name,used_memory --format=csv
-```
-
-`/metrics` carries `llamacpp:spec_decode_num_{drafts,draft_tokens,accepted_tokens}_total`,
-which is the cheapest way to confirm MTP is running, and
-`llamacpp:kv_cache_usage_ratio` for how full the slot is.
-
-## Troubleshooting
-
-| symptom | cause |
-|---|---|
-| unit restarts immediately, `frank-small: missing <path>` in the journal | the model, the projector or the binary is absent — the serve script refuses to start rather than half-load; finish the download |
-| `500 … Jinja Exception: Unexpected reasoning effort <level>` | a client sent an effort the template does not accept; map it (see the client contract) |
-| `500 tools param requires --jinja flag` | impossible here (`--jinja` is pinned) — you are looking at a *different* endpoint, most likely the retired ik runtime's logs |
-| `finish_reason: "length"` with empty content | the budget ran out inside the thinking block; raise `max_tokens` |
-| a request that waits, then answers normally | the single slot was busy (or the KV is nearly full); raise `PARALLEL` or expect serialization |
-| CUDA OOM at load | another process holds the 3090, or the KV was widened past ~9.5 GiB; check `nvidia-smi` and re-derive from the sizing table |
-
-## Rollback
-
-`frank-small` is a new endpoint, so rollback is retirement:
-`systemctl disable --now frank-small.service`, move
-`/etc/systemd/system/frank-small.service` into `~/.AGENTTRASH/`, and remove the
-harness provider + the `smol`/`tiny`/`task`/`vision` role entries. The old
-`frank-small` (FreeToken on Dankserver, retired 2026-09-15) is a different
-service with a different card — see [[freetoken-frank-small]].
-
-## Citations
-
- [ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF](https://huggingface.co/ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF) — the quantized checkpoints, the `-mtp` builds and the projector
- [Qwen/Qwen3.8-27B](https://huggingface.co/Qwen/Qwen3.8-27B) — base model: 64 layers, Gated DeltaNet/Gated Attention layout, 262,144 native context
- [ggml-org/llama.cpp](https://github.com/ggml-org/llama.cpp) — `--spec-type draft-mtp`, the MTP context type, and the Qwen3-VL projector path
@@ -1,7 +1,7 @@
 ---
 type: reference
 title: frank-small on Dankclaw — vLLM, Gemma4-12B QAT w4a16 (compressed-tensors)
-description: The active frank-small runbook — vLLM on Dankclaw's RTX 3090 serving google/gemma-4-12B-it-qat-w4a16-ct (Marlin int4 weights, bf16 KV, 262k window) with Google's bf16 MTP assistant, the audio-capable derived image, the no-nvidia-container-toolkit device mounts, measured throughput, and the drafter-Marlin bug that killed the w4a16 assistant.
+description: "The RETIRED frank-small runbook (2026-09-19, same-day revert; rollback target) — vLLM on Dankclaw's RTX 3090 serving google/gemma-4-12B-it-qat-w4a16-ct (Marlin int4 weights, bf16 KV, 262k window) with Google's bf16 MTP assistant, the audio-capable derived image, the no-nvidia-container-toolkit device mounts, measured throughput, and the drafter-Marlin bug that killed the w4a16 assistant. Replaced by [[frank-small-gemma4-12b]] on the same day."
 tags:
  - llm
  - frank-small
@@ -17,11 +17,12 @@ timestamp: 2026-09-19

 # frank-small — vLLM on the RTX 3090 (Gemma4-12B QAT w4a16)

-> **ACTIVE since 2026-09-19.** Replaces [[frank-small-gemma4-12b]]
-> (llama.cpp, now the rollback target). Same port (7771), same alias
-> (`frank-small`), same client contract surface — the runtime, the model
-> payload, the drafter and the service shape (root systemd unit → rootless
-> Quadlet) all changed.
+> **RETIRED 2026-09-19 (same-day revert).** This vLLM deployment was cut
+> back to [[frank-small-gemma4-12b]] (llama.cpp, now the ACTIVE runbook)
+> the same day it landed — this page is the rollback target. Same port
+> (7771), same alias (`frank-small`), same client contract surface — the
+> runtime, the model payload, the drafter and the service shape (root
+> systemd unit → rootless Quadlet) all changed.

 ## Identity

@@ -1,7 +1,7 @@
 ---
 type: reference
-title: halogen-flash-server on Dankclaw (frank-large)
-description: The RETIRED frank-large runbook (2026-09-16 → 09-19, rollback target) — peonist-ai/halogen-flash-server 0.5.8 serving Qwen3.8-Flash-Next (w4b) on the Dankclaw gfx1151 box: container shape, deployed flags, the memory model, the client contract, and the failure modes worth knowing. Replaced by [[frank-large-ciru-orca]] on 2026-09-19.
+title: "halogen-flash-server on Dankclaw (frank-large)"
+description: "The RETIRED frank-large runbook (2026-09-16 → 09-19, rollback target) — peonist-ai/halogen-flash-server 0.5.8 serving Qwen3.8-Flash-Next (w4b) on the Dankclaw gfx1151 box: container shape, deployed flags, the memory model, the client contract, and the failure modes worth knowing. Replaced by [[frank-large-ciru-orca]] on 2026-09-19."
 tags:
  - llm
  - halogen
@@ -13,467 +13,3 @@ tags:
  - rocm
 timestamp: 2026-09-13
 ---
-
-# halogen-flash-server on Dankclaw (frank-large)
-
-> **RETIRED 2026-09-19 — replaced by [[frank-large-ciru-orca]]** (the CIRU v4.4.1
-> runtime serving jcbtc Qwen3.8-Flash-CIRU-STRIX-Orca). This runbook is the
-> rollback target. It was ACTIVE again since 2026-09-16: it replaced the
-> llama.cpp / kingjones30-ROCmFPX STRIX deployment on 2026-09-11 (see
-> [[kingjones-strix-rocmfp4]]), was displaced for two days by ik_llama.cpp (see
-> [[frank-large-ik-uncensored]]), and came back as the shape the fleet asked for:
-> two 200k-token conversations rather than the native-context pair. It was
-> deployed as a rootless Quadlet container,
-> `dankclaw/frank-large/quadlet/frank-large.container`, port 7770.
-
-The engine is a from-scratch serving runtime for the Qwen3.8-Flash-Next
-architecture on AMD's gfx1151 (Strix Halo) — the same silicon this host runs.
-Upstream's own comparison table measures it at **roughly 4x the prefill
-throughput** of the llama.cpp-family runtimes it ships against (EngramHalo.cpp,
-ROCmFP4, CIRU-IU4) on a 32K prompt, and prefill is most of the wall clock on any
-prompt with real context. That, not decode, is the reason for the swap.
-
-## Shape
-
-One container runs both halves of the stack:
-
-| half | port | reachable |
-|---|---|---|
-| engine | 8730 | container loopback only — **unauthenticated, never publish it** |
-| OpenAI front-end | 8731 | published as host **7770** |
-
-The `all` mode is upstream's default and the shape a single-container host
-wants: the front-end talks to the engine over loopback inside the container, so
-there is one process tree, one log, and one published port.
-
-`HALOGEN_MODEL_ID=frank-large` sets the id reported at `/v1/models`, in
-`/health`, and in every response body. Requests are **never rejected for naming
-a different model** — the server serves whatever asks. Keeping the fleet alias
-is therefore free, and it means Hermes, the harness config, the OpenWebUI
-discovery, the `services.json` probe and the docs needed no rename.
-
-## Weights
-
-`peonist-ai/halogen-qwen3.8-flash-next` (HuggingFace, public and ungated):
-
-| file | size | role |
-|---|---|---|
-| `qwen38-flash-next-w4b.hgn` | 115.55 GiB | the checkpoint |
-| `qwen38-flash-next-w4b.overlay.hgn` | 2.40 GiB | **quality sidecar** — picked up automatically; the engine warns at startup if it is missing. **Refreshed 2026-09-16**: the sidecars are versioned *in place*, so a stale local copy is invisible from its file name — this one had read 2.31 GiB since it was first fetched while the Hub had moved on |
-| `qwen38-flash-next-vision.hgn` | 0.84 GiB | vision tower (see below) |
-| `tokenizer/` | ~21 MB | flat tokenizer dir; found beside the checkpoint, so no extra mount |
-
-`HALOGEN_DOWNLOAD` fetches the whole repo into the bind mount on **first start
-only** — the entrypoint acts only when the checkpoint is absent, so a restart
-never re-downloads, and `huggingface_hub` resumes an interrupted pull. The
-mount must be read-write for this. Roughly 127 GB; on Dankclaw it lands in
-`/home/dank/models/halogen`.
-
-**Refreshing the weights.** Delete the mount's contents (or the whole
-directory, then recreate it **empty**) and start the unit: the entrypoint sees
-the checkpoint absent and re-fetches the repo, resuming if interrupted. Two
-traps. First, the bind mount's source directory must *exist* — podman refuses
-the container with `Error: statfs /home/dank/models/halogen: no such file or
-directory` and the unit loops on `Restart=on-failure`; `mkdir -p` the empty
-directory and the fetch starts on the next start. Second, "refresh" only means
-"fetch what the Hub has now": on 2026-09-16 the re-fetch brought newer sidecars
-(`overlay.hgn` 2,477,677,120 → **2,572,466,560** bytes, `overlay-speed.hgn`
-2,383,306,048 → **2,478,095,488**) while the checkpoint returned
-byte-identical in size (124,068,083,904) — so an update can live entirely in
-the sidecars, and equal file sizes are not proof of an unchanged repo.
-
-`overlay-speed.hgn` (2.31 GiB on the Hub; 2.22 GiB before the 2026-09-16
-refresh) comes along with the repo but is unused: it trades the `o_proj`
-calibration for about 2 % of decode, and the quality sidecar is the default for
-a reason.
-
-## Deployed flags
-
-Everything else is configured by environment variable — there is no config
-file, and the flags are read once at startup. These are the departures from
-upstream's defaults:
-
-| flag | value | why |
-|---|---|---|
-| `HALOGEN_MODEL_ID` | `frank-large` | fleet identity; no client rename |
-| `HALOGEN_CTX` | `200192` | **set 2026-09-16**: exactly half the pool, so two conversations of the full window are always resident. Not a round 200,000 — pool/2 is. The model's native window is 262,144, so this is a real reduction in the per-conversation ceiling, and a client asking for more gets a 400 (never a silent truncation). The harness declares 200192 for `frank-large` |
-| `HALOGEN_KV_POOL_POSITIONS` | `400384` | **retuned 2026-09-16** (from 401,408): 2 x 200,192. Reserves ~11.0 GiB of KV, confirmed by the boot ledger as `400384 positions, ~27.2 GiB (16.2 GiB of it for everything but the pool, at max_tok 24576)` against a 33.5 GiB device budget — **no clamp**. Slots still oversubscribe the pool for any real request (admission reserves prompt + `max_tokens`), so a near-full pair that is both generating is pool-bound. `HALOGEN_KV_POOL_FIT` guards the boot: read the ledger, not this column |
-| `HALOGEN_KV_SLOTS` | `2` | **lowered from 3 on 2026-09-16** — one slot per half of the pool, and two is the fleet's shape now that Dankclaw serves two endpoints. Each slot is 111.0 MiB of O(1) state, so this hands ~111 MiB back as well |
-| `HALOGEN_MAX_TOK` | `24576` | **halved from the 32768 default on 2026-09-16** for frank-small, its co-tenant on the same host. The arena's variable term goes `8.40 x 2 → 8.40 x 1.5`, handing ~4.2 GiB back at roughly a 4-5 % prefill cost — the same lever at 16,384 costs 8-9 %. The vendor's own notes call 24,576 the largest arena that allocated with headroom to spare, so it stays inside their tested range. 65536 still cannot boot. It sizes a per-call PREFILL CHUNK, not the prompt ceiling: a 200k conversation is prefilled in arena-sized pieces |
-| `HALOGEN_HOST_RESERVE_GIB` | `20` | **raised from 15 with the same retune**, because the arena shrank: `budget = 121.2 − 67.7 − 20 = 33.5 GiB` against a measured need of ~27.2, so the fit check kept ~6 GiB of margin and did not clamp. It is **not** an allocation: raising it cannot return memory to the host, only take the pool away (see the ceiling section below) |
-| `HALOGEN_VISION_TOWER` | *unset* | **removed 2026-09-16.** The tower charges 1.97 GiB against the same device budget as the pool and the arena, and the fleet's image path moved to [[frank-small-qwen38-gsq-rco]] (llama.cpp with `--mmproj`). This server refuses image content parts, and `models.yml` declares `frank-large` text-only to match |
-| `HALOGEN_DOWNLOAD` | `peonist-ai/halogen-qwen3.8-flash-next` | first-start fetch |
-| `HALOGEN_VERBOSE` | `1` | verbose engine logging — the fullest startup account, on permanently so an incident report needs no restart-into-debug |
-
-Everything else is upstream's default, deliberately: prompt cache mode `2`
-(two entries per conversation — end-of-system-prompt and end-of-history — which
-already covers the shared-prefix case), `HALOGEN_CACHE_INPLACE=1` (the snapshot
-stays ~115 MiB at any depth), the speculative drafter `1` (byte-identical at
-temperature 0, and it stands down whenever a second conversation is
-generating), the quality sidecar, `HALOGEN_ADMIT_CHUNK=0` (a non-zero value
-gives up the byte-identity property), and the default `HALOGEN_MAX_TOKENS_CAP=65536`.
-
-## Memory model
-
-Three numbers explain the whole box:
-
- **~68 GiB pinned weights.** The checkpoint is mapped and registered with the
-  GPU in place, never copied. The kernel counts the locked pages as reclaimable
-  file cache, so `free`, `MemAvailable` and every dashboard built on them
-  **overstate free memory by the size of the model**. The server prints the
-  corrected figure at startup; believe that line, not Grafana.
- **~28 KiB per KV position** (measured: 262,144 positions = 6,656 MiB of KV,
-  ~7.2 GiB with block scratch), plus the prefill arena. The 2026-09-16 ledger
-  reads the two together in the engine's own words: `400384 positions, ~27.2
-  GiB (16.2 GiB of it for everything but the pool, at max_tok 24576)` — so
-  ~11.0 GiB of pool and 16.2 GiB of arena, working slack and (no longer) a
-  vision tower. The engine's device-side budget stops at roughly 47 GiB on a
-  128 GB machine *with the weights pinned* — and lower on a machine carrying
-  more besides this server, which Dankclaw is.
- **`HALOGEN_HOST_RESERVE_GIB`** is what is left for the page cache the lookup
-  table is read through. Too little and long prompts crawl with the disk busy
-  and the process in uninterruptible sleep — that is a cache problem, not a
-  memory problem, and the fix is a *smaller* pool, not a bigger one.
-
-`HALOGEN_KV_POOL_FIT=1` (default) measures that budget at startup and lowers
-the pool itself if the configured one will not fit, printing what it chose.
-Read those lines on every restart; they are the ground truth for sizing.
-
-**Measured ledger, seven restarts, read live 2026-09-12.** The engine's own
-startup account, including three deliberate experiments:
-
-| pool | max_tok | reserve | KV | arena | need | budget | verdict |
-|---|---|---|---|---|---|---|---|
-| 262144 (baseline) | 32768 | 24 | 7.2 | 20.6 | 29.8 | 29.5 | boots 0.3 over — prints **no** sizing lines |
-| 524288 | 16384 | 24 | 14.4 | 12.2 | 28.6 | 29.5 | fits, 0.9 spare |
-| 524288 | 32768 | 24 | 14.4 | 20.6 | 37.0 | 29.5 | clamped to 262144, deploy still green |
-| 524288 | 65536 | 24 | — | 37.3 | 46.5 | 29.5 | **OOM — will not boot** |
-| 524288 | 32768 | **15** | 14.4 | 20.6 | 37.0 | **38.5** | **deployed — both features** |
-| 524288 | 32768 | 16 | 14.4 | 20.6 | 37.0 | 37.5 | fits, footprint unchanged — see the ceiling below |
-| **400384** | **24576** | **20** | **11.0** | **16.4** | **27.2** | **33.5** | **deployed 2026-09-16 — ledger need 27.2, no clamp** |
-| 401408 | 32768 | 15 | 11.0 | 20.6 | 33.4 | 38.5 | 2026-09-13 → 09-16, superseded by the retune |
-
-### The sizing rule
-
-One budget covers the pool, the arena **and** the vision tower together:
-
-```
-device budget = host RAM - resident weights - HALOGEN_HOST_RESERVE_GIB
-              = 121.2    - 67.7             - reserve
-                                    reserve 24 -> 29.5 GiB
-                                    reserve 15 -> 38.5 GiB
-
-need = KV (28 KiB/position) + arena + vision (1.97 measured)
-arena = 3.80 GiB + 8.40 GiB x (HALOGEN_MAX_TOK / 16384)
-```
-
-The `121` the engine prints is rounded **down** from `MemTotal 127104060 kB =
-121.2159 GiB`, so subtracting the displayed figures gives 29.3 rather than 29.5.
-Use the unrounded value. Two of the three terms are fixed hardware and
-checkpoint facts; `HALOGEN_HOST_RESERVE_GIB` is the only one you own — which
-buys the *fit margin*, not the host's memory. It is a term in this inequality,
-not an allocation, and the section at the end of this page shows the live test
-that proves it.
-
-Trust the engine's printed requirement over this model wherever they disagree:
-a boot clamped at reserve 24 stated `524288 positions need ~35.0 GiB (plus 1.5
-of margin)`, i.e. **budget >= 36.5**, against a modelled 37.0 here. Close
-enough to plan with, and the authoritative number to cut the other way.
-
-The arena term is **linear in `HALOGEN_MAX_TOK`** and predicts every measured
-point to 0.1 GiB — including upstream's three published "on the device" figures
-(524,288 → ~35, 786,432 → ~42, 1,048,576 at `max_tok=16384` → ~41), which are
-**pool + arena totals at the default `max_tok`**, not pool alone. That is why
-their largest pool "needs `HALOGEN_MAX_TOK=16384`": at 32768 a 1M pool would be
-49.4. Reading those figures as pool-only is the trap that makes a doubling look
-affordable when it is not.
-
-At the baseline reserve of 24 the doubled pool leaves `29.5 - 1.97 - 14.4 =
-13.1` for the arena, so `max_tok <= 18,130` and **16384 is that reserve's
-ceiling, not a conservative setting.** The corner check agrees: at 32768 only
-7.0 GiB is left for the KV, ~254k positions, and the pool drops back toward
-262144. Cut the reserve to 15 and the same arithmetic gives `38.5 - 1.97 - 14.4
-= 22.1` for the arena — `max_tok <= 35,700`, which is exactly why 32768 now
-fits unmodified.
-
-### What the reserve actually costs
-
-The device budget is not the whole story. Memory position by configuration
-(401,408 is the 2026-09-13 deployed row, read from its boot ledger):
-
-| config | held | host left | contiguous 2 MiB blocks |
-|---|---|---|---|
-| 262144 / 32768, reserve 24 | 96.3 | 12.1 GiB | 2081 |
-| 524288 / 16384, reserve 24 | 94.6 | 13.8 GiB | 3742 |
-| 524288 / 32768, reserve **16** | **103.5** | **7.1 GiB** | 293 |
-| **401408 / 32768, reserve 15** | **100.0** | **9.1 GiB** | **1396** |
-| **400384 / 24576, reserve 20** | **95.4** | **14.7 GiB** | **475** |
-
-At the doubled setting (2026-09-12) the server held 103.5 of 121 GiB and the
-engine volunteered the stall note. It did not print it at the 401,408 trim, and
-it prints it again after the 2026-09-16 retune. Three data points, and they do
-not line up with a "free memory below X" reading: 310 blocks printed, 1396 did
-not, 475 did. So treat the note as "the machine is tight" and read the
-`host memory left` line rather than waiting to be told:
-
-```
-halogen: NOTE this server now holds most of the machine, and 310 contiguous
-2 MiB blocks remain. It runs best on a host of its own. If other large
-processes share this machine they will compete for what is left, and when it
-runs out BOTH they and this server can stall for minutes at a time at full
-CPU with no disk activity and no output, which is not a crash and needs no
-restart.
-```
-
-That is the failure mode to watch, and it is the least diagnosable of the
-three: not a silent clamp, not a boot refusal, but a minutes-long freeze that
-looks like a hang, for the neighbours as much as for frank-large. `free` is
-useless for spotting it — `MemAvailable` still reads ~74 GiB because the
-page cache counts as reclaimable. Watch the `host memory left for everything
-else` line and the contiguous-block count at boot instead.
-
-28 co-tenant containers (~4.3 GiB working set) stayed up through the change,
-so it is survivable.
-
-### The reserve cannot buy air — ceiling 16
-
-The last row above is the experiment that proves it. Raising the reserve from
-15 to 16 left `held` and `host left` **byte-for-byte identical** (103.5 / 7.1
-GiB). `HALOGEN_HOST_RESERVE_GIB` feeds the fit-check budget and is *not* an
-allocation; the actual footprint is `weights + KV + arena + vision`, none of
-which the reserve touches. So the knob has exactly two regimes:
-
-| reserve | budget | vs required 36.5 | effect |
-|---|---|---|---|
-| 15 | 38.5 | +2.0 | deployed |
-| 16 | 37.5 | +1.0 | no change to footprint, thinner margin — pointless |
-| 17 | 36.5 | +0.02 | knife edge, inside the noise of the 67.7 term |
-| **18** | **35.5** | **−1.0** | **clamps the pool to 262144** |
-| 24 | 29.5 | −7.0 | clamps the pool to 262144 |
-
-The 36.5 requirement is the engine's own, read from a clamped boot:
-`524288 positions need ~35.0 GiB (plus 1.5 of margin)`. Above 16 the only
-thing a bigger reserve can do is take the doubling away.
-
-**To actually free host memory, change the pool or the arena**, and the reserve
-then becomes free to rise: `HALOGEN_KV_POOL_POSITIONS=262144` hands back
-7.2 GiB from the doubled pool, `HALOGEN_MAX_TOK=16384` hands back 8.4. Both
-are one-line edits that the fit check will confirm on the next boot. The
-2026-09-13 trim to 401,408 was this remedy applied at half strength —
-`HALOGEN_MAX_TOK` untouched, and on the host-left metric it bought +2.0 GiB
-with a 4.5x improvement in contiguous blocks (310 → 1396), less than the
-modelled 3.4 because the engine holds the freed KV as working slack.
-
-### The `WARNING` line only appears when the fit check had to move
-
-This one cost me a wrong conclusion, so it is worth stating plainly. When the
-configured pool already fits, the engine reserves it **and prints nothing** —
-the 09-11 baseline boot goes straight from the pre-flight line to
-`checkpoint: mapped` with no `kv pool:` sizing block at all. The sizing block and
-its `WARNING even 262144 positions ... host RAM cannot spare it` appear **only
-after a clamp**.
-
-So the absence of a warning in old logs is not evidence of headroom. Both
-`262144/32768` rows above need 29.8 against a 29.5 budget — 0.27 over — and the
-baseline has lived in that state since it was deployed without incident,
-because the 24 GiB reserve is itself generous (the host reports 12–14 GiB free
-at that point). The engine's budget is a conservative floor, not a hard wall.
-
-### Two failure modes, and only one is dangerous
-
-Raising `max_tok` past the ceiling does not shrink the pool — the arena alone
-outgrows the budget, and the server cannot allocate at all:
-
-```
-kv pool: 524288 positions need ~51.7 GiB ... LOWERING THE POOL TO 262144 (~44.5 GiB).
-kv pool: WARNING even 262144 positions (the context, the smallest pool that can
-         serve one request) needs ~44.5 GiB and host RAM cannot spare it.
-dmalloc: FAILED requesting 0.250 GiB after 39.970 GiB in 566 allocations (out of memory)
-```
-
-This is the quiet one: the unit stays `active`, the container reports
-`Up (starting)`, and `deploy.sh` sits in its health wait for the whole
-`LLM_WAIT` rather than failing fast. It confirms upstream's warning that a
-large `HALOGEN_MAX_TOK` means "the server will not start", not "it will be
-slower".
-
-Under-running the ceiling is far more benign and far easier to miss. At 524288
-with the arena back at 32768, `HALOGEN_KV_POOL_FIT` lowered the pool and
-`deploy.sh` still reported healthy in 12 seconds:
-
-```
-kv pool: 524288 positions need ~35.0 GiB (plus 1.5 of margin) and host
-         RAM cannot spare it. LOWERING THE POOL TO 262144 (~27.8 GiB).
-```
-
-A green health check does not mean the pool you asked for is the pool you got —
-read the `kv pool:` lines after every restart.
-
-The 47.7 GiB n-gram lookup table is never held in RAM; it is read through
-the file cache. The 2.0 GiB the firmware carves for the iGPU appears in no
-`/proc/meminfo` figure at all.
-
-**Co-tenancy.** `HALOGEN_FLASH_PIN_TRUNK=1` (default) is what the published
-decode numbers were measured with, and it means this container holds most of
-the host. If a co-tenant then takes the remainder, both it and the engine can
-stall for minutes at a time at full CPU with no disk activity and no output.
-That is documented behaviour, not a crash, and it needs no restart.
-`HALOGEN_FLASH_PIN_TRUNK=0` gives the memory back and costs several times the
-decode speed — a last resort, not a tuning option.
-
-## Client contract
-
-The differences from llama.cpp that bite are all in the request, not the
-deployment:
-
- **No server-side sampler defaults.** `temperature` absent or `0` is
-  **greedy** decode; above 0 the request samples on the same drafter. Supported
-  fields: `temperature`, `top_p`, `top_k`, `min_p`, `seed`,
-  `presence_penalty`, `frequency_penalty`, `logit_bias`, `logprobs`. Refused
-  with a 400: `top_logprobs`, `logprobs` with `stream: true`, `n > 1`, and any
-  value outside its defined range (values are refused, never clamped).
-  The fleet runs greedy for clients that stay silent — deterministic, and
-  byte-identical with the drafter.
- **The token budget covers thinking.** The default is **8192**, and a budget
-  that runs out mid-thought returns `finish_reason: "length"` with an **empty
-  `content`** and the partial reasoning in `reasoning_content` — which most
-  clients do not display. Above the cap (`HALOGEN_MAX_TOKENS_CAP`, 65536) a
-  request is a 400 rather than a silent truncation. `max_tokens`,
-  `max_completion_tokens` and `max_output_tokens` all work and must agree if
-  more than one is sent.
- **A budget above (window − compaction-threshold) is a guaranteed 400 zone.**
-  The fit check (`prompt + max_tokens <= HALOGEN_CTX`) refuses, and the harness
-  only compacts at its own threshold — with OMP at 70 % of a 200,192 window
-  (140,134), any prompt in (136,192, 140,134] with a 64,000 budget overflows
-  the window while sitting below the compaction trigger. That collided live on
-  2026-09-17 (prompt 138,898 + 64,000 = 202,898 > 200,192). The fix is
-  harness-side: keep the declared `maxTokens` at or under
-  `window − threshold` (57,344 chosen) so no pre-compaction request can miss
-  the fit check. Do not fix it by raising `HALOGEN_CTX` past pool/2 — that
-  breaks the two-resident-conversations property.
- **`reasoning_effort`** is accepted at the top level and inside
-  `chat_template_kwargs`: `minimal`, `low`, `medium`, `high`, `xhigh`. The
-  model's default is `xhigh`. `minimal` aliases the template's `low` and still
-  reasons; `enable_thinking: false` turns thinking off; `preserve_thinking` is
-  honoured. `preserve_reasoning` is a **400** — it is the llama.cpp-era key.
- `/v1/models`, `/health` and completions all report the served id, and
-  `reasoning_content` carries the reasoning trace.
-
-**Pool admission is the one arithmetic trap.** A request reserves its prompt
-**plus** `max_tokens` positions when it is admitted; if the pool cannot hold it
-the request waits in arrival order, and after the queue timeout (3600 s) it
-gets a `503 engine_busy`. A prompt large enough that prompt + budget exceeds
-the pool can therefore never be admitted at all. The 2026-09-16 shape is a
-400,384-position pool with `HALOGEN_CTX` 200,192 and two slots: two
-conversations at the ceiling fit the pool exactly, so a pair that is both
-near-full and both generating is pool-bound — the second waits for the first to
-release its region. The pool binds **how many** conversations stay resident,
-not how long one may be; the older 524,288 / 401,408 numbers live in the ledger
-tables above.
-
-## Vision
-
-**None here since 2026-09-16.** `HALOGEN_VISION_TOWER` is deliberately unset:
-the tower is a 1.97 GiB charge against the same device budget as the pool and
-the arena, and the fleet's image path moved to [[frank-small-qwen38-gsq-rco]],
-which loads the BF16 projector alongside its own weights. This server refuses
-image content parts and says why, rather than ignoring them:
-
-```json
-{"error": {"message": "this server does not accept images: the engine was
-started without a vision tower (HALOGEN_VISION_TOWER)", "type":
-"invalid_request_error"}}
-```
-
-The llama.cpp side handles images the same way this engine used to: a `data:`
-URL or bare base64, about 2,040 tokens of context per image, and `http(s)` URLs
-refused deliberately (fetching one would make the server issue outbound
-requests to wherever a client pointed it). Audio and video have no path
-anywhere in the fleet — that left with [[frank-npu]].
-
-## Deploy and operate
-
-```bash
-./scripts/deploy.sh dankclaw --service frank-large
-```
-
-The handler retires the ik-era **root** unit first (stop, disable, move
-`/etc/systemd/system/frank-large.service` into
-`~/.AGENTTRASH/20260916-halogen-restore/host-units` — never `rm`; this host's
-`disable` may delete the file outright, which the handler now reports), stops
-the Quadlet's own unit if it is running, then pulls
-`ghcr.io/peonist-ai/halogen-flash-server:0.5.8`, re-tags it to the
-`localhost/` alias the unit references, and **waits for port 7770 to be free**
-before installing and starting anything.
-
-That last gate is not paranoia. On the first cutover attempt the ik server
-logged "Deactivated successfully", and pasta — rootless podman's userspace
-network stack, which binds the published port at container CREATE — still
-reported `Address already in use` a second later, exiting the unit with code
-126. Only `Restart=on-failure` recovered it, minutes later. A deploy that
-starts a doomed container has not deployed anything, so `wait_port_free` polls
-every socket on the port (listener, accepted, TIME_WAIT) and fails loudly with
-the socket table if it never clears.
-
-Boot persistence comes from `[Install] WantedBy=default.target` in the unit
-file plus linger on the deploy user — generated units **cannot** be
-`systemctl --user enable`d on this host.
-
-```bash
-systemctl --user status frank-large.service     # the generated unit
-podman logs frank-large | head -40              # version banner + memory ledger
-curl -s http://127.0.0.1:7770/health            # 200 only after the engine PONGs
-curl -s http://127.0.0.1:7770/v1/models
-```
-
-`/health` asks the **engine** for a PONG before answering, so a 200 means
-"ready to serve", not merely "process alive". During the first-start download
-it stays unhealthy by design.
-
-## Troubleshooting
-
-| symptom | cause |
-|---|---|
-| `dmalloc: FAILED requesting … (out of memory)` then `HIP … out of memory` | the KV pool did not fit. `HALOGEN_KV_SLOTS` will not fix it (slots share the pool); lower `HALOGEN_KV_POOL_POSITIONS` to 262144, then `HALOGEN_MAX_TOK` to 16384 (~9 % prefill) |
-| starts, answers short prompts, collapses to a few tok/s on a long one, disk busy, process in uninterruptible sleep | short of **file cache**, not memory — lower the pool and/or raise `HALOGEN_HOST_RESERVE_GIB` |
-| long replies that look cut off | read `finish_reason`: `stop` means you have the whole answer, `length` means you ran out of budget (thinking counts) |
-| a request that never returns, then `503 engine_busy` | the pool could not reserve prompt + `max_tokens`; see the admission trap above |
-| both the engine and a co-tenant stall for minutes at full CPU | the pinned-trunk trade; see Memory model |
-| `free`, `MemAvailable`, Grafana all wrong by ~68 GiB | expected — locked weights are counted as reclaimable |
-
-`HALOGEN_VERBOSE=1` ships in the unit, so the fullest startup account is
-already in the log when something goes wrong. `HALOGEN_DMALLOC_LOG` is
-deliberately **not** set: it logs every allocation over 64 MB, which is
-steady-state noise on a shared host — raise it only for a live OOM report.
-The lines worth grepping first:
-
-```bash
-podman logs frank-large 2>&1 | grep -E '^(dmalloc|kv pool):'
-```
-
-Retrieval: `podman logs frank-large` covers the **current** instance only —
-the Quadlet runs `--replace --rm`, so everything before the last recreate is
-gone from the container log. `journalctl --user -u frank-large.service` is
-the durable record. Both are rootless: `sudo podman logs frank-large` looks
-in root's storage and answers "no such container".
-
-## Rollback
-
-The predecessor is intact: the ik payload stays at
-`/home/dank/models/mradermacher-Qwen3.8-Flash-Next-Uncensored-i1-Q4_K_M`, the
-ik build at `/home/dank/GitPackages/ik_llama.cpp/build`, and its unit, env,
-serve script and the Qwen-Sharp template are archived in the repo under
-`scripts/.AGENTTRASH/*.ik-llama.cpp` and `*.ik-llama-qwen-sharp`. Rolling back
-is a `git revert` of this change set plus a deploy — **but the Quadlet must be
-removed from `~/.config/containers/systemd/` first**, or the restarted root
-unit and the still-installed container both claim port 7770. The port gate in
-the handler now covers exactly that race. See [[frank-large-ik-uncensored]].
-
-The `amd_iommu=off` kernel argument that came with this change is separate and
-reverts on its own — see [[dankclaw-iommu-translation-off]].
-
-## Citations
-
- [peonist-ai/halogen-flash-server](https://github.com/peonist-ai/halogen-flash-server) — README, `docs/FLAGS.md`, `deploy/entrypoint.sh`, `docker-compose.yml`, CHANGELOG (v0.5.8)
- [peonist-ai/halogen-qwen3.8-flash-next](https://huggingface.co/peonist-ai/halogen-qwen3.8-flash-next) — weights and file inventory
- [Qwen/Qwen3.8-Flash-Next](https://huggingface.co/Qwen/Qwen3.8-Flash-Next) — base model
@@ -1,3 +1,13 @@
+---
+type: reference
+description: "**Status (historical, 2026-09-13):** was to deploy on Dankclaw."
+tags:
+  - llm
+  - rocm
+  - frank
+timestamp: 2026-09-15
+---
+
 # frank-large: kingjones Qwen3.8-27B-ROCmFPX-Q8_0-AGENT

 > **SUPERSEDED 2026-09-15.** The Q8_0_ROCMFPX-AGENT checkpoint (with the
@@ -15,12 +15,14 @@ timestamp: 2026-08-24

 # kingjones Qwen3.8-27B ROCmFP4 STRIX MTP on Dankclaw

-> **Active again 2026-09-01.** The kingjones ROCmFP4 STRIX MTP payload is
-> restored as `frank-large` after the Q8_0 AGENT experiment. The Q8_0
-> model was retired because it crashed before serving. The Q6 AGENT model,
-> standalone MTP draft, Q8 projector, and PromptForge runtime are now active.
+> **RETIRED 2026-08-29.** This payload was `frank-large` from 2026-08-25
+> until the Q8_0 AGENT experiment on 2026-08-29 (it was restored again
+> 2026-09-01 and served until 2026-08-30, when [[agentionai-fast-imatrix]]
+> took over). The Q8_0 model was retired because it crashed before
+> serving. Kept as the historical record of the Q6 AGENT model, standalone
+> MTP draft, Q8 projector, and PromptForge runtime.

-> This page documents the active model and its runbook.
+> This page documents a retired model and its runbook.

 `frank-large` (port 7770) serves
 `Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP` from `kingjones777` on Hugging
@@ -1,3 +1,11 @@
+---
+type: reference
+description: "`bailingmoe3-support` (commit `db4480bc8`, build b10411) at `/home/dank/llama.cpp-ling`."
+tags:
+  - llm
+timestamp: 2026-09-03
+---
+
 # Ling-3.0-flash Q6_K Trial on Dankclaw

 > **RETIRED 2026-08-17.** The `frank-large-ling` unit is disabled under the boot policy. At this page's retirement date `frank-large` ran the Qwen3.8-27B PP head (since superseded by the STRIX deployment — see [[kingjones-strix-rocmfp4]]), not the Qwen3.6-27B config described in the (obsolete) switch-back section below.
@@ -1,3 +1,13 @@
+---
+type: guide
+description: "This page was the plan for a vLLM-ROCm deployment of `jcbtc/Ling-3.0-Flash-CIRU-int4-Strix-native` (INT4 W4A16, ~62 GB) on"
+tags:
+  - llm
+  - vllm
+  - rocm
+timestamp: 2026-09-03
+---
+
 # Ling-3.0-flash on Dankclaw — vLLM-ROCm Deployment Plan (never shipped)

 This page was the plan for a vLLM-ROCm deployment of
@@ -1,3 +1,12 @@
+---
+type: reference
+description: "This page records the speculative-decoding benchmark history and the"
+tags:
+  - llm
+  - speculative
+timestamp: 2026-09-03
+---
+
 > **SUPERSEDED 2026-08-17.** The Frank redo retired this topology.
 > See [frank-redo-2026-08-17.md](frank-redo-2026-08-17.md) and
 > [deployment.md](deployment.md) for the active configuration.
@@ -1,3 +1,11 @@
+---
+type: reference
+description: "This page answers one question: **how do we make Qwen3.8-27B serve faster on"
+tags:
+  - llm
+timestamp: 2026-09-03
+---
+
 # Qwen3.8-27B on Strix Halo: Speed Without Lowering the Quant

 This page answers one question: **how do we make Qwen3.8-27B serve faster on
@@ -1,3 +1,12 @@
+---
+type: reference
+description: "vLLM v0.26.0 (latest stable, released July 25, 2026) supports DFlash speculative decoding with the `--spec-method dflash` and `--spec-model` flags."
+tags:
+  - llm
+  - vllm
+timestamp: 2026-08-28
+---
+
 # Research: vLLM Parameters for Specified Configuration

 ## Summary